Compare commits

..
878 Commits
Author SHA1 Message Date
antirez 063e0a8810 Redis 2.9.102 (3.0.0 Release Candidate 2). 2015-01-13 16:35:36 +01:00
antirez df1a7fc4fe Cluster: fetch my IP even if msg is not MEET for the first time.
In order to avoid that misconfigured cluster nodes at some time may
force an IP update on other nodes, it is required that nodes update
their own address only on MEET messages. However it does not make sense
to do this the first time a node is contacted and yet does not have an
IP, we just risk that myself->ip remains not assigned if there are
messages lost or cluster creation procedures that don't make sure
everybody is targeted by at least one incoming MEET message.

Also fix the logging of the IP switch avoiding the :-1 tail.
2015-01-13 16:23:48 +01:00
antirez 45e2a26ded Cluster: clusterMsgDataGossip structure, explict padding + minor stuff.
Also explicitly set version to 0, add a protocol version define, improve
comments in the gossip structure.

Note that the structure layout is the same after the change, we are just
making the padding explicit with an additional not used 16 bits field.
So this commit is still able to talk with the previous versions of
cluster nodes.
2015-01-13 16:23:44 +01:00
antirez 799a3ccac1 Suppress valgrind error about write sending uninitialized data.
Valgrind checks that the buffers we transfer via syscalls are all
composed of bytes actually initialized. This is useful, it makes we able
to avoid leaking informations in non initialized parts fo messages
transferred to other hosts. This commit fixes one of such issues.
2015-01-13 16:23:40 +01:00
antirez 1584c7a31b Cluster: initialize mf_end.
Can't be initialized by resetManualFailover() since it's actual state
the function uses, so we need to initialize it at startup time. Not
really a bug in practical terms, but showed up into valgrind and is not
technically correct anyway.
2015-01-12 15:40:30 +01:00
antirez 8be8780d4a Add "-lrt" in Makefile for Solaris.
This fix is from @NanXiao, however I was not able to retain authorship
because the Pull Request original repository was removed.
2015-01-09 11:54:50 +01:00
antirez f2348a2b7f Check for __sun macro in solarisfixes.h, not in includers. 2015-01-09 11:22:14 +01:00
antirez 7c283c5218 Cluster test: also write from Lua script in resharding test. 2015-01-09 11:06:38 +01:00
antirez 6f3553a2fe Prevent Lua scripts from violating Redis Cluster keyspace access rules.
Before this commit scripts were able to access / create keys outside the
set of hash slots served by the local node.
2015-01-09 10:39:05 +01:00
Matt Stancliffandantirez 67cc0d0851 Remove end of line whitespace from redis-trib 2015-01-08 19:46:02 +01:00
Matt Stancliffandantirez d1b607aa66 Fix redis-trib cluster create
Under certain conditions the node list wasn't being fully populated
and 'create' would fail trying to call methods on nil objects.
2015-01-08 19:46:02 +01:00
Matt Stancliffandantirez 6742dab413 Improve redis-trib replica assignment
This tiny bit of code has gone through so many revisions.  Hopefully
it's more correct now.

Fixes #2204
2015-01-08 19:46:02 +01:00
antirez 2247189e3e README section about make distclean reworded / extended. 2015-01-08 16:35:39 +01:00
Jungtaek Limandantirez eff9e987d3 Explain make distclean which seems not well known 2015-01-08 16:35:34 +01:00
antirez 198fb1621a Advertise Redis Cluster as experimental in redis.conf. 2015-01-08 14:41:26 +01:00
antirez 1e8f1577c5 INFO loading stats: three fixes.
1. Server unxtime may remain not updated while loading AOF, so ETA is
not updated correctly.

2. Number of processed byte was not initialized.

3. Possible division by zero condition (likely cause of issue #1932).
2014-12-23 14:54:31 +01:00
Alon Diamantandantirez aad0c512cc Fixed memory leaks in rdbSaveToSlavesSockets() 2014-12-22 11:01:17 +01:00
antirez 9cb99f547d Fix adjustOpenFilesLimit() logging to match real state.
Fixes issue #2225.
2014-12-19 10:05:07 +01:00
Matt Stancliffandantirez 8bce654246 Cluster: Notify user on accept error
If we woke up to accept a connection, but we can't
accept it, inform the user of the error going on
with their networking.

(The previous message was the same for success or error!)
2014-12-17 17:48:55 +01:00
Rhommel Lamasandantirez 6aba87be82 Add redis-sentinel symlink to a relative path 2014-12-17 12:27:41 +01:00
antirez 86213b4e03 Fix comment in clusterHandleSlaveFailover(). 2014-12-16 15:03:23 +01:00
antirez 2c6dc9f15f Make sure buffer is enough in clusterSendPing(). 2014-12-15 10:18:29 +01:00
antirez 722faed62a Don't log admin commands in MONITOR.
Otherwise there are security risks, especially when providing Redis as a
service, the user may "sniff" for admin commands renamed to an
unguessable string via rename-command in redis.conf.
2014-12-13 08:58:34 +01:00
antirez cc7b9171ca List of commands flagged as admin commands modified.
The old list did not made much sense... and the flag is currently not
used at all, so no side effects.
2014-12-13 08:58:34 +01:00
Rhommel Lamasandantirez 31bf7976b5 Add symlink to redis-sentinel during make install 2014-12-12 22:59:28 +01:00
antirez a419537800 Lua cmsgpack lib updated to latest version.
It fixes a bad bug that crashes the server in certain conditions
as shown in issue #2210.
2014-12-12 22:57:07 +01:00
antirez 814cd40215 SORT: Don't sort Set elements if not needed.
Related to #2094.
2014-12-11 15:58:29 +01:00
Matt Stancliffandantirez 77bbc0dfad Fix zero-ordering SORT when called against lists
People mostly use SORT against lists, but our prior
behavior was pretending lists were an unordered bag
requiring a forced-sort when no sort was requested.

We can just use the native list ordering to ensure
consistency across replicaion and scripting calls.

Closes #2079
Closes #545 (again)
2014-12-11 15:58:29 +01:00
zhanghaileiandantirez e1524f5010 FIXED redis-benchmark's idle mode.With idle mode shouldn't create write event 2014-12-11 15:21:42 +01:00
Ben Dowlingandantirez 94d9482b61 Update redis_init_script.tpl
status command currently reports success when redis has crashed and the pid file still exists. Changing to check the actual process is running.
2014-12-11 15:21:28 +01:00
Nan Xiaoandantirez cc14d7f51d Fix function prototype in redis-cli.c.
Fix function prototype in redis-cli.c.
2014-12-11 15:09:20 +01:00
Matt Stancliffandantirez c13d787d13 Fix redis-trib import SCAN call 2014-12-11 15:07:49 +01:00
h0x91bandantirez b3349204b8 Reconnect redis-cli when cluster return "moved"
if redis works in cluster-mode and redis-cli was run with argv, reconnect if needs.
    example:
    ./redis-cli set foo bar

    if return is MOVED redis-cli just do nothing.
2014-12-11 15:07:49 +01:00
Mariano Pérez Rodríguezandantirez 0f1107432a Typo fixes in redis.conf
Fix two typos in redis.conf:

- "trnasfers" --> "transfers"
- "enalbed" --> "enabled"
2014-12-11 15:07:49 +01:00
Sun Heandantirez 2f52a1191f zipmap.c: update comments above 2014-12-09 16:01:11 +01:00
Deepak Vermaandantirez c86b031572 replaced // comments #2150 2014-12-09 15:26:22 +01:00
azure provisioned userandantirez 0aac7ad8ff redis-benchmark AUTH command to be discarded after the first send #2150 2014-12-09 15:26:22 +01:00
Sun Heandantirez c5b449ea55 sds.c: Correct two spelling mistakes in comments 2014-12-09 15:14:09 +01:00
Sun Heandantirez 559cefce13 sds.c/sdscatvprintf: set va_end to finish va_list cpy 2014-12-09 15:14:09 +01:00
Sun Heandantirez 3959fa2c02 sds.c: Correct some comments 2014-12-09 15:14:09 +01:00
antirez 24ae9a4a82 redis-benchmark: default num of requests is now 100000.
10000 completes in a too short time and may easily provide unreliable
figures because of tiny duration.
2014-12-09 13:02:10 +01:00
Matthias Petschickandantirez 97d31c63e8 fix benchmark memleak in loop mode 2014-12-09 13:02:10 +01:00
antirez 73996c8615 Cluster PUBLISH message: fix totlen count.
bulk_data field size was not removed from the count. It is not possible
to declare it simply as 'char bulk_data[]' since the structure is nested
into another structure.
2014-12-09 13:01:47 +01:00
Serghei Iakovlevandantirez e4d4eff15e Update whatisdoing.sh
Improved getting pid
2014-12-09 13:01:25 +01:00
antirez 2533e2994e Include stropts only if __sun is defined. 2014-12-09 12:59:36 +01:00
Jan-Erik Redigerandantirez 52bc358c11 Fix implicit declaration of ioctl on Solaris 2014-12-09 12:57:56 +01:00
Sun Heandantirez aa130caeae bitops.c/bitopCommand: skip short minlen for FAST PATH 2014-12-09 12:46:43 +01:00
Sun Heandantirez 5fc55a1afa bitops.c/redisPopcount: little optimization in loop 2014-12-09 12:46:43 +01:00
Johan Bergströmandantirez 5ab2c31189 Silence _BSD_SOURCE warnings in glibc 2.20 and forward
See https://sourceware.org/glibc/wiki/Release/2.20#Packaging_Changes
2014-12-09 12:24:28 +01:00
antirez cfe04da17d Mark whatisdoing.sh as deprecated in top-comment. 2014-12-09 12:18:39 +01:00
Serghei Iakovlevandantirez 81eed2f5f6 getting pid fixes
```sh
$ ~ pidof redis-server
# nothing

$ ~ ps aux | grep [r]edis
redis      593  0.0  0.0  36900  5564 ?        Ssl  Dec02   1:37 /usr/bin/redis-server 127.0.0.1:6379
klay     15927  0.0  0.0  16772  6068 pts/6    S+   13:58   0:00 redis-cli

$ ~ uname -a
Linux edge 3.17.4-1-ARCH #1 SMP PREEMPT Fri Nov 21 21:14:42 CET 2014 x86_64 GNU/Linux
```
2014-12-09 12:09:34 +01:00
Sun Heandantirez bafd82167b sparkline.c: AddSample skip Empty label 2014-12-09 11:28:50 +01:00
Sun Heandantirez 6370afc281 sparkline.c: mov label-ini into the AddSample Function 2014-12-09 11:28:50 +01:00
Jan-Erik Redigerandantirez c8305accc3 Only ignore sigpipe in interactive mode
This allows shell pipes to correctly end redis-cli.

Ref #2066
2014-12-09 09:01:08 +01:00
antirez edd7ab325a Simplify lua_cmsgpack macro and fix build on old Linux distros.
Thanks to @badboy for the help in checking the build after the fix.
2014-12-05 10:52:54 +01:00
antirez e7d6f51dcb Check that tcp-backlog is matched by /proc/sys/net/core/somaxconn. 2014-12-04 11:07:52 +01:00
antirez 8a67e5d456 Network bandwidth tracking + refactoring.
Track bandwidth used by clients and replication (but diskless
replication is not tracked since the actual transfer happens in the
child process).

This includes a refactoring that makes tracking new instantaneous
metrics simpler.
2014-12-03 12:17:15 +01:00
antirez 8fb0d455d4 Don't show the ASCII logo if syslog is enabled.
Closes issue #1935.
2014-12-03 10:52:07 +01:00
antirez 237b37a727 Regression test for issue #2175. 2014-12-03 10:41:03 +01:00
antirez b42e00065f Handle infinite explicitly in createStringObjectFromLongLong(). 2014-12-03 10:41:03 +01:00
antirez b6b7bc0bff Use exp format and more precision output for ZSCAN.
Ref: issue #2175
2014-12-03 10:41:03 +01:00
antirez 6a0b1b5b43 Over 80 chars comment trimmed in pfcountCommand(). 2014-12-02 17:03:20 +01:00
antirez 69efb59ac7 Mark PFCOUNT as read-only, even if not true.
PFCOUNT is technically speaking a write command, since the cached value
of the HLL is exposed in the data structure (design error, mea culpa), and
can be modified by PFCOUNT.

However if we flag PFCOUNT as "w", read only slaves can't execute the
command, which is a problem since there are environments where slaves
are used to scale PFCOUNT reads.

Nor it is possible to just prevent PFCOUNT to modify the data structure
in slaves, since without the cache we lose too much efficiency.

So while this commit allows slaves to create a temporary inconsistency
(the strings representing the HLLs in the master and slave can be
different in certain moments) it is actually harmless.

In the long run this should be probably fixed by turning the HLL into a
more opaque representation, for example by storing the cached value in
the part of the string which is not exposed (this should be possible
with SDS strings).
2014-12-02 17:01:53 +01:00
antirez 2e4d043c20 Mark diskless replication as experimental in redis.conf. 2014-12-02 14:13:42 +01:00
antirez bed4108b72 Test: wait for actual startup in start_server.
start_server now uses return value from Tcl exec to get the server pid,
however this introduces errors that depend from timing: a lot of the
testing code base assumed the server to be actually up and running when
server_start returns.

So the old code that waits to see the pid in the log file was restored.
2014-11-28 12:58:11 +01:00
antirez edd1ea2a68 Test: try to cleanup still running Redis instances on exit.
It's hard to run the Redis test continuously if it leaks processes on
exceptions / errors.
2014-11-28 12:58:11 +01:00
antirez 4d316967e8 Test framework: exit on timeout with report.
There was no sane way to detect tests that may never end because of
Redis bugs or tests bugs.
2014-11-28 12:58:11 +01:00
antirez 4c1b3add7f Fix DEBUG OBJECT lru field to report seconds.
Because of (not so) recent Redis changes, now the LRU internally
reported unit is milliseconds, not seconds, but the DEBUG OBJECT output
was still claiming seconds while providing milliseconds.
However OBJECT IDLETIME was working as expected, which is the correct
API to use.
2014-11-26 16:40:04 +01:00
antirez 1490809f1f Document redis-cli --stat in --help output. 2014-11-25 18:25:55 +01:00
Matt Stancliffandantirez b1c426ef73 Fix lua-cmsgpack 64 bit integer on 32 bit platform
This syncs lua-cmsgpack with the mattsta/lua-cmsgpack upstream.

Fixes #2161
2014-11-25 15:56:55 +01:00
antirez f2876f6cf7 Avoid valgrind memory leak false positive in processInlineBuffer().
zmalloc(0) cauesd to actually trigger a non-zero allocation since with
standard libc malloc we have our own zmalloc header for memory tracking,
but at the same time the returned pointer is at the end of the block and
not in the middle. This triggers a false positive when testing with
valgrind.

When the inline protocol args count is 0, we now avoid reallocating
c->argv, preventing the issue to happen.
2014-11-25 14:51:00 +01:00
antirez 3765a5d9ba Attempt to prevent false positives in replication test. 2014-11-24 11:55:12 +01:00
antirez 9466f3b326 scripting.tcl tests order fixed to match unstable branch. 2014-11-14 17:23:40 +01:00
antirez 4861457d1a lua_cjson.c Lua includes: angled -> quoted. 2014-11-14 17:19:58 +01:00
antirez 6b53b32ba2 Fix non-linux builds error introduced with THP checks. 2014-11-14 17:19:58 +01:00
Matt Stancliffandantirez 8934c48075 Lua: add cmsgpack scripting tests
Basically: test to make sure we can load cmsgpack
and do some sanity checks to make sure pack/unpack works
properly.  We also have a bonus test for circular encoding
and decoding because I was curious how it worked.
2014-11-14 17:19:58 +01:00
Matt Stancliffandantirez 8c3bcbfcc4 Lua: upgrade cmsgpack to 0.4.0
Main reasons for upgrade:
  - Remove a warning when building Redis
  - Add multi pack/unpack
  - Improve memory usage and use Lua allocator properly
  - Fix some edge case encoding/decoding bugs
2014-11-14 17:19:58 +01:00
Matt Stancliffandantirez aa1a0d6f17 Lua: remove new warning added by cjson header
clang doesn't like "extern inline" when no definition
is given right away.
2014-11-14 17:19:58 +01:00
Matt Stancliffandantirez 546db94c09 Lua: Use Redis solaris compatability for cjson too
cjson calls isinf, but some Solaris versions don't have isinf
even with the attempted fix we have in deps/Makefile.

We can harmlessly include the Redis solarisfixes.h header to
give cjson proper isinf.

Note: cjson has a compile-time setting for using their own defined
isinf, but the Redis definition in solarisfixes.h is more complete.

Fixes antirez#1620
2014-11-14 17:19:58 +01:00
Matt Stancliffandantirez 03ca2231f0 Lua: Upgrade cjson to 2.1.0 (2012-03-01)
The new cjson has some improvements over our current version including
increased platform compatability, a new resource limit to restrict
decode depth, and better invalid number handling.

One minor change was required to deps/Makefile because this version
of cjson doesn't export itself globally, so we added a quick little
define of -DENABLE_CJSON_GLOBAL.

cjson now has an optional higher performing float parsing interface,
but we are not including it (g_fmt.c, dtoa.c) because it requires
endianness declaration during compile time.

This commit is exactly lua_cjson.c from 2.1.0 with one minor
change of altering the two Lua includes for local search
instead of system-wide importing.
2014-11-14 17:19:58 +01:00
Matt Stancliffandantirez 330ebea470 Lua: add cjson scripting test
Two simple decode tests added mainly to check that
the 'cjson' global gets registered and is usable.
2014-11-14 17:19:52 +01:00
Matt Stancliffandantirez d071e5feab Lua: Add bitop
A few people have written custom C commands because bit
manipulation isn't exposed through Lua.  Let's give
them Mike Pall's bitop.

This adds bitop 1.0.2 (2012-05-08) from http://bitop.luajit.org/

bitop is imported as "bit" into the global namespace.

New Lua commands: bit.tobit, bit.tohex, bit.bnot, bit.band, bit.bor, bit.bxor,
bit.lshift, bit.rshift, bit.arshift, bit.rol, bit.ror, bit.bswap

Verification of working (the asserts would abort on error, so (nil) is correct):
127.0.0.1:6379> eval "assert(bit.tobit(1) == 1); assert(bit.band(1) == 1); assert(bit.bxor(1,2) == 3); assert(bit.bor(1,2,4,8,16,32,64,128) == 255)" 0
(nil)
127.0.0.1:6379> eval 'assert(0x7fffffff == 2147483647, "broken hex literals"); assert(0xffffffff == -1 or 0xffffffff == 2^32-1, "broken hex literals"); assert(tostring(-1) == "-1", "broken tostring()"); assert(tostring(0xffffffff) == "-1" or tostring(0xffffffff) == "4294967295", "broken tostring()")' 0
(nil)

Tests also integrated into the scripting tests and can be run with:
./runtest --single unit/scripting

Tests are excerpted from `bittest.lua` included in the bitop distribution.
2014-11-14 17:18:43 +01:00
antirez c623f0ac50 THP detection for LATENCY DOCTOR. 2014-11-12 11:17:44 +01:00
antirez 868f40c165 Check THP support at startup and warn about it. 2014-11-12 11:17:44 +01:00
antirez e293cfcffb THP detection / reporting functions added. 2014-11-12 11:17:44 +01:00
antirez 90911905f8 Diskless SYNC: fix RDB EOF detection.
RDB EOF detection was relying on the final part of the RDB transfer to
be a magic 40 bytes EOF marker. However as the slave is put online
immediately, and because of sockets timeouts, the replication stream is
actually contiguous with the RDB file.

This means that to detect the EOF correctly we should either:

1) Scan all the stream searching for the mark. Sucks CPU-wise.
2) Start to send the replication stream only after an acknowledge.
3) Implement a proper chunked encoding.

For now solution "2" was picked, so the master does not start to send
ASAP the stream of commands in the case of diskless replication. We wait
for the first REPLCONF ACK command from the slave, that certifies us
that the slave correctly loaded the RDB file and is ready to get more
data.
2014-11-12 09:51:29 +01:00
antirez 78ac9b7f11 Disconnect timedout slave: regression introduced with diskless repl. 2014-11-12 09:51:29 +01:00
Matt Stancliffandantirez 75e68625f1 Parse cluster state file in IPv6 compatible way
We need to pick the port based on the _last_ colon, not the first one.
2014-10-31 10:39:33 +01:00
Matt Stancliffandantirez bb0ff49ea3 Fix redis-trib.rb IP:Port disassembly for IPv6
IP format is now any of:
  - 127.0.0.1:6379
  - ::1:6379
2014-10-31 10:39:33 +01:00
Matt Stancliffandantirez f1a6f78024 Networking: add more outbound IP binding fixes
Same as the original bind fixes (we just missed these the
first time around).

This helps Redis not automatically send
connections from the first IP on an interface if we are bound
to a specific IP address (e.g. with multiple IP aliases on one
interface, you want to send from _your_ IP, not from the first IP
on the interface).
2014-10-31 10:02:42 +01:00
Matt Stancliffandantirez e8f7899ae2 redis-cli: ignore SIGPIPE network errors
Closes #2066
2014-10-30 12:02:28 +01:00
antirez 2480315f2a redis-cli: add missing newline in error message. 2014-10-29 14:27:15 +01:00
antirez f0124c1d90 Fix DEBUG POPULATE warning for lack of casting. 2014-10-29 14:27:15 +01:00
antirez 6231c42ed6 Diskless replication: missing listRewind() added.
This caused BGSAVE to be triggered a second time without any need when
we switch from socket to disk target via the command

    CONFIG SET repl-diskless-sync no

and there is already a slave waiting for the BGSAVE to start.
Also comments clarified about what is happening.
2014-10-29 14:26:48 +01:00
antirez 18698b5269 Log slave ip:port in more log messages. 2014-10-29 14:26:48 +01:00
antirez f687976601 Use new slave name function for diskless repl reporting. 2014-10-29 14:26:48 +01:00
antirez 2fd3fc7f01 Added a function to get slave name for logs. 2014-10-29 14:26:48 +01:00
antirez e1c0a25ca4 Diskless replication: log BGSAVE delay only when it is non-zero. 2014-10-29 14:26:48 +01:00
antirez fc486a585c Document repl-diskless-sync-delay in redis.conf. 2014-10-29 14:26:48 +01:00
antirez 6bbeb8c535 Diskless sync delay is now configurable. 2014-10-29 14:26:48 +01:00
antirez 2882f819f4 Remove duplicated log message about starting BGSAVE. 2014-10-29 14:26:48 +01:00
antirez 52f8f4a065 Diskless replication documented inside example redis.conf. 2014-10-29 14:26:48 +01:00
antirez ec3e239794 Diskless replication tested with the multiple slaves consistency test. 2014-10-29 14:26:48 +01:00
antirez 575c80041c Diskless replication: child -> parent communication improved.
Child now reports full info to the parent including IDs of slaves in
failure state and exit code.
2014-10-29 14:26:48 +01:00
antirez b3550808ea Translate rio fdset target EWOULDBLOCK error into ETIMEDOUT.
EWOULDBLOCK with the fdset rio target is returned when we try to write
but the send timeout socket option triggered an error. Better to
translate the error in something the user can actually recognize as a
timeout.
2014-10-29 14:26:48 +01:00
antirez 0e3c7e3223 Diskless replication: set / reset socket send timeout.
We need to avoid that a child -> slaves transfer can continue forever.
We use the same timeout used as global replication timeout, which is
documented to also affect I/O operations during bulk transfers.
2014-10-29 14:26:48 +01:00
antirez a902919ee3 anet.c: new API anetSendTimeout(). 2014-10-29 14:26:48 +01:00
antirez cd714ba510 Diskless replication: less debugging printfs around. 2014-10-29 14:26:48 +01:00
antirez 5012b0f1e9 rio.c fdset write() method fixed: wrong type for return value. 2014-10-29 14:26:48 +01:00
antirez 5fb51545ce rio fdset target: handle short writes.
While the socket is set in blocking mode, we still can get short writes
writing to a socket.
2014-10-29 14:26:48 +01:00
antirez 5291f7b791 anet.c: API to set sockets back to blocking mode. 2014-10-29 14:26:48 +01:00
antirez 55e9a71317 Diskless replication: rio fdset target new supports buffering.
To perform a socket write() for each RDB rio API write call was
extremely unefficient, so now rio has minimal buffering capabilities.
Writes are accumulated into a buffer and only when a given limit is
reacehd are actually wrote to the N slaves FDs.

Trivia: rio lacked support for buffering since our targets were:

1) Memory buffers.
2) C standard I/O.

Both were buffered already.
2014-10-29 14:26:48 +01:00
antirez 34b0158772 Diskless replication: Various fixes to backgroundSaveDoneHandlerSocket() 2014-10-29 14:26:48 +01:00
antirez ff6f015b2b Diskless replication: don't send "\n" pings to slaves.
This is useful for normal replication in order to refresh the slave
when we are persisting on disk, but for diskless replication the
child is already receiving data while in WAIT_BGSAVE_END state.
2014-10-29 14:26:47 +01:00
antirez 9512211548 Diskless replication: remove 40 bytes EOF mark from end of RDB file. 2014-10-29 14:26:47 +01:00
antirez dbfdd9917b Diskless replication: swap inverted branches to compute read len. 2014-10-29 14:26:47 +01:00
antirez d1b54c4607 Diskless replication: don't enter the read-payload branch forever. 2014-10-29 14:26:47 +01:00
antirez 3b6706848b syncReadLine(): actually enforce buffer size limits. 2014-10-29 14:26:47 +01:00
antirez b699c8ea63 Diskless replication: EOF:<mark> streaming support slave side. 2014-10-29 14:26:47 +01:00
antirez 1472c68212 Diskless replication: redis.conf and CONFIG SET/GET support. 2014-10-29 14:26:47 +01:00
antirez 78adcfe139 Diskless replication: trigger a BGSAVE after a config change.
If we turn from diskless to disk-based replication via CONFIG SET, we
need a way to start a BGSAVE if there are slaves alerady waiting for a
BGSAVE to start. Normally with disk-based replication we do it as soon
as the previous child exits, but when there is a configuration change
via CONFIG SET, we may have slaves in WAIT_BGSAVE_START state without
an RDB background process currently active.
2014-10-29 14:26:47 +01:00
antirez 4d5f4684c1 Diskless replication flag renamed repl_diskless -> repl_diskless_sync. 2014-10-29 14:26:47 +01:00
antirez 989e2ae2e8 Diskless replication: trigger diskless RDB transfer if needed. 2014-10-29 14:26:47 +01:00
antirez 94e9dd6b56 Diskless replication: handle putting the slave online. 2014-10-29 14:26:47 +01:00
antirez 46a0509ef0 Diskless replication: read report from child. 2014-10-29 14:26:47 +01:00
antirez cdd975587c Diskless replication: child writes report to parent. 2014-10-29 14:26:47 +01:00
antirez 9563942f4f rio.c fdset target: tolerate (and report) a subset of FDs in error.
Fdset target is used when we want to write an RDB file directly to
slave's sockets. In this setup as long as there is a single slave that
is still receiving our payload, we want to continue sennding instead of
aborting. However rio calls should abort of no FD is ok.

Also we want the errors reported so that we can signal the parent who is
ok and who is broken, so there is a new set integers with the state of
each fd. Zero is ok, non-zero is the errno of the failure, if avaialble,
or a generic EIO.
2014-10-29 14:26:47 +01:00
antirez eaeeb1216d Diskless replication: parent-child pipe and a few TODOs. 2014-10-29 14:26:47 +01:00
antirez 30231ea005 Diskless replication: RDB -> slaves transfer draft implementation. 2014-10-29 14:26:47 +01:00
antirez 0d40744918 rio.c: draft implementation of fdset target implemented. 2014-10-29 14:26:47 +01:00
antirez e477832257 rio.c refactoring before adding a new target. 2014-10-29 14:26:47 +01:00
antirez a71beaa0cb Add some comments in syncCommand() to clarify RDB target. 2014-10-29 14:26:47 +01:00
antirez 052da718ad Define different types of RDB childs.
We need to remember what is the saving strategy of the current RDB child
process, since the configuration may be modified at runtime via CONFIG
SET and still we'll need to understand, when the child exists, what to
do and for what goal the process was initiated: to create an RDB file
on disk or to write stuff directly to slave's sockets.
2014-10-29 14:26:47 +01:00
antirez b919d952d5 RDB file creation refactored to target non-disk target. 2014-10-29 14:26:47 +01:00
antirez d15ffcd65c Redis 2.9.101 (3.0.0 Release Candidate 1). 2014-10-09 11:06:02 +02:00
antirez c6226f262f Cluster: process gossip section only for known nodes.
With the exception of nodes sending MEET packets: we have to trust them
since they can send us MEET packets only when the cluster is initially
created or because sysadmin manual action.
2014-10-09 10:52:28 +02:00
antirez 419eb18505 Cluster: fix logic to detect we are among a minority.
In the cluster evaluation function we are supposed to set the cluster
state as "fail" if we are among a minority, however the code was not
detecting to be into a minority partition if exactly half the masters
were reachable, which is a minority.
2014-10-09 10:52:28 +02:00
antirez 07963b21e3 Cluster test: helpers/onlydots.tcl: detect EOF and exit. 2014-10-08 10:17:23 +02:00
antirez 079662fca9 Cluster test: less console-spammy resharding test. 2014-10-08 10:17:23 +02:00
antirez 6da71d65ff Cluster: nodes.conf added to git ignore list. 2014-10-08 09:12:43 +02:00
antirez 9a867b686a Cluster: more chatty slaves when failover is stalled. 2014-10-08 09:12:43 +02:00
antirez eedfbdec1d Linenoise README updated to match source code. 2014-10-06 10:07:02 +02:00
antirez cf2815b60e Test: check that INCR object sharing works as expected. 2014-10-06 10:07:02 +02:00
antirez b97644881c INCR: Modify incremented object in-place when possible.
However we don't try to do this if the integer is already inside a range
representable with a shared integer.

The performance gain appears to be around ~15% in micro benchmarks,
however in the long run this also helps to improve locality, so should
have more, hard to measure, benefits.
2014-10-06 10:07:02 +02:00
Miguel Parramonandantirez 6e6749c83d redis.conf small grammar fix
😄

Closes #2034
2014-10-06 10:07:02 +02:00
Ezequiel Lovelleandantirez 188bb04282 Fix typo
Closes #2029
2014-10-06 10:07:02 +02:00
Aniruddh Chaturvediandantirez 9a38820f1e Fix typo in unit test
Closes #2005
2014-10-06 10:07:02 +02:00
Gregory Petrosyanandantirez 2393adda02 Fix typos in comments
Closes #2002
2014-10-06 10:07:02 +02:00
Matt Stancliffandantirez 709ee6e818 Add missing 'by' 2014-10-06 10:07:02 +02:00
Juarez Bochiandantirez c2644c8a35 Fix typo in scripting.c comment
Closes #1960
2014-10-06 10:07:02 +02:00
T.J. Schuckandantirez f30136f206 Fix typo radis -> redis
Closes #1938
2014-10-06 10:07:02 +02:00
Matt Stancliffandantirez eb19819d93 Cleanup wording of dictScan() comment
Some language in the comment was difficult
to understand, so this commit: clarifies wording, removes
unnecessary words, and relocates some dependent clauses
closer to what they actually describe.

I also tried to break up longer chains of thought
(if X, then Y, and Q, and also F, so obviously M)
into more manageable chunks for ease of understanding.
2014-10-06 10:07:02 +02:00
Xiaojie Zhangandantirez d66a3831b4 Fix comment inconsistencies in ziplist.c
Closes #1523
2014-10-06 10:07:02 +02:00
Jan-Erik Redigerandantirez 213e819293 Fix sentinel.conf typo
Closes #1739
2014-10-06 10:07:02 +02:00
Matt Stancliffandantirez a19cdc5138 Remove trailing spaces from tests 2014-10-06 10:07:02 +02:00
Matt Stancliffandantirez f4c4a0fc33 Fix spelling in some test cases 2014-10-06 10:07:01 +02:00
Matt Stancliffandantirez 47fde270b6 Spell software correctly 2014-10-06 10:07:01 +02:00
Matt Stancliffandantirez bd62c95200 Clean up text throughout project
- Remove trailing newlines from redis.conf
  - Fix comment misspelling
  - Clarifies zipEncodeLength usage and a C API mention (#1243, #1242)
  - Fix cluster typos (inspired by @papanikge #1507)
  - Fix rewite -> rewrite in a few places (inspired by #682)

Closes #1243, #1242, #1507
2014-10-06 10:07:01 +02:00
Michael Parkerandantirez c51baf646a Fix hash table size in comment for dictScan
Closes #1351
2014-10-06 10:07:01 +02:00
Ted Nymanandantirez 7e055d4e9c Fix on-place -> in-place
Closes #1373
2014-10-06 10:07:01 +02:00
xuxiangandantirez 189c39607a Fix typo in redis.h
Closes #1386
2014-10-06 10:07:01 +02:00
Benandantirez c4aa0e34d0 Grammar and typo fixes in redis.conf
Closes #1441
2014-10-06 10:07:01 +02:00
Aaron Rutkovskyandantirez cc4162ff44 Fix typos
Closes #1513
2014-10-06 10:07:01 +02:00
Jan-Erik Redigerandantirez 01bdf82cf2 Fix typo: ad -> and
Closes #1537
2014-10-06 10:07:01 +02:00
Agis Anastasopoulosandantirez 76b3e827ee Fix typo
Closes #1544
2014-10-06 10:07:01 +02:00
Juarez Bochiandantirez 84846beaae Fix typo
Closes #1682
2014-10-06 10:06:43 +02:00
Jan-Erik Redigerandantirez ad7104724b Fix typo in redis.conf
Closes #1713
2014-10-06 10:06:43 +02:00
zionwuandantirez f5c20f382c Fix incorrect comments
error != success; and 0 != number of bytes written

Closes #1806
2014-10-06 10:06:43 +02:00
Jan-Erik Redigerandantirez 1a309ad8b1 Fix many small typos
Closes #1871
2014-10-06 10:06:43 +02:00
Matt Stancliffandantirez 3b544712a8 Update linenoise accessory files
The core linenoise code was being backported, but not
the README or example.  It's less confusing for users
if everything matches across directories.

Fix inspired by @thrig

Closes #1872
2014-10-06 10:06:43 +02:00
Lynnandantirez 3123a7b93b Remove surplus double quotes
Closes #1877
2014-10-06 10:06:43 +02:00
Manuel Meurerandantirez d26fe21007 Improve disable save comment in redis.conf
Closes #1897
2014-10-06 10:06:43 +02:00
antirez de03925230 redis-trib should not abort easily on connection issues. 2014-10-06 10:06:43 +02:00
antirez 0ad99ccf75 DEBUG POPULATE two args form implemented.
The old DEBUG POPULATE form for automatic creation of test keys is:

    DEBUG POPULATE <count>

Now an additional form is available:

    DEBUG POPULATE <count> <prefix>

When prefix is not specified, it defaults to "key", so the keys are
named incrementally from key:0 to key:<count-1>. Otherwise the specified
prefix is used instead of "key".

The command is useful in order to populate different Redis instances
with key names guaranteed to don't collide. There are other debugging
uses, for example it is possible to add additional N keys using a count
of N and a random prefix at every call.
2014-10-06 10:06:43 +02:00
antirez 256deeca85 Linenoise lib updated. 2014-09-22 14:56:34 +02:00
Matt Stancliffandantirez beafebb4f3 Fix hiredis getaddrinfo leak
Fixed in Redis by 1a5e5b6, but since that part of code
is largely copy/paste from Redis, the fix needs to be
ported over too.

Closes #2012
2014-09-19 14:22:00 +02:00
antirez 3f64424e3e anetPeerToString(): make unix sockets distinguishable.
Following the CLIENT LIST output format, we prefix the unix socket
address with a "/" so that it is different than an IPv4/6 address.
This makes parsing simpler.

Related to #2010.
2014-09-19 14:22:00 +02:00
antirez c62903bb75 anetPeerToString() refactoring and more explicit checks.
Related to PR #2010.
2014-09-19 14:22:00 +02:00
Yossi Gottliebandantirez 7b2d32d930 Unix domain sockets incorrectly listed as IPv6. 2014-09-19 14:22:00 +02:00
antirez 015cbf3015 Cluster: claim ping_sent time even if we can't connect.
This fixes a potential bug that was never observed in practice since
what happens is that the asynchronous connect returns ok (to fail later,
calling the handler) every time, so a ping is queued, and sent_ping
happens to always be populated.

Howver technically connect(2) with a non blocking socket may return an
error synchronously, so before this fix the code was not correct.
2014-09-19 14:22:00 +02:00
antirez d4c3c1248f Cluster: new option to work with partial slots coverage. 2014-09-19 14:22:00 +02:00
antirez fbaa438830 Better truncated AOF loading tests.
Now there are tests to write more data after loading a truncated AOF,
testing that the loaded data is correct, appending more, and testing
again.
2014-09-17 10:01:02 +02:00
antirez c698616bf8 Seek at the end of AOF after truncate call.
It is not clear if files open in append only mode will automatically fix
their offset after a truncate(2) operation. This commit makes sure that
we reposition the AOF file descriptor offset at the end of the file
after a truncated AOF is loaded and trimmed to the last valid command.
2014-09-17 10:01:02 +02:00
antirez ac76d56e6f On AOF end of file, truncate the AOF to last valid command.
Recently we introduced the ability to load truncated AOFs, but
unfortuantely the support was broken since the server, after loading the
truncated AOF, continues appending to the file that is corrupted at the
end. The problem is fixed only in the next AOF rewrite.

This commit fixes the issue by truncating the AOF to the last valid
opcode, and aborting if it is not possible to truncate the file
correctly.
2014-09-17 10:01:01 +02:00
antirez ef61ab6347 Don't propagate SAVE.
This is a general fix (check that dirty delta is positive) but actually
should have as the only effect fixing the SAVE propagation to
AOF and slaves.
2014-09-17 10:01:01 +02:00
antirez 364ee8322e Cluster test: unit 09, Pub/Sub across the cluster. 2014-09-17 10:01:01 +02:00
antirez 08c95d6ae2 Cluster test: unit 08, update messages. 2014-09-17 10:01:01 +02:00
antirez 60889a3691 Sentinel sentinelGetLeader() top comment improved. 2014-09-11 19:28:12 +02:00
antirez 7a41df06a6 Sentinel: fix computation of total number of votes.
The code to check the number of voters was never updated to follow the new
Sentinel specification, so the number of voters was computed using only
the set of Sentinels that provided a vote.

This means that there is a changing majority on partitions, even if
usually the issue is not triggered because of the configured quorum
check (what was broken was the other implicit check that requires anyway
half of the known sentinels to agree in order to start a failover).
2014-09-11 19:28:12 +02:00
antirez 96c7e94d1f luaRedisGenericCommand() cached argv handling simplified.
As discussed in issue #1945.
2014-09-10 15:47:00 +02:00
xiaostandantirez 0e54d970b3 Limit the *SCAN command dictScan iterations
*SCAN will cause redis server to hang for seconds
after millions of keys was deleted by SCAN/DEL pairs
2014-09-10 09:58:18 +02:00
antirez 290d33d68e Sentinel: don't set announce-ip if is empty. 2014-09-10 09:26:23 +02:00
antirez a3adeb284e Sentinel: clarify announce-ip/port options in sentinel.conf. 2014-09-10 09:26:23 +02:00
antirez e2eb5c8c83 Sentinel: announce ip/port changes + rewrite.
The original implementation was modified in order to allow to
selectively announce a different IP or port, and to rewrite the two
options in the config file after a rewrite.
2014-09-10 09:26:23 +02:00
Dara Kongandantirez 869a97b127 sentinel: Decouple bind address from address sent to other sentinels
There are instances such as EC2 where the bind address is private
(behind a NAT) and cannot be accessible from WAN.

https://groups.google.com/d/msg/redis-db/PVVvjO4nMd0/P3oWC036v3cJ
2014-09-10 09:26:23 +02:00
antirez 3d2c82b489 Tests for aof-load-truncated = yes. 2014-09-08 10:57:03 +02:00
antirez ea6082f1d5 AOF tests fixed turning aof-load-truncated to no.
When aof-load-truncated option was introduced, with a default of "yes",
the past behavior of the server to abort with trunncated AOF changed, so
we need to explicitly configure the tests to abort with truncated AOF
by setting the option to no.
2014-09-08 10:57:03 +02:00
antirez c65af33c01 Make aof-load-truncated option actually configurable. 2014-09-08 10:57:03 +02:00
antirez 24b3f78f21 Correct cleanup when aborting AOF loading.
Because of the new ability to start with a truncated AOF, we need
to correctly release all the memory on EOF error. Otherwise there is a
small leak, that is not really a problem, but causes a false positive in
the tests that detect memory leaks.
2014-09-08 10:57:03 +02:00
antirez ea1de5cdf9 AOF ability to load truncated files. 2014-09-08 10:57:03 +02:00
antirez 46ec968326 Test AOF format error detection. 2014-09-08 10:57:03 +02:00
antirez a4ec67290e AOF loading: split handling of format errors from unexpected EOF. 2014-09-08 10:57:03 +02:00
Matt Stancliffandantirez 5bb0da34ba Return empty string if GETRANGE of empty string
Previously, GETRANGE of a key containing nothing ("")
would allocate a large (size_t)-1 return value causing
crashes on 32bit builds when it tried to allocate the
4 GB return string.
2014-09-04 11:47:25 +02:00
Matt Stancliffandantirez 682914153c Increase size of range request in getrange
32 bit builds don't have a big enough long to capture
the same range as a 64 bit build.  If we use "long long"
we get proper size limits everywhere.

Also updates size of unsigned comparison to fit new size of `end`.

Fixes #1981
2014-09-04 11:46:59 +02:00
Matt Stancliffandantirez e713dc5a43 Sentinel: Abort Hello quicker if not connected
We can save a little work by aborting when we enter the function
if we're disconnected.
2014-09-01 16:35:06 +02:00
Paddy Byersandantirez 98bd223edd Add regression test for issue #1939 2014-09-01 10:42:29 +02:00
Paddy Byersandantirez b790e1d7e5 Store the length of the static argv when first allocated. 2014-09-01 10:42:29 +02:00
antirez a3f3f823d1 Fix dictRehash assert casting type.
Also related to #1929.
2014-08-26 10:41:03 +02:00
antirez 20f20b832a Cast to right type in dictNext().
This closes issue #1929, the other part was fixed in the context of issue
2014-08-26 10:41:03 +02:00
antirez 723a090861 Use long for rehash and iterator index in dict.h.
This allows to support datasets with more than 2 billion of keys
(possible in very large memory instances, this bug was actually
reported).

Closes issue #1814.
2014-08-26 10:41:03 +02:00
antirez dfa7ff1c4c Missing va_end also added in hiredis copy of sds.c. 2014-08-26 10:41:03 +02:00
Mariano Pérez Rodríguezandantirez d15c6e5e33 Missing assert removal from sdsIncrLen()
Companion for 8eeb1802ec, but dealing with hiredis.
2014-08-26 10:41:03 +02:00
antirez 85b20f1f70 Use modern typedef form in cluster.h. 2014-08-26 10:41:03 +02:00
antirez 6f98a9cd2c No longer useful assert removed from sdsIncrLen(). 2014-08-26 10:41:03 +02:00
Matt Stancliffandantirez 660a2f87a1 Fix redis-benchmark abort output 2014-08-26 10:41:03 +02:00
Matt Stancliffandantirez 601e06da8a Deny CLIENT command in scripts
We don't want scripts doing CLIENT SETNAME
or CLIENT KILL or CLIENT LIST or CLIENT PAUSE.

Originally reported by Chris Wj then proper
action inspired by Itamar Haber.

Reference: https://groups.google.com/forum/#!topic/redis-db/09B2EYwyVgk
2014-08-26 10:41:03 +02:00
Matt Stancliffandantirez 01e0112def redis-benchmark: Remove unused buffer
This just deletes old code that didn't get removed when
logic changed.  We were setting offsets that never
got read anywhere.

Since clients are now just cloned, we don't need to track
per-client buffer offsets anywhere because they are all
the same from the original client.
2014-08-26 10:41:03 +02:00
CharSyamandantirez 75ce984723 redis-benchmark: add auth option
Closes #1097
2014-08-26 10:41:03 +02:00
Matt Stancliffandantirez 29af582617 Rename two 'buf' vars to 'ip' for better clarity
Clearly ip[32] is wrong, but it's less clear that buf[32] was wrong
without further reading.
2014-08-26 10:41:03 +02:00
Eiichi Satoandantirez 4e062ec82e Sentinel: fix bufsize to support IPv6 address
Closes #1914
2014-08-26 10:41:03 +02:00
Jan-Erik Redigerandantirez ef2cb6c722 Use correct github url to find commands.json
Once this is merged:
  - merge the latest changes to commands.json in antirez/redis-doc
  - re-run: utils/generate-command-help.rb > src/help.h

Then we'll have nice and easy tab-completed help in redis-cli again.

Closes #1909
2014-08-26 10:41:03 +02:00
Mariano Pérez Rodríguezandantirez 4f29e859cf Stop tests from leaving a black background
Uses ANSI "default background" color code after closing tests
so any non-black terminals don't remain polluted.

Fixes #1649
Closes #1912
2014-08-26 10:41:03 +02:00
Matt Stancliffandantirez 21fb95b91e Sentinel: Reject config from STDIN
Sentinel needs to die with a more accurate error message
when attempted to open a config from STDIN.

See: https://groups.google.com/forum/#!topic/redis-db/sYx7VNMWaNM
2014-08-26 10:41:03 +02:00
Matt Stancliffandantirez 3e22384193 Cluster: Fix segfault if cluster config corrupt
This commit adds a size check after initial config
line parsing to make sure we have *at least* 8 arguments
per line.

Also, instead of asserting for cluster->myself, we just test
and error out normally (since the error does a hard exit anyway).

Closes #1597
2014-08-26 10:41:03 +02:00
Matt Stancliffandantirez 29ff27d430 Fix memory leak in cluster config parsing
The continue stop us from triggering the
free after the long line for loop, so add it
earlier.
2014-08-26 10:41:03 +02:00
Matt Stancliffandantirez 7df0636514 Remove unused global variable
It has an important name, but nothing uses it...
2014-08-26 10:41:03 +02:00
antirez d1e91c3fcc Fix invalid expire error for SET family commands. 2014-08-26 10:41:03 +02:00
Cong Dingandantirez 9d046af1e5 Remove unused function
Closes #878
2014-08-26 10:41:03 +02:00
David Palmandantirez 55672e06fd Extend range of bytesToHuman to TB and PB
Also adds a fallthrough case for when given
large values (like overflow numbers of 2^64 by mistake).

Closes #858
2014-08-26 10:41:03 +02:00
yoavandantirez 512cc03b55 Add error check for writing RDB checksum
Closes #857
2014-08-26 10:41:03 +02:00
Dov Murikandantirez 9da7141dc6 redis-cli: fix prompt after shutdown command
Fix redis-cli prompt to state "not connected" after a SHUTDOWN command
is sent.
2014-08-26 10:41:02 +02:00
xiaoyuandantirez b88fe07bc0 Clarify argument to dict macro
d is more clear because the type of argument is dict not dictht

Closes #513
2014-08-26 10:41:02 +02:00
Matt Stancliffandantirez d409b5acd3 Clarify existing slot wording on cluster start 2014-08-26 10:41:02 +02:00
antirez 125ac18944 Fix undefined behavior in ll2string().
The bug was found by @CAFxX, thanks!
See issue #1940.
2014-08-26 10:41:02 +02:00
antirez d34fade2da Remove warnings and improve integer sign correctness. 2014-08-26 10:41:02 +02:00
antirez 5d6c0b1be3 Add -W to compilation flags. 2014-08-26 10:41:02 +02:00
antirez 3566020acf Use unsigned integers in SDS header.
This raises the max string to 4GB without any downside.
2014-08-26 10:41:02 +02:00
Pieter Noordhuisandantirez 327831e59e va_copy must be matched by va_end
Hat tip to @rfuchs. See: https://github.com/redis/hiredis/pull/178.

Fixes #1187
2014-08-26 10:41:02 +02:00
charsyamandantirez 748a36cf04 Remove duplicate prototypes in redis.h
Also moves acceptHandler() to be near the other accept...() functions.

Closes #1105
2014-08-26 10:41:02 +02:00
NanXiaoandantirez f5e8d29990 Use resolv library in Solaris
For some Solaris flavours, the inet_aton in in resolv library.
Not linking this library will introduce link error.

Improves compatability with older Solaris and still
works on new Solaris.

Closes #1092
2014-08-26 10:41:02 +02:00
Daniel Priceandantirez 3e6578e32f Define AR to help with lua cross-compilation
Closes #997
2014-08-26 10:41:02 +02:00
antirez 990ec8dfc1 representRedisNodeFlags() moved into right code section.
The funciton was also modified in order to be more standalone and
produce an output without trailing spaces, making the reuse simpler.
The global variable was renamed in cammel case as most other Redis
globals, except the main ones we refer too many times, like 'server'.
2014-08-26 10:41:02 +02:00
charsyamandantirez 2c2204e050 Refactor cluster flag printing
Less copy/paste code duplication.

Closes #952
2014-08-26 10:41:02 +02:00
Matt Stancliffandantirez 4b6e73a75c Cleanup double semicolons
Closes #1161
2014-08-26 10:41:02 +02:00
SungBin_Hongandantirez 987127c6e7 Free memory in clusterLoadConfig error handler
Closes #1327
2014-08-26 10:41:02 +02:00
Chris Lambandantirez 194b0525da redis-check-dump: Prevent segfault if can't malloc
Found by The Mayhem Team (Alexandre Rebert, Thanassis Avgerinos,
Sang Kil Cha, David Brumley, Manuel Egele) Cylab, Carnegie Mellon
University. See http://bugs.debian.org/716259 for more.

Signed-off-by: Chris Lamb <lamby@debian.org>

Fixes #1191
2014-08-26 10:41:02 +02:00
charsyamandantirez 74cde68c25 Remove unused LINE_BUFLEN definition
Closes #1129
2014-08-26 10:41:02 +02:00
Matt Stancliffandantirez 17438f7cf4 Reject MOVE to non-integer DBs
Previously, "MOVE key somestring" would move the key to
DB 0 which is just unexpected and wrong.
String as DB == error.

Test added too.

Modified by @antirez in order to use the getLongLongFromObject() API
instead of strtol().

Fixes #1428
2014-08-26 10:41:02 +02:00
Matt Stancliffandantirez 961e5e201f pubsub: Return integers for NUMSUB, not strings
Also adds test for numsub — due to tcl being tcl,
it doesn't capture the "numberness" of the fix,
but now we at least have one test case for numsub.

Closes #1561
2014-08-26 10:41:02 +02:00
Xiaojie Zhangandantirez b1a6a58942 Avoid unnecessary decoding in ziplist.c
Closes #1519
2014-08-26 10:41:02 +02:00
Samuel Chouandantirez 61ce627f3c Improve accuracy of HAVE_ATOMIC dependency check
[I had to split out the clang check due to
 clang *really* not liking the __GLIBC_PREREQ macro; -matt]

Closes #1456
2014-08-26 10:41:02 +02:00
Jim Howardandantirez 6ef09f6aea Remove redundant event loop fd processing
Reason:
https://github.com/antirez/redis/pull/1450#issuecomment-30432925

Closes #1450
2014-08-26 10:41:02 +02:00
Xian Liandantirez 5f7866dcce memtest: Add missing free()
Closes #1614
2014-08-26 10:41:02 +02:00
Joshua Andersonandantirez e7e8382597 src/Makefile: Reword "to run make test" message
Closes #1610
2014-08-26 10:41:02 +02:00
cubicdaiyaandantirez d1bf997d1c Use 'void' for zero-argument functions
According to the C standard,
it is desirable to give the type 'void'
to functions have no argument.

Closes #1631
2014-08-26 10:41:02 +02:00
Matt Stancliffandantirez 0f732b96f7 Fix assert technical correctness
dictAdd returns DICT_OK, not REDIS_OK. They both
have the same underlying values, so it works even though
the code is technically wrong.

Fixes #1512
2014-08-26 10:41:02 +02:00
Jan-Erik Redigerandantirez 71955c513a install_server.sh: add missing bang
This was discovered by _bodya and reported in the IRC channel.
Everything worked fine as these scripts are always executed as shell
scripts.

Closes #1728
2014-08-26 10:41:02 +02:00
Edgars Irmejsandantirez b6b6bcbe70 Change unixsocketperm comment to 700 from 755
According to unix manuals, "Connecting to the socket object requires
read/write permission." -- mode 755 is useless for anybody
other than the owner.

Fixes #1696
2014-08-26 10:41:02 +02:00
ripcurld00dandantirez 9f98303bdb redis-check-dump: use names instead of magic nums
Use constants to avoid magic numbers in `types`, which is an array
that stores the names of `REDIS` types.

Closes #1681
2014-08-26 10:41:02 +02:00
Kashif Rasulandantirez 54e054fff3 Fix issues raised by clang analyzer
Modified by @antirez since the original fix to genInfoString() looked
weak. Probably the clang analyzer complained about `section` being
possibly NULL, and strcasecmp() called with a NULL pointer. In the
practice this can never happen, still for the sake of correctness
the right fix is not to modify only the first call, but to set `section`
to the value of "default" if it happens to be NULL.

Closes #1660
2014-08-26 10:41:02 +02:00
antirez 59c084860d Force quit when receiving a second SIGINT.
Also quit ASAP when we are still loading a DB, since care is not needed
in this special condition, especially for a SIGINT.
2014-08-26 10:41:01 +02:00
Matt Stancliffandantirez 5bc57330cd Add graceful exit when Ctrl-C is received 2014-08-26 10:41:01 +02:00
Matt Stancliffandantirez 56558ba989 Fix intset midpoint selection
The classic (min+max)/2 is provably unsafe.  Fixed
as recommended in research:
http://googleresearch.blogspot.com/2006/06/extra-extra-read-all-about-it-nearly.html

Fix inspired by @wjin, but I used a different approach.
(later, I found @kuebler fixed the same issue too).

Fixes #1741, #1602
2014-08-26 10:41:01 +02:00
Matt Stancliffandantirez 14cedd4279 Fix key extraction for SORT
We only want to use the last STORE key, but we have to record
we actually found a STORE key so we can increment the final return
key count.

Test added to prevent further regression.

Closes #1883, #1645, #1647
2014-08-26 10:41:01 +02:00
Jan-Erik Redigerandantirez b7a4cfb4b3 Handle large getrange requests
Previously the end was casted to a smaller type
which resulted in a wrong check and failed
with values larger than handled by unsigned.

Closes #1847, #1844
2014-08-26 10:41:01 +02:00
Matt Stancliffandantirez 9d4ebed9e9 scripting: no eval with negative key count
Negative key count causes segfault in Lua functions.

Fixes #1842
Closes #1843
2014-08-26 10:41:01 +02:00
kingsumosandantirez 9ac7787838 cluster: fix node connection memory leak
Cluster leaks memory while connecting due to missing freeaddrinfo()

(Commit modified by @antirez. The freeaddrinfo() call was misplaced so
 in case of no address was bound, the memory leak was still there).

Closes #1801
2014-08-26 10:41:01 +02:00
Jan-Erik Redigerandantirez bc9a5bbed4 redis-cli: fix latency result output
(Cleaned up a little by @mattsta)

Closes #1774
2014-08-26 10:41:01 +02:00
Wei Jinandantirez 4215bf453a Remove redundant else/return block
Fixes #1741
2014-08-26 10:41:01 +02:00
antirez 409f5d71e7 Clarify AIX "#undef hz". 2014-08-26 10:41:01 +02:00
siahlandantirez a95888ecb4 Add support for compiling on AIX
Closes #1900
2014-08-26 10:41:01 +02:00
Matt Stancliffandantirez 161a5acf75 redis-cli: Add --no-raw option
Some people need formatted output even when they have no
interactive tty.

Fixes #760
2014-08-26 10:41:01 +02:00
Matt Stancliffandantirez 7693b84b35 redis-cli: stop showing incorrectly selected DB
Previously redis-cli would happily show "-1" or "99999"
as valid DB choices.

Now, if the SELECT call returned an error, we don't update
the DB number in the CLI.

Inspired by @anupshendkar in #1313

Fixes #566, #1313
2014-08-26 10:41:01 +02:00
Matt Stancliffandantirez 839608569e redis-cli: Re-attach selected DB after auth
Previously, if you did SELECT then AUTH, redis-cli
would show your SELECT'd db even though it didn't
happen.

Note: running into this situation is a (hopefully) very limited
used case of people using multiple DBs and AUTH all at the same
time.

Fixes antirez#1639
2014-08-26 10:41:01 +02:00
antirez f3b0cbd15c Redis 2.9.57 (Redis 3.0.0 beta-8). 2014-07-29 10:17:48 +02:00
Yossi Gottliebandantirez 8ea459cc9b Fail SYNC if background save child aborted due to a signal. 2014-07-28 14:55:10 +02:00
vpsandantirez c2cfdc2c38 * fixed doc URL for keyspace events 2014-07-28 14:55:10 +02:00
antirez f2cb04e18c Cluster test: unit 07, replicas migration. 2014-07-28 14:55:10 +02:00
antirez f2fd98bef9 Cluster test: unit 06 cluster creation comment fixed. 2014-07-28 14:55:10 +02:00
Matt Robenoltandantirez 9d30ffa405 Fix spelling of "stand alone" in ascii art
Also unified the logic to match `genRedisInfoString`
2014-07-28 14:55:10 +02:00
antirez 582d8c0c66 Cluster test: node reachability condition fixed. 2014-07-28 14:55:10 +02:00
antirez f54162074e Cluster test: init test, be patient during restarts. 2014-07-28 14:55:10 +02:00
antirez 999205736b Cluster test: 03 unit speedup, don't send WAIT to slaves. 2014-07-28 14:55:10 +02:00
antirez b43c44a196 Cluster test: unit 06, slaves with stale data can't failover. 2014-07-28 14:55:10 +02:00
antirez bf267d3083 Example redis.conf: improve slaveof description. 2014-07-28 14:55:10 +02:00
antirez 1400b6679a Cluster test: new unit 05, slave selection. 2014-07-28 14:55:10 +02:00
antirez c5bd330f9e Cluster: don't migrate to a master that never had slaves.
Replica migration algorithm modified so that slaves never try to migrate
to masters that were never configured to have slaves in the past.
We want the algorithm to take care of masters that remained without
*working* slaves, but that used to have slaves according to the cluster
configuration.
2014-07-28 14:55:10 +02:00
antirez 5a4a86707e Fix util.c compilation by including stdint.h. 2014-07-23 18:01:58 +02:00
antirez d6f309d4c0 Faster ll2string() implementation.
Based on ideas documented in this blog post:

https://www.facebook.com/notes/facebook-engineering/three-optimization-tips-for-c/10151361643253920

The original code was modified to handle signed integers, reformetted to
fit inside the Redis code base, and was stress-tested with a program
in order to validate the implementation against snprintf().

Redis was measured to be measurably faster from the point of view of
clients in real-world operations because of this change, since sometimes
number to string conversion is used extensively (for example every time
a GET results into an integer encoded object to be returned to the
user).
2014-07-23 14:58:28 +02:00
antirez dd50295de6 PFSELFTEST: less false positives.
This is just a quickfix, for the nature of the test the right way to fix
it is to average the error of N runs, since otherwise it is always
possible to get a false positive with a bad run, or to minimize too much
this possibility we may end testing with too much "large" error ranges.
2014-07-23 11:45:08 +02:00
antirez 42f4a83ed7 Test: check sorted set elements order after union.
This is not a regression but issue #1786 showed the need for this test.
2014-07-23 09:41:01 +02:00
antirez b46271cc78 Remove useless var and check in zunionInterGenericCommand(). 2014-07-23 09:41:01 +02:00
antirez 5e88bd0526 ZUNIONSTORE reimplemented for speed.
The user @kjmph provided excellent ideas to improve speed of ZUNIONSTORE
(in certain cases by many order of magnitude), together with an
implementation of the ideas.

While the ideas were sounding, the implementation could be improved both
in terms of speed and clearness, so that's my attempt at reimplementing
the speedup proposed, trying to improve by directly using just a
dictionary with an embedded score inside, and reusing the single-pass
aggregate + order-later approach.

Note that you can't apply this commit without applying the previous
commit in this branch that adds a double in the dictEntry value union.

Issue #1786.
2014-07-23 09:41:01 +02:00
antirez 36498152f5 Add double field in dict.c entry value union. 2014-07-23 09:41:01 +02:00
antirez 955b25624e CLUSTER RESET: Flush dataset if node is a slave.
For non-empty masters, CLUSTER RESET is denied, and the user requires to
start to reset a node by explicitly clearing it with FLUSHALL.
However CLUSTER RESET when executed with slaves don't have this
restrictions since data is just a replica of the master, and with
read-only slaves it is also not possible to remove the data set. However
the node was turned from slave to master after a reset, without touching
the old slave data. This is 99.99% of times not appropriate and forces
full resets to follow this path to work with both slave and master
nodes:

    FLUSHALL
    CLUSTER RESET HARD
    FLUSHALL

Since we need the first flushall for masters, and the second for slaves.

This commit changes the behavior so that CLUSTER RESET removes the data set
of a slave node during a reset, in the moment it gets turned into a master,
so the new pattern is simply:

    FLUSHALL (that may fail for slaves)
    CLUSTER RESET
2014-07-23 09:41:01 +02:00
antirez e8199b67e3 Cluster test: use larger keyspace in resharding test. 2014-07-23 09:41:01 +02:00
antirez b4937427db Cluster test: unit 04, consistency during resharding. 2014-07-23 09:41:01 +02:00
antirez 16a66581fe Cluster: fix redis-trib --from all. 2014-07-23 09:41:01 +02:00
antirez ec054b36c3 redis-trib: allow to reshard in non-interactive way.
The introduction of --from --to --slots --yes options allow to reshard
from cli in an automated way from scripts. The code is ugly and needs
refactoring as soon as we get it in RC / stable release.
2014-07-23 09:41:01 +02:00
antirez e6ac6a170b Cluster test: test that writes are retained during failovers.
In the test we use WAIT when the master and slave are up, and only later the
partition is created killing the master, so we are sure we don't incur
in failure modes that may lose writes in this test: the goal here is to
make sure that the elected slave was replicating correctly with the
master.
2014-07-23 09:41:01 +02:00
antirez d080eea03d Test: Pub/Sub PING. 2014-07-18 12:20:56 +02:00
antirez e236199be8 Test: small integer sharing depends on maxmemory policy. 2014-07-18 12:20:56 +02:00
antirez 29e1c431bd PING: backward compatible error for wrong number of args. 2014-07-18 12:20:56 +02:00
antirez 2570eeb711 tryObjectEncoding(): use shared objects with maxmemory and non-LRU policy.
In order to make sure every object has its own private LRU counter, when
maxmemory is enabled tryObjectEncoding() does not use the pool of shared
integers. However when the policy is not LRU-based, it does not make
sense to do so, and it is much better to save memory using shared
integers.
2014-07-18 12:20:56 +02:00
antirez 2264b9816c Variadic PING with support for Pub/Sub.
PING can now be called with an additional arugment, behaving exactly
like the ECHO command. PING can now also be called in Pub/Sub mode (with
one more more subscriptions to channels / patterns) in order to trigger
the delivery of an asynchronous pong message with the optional payload.

This fixes issue #420.
2014-07-18 12:20:56 +02:00
antirez 23c1408963 PubSub clients refactoring and new PUBSUB flag.
The code tested many times if a client had active Pub/Sub subscriptions
by checking the length of a list and dictionary where the patterns and
channels are stored. This was substituted with a client flag called
REDIS_PUBSUB that is simpler to test for. Moreover in order to manage
this flag some code was refactored.

This commit is believed to have no effects in the behavior of the
server.
2014-07-18 12:20:56 +02:00
antirez 63f6351858 RDB: load string objects directly as EMBSTR objects when possible. 2014-07-18 12:20:56 +02:00
michael-grunderandantirez d75fc09354 Fix OBJECT arity
Previously, the command definition for the OBJECT command specified
a minimum of two args (and that it was variadic), which meant that
if you sent this:

OBJECT foo

When cluster was enabled, it would result in an assertion/SEGFAULT
when Redis was attempting to extract keys.

It appears that OBJECT is not variadic, and only ever takes 3 args.

https://gist.github.com/michael-grunder/25960ce1508396d0d36a
2014-07-18 12:20:56 +02:00
antirez a16feb99f2 Fix typo in LATENCY DOCTOR output. 2014-07-18 12:20:56 +02:00
antirez 639674b309 Test: more reliable AOF rewrite test under write load. 2014-07-18 12:20:56 +02:00
antirez 0e22e4ae91 LATENCY command / monitor basic tests. 2014-07-18 12:20:56 +02:00
antirez 034f39f14d LATENCY HISTORY returns empty array if no data is available.
Previously we returned an error.
2014-07-18 12:20:56 +02:00
antirez ef7c1fbcef Test: enable latency monitor in the default config.
This way as a side effect of running the test we also stress the latency
monitor data collection.
2014-07-18 12:20:56 +02:00
antirez 93c6ea6c6f Test: AOF rewrite during write load. 2014-07-18 12:20:56 +02:00
antirez 959667ffe1 Fixed a few missing newline in createLatencyReport(). 2014-07-18 12:20:56 +02:00
antirez c9ddd141dd createLatencyReport(): compile before commit avoids commits. 2014-07-18 12:20:56 +02:00
antirez 1b6d614f2c createLatencyReport(), fix mount option name. 2014-07-18 12:20:56 +02:00
antirez 44de3a9cfb createLatencyReport(): initialize all advices to zero. 2014-07-18 12:20:56 +02:00
antirez a117fd2a44 LATENCY DOCTOR first implementation complete. 2014-07-18 12:20:56 +02:00
antirez 31e1e30aa6 Fix analyzeLatencyForEvent() MAD computation. 2014-07-18 12:20:56 +02:00
antirez 9ed02e137c LATENCY DOCTOR: initial draft and events summary output. 2014-07-18 12:20:56 +02:00
antirez c7f1aa74d7 Latency: low level time series analysis implemented. 2014-07-18 12:20:56 +02:00
antirez 9a037792b6 LATENCY SAMPLES renamed LATENCY HISTORY. 2014-07-18 12:20:56 +02:00
antirez 7c6a028a3a LATENCY RESET implemented. 2014-07-18 12:20:56 +02:00
antirez cc7b10cb52 Better "final read from parent" algorithm in rewriteAppendOnlyFile*(.
We now wait up to 1 second for diff data to come from the parent,
however we use poll(2) to wait for more data, and use a counter of
contiguous failures to get data for N times (set to 20 experimentally
after different tests) as an early stop condition to avoid wasting 1
second when the write traffic is too low.
2014-07-18 12:20:56 +02:00
antirez d721e6e5c3 Log AOF diff sizes in MBs instead of bytes. 2014-07-18 12:20:56 +02:00
antirez e615539cc7 aofChildWriteDiffData() better handling of free blocks. 2014-07-18 12:20:55 +02:00
antirez f0a58bfaba redis-benchmark: abort when all clients are disconnected. 2014-07-18 12:20:55 +02:00
antirez d3514edd3e Fine tuning of aofReadDiffFromParent() calls trigger. 2014-07-18 12:20:55 +02:00
antirez 180c9143b9 Use amount of I/O not ops to trigger aofReadDiffFromParent(). 2014-07-18 12:20:55 +02:00
antirez e05cd36326 aofChildWriteDiffData(): write as much as possible. 2014-07-18 12:20:55 +02:00
antirez 9494c84523 Handle write failure in AOF parent -> child ACK. 2014-07-18 12:20:55 +02:00
antirez 896256210f Use a timeout when reading parent ack from AOF child. 2014-07-18 12:20:55 +02:00
antirez 576056d973 Send AOF diffs from parent to child to improve latency. 2014-07-18 12:20:55 +02:00
antirez ca2db2bcb9 Use fsync instead of aof_fsync in final AOF sync.
This happens in the child process so we don't care about latency:
better to sync metadata as well.
2014-07-18 12:20:55 +02:00
antirez 74423be7a5 Fixed conditional for aof-write-pending-fsync latency event selection. 2014-07-18 12:20:55 +02:00
antirez 7989b612af Dependencies updated. 2014-07-18 12:20:55 +02:00
antirez 289e83f1c9 Cast void* to char* to avoid waring in latencyCommand(). 2014-07-18 12:20:55 +02:00
antirez a7dc975a81 Properly initialize min/max in latency.c. 2014-07-18 12:20:55 +02:00
antirez b2d6266521 latencyStartMonitor() modified to avoid warnings. 2014-07-18 12:20:55 +02:00
antirez cd5c13e12f Latency monitor: specialize delayed aof writes events. 2014-07-18 12:20:55 +02:00
antirez 231e016235 LATENCY GRAPH: filling under the curve is more readable. 2014-07-18 12:20:55 +02:00
antirez 3bd6917b0d LATENCY GRAPH implemented. 2014-07-18 12:20:55 +02:00
antirez 6fb6609600 latencyTimeSeries structure max field type fixed. 2014-07-18 12:20:55 +02:00
antirez 9bc408493e Free labels in freeSparklineSequence(). 2014-07-18 12:20:55 +02:00
antirez b58c960eb2 LATENCY LATEST: add the max field. 2014-07-18 12:20:55 +02:00
antirez a9d9898680 Latency monitor trheshold value is now configurable.
This commit adds both support for redis.conf and CONFIG SET/GET.
2014-07-18 12:20:55 +02:00
antirez b71a980687 ASCII sparklines generation API. 2014-07-18 12:20:55 +02:00
antirez e817f6c013 License added to latency.h. 2014-07-18 12:20:55 +02:00
antirez 0f674f3368 Latency monitor turned off by default.
It is not a good idea to bloat the code with gettimeofday() calls if the
instance is working well, and turning monitoring on at runtime is a
joke.
2014-07-18 12:20:55 +02:00
antirez 2b5a0e8066 Latency monitor: more hooks around the code. 2014-07-18 12:20:55 +02:00
antirez 372484ab8e Latency monitor: don't add new samples in the same second.
Instead we update the old sample with the new latency if it is greater.
2014-07-18 12:20:55 +02:00
antirez 5e56590198 LATENCY LATEST implemented. 2014-07-18 12:20:55 +02:00
antirez 485a5908c0 Latency monitor: command duration is in useconds. Convert. 2014-07-18 12:20:55 +02:00
antirez 41c9d1c08f LATENCY SAMPLES implemented. 2014-07-18 12:20:55 +02:00
antirez 44cf7395a2 Latency monitor: collect slow commands.
We introduce the distinction between slow and fast commands since those
are two different sources of latency. An O(1) or O(log N) command without
side effects (can't trigger deletion of large objects as a side effect of
its execution) if delayed is a symptom of inherent latency of the system.

A non-fast command (commands that may run large O(N) computations) if
delayed may just mean that the user is executing slow operations.

The advices LATENCY should provide in this two different cases are
different, so we log the two classes of commands in a separated way.
2014-07-18 12:20:55 +02:00
antirez 62ca4bd438 Latency monitor: basic samples collection. 2014-07-18 12:20:55 +02:00
antirez 4a7fca7c07 Fix Solaris compilation due to ctime_r() call.
Introduced in Redis 2.8.10 because of a change in Sentinel.
This closes issue #1837.
2014-06-30 16:28:22 +02:00
antirez f884be6854 Redis 2.9.56 (Redis 3.0.0 beta-7). 2014-06-30 12:11:22 +02:00
antirez 05b6b147ec Test: find_available_port: check that cluster port is free as well.
The function will only return ports that have also port+10000 free, so
that Redis Cluster instances can be executed at the returned port.
2014-06-30 12:11:22 +02:00
antirez a5aa3b6491 Test: fix instances.tcl restart_instance abort error. 2014-06-30 12:11:22 +02:00
antirez b18c92b9e7 DEBUG CMDKEYS moved to COMMAND GETKEYS. 2014-06-27 18:39:33 +02:00
antirez d3f43db20f COMMAND COUNT subcommand added. 2014-06-27 18:39:33 +02:00
antirez b205119810 COMMAND: fix argument parsing.
This fixes detection of wrong subcommand (that resulted in the default
all-commands output instead) and allows COMMAND INFO to be called
without arguments (resulting into an empty array) which is useful in
programmtically generated calls like the following (in Ruby):

    redis.commands("command","info",*mycommands)

Note: mycommands may be empty.
2014-06-27 18:39:33 +02:00
antirez 3364b2d12b COMMANDS command renamed COMMAND. 2014-06-27 18:39:33 +02:00
antirez 0aab8eb1e3 COMMANDS command: remove static + aesthetic changes.
Static was removed since it is needed in order to get symbols in stack
traces. Minor changes in the source code were operated to make it more
similar to the existing Redis code base.
2014-06-27 18:39:33 +02:00
Matt Stancliffandantirez 60d62a99d2 Cluster: Add COMMANDS command
COMMANDS returns a nested multibulk reply for each
command in the command table.  The reply for each
command contains:
  - command name
  - arity
  - array of command flags
  - start key position
  - end key position
  - key offset step
  - optional: if the keys are not deterministic and
    Redis uses an internal key evaluation function,
    the 6th field appears and is defined as a status
    reply of: REQUIRES ARGUMENT PARSING

Cluster clients need to know where the keys are in each
command to implement proper routing to cluster nodes.

Redis commands can have multiple keys, keys at offset steps, or other
issues where you can't always assume the first element after
the command name is the cluster routing key.

Using the information exposed by COMMANDS, client implementations
can have live, accurate key extraction details for all commands.

Also implements COMMANDS INFO [commands...] to return only a
specific set of commands instead of all 160+ commands live in Redis.
2014-06-27 18:39:33 +02:00
antirez e530c2ccb2 Fixed assert conditional in ROLE command test. 2014-06-26 22:13:54 +02:00
antirez b411678679 No more trailing spaces in Redis source code. 2014-06-26 18:48:47 +02:00
antirez b4d378f137 CLIENT KILL: don't kill the master as a normal client.
Technically the problem is due to the client type API that does not
return a special value for the master, however fixing it locally in the
CLIENT KILL command is better currently because otherwise we would
introduce a new output buffer limit class as a side effect.
2014-06-26 18:43:28 +02:00
antirez 76e7673259 Remove infinite loop from PSYNC test.
Added for debugging and forgot there.
2014-06-26 18:43:28 +02:00
antirez 2e01ac52c7 Test: hopefully more robust PSYNC test.
This is supposed to fix issue #1417, but we'll know if this is enough
only after a couple of runs of the CI test without false positives.
2014-06-26 18:43:28 +02:00
Matt Stancliffandantirez 77fb9fd4fd Allow __powerpc__ to define HAVE_ATOMIC too
From mailing list post https://groups.google.com/forum/#!topic/redis-db/D3k7KmJmYgM

In the file “config.h”, the definition HAVE_ATOMIC is used to indicate
if an architecture on which redis is implemented supports atomic
synchronization primitives.  Powerpc  supports atomic synchronization
primitives, however, it is not listed as one of the architectures
supported in config.h. This patch  adds the __powerpc__ to the list of
architectures supporting these primitives. The improvement of redis
due to the atomic synchronization on powerpc is significant,
around 30% to 40%, over the default implementation using pthreads.

This proposal adds __powerpc__ to the list of architectures designated
to support atomic builtins.
2014-06-26 15:16:23 +02:00
Matt Stancliffandantirez 27e30e2f8b Allow atomic memory count update with C11 builtins
From mailing list post https://groups.google.com/forum/#!topic/redis-db/QLjiQe4D7LA

In zmalloc.c the following primitives are currently used
to synchronize access to single global variable:
__sync_add_and_fetch
__sync_sub_and_fetch

In some architectures such as powerpc these primitives are overhead
intensive. More efficient C11 __atomic builtins are available with
newer GCC versions, see
http://gcc.gnu.org/onlinedocs/gcc-4.8.2/gcc/_005f_005fatomic-Builtins.html#_005f_005fatomic-Builtins

By substituting the following  __atomic… builtins:
__atomic_add_fetch
__atomic_sub_fetch

the performance improvement on certain architectures such as powerpc can be significant,
around 10% to 15%, over the implementation using __sync builtins while there is only slight uptick on
Intel architectures because it was already enforcing Intel Strongly ordered memory semantics.

The selection of __atomic built-ins can be predicated on the definition of ATOMIC_RELAXED
which Is available on in gcc 4.8.2 and later versions.
2014-06-26 15:16:23 +02:00
Matt Stancliffandantirez 4d14f2744a Use predefined macro for used_memory() update 2014-06-26 15:16:23 +02:00
antirez a7bef1bb7c CLUSTER SLOTS: don't output failing slaves.
While we have to output failing masters in order to provide an accurate
map (that may be the one of a Redis Cluster in down state because not
all slots are served by a working master), to provide slaves in FAIL
state is not a good idea since those are not necesarely needed, and the
client will likely incur into a latency penalty trying to connect with a
slave which is down.

Note that this means that CLUSTER SLOTS does not provide a *complete*
map of slaves, however this would not be of any help since slaves may be
added later, and a client that needs to scale reads and requires to
stay updated with the list of slaves, need to do a refresh of the map
from time to time, anyway.
2014-06-26 15:16:23 +02:00
antirez 94bbcf400d CLUSTER SLOTS: main loop should skip only slaves and zero slot masters. 2014-06-26 15:16:23 +02:00
Matt Stancliffandantirez 237188bf1b Cluster: Add CLUSTER SLOTS command
CLUSTER SLOTS returns a Redis-formatted mapping from
slot ranges to IP/Port pairs serving that slot range.

The outer return elements group return values by slot ranges.

The first two entires in each result are the min and max slots for the range.

The third entry in each result is guaranteed to be either
an IP/Port of the master for that slot range - OR - null
if that slot range, for some reason, has no master

The 4th and higher entries in each result are replica instances
for the slot range.

Output comparison:
127.0.0.1:7001> cluster nodes
f853501ec8ae1618df0e0f0e86fd7abcfca36207 127.0.0.1:7001 myself,master - 0 0 2 connected 4096-8191
5a2caa782042187277647661ffc5da739b3e0805 127.0.0.1:7005 slave f853501ec8ae1618df0e0f0e86fd7abcfca36207 0 1402622415859 6 connected
6c70b49813e2ffc9dd4b8ec1e108276566fcf59f 127.0.0.1:7007 slave 26f4729ca0a5a992822667fc16b5220b13368f32 0 1402622415357 8 connected
2bd5a0e3bb7afb2b56a2120d3fef2f2e4333de1d 127.0.0.1:7006 slave 32adf4b8474fdc938189dba00dc8ed60ce635b0f 0 1402622419373 7 connected
5a9450e8279df36ff8e6bb1c139ce4d5268d1390 127.0.0.1:7000 master - 0 1402622418872 1 connected 0-4095
32adf4b8474fdc938189dba00dc8ed60ce635b0f 127.0.0.1:7002 master - 0 1402622419874 3 connected 8192-12287
5db7d05c245267afdfe48c83e7de899348d2bdb6 127.0.0.1:7004 slave 5a9450e8279df36ff8e6bb1c139ce4d5268d1390 0 1402622417867 5 connected
26f4729ca0a5a992822667fc16b5220b13368f32 127.0.0.1:7003 master - 0 1402622420877 4 connected 12288-16383

127.0.0.1:7001> cluster slots
1) 1) (integer) 0
   2) (integer) 4095
   3) 1) "127.0.0.1"
      2) (integer) 7000
   4) 1) "127.0.0.1"
      2) (integer) 7004
2) 1) (integer) 12288
   2) (integer) 16383
   3) 1) "127.0.0.1"
      2) (integer) 7003
   4) 1) "127.0.0.1"
      2) (integer) 7007
3) 1) (integer) 4096
   2) (integer) 8191
   3) 1) "127.0.0.1"
      2) (integer) 7001
   4) 1) "127.0.0.1"
      2) (integer) 7005
4) 1) (integer) 8192
   2) (integer) 12287
   3) 1) "127.0.0.1"
      2) (integer) 7002
   4) 1) "127.0.0.1"
      2) (integer) 7006
2014-06-26 15:16:23 +02:00
antirez 8fdc857a9f Cluster: myself->ip autodiscovery.
Instead of having an hardcoded IP address in the node configuration, we
autodiscover it via MEET messages for automatic update when the node is
restarted with a different IP address.

This mechanism was discussed in the context of PR #1782.
2014-06-26 15:16:23 +02:00
antirez b56849cd82 Old form of CLIENT KILL should still allow suicide. 2014-06-24 12:49:26 +02:00
antirez 6adf0e092c Sentinel test: more correct sentinels config reset.
In the initialization test for each instance we used to unregister the
old master and register it again to clear the config.
However there is a race condition doing this: as soon as we unregister
and re-register "mymaster", another Sentinel can update the new
configuration with the old state because of gossip "hello" messages.

So the correct procedure is instead, unregister "mymaster" from all the
sentinel instances, and re-register it everywhere again.
2014-06-23 14:07:45 +02:00
antirez 2f778b1f6b Sentinel implementation of ROLE. 2014-06-23 12:07:52 +02:00
antirez 8322ede059 Silence different signs comparison warning in sds.c. 2014-06-23 11:54:46 +02:00
antirez 26291dc7f3 Use Redis updated sds.c for deps/hiredis. 2014-06-23 11:54:46 +02:00
Matt Stancliffandantirez 02b94801ad Sentinel: bind source address
Some deployments need traffic sent from a specific address.  This
change uses the same policy as Cluster where the first listed bindaddr
becomes the source address for outgoing Sentinel communication.

Fixes #1667
2014-06-23 11:54:45 +02:00
Matt Stancliffandantirez a2632f2692 Add REDIS_BIND_ADDR access macro
We need to access (bindaddr[0] || NULL) in a few places, so centralize
access with a nice macro.
2014-06-23 11:54:45 +02:00
antirez ba472111d3 Dummy zmalloc.h restored into deps/hiredis. 2014-06-23 11:54:45 +02:00
Matt Stancliffandantirez b95332a6a6 hiredis: Update to latest version
This is hiredis f225c276be7fd0646019b51023e3f41566633dfe

This update includes all changes that diverged inside of Redis since
the last update.  This version also allows optional source address
binding for connections which we need for some Sentinel deployments.
2014-06-23 11:54:45 +02:00
Matt Stancliffandantirez ddad61bfcc Cancel SHUTDOWN if initial AOF is being written
Fixes #1826 (and many other reports of the same problem)
2014-06-23 10:09:22 +02:00
antirez 5eb5a06078 Basic tests for the ROLE command. 2014-06-23 09:09:07 +02:00
antirez 82caeee12e Allow to call ROLE in LOADING state. 2014-06-21 15:18:33 +02:00
antirez 96f556b27e ROLE command: array len fixed for slave output. 2014-06-21 15:18:33 +02:00
antirez 5c988b334e Jemalloc updated to 3.6.0.
Not a single bug in about 3 months, and our previous version was
too old (3.2.0).
2014-06-21 15:18:33 +02:00
antirez f94d46ba9a Jemalloc update script added. 2014-06-21 15:18:33 +02:00
antirez 9eb02c6a1e Cluster: clear NOADDR flag when updating node address. 2014-06-21 15:18:33 +02:00
antirez 52456bf8c3 Sentinel test: tolerate larger delays in init tests. 2014-06-21 15:18:33 +02:00
antirez 716a77d347 Sentinel test: unit 02, avoid some time related false positives. 2014-06-21 15:18:33 +02:00
antirez 54756adf44 Sentinel: send hello messages ASAP after config change.
Eventual configuration convergence is guaranteed by our periodic hello
messages to all the instances, however when there are important notices
to share, better make a phone call. With this commit we force an hello
message to other Sentinal and Redis instances within the next 100
milliseconds of a config update, which is practically better than
waiting a few seconds.
2014-06-21 15:18:33 +02:00
antirez 32adcda857 Sentinel test: add manual failover test. 2014-06-21 15:18:33 +02:00
antirez 6588ef2944 Sentinel: handle SRI_PROMOTED flag correctly.
Lack of check of the SRI_PROMOTED flag caused Sentienl to act with the
promoted slave turned into a master during failover like if it was a
normal instance.

Normally this problem was not apparent because during real failovers the
old master is down so the bugged code path was not entered, however with
manual failovers via the SENTINEL FAILOVER command, the problem was
easily triggered.

This commit prevents promoted slaves from getting reconfigured, moreover
we now explicitly check that during a failover the slave turning into a
master is the one we selected for promotion and not a different one.
2014-06-21 15:18:33 +02:00
antirez 64e618d10c Test: use higher level redis.tcl proc to read replies. 2014-06-21 15:18:33 +02:00
antirez 28c86fe463 Sentienl/Redis test: enable link reconnection in instances.tcl. 2014-06-21 15:18:32 +02:00
antirez 3cba0c3ebb Test: Tcl client initial support for automatic reconnection. 2014-06-21 15:18:32 +02:00
antirez 256046fdb4 Test: tcl client, unset deferred var on close. 2014-06-21 15:18:32 +02:00
Matt Stancliffandantirez 4dfc18596e Add correct exit value to failed tests 2014-06-21 15:18:32 +02:00
Alex Suraciandantirez 2b1047ffd3 add missing signal.h include 2014-06-21 15:18:32 +02:00
Matt Stancliffandantirez 65bab61159 Add SIGINT handler to cli --intrinsic-latency
If we run a long latency session and want to Ctrl-C out of it,
it's nice to still get the summary output at the end.
2014-06-21 15:18:32 +02:00
antirez 8d75a20ab1 Sentinel: send SLAVEOF with MULTI, CLIENT KILL, CONFIG REWRITE.
This implements the new Sentinel-Client protocol for the Sentinel part:
now instances are reconfigured using a transaction that ensures that the
config is rewritten in the target instance, and that clients lose the
connection with the instance, in order to be forced to: ask Sentinel,
reconnect to the instance, and verify the instance role with the new
ROLE command.
2014-06-21 15:18:32 +02:00
antirez 52f1edb2ab CLIENT KILL API modified.
Added a new SKIPME option that is true by default, that prevents the
client sending the command to be killed, unless SKIPME NO is sent.
2014-06-21 15:18:32 +02:00
antirez 7e9863637c CLIENT KILL: fix closing link of the current client. 2014-06-21 15:18:32 +02:00
antirez ca262a58c6 New features for CLIENT KILL. 2014-06-21 15:18:32 +02:00
antirez b7dd451892 Assign an unique non-repeating ID to each new client.
This will be used by CLIENT KILL and is also a good way to ensure a
given client is still the same across CLIENT LIST calls.

The output of CLIENT LIST was modified to include the new ID, but this
change is considered to be backward compatible as the API does not imply
you can do positional parsing, since each filed as a different name.
2014-06-21 15:18:32 +02:00
antirez d97d1a6430 Client types generalized.
Because of output buffer limits Redis internals had this idea of type of
clients: normal, pubsub, slave. It is possible to set different output
buffer limits for the three kinds of clients.

However all the macros and API were named after output buffer limit
classes, while the idea of a client type is a generic one that can be
reused.

This commit does two things:

1) Rename the API and defines with more general names.
2) Change the class of clients executing the MONITOR command from "slave"
   to "normal".

"2" is a good idea because you want to have very special settings for
slaves, that are not a good idea for MONITOR clients that are instead
normal clients even if they are conceptually slave-alike (since it is a
push protocol).

The backward-compatibility breakage resulting from "2" is considered to
be minimal to care, since MONITOR is a debugging command, and because
anyway this change is not going to break the format or the behavior, but
just when a connection is closed on big output buffer issues.
2014-06-21 15:18:32 +02:00
antirez 59cfa49c21 Scripting: regression test for issue #1811. 2014-06-21 15:18:32 +02:00
antirez 2b57d96ea0 Fix semantics of Lua calls to SELECT.
Lua scripts are executed in the context of the currently selected
database (as selected by the caller of the script).

However Lua scripts are also free to use the SELECT command in order to
affect other DBs. When SELECT is called frm Lua, the old behavior, before
this commit, was to automatically set the Lua caller selected DB to the
last DB selected by Lua. See for example the following sequence of
commands:

    SELECT 0
    SET x 10
    EVAL "redis.call('select','1')" 0
    SET x 20

Before this commit after the execution of this sequence of commands,
we'll have x=10 in DB 0, and x=20 in DB 1.

Because of the problem above, there was a bug affecting replication of
Lua scripts, because of the actual implementation of replication. It was
possible to fix the implementation of Lua scripts in order to fix the
issue, but looking closely, the bug is the consequence of the behavior
of Lua ability to set the caller's DB.

Under the old semantics, a script selecting a different DB, has no simple
ways to restore the state and select back the previously selected DB.
Moreover the script auhtor must remember that the restore is needed,
otherwise the new commands executed by the caller, will be executed in
the context of a different DB.

So this commit fixes both the replication issue, and this hard-to-use
semantics, by removing the ability of Lua, after the script execution,
to force the caller to switch to the DB selected by the Lua script.

The new behavior of the previous sequence of commadns is to just set
X=20 in DB 0. However Lua scripts are still capable of writing / reading
from different DBs if needed.

WARNING: This is a semantical change that will break programs that are
conceived to select the client selected DB via Lua scripts.

This fixes issue #1811.
2014-06-21 15:18:32 +02:00
antirez 62bddfe3af Cluser test: write multiple keys in unit 03.
The write-failover-read test is better if we write 100 keys so that we
are sure all the nodes are affected by the write.
2014-06-21 15:18:32 +02:00
antirez 1381a6bf3d Cluster test: rewrite config when resetting nodes. 2014-06-21 15:18:32 +02:00
antirez 0f4b925de5 Scripting: Fix for a #1118 regression simplified.
It is more straightforward to just test for a numerical type avoiding
Lua's automatic conversion. The code is technically more correct now,
however Lua should automatically convert to number only if the original
type is a string that "looks like a number", and not from other types,
so practically speaking the fix is identical AFAIK.
2014-06-21 15:18:32 +02:00
Matt Stancliffandantirez 9ca83c82bf Scripting: Fix regression from #1118
The new check-for-number behavior of Lua arguments broke
users who use large strings of just integers.

The Lua number check would convert the string to a number, but
that breaks user data because
Lua numbers have limited precision compared to an arbitrarily
precise number wrapped in a string.

Regression fixed and new test added.

Fixes #1118 again.
2014-06-21 15:18:32 +02:00
antirez 6b566ab78a Cluster: fix an error message when logging failover auth denied. 2014-06-21 15:18:32 +02:00
antirez 326145b9ef Cluster: better comment for clusterSendFailoverAuthIfNeeded() epoch test. 2014-06-21 15:18:32 +02:00
antirez 053bcf034e Cluster test: set config epochs when creating clusters. 2014-06-21 15:18:32 +02:00
antirez 78a687ec17 Cluster: log granted failover authorizations. 2014-06-21 15:18:32 +02:00
antirez 61910a6708 Cluster: log configEpoch updates to myself. 2014-06-21 15:18:32 +02:00
antirez 111a5556aa Cluster: log when a master denies a failover auth. 2014-06-21 15:18:32 +02:00
antirez 6cd068d421 Cluster test: unit 03, check epochs correctness. 2014-06-21 15:18:32 +02:00
antirez 90b4f91e1a Cluster test: better failover detection in unit 03. 2014-06-21 15:18:32 +02:00
antirez 9912a60d01 Cluster test: added show-redis-logs command in the interactive console. 2014-06-21 15:18:32 +02:00
antirez 7079ef0341 Cluster test: new unit 03, failover loop stress testing. 2014-06-21 15:18:32 +02:00
antirez 1d896303af Cluster test: 02 unit more reliable waiting for slave sync. 2014-06-21 15:18:32 +02:00
antirez d7a4025828 Cluster test: unit 02 should wait for failover. 2014-06-21 15:18:31 +02:00
antirez 33f09f0ff3 Cluster test: check master -> slave role switch. 2014-06-21 15:18:31 +02:00
antirez 5cdefb7c99 Cluster: cluster_my_epoch added to CLUSTER INFO output. 2014-06-21 15:18:31 +02:00
antirez 2213f283c5 ROLE output improved for slaves.
Info about the replication state with the master added.
2014-06-21 15:18:31 +02:00
antirez 9b460e803a ROLE command added.
The new ROLE command is designed in order to provide a client with
informations about the replication in a fast and easy to use way
compared to the INFO command where the same information is also
available.
2014-06-21 15:18:31 +02:00
antirez ea5335fb9b Redis 2.9.55 (Redis 3.0.0 beta-6). 2014-06-09 16:01:12 +02:00
Matt Stancliffandantirez 726d343ad7 Fix lack of strtold under Cygwin
Renaming strtold to strtod then casting
the result is the standard way of dealing with
no strtold in Cygwin.
2014-06-09 11:43:17 +02:00
Matt Stancliffandantirez 28fef5c57b Fix lack of SA_ONSTACK under Cygwin
Fixes #232
2014-06-09 11:43:17 +02:00
Matt Stancliffandantirez 7fc1fc8ca7 Fix blocking operations from missing new lists
Behrad Zari discovered [1] and Josiah reported [2]: if you block
and wait for a list to exist, but the list creates from
a non-push command, the blocked client never gets notified.

This commit adds notification of blocked clients into
the DB layer and away from individual commands.

Lists can be created by [LR]PUSH, SORT..STORE, RENAME, MOVE,
and RESTORE.  Previously, blocked client notifications were
only triggered by [LR]PUSH.  Your client would never get
notified if a list were created by SORT..STORE or RENAME or
a RESTORE, etc.

Blocked client notification now happens in one unified place:
  - dbAdd() triggers notification when adding a list to the DB

Two new tests are added that fail prior to this commit.

All test pass.

Fixes #1668

[1]: https://groups.google.com/forum/#!topic/redis-db/k4oWfMkN1NU
[2]: #1668
2014-06-09 11:38:55 +02:00
antirez 8b059f06d7 Cluster: check that configEpoch never goes back.
Since there are ways to alter the configEpoch outside of the failover
procedure (for exampel CLUSTER SET-CONFIG-EPOCH and via the configEpoch
collision resolution algorithm), make always sure, before replacing our
configEpoch with a new one, that it is greater than the current one.
2014-06-07 14:45:53 +02:00
antirez 67029323a3 Cluster: SET-CONFIG-EPOCH should update currentEpoch.
SET-CONFIG-EPOCH, used by redis-trib at cluster creation time, failed to
update the currentEpoch, making it possible after a failover for a
server to set its configEpoch to a value smaller than the current one
(since configEpochs are obtained using currentEpoch).

The bug totally break the Redis Cluster algorithms and protocols
allowing for permanent split brain conditions about the slots
configuration as shown in issue #1799.
2014-06-07 14:28:21 +02:00
Andy Grunwaldandantirez 531e577e2d Fixed typo in word avarege in result message of --intrinsic-latency analyzer 2014-06-06 11:20:00 +02:00
Jan-Erik Redigerandantirez eca41f48d3 Small typo fixed 2014-06-06 10:45:44 +02:00
yoavandantirez a20d5c50d0 Fix eval usage in tests to conform with eval semantics 2014-06-06 10:44:04 +02:00
zionwuandantirez a970f5ca70 fix issue 1787 2014-06-06 10:38:01 +02:00
antirez 9f5ab8699d Don't process min-slaves-to-write for slaves.
Replication is totally broken when a slave has this option, since it
stops accepting updates from masters.

This fixes issue #1434.
2014-06-05 10:50:33 +02:00
antirez cfe2d563bc Tests for min-slaves-* feature. 2014-06-05 10:50:32 +02:00
antirez 751c8698d4 Fixed dbuf variable scope in luaRedisGenericCommand().
I'm not sure if while the visibility is the inner block, the fact we
point to 'dbuf' is a problem or not, probably the stack var isx
guaranteed to live until the function returns. However obvious code is
better anyway.
2014-06-04 18:58:56 +02:00
antirez b71720772e Regression test for issue #1118. 2014-06-04 18:51:33 +02:00
antirez c3967f42d7 Scripting: better Lua number -> string conversion in luaRedisGenericCommand().
The lua_to*string() family of functions use a non optimal format
specifier when converting integers to strings. This has both the problem
of the number being converted in exponential notation, which we don't
use as a Redis return value when floating point numbers are involed,
and, moreover, there is a loss of precision since the default format
specifier is not able to represent numbers that must be represented
exactly in the IEEE 754 number mantissa.

The new code handles it as a special case using a saner conversion.

This fixes issue #1118.
2014-06-04 18:45:45 +02:00
antirez 3614176349 More trailing spaces in sentinel.c removed. 2014-05-28 15:46:11 +02:00
antirez 53bc8a77d1 Cluster test: add tmp dir to Git repo. 2014-05-26 18:06:48 +02:00
Matt Stancliffandantirez 033d717295 Fix test framework to detect proper server PID
Previously the PID format was:
[PID] Timestamp

But it recently changed to:
PID:X Timestamp

The tcl testing framework was grabbing the PID from \[\d+\], but
that's not valid anymore.

Now we grab the pid from "PID: <PID>" in the part of Redis startup
output to the right of the ASCII logo.
2014-05-26 17:58:49 +02:00
Matt Stancliffandantirez 186adefd56 Disable recursive watchdog signal handler
If we are in the signal handler, we don't want to handle
the signal again.  In extreme cases, this can cause a stack overflow
and segfault Redis.

Fixes #1771
2014-05-26 17:53:37 +02:00
antirez 4fe4c4b34d Redis 2.9.54 (Redis 3.0.0 beta-5). 2014-05-26 16:38:11 +02:00
antirez eea0d41fb9 Cluster: always allow ok -> fail switch in clusterUpdateState().
There is a time defined by REDIS_CLUSTER_WRITABLE_DELAY where fail -> ok
switch is not possible after startup as a master for some time, however
the contrary (ok -> fail) should always be possible.
2014-05-26 16:36:14 +02:00
antirez 672e2672aa Cluster test: catch FLUSHALL errors on node reset.
FLUSHALL will fail on read-only slaves, but there the command is not
needed in order to reset the instance with CLUSTER RESET so errors can
be ignored.
2014-05-26 16:36:11 +02:00
antirez df0c8c28ad Sentinel example config: explain you don't need to specify slaves. 2014-05-26 10:18:04 +02:00
antirez b5f0d0d6f9 Redis 2.9.53 (Redis 3.0.0 beta-4). 2014-05-23 16:08:38 +02:00
antirez 7f8ea7a50f Cluster test: basic failover unit added. 2014-05-23 15:54:53 +02:00
antirez 29aed9c48f Cluster test: move basic read/write test into a procedure. 2014-05-23 15:54:53 +02:00
antirez 81af783fec Cluster test: more reliable 01-faildet unit.
Do things in a sequence that prevents failover during failure detection.
2014-05-23 15:54:53 +02:00
antirez e62e3c4cb2 3.0 Changelog typo fixed. lost -> list. 2014-05-23 09:49:45 +02:00
antirez 5697f4f85d Document log format change in 3.0 Changelog. 2014-05-23 09:28:32 +02:00
antirez 90bbf1159b redisLogFromHandler() format changed to match new logs format. 2014-05-23 09:26:04 +02:00
antirez 9996c37e74 Tag every log line with role.
Every log contains, just after the pid, a single character that provides
information about the role of an instance:

S - Slave
M - Master
C - Writing child
X - Sentinel
2014-05-23 09:26:04 +02:00
antirez 33f63ff988 Cluster: slave validity factor is now user configurable.
Check the commit changes in the example redis.conf for more information.
2014-05-23 09:26:04 +02:00
antirez 308c570e04 Test: AOF test false positive when running in slow hosts.
The bug was triggered by running the test with Valgrind (which is a lot
slower and more sensible to timing issues) after the recent changes
that made Redis more promptly able to reply with the -LOADING error.
2014-05-23 09:26:04 +02:00
antirez 1d70312028 Test: dump.tcl fixed for RESTORE new error msg. 2014-05-23 09:26:04 +02:00
antirez 0e1b5ff63b Fix an error in redis-trib where we always talk with same node.
While iterating the list of nodes we want to set the slot as stable in
the current node, not always in the first node of the list.
2014-05-23 09:26:04 +02:00
antirez c268d1ebd5 redis-trib fix improved: move keys from N nodes to owner. 2014-05-23 09:26:04 +02:00
antirez 32ca949c76 redis-trib fix: use MIGRATE REPLACE when fixing slots.
This fixes issue #1765.
2014-05-23 09:26:04 +02:00
antirez 122157f220 Regression test for issue #1764. 2014-05-20 17:45:50 +02:00
antirez d357e3ee8f Remove trailing spaces from scripting.c 2014-05-20 17:45:50 +02:00
antirez d0c84acc97 Remove trailing spaces from sentinel.c. 2014-05-20 17:45:50 +02:00
michael-grunderandantirez 6f49395148 Fix LUA_OBJCACHE segfault.
When scanning the argument list inside of a redis.call() invocation
for pre-cached values, there was no check being done that the
argument we were on was in fact within the bounds of the cache size.

So if a redis.call() command was ever executed with more than 32
arguments (current cache size #define setting) redis-server could
segfault.
2014-05-20 17:45:50 +02:00
antirez 440ee603b7 Cluster test: better failure detection test and framework improvements. 2014-05-20 17:45:50 +02:00
antirez a7fd2eb594 Cluster test: failure detection initial tests. 2014-05-20 17:45:50 +02:00
antirez a82c000bdb Cluster test: proper initialization at unit startup. 2014-05-20 17:45:50 +02:00
antirez 8c6e86805c Cluster: use clusterSetNodeAsMaster() during slave failover.
clusterHandleSlaveFailover() was reimplementing what
clusterSetNodeAsMaster() without any good reason.
2014-05-20 17:45:50 +02:00
antirez 41a7241603 Cluster: clear todo_before_sleep flags when executing actions.
Thanks to this change, when there is some code like:

    clusterDoBeforeSleep(CLUSTER_TODO_UPDATE_STATE|...);
    ... and later before returning to the event loop ...
    clusterUpdateState();

The clusterUpdateState() function will clar the flag and will not be
repeated in the clusterBeforeSleep() function. This especially important
for config save/fsync flags which are slow to execute and not a good
idea to repeat without a good reason.

This is implemented for all the CLUSTER_TODO flags.
2014-05-20 17:45:50 +02:00
antirez 5685d15af1 Fixed typo in CLUSTER RESET implementation. 2014-05-20 17:45:50 +02:00
antirez 5efa5501e8 CLUSTER RESET implemented.
The new command is able to reset a cluster node so that it starts again
as a fresh node. By default the command performs a soft reset (the same
as calling it as CLUSTER RESET SOFT), and the following steps are
performed:

1) All slots are set as unassigned.
2) The list of known nodes is flushed.
3) Node is set as master if it is a slave.

When an hard reset is performed with CLUSTER RESET HARD the following
additional operations are performed:

4) A new Node ID is created at random.
5) Epochs are set to 0.

CLUSTER RESET is useful both when the sysadmin wants to reconfigure a
node with a different role (for example turning a slave into a master)
and for testing purposes.

It also may play a role in automatically provisioned Redis Clusters,
since it allows to reset a node back to the initial state in order to be
reconfigured.
2014-05-20 17:45:50 +02:00
antirez 687f84a377 Remove trailing spaces from cluster.c file. 2014-05-20 17:45:50 +02:00
antirez d279024081 Cluster test: added function assert_cluster_state. 2014-05-20 17:45:50 +02:00
antirez be1594905f Cluster: don't accept cluster bus connections during startup. 2014-05-20 17:45:50 +02:00
antirez b8a71e5a77 Cluster: better handling of stolen slots.
The previous code handling a lost slot (by another master with an higher
configuration for the slot) was defensive, considering it an error and
putting the cluster in an odd state requiring redis-cli fix.

This was changed, because actually this only happens either in a
legitimate way, with failovers, or when the admin messed with the config
in order to reconfigure the cluster. So the new code instead will try to
make sure that the keys stored match the new slots map, by removing all
the keys in the slots we lost ownership from.

The function that deletes the keys from the lost slots is called only
if the node does not lose all its slots (resulting in a reconfiguration
as a slave of the node that got ownership). This is an optimization
since the replication code will anyway flush all the instance data in
a faster way.
2014-05-20 17:45:49 +02:00
antirez f727000f0c cluster.tcl: fix redis links leak in refresh_nodes_map. 2014-05-20 17:45:49 +02:00
antirez 4db4a5d5d6 cluster.tcl: saner error handling.
Better handling of connection errors in order to update the table and
recovery, populate the startup nodes table after fetching the list of
nodes.

More work to do about it, it is still not as reliable as
redis-rb-cluster implementation which is the minimal reference
implementation for Redis Cluster clients.
2014-05-20 17:45:49 +02:00
antirez d9227938c8 redis.tcl: return I/O error message when peer closes connection. 2014-05-20 17:45:49 +02:00
antirez e10ee0728a Cluster: fixed data_age computation / check integer overflow. 2014-05-20 17:45:49 +02:00
antirez e84dcabf83 Cluster: forced failover implemented.
Using CLUSTER FAILOVER FORCE it is now possible to failover a master in
a forced way, which means:

1) No check to understand if the master is up is performed.
2) No data age of the slave is checked. Evan a slave with very old data
   can manually failover a master in this way.
3) No chat with the master is attempted to reach its replication offset:
   the master can just be down.
2014-05-20 17:45:49 +02:00
antirez b5cdd42bf3 Cluster: bypass data_age check for manual failovers.
Automatic failovers only happen in Redis Cluster if the slave trying to
be elected was disconnected from its master for no more than 10 times
the node-timeout value. However there should be no such a check for
manual failovers, since these are initiated by the sysadmin that, in
theory, knows what she is doing when a slave is selected to be promoted.
2014-05-20 17:45:49 +02:00
antirez 26eb7ec731 HyperLogLog regression test for issue #1762. 2014-05-19 15:45:22 +02:00
Mike Trinkalaandantirez 73737c4b85 Correct the HyperLogLog stale cache flag to prevent unnecessary computations.
Set the MSB as documented.
2014-05-19 15:45:22 +02:00
Akos Vandraandantirez 15a1c261ea Fixed possible buffer overflow bug if RDB file is corrupted.
(Note: commit message modified by @antirez for clarity).
2014-05-12 15:56:17 +02:00
Akos Vandraandantirez ef92a30767 fixed possible buffer overflow error 2014-05-12 15:56:17 +02:00
antirez d3ddeff2d2 redis-trib create: use CONFIG SET-CONFIG-EPOCH before joining the cluster.
This way there is no need for the conflict resolution algo to be used in
order to start with a cluster where each node has a different
configEpoch.
2014-05-12 15:56:17 +02:00
antirez ce5e625a5e Sentinel: Add "dir /tmp" directive in example sentinel.conf. 2014-05-12 15:56:17 +02:00
antirez 34b68b7e89 redis-trib import: trap MIGRATE errors. 2014-05-12 15:56:17 +02:00
antirez bdfa26fb51 redis-trib.rb: MIGRATE hardcoded timeout set to 15 sec.
Will be configurable / adaptive at some point but let's start with a
saner value compared to 1 sec which is not a good idea for big data
structures stored into a single key.
2014-05-12 15:56:17 +02:00
antirez 0a707babb1 RESTORE: reply with -BUSYKEY special error code.
The error when the target key is busy was a generic one, while it makes
sense to be able to distinguish between the target key busy error and
the others easily.
2014-05-12 15:56:17 +02:00
antirez 85d2a8091c Cluster: initial ability to import data from standalone instance. 2014-05-12 15:56:17 +02:00
antirez fafe29cd4c CLUSTER MEET: better error messages when address is invalid.
Fixes issue #1734.
2014-05-12 15:56:17 +02:00
antirez 5fef9b63d2 redis-trib: allow support for mandatory options. 2014-05-12 15:56:16 +02:00
antirez 8522f295fa DEBUG POPULATE: call dictExpand() to avoid useless rehashing. 2014-05-09 15:05:26 +02:00
antirez 83c92d50a6 Cluster: bulk-accept new nodes connections.
The same change was operated for normal client connections. This is
important for Cluster as well, since when a node rejoins the cluster,
when a partition heals or after a restart, it gets flooded with new
connection attempts by all the other nodes trying to form a full
mesh again.
2014-05-09 15:05:26 +02:00
antirez ed18587f52 Cluster: clusterAcceptHandler() comments updated to match the code. 2014-05-09 15:05:26 +02:00
antirez 6eadeb7236 Sentinel: log when a failover will be attempted again.
When a Sentinel performs a failover (successful or not), or when a
Sentinel votes for a different Sentinel trying to start a failover, it
sets a min delay before it will try to get elected for a failover.

While not strictly needed, because if multiple Sentinels will try
to failover the same master at the same time, only one configuration
will eventually win, this serialization is practically very useful.
Normal failovers are cleaner: one Sentinel starts to failover, the
others update their config when the Sentinel performing the failover
is able to get the selected slave to move from the role of slave to the
one of master.

However currently this timeout was implicit, so users could see
Sentinels not reacting, after a failed failover, for some time, without
giving any feedback in the logs to the poor sysadmin waiting for clues.

This commit makes Sentinels more verbose about the delay: when a master
is down and a failover attempt is not performed because the delay has
still not elaped, something like that will be logged:

    Next failover delay: I will not start a failover
    before Thu May  8 16:48:59 2014
2014-05-08 16:49:41 +02:00
antirez c738f0f4fc Sentinel: generate +config-update-from event when a new config is received.
This event makes clear, before the switch-master event is generated,
that a Sentinel received a configuration update from another Sentinel.
2014-05-08 16:49:41 +02:00
antirez d8ffd2b2ba REDIS_ENCODING_EMBSTR_SIZE_LIMIT set to 39.
The new value is the limit for the robj + SDS header + string +
null-term to stay inside the 64 bytes Jemalloc arena in 64 bits
systems.
2014-05-08 15:40:43 +02:00
antirez 35b94422f6 Scripting test: check that Lua can call commands rewirting argv.
SPOP, tested in the new test, is among the commands rewritng the
client->argv argument vector (it gets rewritten as SREM) for command
replication purposes.

Because of recent optimizations to client->argv caching in the context
of the Lua internal Redis client, it is important to test for SPOP to be
callable from Lua without bad effects to the other commands.
2014-05-07 16:12:38 +02:00
antirez 1255460c96 Test: handle new osx 'leaks' error.
Sometimes the process is still there but no longer in a state that can
be checked (after being killed). This used to happen after a call to
SHUTDOWN NOSAVE in the scripting unit, causing a false positive.
2014-05-07 16:12:38 +02:00
antirez baf930176b Scripting: objects caching for Lua c->argv creation.
Reusing small objects when possible is a major speedup under certain
conditions, since it is able to avoid the malloc/free pattern that
otherwise is performed for every argument in the client command vector.
2014-05-07 16:12:38 +02:00
antirez 77503edf10 Scripting: Use faster API for Lua client c->argv creation.
Replace the three calls to Lua API lua_tostring, lua_lua_strlen,
and lua_isstring, with a single call to lua_tolstring.

~ 5% consistent speed gain measured.
2014-05-07 16:12:38 +02:00
antirez d757d400c1 Scripting: don't call lua_gc() after Lua script run.
Calling lua_gc() after every script execution is too expensive, and
apparently does not make the execution smoother: the same peak latency
was measured before and after the commit.

This change accounts for scripts execution speedup in the order of 10%.
2014-05-07 16:12:38 +02:00
antirez ef164836b1 Scripting: cache argv in luaRedisGenericCommand().
~ 4% consistently measured speed improvement.
2014-05-07 16:12:38 +02:00
antirez 5c291ba9f6 Fixed missing c->bufpos reset in luaRedisGenericCommand().
Bug introduced when adding a fast path to avoid copying the reply buffer
for small replies that fit into the client static buffer.
2014-05-07 16:12:38 +02:00
antirez 4b1cb0c44f Scripting: replace tolower() with faster code in evalGenericCommand().
The function showed up consuming a non trivial amount of time in the
profiler output. After this change benchmarking gives a 6% speed
improvement that can be consistently measured.
2014-05-07 16:12:38 +02:00
antirez b1f3bf8858 Scripting: luaRedisGenericCommand() fast path for buffer-only replies.
When the reply is only contained in the client static output buffer, use
a fast path avoiding the dynamic allocation of an SDS string to
concatenate the client reply objects.
2014-05-07 16:12:38 +02:00
antirez 473a8bd7d8 Define HAVE_ATOMIC for clang. 2014-05-07 16:12:38 +02:00
antirez 650b48ea7c Scripting: simpler reply buffer creation in luaRedisGenericCommand().
It if faster to just create the string with a single sdsnewlen() call.
If c->bufpos is zero, the call will simply be like sdsemtpy().
2014-05-07 16:12:38 +02:00
antirez b7a6a6c486 Redis 2.9.52 (Redis 3.0.0 beta-3). 2014-05-05 10:29:00 +02:00
antirez 402b1170bc Test: cluster/base, check that we can write/read from cluster. 2014-05-05 09:37:39 +02:00
antirez f188fa4a15 Cluster: Tcl cluster client: handle MOVED/ASK. 2014-05-05 09:37:39 +02:00
antirez 5069e0b9cd Cluster: Tcl cluster client: slots-nodes map and close method.
Now the client is able to actually run commands in a Redis Cluster
assuming the slots->nodes map is stable.
2014-05-05 09:37:39 +02:00
antirez 9fe28612b9 Cluster: Tcl cluster client: build nodes representation. 2014-05-05 09:37:39 +02:00
antirez a5b7065bdf Cluster: Tcl cluster client: get nodes description. 2014-05-05 09:37:39 +02:00
antirez 1826167519 Cluster: Tcl cluster client key -> hashslot. 2014-05-05 09:37:39 +02:00
antirez 43570e9319 Cluster test: Tcl cluster library initial skeleton. 2014-05-05 09:37:39 +02:00
antirez 4ca7c5bcde Cluster test: check for state=ok after slot allocation. 2014-05-05 09:37:39 +02:00
antirez c4c7389fea CLUSTER SET-CONFIG-EPOCH implemented.
Initially Redis Cluster accepted that after cluster creation all the
nodes were at configEpoch 0, evolving from zero as failovers happen.

However later the semantic was made more strict in order to make sure a
cluster has always all the master nodes with a different configEpoch,
which is more robust in some corner case (especially resulting from
errors by the system administrator).

To assign different configEpochs to different nodes at startup was a
task performed naturally by the config conflicts resolution algorithm
(see the Cluster specification). However this works well only for small
clusters or when there are actually just a few collisions, since it is
designed for exceptional cases.

When a large cluster is created hundred of nodes can be at epoch 0, so
the conflict resolution code is slow to provide an unique config to each
node. For this reason this new command was introduced. It can be called
only when a node is totally fresh: no other nodes known, and configEpoch
set to zero, so it is safe even against misuses.

redis-trib will use the new command in order to start the cluster
already setting an incremental unique config to every node.
2014-05-05 09:37:39 +02:00
antirez 6fc17adcd8 Cluster test: slots allocation. 2014-05-05 09:37:39 +02:00
antirez 0906ca8a3e Cluster test: use 20 instances.
This makes tests a bit slower, but it is better to test things at a
decent scale instead of using just a few nodes, and for a few tests we
actually need so many nodes.
2014-05-05 09:37:39 +02:00
antirez 402815cc3f Cluster / Sentinel test: instances count moved to run.tcl. 2014-05-05 09:37:39 +02:00
antirez 6e60e2467b Cluster test: config epoch conflict resolution. 2014-05-05 09:37:39 +02:00
antirez b1f6f2ace0 Cluster test: auto-discovery to form full mesh. 2014-05-05 09:37:39 +02:00
antirez e244bee60c Cluster test: check that every node has a different ID. 2014-05-05 09:37:39 +02:00
antirez 6b4e9558bd Cluster test: basic cluster nodes info access functions. 2014-05-05 09:37:39 +02:00
antirez a11ae385e9 CLIENT LIST speedup via peerid caching + smart allocation.
This commit adds peer ID caching in the client structure plus an API
change and the use of sdsMakeRoomFor() in order to improve the
reallocation pattern to generate the CLIENT LIST output.

Both the changes account for a very significant speedup.
2014-04-28 18:17:02 +02:00
antirez bb00154f0e Use sdscatfmt() in getClientInfoString() to make it faster. 2014-04-28 18:17:02 +02:00
antirez e285a57073 Added new sdscatfmt() %u and %U format specifiers.
This commit also fixes a bug in the implementation of sdscatfmt()
resulting from stale references to the SDS string header after
sdsMakeRoomFor() calls.
2014-04-28 18:17:02 +02:00
antirez 0ab06bb500 sdscatfmt() added to SDS library.
sdscatprintf() relies on printf() family libc functions and is sometimes
too slow in critical code paths. sdscatfmt() is an alternative which is:

1) Far less capable.
2) Format specifier uncompatible.
3) Faster.

It is suitable to be used in those speed critical code paths such as
CLIENT LIST output generation.
2014-04-28 18:17:02 +02:00
antirez eeab60d1d5 Test: fixed scripting.tcl test false positive. 2014-04-28 18:17:02 +02:00
antirez cdd2bd5632 Process events with processEventsWhileBlocked() when blocked.
When we are blocked and a few events a processed from time to time, it
is smarter to call the event handler a few times in order to handle the
accept, read, write, close cycle of a client in a single pass, otherwise
there is too much latency added for clients to receive a reply while the
server is busy in some way (for example during the DB loading).
2014-04-28 18:17:02 +02:00
antirez 357b039fdb Accept multiple clients per iteration.
When the listening sockets readable event is fired, we have the chance
to accept multiple clients instead of accepting a single one. This makes
Redis more responsive when there is a mass-connect event (for example
after the server startup), and in workloads where a connect-disconnect
pattern is used often, so that multiple clients are waiting to be
accepted continuously.

As a side effect, this commit makes the LOADING, BUSY, and similar
errors much faster to deliver to the client, making Redis more
responsive when there is to return errors to inform the clients that the
server is blocked in an not interruptible operation.
2014-04-28 18:17:02 +02:00
antirez ba86119414 AE_ERR -> ANET_ERR in acceptUnixHandler().
No actual changes since the value is the same.
2014-04-28 18:17:02 +02:00
antirez a8464828b6 Redis Cluster test framework skeleton. 2014-04-28 18:17:02 +02:00
antirez 3acae3dee3 While ANET_ERR is -1, check syscall retval for -1 itself. 2014-04-28 18:17:02 +02:00
antirez b008863edb clusterLoadConfig() REDIS_ERR retval semantics refined.
We should return REDIS_ERR to signal we can't read the configuration
because there is no config file only after checking errno, othewise
we risk to rewrite an existing file that was not accessible for some
other reason.
2014-04-28 18:17:02 +02:00
antirez 71d71814a9 Lock nodes.conf to avoid multiple processes using the same file.
This was a common source of problems among users.
The solution adopted is not bullet-proof as if the user deletes the
nodes.conf file manually, and starts a new instance with the same
nodes.conf file path, two instances will use the same file. However
following this reasoning the user may drop a nuclear bomb into the
datacenter as well.
2014-04-28 18:17:02 +02:00
antirez a0050e5030 Sentinel test files / directories layout improved.
The test now runs in a self-contained directory.
The general abstractions to run the tests in an environment where
mutliple instances are executed at the same time was extrapolated into
instances.tcl, that will be reused to test Redis Cluster.
2014-04-28 18:17:02 +02:00
antirez 3bff0291d8 Test: spawn_instance now supports additional config. 2014-04-28 18:17:02 +02:00
Matt Stancliffandantirez 08174e730f Add test for deleting an expired key
Verify proper expire-before-delete behavior.

This test passes with the expire-before-delete commit and fails
without it.
2014-04-23 16:14:34 +02:00
Matt Stancliffandantirez e687621e15 Check key expiration before deleting
Deleting an expired key should return 0, not success.

Fixes #1648
2014-04-23 16:14:34 +02:00
Glauber Costaandantirez 0c8fdb349c fix null pointer access with no file pointer
I happen to be working on a system that lacks urandom. While the code does try
to handle this case and artificially create some bytes if the file pointer is
empty, it does try to close it unconditionally, leading to a segfault.
2014-04-23 15:35:07 +02:00
kingsumosandantirez 19ac9af09f fix cluster node description showing wrong slot allocation 2014-04-23 15:35:07 +02:00
Kevin Menardandantirez 7b25ff4bf7 Fixed typos. 2014-04-23 15:34:47 +02:00
antirez 7b670805cf Missing return REDIS_ERR added to processMultibulkBuffer().
When we set a protocol error we should return with REDIS_ERR to let the
caller know it should stop processing the client.

Bug found in a code auditing related to issue #1699.
2014-04-23 10:21:30 +02:00
antirez e99e45be38 redis-cli help.h updated. 2014-04-22 16:14:59 +02:00
antirez 25f72e9e0d generate-command-help.rb updated with new hyperloglog group. 2014-04-22 16:14:59 +02:00
antirez aff94a706e Fuzzy test for ZREMRANGEBYLEX added. 2014-04-18 16:14:34 +02:00
antirez 5b5dd73093 ZREMRANGEBYLEX memory leak removed calling zslFreeLexRange(). 2014-04-18 16:14:34 +02:00
antirez ca6c793844 PFCOUNT multi-key test added. 2014-04-18 16:14:34 +02:00
antirez 82556c1687 Speedup hllRawSum() processing 8 bytes per iteration.
The internal HLL raw encoding used by PFCOUNT when merging multiple keys
is aligned to 8 bits (1 byte per register) so we can exploit this to
improve performances by processing multiple bytes per iteration.

In benchmarks the new code was several times faster with HLLs with many
registers set to zero, while no slowdown was observed with populated
HLLs.
2014-04-18 16:14:34 +02:00
antirez 5e928eff64 Speedup SUM(2^-reg[m]) in HyperLogLog computation.
When the register is set to zero, we need to add 2^-0 to E, which is 1,
but it is faster to just add 'ez' at the end, which is the number of
registers set to zero, a value we need to compute anyway.
2014-04-18 16:14:34 +02:00
antirez 00726b2e0e PFCOUNT support for multi-key union. 2014-04-18 16:14:34 +02:00
antirez 57b34a73c2 HyperLogLog low level merge extracted from PFMERGE. 2014-04-18 16:14:34 +02:00
antirez 816e22bc6f ZREMRANGEBYLEX implemented. 2014-04-18 16:14:34 +02:00
antirez d28b239fe9 Always pass sorted set range objects by reference. 2014-04-18 16:14:34 +02:00
antirez 69d0585d61 ZREMRANGE* commands refactored into a single generic function. 2014-04-18 16:14:34 +02:00
antirez 53ca563ff7 Sorted set lex ranges stress tester. 2014-04-18 16:14:34 +02:00
antirez 5e1224dc1d Basic ZRANGEBYLEX / ZLEXCOUNT tests. 2014-04-17 00:08:35 +02:00
antirez 964faba650 Pass by pointer and release of lex ranges.
Given that the code was written with a 2 years pause... something
strange happened in the middle. So there was no function to free a
lex range min/max objects, and in some places the range was passed by
value.
2014-04-17 00:08:35 +02:00
antirez 5e21acec54 ZLEXCOUNT implemented.
Like ZCOUNT for lexicographical ranges.
2014-04-16 15:09:47 +02:00
antirez 00aa96f34b More HyperLogLog tests. 2014-04-16 15:09:47 +02:00
antirez a3848a7429 HyperLogLog invalid representation error code set to INVALIDOBJ. 2014-04-16 15:09:47 +02:00
antirez 030336d4ef PFDEBUG TODENSE added.
Converts HyperLogLogs from sparse to dense. Used for testing.
2014-04-16 15:09:47 +02:00
antirez e4c2afb9c0 User-defined switch point between sparse-dense HLL encodings. 2014-04-16 15:09:47 +02:00
antirez e19e078c77 PFSELFTEST improved with sparse encoding checks. 2014-04-16 15:09:47 +02:00
antirez 66f67208d3 PFDEBUG ENCODING added. 2014-04-16 15:09:47 +02:00
antirez db929041b5 Set HLL_SPARSE_MAX to 3000.
After running a few benchmarks, 3000 looks like a reasonable value to
keep HLLs with a few thousand elements small while the CPU cost is
still not huge.

This covers all the cases where the dense representation would use N
orders of magnitude more space, like in the case of many HLLs with
carinality of a few tens or hundreds.

It is not impossible that in the future this gets user configurable,
however it is easy to pick an unreasoable value just looking at savings
in the space dimension without checking what happens in the time
dimension.
2014-04-16 15:09:47 +02:00
antirez 90b67e416a Error message for invalid HLL objects unified. 2014-04-16 15:09:47 +02:00
antirez fdace9bc50 PFMERGE fixed to work with sparse encoding. 2014-04-16 15:09:47 +02:00
antirez 64fbd15706 Mark PFDEBUG as write command in the commands table.
It is safer since it is able to have side effects.
2014-04-16 15:09:47 +02:00
antirez 159926a170 Correctly replicate PFDEBUG GETREG.
Even if it is a debugging command, make sure that when it forces a
change in encoding, the command is propagated.
2014-04-16 15:09:47 +02:00
antirez fc33779283 Added assertion in hllSparseAdd() when promotion to dense occurs.
If we converted to dense, a register must be updated in the dense
representation.
2014-04-16 15:09:47 +02:00
antirez 67e15a0ff7 hllSparseAdd(): speed optimization.
Mostly by reordering opcodes check conditional by frequency of opcodes
in larger sparse-encoded HLLs.
2014-04-16 15:09:47 +02:00
antirez ed52bbe134 Detect corrupted sparse HLLs in hllSparseSum(). 2014-04-16 15:09:46 +02:00
antirez fd137b0781 hllSparseAdd(): faster code removing conditional.
Bottleneck found profiling. Big run time improvement found when testing
after the change.
2014-04-16 15:09:46 +02:00
antirez 8b3d5a8eb0 Comment typo in hllSparseAdd(). first -> fits. 2014-04-16 15:09:46 +02:00
antirez 46f9a78d9c Merge adjacent VAL opcodes in hllSparseAdd().
As more values are added splitting ZERO or XZERO opcodes, try to merge
adjacent VAL opcodes if they have the same value.
2014-04-16 15:09:46 +02:00
antirez 9f12467c9c More robust HLL_SPARSE macros protecting 'p' with parens.
Now the macros will work with arguments such as "ptr+1".
2014-04-16 15:09:46 +02:00
antirez 10bc0aea8d hllSparseAdd() opcode seek stop condition fixed. 2014-04-16 15:09:46 +02:00
antirez 8be9394ec7 Fixed error message generation in PFDEBUG GETREG.
Bulk length for registers was emitted too early, so if there was a bug
the reply looked like a long array with just one element, blocking the
client as result.
2014-04-16 15:09:46 +02:00
antirez 8cc09325c8 Fixed memmove() count in hllSparseAdd(). 2014-04-16 15:09:46 +02:00
antirez da084bdf84 hllSparseAdd(): more correct dense conversion conditional.
We want to promote if the total string size exceeds the resulting size
after the upgrade.
2014-04-16 15:09:46 +02:00
antirez a7dbd28cff hllSparseToDense(): sanity check added.
The function checks if all the HLL_REGISTERS were processed during the
convertion from sparse to dense encoding, returning REDIS_OK or
REDIS_ERR to signal a corruption problem.

A bug in PFDEBUG GETREG was fixed: when the object is converted to the
dense representation we need to reassign the new pointer to the header
structure pointer.
2014-04-16 15:09:46 +02:00
antirez cf500998e8 PFDEBUG DECODE added.
Provides a human readable description of the opcodes composing a
run-length encoded HLL (sparse encoding).
The command is only useful for debugging / development tasks.
2014-04-16 15:09:46 +02:00
antirez 2c4a1eccda PFDEBUG added, PFGETREG removed.
PFDEBUG will be the interface to do debugging tasks with a key
containing an HLL object.
2014-04-16 15:09:46 +02:00
antirez 5786846122 hllSparseToDense API changed to take ref to object.
The new API takes directly the object doing everything needed to
turn it into a dense representation, including setting the new
representation as object->ptr.
2014-04-16 15:09:46 +02:00
antirez 68fb3019a0 hllSparseAdd() sanity check for span != 0 added. 2014-04-16 15:09:46 +02:00
antirez e7e6aa49d0 Fix hllSparseAdd() new sequence replacement when next is NULL.
sdsIncrLen() must be called anyway even if we are replacing the last
oppcode of the sparse representation.
2014-04-16 15:09:46 +02:00
antirez b964380817 Fix seqlen computation in hllSparseAdd(). 2014-04-16 15:09:46 +02:00
antirez 1c6671ab90 Abstract hllSparseAdd() / hllDenseAdd() via hllAdd(). 2014-04-16 15:09:46 +02:00
antirez e3f5a38695 hllSparseSum(): multiply 1 * runlen for zero entries. 2014-04-16 15:09:46 +02:00
antirez 5b69b984b3 Macro HLL_SPARSE_XZERO_LEN fixed. 2014-04-16 15:09:46 +02:00
antirez 2c1fa11124 Fix HLL sparse object creation #2.
Two vars initialized to wrong values in createHLLObject().
2014-04-16 15:09:46 +02:00
antirez 0b9f09c4e0 Increment pointer while iterating sparse HLL object. 2014-04-16 15:09:46 +02:00
antirez e526d32635 Fix HLL sparse object creation.
The function didn't considered the fact that each XZERO opcode is
two bytes.
2014-04-16 15:09:46 +02:00
antirez ccbabf01e1 Create HyperLogLog objects with sparse encoding. 2014-04-16 15:09:46 +02:00
antirez df5782f4d0 HyperLogLog sparse to dense conversion function. 2014-04-16 15:09:46 +02:00
antirez e872394176 HyperLogLog sparse representation initial implementation.
Code never tested, but the basic layout is shaped in this commit.
Also missing:

1) Sparse -> Dense conversion function.
2) New HLL object creation using the sparse representation.
3) Implementation of PFMERGE for the sparse representation.
2014-04-16 15:09:46 +02:00
antirez 4e12ecdba5 hllCount() refactored to support multiple representations. 2014-04-16 15:09:46 +02:00
antirez 05b5032302 hllAdd() refactored into two functions.
Also dense representation access macro renamed accordingly.
2014-04-16 15:09:46 +02:00
antirez 4407ca6fc8 HyperLogLog refactoring to support different encodings.
Metadata are now placed at the start of the representation as an header.
There is a proper structure to access the representation.
Still work to do in order to truly abstract the implementation from the
representation, commands still work assuming dense representation.
2014-04-16 15:09:46 +02:00
antirez 099ae637ab HyperLogLog sparse representation slightly modified.
After running a few simulations with different alternative encodings,
it was found that the VAL opcode performs better using 5 bits for the
value and 2 bits for the run length, at least for cardinalities in the
range of interest.
2014-04-16 15:09:46 +02:00
antirez 2b531f729d HyperLogLog sparse representation description and macros. 2014-04-16 15:09:46 +02:00
antirez 437cddee0d Add casting to match printf format.
adjustOpenFilesLimit() and clusterUpdateSlotsWithConfig() that were
assuming uint64_t is the same as unsigned long long, which is true
probably for all the systems out there that we target, but still GCC
emitted a warning since technically they are two different types.
2014-04-16 15:09:46 +02:00
antirez 109ec9c6a5 ZRANGEBYLEX and ZREVRANGEBYLEX implementation. 2014-04-16 15:09:46 +02:00
antirez f5bdaf366e PFCOUNT: always unshare/decode the object.
This will be a non-op most of the times since the object will be
unshared / decoded, however it is more technically correct to start this
way since the object may be decoded even in the read-only code path.
2014-04-16 15:09:46 +02:00
antirez ae6103ebf6 tryObjectEncoding() refactoring.
We also avoid to re-create an object that is already in EMBSTR encoding.
2014-04-16 15:09:46 +02:00
antirez 63ba5f5a8c Changed HyperLogLog hash seed to a non-zero value.
Using a seed of zero has the side effect of having the empty string
hashing to what is a very special case in the context of HyperLogLog: a
very long run of zeroes.

This did not influenced the correctness of the result with 16k registers
because of the harmonic mean, but still it is inconvenient that a so
obvious value maps to a so special hash.

The seed 0xadc83b19 is used instead, which is the first 64 bits of the
SHA1 of the empty string.

Reference: issue #1657.
2014-04-16 15:09:46 +02:00
antirez e0389f031e Initial HyperLogLog tests. 2014-04-16 15:09:45 +02:00
antirez 5056450b02 Return "WRONGTYPE" error on PF* type mismatch. 2014-04-16 15:09:45 +02:00
antirez f553fc442f Fix PFADD infinite loop.
We need to guarantee that the last bit is 1, otherwise an element may
hash to just zeroes with probability 1/(2^64) and trigger an infinite
loop.

See issue #1657.
2014-04-16 15:09:45 +02:00
antirez 19938b9103 Make hll-gnuplot-graph.rb callable from cli. 2014-04-16 15:09:45 +02:00
antirez 628b9fa535 Remove HyperLogLog type checking duplicated code. 2014-04-16 15:09:45 +02:00
antirez aecb59b0b2 PFGETREG added for testing purposes.
The new command allows to get a dump of the registers stored
into an HyperLogLog data structure for testing / debugging purposes.
2014-04-16 15:09:45 +02:00
antirez e6fda1c9ed PFCOUNT: unshare the object when cached cardinality is modified. 2014-04-16 15:09:45 +02:00
antirez 9c7bb351ab PFSELFTEST improved to test the approximation error. 2014-04-16 15:09:45 +02:00
antirez 80d7f7081c hll-gnuplot-graph.rb improved with new filter.
The function to generate graphs is also more flexible as now includes
step and max value. The step of the samples generation function is no
longer limited to min step of 1000.
2014-04-16 15:09:45 +02:00
antirez 6415fe5623 HyperLogLog: added magic / version.
This will allow future changes like compressed representations.
Currently the magic is not checked for performance reasons but this may
change in the future, for example if we add new types encoded in strings
that may have the same size of HyperLogLogs.
2014-04-16 15:09:45 +02:00
Raymond Myersandantirez f403fca654 Fixed pfadd/pfcount commands emitting hll* events instead of pf* events 2014-04-16 15:09:45 +02:00
Raymond Myersandantirez 21c9372454 Change HLL* to PF* in error messages 2014-04-16 15:09:45 +02:00
antirez f307a51049 Include redis.h before other stuff in hyperloglog.c.
Otherwise fmacros.h is included later and this may break compilation on
different systems.
2014-04-16 15:09:45 +02:00
antirez 08516c1ab6 HyperLogLog API prefix modified from "P" to "PF".
Using both the initials of Philippe Flajolet instead of just "P".
2014-04-16 15:09:45 +02:00
antirez 0c01e293ef Makefile.dep updated with hyperloglog.o deps. 2014-04-16 15:09:45 +02:00
antirez e262442fd4 HyperLogLog: make API use the P prefix in honor of Philippe Flajolet. 2014-04-16 15:09:45 +02:00
antirez 8ad9123a14 HLLMERGE fixed by adding a... missing loop! 2014-04-16 15:09:45 +02:00
antirez 9cb2b5003f hll-gnuplot-graph.rb: added new filter "all". 2014-04-16 15:09:45 +02:00
antirez 07daaa6ec4 HyperLogLog apply bias correction using a polynomial.
Better results can be achieved by compensating for the bias of the raw
approximation just after 2.5m (when LINEARCOUNTING is no longer used) by
using a polynomial that approximates the bias at a given cardinality.

The curve used was found using this web page:

    http://www.xuru.org/rt/PR.asp

That performs polynomial regression given a set of values.
2014-04-16 15:09:45 +02:00
antirez 691846e38d HLLMERGE implemented.
Merge N HLL data structures by selecting the max value for every
M[i] register among the set of HLLs.
2014-04-16 15:09:45 +02:00
antirez 7f18689a00 HLLCOUNT is technically a write command
When we update the cached value, we need to propagate the command and
signal the key as modified for WATCH.
2014-04-16 15:09:45 +02:00
antirez 6672e101e8 HLLADD: propagate write when only variable name is given.
The following form is given:

    HLLADD myhll

No element is provided in the above case so if 'myhll' var does not
exist the result is to just create an empty HLL structure, and no update
will be performed on the registers.

In this case, the DB should still be set dirty and the command
propagated.
2014-04-16 15:09:45 +02:00
antirez df2c02785a hll-gnuplot-graph.rb: Use |error| when filter is :max 2014-04-16 15:09:45 +02:00
antirez dc774e1bdc Ignore txt files inside utils/hyperloglog.
Those are generated to trace graphs using gnuplot.
2014-04-16 15:09:45 +02:00
antirez 00cb4c5dee HyperLogLog: use LINEARCOUNTING up to 3m.
The HyperLogLog original paper suggests using LINEARCOUNTING for
cardinalities < 2.5m, however for P=14 the median / max error
curves show that a value of '3' is the best pick for m = 16384.
2014-04-16 15:09:45 +02:00
antirez 2c1b4dcd40 hll-gnuplot-graph.rb added to plot HyperLogLog error graphs. 2014-04-16 15:09:45 +02:00
antirez 7acccc7ef9 HyperLogLog approximated cardinality caching.
The more we add elements to an HyperLogLog counter, the smaller is
the probability that we actually update some register.

From this observation it is easy to see how it is possible to use
caching of a previously computed cardinality and reuse it to serve
HLLCOUNT queries as long as no register was updated in the data
structure.

This commit does exactly this by using just additional 8 bytes for the
data structure to store a 64 bit unsigned integer value cached
cardinality. When the most significant bit of the 64 bit integer is set,
it means that the value computed is no longer usable since at least a
single register was modified and we need to recompute it at the next
call of HLLCOUNT.

The value is always stored in little endian format regardless of the
actual CPU endianess.
2014-04-16 15:09:45 +02:00
antirez 8be30eca17 String value unsharing refactored into proper function.
All the Redis functions that need to modify the string value of a key in
a destructive way (APPEND, SETBIT, SETRANGE, ...) require to make the
object unshared (if refcount > 1) and encoded in raw format (if encoding
is not already REDIS_ENCODING_RAW).

This was cut & pasted many times in multiple places of the code. This
commit puts the small logic needed into a function called
dbUnshareStringValue().
2014-04-16 15:09:45 +02:00
antirez f3aa086b6d Use endian neutral hash function for HyperLogLog.
We need to be sure that you can save a dataset in a Redis instance,
reload it in a different architecture, and continue to count in the same
HyperLogLog structure.

So 32 and 64 bit, little or bit endian, must all guarantee to output the
same hash for the same element.
2014-04-16 15:09:45 +02:00
antirez f704678d07 HyperLogLog internal representation modified.
The new representation is more obvious, starting from the LSB of the
first byte and using bits going to MSB, and passing to next byte as
needed.

There was also a subtle error: first two bits were unused, everything
was carried over on the right of two bits, even if it worked because of
the code requirement of always having a byte more at the end.

During the rewrite the code was made safer trying to avoid undefined
behavior due to shifting an uint8_t for more than 8 bits.
2014-04-16 15:09:45 +02:00
antirez c1f97815ca Remove a few useless operations from hllCount() fast path. 2014-04-16 15:09:45 +02:00
antirez c16f638dc5 HLLCOUNT 3x faster taking fast path for default params. 2014-04-16 15:09:45 +02:00
antirez e6d57660ed Use processor base types in HLL_(GET|SET)_REGISTER.
This speedups the macros by a noticeable factor.
2014-04-16 15:09:45 +02:00
antirez 650b21fb2c HyperLogLog: use precomputed table for 2^(-M[i]). 2014-04-16 15:09:45 +02:00
antirez 6aa1b23269 hll-err.rb: speedup using pipelining. 2014-04-16 15:09:44 +02:00
antirez a0b27c329a hll-err.rb added to test error rate of Redis HyperLogLog. 2014-04-16 15:09:44 +02:00
antirez 7d04f823e1 HyperLogLog algorithm fixed in two ways.
There was an error in the computation of 2^register, and the sequence of
zeroes computed after the hashing did not included the "1".
2014-04-16 15:09:44 +02:00
antirez 93e5876a72 HLLCOUNT implemented. 2014-04-16 15:09:44 +02:00
antirez bb57749a86 HLLADD implemented. 2014-04-16 15:09:44 +02:00
antirez 0faae58091 hllAdd() low level HyperLogLog "add" implemented. 2014-04-16 15:09:44 +02:00
antirez c03e7ceaaa HyperLogLog: redefine constants using "P". 2014-04-16 15:09:44 +02:00
antirez f4839dfb27 HLL_SET_REGISTER fixed.
There was an error in the first version of the macro.
Now the HLLSELFTEST test reports success.
2014-04-16 15:09:44 +02:00
antirez 87dd31427c Use REDIS_HLL_REGISTER_MAX when possible. 2014-04-16 15:09:44 +02:00
antirez 3610af42b2 HLL_(SET|GET)_REGISTER types fixed. 2014-04-16 15:09:44 +02:00
antirez c7344144f4 HLLSELFTEST command implemented.
To test the bitfield array of counters set/get macros from the Redis Tcl
suite is hard, so a specialized command that is able to test the
internals was developed.
2014-04-16 15:09:44 +02:00
antirez be20651435 HyperLogLog: initial sketch of registers access. 2014-04-16 15:09:44 +02:00
antirez 8c6ce3dd6a Cluster: last_vote_epoch -> lastVoteEpoch.
Use cammel case for epochs that are persisted on disk.
2014-04-16 15:09:44 +02:00
antirez 1080e272de Cluster: save/restore vars that must persist after recovery.
This fixes issue #1479.
2014-04-16 15:09:44 +02:00
antirez 47fbbd9fbc Cluster: handshake "already known" error logged to VERBOSE.
This is not really an error but something that always happens for
example when creating a new cluster, or if the sysadmin rejoins manually
a node that is already known.

Since useless logs don't help, moved to VERBOSE level.
2014-04-16 15:09:44 +02:00
antirez 6875a15858 Cluster: clusterHandleConfigEpochCollision() fixed.
New config epochs must always be obtained incrementing the currentEpoch,
that is itself guaranteed to be >= the max configEpoch currently known
to the node.
2014-04-16 15:09:44 +02:00
antirez 74fd89b321 Cluster: better logging for clusterUpdateSlotsConfigWith(). 2014-04-16 15:09:44 +02:00
antirez cf8f72c184 Cluster: CLUSTER SETSLOT implementation comment updated.
Update the comment since the implementation details changed.
2014-04-16 15:09:44 +02:00
antirez 6837863902 Cluster: redis-trib cluster allocation more even across nodes.
redis-trib used to allocate slots not considering fractions of nodes
when computing the slots_per_node amount. So the fractional part was
carried over till the end of the allocation, where the last node
received a few more slots than any other (or a lot more if the cluster
was composed of many nodes).

The computation was changed to allocate slots more evenly when they are
not exactly divisible for the number of masters we have.
2014-04-16 15:09:44 +02:00
antirez 431573ae98 Cluster: configEpoch collisions resolution.
The slave election in Redis Cluster guarantees that slaves promoted to
masters always end with unique config epochs, however failures during
manual reshardings, software bugs and operational errors may in theory
cause two nodes to have the same configEpoch.

This commit introduces a mechanism to eventually always end with different
configEpochs if a collision ever happens.

As a (wanted) side effect, this also ensures that after a new cluster
is created, all nodes will end with a different configEpoch automatically.
2014-04-16 15:09:44 +02:00
antirez 8a3dde208d Cluster: stay within 80 cols. 2014-04-16 15:09:44 +02:00
antirez 046e691a6f Fix off by one bug in freeMemoryIfNeeded() eviction pool.
Bug found by the continuous integration test running the Redis
with valgrind:

==6245== Invalid read of size 8
==6245==    at 0x4C2DEEF: memcpy@GLIBC_2.2.5 (mc_replace_strmem.c:876)
==6245==    by 0x41F9E6: freeMemoryIfNeeded (redis.c:3010)
==6245==    by 0x41D2CC: processCommand (redis.c:2069)

memmove() size argument was accounting for an extra element, going
outside the bounds of the array.
2014-03-25 10:32:50 +01:00
antirez 74d86dc13b Test: do not complain when "leaks" can't run because process died. 2014-03-25 09:34:07 +01:00
antirez c69c40a4ae adjustOpenFilesLimit() refactoring.
In this commit:
* Decrement steps are semantically differentiated from the reserved FDs.
  Previously both values were 32 but the meaning was different.
* Make it clear that we save setrlimit errno.
* Don't explicitly handle wrapping of 'f', but prevent it from
  happening.
* Add comments to make the function flow more readable.

This integrates PR #1630
2014-03-25 09:07:17 +01:00
Matt Stancliffandantirez b2a02cc92d Fix potentially incorrect errno usage
errno may be reset by the previous call to redisLog, so capture
the original value for proper error reporting.
2014-03-25 09:07:17 +01:00
Matt Stancliffandantirez 01fe750c6a Add REDIS_MIN_RESERVED_FDS define for open fds
Also update the original REDIS_EVENTLOOP_FDSET_INCR to
include REDIS_MIN_RESERVED_FDS. REDIS_EVENTLOOP_FDSET_INCR
exists to make sure more than (maxclients+RESERVED) entries
are allocated, but we can only guarantee that if we include
the current value of REDIS_MIN_RESERVED_FDS as a minimum
for the INCR size.
2014-03-25 09:07:17 +01:00
Matt Stancliffandantirez 1e7b99809b Fix infinite loop on startup if ulimit too low
Fun fact: rlim_t is an unsigned long long on all platforms.

Continually subtracting from a rlim_t makes it get smaller
and smaller until it wraps, then you're up to 2^64-1.

This was causing an infinite loop on Redis startup if
your ulimit was extremely (almost comically) low.

The case of (f > oldlimit) would never be met in a case like:

    f = 150
    while (f > 20) f -= 128

Since f is unsigned, it can't go negative and would
take on values of:

    Iteration 1: 150 - 128 => 22
    Iteration 2:  22 - 128 => 18446744073709551510
    Iterations 3-∞: ...

To catch the wraparound, we use the previous value of f
stored in limit.rlimit_cur.  If we subtract from f and
get a larger number than the value it had previously,
we print an error and exit since we don't have enough
file descriptors to help the user at this point.

Thanks to @bs3g for the inspiration to fix this problem.
Patches existed from @bs3g at antirez#1227, but I needed to repair a few other
parts of Redis simultaneously, so I didn't get a chance to use them.
2014-03-25 09:07:17 +01:00
Matt Stancliffandantirez f701a347b8 Improve error handling around setting ulimits
The log messages about open file limits have always
been slightly opaque and confusing.  Here's an attempt to
fix their wording, detail, and meaning.  Users will have a
better understanding of how to fix very common problems
with these reworded messages.

Also, we handle a new error case when maxclients becomes less
than one, essentially rendering the server unusable.  We
now exit on startup instead of leaving the user with a server
unable to handle any connections.

This fixes antirez#356 as well.
2014-03-25 09:07:17 +01:00
Matt Stancliffandantirez 6f4be45997 Replace magic 32 with REDIS_EVENTLOOP_FDSET_INCR
32 was the additional number of file descriptors Redis
would reserve when managing a too-low ulimit.  The
number 32 was in too many places statically, so now
we use a macro instead that looks more appropriate.

When Redis sets up the server event loop, it uses:
    server.maxclients+REDIS_EVENTLOOP_FDSET_INCR

So, when reserving file descriptors, it makes sense to
reserve at least REDIS_EVENTLOOP_FDSET_INCR FDs instead
of only 32.  Currently, REDIS_EVENTLOOP_FDSET_INCR is
set to 128 in redis.h.

Also, I replaced the static 128 in the while f < old loop
with REDIS_EVENTLOOP_FDSET_INCR as well, which results
in no change since it was already 128.

Impact: Users now need at least maxclients+128 as
their open file limit instead of maxclients+32 to obtain
actual "maxclients" number of clients.  Redis will carve
the extra REDIS_EVENTLOOP_FDSET_INCR file descriptors it
needs out of the "maxclients" range instead of failing
to start (unless the local ulimit -n is too low to accomidate
the request).
2014-03-25 09:07:17 +01:00
Matt Stancliffandantirez 49b576cb6c Fix maxclients error handling
Everywhere in the Redis code base, maxclients is treated
as an int with (int)maxclients or `maxclients = atoi(source)`,
so let's make maxclients an int.

This fixes a bug where someone could specify a negative maxclients
on startup and it would work (as well as set maxclients very high)
because:

    unsigned int maxclients;
    char *update = "-300";
    maxclients = atoi(update);
    if (maxclients < 1) goto fail;

But, (maxclients < 1) can only catch the case when maxclients
is exactly 0.  maxclients happily sets itself to -300, which isn't
-300, but rather 4294966996, which isn't < 1, so... everything
"worked."

maxclients config parsing checks for the case of < 1, but maxclients
CONFIG SET parsing was checking for case of < 0 (allowing
maxclients to be set to 0).  CONFIG SET parsing is now updated to
match config parsing of < 1.

It's tempting to add a MINIMUM_CLIENTS define, but... I didn't.

These changes were inspired by antirez#356, but this doesn't
fix that issue.
2014-03-25 09:07:17 +01:00
antirez 317ec18214 Fixed undefined variable value with certain code paths.
In sentinelFlushConfig() fd could be undefined when the following if
statement was true:

        if (rewrite_status == -1) goto werr;

This could cause random file descriptors to get closed.
2014-03-24 21:10:00 +01:00
Matt Stancliffandantirez 19ecf7cdba Use LRU_CLOCK() instead of function getLRUClock()
lookupKey() uses LRU_CLOCK(), so it seems object
creation should use LRU_CLOCK() too.
2014-03-24 21:10:00 +01:00
Matt Stancliffandantirez 73c2bcca85 Sentinel: Notify user when config can't be saved 2014-03-24 21:10:00 +01:00
Matt Stancliffandantirez 65e0452886 Fix data loss when save AOF/RDB with no free space
Previously, the (!fp) would only catch lack of free space
under OS X.  Linux waits to discover it can't write until
it actually writes contents to disk.

(fwrite() returns success even if the underlying file
has no free space to write into.  All the errors
only show up at flush/sync/close time.)

Fixes antirez/redis#1604
2014-03-24 21:10:00 +01:00
Matt Stancliffandantirez 4af72d326e Cluster: Restore proper trib master iteration
This got removed in 2e5c394 during a new feature addition.

The prior commit had "break if masters.length == masters_count"
but we are guaranteed to aready have that condition met since
otherwise we would haven't gotten this far.

Without this break statement, it's possible some masters may
be forgotten and have zero replicas while other masters have
more than their requested number of replicas.

Thanks to carlos for pointing out this regression at:
https://groups.google.com/forum/#!topic/redis-db/_WVVqDw5B7c
2014-03-24 21:10:00 +01:00
Matt Stancliffandantirez ee18135a19 Cluster: Fix trib create when masters==replicas
This bug was introduced in 2e5c394f during a refactor.

It took me a while to understand what was going on with
the code, so I've refactored it further by:
  - Replacing boolean values with meaningful symbols
  - Replacing 'i' with a meaningful variable name
  - Adding the proper abort check
  - Factoring out now duplicated conditionals
  - Adding optional verbose logging (we're inside *four*
    different looping constructs, so it takes a while to
    figure out where all the moving parts are)
  - Updating comment for the section

This fixes a problem when the number of master instances
equaled the number of replica instances.  Before, when
there were equal numbers of both, nodes_count would go to
zero, but the while loop would spin in i < @replicas because
i would never be updated (because the nodes_list of each ip
was length == 0, which triggered an endless loop of
next -> i = 0 -> 0 < 1? -> true -> next -> i = 0 ...)

Thanks to carlo who found this problem at:
https://groups.google.com/forum/#!topic/redis-db/_WVVqDw5B7c
2014-03-24 21:10:00 +01:00
Jan-Erik Redigerandantirez 9fa96697a9 Fixed a few typos. 2014-03-24 21:10:00 +01:00
Matt Stancliffandantirez a7478079a1 Cluster: remove variable causing warning
GCC-4.9 warned about this, but clang didn't.

This commit fixes warning:
sentinel.c: In function 'sentinelReceiveHelloMessages':
sentinel.c:2156:43: warning: variable 'master' set but not used [-Wunused-but-set-variable]
     sentinelRedisInstance *ri = c->data, *master;
2014-03-24 21:10:00 +01:00
Jan-Erik Redigerandantirez 46904ae22b Finally fix the install_server.sh script.
Includes changes from a dozen bug reports and pull requests.
Was tested on Ubuntu, Debian and CentOS.
2014-03-24 18:12:55 +01:00
antirez 2dd8c46249 Sample and cache RSS in serverCron().
Obtaining the RSS (Resident Set Size) info is slow in Linux and OSX.
This slowed down the generation of the INFO 'memory' section.

Since the RSS does not require to be a real-time measurement, we
now sample it with server.hz frequency (10 times per second by default)
and use this value both to show the INFO rss field and to compute the
fragmentation ratio.

Practically this does not make any difference for memory profiling of
Redis but speeds up the INFO call significantly.
2014-03-24 12:03:41 +01:00
antirez ae211be98b sdscatvprintf(): Try to use a static buffer.
For small content the function now tries to use a static buffer to avoid
a malloc/free cycle that is too costly when the function is used in the
context of performance critical code path such as INFO output generation.

This change was verified to have positive effects in the execution speed
of the INFO command.
2014-03-24 10:23:55 +01:00
antirez 571d6b01de Cache uname() output across INFO calls.
Uname was profiled to be a slow syscall. It produces always the same
output in the context of a single execution of Redis, so calling it at
every INFO output generation does not make too much sense.

The uname utsname structure was modified as a static variable. At the
same time a static integer was added to check if we need to call uname
the first time.
2014-03-24 10:02:55 +01:00
antirez 06b2a389eb sdscatvprintf(): guess buflen using format length.
sdscatvprintf() uses a loop where it tries to output the formatted
string in a buffer of the initial length, if there was not enough room,
a buffer of doubled size is tried and so forth.

The initial guess for the buffer length was very poor, an hardcoded
"16". This caused the printf to be processed multiple times without a
good reason. Given that printf functions are already not fast, the
overhead was significant.

The new heuristic is to use a buffer 4 times the length of the format
buffer, and 32 as minimal size. This appears to be a good balance for
typical uses of the function inside the Redis code base.

This change improved INFO command performances 3 times.
2014-03-24 09:44:49 +01:00
antirez fd70c68c97 Add test-lru.rb to utils.
This is a program useful to evaluate the Redis LRU algorithm behavior.
2014-03-21 09:57:02 +01:00
antirez 24c350245d Use getLRUClock() instead of server.lruclock to create objects.
Thanks to Matt Stancliff for noticing this error. It was in the original
code but somehow I managed to remove the change from the commit...
2014-03-21 09:57:02 +01:00
antirez 7f274e3bfe The default maxmemory policy is now noeviction.
This is safer as by default maxmemory should just set a memory limit
without any key to be deleted, unless the policy is set to something
more relaxed.
2014-03-21 09:57:02 +01:00
antirez a219339ee0 Use 24 bits for the lru object field and improve resolution.
There were 2 spare bits inside the Redis object structure that are now
used in order to enlarge 4x the range of the LRU field.

At the same time the resolution was improved from 10 to 1 second: this
still provides 194 days before the LRU counter overflows (restarting from
zero).

This is not a problem since it only causes lack of eviction precision for
objects not touched for a very long time, and the lack of precision is
only temporary.
2014-03-21 09:57:02 +01:00
antirez 9928b53102 Default LRU samples is now 5. 2014-03-21 09:57:02 +01:00
antirez fd5e8c0132 Use new dictGetRandomKeys() API to get samples for eviction.
The eviction quality degradates a bit in my tests, but since the API is
faster, it allows to raise the number of samples, and overall is a win.
2014-03-21 09:57:02 +01:00
antirez 10c8d86242 struct dictEntry -> dictEntry. 2014-03-21 09:57:02 +01:00
antirez 26292670fa Added dictGetRandomKeys() to dict.c: mass get random entries.
This new function is useful to get a number of random entries from an
hash table when we just need to do some sampling without particularly
good distribution.

It just jumps at a random place of the hash table and returns the first
N items encountered by scanning linearly.

The main usefulness of this function is to speedup Redis internal
sampling of the key space, for example for key eviction or expiry.
2014-03-21 09:57:02 +01:00
antirez c641074a21 LRU eviction pool implementation.
This is an improvement over the previous eviction algorithm where we use
an eviction pool that is persistent across evictions of keys, and gets
populated with the best candidates for evictions found so far.

It allows to approximate LRU eviction at a given number of samples
better than the previous algorithm used.
2014-03-21 09:57:02 +01:00
antirez c9ac817cc0 Fix OBJECT IDLETIME return value converting to seconds.
estimateObjectIdleTime() returns a value in milliseconds now, so we need
to scale the output of OBJECT IDLETIME to seconds.
2014-03-21 09:57:02 +01:00
antirez 205c2ccc0c Obtain LRU clock in a resolution dependent way.
For testing purposes it is handy to have a very high resolution of the
LRU clock, so that it is possible to experiment with scripts running in
just a few seconds how the eviction algorithms works.

This commit allows Redis to use the cached LRU clock, or a value
computed on demand, depending on the resolution. So normally we have the
good performance of a precomputed value, and a clock that wraps in many
days using the normal resolution, but if needed, changing a define will
switch behavior to an high resolution LRU clock.
2014-03-21 09:57:02 +01:00
antirez 561e793472 Specify lruclock in redisServer structure via REDIS_LRU_BITS.
The padding field was totally useless: removed.
2014-03-21 09:57:02 +01:00
antirez 8f0b74910a Specify LRU resolution in milliseconds. 2014-03-21 09:57:02 +01:00
antirez 63aacbe86c Set LRU parameters via REDIS_LRU_BITS define. 2014-03-21 09:57:02 +01:00
antirez 8b6a674a5d Unify stats reset for CONFIG RESETSTAT / initServer().
Now CONFIG RESETSTAT makes sure to reset all the fields, and in the
future it will be simpler to avoid missing new fields.
2014-03-21 09:57:02 +01:00
antirez 128dcee43d Sentinel: sentinelRefreshInstanceInfo() minor refactoring.
Test sentinel.tilt condition on top and return if it is true.
This allows to remove the check for the tilt condition in the remaining
code paths of the function.
2014-03-21 09:57:02 +01:00
antirez b104015fb3 Sentinel test: 02 unit better coverage + refactoring. 2014-03-21 09:57:02 +01:00
antirez f5f281f906 Sentinel test: foreach_instance_id implements 'break'. 2014-03-21 09:57:02 +01:00
antirez f308f67703 Sentinel: instance_is_killed proc added to sentinel.tcl. 2014-03-21 09:57:02 +01:00
antirez 0774d4928a Sentinel: propagate down-after-ms changes to slaves and sentinels. 2014-03-21 09:57:02 +01:00
antirez a86e24de93 Sentinel: down-after-milliseconds is not master-specific.
addReplySentinelRedisInstance() modified so that this field is displayed
for all the kind of instances: Sentinels, Masters, Slaves.
2014-03-21 09:57:02 +01:00
antirez 9997b51ff8 Sentinel failure detection implementation improved.
Failure detection in Sentinel is ping-pong based. It used to work by
remembering the last time a valid PONG reply was received, and checking
if the reception time was too old compared to the current current time.

PINGs were sent at a fixed interval of 1 second.

This works in a decent way, but does not scale well when we want to set
very small values of "down-after-milliseconds" (this is the node
timeout basically).

This commit reiplements the failure detection making a number of
changes. Some changes are inspired to Redis Cluster failure detection
code:

* A new last_ping_time field is added in representation of instances.
  If non zero, we have an active ping that was sent at the specified
  time. When a valid reply to ping is received, the field is zeroed
  again.
* last_ping_time is not reset when we reconnect the link or send a new
  ping, so from our point of view it represents the time we started
  waiting for the instance to reply to our pings without receiving a
  reply.
* last_ping_time is now used in order to check if the instance is
  timed out. This means that we can have a node timeout of 100
  milliseconds and yet the system will work well since the new check is
  not bound to the period used to send pings.
* Pings are now sent every second, or often if the value of
  down-after-milliseconds is less than one second. With a lower limit of
  10 HZ ping frequency.
* Link reconnection code was improved. This is used in order to try to
  reconnect the link when we are at 50% of the node timeout without a
  valid reply received yet. However the old code triggered unnecessary
  reconnections when the node timeout was very small. Now that should be
  ok.

The new code passes the tests but more testing is needed and more unit
tests stressing the failure detector, so currently this is merged only
in the unstable branch.
2014-03-21 09:57:02 +01:00
antirez 04cd7f773d Sentinel: use CLIENT SETNAME when connecting to Redis.
This makes debugging / monitoring of Sentinels simpler since you can
identify sentinels in CLIENT LIST output of Redis instances.
2014-03-21 09:57:02 +01:00
antirez 32f188cdf7 redis-trib: call MIGRATE via r.client.call as fix for redis-rb API changes.
See issue #1593.

Thanks to @badboy for suggesting the direct client.call fix.
2014-03-15 11:02:59 +01:00
Matt Stancliffandantirez 0c8f0079f9 Fix segfault from accessing array out of bounds
argc == 2; argv[2] == crash
2014-03-14 22:57:29 +01:00
antirez 668a95a583 Sentinel: be safe under crash-recovery assumptions.
Sentinel's main safety argument is that there are no two configurations
for the same master with the same version (configuration epoch).

For this to be true Sentinels require to be authorized by a majority.
Additionally Sentinels require to do two important things:

* Never vote again for the same epoch.
* Never exchange an old vote for a fresh one.

The first prerequisite, in a crash-recovery system model, requires to
persist the master->leader_epoch on durable storage before to reply to
messages. This was not the case.

We also make sure to persist the current epoch in order to never reply
to stale votes requests from other Sentinels, after a recovery.

The configuration is persisted by making use of fsync(), this is
considered in the context of this code a good enough guarantee that
after a restart our durable state is restored, however this may not
always be the case depending on the kind of hardware and operating
system used.
2014-03-14 22:57:25 +01:00
antirez ca2a3b3240 Sentinel: fake PUBLISH command to receive HELLO messages.
Now the way HELLO messages are received is unified.
Now it is no longer needed for Sentinels to converge to the higher
configuration for a master to be able to chat via some Redis instance,
the are able to directly exchanges configurations.

Note that this commit does not include the (trivial) change needed to
send HELLO messages to Sentinel instances as well, since for an error I
committed the change in the previous commit that refactored hello
messages processing into a separated function.
2014-03-14 11:09:32 +01:00
antirez ee513ee958 Sentinel: HELLO processing refactored into sentinelProcessHelloMessage(). 2014-03-14 11:09:32 +01:00
antirez 9e82461af0 Linenoise updated, multiline mode enabled in redis-cli. 2014-03-13 15:11:41 +01:00
antirez e0ef510630 Redis 2.9.51 (Redis 3.0.0 beta-2). 2014-03-11 16:13:39 +01:00
antirez e917de12d0 Cluster: flag the transaction as dirty for the new redirections. 2014-03-11 15:19:00 +01:00
antirez 07036ad45e redis-trib: new subcommand 'call'. Exec command in all nodes.
Example:

./redis-trib.rb call 192.168.1.11:7000 config get cluster-node-timeout
2014-03-11 15:02:41 +01:00
antirez a02424e29b redis-trib: create subcommand is now able to assign spare slaves.
Example: if the user will try to configure a cluster with 9 nodes,
asking for 1 slave for master, redis-trib will configure a 4 masters
cluster with 1 slave each as usually, but this time will assign the
spare node as a slave of one of the masters.
2014-03-11 15:02:41 +01:00
antirez 412a759a83 Cluster: update node configEpoch on UPDATE messages.
The UPDATE message contains the configEpoch of the node configuration
advertised in the packet. Update it if needed.
2014-03-11 15:02:41 +01:00
antirez 48702e0011 Cluster: set slot error if we receive an update for a busy slot.
By manually modifying nodes configurations in random ways, it is possible
to create the following scenario:

A is serving keys for slot 10
B is manually configured to serve keys for slot 10

A receives an update from B (or another node) where it is informed that
the slot 10 is now claimed by B with a greater configuration epoch,
however A still has keys from slot 10.

With this commit A will put the slot in error setting it in IMPORTING
state, so that redis-trib can detect the issue.
2014-03-11 15:02:41 +01:00
antirez efd0346ea3 Cluster: clarified a comment in clusterUpdateSlotsConfigWith(). 2014-03-11 15:02:41 +01:00
antirez 47e3f1f16c Cluster: flush importing/migrating state when master is turned into slave. 2014-03-11 15:02:41 +01:00
antirez 117557192e Cluster: clusterCloseAllSlots() added. 2014-03-11 15:02:41 +01:00
antirez 01eee56f4f DEBUG ERROR implemented.
The new "error" subcommand of the DEBUG command can reply with an user
selected error, specified as its sole argument:

    DEBUG ERROR "LOADING please wait..."

The error is generated just prefixing the command argument with a "-"
character, and replacing newlines with spaces (since error replies can't
include newlines).

The goal of the command is to help in Client libraries unit tests by
making simple to simulate a command call triggering a given error.
2014-03-11 11:10:33 +01:00
antirez ab8e1bbcdc DEBUG CMDKEYS: provide some guarantee to getKeysFromCommand().
getKeysFromCommand() is designed to be called with the command arguments
passing the basic arity checks described in the command table.

DEBUG CMDKEYS must provide the same guarantees for calling
getKeysFromCommand() to be safe.
2014-03-11 11:10:33 +01:00
antirez 94129415bf Cluster: make sortGetKeys() able to handle multiple STORE options.
It does not make sense to pass multiple store options, so, better to
handle it ;-)
2014-03-11 11:10:33 +01:00
antirez db0d9f4326 DEBUG CMDKEYS added for getKeysFromCommand() testing.
Examples:

    redis 127.0.0.1:6379> debug cmdkeys set foo bar
    1) "foo"
    redis 127.0.0.1:6379> debug cmdkeys mget a b c
    1) "a"
    2) "b"
    3) "c"
    redis 127.0.0.1:6379> debug cmdkeys zunionstore foo 2 a b
    1) "a"
    2) "b"
    3) "foo"
    redis 127.0.0.1:6379> debug cmdkeys ping
    (empty list or set)
2014-03-11 11:10:33 +01:00
antirez 3b80e0a41d Cluster: don't allow BY option of SORT as well.
There is the exception of a "constant" BY pattern that is used in order
to signal to don't sort at all. In this case no lookup is needed so it
is possible to support this case in Cluster mode.
2014-03-11 11:10:33 +01:00
antirez 399fca8f45 Cluster: SORT get keys helper implemented. 2014-03-11 11:10:33 +01:00
antirez 0c38b3c934 Cluster: evalGetKeys() fixed: was not setting keys count. 2014-03-11 11:10:33 +01:00
antirez d7451a0110 Cluster: don't allow GET option in cluster mode.
The commit also refactors a bit the error handling during SORT option
parsing.
2014-03-11 11:10:33 +01:00
antirez c3aff0c20a Fixed memory leak in SORT LIMIT option argument parsing on error. 2014-03-11 11:10:33 +01:00
antirez fdf737e132 Cluster: getKeysFromCommand() top comment improved. 2014-03-11 11:10:33 +01:00
antirez 81efa0d296 Cluster: evalGetKey() added for EVAL/EVALSHA.
Previously we used zunionInterGetKeys(), however after this function was
fixed to account for the destination key (not needed when the API was
designed for "diskstore") the two set of commands can no longer be served
by an unique keys-extraction function.
2014-03-11 11:10:09 +01:00
antirez 0618d26b89 Cluster: getKeysFromCommand() and related: top-comments added. 2014-03-11 11:10:09 +01:00
antirez a2a72b87e0 Cluster: getKeysFromCommand() API cleaned up.
This API originated from the "diskstore" experiment, not for Redis
Cluster itself, so there were legacy/useless things trying to
differentiate between keys that are going to be overwritten and keys
that need to be fetched from disk (preloaded).

All useless with Cluster, so removed with the result of code
simplification.
2014-03-11 11:10:09 +01:00
antirez c8485703f5 Cluster: some zunionInterGetKeys() comment trimmed.
Everything was pretty clear again from the initial statements.
2014-03-11 11:10:09 +01:00
antirez d610d2343d Cluster: abort on port too high error.
It also fixes multi-line comment style to be consistent with the rest of
the code base.

Related to #1555.
2014-03-11 11:10:09 +01:00
antirez 2a951ce502 Cluster: be explicit about passing NULL as bind addr for connect.
The code was already correct but it was using that bindaddr[0] is set to
NULL as a side effect of current implementation if no bind address is
configured. This is not guarnteed to hold true in the future.
2014-03-11 11:10:09 +01:00
antirez 9c9914d779 Cluster: log error when anetTcpNonBlockBindConnect() fails. 2014-03-11 11:10:09 +01:00
antirez 3119f4f694 Cluster: better timeout and retry time for failover.
When node-timeout is too small, in the order of a few milliseconds,
there is no way the voting process can terminate during that time, so we
set a lower limit for the failover timeout of two seconds.

The retry time is set to two times the failover timeout time, so it is
at least 4 seconds.
2014-03-11 11:10:09 +01:00
Matt Stancliffandantirez 14f77b343a Fix key extraction for z{union,inter}store
The previous implementation wasn't taking into account
the storage key in position 1 being a requirement (it
was only counting the source keys in positions 3 to N).

Fixes antirez/redis#1581
2014-03-11 11:10:09 +01:00
antirez 63fc5dc8b1 Typo in sentinel.conf, exists -> exits. 2014-03-11 11:10:09 +01:00
antirez afe28cfd75 Cluster: fix conditional generating TRYAGAIN error. 2014-03-11 11:10:09 +01:00
antirez aa5898f53e Redis Cluster: support for multi-key operations. 2014-03-11 11:10:09 +01:00
Matt Stancliffandantirez c0915ad1a0 Reset op_sec_last_sample_ops when reset requested
This value needs to be set to zero (in addition to
stat_numcommands) or else people may see
a negative operations per second count after they
run CONFIG RESETSTAT.

Fixes antirez/redis#1577
2014-03-11 11:10:09 +01:00
Matt Stancliffandantirez 4b3c87a027 Remove redundant IP length definition
REDIS_CLUSTER_IPLEN had the same value as
REDIS_IP_STR_LEN.  They were both #define'd
to the same INET6_ADDRSTRLEN.
2014-03-11 11:10:09 +01:00
Matt Stancliffandantirez 7c8964a8cf Remove some redundant code
Function nodeIp2String in cluster.c is exactly
anetPeerToString with a pre-extracted fd.
2014-03-11 11:09:37 +01:00
Matt Stancliffandantirez 7c359449d5 Fix return value check for anetTcpAccept
anetTcpAccept returns ANET_ERR, not AE_ERR.

This isn't a physical error since both ANET_ERR
and AE_ERR are -1, but better to be consistent.
2014-03-11 11:09:37 +01:00
Jan-Erik Redigerandantirez 6766fc561e Small typo fixed 2014-03-11 11:09:37 +01:00
Matt Stancliffandantirez 9a7cf31960 Bind source address for cluster communication
The first address specified as a bind parameter
(server.bindaddr[0]) gets used as the source IP
for cluster communication.

If no bind address is specified by the user, the
behavior is unchanged.

This patch allows multiple Redis Cluster instances
to communicate when running on the same interface
of the same host.
2014-03-11 11:09:37 +01:00
zhanghaileiandantirez 503938022f refer to updateLRUClock's comment REDIS_LRU_CLOCK_MAX is 22 bits,but #define REDIS_LRU_CLOCK_MAX ((1<<21)-1) only 21 bits 2014-03-11 11:09:37 +01:00
zhanghaileiandantirez 7eec424953 FIXED a typo more thank should be more than 2014-03-11 11:09:37 +01:00
zhanghaileiandantirez 0abe98cb4d According to context,the size should be 16 rather than 64 2014-03-11 11:09:37 +01:00
Matt Stancliffandantirez a0ea8f235e Cluster: error out quicker if port is unusable
The default cluster control port is 10,000 ports higher than
the base Redis port.  If Redis is started on a too-high port,
Cluster can't start and everything will exit later anyway.
2014-03-11 11:09:37 +01:00
Matt Stancliffandantirez 6f4b5ef6d5 Fix "can't bind to address" error reporting.
Report the actual port used for the listening attempt instead of
server.port.

Originally, Redis would just listen on server.port.
But, with clustering, Redis uses a Cluster Port too,
so we can't say server.port is always where we are listening.

If you tried to launch Redis with a too-high port number (any
port where Port+10000 > 65535), Redis would refuse to start, but
only print an error saying it can't connect to the Redis port.

This patch fixes much confusions.
2014-03-11 11:09:37 +01:00
antirez 4d5ba5962c Cast saveparams[].seconds to long for %ld format specifier. 2014-03-05 11:26:46 +01:00
antirez ef4e465316 Sentinel test: set less time sensitive defaults.
This commit sets the failover timeout to 30 seconds instead of the 180
seconds default, and allows to reconfigure multiple slaves at the same
time.

This makes tests less sensible to timing, with the result that there are
less false positives due to normal behaviors that require time to
succeed or to be retried.

However the long term solution is probably some way in order to detect
when a test failed because of timing issues (for example split brain
during leader election) and retry it.
2014-03-05 10:22:07 +01:00
antirez 313f8831ed Sentinel: more aggressive failover start desynchronization.
Sentinel needs to avoid split brain conditions due to multiple sentinels
trying to get voted at the exact same time.

So far some desynchronization was provided by fluctuating server.hz,
that is the frequency of the timer function call. However the
desynchonization provided in this way was not enough when using many
Sentinel instances, especially when a large quorum value is used in
order to force a greater degree of agreement (more than N/2+1).

It was verified that it was likely to trigger a split brain
condition, forcing the system to try again after a timeout.
Usually the system will succeed after a few retries, but this is not
optimal.

This commit desynchronizes instances in a more effective way to make it
likely that the first attempt will be successful.
2014-03-05 10:22:07 +01:00
antirez 5ee2394474 CONFIG REWRITE should be logged at WARNING level. 2014-03-05 10:22:07 +01:00
antirez fa8aca5236 Sentinel test: debugging console improved. 2014-03-05 10:22:07 +01:00
antirez cf746c398c Sentinel test: initial debugging console. 2014-03-05 10:22:07 +01:00
antirez cbf51b60a2 Sentinel test: be more patient in create_redis_master_slave_cluster. 2014-03-05 10:22:07 +01:00
antirez 23a12bb8c1 Sentiel test: add test start time in output. 2014-03-05 10:22:07 +01:00
antirez d2e16801f0 Cluster: invalidate current transaction on redirections. 2014-03-05 10:22:07 +01:00
antirez 21557a0d8b Sentinel test: use 1000 as retry in initial 00 unit test. 2014-03-05 10:22:07 +01:00
antirez 290a5d7ff6 Sentinel test: initial tests in 03 unit. 2014-03-05 10:22:07 +01:00
antirez b2668466f4 Sentinel test: foreach_instance_id now supports 'continue'. 2014-03-05 10:22:07 +01:00
antirez a76bca16e6 Sentienl test: fixed typo in unit 03 top comment. 2014-03-05 10:22:07 +01:00
antirez 1d9eb47f9d Document why we update peak memory in INFO. 2014-03-05 10:22:07 +01:00
antirez e4833ed8bf Fix configEpoch assignment when a cluster slot gets "closed".
This is still code to rework in order to use agreement to obtain a new
configEpoch when a slot is migrated, however this commit handles the
special case that happens when the nodes are just started and everybody
has a configEpoch of 0. In this special condition to have the maximum
configEpoch is not enough as the special epoch 0 is not unique (all the
others are).

This does not fixes the intrinsic race condition of a failover happening
while we are resharding, that will be addressed later.
2014-03-05 10:22:07 +01:00
Matt Stancliffandantirez 7c092b679f Force INFO used_memory_peak to match peak memory
used_memory_peak only updates in serverCron every server.hz,
but Redis can use more memory and a user can request memory
INFO before used_memory_peak gets updated in the next
cron run.

This patch updates used_memory_peak to the current
memory usage if the current memory usage is higher
than the recorded used_memory_peak value.

(And it only calls zmalloc_used_memory() once instead of
twice as it was doing before.)
2014-03-05 10:22:07 +01:00
michael-grunderandantirez 23addbb5a3 Improved bigkeys with progress, pipelining and summary
This commit reworks the redis-cli --bigkeys command to provide more
information about our progress as well as output summary information
when we're done.

 - We now show an approximate percentage completion as we go
 - Hiredis pipelining is used for TYPE and SIZE retreival
 - A summary of keyspace distribution and overall breakout at the end
2014-03-05 10:22:07 +01:00
antirez a46811693d Sentinel test: Makefile target added. 2014-02-28 16:00:14 +01:00
antirez 32db37c3c4 BITPOS fuzzy testing. 2014-02-27 15:52:43 +01:00
antirez 1d62f83364 Basic BITPOS tests. 2014-02-27 15:52:43 +01:00
antirez 9104f1e672 warnigns -> warnings in redisBitpos(). 2014-02-27 15:52:43 +01:00
antirez eacc0951a2 More consistent BITPOS behavior with bit=0 and ranges.
With the new behavior it is possible to specify just the start in the
range (the end will be assumed to be the first byte), or it is possible
to specify both start and end.

This is useful to change the behavior of the command when looking for
zeros inside a string.

1) If the user specifies both start and end, and no 0 is found inside
   the range, the command returns -1.

2) If instead no range is specified, or just the start is given, even
   if in the actual string no 0 bit is found, the command returns the
   first bit on the right after the end of the string.

So for example if the string stored at key foo is "\xff\xff":

    BITPOS foo (returns 16)
    BITPOS foo 0 -1 (returns -1)
    BITPOS foo 0 (returns 16)

The idea is that when no end is given the user is just looking for the
first bit that is zero and can be set to 1 with SETBIT, as it is
"available". Instead when a specific range is given, we just look for a
zero within the boundaries of the range.
2014-02-27 15:52:43 +01:00
antirez 1f8005ca09 Initial implementation of BITPOS.
It appears to work but more stress testing, and both unit tests and
fuzzy testing, is needed in order to ensure the implementation is sane.
2014-02-27 15:52:43 +01:00
antirez cbeaf17374 Added two more BITCOUNT tests stressing misaligned access. 2014-02-27 15:52:43 +01:00
antirez 9c206cc8e8 BITCOUNT fuzzy test with random start/end added.
It was verified in practice that this test is able to stress much more
the implementation by introducing errors that were only trivially to
detect with different offsets but impossible to detect starting always
at zero and counting bits the full length of the string.
2014-02-27 15:52:43 +01:00
antirez 24265edb6c Fix misaligned word access in redisPopcount(). 2014-02-27 15:52:43 +01:00
Matt Stancliffandantirez 7e274194bf Fix IP representation in clusterMsgDataGossip 2014-02-27 15:52:43 +01:00
antirez d242b698f4 Sentinel test: add stub for unit 04. 2014-02-25 15:55:21 +01:00
antirez cd8b149dff Sentinel test: added TODO items in 02 unit. 2014-02-25 15:55:21 +01:00
antirez c12ebb4633 Sentinel test: check role at end of unit 01. 2014-02-25 15:09:46 +01:00
michael-grunderandantirez a2b3f2eae2 Update --bigkeys to use SCAN
This commit changes the findBigKeys() function in redis-cli.c to use the new
SCAN command for iterating the keyspace, rather than RANDOMKEY.  Because we
can know when we're done using SCAN, it will exit after exhausting the keyspace.
2014-02-25 15:09:46 +01:00
antirez cb7ccc9ad0 Sentinel test: kill masters instead of using DEBUG SLEEP in all tests. 2014-02-25 15:09:46 +01:00
antirez 48fa34bf9e redis-cli: also remove useless uint8_t. 2014-02-25 15:09:46 +01:00
antirez ff20d05650 redis-cli: don't use uint64_t where actually not needed.
The computation is just something to take the CPU busy, no need to use a
specific type. Since stdint.h was not included this prevented
compilation on certain systems.
2014-02-25 15:09:46 +01:00
antirez 6a95ddb248 redis-cli: check argument existence for --pattern. 2014-02-25 15:09:46 +01:00
antirez 2431b63ff4 redis-cli: --intrinsic-latency run mode added. 2014-02-25 15:09:46 +01:00
antirez 68e9597e8a redis-cli: added comments to split program in parts. 2014-02-25 15:09:46 +01:00
antirez 31ed0911fa Sentinel test: restart instances left killed by previous unit.
An unit can abort in the middle for an error. The next unit should not
assume that the instances are in a clean state, and must restart what
was left killed.
2014-02-25 10:24:16 +01:00
antirez edf48d17a2 Sentinel test: jump to next unit on test failure.
Sentinel tests are designed to be dependent on the previous tests in the
same unit, so usually we can't continue with the next test in the same
unit if a previous test failed.
2014-02-25 10:24:16 +01:00
antirez 7f521d764f Sentinel test: test majority crashing Sentinels.
The test was previously performed by removing the master from the
Sentinel monitored masters. The test with the Sentinels crashed is
more similar to real-world partitions / failures.
2014-02-25 10:24:16 +01:00
antirez 491bcbaf73 Sentinel test: restart_instance should refresh pid attrib.
Also kill_instance was modified to warn when a test will try to kill the
same instance multiple times for error.
2014-02-25 10:24:16 +01:00
antirez 55643361f1 Sentinel test: more stuff mored from 00-base to init.
The area a number of mandatory tests to craete a stable setup for
testing that is not too sensitive to timing issues. All those tests
moved to includes/init-tests, and marked as (init).
2014-02-25 10:24:16 +01:00
antirez 4f6ed3412e Sentinel: log quorum with +monitor event. 2014-02-25 10:24:16 +01:00
antirez 55a2e10b03 Sentinel test: removed useless code to set SDOWN timeout.
The new common initialization code used to start a new unit already set
the timeout to 2000 milliseconds.
2014-02-25 10:24:16 +01:00
antirez 56e18ba4f6 Sentinel: generate +monitor events at startup. 2014-02-25 10:24:16 +01:00
antirez cd68a1d45a Sentinel: log +monitor and +set events.
Now that we have a runtime configuration system, it is very important to
be able to log how the Sentinel configuration changes over time because
of API calls.
2014-02-25 10:24:16 +01:00
antirez 4af2acf2b0 Sentinel: added missing exit(1) after checking for config file. 2014-02-25 10:24:16 +01:00
antirez fecfd721df Sentinel test: tmp dir and gitignore added. 2014-02-25 10:24:16 +01:00
antirez b18109e351 Sentinel test: minor fixes to --pause-on-error. 2014-02-25 10:24:16 +01:00
antirez 2ed707bcaa Sentinel test: --pause-on-error option added.
Pause the test with running instances available for state inspection on
error.
2014-02-25 10:24:16 +01:00
antirez 2666f737b0 Sentinel test: added empty units to fill later. 2014-02-25 10:24:16 +01:00
antirez 6e2e6d5b8c Sentinel: IDONTKNOW error removed.
This error was conceived for the older version of Sentinel that worked
via master redirection and that was not able to get configuration
updates from other Sentinels via the Pub/Sub channel of masters or
slaves.

This reply does not make sense today, every Sentinel should reply with
the best information it has currently. The error will make even more
sense in the future since the plan is to allow Sentinels to update the
configuration of other Sentinels via gossip with a direct chat without
the prerequisite that they have at least a monitored instance in common.
2014-02-25 10:24:16 +01:00
antirez ec593b582e Sentinel test: framework improved and conf-update unit added.
It is now possible to kill and restart sentinel or redis instances for
more real-world testing.

The 01 unit tests the capability of Sentinel to update the configuration
of Sentinels rejoining the cluster, however the test is pretty trivial
and more tests should be added.
2014-02-25 10:24:16 +01:00
Matt Stancliffandantirez 5a8c9f94a6 Add cluster or sentinel to proc title
If you launch redis with `redis-server --sentinel` then
in a ps, your output only says "redis-server IP:Port" — this
patch changes the proc title to include [sentinel] or
[cluster] depending on the current server mode:
e.g.  "redis-server IP:Port [sentinel]"
      "redis-server IP:Port [cluster]"
2014-02-25 10:24:16 +01:00
antirez ef01bbe1f5 Sentinel test: move init tests as includes.
Most units will start with these two basic tests to create an
environment where the real tests are ran.
2014-02-25 10:24:16 +01:00
antirez 2b4bc4f900 Sentinel test: ability to run just a subset of test files. 2014-02-25 10:24:16 +01:00
Matt Stancliffandantirez 21a7f9e7ef Auto-enter slaveMode when SYNC from redis-cli
If someone asks for SYNC or PSYNC from redis-cli,
automatically enter slaveMode (as if they ran
redis-cli --slave) and continue printing the replication
stream until either they Ctrl-C or the master gets disconnected.
2014-02-25 10:24:16 +01:00
antirez 48d74f2039 Sentinel: report instances role switch events.
This is useful mostly for debugging of issues.
2014-02-20 12:28:18 +01:00
antirez 8a6774e209 Sentinel test: some reliability fixes to 00-base tests. 2014-02-20 12:28:18 +01:00
antirez 6bb42b5bfe Sentinel test: check that role matches at end of 00-base. 2014-02-20 12:28:18 +01:00
antirez 37be5987e6 Sentinel test: ODOWN and agreement. 2014-02-20 12:28:18 +01:00
antirez b037c897ae Sentinel test: check reconfig of slaves and old master. 2014-02-20 12:28:18 +01:00
antirez f7cb83778b Sentinel test: basic failover tested. Framework improvements. 2014-02-20 12:28:18 +01:00
antirez a11749f9f6 Sentinel test: basic tests for MONITOR and auto-discovery. 2014-02-20 12:28:18 +01:00
antirez 8292dd50cd Sentinel test: info fields, master-slave setup, fixes. 2014-02-20 12:28:18 +01:00
antirez 68a3d5602e Prefix test file names with numbers to force exec order. 2014-02-20 12:28:18 +01:00
antirez bbbe68aa0d Sentinel test: provide basic commands to access instances. 2014-02-20 12:28:18 +01:00
antirez 6e4662e479 Sentinel: SENTINEL_SLAVE_RECONF_RETRY_PERIOD -> RECONF_TIMEOUT
Rename define to match the new meaning.
2014-02-18 10:30:47 +01:00
antirez bd31fcf16e Sentinel: fix slave promotion timeout.
If we can't reconfigure a slave in time during failover, go forward as
anyway the slave will be fixed by Sentinels in the future, once they
detect it is misconfigured.

Otherwise a failover in progress may never terminate if for some reason
the slave is uncapable to sync with the master while at the same time
it is not disconnected.
2014-02-18 10:30:47 +01:00
antirez 2a35abcf12 Sentinel: initial testing framework.
Nothing tested at all so far... Just the infrastructure spawning N
Sentinels and N Redis instances that the test will use again and again.
2014-02-17 17:39:23 +01:00
antirez c1786693b1 Test: colorstr moved to util.tcl. 2014-02-17 17:39:19 +01:00
antirez 48af4d4f26 Test: code to test server availability refactored.
Some inline test moved into server_is_up procedure.
Also find_available_port was moved into util since it is going
to be used for the Sentinel test as well.
2014-02-17 17:39:15 +01:00
antirez 58b6dd9beb Get absoulte config file path before processig 'dir'.
The code tried to obtain the configuration file absolute path after
processing the configuration file. However if config file was a relative
path and a "dir" statement was processed reading the config, the absolute
path obtained was wrong.

With this fix the absolute path is obtained before processing the
configuration while the server is still in the original directory where
it was executed.
2014-02-17 17:39:12 +01:00
antirez c36a5dce54 Sentinel: better specify startup errors due to config file.
Now it logs the file name if it is not accessible. Also there is a
different error for the missing config file case, and for the non
writable file case.
2014-02-17 17:39:09 +01:00
antirez 3c1672da7d Update cached time in rdbLoad() callback.
server.unixtime and server.mstime are cached less precise timestamps
that we use every time we don't need an accurate time representation and
a syscall would be too slow for the number of calls we require.

Such an example is the initialization and update process of the last
interaction time with the client, that is used for timeouts.

However rdbLoad() can take some time to load the DB, but at the same
time it did not updated the time during DB loading. This resulted in the
bug described in issue #1535, where in the replication process the slave
loads the DB, creates the redisClient representation of its master, but
the timestamp is so old that the master, under certain conditions, is
sensed as already "timed out".

Thanks to @yoav-steinberg and Redis Labs Inc for the bug report and
analysis.
2014-02-13 15:13:38 +01:00
antirez 116617c5e7 Log when CONFIG REWRITE goes bad. 2014-02-13 14:33:53 +01:00
antirez 767846dc5b Test: regression for issue #1549.
It was verified that reverting the commit that fixes the bug, the test
no longer passes.
2014-02-13 12:27:10 +01:00
antirez 14143fbede Fix script cache bug in the scripting engine.
This commit fixes a serious Lua scripting replication issue, described
by Github issue #1549. The root cause of the problem is that scripts
were put inside the script cache, assuming that slaves and AOF already
contained it, even if the scripts sometimes produced no changes in the
data set, and were not actaully propagated to AOF/slaves.

Example:

    eval "if tonumber(KEYS[1]) > 0 then redis.call('incr', 'x') end" 1 0

Then:

    evalsha <sha1 step 1 script> 1 0

At this step sha1 of the script is added to the replication script cache
(the script is marked as known to the slaves) and EVALSHA command is
transformed to EVAL. However it is not dirty (there is no changes to db),
so it is not propagated to the slaves. Then the script is called again:

    evalsha <sha1 step 1 script> 1 1

At this step master checks that the script already exists in the
replication script cache and doesn't transform it to EVAL command. It is
dirty and propagated to the slaves, but they fail to evaluate the script
as they don't have it in the script cache.

The fix is trivial and just uses the new API to force the propagation of
the executed command regardless of the dirty state of the data set.

Thank you to @minus-infinity on Github for finding the issue,
understanding the root cause, and fixing it.
2014-02-13 12:16:31 +01:00
antirez 0296aab6da AOF write error: retry with a frequency of 1 hz. 2014-02-12 16:57:08 +01:00
antirez dd73a7bf43 AOF: don't abort on write errors unless fsync is 'always'.
A system similar to the RDB write error handling is used, in which when
we can't write to the AOF file, writes are no longer accepted until we
are able to write again.

For fsync == always we still abort on errors since there is currently no
easy way to avoid replying with success to the user otherwise, and this
would violate the contract with the user of only acknowledging data
already secured on disk.
2014-02-12 16:57:04 +01:00
antirez 910b6d34f2 Redis 2.9.50 (Redis 3.0.0 beta-1) 2014-02-11 10:47:54 +01:00
antirez 0725988a07 Cluster: clusterDelNode(): remove node from master's slaves. 2014-02-11 10:34:14 +01:00
antirez 4513d8fcd4 Cluster: UPDATE messages are the norm and verbose.
Logging them at WARNING level was of little utility and of sure disturb.
2014-02-11 10:22:05 +01:00
antirez 9721255178 Cluster: redis-trib fix: handling of another trivial case. 2014-02-11 10:22:02 +01:00
antirez 6d550f2de4 Cluster: configEpoch assignment in SETNODE improved.
Avoid to trash a configEpoch for every slot migrated if this node has
already the max configEpoch across the cluster.

Still work to do in this area but this avoids both ending with a very
high configEpoch without any reason and to flood the system with fsyncs.
2014-02-11 10:21:58 +01:00
antirez 585e9fb886 Cluster: clusterSetStartupEpoch() made more generally useful.
The actual goal of the function was to get the max configEpoch found in
the cluster, so make it general by removing the assignment of the max
epoch to currentEpoch that is useful only at startup.
2014-02-11 10:21:55 +01:00
antirez 8b5196addf Cluster: always increment the configEpoch in SETNODE after import.
Removed a stale conditional preventing the configEpoch from incrementing
after the import in certain conditions. Since the master got a new slot
it should always claim a new configuration.
2014-02-11 10:21:52 +01:00
antirez 2e3f6b0fb3 Cluster: on resharding upgrade version of receiving node.
The node receiving the hash slot needs to have a version that wins over
the other versions in order to force the ownership of the slot.

However the current code is far from perfect since a failover can happen
during the manual resharding. The fix is a work in progress but the
bottom line is that the new version must either be voted as usually,
set by redis-trib manually after it makes sure can't be used by other
nodes, or reserved configEpochs could be used for manual operations (for
example odd versions could be never used by slaves and are always used
by CLUSTER SETSLOT NODE).
2014-02-11 00:39:24 +01:00
antirez a221ae5ce2 Cluster: fsync at every SETSLOT command puts too pressure on disks.
During slots migration redis-trib can send a number of SETSLOT commands.
Fsyncing every time is a bit too much in production as verified
empirically.

To make sure configs are fsynced on all nodes after a resharding
redis-trib may send something like CLUSTER CONFSYNC.

In this case fsyncs were not providing too much value since anyway
processes can crash in the middle of the resharding of an hash slot, and
redis-trib should be able to recover from this condition anyway.
2014-02-11 00:39:20 +01:00
antirez 77c6fa65f1 Cluster: conditions to clear "migrating" on slot for SETSLOT ... NODE changed.
If the slot is manually assigned to another node, clear the migrating
status regardless of the fact it was previously assigned to us or not,
as long as we no longer have keys for this slot.

This avoid a race during slots migration that may leave the slot in
migrating status in the source node, since it received an update message
from the destination node that is already claiming the slot.

This way we are sure that redis-trib at the end of the slot migration is
always able to close the slot correctly.
2014-02-11 00:39:14 +01:00
antirez f406a09495 Cluster: remove debugging xputs from redis-trib. 2014-02-11 00:39:11 +01:00
antirez 01de246843 Cluster: redis-trib fix: cover new case of open slot.
The case is the trivial one a single node claiming the slot as
migrating, without nodes claiming it as importing.
2014-02-11 00:39:08 +01:00
antirez 8f25428772 redis-trib: log event after we have reference to 'master'. 2014-02-11 00:39:05 +01:00
antirez cc97305ec3 Cluster: don't update slave's master if we don't know it.
There is no way we can update the slave's node->slaveof pointer if we
don't know the master (no node with such an ID in our tables).
2014-02-11 00:39:02 +01:00
antirez fa6f4f21c3 Cluster: ignore slot config changes if we are importing it. 2014-02-11 00:38:59 +01:00
antirez 30214fff3e Cluster: update configEpoch after manually messing with slots. 2014-02-11 00:38:56 +01:00
antirez cfcb09bf76 Cluster: redis-trib, more info about open slots error. 2014-02-11 00:38:53 +01:00
antirez 8e12fae05e Cluster: fixed inverted arguments in logging function call. 2014-02-10 17:21:17 +01:00
antirez 6a01545744 Cluster: clear the FAIL status for masters without slots.
Masters without slots don't participate to the cluster but just do
redirections, no need to take them in FAIL state if they are back
reachable.
2014-02-10 17:19:16 +01:00
antirez 969a4f1db3 Cluster: replica migration should only work for masters serving slots. 2014-02-10 17:08:47 +01:00
antirez cb92a1ef08 Cluster: redis-trib del-node variable typo fixed. 2014-02-10 16:59:17 +01:00
antirez 6987a95952 Cluster: clusterReadHandler() fixed to work with new message header. 2014-02-10 16:28:44 +01:00
antirez ad85f520f8 Cluster: don't propagate PUBLISH two times.
PUBLISH both published messages via Cluster bus and replication when
cluster was enabled, resulting in duplicated message in the slave.
2014-02-10 16:05:26 +01:00
antirez b82b66b51d Cluster: signature changed to "RCmb" (Redis Cluster message bus).
Sounds better after all.
2014-02-10 16:05:22 +01:00
antirez b6e04f5584 Cluster: discard bus messages with version != 0. 2014-02-10 16:05:18 +01:00
antirez 0ee1a78c86 Cluster: added signature + version in bus packets. 2014-02-10 16:05:15 +01:00
antirez ff6a75a0c1 3.0 release notes added. 2014-02-10 15:36:02 +01:00
581 changed files with 41097 additions and 115206 deletions
-2
View File
@@ -4,7 +4,6 @@
dump.rdb
redis-benchmark
redis-check-aof
redis-check-rdb
redis-check-dump
redis-cli
redis-sentinel
@@ -27,4 +26,3 @@ deps/lua/src/liblua.a
.make-*
.prerequisites
*.dSYM
Makefile.dep
+419 -11
View File
@@ -1,16 +1,424 @@
Hello! This file is just a placeholder, since this is the "unstable" branch
of Redis, the place where all the development happens.
Redis 3.0 release notes
=======================
There is no release notes for this branch, it gets forked into another branch
every time there is a partial feature freeze in order to eventually create
a new stable release.
WARNING: Redis 3.0 is currently a BETA not suitable for production environments.
Usually "unstable" is stable enough for you to use it in development environments
however you should never use it in production environments. It is possible
to download the latest stable release here:
--------------------------------------------------------------------------------
Upgrade urgency levels:
http://download.redis.io/releases/redis-stable.tar.gz
LOW: No need to upgrade unless there are new features you want to use.
MODERATE: Program an upgrade of the server, but it's not urgent.
HIGH: There is a critical bug that may affect a subset of users. Upgrade!
CRITICAL: There is a critical bug affecting MOST USERS. Upgrade ASAP.
--------------------------------------------------------------------------------
More information is available at http://redis.io
--[ Redis 3.0.0 RC2 (version 2.9.102) ] Release date: 13 jan 2014
Happy hacking!
Upgrade urgency: LOW.
This is the second release candidate of Redis Cluster. The major changes
are back porting of things implemented into the unstable branch while
this was still possible (with the new development model adopted only
bug fixes will be merged in the future).
RC2 also fixes a few Redis Cluster non critical bugs.
>> General changes
* [FIX] A number of minor bug fixes.
* [NEW] Diskless replication backportede.
* [NEW] Lua bitops and updated cmsgpack backported.
* [NEW] Transparent Huge Pages warnings and reporting backported.
>> Cluster changes
* [FIX] Fix PUBLISH cluster bus message count field.
* [FIX] It is no longer possible to write outside node hash slots using Lua.
* [FIX] Valgrind warnings (no actual bugs).
* [FIX] Less strict in acceptiong myself->ip if it's not populated.
* [NEW] Better testing of Lua scripts.
>> Sentinel changes
No changes to Sentinel.
--[ Redis 3.0.0 RC1 (version 2.9.101) ] Release date: 9 oct 2014
This is the first release candidate of Redis Cluster.
>> General changes
* [FIX] An very large number of small fixes, old and new, merged in the
context of a the issue #1906. Please see the issue page here
for exact credits: https://github.com/antirez/redis/pull/1906
of each commit. (Matt Stancliff and many others).
* [FIX] SAVE is no longer propagated to AOF / slaves.
* [FIX] GETRANGE test no longer fails for 32 bit builds (Matt Stancliff).
* [FIX] Limit SCAN latency when the hash table is in an odd state (very few
populted buckets because rehashing is in progress). (Xiaost and
Salvatore Sanfilippo)
* [NEW] Redis is now able to load truncated AOF files without requiring a
redis-check-aof utility run. The default now is to load truncated
(but apparently not corrupted) AOFs, you can change this in redis.conf.
(Salvatore Sanfilippo).
* [NEW] DEBUG POPULATE two args form implemented. It is now possible to
call it with DEBUG POPULATE <count> <prefix>. Default prefix
is "key:" as usually.
* [NEW] INCR: Modify incremented object in-place when possible. This results
in speed improvements + possibly better memory locality.
>> Cluster changes
* [FIX] Cluster: claim ping_sent time even if we can't connect.
* [FIX] redis-trib should not abort easily on connection issues.
* [FIX] Cluster test: less console-spammy resharding test.
* [FIX] Fix logic to detect we are among a minority.
* [FIX] Process gossip section only for known nodes.
* [NEW] Redis Cluster is stable and tested enough, there is a clear MVP,
so it was promoted from beta to stable.
* [NEW] New unit 09, Pub/Sub across the cluster.
* [NEW] New unit 08, update messages.
* [NEW] New cluster option to work with partial slots coverage.
* [NEW] More chatty cluster slaves when failover is stalled. They log reason
with rate limiting, only when reason changes or a given time
has elapsed.
>> Sentinel changes
* [FIX] Sentinel critical bug fixed: the absolute majority was computed in a
wrong way because of a programming error. Now the implementation does
what the specification says and the majority to authorize a failover
(that should not be confused with the ODOWN quorum) is the majority of
*all* the Sentinels ever seen for a given master, regardless of their
current state.
* [FIX] Resolved a memory leak in the hiredis library causing a memory leak
in Redis Sentinel when a monitored instance or another Sentinel is
unavailable. Every reconnection attempt will leak a small amount of
memory, but in the long run the process can reach a considerable size.
* [NEW] Sentinel: ability to announce itself with an arbitrary IP/port to work
in the context of natted networks. However this is probably still
not enough since there is no equivalent mechanism for slaves listed
in the master INFO output. (Dara Kong and Salvatore Sanfilippo)
--[ Redis 3.0.0 Beta 8 (version 2.9.57) ] Release date: 29 jul 2014
This is the 8th beta of Redis 3.0.0.
>> General changes
* [FIX] Solaris compilation issues. (Matt Stancliff, Salvatore Sanfilippo)
* [FIX] Allow shared integer objects if maxmemory policy is not LRU based.
(Salvatore Sanfilippo)
* [FIX] PFSELFTEST: less false positives. (Salvatore Sanfilippo)
* [FIX] Fail SYNC if background save child aborted due to a signal. (Yossi
Gottlieb)
* [NEW] Latency framework backported from unstable branch. (Salvatore
Sanfilippo)
* [NEW] AOF rewrite improved for latency. (Salvatore Sanfilippo)
* [NEW] Pub/Sub PING. (Salvatore Sanfilippo)
* [NEW] Much faster ZUNIONSTORE. (Kyle Hubert, Salvatore Sanfilippo)
* [NEW] Faster ll2string() implementation. (Salvatore Sanfilippo)
>> Cluster changes
* [FIX] CLUSTER RESET: Flush slave dataset on reset. (Salvatore Sanfilippo)
* [FIX] Replica migration: don't migrate to masters that never had slaves
in the past, but only to masters that remained orphaned after
failure events. (Salvatore Sanfilippo)
* [NEW] redis-trib: allow to reshard in non-interactive way. (Salvatore
Sanfilippo)
* [NEW] Cluster test: unit 04, check consistency during resharding. (Salvatore
Sanfilippo)
* [NEW] Cluster test: unit 05, slave selection. (Salvatore Sanfilippo)
* [NEW] Cluster test: unit 06, slaves with stale data can't failover. (Salvatore
Sanfilippo)
* [NEW] Cluster test: unit 07, replicas migration. (Salvatore Sanfilippo)
>> Sentinel changes
* No Sentinel changes in this release.
--[ Redis 3.0.0 Beta 7 (version 2.9.56) ] Release date: 30 jun 2014
This is the 7th beta of Redis 3.0.0.
>> General changes
* [FIX] Scripting fixes backported from unstable, see Redis 2.8.12 changelog
for more info. (Salvatore Sanfilippo)
* [FIX] Cancel SHUTDOWN if initial AOF is being written. (Matt Stancliff)
* [NEW] New command: COMMAND, for commands introspection (Matt Stancliff &
Salvatore Sanfilippo)
* [NEW] hiredis: Update to latest version. (Matt Stancliff)
* [NEW] Jemalloc updated to 3.6.0. (Salvatore Sanfilippo)
>> Cluster changes
* [FIX] Cluster: clear NOADDR flag when updating node address.
(Salvatore Sanfilippo)
* [NEW] New CLUSTER SLOTS command to simplify Cluster clients operations.
(Matt Stancliff)
* [NEW] More Cluster tests. (Salvatore Sanfilippo)
* [NEW] Log when failover authorization are granted / denied.
(Salvatore Sanfilippo)
>> Sentinel changes
* [FIX] A few Sentinel bugs fixed and improvements, see Redis 2.8.12
changelog for more info. (Salvatore Sanfilippo & Matt Stancliff)
* [NEW] New Sentinel-Client handshake protocol, ROLE command, CLIENT KILL,
all backported to 3.0 branch. (Salvatore Sanfilippo)
--[ Redis 3.0.0 Beta 6 (version 2.9.55) ] Release date: 9 jun 2014
This is the 6th beta of Redis 3.0.0.
>> General changes
* [FIX] Fix software watchdog signal handler crash due to re-entering.
* [FIX] Better Lua number -> string conversion for Lua scripts.
* [FIX] Serious replication bug when min-slaves-* feature is used in slaves
configuration fixed.
* [FIX] Blocking pop on lists now works when the list is created by commands
other than *PUSH* (for example SORT STORE).
>> Cluster changes
* [FIX] CRITICAL: For a bug in the implementation of CLUSTER SET-CONFIG-EPOCH
introduced with beta-3 (especially beta-4 where the command
is actually used by redis-trib), a configEpoch for a node could
jump backward, breaking the eventual consistency property of the
slots -> nodes mapping in the cluster.
>> Sentinel changes
* No changes for Sentinel in this release.
--[ Redis 3.0.0 Beta 5 (version 2.9.54) ] Release date: 26 may 2014
This is the 5th beta of Redis 3.0.0. It does not include any real
worthwhile change (just three days passed since the previous beta), but
fixes two stupid bugs preventing cluster tests to pass.
--[ Redis 3.0.0 Beta 4 (version 2.9.53) ] Release date: 23 may 2014
This is the fourth beta of Redis 3.0.0.
>> General changes
* [NEW] Scripting engine performances improvements.
* [NEW] Log format slightly changed to report current node role.
* [FIX] Correct the HyperLogLog stale cache flag to prevent unnecessary
computation.
>> Cluster changes
* [NEW] redis-trib: ability to import data from standalone Redis instances.
* [NEW] redis-trib: "fix" subcommand much better at fixing errors.
* [NEW] CLUSTER FAILOVER FORCE implemented.
* [NEW] CLUSTER RESET implemented, it is now possible to completely reset
nodes and create a new cluster without restarting anything.
* [NEW] Slave validity factor (max estimated data age to still failover)
is now under the control of the user, and can be configured via
redis.conf or CONFIG SET. Option name cluster-slave-validity-factor.
* [NEW] Cluster test: failure detection and failover initial tests.
* [NEW] CLUSTER MEET: better error messages when address is invalid.
* [NEW] Bulk-accept new Cluster nodes in the Cluster bus instead of
performing just a single accept() per event fired.
* [FIX] Bypass data_age check for manual failovers.
* [FIX] Fixed data_age computation / check integer overflow.
* [FIX] Various fixes to Tcl client.tcl Redis Cluster client used in tests.
* [FIX] Better handling of stolen slots.
* [FIX] Don't accept cluster bus connections during startup.
>> Sentinel changes
* [NEW] Generate +config-update-from event when a new config is received.
* [NEW] Log when a failover will be re-attempted again.
* [FIX] Sentinel: Add "dir /tmp" directive in example sentinel.conf.
--[ Redis 3.0.0 Beta 3 (version 2.9.52) ] Release date: 5 may 2014
This is the third beta of Redis 3.0.0.
>> General changes
* [NEW] New data structure: the HyperLogLog (see 2.8 release notes).
* [NEW] Lexicograhical range queries in sorted sets (see 2.8 release notes).
* [NEW] LRU algorithm precision greatly improved.
* [FIX] Redis is now much more responsive to reply with LOADING / BUSY errors.
>> Cluster changes
* [NEW] Cluster testing framework and initial tests.
* [NEW] Cluster epochs collision resolution (make Redis Cluster more resilient
to admin and programming errors).
* [NEW] Persist / fsync some global state to ensure correct crash-recovery
semantics.
* [NEW] New command introduced: CLUSTER SET-CONFIG-EPOCH, still not used
by redis-trib. Will be used to speedup the assignment of unique
epochs to different nodes at cluster creation time. For now this is
handled as a side effect of the cluster epochs collision resolution.
* [FIX] Different fixes to redis-trib cluster creation.
* [FIX] Fix an error in the CLUSTER NODES output for nodes slots.
>> Sentinel changes
* [NEW] Sentinels are now able to send update messages in a peer-to-peer
fashion even if no Redis instances are available. Now the Sentinel
liveness property that the most updated configuration in a given
partition is propagated to all the Sentinels is extended to partitions
without reachable instances.
* [NEW] Sentinel safety properties are now ensured in a crash-recovery system
model since some state is persisted on disk before replying to other
nodes, and reloaded at startup.
* [NEW] Sentinel now uses CLIENT SETNAME so that it is easy to identify
Sentinels using CLIENT LIST among other clients.
* [NEW] Sentinel failure detection and reconnection code improved.
--[ Redis 3.0.0 Beta 2 (version 2.9.51) ] Release date: 11 mar 2014
This is the second beta of Redis 3.0.0.
>> General changes
* [FIX] Sometimes the absolute config file path was obtained in a wrong way.
This happened when there was a "dir" directive inside the config file
and at the same time the configuration file was given as a relative
path to redis-server or redis-sentinel executables.
* [FIX] redis-cli: Automatically enter --slave mode when SYNC or PSYNC are
called during an interactive session.
* [FIX] BITCOUNT: fixed unaligned access causing issues in sparc and other
archs not capable of dealing with unaligned accesses. This also makes
the code faster in archs where unaligned accesses are allowed.
* [FIX] Force INFO used_memory_peak to match peak memory. This generated some
confusion among users even if it was not an actual bug.
* [FIX] Fixed an critical EVALSHA script cache bug: scripts executed may not
propagate to AOF / Slaves correctly under certain conditions.
See issue #1549 at Github for more information.
* [FIX] Fixed multiple bugs resulting into closing the link with master or slave
during replication without good reasons. This will result in useless
resynchronizations, or infinite loops where the replication link can't
be established.
* [FIX] Don't count the time needed to populate the buffers of clients waiting
in MONITOR mode when populating the Slow Log entries.
* [NEW] Redis-cli updated to use SCAN instead of random sampling via
RANDOMKEY in order to implement --bigkeys feature. Moreover the
implementation now supports pipelining and reports more information
at the end of the scan. Much faster, much better. A special thank
you to Michael Grunder for this improvement.
* [NEW] redis-cli now supports a new --intrinsic-latency mode that is able
to meter the latency of a system due to kernel / hypervisor.
How to use it is explained at http://redis.io/topics/latency.
* [NEW] New command BITPOS: find first bit set or clear in a bitmap.
* [NEW] CONFIG REWRITE calls are now logged.
* [NEW] AOF write errors (like no space on device) no longer abort Redis if the
fsync policy is none or every second. The database enters a read-only
mode where every write is refused with an error. Normal operations are
restored as soon as Redis is able to append again data to the AOF file.
* [NEW] Sentinel now accepts SHUTDOWN command.
>> Cluster changes
* [FIX] Bind the first interface listed in the "bind" configuration directive
if any, in order to perform outgoing connections. This fixes Cluster
usage when an address is bound but there are multiple interfaces that
may be used to connect with other nodes.
* [FIX] When an "Importing" slot is closed via CLUSTER SETSLOT NODE ...
increment the configEpoch in the special case it is zero.
* [FIX] Current transaction is invalidated on redirection errors.
* [FIX] Abort if port does not allow for a valid cluster bus port that is
always at fixed +10000 offset.
* [FIX] Keys extraction algorithm fixed for ZUNIONSTORE/ZINTERSTORE and SORT.
* [FIX] Better failover timeout and retry times: failover should now work
reliabily when node-timeout is very small (a few milliseconds).
* [FIX] Don't allow SORT GET/BY options in Cluster mode.
* [FIX] Clear importing/migrating state when turning from master to slave role.
* [FIX] Set slot error if we receive an update for a busy slot.
* [FIX] Update node configEpoch on UPDATE messages.
* [NEW] Support multi-key operations as long as keys resolve to the same
hash slot, and the slot is not migrating, or it is migrating but all
the mentioned keys are available.
* [NEW] New DEBUG command CMDKEYS available to debug / test keys identification
in Redis commands.
* [NEW] redis-trib: create subcommand is now able to assign spare slaves.
* [NEW] redis-trib: new subcommand 'call'. Exec command in all nodes.
>> Sentinel changes
* [FIX] Sentinel "IDONTKNOW" error removed as it does not made sense with the
new Sentinel design. This error was actually a fix for a design error
in the first implementation of Sentinel.
* [FIX] Sentinel: added a missing exit() call to abort after config file
checks at startup. This error was introduced with an improvement in
a previous 2.8 release.
* [FIX] Sentinel: better nodes fail over start time desynchronization to avoid
split-brain during the voting process needed to get authorization to
fail over. This means the system is less likely to need to retry
and will fail over faster. No changes in behavior / correctness.
* [NEW] Sentinel unit tests and framework. More tests needed and units must
be improved in order to have less false positives, but it is a start
and features a debugging console that is useful to fix tests or to
inspect bugs causing tests failures.
* [NEW] New Sentinel events: +/-monitor and +set used to monitor when an
instance to monitor is added or removed, or when a configuration
is modified via SENTINEL SET.
--[ Redis 3.0.0 Beta 1 (version 2.9.50) ] Release date: 11 Feb 2014
This is the first beta of Redis 3.0.0.
The following is a list of improvements in Redis 3.0, compared to Redis 2.8.
* [NEW] Redis Cluster: a distributed implementation of a subset of Redis.
* [NEW] New "embedded string" object encoding resulting in less cache
misses. Big speed gain under certain work loads.
* [NEW] WAIT command to block waiting for a write to be transmitted to
the specified number of slaves.
* [NEW] MIGRATE connection caching. Much faster keys migraitons.
* [NEW] MIGARTE new options COPY and REPLACE.
* [NEW] CLIENT PAUSE command: stop processing client requests for a
specified amount of time.
Migrating from 2.8 to 3.0
=========================
Redis 3.0 is mostly a strict subset of 2.8, you should not have any problem
upgrading your application from 2.8 to 3.0. However this is a list of small
non-backward compatible changes introduced in the 3.0 release:
* The log format was modified. The prefix of each line included the pid
in the following format [1234]. Now instead it is 1234:? Where
'?' is actually the role of the instance. M for master, S for slave, C
if this process is a saving child (for RDB/AOF), and X for Sentinel.
--------------------------------------------------------------------------------
Credits: Where not specified the implementation and design is done by
Salvatore Sanfilippo. Thanks to Pivotal for making all this possible.
Also many thanks to all the other contributors and the amazing community
we have.
See commit messages for more credits.
Cheers,
Salvatore
+8 -14
View File
@@ -8,33 +8,27 @@ each source file that you contribute.
# IMPORTANT: HOW TO USE REDIS GITHUB ISSUES
* Github issues SHOULD ONLY BE USED to report bugs, and for DETAILED feature
requests. Everything else belongs to the Redis Google Group:
https://groups.google.com/forum/m/#!forum/Redis-db
requests. Everything else belongs to the Redis Google Group.
PLEASE DO NOT POST GENERAL QUESTIONS that are not about bugs or suspected
bugs in the Github issues system. We'll be very happy to help you and provide
all the support at the Reddit sub:
all the support in the Redis Google Group.
http://reddit.com/r/redis
There is also an active community of Redis users at Stack Overflow:
http://stackoverflow.com/questions/tagged/redis
Redis Google Group address:
https://groups.google.com/forum/?fromgroups#!forum/redis-db
# How to provide a patch for a new feature
1. If it is a major feature or a semantical change, please post it as a new submission in r/redis on Reddit at http://reddit.com/r/redis. Try to be passionate about why the feature is needed, make users upvote your proposal to gain traction and so forth. Read feedbacks about the community. But in this first step **please don't write code yet**.
1. Drop a message to the Redis Google Group with a proposal of semantics/API.
2. If in step 1 you get an acknowledgment from the project leaders, use the
2. If in step 1 you get an acknowledge from the project leaders, use the
following procedure to submit a patch:
a. Fork Redis on github ( http://help.github.com/fork-a-repo/ )
b. Create a topic branch (git checkout -b my_branch)
c. Push to your branch (git push origin my_branch)
d. Initiate a pull request on github ( https://help.github.com/articles/creating-a-pull-request/ )
d. Initiate a pull request on github ( http://help.github.com/send-pull-requests/ )
e. Done :)
For minor fixes just open a pull request on Github.
Thanks!
+1 -1
View File
@@ -1,4 +1,4 @@
Copyright (c) 2006-2015, Salvatore Sanfilippo
Copyright (c) 2006-2014, Salvatore Sanfilippo
All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
+173
View File
@@ -0,0 +1,173 @@
Where to find complete Redis documentation?
-------------------------------------------
This README is just a fast "quick start" document. You can find more detailed
documentation at http://redis.io
Building Redis
--------------
Redis can be compiled and used on Linux, OSX, OpenBSD, NetBSD, FreeBSD.
We support big endian and little endian architectures.
It may compile on Solaris derived systems (for instance SmartOS) but our
support for this platform is "best effort" and Redis is not guaranteed to
work as well as in Linux, OSX, and *BSD there.
It is as simple as:
% make
You can run a 32 bit Redis binary using:
% make 32bit
After building Redis is a good idea to test it, using:
% make test
Fixing build problems with dependencies or cached build options
—--------
Redis has some dependencies which are included into the "deps" directory.
"make" does not rebuild dependencies automatically, even if something in the
source code of dependencies is changes.
When you update the source code with `git pull` or when code inside the
dependencies tree is modified in any other way, make sure to use the following
command in order to really clean everything and rebuild from scratch:
make distclean
This will clean: jemalloc, lua, hiredis, linenoise.
Also if you force certain build options like 32bit target, no C compiler
optimizations (for debugging purposes), and other similar build time options,
those options are cached indefinitely until you issue a "make distclean"
command.
Fixing problems building 32 bit binaries
---------
If after building Redis with a 32 bit target you need to rebuild it
with a 64 bit target, or the other way around, you need to perform a
"make distclean" in the root directory of the Redis distribution.
In case of build errors when trying to build a 32 bit binary of Redis, try
the following steps:
* Install the packages libc6-dev-i386 (also try g++-multilib).
* Try using the following command line instead of "make 32bit":
make CFLAGS="-m32 -march=native" LDFLAGS="-m32"
Allocator
---------
Selecting a non-default memory allocator when building Redis is done by setting
the `MALLOC` environment variable. Redis is compiled and linked against libc
malloc by default, with the exception of jemalloc being the default on Linux
systems. This default was picked because jemalloc has proven to have fewer
fragmentation problems than libc malloc.
To force compiling against libc malloc, use:
% make MALLOC=libc
To compile against jemalloc on Mac OS X systems, use:
% make MALLOC=jemalloc
Verbose build
-------------
Redis will build with a user friendly colorized output by default.
If you want to see a more verbose output use the following:
% make V=1
Running Redis
-------------
To run Redis with the default configuration just type:
% cd src
% ./redis-server
If you want to provide your redis.conf, you have to run it using an additional
parameter (the path of the configuration file):
% cd src
% ./redis-server /path/to/redis.conf
It is possible to alter the Redis configuration passing parameters directly
as options using the command line. Examples:
% ./redis-server --port 9999 --slaveof 127.0.0.1 6379
% ./redis-server /etc/redis/6379.conf --loglevel debug
All the options in redis.conf are also supported as options using the command
line, with exactly the same name.
Playing with Redis
------------------
You can use redis-cli to play with Redis. Start a redis-server instance,
then in another terminal try the following:
% cd src
% ./redis-cli
redis> ping
PONG
redis> set foo bar
OK
redis> get foo
"bar"
redis> incr mycounter
(integer) 1
redis> incr mycounter
(integer) 2
redis>
You can find the list of all the available commands here:
http://redis.io/commands
Installing Redis
-----------------
In order to install Redis binaries into /usr/local/bin just use:
% make install
You can use "make PREFIX=/some/other/directory install" if you wish to use a
different destination.
Make install will just install binaries in your system, but will not configure
init scripts and configuration files in the appropriate place. This is not
needed if you want just to play a bit with Redis, but if you are installing
it the proper way for a production system, we have a script doing this
for Ubuntu and Debian systems:
% cd utils
% ./install_server.sh
The script will ask you a few questions and will setup everything you need
to run Redis properly as a background daemon that will start again on
system reboots.
You'll be able to stop and start Redis using the script named
/etc/init.d/redis_<portnumber>, for instance /etc/init.d/redis_6379.
Code contributions
---
Note: by contributing code to the Redis project in any form, including sending
a pull request via Github, a code fragment or patch via private email or
public discussion groups, you agree to release your code under the terms
of the BSD license that you can find in the COPYING file included in the Redis
source distribution.
Please see the CONTRIBUTING file in this source distribution for more
information.
Enjoy!
-446
View File
@@ -1,446 +0,0 @@
This README is just a fast *quick start* document. You can find more detailed documentation at [redis.io](https://redis.io).
What is Redis?
--------------
Redis is often referred as a *data structures* server. What this means is that Redis provides access to mutable data structures via a set of commands, which are sent using a *server-client* model with TCP sockets and a simple protocol. So different processes can query and modify the same data structures in a shared way.
Data structures implemented into Redis have a few special properties:
* Redis cares to store them on disk, even if they are always served and modified into the server memory. This means that Redis is fast, but that is also non-volatile.
* Implementation of data structures stress on memory efficiency, so data structures inside Redis will likely use less memory compared to the same data structure modeled using an high level programming language.
* Redis offers a number of features that are natural to find in a database, like replication, tunable levels of durability, cluster, high availability.
Another good example is to think of Redis as a more complex version of memcached, where the operations are not just SETs and GETs, but operations to work with complex data types like Lists, Sets, ordered data structures, and so forth.
If you want to know more, this is a list of selected starting points:
* Introduction to Redis data types. http://redis.io/topics/data-types-intro
* Try Redis directly inside your browser. http://try.redis.io
* The full list of Redis commands. http://redis.io/commands
* There is much more inside the Redis official documentation. http://redis.io/documentation
Building Redis
--------------
Redis can be compiled and used on Linux, OSX, OpenBSD, NetBSD, FreeBSD.
We support big endian and little endian architectures, and both 32 bit
and 64 bit systems.
It may compile on Solaris derived systems (for instance SmartOS) but our
support for this platform is *best effort* and Redis is not guaranteed to
work as well as in Linux, OSX, and \*BSD there.
It is as simple as:
% make
You can run a 32 bit Redis binary using:
% make 32bit
After building Redis, it is a good idea to test it using:
% make test
Fixing build problems with dependencies or cached build options
---------
Redis has some dependencies which are included into the `deps` directory.
`make` does not automatically rebuild dependencies even if something in
the source code of dependencies changes.
When you update the source code with `git pull` or when code inside the
dependencies tree is modified in any other way, make sure to use the following
command in order to really clean everything and rebuild from scratch:
make distclean
This will clean: jemalloc, lua, hiredis, linenoise.
Also if you force certain build options like 32bit target, no C compiler
optimizations (for debugging purposes), and other similar build time options,
those options are cached indefinitely until you issue a `make distclean`
command.
Fixing problems building 32 bit binaries
---------
If after building Redis with a 32 bit target you need to rebuild it
with a 64 bit target, or the other way around, you need to perform a
`make distclean` in the root directory of the Redis distribution.
In case of build errors when trying to build a 32 bit binary of Redis, try
the following steps:
* Install the packages libc6-dev-i386 (also try g++-multilib).
* Try using the following command line instead of `make 32bit`:
`make CFLAGS="-m32 -march=native" LDFLAGS="-m32"`
Allocator
---------
Selecting a non-default memory allocator when building Redis is done by setting
the `MALLOC` environment variable. Redis is compiled and linked against libc
malloc by default, with the exception of jemalloc being the default on Linux
systems. This default was picked because jemalloc has proven to have fewer
fragmentation problems than libc malloc.
To force compiling against libc malloc, use:
% make MALLOC=libc
To compile against jemalloc on Mac OS X systems, use:
% make MALLOC=jemalloc
Verbose build
-------------
Redis will build with a user friendly colorized output by default.
If you want to see a more verbose output use the following:
% make V=1
Running Redis
-------------
To run Redis with the default configuration just type:
% cd src
% ./redis-server
If you want to provide your redis.conf, you have to run it using an additional
parameter (the path of the configuration file):
% cd src
% ./redis-server /path/to/redis.conf
It is possible to alter the Redis configuration by passing parameters directly
as options using the command line. Examples:
% ./redis-server --port 9999 --replicaof 127.0.0.1 6379
% ./redis-server /etc/redis/6379.conf --loglevel debug
All the options in redis.conf are also supported as options using the command
line, with exactly the same name.
Playing with Redis
------------------
You can use redis-cli to play with Redis. Start a redis-server instance,
then in another terminal try the following:
% cd src
% ./redis-cli
redis> ping
PONG
redis> set foo bar
OK
redis> get foo
"bar"
redis> incr mycounter
(integer) 1
redis> incr mycounter
(integer) 2
redis>
You can find the list of all the available commands at http://redis.io/commands.
Installing Redis
-----------------
In order to install Redis binaries into /usr/local/bin just use:
% make install
You can use `make PREFIX=/some/other/directory install` if you wish to use a
different destination.
Make install will just install binaries in your system, but will not configure
init scripts and configuration files in the appropriate place. This is not
needed if you want just to play a bit with Redis, but if you are installing
it the proper way for a production system, we have a script doing this
for Ubuntu and Debian systems:
% cd utils
% ./install_server.sh
The script will ask you a few questions and will setup everything you need
to run Redis properly as a background daemon that will start again on
system reboots.
You'll be able to stop and start Redis using the script named
`/etc/init.d/redis_<portnumber>`, for instance `/etc/init.d/redis_6379`.
Code contributions
-----------------
Note: by contributing code to the Redis project in any form, including sending
a pull request via Github, a code fragment or patch via private email or
public discussion groups, you agree to release your code under the terms
of the BSD license that you can find in the [COPYING][1] file included in the Redis
source distribution.
Please see the [CONTRIBUTING][2] file in this source distribution for more
information.
[1]: https://github.com/antirez/redis/blob/unstable/COPYING
[2]: https://github.com/antirez/redis/blob/unstable/CONTRIBUTING
Redis internals
===
If you are reading this README you are likely in front of a Github page
or you just untarred the Redis distribution tar ball. In both the cases
you are basically one step away from the source code, so here we explain
the Redis source code layout, what is in each file as a general idea, the
most important functions and structures inside the Redis server and so forth.
We keep all the discussion at a high level without digging into the details
since this document would be huge otherwise and our code base changes
continuously, but a general idea should be a good starting point to
understand more. Moreover most of the code is heavily commented and easy
to follow.
Source code layout
---
The Redis root directory just contains this README, the Makefile which
calls the real Makefile inside the `src` directory and an example
configuration for Redis and Sentinel. You can find a few shell
scripts that are used in order to execute the Redis, Redis Cluster and
Redis Sentinel unit tests, which are implemented inside the `tests`
directory.
Inside the root are the following important directories:
* `src`: contains the Redis implementation, written in C.
* `tests`: contains the unit tests, implemented in Tcl.
* `deps`: contains libraries Redis uses. Everything needed to compile Redis is inside this directory; your system just needs to provide `libc`, a POSIX compatible interface and a C compiler. Notably `deps` contains a copy of `jemalloc`, which is the default allocator of Redis under Linux. Note that under `deps` there are also things which started with the Redis project, but for which the main repository is not `antirez/redis`. An exception to this rule is `deps/geohash-int` which is the low level geocoding library used by Redis: it originated from a different project, but at this point it diverged so much that it is developed as a separated entity directly inside the Redis repository.
There are a few more directories but they are not very important for our goals
here. We'll focus mostly on `src`, where the Redis implementation is contained,
exploring what there is inside each file. The order in which files are
exposed is the logical one to follow in order to disclose different layers
of complexity incrementally.
Note: lately Redis was refactored quite a bit. Function names and file
names have been changed, so you may find that this documentation reflects the
`unstable` branch more closely. For instance in Redis 3.0 the `server.c`
and `server.h` files were named `redis.c` and `redis.h`. However the overall
structure is the same. Keep in mind that all the new developments and pull
requests should be performed against the `unstable` branch.
server.h
---
The simplest way to understand how a program works is to understand the
data structures it uses. So we'll start from the main header file of
Redis, which is `server.h`.
All the server configuration and in general all the shared state is
defined in a global structure called `server`, of type `struct redisServer`.
A few important fields in this structure are:
* `server.db` is an array of Redis databases, where data is stored.
* `server.commands` is the command table.
* `server.clients` is a linked list of clients connected to the server.
* `server.master` is a special client, the master, if the instance is a replica.
There are tons of other fields. Most fields are commented directly inside
the structure definition.
Another important Redis data structure is the one defining a client.
In the past it was called `redisClient`, now just `client`. The structure
has many fields, here we'll just show the main ones:
struct client {
int fd;
sds querybuf;
int argc;
robj **argv;
redisDb *db;
int flags;
list *reply;
char buf[PROTO_REPLY_CHUNK_BYTES];
... many other fields ...
}
The client structure defines a *connected client*:
* The `fd` field is the client socket file descriptor.
* `argc` and `argv` are populated with the command the client is executing, so that functions implementing a given Redis command can read the arguments.
* `querybuf` accumulates the requests from the client, which are parsed by the Redis server according to the Redis protocol and executed by calling the implementations of the commands the client is executing.
* `reply` and `buf` are dynamic and static buffers that accumulate the replies the server sends to the client. These buffers are incrementally written to the socket as soon as the file descriptor is writable.
As you can see in the client structure above, arguments in a command
are described as `robj` structures. The following is the full `robj`
structure, which defines a *Redis object*:
typedef struct redisObject {
unsigned type:4;
unsigned encoding:4;
unsigned lru:LRU_BITS; /* lru time (relative to server.lruclock) */
int refcount;
void *ptr;
} robj;
Basically this structure can represent all the basic Redis data types like
strings, lists, sets, sorted sets and so forth. The interesting thing is that
it has a `type` field, so that it is possible to know what type a given
object has, and a `refcount`, so that the same object can be referenced
in multiple places without allocating it multiple times. Finally the `ptr`
field points to the actual representation of the object, which might vary
even for the same type, depending on the `encoding` used.
Redis objects are used extensively in the Redis internals, however in order
to avoid the overhead of indirect accesses, recently in many places
we just use plain dynamic strings not wrapped inside a Redis object.
server.c
---
This is the entry point of the Redis server, where the `main()` function
is defined. The following are the most important steps in order to startup
the Redis server.
* `initServerConfig()` setups the default values of the `server` structure.
* `initServer()` allocates the data structures needed to operate, setup the listening socket, and so forth.
* `aeMain()` starts the event loop which listens for new connections.
There are two special functions called periodically by the event loop:
1. `serverCron()` is called periodically (according to `server.hz` frequency), and performs tasks that must be performed from time to time, like checking for timedout clients.
2. `beforeSleep()` is called every time the event loop fired, Redis served a few requests, and is returning back into the event loop.
Inside server.c you can find code that handles other vital things of the Redis server:
* `call()` is used in order to call a given command in the context of a given client.
* `activeExpireCycle()` handles eviciton of keys with a time to live set via the `EXPIRE` command.
* `freeMemoryIfNeeded()` is called when a new write command should be performed but Redis is out of memory according to the `maxmemory` directive.
* The global variable `redisCommandTable` defines all the Redis commands, specifying the name of the command, the function implementing the command, the number of arguments required, and other properties of each command.
networking.c
---
This file defines all the I/O functions with clients, masters and replicas
(which in Redis are just special clients):
* `createClient()` allocates and initializes a new client.
* the `addReply*()` family of functions are used by commands implementations in order to append data to the client structure, that will be transmitted to the client as a reply for a given command executed.
* `writeToClient()` transmits the data pending in the output buffers to the client and is called by the *writable event handler* `sendReplyToClient()`.
* `readQueryFromClient()` is the *readable event handler* and accumulates data from read from the client into the query buffer.
* `processInputBuffer()` is the entry point in order to parse the client query buffer according to the Redis protocol. Once commands are ready to be processed, it calls `processCommand()` which is defined inside `server.c` in order to actually execute the command.
* `freeClient()` deallocates, disconnects and removes a client.
aof.c and rdb.c
---
As you can guess from the names these files implement the RDB and AOF
persistence for Redis. Redis uses a persistence model based on the `fork()`
system call in order to create a thread with the same (shared) memory
content of the main Redis thread. This secondary thread dumps the content
of the memory on disk. This is used by `rdb.c` to create the snapshots
on disk and by `aof.c` in order to perform the AOF rewrite when the
append only file gets too big.
The implementation inside `aof.c` has additional functions in order to
implement an API that allows commands to append new commands into the AOF
file as clients execute them.
The `call()` function defined inside `server.c` is responsible to call
the functions that in turn will write the commands into the AOF.
db.c
---
Certain Redis commands operate on specific data types, others are general.
Examples of generic commands are `DEL` and `EXPIRE`. They operate on keys
and not on their values specifically. All those generic commands are
defined inside `db.c`.
Moreover `db.c` implements an API in order to perform certain operations
on the Redis dataset without directly accessing the internal data structures.
The most important functions inside `db.c` which are used in many commands
implementations are the following:
* `lookupKeyRead()` and `lookupKeyWrite()` are used in order to get a pointer to the value associated to a given key, or `NULL` if the key does not exist.
* `dbAdd()` and its higher level counterpart `setKey()` create a new key in a Redis database.
* `dbDelete()` removes a key and its associated value.
* `emptyDb()` removes an entire single database or all the databases defined.
The rest of the file implements the generic commands exposed to the client.
object.c
---
The `robj` structure defining Redis objects was already described. Inside
`object.c` there are all the functions that operate with Redis objects at
a basic level, like functions to allocate new objects, handle the reference
counting and so forth. Notable functions inside this file:
* `incrRefcount()` and `decrRefCount()` are used in order to increment or decrement an object reference count. When it drops to 0 the object is finally freed.
* `createObject()` allocates a new object. There are also specialized functions to allocate string objects having a specific content, like `createStringObjectFromLongLong()` and similar functions.
This file also implements the `OBJECT` command.
replication.c
---
This is one of the most complex files inside Redis, it is recommended to
approach it only after getting a bit familiar with the rest of the code base.
In this file there is the implementation of both the master and replica role
of Redis.
One of the most important functions inside this file is `replicationFeedSlaves()` that writes commands to the clients representing replica instances connected
to our master, so that the replicas can get the writes performed by the clients:
this way their data set will remain synchronized with the one in the master.
This file also implements both the `SYNC` and `PSYNC` commands that are
used in order to perform the first synchronization between masters and
replicas, or to continue the replication after a disconnection.
Other C files
---
* `t_hash.c`, `t_list.c`, `t_set.c`, `t_string.c` and `t_zset.c` contains the implementation of the Redis data types. They implement both an API to access a given data type, and the client commands implementations for these data types.
* `ae.c` implements the Redis event loop, it's a self contained library which is simple to read and understand.
* `sds.c` is the Redis string library, check http://github.com/antirez/sds for more information.
* `anet.c` is a library to use POSIX networking in a simpler way compared to the raw interface exposed by the kernel.
* `dict.c` is an implementation of a non-blocking hash table which rehashes incrementally.
* `scripting.c` implements Lua scripting. It is completely self contained from the rest of the Redis implementation and is simple enough to understand if you are familar with the Lua API.
* `cluster.c` implements the Redis Cluster. Probably a good read only after being very familiar with the rest of the Redis code base. If you want to read `cluster.c` make sure to read the [Redis Cluster specification][3].
[3]: http://redis.io/topics/cluster-spec
Anatomy of a Redis command
---
All the Redis commands are defined in the following way:
void foobarCommand(client *c) {
printf("%s",c->argv[1]->ptr); /* Do something with the argument. */
addReply(c,shared.ok); /* Reply something to the client. */
}
The command is then referenced inside `server.c` in the command table:
{"foobar",foobarCommand,2,"rtF",0,NULL,0,0,0,0,0},
In the above example `2` is the number of arguments the command takes,
while `"rtF"` are the command flags, as documented in the command table
top comment inside `server.c`.
After the command operates in some way, it returns a reply to the client,
usually using `addReply()` or a similar function defined inside `networking.c`.
There are tons of commands implementations inside the Redis source code
that can serve as examples of actual commands implementations. To write
a few toy commands can be a good exercise to familiarize with the code base.
There are also many other files not described here, but it is useless to
cover everything. We want to just help you with the first steps.
Eventually you'll find your way inside the Redis code base :-)
Enjoy!
+2 -2
View File
@@ -58,7 +58,7 @@ ifeq ($(uname_S),SunOS)
LUA_CFLAGS= -D__C99FEATURES__=1
endif
LUA_CFLAGS+= -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' $(CFLAGS)
LUA_CFLAGS+= -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL $(CFLAGS)
LUA_LDFLAGS+= $(LDFLAGS)
# lua's Makefile defines AR="ar rcu", which is unusual, and makes it more
# challenging to cross-compile lua (and redis). These defines make it easier
@@ -77,7 +77,7 @@ JEMALLOC_LDFLAGS= $(LDFLAGS)
jemalloc: .make-prerequisites
@printf '%b %b\n' $(MAKECOLOR)MAKE$(ENDCOLOR) $(BINCOLOR)$@$(ENDCOLOR)
cd jemalloc && ./configure --with-version=5.1.0-0-g0 --with-lg-quantum=3 --with-jemalloc-prefix=je_ --enable-cc-silence CFLAGS="$(JEMALLOC_CFLAGS)" LDFLAGS="$(JEMALLOC_LDFLAGS)"
cd jemalloc && ./configure --with-jemalloc-prefix=je_ --enable-cc-silence CFLAGS="$(JEMALLOC_CFLAGS)" LDFLAGS="$(JEMALLOC_LDFLAGS)"
cd jemalloc && $(MAKE) CFLAGS="$(JEMALLOC_CFLAGS)" LDFLAGS="$(JEMALLOC_LDFLAGS)" lib/libjemalloc.a
.PHONY: jemalloc
-88
View File
@@ -1,88 +0,0 @@
This directory contains all Redis dependencies, except for the libc that
should be provided by the operating system.
* **Jemalloc** is our memory allocator, used as replacement for libc malloc on Linux by default. It has good performances and excellent fragmentation behavior. This component is upgraded from time to time.
* **geohash-int** is inside the dependencies directory but is actually part of the Redis project, since it is our private fork (heavily modified) of a library initially developed for Ardb, which is in turn a fork of Redis.
* **hiredis** is the official C client library for Redis. It is used by redis-cli, redis-benchmark and Redis Sentinel. It is part of the Redis official ecosystem but is developed externally from the Redis repository, so we just upgrade it as needed.
* **linenoise** is a readline replacement. It is developed by the same authors of Redis but is managed as a separated project and updated as needed.
* **lua** is Lua 5.1 with minor changes for security and additional libraries.
How to upgrade the above dependencies
===
Jemalloc
---
Jemalloc is modified with changes that allow us to implement the Redis
active defragmentation logic. However this feature of Redis is not mandatory
and Redis is able to understand if the Jemalloc version it is compiled
against supports such Redis-specific modifications. So in theory, if you
are not interested in the active defragmentation, you can replace Jemalloc
just following tose steps:
1. Remove the jemalloc directory.
2. Substitute it with the new jemalloc source tree.
3. Edit the Makefile localted in the same directory as the README you are
reading, and change the --with-version in the Jemalloc configure script
options with the version you are using. This is required because otherwise
Jemalloc configuration script is broken and will not work nested in another
git repository.
However note that we change Jemalloc settings via the `configure` script of Jemalloc using the `--with-lg-quantum` option, setting it to the value of 3 instead of 4. This provides us with more size classes that better suit the Redis data structures, in order to gain memory efficiency.
If you want to upgrade Jemalloc while also providing support for
active defragmentation, in addition to the above steps you need to perform
the following additional steps:
5. In Jemalloc three, file `include/jemalloc/jemalloc_macros.h.in`, make sure
to add `#define JEMALLOC_FRAG_HINT`.
6. Implement the function `je_get_defrag_hint()` inside `src/jemalloc.c`. You
can see how it is implemented in the current Jemalloc source tree shipped
with Redis, and rewrite it according to the new Jemalloc internals, if they
changed, otherwise you could just copy the old implementation if you are
upgrading just to a similar version of Jemalloc.
Geohash
---
This is never upgraded since it's part of the Redis project. If there are changes to merge from Ardb there is the need to manually check differences, but at this point the source code is pretty different.
Hiredis
---
Hiredis uses the SDS string library, that must be the same version used inside Redis itself. Hiredis is also very critical for Sentinel. Historically Redis often used forked versions of hiredis in a way or the other. In order to upgrade it is advised to take a lot of care:
1. Check with diff if hiredis API changed and what impact it could have in Redis.
2. Make sure thet the SDS library inside Hiredis and inside Redis are compatible.
3. After the upgrade, run the Redis Sentinel test.
4. Check manually that redis-cli and redis-benchmark behave as expecteed, since we have no tests for CLI utilities currently.
Linenoise
---
Linenoise is rarely upgraded as needed. The upgrade process is trivial since
Redis uses a non modified version of linenoise, so to upgrade just do the
following:
1. Remove the linenoise directory.
2. Substitute it with the new linenoise source tree.
Lua
---
We use Lua 5.1 and no upgrade is planned currently, since we don't want to break
Lua scripts for new Lua features: in the context of Redis Lua scripts the
capabilities of 5.1 are usually more than enough, the release is rock solid,
and we definitely don't want to break old scripts.
So upgrading of Lua is up to the Redis project maintainers and should be a
manual procedure performed by taking a diff between the different versions.
Currently we have at least the following differences between official Lua 5.1
and our version:
1. Makefile is modified to allow a different compiler than GCC.
2. We have the implementation source code, and directly link to the following external libraries: `lua_cjson.o`, `lua_struct.o`, `lua_cmsgpack.o` and `lua_bit.o`.
3. There is a security fix in `ldo.c`, line 498: The check for `LUA_SIGNATURE[0]` is removed in order toa void direct bytecode execution.
-1
View File
@@ -4,4 +4,3 @@
/*.so
/*.dylib
/*.a
/*.pc
+1 -34
View File
@@ -1,39 +1,6 @@
language: c
sudo: false
compiler:
- gcc
- clang
os:
- linux
- osx
before_script:
- if [ "$TRAVIS_OS_NAME" == "osx" ] ; then brew update; brew install redis; fi
addons:
apt:
packages:
- libc6-dbg
- libc6-dev
- libc6:i386
- libc6-dev-i386
- libc6-dbg:i386
- gcc-multilib
- valgrind
env:
- CFLAGS="-Werror"
- PRE="valgrind --track-origins=yes --leak-check=full"
- TARGET="32bit" TARGET_VARS="32bit-vars" CFLAGS="-Werror"
- TARGET="32bit" TARGET_VARS="32bit-vars" PRE="valgrind --track-origins=yes --leak-check=full"
matrix:
exclude:
- os: osx
env: PRE="valgrind --track-origins=yes --leak-check=full"
- os: osx
env: TARGET="32bit" TARGET_VARS="32bit-vars" PRE="valgrind --track-origins=yes --leak-check=full"
script: make $TARGET CFLAGS="$CFLAGS" && make check PRE="$PRE" && make $TARGET_VARS hiredis-example
script: make && make check
-125
View File
@@ -1,128 +1,3 @@
### 1.0.0 (unreleased)
**Fixes**:
* Catch a buffer overflow when formatting the error message
* Import latest upstream sds. This breaks applications that are linked against the old hiredis v0.13
* Fix warnings, when compiled with -Wshadow
* Make hiredis compile in Cygwin on Windows, now CI-tested
**BREAKING CHANGES**:
* Change `redisReply.len` to `size_t`, as it denotes the the size of a string
User code should compare this to `size_t` values as well.
If it was used to compare to other values, casting might be necessary or can be removed, if casting was applied before.
* Remove backwards compatibility macro's
This removes the following old function aliases, use the new name now:
| Old | New |
| --------------------------- | ---------------------- |
| redisReplyReaderCreate | redisReaderCreate |
| redisReplyReaderCreate | redisReaderCreate |
| redisReplyReaderFree | redisReaderFree |
| redisReplyReaderFeed | redisReaderFeed |
| redisReplyReaderGetReply | redisReaderGetReply |
| redisReplyReaderSetPrivdata | redisReaderSetPrivdata |
| redisReplyReaderGetObject | redisReaderGetObject |
| redisReplyReaderGetError | redisReaderGetError |
* The `DEBUG` variable in the Makefile was renamed to `DEBUG_FLAGS`
Previously it broke some builds for people that had `DEBUG` set to some arbitrary value,
due to debugging other software.
By renaming we avoid unintentional name clashes.
Simply rename `DEBUG` to `DEBUG_FLAGS` in your environment to make it working again.
### 0.13.3 (2015-09-16)
* Revert "Clear `REDIS_CONNECTED` flag when connection is closed".
* Make tests pass on FreeBSD (Thanks, Giacomo Olgeni)
If the `REDIS_CONNECTED` flag is cleared,
the async onDisconnect callback function will never be called.
This causes problems as the disconnect is never reported back to the user.
### 0.13.2 (2015-08-25)
* Prevent crash on pending replies in async code (Thanks, @switch-st)
* Clear `REDIS_CONNECTED` flag when connection is closed (Thanks, Jerry Jacobs)
* Add MacOS X addapter (Thanks, @dizzus)
* Add Qt adapter (Thanks, Pietro Cerutti)
* Add Ivykis adapter (Thanks, Gergely Nagy)
All adapters are provided as is and are only tested where possible.
### 0.13.1 (2015-05-03)
This is a bug fix release.
The new `reconnect` method introduced new struct members, which clashed with pre-defined names in pre-C99 code.
Another commit forced C99 compilation just to make it work, but of course this is not desirable for outside projects.
Other non-C99 code can now use hiredis as usual again.
Sorry for the inconvenience.
* Fix memory leak in async reply handling (Salvatore Sanfilippo)
* Rename struct member to avoid name clash with pre-c99 code (Alex Balashov, ncopa)
### 0.13.0 (2015-04-16)
This release adds a minimal Windows compatibility layer.
The parser, standalone since v0.12.0, can now be compiled on Windows
(and thus used in other client libraries as well)
* Windows compatibility layer for parser code (tzickel)
* Properly escape data printed to PKGCONF file (Dan Skorupski)
* Fix tests when assert() undefined (Keith Bennett, Matt Stancliff)
* Implement a reconnect method for the client context, this changes the structure of `redisContext` (Aaron Bedra)
### 0.12.1 (2015-01-26)
* Fix `make install`: DESTDIR support, install all required files, install PKGCONF in proper location
* Fix `make test` as 32 bit build on 64 bit platform
### 0.12.0 (2015-01-22)
* Add optional KeepAlive support
* Try again on EINTR errors
* Add libuv adapter
* Add IPv6 support
* Remove possiblity of multiple close on same fd
* Add ability to bind source address on connect
* Add redisConnectFd() and redisFreeKeepFd()
* Fix getaddrinfo() memory leak
* Free string if it is unused (fixes memory leak)
* Improve redisAppendCommandArgv performance 2.5x
* Add support for SO_REUSEADDR
* Fix redisvFormatCommand format parsing
* Add GLib 2.0 adapter
* Refactor reading code into read.c
* Fix errno error buffers to not clobber errors
* Generate pkgconf during build
* Silence _BSD_SOURCE warnings
* Improve digit counting for multibulk creation
### 0.11.0
* Increase the maximum multi-bulk reply depth to 7.
+28 -79
View File
@@ -3,25 +3,13 @@
# Copyright (C) 2010-2011 Pieter Noordhuis <pcnoordhuis at gmail dot com>
# This file is released under the BSD license, see the COPYING file
OBJ=net.o hiredis.o sds.o async.o read.o
EXAMPLES=hiredis-example hiredis-example-libevent hiredis-example-libev hiredis-example-glib
OBJ=net.o hiredis.o sds.o async.o
EXAMPLES=hiredis-example hiredis-example-libevent hiredis-example-libev
TESTS=hiredis-test
LIBNAME=libhiredis
PKGCONFNAME=hiredis.pc
HIREDIS_MAJOR=$(shell grep HIREDIS_MAJOR hiredis.h | awk '{print $$3}')
HIREDIS_MINOR=$(shell grep HIREDIS_MINOR hiredis.h | awk '{print $$3}')
HIREDIS_PATCH=$(shell grep HIREDIS_PATCH hiredis.h | awk '{print $$3}')
HIREDIS_SONAME=$(shell grep HIREDIS_SONAME hiredis.h | awk '{print $$3}')
# Installation related variables and target
PREFIX?=/usr/local
INCLUDE_PATH?=include/hiredis
LIBRARY_PATH?=lib
PKGCONF_PATH?=pkgconfig
INSTALL_INCLUDE_PATH= $(DESTDIR)$(PREFIX)/$(INCLUDE_PATH)
INSTALL_LIBRARY_PATH= $(DESTDIR)$(PREFIX)/$(LIBRARY_PATH)
INSTALL_PKGCONF_PATH= $(INSTALL_LIBRARY_PATH)/$(PKGCONF_PATH)
HIREDIS_MAJOR=0
HIREDIS_MINOR=11
# redis-server configuration used for testing
REDIS_PORT=56379
@@ -37,16 +25,15 @@ export REDIS_TEST_CONFIG
# Fallback to gcc when $CC is not in $PATH.
CC:=$(shell sh -c 'type $(CC) >/dev/null 2>/dev/null && echo $(CC) || echo gcc')
CXX:=$(shell sh -c 'type $(CXX) >/dev/null 2>/dev/null && echo $(CXX) || echo g++')
OPTIMIZATION?=-O3
WARNINGS=-Wall -W -Wstrict-prototypes -Wwrite-strings
DEBUG_FLAGS?= -g -ggdb
REAL_CFLAGS=$(OPTIMIZATION) -fPIC $(CFLAGS) $(WARNINGS) $(DEBUG_FLAGS) $(ARCH)
DEBUG?= -g -ggdb
REAL_CFLAGS=$(OPTIMIZATION) -fPIC $(CFLAGS) $(WARNINGS) $(DEBUG) $(ARCH)
REAL_LDFLAGS=$(LDFLAGS) $(ARCH)
DYLIBSUFFIX=so
STLIBSUFFIX=a
DYLIB_MINOR_NAME=$(LIBNAME).$(DYLIBSUFFIX).$(HIREDIS_SONAME)
DYLIB_MINOR_NAME=$(LIBNAME).$(DYLIBSUFFIX).$(HIREDIS_MAJOR).$(HIREDIS_MINOR)
DYLIB_MAJOR_NAME=$(LIBNAME).$(DYLIBSUFFIX).$(HIREDIS_MAJOR)
DYLIBNAME=$(LIBNAME).$(DYLIBSUFFIX)
DYLIB_MAKE_CMD=$(CC) -shared -Wl,-soname,$(DYLIB_MINOR_NAME) -o $(DYLIBNAME) $(LDFLAGS)
@@ -62,20 +49,19 @@ ifeq ($(uname_S),SunOS)
endif
ifeq ($(uname_S),Darwin)
DYLIBSUFFIX=dylib
DYLIB_MINOR_NAME=$(LIBNAME).$(HIREDIS_SONAME).$(DYLIBSUFFIX)
DYLIB_MINOR_NAME=$(LIBNAME).$(HIREDIS_MAJOR).$(HIREDIS_MINOR).$(DYLIBSUFFIX)
DYLIB_MAJOR_NAME=$(LIBNAME).$(HIREDIS_MAJOR).$(DYLIBSUFFIX)
DYLIB_MAKE_CMD=$(CC) -shared -Wl,-install_name,$(DYLIB_MINOR_NAME) -o $(DYLIBNAME) $(LDFLAGS)
endif
all: $(DYLIBNAME) $(STLIBNAME) hiredis-test $(PKGCONFNAME)
all: $(DYLIBNAME)
# Deps (use make dep to generate this)
async.o: async.c fmacros.h async.h hiredis.h read.h sds.h net.h dict.c dict.h
dict.o: dict.c fmacros.h dict.h
hiredis.o: hiredis.c fmacros.h hiredis.h read.h sds.h net.h
net.o: net.c fmacros.h net.h hiredis.h read.h sds.h
read.o: read.c fmacros.h read.h sds.h
net.o: net.c fmacros.h net.h hiredis.h
async.o: async.c async.h hiredis.h sds.h dict.c dict.h
hiredis.o: hiredis.c fmacros.h hiredis.h net.h sds.h
sds.o: sds.c sds.h
test.o: test.c fmacros.h hiredis.h read.h sds.h
test.o: test.c hiredis.h
$(DYLIBNAME): $(OBJ)
$(DYLIB_MAKE_CMD) $(OBJ)
@@ -93,15 +79,6 @@ hiredis-example-libevent: examples/example-libevent.c adapters/libevent.h $(STLI
hiredis-example-libev: examples/example-libev.c adapters/libev.h $(STLIBNAME)
$(CC) -o examples/$@ $(REAL_CFLAGS) $(REAL_LDFLAGS) -I. $< -lev $(STLIBNAME)
hiredis-example-glib: examples/example-glib.c adapters/glib.h $(STLIBNAME)
$(CC) -o examples/$@ $(REAL_CFLAGS) $(REAL_LDFLAGS) $(shell pkg-config --cflags --libs glib-2.0) -I. $< $(STLIBNAME)
hiredis-example-ivykis: examples/example-ivykis.c adapters/ivykis.h $(STLIBNAME)
$(CC) -o examples/$@ $(REAL_CFLAGS) $(REAL_LDFLAGS) -I. $< -livykis $(STLIBNAME)
hiredis-example-macosx: examples/example-macosx.c adapters/macosx.h $(STLIBNAME)
$(CC) -o examples/$@ $(REAL_CFLAGS) $(REAL_LDFLAGS) -I. $< -framework CoreFoundation $(STLIBNAME)
ifndef AE_DIR
hiredis-example-ae:
@echo "Please specify AE_DIR (e.g. <redis repository>/src)"
@@ -117,20 +94,7 @@ hiredis-example-libuv:
@false
else
hiredis-example-libuv: examples/example-libuv.c adapters/libuv.h $(STLIBNAME)
$(CC) -o examples/$@ $(REAL_CFLAGS) $(REAL_LDFLAGS) -I. -I$(LIBUV_DIR)/include $< $(LIBUV_DIR)/.libs/libuv.a -lpthread -lrt $(STLIBNAME)
endif
ifeq ($(and $(QT_MOC),$(QT_INCLUDE_DIR),$(QT_LIBRARY_DIR)),)
hiredis-example-qt:
@echo "Please specify QT_MOC, QT_INCLUDE_DIR AND QT_LIBRARY_DIR"
@false
else
hiredis-example-qt: examples/example-qt.cpp adapters/qt.h $(STLIBNAME)
$(QT_MOC) adapters/qt.h -I. -I$(QT_INCLUDE_DIR) -I$(QT_INCLUDE_DIR)/QtCore | \
$(CXX) -x c++ -o qt-adapter-moc.o -c - $(REAL_CFLAGS) -I. -I$(QT_INCLUDE_DIR) -I$(QT_INCLUDE_DIR)/QtCore
$(QT_MOC) examples/example-qt.h -I. -I$(QT_INCLUDE_DIR) -I$(QT_INCLUDE_DIR)/QtCore | \
$(CXX) -x c++ -o qt-example-moc.o -c - $(REAL_CFLAGS) -I. -I$(QT_INCLUDE_DIR) -I$(QT_INCLUDE_DIR)/QtCore
$(CXX) -o examples/$@ $(REAL_CFLAGS) $(REAL_LDFLAGS) -I. -I$(QT_INCLUDE_DIR) -I$(QT_INCLUDE_DIR)/QtCore -L$(QT_LIBRARY_DIR) qt-adapter-moc.o qt-example-moc.o $< -pthread $(STLIBNAME) -lQtCore
$(CC) -o examples/$@ $(REAL_CFLAGS) $(REAL_LDFLAGS) -I. -I$(LIBUV_DIR)/include $< $(LIBUV_DIR)/.libs/libuv.a -lpthread $(STLIBNAME)
endif
hiredis-example: examples/example.c $(STLIBNAME)
@@ -139,16 +103,14 @@ hiredis-example: examples/example.c $(STLIBNAME)
examples: $(EXAMPLES)
hiredis-test: test.o $(STLIBNAME)
hiredis-%: %.o $(STLIBNAME)
$(CC) $(REAL_CFLAGS) -o $@ $(REAL_LDFLAGS) $< $(STLIBNAME)
$(CC) -o $@ $(REAL_LDFLAGS) $< $(STLIBNAME)
test: hiredis-test
./hiredis-test
check: hiredis-test
@echo "$$REDIS_TEST_CONFIG" | $(REDIS_SERVER) -
$(PRE) ./hiredis-test -h 127.0.0.1 -p $(REDIS_PORT) -s /tmp/hiredis-test-redis.sock || \
./hiredis-test -h 127.0.0.1 -p $(REDIS_PORT) -s /tmp/hiredis-test-redis.sock || \
( kill `cat /tmp/hiredis-test-redis.pid` && false )
kill `cat /tmp/hiredis-test-redis.pid`
@@ -156,38 +118,29 @@ check: hiredis-test
$(CC) -std=c99 -pedantic -c $(REAL_CFLAGS) $<
clean:
rm -rf $(DYLIBNAME) $(STLIBNAME) $(TESTS) $(PKGCONFNAME) examples/hiredis-example* *.o *.gcda *.gcno *.gcov
rm -rf $(DYLIBNAME) $(STLIBNAME) $(TESTS) examples/hiredis-example* *.o *.gcda *.gcno *.gcov
dep:
$(CC) -MM *.c
# Installation related variables and target
PREFIX?=/usr/local
INSTALL_INCLUDE_PATH= $(PREFIX)/include/hiredis
INSTALL_LIBRARY_PATH= $(PREFIX)/lib
ifeq ($(uname_S),SunOS)
INSTALL?= cp -r
endif
INSTALL?= cp -a
$(PKGCONFNAME): hiredis.h
@echo "Generating $@ for pkgconfig..."
@echo prefix=$(PREFIX) > $@
@echo exec_prefix=\$${prefix} >> $@
@echo libdir=$(PREFIX)/$(LIBRARY_PATH) >> $@
@echo includedir=$(PREFIX)/$(INCLUDE_PATH) >> $@
@echo >> $@
@echo Name: hiredis >> $@
@echo Description: Minimalistic C client library for Redis. >> $@
@echo Version: $(HIREDIS_MAJOR).$(HIREDIS_MINOR).$(HIREDIS_PATCH) >> $@
@echo Libs: -L\$${libdir} -lhiredis >> $@
@echo Cflags: -I\$${includedir} -D_FILE_OFFSET_BITS=64 >> $@
install: $(DYLIBNAME) $(STLIBNAME) $(PKGCONFNAME)
install: $(DYLIBNAME) $(STLIBNAME)
mkdir -p $(INSTALL_INCLUDE_PATH) $(INSTALL_LIBRARY_PATH)
$(INSTALL) hiredis.h async.h read.h sds.h adapters $(INSTALL_INCLUDE_PATH)
$(INSTALL) hiredis.h async.h adapters $(INSTALL_INCLUDE_PATH)
$(INSTALL) $(DYLIBNAME) $(INSTALL_LIBRARY_PATH)/$(DYLIB_MINOR_NAME)
cd $(INSTALL_LIBRARY_PATH) && ln -sf $(DYLIB_MINOR_NAME) $(DYLIBNAME)
cd $(INSTALL_LIBRARY_PATH) && ln -sf $(DYLIB_MINOR_NAME) $(DYLIB_MAJOR_NAME)
cd $(INSTALL_LIBRARY_PATH) && ln -sf $(DYLIB_MAJOR_NAME) $(DYLIBNAME)
$(INSTALL) $(STLIBNAME) $(INSTALL_LIBRARY_PATH)
mkdir -p $(INSTALL_PKGCONF_PATH)
$(INSTALL) $(PKGCONFNAME) $(INSTALL_PKGCONF_PATH)
32bit:
@echo ""
@@ -195,10 +148,6 @@ install: $(DYLIBNAME) $(STLIBNAME) $(PKGCONFNAME)
@echo ""
$(MAKE) CFLAGS="-m32" LDFLAGS="-m32"
32bit-vars:
$(eval CFLAGS=-m32)
$(eval LDFLAGS=-m32)
gprof:
$(MAKE) CFLAGS="-pg" LDFLAGS="-pg"
@@ -214,4 +163,4 @@ coverage: gcov
noopt:
$(MAKE) OPTIMIZATION=""
.PHONY: all test check clean dep install 32bit 32bit-vars gprof gcov noopt
.PHONY: all test check clean dep install 32bit gprof gcov noopt
+89 -116
View File
@@ -1,13 +1,11 @@
[![Build Status](https://travis-ci.org/redis/hiredis.png)](https://travis-ci.org/redis/hiredis)
**This Readme reflects the latest changed in the master branch. See [v0.13.3](https://github.com/redis/hiredis/tree/v0.13.3) for the Readme and documentation for the latest release.**
# HIREDIS
Hiredis is a minimalistic C client library for the [Redis](http://redis.io/) database.
It is minimalistic because it just adds minimal support for the protocol, but
at the same time it uses a high level printf-alike API in order to make it
at the same time it uses an high level printf-alike API in order to make it
much higher level than otherwise suggested by its minimal code base and the
lack of explicit bindings for every Redis command.
@@ -22,15 +20,7 @@ Redis version >= 1.2.0.
The library comes with multiple APIs. There is the
*synchronous API*, the *asynchronous API* and the *reply parsing API*.
## Upgrading to `1.0.0`
Version 1.0.0 marks a stable release of hiredis.
It includes some minor breaking changes, mostly to make the exposed API more uniform and self-explanatory.
It also bundles the updated `sds` library, to sync up with upstream and Redis.
For most applications a recompile against the new hiredis should be enough.
For code changes see the [Changelog](CHANGELOG.md).
## Upgrading from `<0.9.0`
## UPGRADING
Version 0.9.0 is a major overhaul of hiredis in every aspect. However, upgrading existing
code using hiredis should not be a big pain. The key thing to keep in mind when
@@ -41,62 +31,51 @@ the stateless 0.0.1 that only has a file descriptor to work with.
To consume the synchronous API, there are only a few function calls that need to be introduced:
```c
redisContext *redisConnect(const char *ip, int port);
void *redisCommand(redisContext *c, const char *format, ...);
void freeReplyObject(void *reply);
```
redisContext *redisConnect(const char *ip, int port);
void *redisCommand(redisContext *c, const char *format, ...);
void freeReplyObject(void *reply);
### Connecting
The function `redisConnect` is used to create a so-called `redisContext`. The
context is where Hiredis holds state for a connection. The `redisContext`
struct has an integer `err` field that is non-zero when the connection is in
struct has an integer `err` field that is non-zero when an the connection is in
an error state. The field `errstr` will contain a string with a description of
the error. More information on errors can be found in the **Errors** section.
After trying to connect to Redis using `redisConnect` you should
check the `err` field to see if establishing the connection was successful:
```c
redisContext *c = redisConnect("127.0.0.1", 6379);
if (c == NULL || c->err) {
if (c) {
redisContext *c = redisConnect("127.0.0.1", 6379);
if (c != NULL && c->err) {
printf("Error: %s\n", c->errstr);
// handle error
} else {
printf("Can't allocate redis context\n");
}
}
```
*Note: A `redisContext` is not thread-safe.*
### Sending commands
There are several ways to issue commands to Redis. The first that will be introduced is
`redisCommand`. This function takes a format similar to printf. In the simplest form,
it is used like this:
```c
reply = redisCommand(context, "SET foo bar");
```
reply = redisCommand(context, "SET foo bar");
The specifier `%s` interpolates a string in the command, and uses `strlen` to
determine the length of the string:
```c
reply = redisCommand(context, "SET foo %s", value);
```
reply = redisCommand(context, "SET foo %s", value);
When you need to pass binary safe strings in a command, the `%b` specifier can be
used. Together with a pointer to the string, it requires a `size_t` length argument
of the string:
```c
reply = redisCommand(context, "SET foo %b", value, (size_t) valuelen);
```
reply = redisCommand(context, "SET foo %b", value, (size_t) valuelen);
Internally, Hiredis splits the command in different arguments and will
convert it to the protocol used to communicate with Redis.
One or more spaces separates arguments, so you can use the specifiers
anywhere in an argument:
```c
reply = redisCommand(context, "SET key:%s %s", myid, value);
```
reply = redisCommand(context, "SET key:%s %s", myid, value);
### Using replies
@@ -135,11 +114,11 @@ was received:
Redis may reply with nested arrays but this is fully supported.
Replies should be freed using the `freeReplyObject()` function.
Note that this function will take care of freeing sub-reply objects
Note that this function will take care of freeing sub-replies objects
contained in arrays and nested arrays, so there is no need for the user to
free the sub replies (it is actually harmful and will corrupt the memory).
**Important:** the current version of hiredis (0.10.0) frees replies when the
**Important:** the current version of hiredis (0.10.0) free's replies when the
asynchronous API is used. This means you should not call `freeReplyObject` when
you use this API. The reply is cleaned up by hiredis _after_ the callback
returns. This behavior will probably change in future releases, so make sure to
@@ -148,19 +127,19 @@ keep an eye on the changelog when upgrading (see issue #39).
### Cleaning up
To disconnect and free the context the following function can be used:
```c
void redisFree(redisContext *c);
```
This function immediately closes the socket and then frees the allocations done in
void redisFree(redisContext *c);
This function immediately closes the socket and then free's the allocations done in
creating the context.
### Sending commands (cont'd)
Together with `redisCommand`, the function `redisCommandArgv` can be used to issue commands.
It has the following prototype:
```c
void *redisCommandArgv(redisContext *c, int argc, const char **argv, const size_t *argvlen);
```
void *redisCommandArgv(redisContext *c, int argc, const char **argv, const size_t *argvlen);
It takes the number of arguments `argc`, an array of strings `argv` and the lengths of the
arguments `argvlen`. For convenience, `argvlen` may be set to `NULL` and the function will
use `strlen(3)` on every argument to determine its length. Obviously, when any of the arguments
@@ -190,10 +169,10 @@ The function `redisGetReply` is exported as part of the Hiredis API and can be u
is expected on the socket. To pipeline commands, the only things that needs to be done is
filling up the output buffer. For this cause, two commands can be used that are identical
to the `redisCommand` family, apart from not returning a reply:
```c
void redisAppendCommand(redisContext *c, const char *format, ...);
void redisAppendCommandArgv(redisContext *c, int argc, const char **argv, const size_t *argvlen);
```
void redisAppendCommand(redisContext *c, const char *format, ...);
void redisAppendCommandArgv(redisContext *c, int argc, const char **argv, const size_t *argvlen);
After calling either function one or more times, `redisGetReply` can be used to receive the
subsequent replies. The return value for this function is either `REDIS_OK` or `REDIS_ERR`, where
the latter means an error occurred while reading a reply. Just as with the other commands,
@@ -201,24 +180,24 @@ the `err` field in the context can be used to find out what the cause of this er
The following examples shows a simple pipeline (resulting in only a single call to `write(2)` and
a single call to `read(2)`):
```c
redisReply *reply;
redisAppendCommand(context,"SET foo bar");
redisAppendCommand(context,"GET foo");
redisGetReply(context,&reply); // reply for SET
freeReplyObject(reply);
redisGetReply(context,&reply); // reply for GET
freeReplyObject(reply);
```
This API can also be used to implement a blocking subscriber:
```c
reply = redisCommand(context,"SUBSCRIBE foo");
freeReplyObject(reply);
while(redisGetReply(context,&reply) == REDIS_OK) {
// consume message
redisReply *reply;
redisAppendCommand(context,"SET foo bar");
redisAppendCommand(context,"GET foo");
redisGetReply(context,&reply); // reply for SET
freeReplyObject(reply);
}
```
redisGetReply(context,&reply); // reply for GET
freeReplyObject(reply);
This API can also be used to implement a blocking subscriber:
reply = redisCommand(context,"SUBSCRIBE foo");
freeReplyObject(reply);
while(redisGetReply(context,&reply) == REDIS_OK) {
// consume message
freeReplyObject(reply);
}
### Errors
When a function call is not successful, depending on the function either `NULL` or `REDIS_ERR` is
@@ -258,62 +237,58 @@ should be checked after creation to see if there were errors creating the connec
Because the connection that will be created is non-blocking, the kernel is not able to
instantly return if the specified host and port is able to accept a connection.
*Note: A `redisAsyncContext` is not thread-safe.*
```c
redisAsyncContext *c = redisAsyncConnect("127.0.0.1", 6379);
if (c->err) {
printf("Error: %s\n", c->errstr);
// handle error
}
```
redisAsyncContext *c = redisAsyncConnect("127.0.0.1", 6379);
if (c->err) {
printf("Error: %s\n", c->errstr);
// handle error
}
The asynchronous context can hold a disconnect callback function that is called when the
connection is disconnected (either because of an error or per user request). This function should
have the following prototype:
```c
void(const redisAsyncContext *c, int status);
```
void(const redisAsyncContext *c, int status);
On a disconnect, the `status` argument is set to `REDIS_OK` when disconnection was initiated by the
user, or `REDIS_ERR` when the disconnection was caused by an error. When it is `REDIS_ERR`, the `err`
field in the context can be accessed to find out the cause of the error.
The context object is always freed after the disconnect callback fired. When a reconnect is needed,
The context object is always free'd after the disconnect callback fired. When a reconnect is needed,
the disconnect callback is a good point to do so.
Setting the disconnect callback can only be done once per context. For subsequent calls it will
return `REDIS_ERR`. The function to set the disconnect callback has the following prototype:
```c
int redisAsyncSetDisconnectCallback(redisAsyncContext *ac, redisDisconnectCallback *fn);
```
int redisAsyncSetDisconnectCallback(redisAsyncContext *ac, redisDisconnectCallback *fn);
### Sending commands and their callbacks
In an asynchronous context, commands are automatically pipelined due to the nature of an event loop.
Therefore, unlike the synchronous API, there is only a single way to send commands.
Because commands are sent to Redis asynchronously, issuing a command requires a callback function
that is called when the reply is received. Reply callbacks should have the following prototype:
```c
void(redisAsyncContext *c, void *reply, void *privdata);
```
void(redisAsyncContext *c, void *reply, void *privdata);
The `privdata` argument can be used to curry arbitrary data to the callback from the point where
the command is initially queued for execution.
The functions that can be used to issue commands in an asynchronous context are:
```c
int redisAsyncCommand(
redisAsyncContext *ac, redisCallbackFn *fn, void *privdata,
const char *format, ...);
int redisAsyncCommandArgv(
redisAsyncContext *ac, redisCallbackFn *fn, void *privdata,
int argc, const char **argv, const size_t *argvlen);
```
int redisAsyncCommand(
redisAsyncContext *ac, redisCallbackFn *fn, void *privdata,
const char *format, ...);
int redisAsyncCommandArgv(
redisAsyncContext *ac, redisCallbackFn *fn, void *privdata,
int argc, const char **argv, const size_t *argvlen);
Both functions work like their blocking counterparts. The return value is `REDIS_OK` when the command
was successfully added to the output buffer and `REDIS_ERR` otherwise. Example: when the connection
is being disconnected per user-request, no new commands may be added to the output buffer and `REDIS_ERR` is
returned on calls to the `redisAsyncCommand` family.
If the reply for a command with a `NULL` callback is read, it is immediately freed. When the callback
for a command is non-`NULL`, the memory is freed immediately following the callback: the reply is only
If the reply for a command with a `NULL` callback is read, it is immediately free'd. When the callback
for a command is non-`NULL`, the memory is free'd immediately following the callback: the reply is only
valid for the duration of the callback.
All pending callbacks are called with a `NULL` reply when the context encountered an error.
@@ -321,14 +296,14 @@ All pending callbacks are called with a `NULL` reply when the context encountere
### Disconnecting
An asynchronous connection can be terminated using:
```c
void redisAsyncDisconnect(redisAsyncContext *ac);
```
void redisAsyncDisconnect(redisAsyncContext *ac);
When this function is called, the connection is **not** immediately terminated. Instead, new
commands are no longer accepted and the connection is only terminated when all pending commands
have been written to the socket, their respective replies have been read and their respective
callbacks have been executed. After this, the disconnection callback is executed with the
`REDIS_OK` status and the context object is freed.
`REDIS_OK` status and the context object is free'd.
### Hooking it up to event library *X*
@@ -341,12 +316,12 @@ Hiredis comes with a reply parsing API that makes it easy for writing higher
level language bindings.
The reply parsing API consists of the following functions:
```c
redisReader *redisReaderCreate(void);
void redisReaderFree(redisReader *reader);
int redisReaderFeed(redisReader *reader, const char *buf, size_t len);
int redisReaderGetReply(redisReader *reader, void **reply);
```
redisReader *redisReaderCreate(void);
void redisReaderFree(redisReader *reader);
int redisReaderFeed(redisReader *reader, const char *buf, size_t len);
int redisReaderGetReply(redisReader *reader, void **reply);
The same set of functions are used internally by hiredis when creating a
normal Redis context, the above API just exposes it to the user for a direct
usage.
@@ -386,7 +361,7 @@ Both when using the Reader API directly or when using it indirectly via a
normal Redis context, the redisReader structure uses a buffer in order to
accumulate data from the server.
Usually this buffer is destroyed when it is empty and is larger than 16
KiB in order to avoid wasting memory in unused buffers
kb in order to avoid wasting memory in unused buffers
However when working with very big payloads destroying the buffer may slow
down performances considerably, so it is possible to modify the max size of
@@ -396,9 +371,9 @@ value for an idle buffer, so the buffer will never get freed.
For instance if you have a normal Redis context you can set the maximum idle
buffer to zero (unlimited) just with:
```c
context->reader->maxbuf = 0;
```
context->reader->maxbuf = 0;
This should be done only in order to maximize performances when working with
large payloads. The context should be set back to `REDIS_READER_MAX_BUF` again
as soon as possible in order to prevent allocation of useless memory.
@@ -406,6 +381,4 @@ as soon as possible in order to prevent allocation of useless memory.
## AUTHORS
Hiredis was written by Salvatore Sanfilippo (antirez at gmail) and
Pieter Noordhuis (pcnoordhuis at gmail) and is released under the BSD license.
Hiredis is currently maintained by Matt Stancliff (matt at genges dot com) and
Jan-Erik Rediger (janerik at fnordig dot com)
Pieter Noordhuis (pcnoordhuis at gmail) and is released under the BSD license.
-153
View File
@@ -1,153 +0,0 @@
#ifndef __HIREDIS_GLIB_H__
#define __HIREDIS_GLIB_H__
#include <glib.h>
#include "../hiredis.h"
#include "../async.h"
typedef struct
{
GSource source;
redisAsyncContext *ac;
GPollFD poll_fd;
} RedisSource;
static void
redis_source_add_read (gpointer data)
{
RedisSource *source = (RedisSource *)data;
g_return_if_fail(source);
source->poll_fd.events |= G_IO_IN;
g_main_context_wakeup(g_source_get_context((GSource *)data));
}
static void
redis_source_del_read (gpointer data)
{
RedisSource *source = (RedisSource *)data;
g_return_if_fail(source);
source->poll_fd.events &= ~G_IO_IN;
g_main_context_wakeup(g_source_get_context((GSource *)data));
}
static void
redis_source_add_write (gpointer data)
{
RedisSource *source = (RedisSource *)data;
g_return_if_fail(source);
source->poll_fd.events |= G_IO_OUT;
g_main_context_wakeup(g_source_get_context((GSource *)data));
}
static void
redis_source_del_write (gpointer data)
{
RedisSource *source = (RedisSource *)data;
g_return_if_fail(source);
source->poll_fd.events &= ~G_IO_OUT;
g_main_context_wakeup(g_source_get_context((GSource *)data));
}
static void
redis_source_cleanup (gpointer data)
{
RedisSource *source = (RedisSource *)data;
g_return_if_fail(source);
redis_source_del_read(source);
redis_source_del_write(source);
/*
* It is not our responsibility to remove ourself from the
* current main loop. However, we will remove the GPollFD.
*/
if (source->poll_fd.fd >= 0) {
g_source_remove_poll((GSource *)data, &source->poll_fd);
source->poll_fd.fd = -1;
}
}
static gboolean
redis_source_prepare (GSource *source,
gint *timeout_)
{
RedisSource *redis = (RedisSource *)source;
*timeout_ = -1;
return !!(redis->poll_fd.events & redis->poll_fd.revents);
}
static gboolean
redis_source_check (GSource *source)
{
RedisSource *redis = (RedisSource *)source;
return !!(redis->poll_fd.events & redis->poll_fd.revents);
}
static gboolean
redis_source_dispatch (GSource *source,
GSourceFunc callback,
gpointer user_data)
{
RedisSource *redis = (RedisSource *)source;
if ((redis->poll_fd.revents & G_IO_OUT)) {
redisAsyncHandleWrite(redis->ac);
redis->poll_fd.revents &= ~G_IO_OUT;
}
if ((redis->poll_fd.revents & G_IO_IN)) {
redisAsyncHandleRead(redis->ac);
redis->poll_fd.revents &= ~G_IO_IN;
}
if (callback) {
return callback(user_data);
}
return TRUE;
}
static void
redis_source_finalize (GSource *source)
{
RedisSource *redis = (RedisSource *)source;
if (redis->poll_fd.fd >= 0) {
g_source_remove_poll(source, &redis->poll_fd);
redis->poll_fd.fd = -1;
}
}
static GSource *
redis_source_new (redisAsyncContext *ac)
{
static GSourceFuncs source_funcs = {
.prepare = redis_source_prepare,
.check = redis_source_check,
.dispatch = redis_source_dispatch,
.finalize = redis_source_finalize,
};
redisContext *c = &ac->c;
RedisSource *source;
g_return_val_if_fail(ac != NULL, NULL);
source = (RedisSource *)g_source_new(&source_funcs, sizeof *source);
source->ac = ac;
source->poll_fd.fd = c->fd;
source->poll_fd.events = 0;
source->poll_fd.revents = 0;
g_source_add_poll((GSource *)source, &source->poll_fd);
ac->ev.addRead = redis_source_add_read;
ac->ev.delRead = redis_source_del_read;
ac->ev.addWrite = redis_source_add_write;
ac->ev.delWrite = redis_source_del_write;
ac->ev.cleanup = redis_source_cleanup;
ac->ev.data = source;
return (GSource *)source;
}
#endif /* __HIREDIS_GLIB_H__ */
-81
View File
@@ -1,81 +0,0 @@
#ifndef __HIREDIS_IVYKIS_H__
#define __HIREDIS_IVYKIS_H__
#include <iv.h>
#include "../hiredis.h"
#include "../async.h"
typedef struct redisIvykisEvents {
redisAsyncContext *context;
struct iv_fd fd;
} redisIvykisEvents;
static void redisIvykisReadEvent(void *arg) {
redisAsyncContext *context = (redisAsyncContext *)arg;
redisAsyncHandleRead(context);
}
static void redisIvykisWriteEvent(void *arg) {
redisAsyncContext *context = (redisAsyncContext *)arg;
redisAsyncHandleWrite(context);
}
static void redisIvykisAddRead(void *privdata) {
redisIvykisEvents *e = (redisIvykisEvents*)privdata;
iv_fd_set_handler_in(&e->fd, redisIvykisReadEvent);
}
static void redisIvykisDelRead(void *privdata) {
redisIvykisEvents *e = (redisIvykisEvents*)privdata;
iv_fd_set_handler_in(&e->fd, NULL);
}
static void redisIvykisAddWrite(void *privdata) {
redisIvykisEvents *e = (redisIvykisEvents*)privdata;
iv_fd_set_handler_out(&e->fd, redisIvykisWriteEvent);
}
static void redisIvykisDelWrite(void *privdata) {
redisIvykisEvents *e = (redisIvykisEvents*)privdata;
iv_fd_set_handler_out(&e->fd, NULL);
}
static void redisIvykisCleanup(void *privdata) {
redisIvykisEvents *e = (redisIvykisEvents*)privdata;
iv_fd_unregister(&e->fd);
free(e);
}
static int redisIvykisAttach(redisAsyncContext *ac) {
redisContext *c = &(ac->c);
redisIvykisEvents *e;
/* Nothing should be attached when something is already attached */
if (ac->ev.data != NULL)
return REDIS_ERR;
/* Create container for context and r/w events */
e = (redisIvykisEvents*)malloc(sizeof(*e));
e->context = ac;
/* Register functions to start/stop listening for events */
ac->ev.addRead = redisIvykisAddRead;
ac->ev.delRead = redisIvykisDelRead;
ac->ev.addWrite = redisIvykisAddWrite;
ac->ev.delWrite = redisIvykisDelWrite;
ac->ev.cleanup = redisIvykisCleanup;
ac->ev.data = e;
/* Initialize and install read/write events */
IV_FD_INIT(&e->fd);
e->fd.fd = c->fd;
e->fd.handler_in = redisIvykisReadEvent;
e->fd.handler_out = redisIvykisWriteEvent;
e->fd.handler_err = NULL;
e->fd.cookie = e->context;
iv_fd_register(&e->fd);
return REDIS_OK;
}
#endif
+12 -12
View File
@@ -30,13 +30,13 @@
#ifndef __HIREDIS_LIBEVENT_H__
#define __HIREDIS_LIBEVENT_H__
#include <event2/event.h>
#include <event.h>
#include "../hiredis.h"
#include "../async.h"
typedef struct redisLibeventEvents {
redisAsyncContext *context;
struct event *rev, *wev;
struct event rev, wev;
} redisLibeventEvents;
static void redisLibeventReadEvent(int fd, short event, void *arg) {
@@ -53,28 +53,28 @@ static void redisLibeventWriteEvent(int fd, short event, void *arg) {
static void redisLibeventAddRead(void *privdata) {
redisLibeventEvents *e = (redisLibeventEvents*)privdata;
event_add(e->rev,NULL);
event_add(&e->rev,NULL);
}
static void redisLibeventDelRead(void *privdata) {
redisLibeventEvents *e = (redisLibeventEvents*)privdata;
event_del(e->rev);
event_del(&e->rev);
}
static void redisLibeventAddWrite(void *privdata) {
redisLibeventEvents *e = (redisLibeventEvents*)privdata;
event_add(e->wev,NULL);
event_add(&e->wev,NULL);
}
static void redisLibeventDelWrite(void *privdata) {
redisLibeventEvents *e = (redisLibeventEvents*)privdata;
event_del(e->wev);
event_del(&e->wev);
}
static void redisLibeventCleanup(void *privdata) {
redisLibeventEvents *e = (redisLibeventEvents*)privdata;
event_del(e->rev);
event_del(e->wev);
event_del(&e->rev);
event_del(&e->wev);
free(e);
}
@@ -99,10 +99,10 @@ static int redisLibeventAttach(redisAsyncContext *ac, struct event_base *base) {
ac->ev.data = e;
/* Initialize and install read/write events */
e->rev = event_new(base, c->fd, EV_READ, redisLibeventReadEvent, e);
e->wev = event_new(base, c->fd, EV_WRITE, redisLibeventWriteEvent, e);
event_add(e->rev, NULL);
event_add(e->wev, NULL);
event_set(&e->rev,c->fd,EV_READ,redisLibeventReadEvent,e);
event_set(&e->wev,c->fd,EV_WRITE,redisLibeventWriteEvent,e);
event_base_set(base,&e->rev);
event_base_set(base,&e->wev);
return REDIS_OK;
}
#endif
+3 -4
View File
@@ -1,6 +1,5 @@
#ifndef __HIREDIS_LIBUV_H__
#define __HIREDIS_LIBUV_H__
#include <stdlib.h>
#include <uv.h>
#include "../hiredis.h"
#include "../async.h"
@@ -12,6 +11,7 @@ typedef struct redisLibuvEvents {
int events;
} redisLibuvEvents;
int redisLibuvAttach(redisAsyncContext*, uv_loop_t*);
static void redisLibuvPoll(uv_poll_t* handle, int status, int events) {
redisLibuvEvents* p = (redisLibuvEvents*)handle->data;
@@ -20,10 +20,10 @@ static void redisLibuvPoll(uv_poll_t* handle, int status, int events) {
return;
}
if (p->context != NULL && (events & UV_READABLE)) {
if (events & UV_READABLE) {
redisAsyncHandleRead(p->context);
}
if (p->context != NULL && (events & UV_WRITABLE)) {
if (events & UV_WRITABLE) {
redisAsyncHandleWrite(p->context);
}
}
@@ -83,7 +83,6 @@ static void on_close(uv_handle_t* handle) {
static void redisLibuvCleanup(void *privdata) {
redisLibuvEvents* p = (redisLibuvEvents*)privdata;
p->context = NULL; // indicate that context might no longer exist
uv_close((uv_handle_t*)&p->handle, on_close);
}
-114
View File
@@ -1,114 +0,0 @@
//
// Created by Дмитрий Бахвалов on 13.07.15.
// Copyright (c) 2015 Dmitry Bakhvalov. All rights reserved.
//
#ifndef __HIREDIS_MACOSX_H__
#define __HIREDIS_MACOSX_H__
#include <CoreFoundation/CoreFoundation.h>
#include "../hiredis.h"
#include "../async.h"
typedef struct {
redisAsyncContext *context;
CFSocketRef socketRef;
CFRunLoopSourceRef sourceRef;
} RedisRunLoop;
static int freeRedisRunLoop(RedisRunLoop* redisRunLoop) {
if( redisRunLoop != NULL ) {
if( redisRunLoop->sourceRef != NULL ) {
CFRunLoopSourceInvalidate(redisRunLoop->sourceRef);
CFRelease(redisRunLoop->sourceRef);
}
if( redisRunLoop->socketRef != NULL ) {
CFSocketInvalidate(redisRunLoop->socketRef);
CFRelease(redisRunLoop->socketRef);
}
free(redisRunLoop);
}
return REDIS_ERR;
}
static void redisMacOSAddRead(void *privdata) {
RedisRunLoop *redisRunLoop = (RedisRunLoop*)privdata;
CFSocketEnableCallBacks(redisRunLoop->socketRef, kCFSocketReadCallBack);
}
static void redisMacOSDelRead(void *privdata) {
RedisRunLoop *redisRunLoop = (RedisRunLoop*)privdata;
CFSocketDisableCallBacks(redisRunLoop->socketRef, kCFSocketReadCallBack);
}
static void redisMacOSAddWrite(void *privdata) {
RedisRunLoop *redisRunLoop = (RedisRunLoop*)privdata;
CFSocketEnableCallBacks(redisRunLoop->socketRef, kCFSocketWriteCallBack);
}
static void redisMacOSDelWrite(void *privdata) {
RedisRunLoop *redisRunLoop = (RedisRunLoop*)privdata;
CFSocketDisableCallBacks(redisRunLoop->socketRef, kCFSocketWriteCallBack);
}
static void redisMacOSCleanup(void *privdata) {
RedisRunLoop *redisRunLoop = (RedisRunLoop*)privdata;
freeRedisRunLoop(redisRunLoop);
}
static void redisMacOSAsyncCallback(CFSocketRef __unused s, CFSocketCallBackType callbackType, CFDataRef __unused address, const void __unused *data, void *info) {
redisAsyncContext* context = (redisAsyncContext*) info;
switch (callbackType) {
case kCFSocketReadCallBack:
redisAsyncHandleRead(context);
break;
case kCFSocketWriteCallBack:
redisAsyncHandleWrite(context);
break;
default:
break;
}
}
static int redisMacOSAttach(redisAsyncContext *redisAsyncCtx, CFRunLoopRef runLoop) {
redisContext *redisCtx = &(redisAsyncCtx->c);
/* Nothing should be attached when something is already attached */
if( redisAsyncCtx->ev.data != NULL ) return REDIS_ERR;
RedisRunLoop* redisRunLoop = (RedisRunLoop*) calloc(1, sizeof(RedisRunLoop));
if( !redisRunLoop ) return REDIS_ERR;
/* Setup redis stuff */
redisRunLoop->context = redisAsyncCtx;
redisAsyncCtx->ev.addRead = redisMacOSAddRead;
redisAsyncCtx->ev.delRead = redisMacOSDelRead;
redisAsyncCtx->ev.addWrite = redisMacOSAddWrite;
redisAsyncCtx->ev.delWrite = redisMacOSDelWrite;
redisAsyncCtx->ev.cleanup = redisMacOSCleanup;
redisAsyncCtx->ev.data = redisRunLoop;
/* Initialize and install read/write events */
CFSocketContext socketCtx = { 0, redisAsyncCtx, NULL, NULL, NULL };
redisRunLoop->socketRef = CFSocketCreateWithNative(NULL, redisCtx->fd,
kCFSocketReadCallBack | kCFSocketWriteCallBack,
redisMacOSAsyncCallback,
&socketCtx);
if( !redisRunLoop->socketRef ) return freeRedisRunLoop(redisRunLoop);
redisRunLoop->sourceRef = CFSocketCreateRunLoopSource(NULL, redisRunLoop->socketRef, 0);
if( !redisRunLoop->sourceRef ) return freeRedisRunLoop(redisRunLoop);
CFRunLoopAddSource(runLoop, redisRunLoop->sourceRef, kCFRunLoopDefaultMode);
return REDIS_OK;
}
#endif
-135
View File
@@ -1,135 +0,0 @@
/*-
* Copyright (C) 2014 Pietro Cerutti <gahr@gahr.ch>
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
#ifndef __HIREDIS_QT_H__
#define __HIREDIS_QT_H__
#include <QSocketNotifier>
#include "../async.h"
static void RedisQtAddRead(void *);
static void RedisQtDelRead(void *);
static void RedisQtAddWrite(void *);
static void RedisQtDelWrite(void *);
static void RedisQtCleanup(void *);
class RedisQtAdapter : public QObject {
Q_OBJECT
friend
void RedisQtAddRead(void * adapter) {
RedisQtAdapter * a = static_cast<RedisQtAdapter *>(adapter);
a->addRead();
}
friend
void RedisQtDelRead(void * adapter) {
RedisQtAdapter * a = static_cast<RedisQtAdapter *>(adapter);
a->delRead();
}
friend
void RedisQtAddWrite(void * adapter) {
RedisQtAdapter * a = static_cast<RedisQtAdapter *>(adapter);
a->addWrite();
}
friend
void RedisQtDelWrite(void * adapter) {
RedisQtAdapter * a = static_cast<RedisQtAdapter *>(adapter);
a->delWrite();
}
friend
void RedisQtCleanup(void * adapter) {
RedisQtAdapter * a = static_cast<RedisQtAdapter *>(adapter);
a->cleanup();
}
public:
RedisQtAdapter(QObject * parent = 0)
: QObject(parent), m_ctx(0), m_read(0), m_write(0) { }
~RedisQtAdapter() {
if (m_ctx != 0) {
m_ctx->ev.data = NULL;
}
}
int setContext(redisAsyncContext * ac) {
if (ac->ev.data != NULL) {
return REDIS_ERR;
}
m_ctx = ac;
m_ctx->ev.data = this;
m_ctx->ev.addRead = RedisQtAddRead;
m_ctx->ev.delRead = RedisQtDelRead;
m_ctx->ev.addWrite = RedisQtAddWrite;
m_ctx->ev.delWrite = RedisQtDelWrite;
m_ctx->ev.cleanup = RedisQtCleanup;
return REDIS_OK;
}
private:
void addRead() {
if (m_read) return;
m_read = new QSocketNotifier(m_ctx->c.fd, QSocketNotifier::Read, 0);
connect(m_read, SIGNAL(activated(int)), this, SLOT(read()));
}
void delRead() {
if (!m_read) return;
delete m_read;
m_read = 0;
}
void addWrite() {
if (m_write) return;
m_write = new QSocketNotifier(m_ctx->c.fd, QSocketNotifier::Write, 0);
connect(m_write, SIGNAL(activated(int)), this, SLOT(write()));
}
void delWrite() {
if (!m_write) return;
delete m_write;
m_write = 0;
}
void cleanup() {
delRead();
delWrite();
}
private slots:
void read() { redisAsyncHandleRead(m_ctx); }
void write() { redisAsyncHandleWrite(m_ctx); }
private:
redisAsyncContext * m_ctx;
QSocketNotifier * m_read;
QSocketNotifier * m_write;
};
#endif /* !__HIREDIS_QT_H__ */
-36
View File
@@ -1,36 +0,0 @@
# Appveyor configuration file for CI build of hiredis on Windows (under Cygwin)
environment:
matrix:
- CYG_ROOT: C:\cygwin64
CYG_SETUP: setup-x86_64.exe
CYG_MIRROR: http://cygwin.mirror.constant.com
CYG_CACHE: C:\cygwin64\var\cache\setup
CYG_BASH: C:\cygwin64\bin\bash
CC: gcc
- CYG_ROOT: C:\cygwin
CYG_SETUP: setup-x86.exe
CYG_MIRROR: http://cygwin.mirror.constant.com
CYG_CACHE: C:\cygwin\var\cache\setup
CYG_BASH: C:\cygwin\bin\bash
CC: gcc
TARGET: 32bit
TARGET_VARS: 32bit-vars
# Cache Cygwin files to speed up build
cache:
- '%CYG_CACHE%'
clone_depth: 1
# Attempt to ensure we don't try to convert line endings to Win32 CRLF as this will cause build to fail
init:
- git config --global core.autocrlf input
# Install needed build dependencies
install:
- ps: 'Start-FileDownload "http://cygwin.com/$env:CYG_SETUP" -FileName "$env:CYG_SETUP"'
- '%CYG_SETUP% --quiet-mode --no-shortcuts --only-site --root "%CYG_ROOT%" --site "%CYG_MIRROR%" --local-package-dir "%CYG_CACHE%" --packages automake,bison,gcc-core,libtool,make,gettext-devel,gettext,intltool,pkg-config,clang,llvm > NUL 2>&1'
- '%CYG_BASH% -lc "cygcheck -dc cygwin"'
build_script:
- 'echo building...'
- '%CYG_BASH% -lc "cd $APPVEYOR_BUILD_FOLDER; exec 0</dev/null; make LDFLAGS=$LDFLAGS CC=$CC $TARGET CFLAGS=$CFLAGS && make LDFLAGS=$LDFLAGS CC=$CC $TARGET_VARS hiredis-example"'
+13 -39
View File
@@ -58,7 +58,7 @@
} while(0);
/* Forward declaration of function in hiredis.c */
int __redisAppendCommand(redisContext *c, const char *cmd, size_t len);
void __redisAppendCommand(redisContext *c, char *cmd, size_t len);
/* Functions managing dictionary of callbacks for pub/sub. */
static unsigned int callbackHash(const void *key) {
@@ -142,9 +142,6 @@ static redisAsyncContext *redisAsyncInitialize(redisContext *c) {
/* We want the error field to be accessible directly instead of requiring
* an indirection to the redisContext struct. */
static void __redisAsyncCopyError(redisAsyncContext *ac) {
if (!ac)
return;
redisContext *c = &(ac->c);
ac->err = c->err;
ac->errstr = c->errstr;
@@ -176,14 +173,6 @@ redisAsyncContext *redisAsyncConnectBind(const char *ip, int port,
return ac;
}
redisAsyncContext *redisAsyncConnectBindWithReuse(const char *ip, int port,
const char *source_addr) {
redisContext *c = redisConnectBindNonBlockWithReuse(ip,port,source_addr);
redisAsyncContext *ac = redisAsyncInitialize(c);
__redisAsyncCopyError(ac);
return ac;
}
redisAsyncContext *redisAsyncConnectUnix(const char *path) {
redisContext *c;
redisAsyncContext *ac;
@@ -418,8 +407,7 @@ void redisProcessCallbacks(redisAsyncContext *ac) {
if (reply == NULL) {
/* When the connection is being disconnected and there are
* no more replies, this is the cue to really disconnect. */
if (c->flags & REDIS_DISCONNECTING && sdslen(c->obuf) == 0
&& ac->replies.head == NULL) {
if (c->flags & REDIS_DISCONNECTING && sdslen(c->obuf) == 0) {
__redisAsyncDisconnect(ac);
return;
}
@@ -455,7 +443,6 @@ void redisProcessCallbacks(redisAsyncContext *ac) {
if (((redisReply*)reply)->type == REDIS_REPLY_ERROR) {
c->err = REDIS_ERR_OTHER;
snprintf(c->errstr,sizeof(c->errstr),"%s",((redisReply*)reply)->str);
c->reader->fn->freeObject(reply);
__redisAsyncDisconnect(ac);
return;
}
@@ -489,7 +476,7 @@ void redisProcessCallbacks(redisAsyncContext *ac) {
}
/* Internal helper function to detect socket status the first time a read or
* write event fires. When connecting was not successful, the connect callback
* write event fires. When connecting was not succesful, the connect callback
* is called with a REDIS_ERR status and the context is free'd. */
static int __redisAsyncHandleConnect(redisAsyncContext *ac) {
redisContext *c = &(ac->c);
@@ -563,8 +550,8 @@ void redisAsyncHandleWrite(redisAsyncContext *ac) {
/* Sets a pointer to the first argument and its length starting at p. Returns
* the number of bytes to skip to get to the following argument. */
static const char *nextArgument(const char *start, const char **str, size_t *len) {
const char *p = start;
static char *nextArgument(char *start, char **str, size_t *len) {
char *p = start;
if (p[0] != '$') {
p = strchr(p,'$');
if (p == NULL) return NULL;
@@ -580,15 +567,14 @@ static const char *nextArgument(const char *start, const char **str, size_t *len
/* Helper function for the redisAsyncCommand* family of functions. Writes a
* formatted command to the output buffer and registers the provided callback
* function with the context. */
static int __redisAsyncCommand(redisAsyncContext *ac, redisCallbackFn *fn, void *privdata, const char *cmd, size_t len) {
static int __redisAsyncCommand(redisAsyncContext *ac, redisCallbackFn *fn, void *privdata, char *cmd, size_t len) {
redisContext *c = &(ac->c);
redisCallback cb;
int pvariant, hasnext;
const char *cstr, *astr;
char *cstr, *astr;
size_t clen, alen;
const char *p;
char *p;
sds sname;
int ret;
/* Don't accept new commands when the connection is about to be closed. */
if (c->flags & (REDIS_DISCONNECTING | REDIS_FREEING)) return REDIS_ERR;
@@ -612,11 +598,9 @@ static int __redisAsyncCommand(redisAsyncContext *ac, redisCallbackFn *fn, void
while ((p = nextArgument(p,&astr,&alen)) != NULL) {
sname = sdsnewlen(astr,alen);
if (pvariant)
ret = dictReplace(ac->sub.patterns,sname,&cb);
dictReplace(ac->sub.patterns,sname,&cb);
else
ret = dictReplace(ac->sub.channels,sname,&cb);
if (ret == 0) sdsfree(sname);
dictReplace(ac->sub.channels,sname,&cb);
}
} else if (strncasecmp(cstr,"unsubscribe\r\n",13) == 0) {
/* It is only useful to call (P)UNSUBSCRIBE when the context is
@@ -652,11 +636,6 @@ int redisvAsyncCommand(redisAsyncContext *ac, redisCallbackFn *fn, void *privdat
int len;
int status;
len = redisvFormatCommand(&cmd,format,ap);
/* We don't want to pass -1 or -2 to future functions as a length. */
if (len < 0)
return REDIS_ERR;
status = __redisAsyncCommand(ac,fn,privdata,cmd,len);
free(cmd);
return status;
@@ -672,16 +651,11 @@ int redisAsyncCommand(redisAsyncContext *ac, redisCallbackFn *fn, void *privdata
}
int redisAsyncCommandArgv(redisAsyncContext *ac, redisCallbackFn *fn, void *privdata, int argc, const char **argv, const size_t *argvlen) {
sds cmd;
char *cmd;
int len;
int status;
len = redisFormatSdsCommandArgv(&cmd,argc,argv,argvlen);
len = redisFormatCommandArgv(&cmd,argc,argv,argvlen);
status = __redisAsyncCommand(ac,fn,privdata,cmd,len);
sdsfree(cmd);
return status;
}
int redisAsyncFormattedCommand(redisAsyncContext *ac, redisCallbackFn *fn, void *privdata, const char *cmd, size_t len) {
int status = __redisAsyncCommand(ac,fn,privdata,cmd,len);
free(cmd);
return status;
}
-3
View File
@@ -103,8 +103,6 @@ typedef struct redisAsyncContext {
/* Functions that proxy to hiredis */
redisAsyncContext *redisAsyncConnect(const char *ip, int port);
redisAsyncContext *redisAsyncConnectBind(const char *ip, int port, const char *source_addr);
redisAsyncContext *redisAsyncConnectBindWithReuse(const char *ip, int port,
const char *source_addr);
redisAsyncContext *redisAsyncConnectUnix(const char *path);
int redisAsyncSetConnectCallback(redisAsyncContext *ac, redisConnectCallback *fn);
int redisAsyncSetDisconnectCallback(redisAsyncContext *ac, redisDisconnectCallback *fn);
@@ -120,7 +118,6 @@ void redisAsyncHandleWrite(redisAsyncContext *ac);
int redisvAsyncCommand(redisAsyncContext *ac, redisCallbackFn *fn, void *privdata, const char *format, va_list ap);
int redisAsyncCommand(redisAsyncContext *ac, redisCallbackFn *fn, void *privdata, const char *format, ...);
int redisAsyncCommandArgv(redisAsyncContext *ac, redisCallbackFn *fn, void *privdata, int argc, const char **argv, const size_t *argvlen);
int redisAsyncFormattedCommand(redisAsyncContext *ac, redisCallbackFn *fn, void *privdata, const char *cmd, size_t len);
#ifdef __cplusplus
}
+2 -2
View File
@@ -161,7 +161,7 @@ static int dictReplace(dict *ht, void *key, void *val) {
dictEntry *entry, auxentry;
/* Try to add the element. If the key
* does not exists dictAdd will succeed. */
* does not exists dictAdd will suceed. */
if (dictAdd(ht, key, val) == DICT_OK)
return 1;
/* It already exists, get the entry */
@@ -293,7 +293,7 @@ static void dictReleaseIterator(dictIterator *iter) {
/* Expand the hash table if needed */
static int _dictExpandIfNeeded(dict *ht) {
/* If the hash table is empty expand it to the initial size,
/* If the hash table is empty expand it to the intial size,
* if the table is "full" dobule its size. */
if (ht->size == 0)
return dictExpand(ht, DICT_HT_INITIAL_SIZE);
-73
View File
@@ -1,73 +0,0 @@
#include <stdlib.h>
#include <hiredis.h>
#include <async.h>
#include <adapters/glib.h>
static GMainLoop *mainloop;
static void
connect_cb (const redisAsyncContext *ac G_GNUC_UNUSED,
int status)
{
if (status != REDIS_OK) {
g_printerr("Failed to connect: %s\n", ac->errstr);
g_main_loop_quit(mainloop);
} else {
g_printerr("Connected...\n");
}
}
static void
disconnect_cb (const redisAsyncContext *ac G_GNUC_UNUSED,
int status)
{
if (status != REDIS_OK) {
g_error("Failed to disconnect: %s", ac->errstr);
} else {
g_printerr("Disconnected...\n");
g_main_loop_quit(mainloop);
}
}
static void
command_cb(redisAsyncContext *ac,
gpointer r,
gpointer user_data G_GNUC_UNUSED)
{
redisReply *reply = r;
if (reply) {
g_print("REPLY: %s\n", reply->str);
}
redisAsyncDisconnect(ac);
}
gint
main (gint argc G_GNUC_UNUSED,
gchar *argv[] G_GNUC_UNUSED)
{
redisAsyncContext *ac;
GMainContext *context = NULL;
GSource *source;
ac = redisAsyncConnect("127.0.0.1", 6379);
if (ac->err) {
g_printerr("%s\n", ac->errstr);
exit(EXIT_FAILURE);
}
source = redis_source_new(ac);
mainloop = g_main_loop_new(context, FALSE);
g_source_attach(source, context);
redisAsyncSetConnectCallback(ac, connect_cb);
redisAsyncSetDisconnectCallback(ac, disconnect_cb);
redisAsyncCommand(ac, command_cb, NULL, "SET key 1234");
redisAsyncCommand(ac, command_cb, NULL, "GET key");
g_main_loop_run(mainloop);
return EXIT_SUCCESS;
}
-58
View File
@@ -1,58 +0,0 @@
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <signal.h>
#include <hiredis.h>
#include <async.h>
#include <adapters/ivykis.h>
void getCallback(redisAsyncContext *c, void *r, void *privdata) {
redisReply *reply = r;
if (reply == NULL) return;
printf("argv[%s]: %s\n", (char*)privdata, reply->str);
/* Disconnect after receiving the reply to GET */
redisAsyncDisconnect(c);
}
void connectCallback(const redisAsyncContext *c, int status) {
if (status != REDIS_OK) {
printf("Error: %s\n", c->errstr);
return;
}
printf("Connected...\n");
}
void disconnectCallback(const redisAsyncContext *c, int status) {
if (status != REDIS_OK) {
printf("Error: %s\n", c->errstr);
return;
}
printf("Disconnected...\n");
}
int main (int argc, char **argv) {
signal(SIGPIPE, SIG_IGN);
iv_init();
redisAsyncContext *c = redisAsyncConnect("127.0.0.1", 6379);
if (c->err) {
/* Let *c leak for now... */
printf("Error: %s\n", c->errstr);
return 1;
}
redisIvykisAttach(c);
redisAsyncSetConnectCallback(c,connectCallback);
redisAsyncSetDisconnectCallback(c,disconnectCallback);
redisAsyncCommand(c, NULL, NULL, "SET key %b", argv[argc-1], strlen(argv[argc-1]));
redisAsyncCommand(c, getCallback, (char*)"end-1", "GET key");
iv_main();
iv_deinit();
return 0;
}
-66
View File
@@ -1,66 +0,0 @@
//
// Created by Дмитрий Бахвалов on 13.07.15.
// Copyright (c) 2015 Dmitry Bakhvalov. All rights reserved.
//
#include <stdio.h>
#include <hiredis.h>
#include <async.h>
#include <adapters/macosx.h>
void getCallback(redisAsyncContext *c, void *r, void *privdata) {
redisReply *reply = r;
if (reply == NULL) return;
printf("argv[%s]: %s\n", (char*)privdata, reply->str);
/* Disconnect after receiving the reply to GET */
redisAsyncDisconnect(c);
}
void connectCallback(const redisAsyncContext *c, int status) {
if (status != REDIS_OK) {
printf("Error: %s\n", c->errstr);
return;
}
printf("Connected...\n");
}
void disconnectCallback(const redisAsyncContext *c, int status) {
if (status != REDIS_OK) {
printf("Error: %s\n", c->errstr);
return;
}
CFRunLoopStop(CFRunLoopGetCurrent());
printf("Disconnected...\n");
}
int main (int argc, char **argv) {
signal(SIGPIPE, SIG_IGN);
CFRunLoopRef loop = CFRunLoopGetCurrent();
if( !loop ) {
printf("Error: Cannot get current run loop\n");
return 1;
}
redisAsyncContext *c = redisAsyncConnect("127.0.0.1", 6379);
if (c->err) {
/* Let *c leak for now... */
printf("Error: %s\n", c->errstr);
return 1;
}
redisMacOSAttach(c, loop);
redisAsyncSetConnectCallback(c,connectCallback);
redisAsyncSetDisconnectCallback(c,disconnectCallback);
redisAsyncCommand(c, NULL, NULL, "SET key %b", argv[argc-1], strlen(argv[argc-1]));
redisAsyncCommand(c, getCallback, (char*)"end-1", "GET key");
CFRunLoopRun();
return 0;
}
-46
View File
@@ -1,46 +0,0 @@
#include <iostream>
using namespace std;
#include <QCoreApplication>
#include <QTimer>
#include "example-qt.h"
void getCallback(redisAsyncContext *, void * r, void * privdata) {
redisReply * reply = static_cast<redisReply *>(r);
ExampleQt * ex = static_cast<ExampleQt *>(privdata);
if (reply == nullptr || ex == nullptr) return;
cout << "key: " << reply->str << endl;
ex->finish();
}
void ExampleQt::run() {
m_ctx = redisAsyncConnect("localhost", 6379);
if (m_ctx->err) {
cerr << "Error: " << m_ctx->errstr << endl;
redisAsyncFree(m_ctx);
emit finished();
}
m_adapter.setContext(m_ctx);
redisAsyncCommand(m_ctx, NULL, NULL, "SET key %s", m_value);
redisAsyncCommand(m_ctx, getCallback, this, "GET key");
}
int main (int argc, char **argv) {
QCoreApplication app(argc, argv);
ExampleQt example(argv[argc-1]);
QObject::connect(&example, SIGNAL(finished()), &app, SLOT(quit()));
QTimer::singleShot(0, &example, SLOT(run()));
return app.exec();
}
-32
View File
@@ -1,32 +0,0 @@
#ifndef __HIREDIS_EXAMPLE_QT_H
#define __HIREDIS_EXAMPLE_QT_H
#include <adapters/qt.h>
class ExampleQt : public QObject {
Q_OBJECT
public:
ExampleQt(const char * value, QObject * parent = 0)
: QObject(parent), m_value(value) {}
signals:
void finished();
public slots:
void run();
private:
void finish() { emit finished(); }
private:
const char * m_value;
redisAsyncContext * m_ctx;
RedisQtAdapter m_adapter;
friend
void getCallback(redisAsyncContext *, void *, void *);
};
#endif /* !__HIREDIS_EXAMPLE_QT_H */
+1 -1
View File
@@ -57,7 +57,7 @@ int main(int argc, char **argv) {
for (j = 0; j < 10; j++) {
char buf[64];
snprintf(buf,64,"%u",j);
snprintf(buf,64,"%d",j);
reply = redisCommand(c,"LPUSH mylist element-%s", buf);
freeReplyObject(reply);
}
+7 -8
View File
@@ -1,24 +1,23 @@
#ifndef __HIREDIS_FMACRO_H
#define __HIREDIS_FMACRO_H
#if defined(__linux__)
#if !defined(_BSD_SOURCE)
#define _BSD_SOURCE
#define _DEFAULT_SOURCE
#endif
#if defined(__CYGWIN__)
#include <sys/cdefs.h>
#if defined(_AIX)
#define _ALL_SOURCE
#endif
#if defined(__sun__)
#define _POSIX_C_SOURCE 200112L
#else
#if !(defined(__APPLE__) && defined(__MACH__)) && !(defined(__FreeBSD__))
#elif defined(__linux__) || defined(__OpenBSD__) || defined(__NetBSD__)
#define _XOPEN_SOURCE 600
#endif
#else
#define _XOPEN_SOURCE
#endif
#if defined(__APPLE__) && defined(__MACH__)
#if __APPLE__ && __MACH__
#define _OSX
#endif
+528 -191
View File
@@ -1,8 +1,6 @@
/*
* Copyright (c) 2009-2011, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2010-2014, Pieter Noordhuis <pcnoordhuis at gmail dot com>
* Copyright (c) 2015, Matt Stancliff <matt at genges dot com>,
* Jan-Erik Rediger <janerik at fnordig dot com>
* Copyright (c) 2010-2011, Pieter Noordhuis <pcnoordhuis at gmail dot com>
*
* All rights reserved.
*
@@ -75,9 +73,6 @@ void freeReplyObject(void *reply) {
redisReply *r = reply;
size_t j;
if (r == NULL)
return;
switch(r->type) {
case REDIS_REPLY_INTEGER:
break; /* Nothing to free */
@@ -188,23 +183,504 @@ static void *createNilObject(const redisReadTask *task) {
return r;
}
/* Return the number of digits of 'v' when converted to string in radix 10.
* Implementation borrowed from link in redis/src/util.c:string2ll(). */
static uint32_t countDigits(uint64_t v) {
uint32_t result = 1;
for (;;) {
if (v < 10) return result;
if (v < 100) return result + 1;
if (v < 1000) return result + 2;
if (v < 10000) return result + 3;
v /= 10000U;
result += 4;
}
static void __redisReaderSetError(redisReader *r, int type, const char *str) {
size_t len;
if (r->reply != NULL && r->fn && r->fn->freeObject) {
r->fn->freeObject(r->reply);
r->reply = NULL;
}
/* Clear input buffer on errors. */
if (r->buf != NULL) {
sdsfree(r->buf);
r->buf = NULL;
r->pos = r->len = 0;
}
/* Reset task stack. */
r->ridx = -1;
/* Set error. */
r->err = type;
len = strlen(str);
len = len < (sizeof(r->errstr)-1) ? len : (sizeof(r->errstr)-1);
memcpy(r->errstr,str,len);
r->errstr[len] = '\0';
}
static size_t chrtos(char *buf, size_t size, char byte) {
size_t len = 0;
switch(byte) {
case '\\':
case '"':
len = snprintf(buf,size,"\"\\%c\"",byte);
break;
case '\n': len = snprintf(buf,size,"\"\\n\""); break;
case '\r': len = snprintf(buf,size,"\"\\r\""); break;
case '\t': len = snprintf(buf,size,"\"\\t\""); break;
case '\a': len = snprintf(buf,size,"\"\\a\""); break;
case '\b': len = snprintf(buf,size,"\"\\b\""); break;
default:
if (isprint(byte))
len = snprintf(buf,size,"\"%c\"",byte);
else
len = snprintf(buf,size,"\"\\x%02x\"",(unsigned char)byte);
break;
}
return len;
}
static void __redisReaderSetErrorProtocolByte(redisReader *r, char byte) {
char cbuf[8], sbuf[128];
chrtos(cbuf,sizeof(cbuf),byte);
snprintf(sbuf,sizeof(sbuf),
"Protocol error, got %s as reply type byte", cbuf);
__redisReaderSetError(r,REDIS_ERR_PROTOCOL,sbuf);
}
static void __redisReaderSetErrorOOM(redisReader *r) {
__redisReaderSetError(r,REDIS_ERR_OOM,"Out of memory");
}
static char *readBytes(redisReader *r, unsigned int bytes) {
char *p;
if (r->len-r->pos >= bytes) {
p = r->buf+r->pos;
r->pos += bytes;
return p;
}
return NULL;
}
/* Find pointer to \r\n. */
static char *seekNewline(char *s, size_t len) {
int pos = 0;
int _len = len-1;
/* Position should be < len-1 because the character at "pos" should be
* followed by a \n. Note that strchr cannot be used because it doesn't
* allow to search a limited length and the buffer that is being searched
* might not have a trailing NULL character. */
while (pos < _len) {
while(pos < _len && s[pos] != '\r') pos++;
if (s[pos] != '\r') {
/* Not found. */
return NULL;
} else {
if (s[pos+1] == '\n') {
/* Found. */
return s+pos;
} else {
/* Continue searching. */
pos++;
}
}
}
return NULL;
}
/* Read a long long value starting at *s, under the assumption that it will be
* terminated by \r\n. Ambiguously returns -1 for unexpected input. */
static long long readLongLong(char *s) {
long long v = 0;
int dec, mult = 1;
char c;
if (*s == '-') {
mult = -1;
s++;
} else if (*s == '+') {
mult = 1;
s++;
}
while ((c = *(s++)) != '\r') {
dec = c - '0';
if (dec >= 0 && dec < 10) {
v *= 10;
v += dec;
} else {
/* Should not happen... */
return -1;
}
}
return mult*v;
}
static char *readLine(redisReader *r, int *_len) {
char *p, *s;
int len;
p = r->buf+r->pos;
s = seekNewline(p,(r->len-r->pos));
if (s != NULL) {
len = s-(r->buf+r->pos);
r->pos += len+2; /* skip \r\n */
if (_len) *_len = len;
return p;
}
return NULL;
}
static void moveToNextTask(redisReader *r) {
redisReadTask *cur, *prv;
while (r->ridx >= 0) {
/* Return a.s.a.p. when the stack is now empty. */
if (r->ridx == 0) {
r->ridx--;
return;
}
cur = &(r->rstack[r->ridx]);
prv = &(r->rstack[r->ridx-1]);
assert(prv->type == REDIS_REPLY_ARRAY);
if (cur->idx == prv->elements-1) {
r->ridx--;
} else {
/* Reset the type because the next item can be anything */
assert(cur->idx < prv->elements);
cur->type = -1;
cur->elements = -1;
cur->idx++;
return;
}
}
}
static int processLineItem(redisReader *r) {
redisReadTask *cur = &(r->rstack[r->ridx]);
void *obj;
char *p;
int len;
if ((p = readLine(r,&len)) != NULL) {
if (cur->type == REDIS_REPLY_INTEGER) {
if (r->fn && r->fn->createInteger)
obj = r->fn->createInteger(cur,readLongLong(p));
else
obj = (void*)REDIS_REPLY_INTEGER;
} else {
/* Type will be error or status. */
if (r->fn && r->fn->createString)
obj = r->fn->createString(cur,p,len);
else
obj = (void*)(size_t)(cur->type);
}
if (obj == NULL) {
__redisReaderSetErrorOOM(r);
return REDIS_ERR;
}
/* Set reply if this is the root object. */
if (r->ridx == 0) r->reply = obj;
moveToNextTask(r);
return REDIS_OK;
}
return REDIS_ERR;
}
static int processBulkItem(redisReader *r) {
redisReadTask *cur = &(r->rstack[r->ridx]);
void *obj = NULL;
char *p, *s;
long len;
unsigned long bytelen;
int success = 0;
p = r->buf+r->pos;
s = seekNewline(p,r->len-r->pos);
if (s != NULL) {
p = r->buf+r->pos;
bytelen = s-(r->buf+r->pos)+2; /* include \r\n */
len = readLongLong(p);
if (len < 0) {
/* The nil object can always be created. */
if (r->fn && r->fn->createNil)
obj = r->fn->createNil(cur);
else
obj = (void*)REDIS_REPLY_NIL;
success = 1;
} else {
/* Only continue when the buffer contains the entire bulk item. */
bytelen += len+2; /* include \r\n */
if (r->pos+bytelen <= r->len) {
if (r->fn && r->fn->createString)
obj = r->fn->createString(cur,s+2,len);
else
obj = (void*)REDIS_REPLY_STRING;
success = 1;
}
}
/* Proceed when obj was created. */
if (success) {
if (obj == NULL) {
__redisReaderSetErrorOOM(r);
return REDIS_ERR;
}
r->pos += bytelen;
/* Set reply if this is the root object. */
if (r->ridx == 0) r->reply = obj;
moveToNextTask(r);
return REDIS_OK;
}
}
return REDIS_ERR;
}
static int processMultiBulkItem(redisReader *r) {
redisReadTask *cur = &(r->rstack[r->ridx]);
void *obj;
char *p;
long elements;
int root = 0;
/* Set error for nested multi bulks with depth > 7 */
if (r->ridx == 8) {
__redisReaderSetError(r,REDIS_ERR_PROTOCOL,
"No support for nested multi bulk replies with depth > 7");
return REDIS_ERR;
}
if ((p = readLine(r,NULL)) != NULL) {
elements = readLongLong(p);
root = (r->ridx == 0);
if (elements == -1) {
if (r->fn && r->fn->createNil)
obj = r->fn->createNil(cur);
else
obj = (void*)REDIS_REPLY_NIL;
if (obj == NULL) {
__redisReaderSetErrorOOM(r);
return REDIS_ERR;
}
moveToNextTask(r);
} else {
if (r->fn && r->fn->createArray)
obj = r->fn->createArray(cur,elements);
else
obj = (void*)REDIS_REPLY_ARRAY;
if (obj == NULL) {
__redisReaderSetErrorOOM(r);
return REDIS_ERR;
}
/* Modify task stack when there are more than 0 elements. */
if (elements > 0) {
cur->elements = elements;
cur->obj = obj;
r->ridx++;
r->rstack[r->ridx].type = -1;
r->rstack[r->ridx].elements = -1;
r->rstack[r->ridx].idx = 0;
r->rstack[r->ridx].obj = NULL;
r->rstack[r->ridx].parent = cur;
r->rstack[r->ridx].privdata = r->privdata;
} else {
moveToNextTask(r);
}
}
/* Set reply if this is the root object. */
if (root) r->reply = obj;
return REDIS_OK;
}
return REDIS_ERR;
}
static int processItem(redisReader *r) {
redisReadTask *cur = &(r->rstack[r->ridx]);
char *p;
/* check if we need to read type */
if (cur->type < 0) {
if ((p = readBytes(r,1)) != NULL) {
switch (p[0]) {
case '-':
cur->type = REDIS_REPLY_ERROR;
break;
case '+':
cur->type = REDIS_REPLY_STATUS;
break;
case ':':
cur->type = REDIS_REPLY_INTEGER;
break;
case '$':
cur->type = REDIS_REPLY_STRING;
break;
case '*':
cur->type = REDIS_REPLY_ARRAY;
break;
default:
__redisReaderSetErrorProtocolByte(r,*p);
return REDIS_ERR;
}
} else {
/* could not consume 1 byte */
return REDIS_ERR;
}
}
/* process typed item */
switch(cur->type) {
case REDIS_REPLY_ERROR:
case REDIS_REPLY_STATUS:
case REDIS_REPLY_INTEGER:
return processLineItem(r);
case REDIS_REPLY_STRING:
return processBulkItem(r);
case REDIS_REPLY_ARRAY:
return processMultiBulkItem(r);
default:
assert(NULL);
return REDIS_ERR; /* Avoid warning. */
}
}
redisReader *redisReaderCreate(void) {
redisReader *r;
r = calloc(sizeof(redisReader),1);
if (r == NULL)
return NULL;
r->err = 0;
r->errstr[0] = '\0';
r->fn = &defaultFunctions;
r->buf = sdsempty();
r->maxbuf = REDIS_READER_MAX_BUF;
if (r->buf == NULL) {
free(r);
return NULL;
}
r->ridx = -1;
return r;
}
void redisReaderFree(redisReader *r) {
if (r->reply != NULL && r->fn && r->fn->freeObject)
r->fn->freeObject(r->reply);
if (r->buf != NULL)
sdsfree(r->buf);
free(r);
}
int redisReaderFeed(redisReader *r, const char *buf, size_t len) {
sds newbuf;
/* Return early when this reader is in an erroneous state. */
if (r->err)
return REDIS_ERR;
/* Copy the provided buffer. */
if (buf != NULL && len >= 1) {
/* Destroy internal buffer when it is empty and is quite large. */
if (r->len == 0 && r->maxbuf != 0 && sdsavail(r->buf) > r->maxbuf) {
sdsfree(r->buf);
r->buf = sdsempty();
r->pos = 0;
/* r->buf should not be NULL since we just free'd a larger one. */
assert(r->buf != NULL);
}
newbuf = sdscatlen(r->buf,buf,len);
if (newbuf == NULL) {
__redisReaderSetErrorOOM(r);
return REDIS_ERR;
}
r->buf = newbuf;
r->len = sdslen(r->buf);
}
return REDIS_OK;
}
int redisReaderGetReply(redisReader *r, void **reply) {
/* Default target pointer to NULL. */
if (reply != NULL)
*reply = NULL;
/* Return early when this reader is in an erroneous state. */
if (r->err)
return REDIS_ERR;
/* When the buffer is empty, there will never be a reply. */
if (r->len == 0)
return REDIS_OK;
/* Set first item to process when the stack is empty. */
if (r->ridx == -1) {
r->rstack[0].type = -1;
r->rstack[0].elements = -1;
r->rstack[0].idx = -1;
r->rstack[0].obj = NULL;
r->rstack[0].parent = NULL;
r->rstack[0].privdata = r->privdata;
r->ridx = 0;
}
/* Process items in reply. */
while (r->ridx >= 0)
if (processItem(r) != REDIS_OK)
break;
/* Return ASAP when an error occurred. */
if (r->err)
return REDIS_ERR;
/* Discard part of the buffer when we've consumed at least 1k, to avoid
* doing unnecessary calls to memmove() in sds.c. */
if (r->pos >= 1024) {
sdsrange(r->buf,r->pos,-1);
r->pos = 0;
r->len = sdslen(r->buf);
}
/* Emit a reply when there is one. */
if (r->ridx == -1) {
if (reply != NULL)
*reply = r->reply;
r->reply = NULL;
}
return REDIS_OK;
}
/* Calculate the number of bytes needed to represent an integer as string. */
static int intlen(int i) {
int len = 0;
if (i < 0) {
len++;
i = -i;
}
do {
len++;
i /= 10;
} while(i);
return len;
}
/* Helper that calculates the bulk length given a certain string length. */
static size_t bulklen(size_t len) {
return 1+countDigits(len)+2+len+2;
return 1+intlen(len)+2+len+2;
}
int redisvFormatCommand(char **target, const char *format, va_list ap) {
@@ -216,7 +692,6 @@ int redisvFormatCommand(char **target, const char *format, va_list ap) {
char **curargv = NULL, **newargv = NULL;
int argc = 0;
int totlen = 0;
int error_type = 0; /* 0 = no error; -1 = memory error; -2 = format error */
int j;
/* Abort if there is not target to set */
@@ -233,19 +708,19 @@ int redisvFormatCommand(char **target, const char *format, va_list ap) {
if (*c == ' ') {
if (touched) {
newargv = realloc(curargv,sizeof(char*)*(argc+1));
if (newargv == NULL) goto memory_err;
if (newargv == NULL) goto err;
curargv = newargv;
curargv[argc++] = curarg;
totlen += bulklen(sdslen(curarg));
/* curarg is put in argv so it can be overwritten. */
curarg = sdsempty();
if (curarg == NULL) goto memory_err;
if (curarg == NULL) goto err;
touched = 0;
}
} else {
newarg = sdscatlen(curarg,c,1);
if (newarg == NULL) goto memory_err;
if (newarg == NULL) goto err;
curarg = newarg;
touched = 1;
}
@@ -276,14 +751,17 @@ int redisvFormatCommand(char **target, const char *format, va_list ap) {
/* Try to detect printf format */
{
static const char intfmts[] = "diouxX";
static const char flags[] = "#0-+ ";
char _format[16];
const char *_p = c+1;
size_t _l = 0;
va_list _cpy;
/* Flags */
while (*_p != '\0' && strchr(flags,*_p) != NULL) _p++;
if (*_p != '\0' && *_p == '#') _p++;
if (*_p != '\0' && *_p == '0') _p++;
if (*_p != '\0' && *_p == '-') _p++;
if (*_p != '\0' && *_p == ' ') _p++;
if (*_p != '\0' && *_p == '+') _p++;
/* Field width */
while (*_p != '\0' && isdigit(*_p)) _p++;
@@ -351,7 +829,7 @@ int redisvFormatCommand(char **target, const char *format, va_list ap) {
fmt_invalid:
va_end(_cpy);
goto format_err;
goto err;
fmt_valid:
_l = (_p+1)-c;
@@ -370,7 +848,7 @@ int redisvFormatCommand(char **target, const char *format, va_list ap) {
}
}
if (newarg == NULL) goto memory_err;
if (newarg == NULL) goto err;
curarg = newarg;
touched = 1;
@@ -382,7 +860,7 @@ int redisvFormatCommand(char **target, const char *format, va_list ap) {
/* Add the last argument if needed */
if (touched) {
newargv = realloc(curargv,sizeof(char*)*(argc+1));
if (newargv == NULL) goto memory_err;
if (newargv == NULL) goto err;
curargv = newargv;
curargv[argc++] = curarg;
totlen += bulklen(sdslen(curarg));
@@ -394,11 +872,11 @@ int redisvFormatCommand(char **target, const char *format, va_list ap) {
curarg = NULL;
/* Add bytes needed to hold multi bulk count */
totlen += 1+countDigits(argc)+2;
totlen += 1+intlen(argc)+2;
/* Build the command at protocol level */
cmd = malloc(totlen+1);
if (cmd == NULL) goto memory_err;
if (cmd == NULL) goto err;
pos = sprintf(cmd,"*%d\r\n",argc);
for (j = 0; j < argc; j++) {
@@ -416,29 +894,20 @@ int redisvFormatCommand(char **target, const char *format, va_list ap) {
*target = cmd;
return totlen;
format_err:
error_type = -2;
goto cleanup;
err:
while(argc--)
sdsfree(curargv[argc]);
free(curargv);
memory_err:
error_type = -1;
goto cleanup;
cleanup:
if (curargv) {
while(argc--)
sdsfree(curargv[argc]);
free(curargv);
}
sdsfree(curarg);
if (curarg != NULL)
sdsfree(curarg);
/* No need to check cmd since it is the last statement that can fail,
* but do it anyway to be as defensive as possible. */
if (cmd != NULL)
free(cmd);
return error_type;
return -1;
}
/* Format a command according to the Redis protocol. This function
@@ -459,69 +928,9 @@ int redisFormatCommand(char **target, const char *format, ...) {
va_start(ap,format);
len = redisvFormatCommand(target,format,ap);
va_end(ap);
/* The API says "-1" means bad result, but we now also return "-2" in some
* cases. Force the return value to always be -1. */
if (len < 0)
len = -1;
return len;
}
/* Format a command according to the Redis protocol using an sds string and
* sdscatfmt for the processing of arguments. This function takes the
* number of arguments, an array with arguments and an array with their
* lengths. If the latter is set to NULL, strlen will be used to compute the
* argument lengths.
*/
int redisFormatSdsCommandArgv(sds *target, int argc, const char **argv,
const size_t *argvlen)
{
sds cmd;
unsigned long long totlen;
int j;
size_t len;
/* Abort on a NULL target */
if (target == NULL)
return -1;
/* Calculate our total size */
totlen = 1+countDigits(argc)+2;
for (j = 0; j < argc; j++) {
len = argvlen ? argvlen[j] : strlen(argv[j]);
totlen += bulklen(len);
}
/* Use an SDS string for command construction */
cmd = sdsempty();
if (cmd == NULL)
return -1;
/* We already know how much storage we need */
cmd = sdsMakeRoomFor(cmd, totlen);
if (cmd == NULL)
return -1;
/* Construct command */
cmd = sdscatfmt(cmd, "*%i\r\n", argc);
for (j=0; j < argc; j++) {
len = argvlen ? argvlen[j] : strlen(argv[j]);
cmd = sdscatfmt(cmd, "$%u\r\n", len);
cmd = sdscatlen(cmd, argv[j], len);
cmd = sdscatlen(cmd, "\r\n", sizeof("\r\n")-1);
}
assert(sdslen(cmd)==totlen);
*target = cmd;
return totlen;
}
void redisFreeSdsCommand(sds cmd) {
sdsfree(cmd);
}
/* Format a command according to the Redis protocol. This function takes the
* number of arguments, an array with arguments and an array with their
* lengths. If the latter is set to NULL, strlen will be used to compute the
@@ -533,12 +942,8 @@ int redisFormatCommandArgv(char **target, int argc, const char **argv, const siz
size_t len;
int totlen, j;
/* Abort on a NULL target */
if (target == NULL)
return -1;
/* Calculate number of bytes needed for the command */
totlen = 1+countDigits(argc)+2;
totlen = 1+intlen(argc)+2;
for (j = 0; j < argc; j++) {
len = argvlen ? argvlen[j] : strlen(argv[j]);
totlen += bulklen(len);
@@ -565,10 +970,6 @@ int redisFormatCommandArgv(char **target, int argc, const char **argv, const siz
return totlen;
}
void redisFreeCommand(char *cmd) {
free(cmd);
}
void __redisSetError(redisContext *c, int type, const char *str) {
size_t len;
@@ -581,14 +982,10 @@ void __redisSetError(redisContext *c, int type, const char *str) {
} else {
/* Only REDIS_ERR_IO may lack a description! */
assert(type == REDIS_ERR_IO);
__redis_strerror_r(errno, c->errstr, sizeof(c->errstr));
strerror_r(errno,c->errstr,sizeof(c->errstr));
}
}
redisReader *redisReaderCreate(void) {
return redisReaderCreateWithFunctions(&defaultFunctions);
}
static redisContext *redisContextInit(void) {
redisContext *c;
@@ -600,72 +997,24 @@ static redisContext *redisContextInit(void) {
c->errstr[0] = '\0';
c->obuf = sdsempty();
c->reader = redisReaderCreate();
c->tcp.host = NULL;
c->tcp.source_addr = NULL;
c->unix_sock.path = NULL;
c->timeout = NULL;
if (c->obuf == NULL || c->reader == NULL) {
redisFree(c);
return NULL;
}
return c;
}
void redisFree(redisContext *c) {
if (c == NULL)
return;
if (c->fd > 0)
close(c->fd);
if (c->obuf != NULL)
sdsfree(c->obuf);
if (c->reader != NULL)
redisReaderFree(c->reader);
if (c->tcp.host)
free(c->tcp.host);
if (c->tcp.source_addr)
free(c->tcp.source_addr);
if (c->unix_sock.path)
free(c->unix_sock.path);
if (c->timeout)
free(c->timeout);
free(c);
}
int redisFreeKeepFd(redisContext *c) {
int fd = c->fd;
c->fd = -1;
redisFree(c);
return fd;
}
int redisReconnect(redisContext *c) {
c->err = 0;
memset(c->errstr, '\0', strlen(c->errstr));
if (c->fd > 0) {
close(c->fd);
}
sdsfree(c->obuf);
redisReaderFree(c->reader);
c->obuf = sdsempty();
c->reader = redisReaderCreate();
if (c->connection_type == REDIS_CONN_TCP) {
return redisContextConnectBindTcp(c, c->tcp.host, c->tcp.port,
c->timeout, c->tcp.source_addr);
} else if (c->connection_type == REDIS_CONN_UNIX) {
return redisContextConnectUnix(c, c->unix_sock.path, c->timeout);
} else {
/* Something bad happened here and shouldn't have. There isn't
enough information in the context to reconnect. */
__redisSetError(c,REDIS_ERR_OTHER,"Not enough information to reconnect");
}
return REDIS_ERR;
int fd = c->fd;
c->fd = -1;
redisFree(c);
return fd;
}
/* Connect to a Redis instance. On error the field error in the returned
@@ -715,15 +1064,6 @@ redisContext *redisConnectBindNonBlock(const char *ip, int port,
return c;
}
redisContext *redisConnectBindNonBlockWithReuse(const char *ip, int port,
const char *source_addr) {
redisContext *c = redisContextInit();
c->flags &= ~REDIS_BLOCK;
c->flags |= REDIS_REUSEADDR;
redisContextConnectBindTcp(c,ip,port,NULL,source_addr);
return c;
}
redisContext *redisConnectUnix(const char *path) {
redisContext *c;
@@ -822,10 +1162,10 @@ int redisBufferRead(redisContext *c) {
/* Write the output buffer to the socket.
*
* Returns REDIS_OK when the buffer is empty, or (a part of) the buffer was
* successfully written to the socket. When the buffer is empty after the
* succesfully written to the socket. When the buffer is empty after the
* write operation, "done" is set to 1 (if given).
*
* Returns REDIS_ERR if an error occurred trying to write and sets
* Returns REDIS_ERR if an error occured trying to write and sets
* c->errstr to hold the appropriate error string.
*/
int redisBufferWrite(redisContext *c, int *done) {
@@ -934,9 +1274,6 @@ int redisvAppendCommand(redisContext *c, const char *format, va_list ap) {
if (len == -1) {
__redisSetError(c,REDIS_ERR_OOM,"Out of memory");
return REDIS_ERR;
} else if (len == -2) {
__redisSetError(c,REDIS_ERR_OTHER,"Invalid format string");
return REDIS_ERR;
}
if (__redisAppendCommand(c,cmd,len) != REDIS_OK) {
@@ -959,21 +1296,21 @@ int redisAppendCommand(redisContext *c, const char *format, ...) {
}
int redisAppendCommandArgv(redisContext *c, int argc, const char **argv, const size_t *argvlen) {
sds cmd;
char *cmd;
int len;
len = redisFormatSdsCommandArgv(&cmd,argc,argv,argvlen);
len = redisFormatCommandArgv(&cmd,argc,argv,argvlen);
if (len == -1) {
__redisSetError(c,REDIS_ERR_OOM,"Out of memory");
return REDIS_ERR;
}
if (__redisAppendCommand(c,cmd,len) != REDIS_OK) {
sdsfree(cmd);
free(cmd);
return REDIS_ERR;
}
sdsfree(cmd);
free(cmd);
return REDIS_OK;
}
@@ -984,7 +1321,7 @@ int redisAppendCommandArgv(redisContext *c, int argc, const char **argv, const s
* context is non-blocking, the "reply" pointer will not be used and the
* command is simply appended to the write buffer.
*
* Returns the reply when a reply was successfully retrieved. Returns NULL
* Returns the reply when a reply was succesfully retrieved. Returns NULL
* otherwise. When NULL is returned in a blocking context, the error field
* in the context will be set.
*/
+75 -78
View File
@@ -1,8 +1,6 @@
/*
* Copyright (c) 2009-2011, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2010-2014, Pieter Noordhuis <pcnoordhuis at gmail dot com>
* Copyright (c) 2015, Matt Stancliff <matt at genges dot com>,
* Jan-Erik Rediger <janerik at fnordig dot com>
* Copyright (c) 2010-2011, Pieter Noordhuis <pcnoordhuis at gmail dot com>
*
* All rights reserved.
*
@@ -33,16 +31,26 @@
#ifndef __HIREDIS_H
#define __HIREDIS_H
#include "read.h"
#include <stdio.h> /* for size_t */
#include <stdarg.h> /* for va_list */
#include <sys/time.h> /* for struct timeval */
#include <stdint.h> /* uintXX_t, etc */
#include "sds.h" /* for sds */
#define HIREDIS_MAJOR 0
#define HIREDIS_MINOR 13
#define HIREDIS_PATCH 3
#define HIREDIS_SONAME 0.13
#define HIREDIS_MINOR 11
#define HIREDIS_PATCH 0
#define REDIS_ERR -1
#define REDIS_OK 0
/* When an error occurs, the err flag in a context is set to hold the type of
* error that occured. REDIS_ERR_IO means there was an I/O error and you
* should use the "errno" variable to find out what is wrong.
* For other values, the "errstr" field will hold a description. */
#define REDIS_ERR_IO 1 /* Error in read or write */
#define REDIS_ERR_EOF 3 /* End of file */
#define REDIS_ERR_PROTOCOL 4 /* Protocol error */
#define REDIS_ERR_OOM 5 /* Out of memory */
#define REDIS_ERR_OTHER 2 /* Everything else... */
/* Connection type can be blocking or non-blocking and is set in the
* least significant bit of the flags field in redisContext. */
@@ -71,39 +79,17 @@
/* Flag that is set when monitor mode is active */
#define REDIS_MONITORING 0x40
/* Flag that is set when we should set SO_REUSEADDR before calling bind() */
#define REDIS_REUSEADDR 0x80
#define REDIS_REPLY_STRING 1
#define REDIS_REPLY_ARRAY 2
#define REDIS_REPLY_INTEGER 3
#define REDIS_REPLY_NIL 4
#define REDIS_REPLY_STATUS 5
#define REDIS_REPLY_ERROR 6
#define REDIS_READER_MAX_BUF (1024*16) /* Default max unused reader buffer. */
#define REDIS_KEEPALIVE_INTERVAL 15 /* seconds */
/* number of times we retry to connect in the case of EADDRNOTAVAIL and
* SO_REUSEADDR is being used. */
#define REDIS_CONNECT_RETRIES 10
/* strerror_r has two completely different prototypes and behaviors
* depending on system issues, so we need to operate on the error buffer
* differently depending on which strerror_r we're using. */
#ifndef _GNU_SOURCE
/* "regular" POSIX strerror_r that does the right thing. */
#define __redis_strerror_r(errno, buf, len) \
do { \
strerror_r((errno), (buf), (len)); \
} while (0)
#else
/* "bad" GNU strerror_r we need to clean up after. */
#define __redis_strerror_r(errno, buf, len) \
do { \
char *err_str = strerror_r((errno), (buf), (len)); \
/* If return value _isn't_ the start of the buffer we passed in, \
* then GNU strerror_r returned an internal static buffer and we \
* need to copy the result into our private buffer. */ \
if (err_str != (buf)) { \
strncpy((buf), err_str, ((len) - 1)); \
buf[(len)-1] = '\0'; \
} \
} while (0)
#endif
#ifdef __cplusplus
extern "C" {
#endif
@@ -112,13 +98,61 @@ extern "C" {
typedef struct redisReply {
int type; /* REDIS_REPLY_* */
long long integer; /* The integer when type is REDIS_REPLY_INTEGER */
size_t len; /* Length of string */
int len; /* Length of string */
char *str; /* Used for both REDIS_REPLY_ERROR and REDIS_REPLY_STRING */
size_t elements; /* number of elements, for REDIS_REPLY_ARRAY */
struct redisReply **element; /* elements vector for REDIS_REPLY_ARRAY */
} redisReply;
typedef struct redisReadTask {
int type;
int elements; /* number of elements in multibulk container */
int idx; /* index in parent (array) object */
void *obj; /* holds user-generated value for a read task */
struct redisReadTask *parent; /* parent task */
void *privdata; /* user-settable arbitrary field */
} redisReadTask;
typedef struct redisReplyObjectFunctions {
void *(*createString)(const redisReadTask*, char*, size_t);
void *(*createArray)(const redisReadTask*, int);
void *(*createInteger)(const redisReadTask*, long long);
void *(*createNil)(const redisReadTask*);
void (*freeObject)(void*);
} redisReplyObjectFunctions;
/* State for the protocol parser */
typedef struct redisReader {
int err; /* Error flags, 0 when there is no error */
char errstr[128]; /* String representation of error when applicable */
char *buf; /* Read buffer */
size_t pos; /* Buffer cursor */
size_t len; /* Buffer length */
size_t maxbuf; /* Max length of unused buffer */
redisReadTask rstack[9];
int ridx; /* Index of current read task */
void *reply; /* Temporary reply pointer */
redisReplyObjectFunctions *fn;
void *privdata;
} redisReader;
/* Public API for the protocol parser. */
redisReader *redisReaderCreate(void);
void redisReaderFree(redisReader *r);
int redisReaderFeed(redisReader *r, const char *buf, size_t len);
int redisReaderGetReply(redisReader *r, void **reply);
/* Backwards compatibility, can be removed on big version bump. */
#define redisReplyReaderCreate redisReaderCreate
#define redisReplyReaderFree redisReaderFree
#define redisReplyReaderFeed redisReaderFeed
#define redisReplyReaderGetReply redisReaderGetReply
#define redisReplyReaderSetPrivdata(_r, _p) (int)(((redisReader*)(_r))->privdata = (_p))
#define redisReplyReaderGetObject(_r) (((redisReader*)(_r))->reply)
#define redisReplyReaderGetError(_r) (((redisReader*)(_r))->errstr)
/* Function to free the reply objects hiredis returns by default. */
void freeReplyObject(void *reply);
@@ -127,14 +161,6 @@ void freeReplyObject(void *reply);
int redisvFormatCommand(char **target, const char *format, va_list ap);
int redisFormatCommand(char **target, const char *format, ...);
int redisFormatCommandArgv(char **target, int argc, const char **argv, const size_t *argvlen);
int redisFormatSdsCommandArgv(sds *target, int argc, const char ** argv, const size_t *argvlen);
void redisFreeCommand(char *cmd);
void redisFreeSdsCommand(sds cmd);
enum redisConnectionType {
REDIS_CONN_TCP,
REDIS_CONN_UNIX
};
/* Context for a connection to Redis */
typedef struct redisContext {
@@ -144,45 +170,16 @@ typedef struct redisContext {
int flags;
char *obuf; /* Write buffer */
redisReader *reader; /* Protocol reader */
enum redisConnectionType connection_type;
struct timeval *timeout;
struct {
char *host;
char *source_addr;
int port;
} tcp;
struct {
char *path;
} unix_sock;
} redisContext;
redisContext *redisConnect(const char *ip, int port);
redisContext *redisConnectWithTimeout(const char *ip, int port, const struct timeval tv);
redisContext *redisConnectNonBlock(const char *ip, int port);
redisContext *redisConnectBindNonBlock(const char *ip, int port,
const char *source_addr);
redisContext *redisConnectBindNonBlockWithReuse(const char *ip, int port,
const char *source_addr);
redisContext *redisConnectBindNonBlock(const char *ip, int port, const char *source_addr);
redisContext *redisConnectUnix(const char *path);
redisContext *redisConnectUnixWithTimeout(const char *path, const struct timeval tv);
redisContext *redisConnectUnixNonBlock(const char *path);
redisContext *redisConnectFd(int fd);
/**
* Reconnect the given context using the saved information.
*
* This re-uses the exact same connect options as in the initial connection.
* host, ip (or path), timeout and bind address are reused,
* flags are used unmodified from the existing context.
*
* Returns REDIS_OK on successful connect or REDIS_ERR otherwise.
*/
int redisReconnect(redisContext *c);
int redisSetTimeout(redisContext *c, const struct timeval tv);
int redisEnableKeepAlive(redisContext *c);
void redisFree(redisContext *c);
+19 -113
View File
@@ -1,9 +1,7 @@
/* Extracted from anet.c to work properly with Hiredis error reporting.
*
* Copyright (c) 2009-2011, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2010-2014, Pieter Noordhuis <pcnoordhuis at gmail dot com>
* Copyright (c) 2015, Matt Stancliff <matt at genges dot com>,
* Jan-Erik Rediger <janerik at fnordig dot com>
* Copyright (c) 2006-2011, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2010-2011, Pieter Noordhuis <pcnoordhuis at gmail dot com>
*
* All rights reserved.
*
@@ -49,7 +47,6 @@
#include <stdio.h>
#include <poll.h>
#include <limits.h>
#include <stdlib.h>
#include "net.h"
#include "sds.h"
@@ -70,7 +67,7 @@ static void __redisSetErrorFromErrno(redisContext *c, int type, const char *pref
if (prefix != NULL)
len = snprintf(buf,sizeof(buf),"%s: ",prefix);
__redis_strerror_r(errno, (char *)(buf + len), sizeof(buf) - len);
strerror_r(errno,buf+len,sizeof(buf)-len);
__redisSetError(c,type,buf);
}
@@ -141,7 +138,7 @@ int redisKeepAlive(redisContext *c, int interval) {
return REDIS_ERR;
}
#else
#if defined(__GLIBC__) && !defined(__FreeBSD_kernel__)
#ifndef __sun
val = interval;
if (setsockopt(fd, IPPROTO_TCP, TCP_KEEPIDLE, &val, sizeof(val)) < 0) {
__redisSetError(c,REDIS_ERR_OTHER,strerror(errno));
@@ -178,15 +175,19 @@ static int redisSetTcpNoDelay(redisContext *c) {
#define __MAX_MSEC (((LONG_MAX) - 999) / 1000)
static int redisContextTimeoutMsec(redisContext *c, long *result)
{
const struct timeval *timeout = c->timeout;
long msec = -1;
static int redisContextWaitReady(redisContext *c, const struct timeval *timeout) {
struct pollfd wfd[1];
long msec;
msec = -1;
wfd[0].fd = c->fd;
wfd[0].events = POLLOUT;
/* Only use timeout when not NULL. */
if (timeout != NULL) {
if (timeout->tv_usec > 1000000 || timeout->tv_sec > __MAX_MSEC) {
*result = msec;
__redisSetErrorFromErrno(c, REDIS_ERR_IO, NULL);
redisContextCloseFd(c);
return REDIS_ERR;
}
@@ -197,16 +198,6 @@ static int redisContextTimeoutMsec(redisContext *c, long *result)
}
}
*result = msec;
return REDIS_OK;
}
static int redisContextWaitReady(redisContext *c, long msec) {
struct pollfd wfd[1];
wfd[0].fd = c->fd;
wfd[0].events = POLLOUT;
if (errno == EINPROGRESS) {
int res;
@@ -265,57 +256,10 @@ int redisContextSetTimeout(redisContext *c, const struct timeval tv) {
static int _redisContextConnectTcp(redisContext *c, const char *addr, int port,
const struct timeval *timeout,
const char *source_addr) {
int s, rv, n;
int s, rv;
char _port[6]; /* strlen("65535"); */
struct addrinfo hints, *servinfo, *bservinfo, *p, *b;
int blocking = (c->flags & REDIS_BLOCK);
int reuseaddr = (c->flags & REDIS_REUSEADDR);
int reuses = 0;
long timeout_msec = -1;
servinfo = NULL;
c->connection_type = REDIS_CONN_TCP;
c->tcp.port = port;
/* We need to take possession of the passed parameters
* to make them reusable for a reconnect.
* We also carefully check we don't free data we already own,
* as in the case of the reconnect method.
*
* This is a bit ugly, but atleast it works and doesn't leak memory.
**/
if (c->tcp.host != addr) {
if (c->tcp.host)
free(c->tcp.host);
c->tcp.host = strdup(addr);
}
if (timeout) {
if (c->timeout != timeout) {
if (c->timeout == NULL)
c->timeout = malloc(sizeof(struct timeval));
memcpy(c->timeout, timeout, sizeof(struct timeval));
}
} else {
if (c->timeout)
free(c->timeout);
c->timeout = NULL;
}
if (redisContextTimeoutMsec(c, &timeout_msec) != REDIS_OK) {
__redisSetError(c, REDIS_ERR_IO, "Invalid timeout specified");
goto error;
}
if (source_addr == NULL) {
free(c->tcp.source_addr);
c->tcp.source_addr = NULL;
} else if (c->tcp.source_addr != source_addr) {
free(c->tcp.source_addr);
c->tcp.source_addr = strdup(source_addr);
}
snprintf(_port, 6, "%d", port);
memset(&hints,0,sizeof(hints));
@@ -327,7 +271,7 @@ static int _redisContextConnectTcp(redisContext *c, const char *addr, int port,
* as this would add latency to every connect. Otherwise a more sensible
* route could be: Use IPv6 if both addresses are available and there is IPv6
* connectivity. */
if ((rv = getaddrinfo(c->tcp.host,_port,&hints,&servinfo)) != 0) {
if ((rv = getaddrinfo(addr,_port,&hints,&servinfo)) != 0) {
hints.ai_family = AF_INET6;
if ((rv = getaddrinfo(addr,_port,&hints,&servinfo)) != 0) {
__redisSetError(c,REDIS_ERR_OTHER,gai_strerror(rv));
@@ -335,31 +279,21 @@ static int _redisContextConnectTcp(redisContext *c, const char *addr, int port,
}
}
for (p = servinfo; p != NULL; p = p->ai_next) {
addrretry:
if ((s = socket(p->ai_family,p->ai_socktype,p->ai_protocol)) == -1)
continue;
c->fd = s;
if (redisSetBlocking(c,0) != REDIS_OK)
goto error;
if (c->tcp.source_addr) {
if (source_addr) {
int bound = 0;
/* Using getaddrinfo saves us from self-determining IPv4 vs IPv6 */
if ((rv = getaddrinfo(c->tcp.source_addr, NULL, &hints, &bservinfo)) != 0) {
if ((rv = getaddrinfo(source_addr, NULL, &hints, &bservinfo)) != 0) {
char buf[128];
snprintf(buf,sizeof(buf),"Can't get addr: %s",gai_strerror(rv));
__redisSetError(c,REDIS_ERR_OTHER,buf);
goto error;
}
if (reuseaddr) {
n = 1;
if (setsockopt(s, SOL_SOCKET, SO_REUSEADDR, (char*) &n,
sizeof(n)) < 0) {
goto error;
}
}
for (b = bservinfo; b != NULL; b = b->ai_next) {
if (bind(s,b->ai_addr,b->ai_addrlen) != -1) {
bound = 1;
@@ -380,15 +314,8 @@ addrretry:
continue;
} else if (errno == EINPROGRESS && !blocking) {
/* This is ok. */
} else if (errno == EADDRNOTAVAIL && reuseaddr) {
if (++reuses >= REDIS_CONNECT_RETRIES) {
goto error;
} else {
redisContextCloseFd(c);
goto addrretry;
}
} else {
if (redisContextWaitReady(c,timeout_msec) != REDIS_OK)
if (redisContextWaitReady(c,timeout) != REDIS_OK)
goto error;
}
}
@@ -429,40 +356,19 @@ int redisContextConnectBindTcp(redisContext *c, const char *addr, int port,
int redisContextConnectUnix(redisContext *c, const char *path, const struct timeval *timeout) {
int blocking = (c->flags & REDIS_BLOCK);
struct sockaddr_un sa;
long timeout_msec = -1;
if (redisCreateSocket(c,AF_LOCAL) < 0)
return REDIS_ERR;
if (redisSetBlocking(c,0) != REDIS_OK)
return REDIS_ERR;
c->connection_type = REDIS_CONN_UNIX;
if (c->unix_sock.path != path)
c->unix_sock.path = strdup(path);
if (timeout) {
if (c->timeout != timeout) {
if (c->timeout == NULL)
c->timeout = malloc(sizeof(struct timeval));
memcpy(c->timeout, timeout, sizeof(struct timeval));
}
} else {
if (c->timeout)
free(c->timeout);
c->timeout = NULL;
}
if (redisContextTimeoutMsec(c,&timeout_msec) != REDIS_OK)
return REDIS_ERR;
sa.sun_family = AF_LOCAL;
strncpy(sa.sun_path,path,sizeof(sa.sun_path)-1);
if (connect(c->fd, (struct sockaddr*)&sa, sizeof(sa)) == -1) {
if (errno == EINPROGRESS && !blocking) {
/* This is ok. */
} else {
if (redisContextWaitReady(c,timeout_msec) != REDIS_OK)
if (redisContextWaitReady(c,timeout) != REDIS_OK)
return REDIS_ERR;
}
}
+3 -5
View File
@@ -1,9 +1,7 @@
/* Extracted from anet.c to work properly with Hiredis error reporting.
*
* Copyright (c) 2009-2011, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2010-2014, Pieter Noordhuis <pcnoordhuis at gmail dot com>
* Copyright (c) 2015, Matt Stancliff <matt at genges dot com>,
* Jan-Erik Rediger <janerik at fnordig dot com>
* Copyright (c) 2006-2011, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2010-2011, Pieter Noordhuis <pcnoordhuis at gmail dot com>
*
* All rights reserved.
*
@@ -37,7 +35,7 @@
#include "hiredis.h"
#if defined(__sun)
#if defined(__sun) || defined(_AIX)
#define AF_LOCAL AF_UNIX
#endif
-525
View File
@@ -1,525 +0,0 @@
/*
* Copyright (c) 2009-2011, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2010-2011, Pieter Noordhuis <pcnoordhuis at gmail dot com>
*
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Redis nor the names of its contributors may be used
* to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#include "fmacros.h"
#include <string.h>
#include <stdlib.h>
#ifndef _MSC_VER
#include <unistd.h>
#endif
#include <assert.h>
#include <errno.h>
#include <ctype.h>
#include "read.h"
#include "sds.h"
static void __redisReaderSetError(redisReader *r, int type, const char *str) {
size_t len;
if (r->reply != NULL && r->fn && r->fn->freeObject) {
r->fn->freeObject(r->reply);
r->reply = NULL;
}
/* Clear input buffer on errors. */
if (r->buf != NULL) {
sdsfree(r->buf);
r->buf = NULL;
r->pos = r->len = 0;
}
/* Reset task stack. */
r->ridx = -1;
/* Set error. */
r->err = type;
len = strlen(str);
len = len < (sizeof(r->errstr)-1) ? len : (sizeof(r->errstr)-1);
memcpy(r->errstr,str,len);
r->errstr[len] = '\0';
}
static size_t chrtos(char *buf, size_t size, char byte) {
size_t len = 0;
switch(byte) {
case '\\':
case '"':
len = snprintf(buf,size,"\"\\%c\"",byte);
break;
case '\n': len = snprintf(buf,size,"\"\\n\""); break;
case '\r': len = snprintf(buf,size,"\"\\r\""); break;
case '\t': len = snprintf(buf,size,"\"\\t\""); break;
case '\a': len = snprintf(buf,size,"\"\\a\""); break;
case '\b': len = snprintf(buf,size,"\"\\b\""); break;
default:
if (isprint(byte))
len = snprintf(buf,size,"\"%c\"",byte);
else
len = snprintf(buf,size,"\"\\x%02x\"",(unsigned char)byte);
break;
}
return len;
}
static void __redisReaderSetErrorProtocolByte(redisReader *r, char byte) {
char cbuf[8], sbuf[128];
chrtos(cbuf,sizeof(cbuf),byte);
snprintf(sbuf,sizeof(sbuf),
"Protocol error, got %s as reply type byte", cbuf);
__redisReaderSetError(r,REDIS_ERR_PROTOCOL,sbuf);
}
static void __redisReaderSetErrorOOM(redisReader *r) {
__redisReaderSetError(r,REDIS_ERR_OOM,"Out of memory");
}
static char *readBytes(redisReader *r, unsigned int bytes) {
char *p;
if (r->len-r->pos >= bytes) {
p = r->buf+r->pos;
r->pos += bytes;
return p;
}
return NULL;
}
/* Find pointer to \r\n. */
static char *seekNewline(char *s, size_t len) {
int pos = 0;
int _len = len-1;
/* Position should be < len-1 because the character at "pos" should be
* followed by a \n. Note that strchr cannot be used because it doesn't
* allow to search a limited length and the buffer that is being searched
* might not have a trailing NULL character. */
while (pos < _len) {
while(pos < _len && s[pos] != '\r') pos++;
if (pos==_len) {
/* Not found. */
return NULL;
} else {
if (s[pos+1] == '\n') {
/* Found. */
return s+pos;
} else {
/* Continue searching. */
pos++;
}
}
}
return NULL;
}
/* Read a long long value starting at *s, under the assumption that it will be
* terminated by \r\n. Ambiguously returns -1 for unexpected input. */
static long long readLongLong(char *s) {
long long v = 0;
int dec, mult = 1;
char c;
if (*s == '-') {
mult = -1;
s++;
} else if (*s == '+') {
mult = 1;
s++;
}
while ((c = *(s++)) != '\r') {
dec = c - '0';
if (dec >= 0 && dec < 10) {
v *= 10;
v += dec;
} else {
/* Should not happen... */
return -1;
}
}
return mult*v;
}
static char *readLine(redisReader *r, int *_len) {
char *p, *s;
int len;
p = r->buf+r->pos;
s = seekNewline(p,(r->len-r->pos));
if (s != NULL) {
len = s-(r->buf+r->pos);
r->pos += len+2; /* skip \r\n */
if (_len) *_len = len;
return p;
}
return NULL;
}
static void moveToNextTask(redisReader *r) {
redisReadTask *cur, *prv;
while (r->ridx >= 0) {
/* Return a.s.a.p. when the stack is now empty. */
if (r->ridx == 0) {
r->ridx--;
return;
}
cur = &(r->rstack[r->ridx]);
prv = &(r->rstack[r->ridx-1]);
assert(prv->type == REDIS_REPLY_ARRAY);
if (cur->idx == prv->elements-1) {
r->ridx--;
} else {
/* Reset the type because the next item can be anything */
assert(cur->idx < prv->elements);
cur->type = -1;
cur->elements = -1;
cur->idx++;
return;
}
}
}
static int processLineItem(redisReader *r) {
redisReadTask *cur = &(r->rstack[r->ridx]);
void *obj;
char *p;
int len;
if ((p = readLine(r,&len)) != NULL) {
if (cur->type == REDIS_REPLY_INTEGER) {
if (r->fn && r->fn->createInteger)
obj = r->fn->createInteger(cur,readLongLong(p));
else
obj = (void*)REDIS_REPLY_INTEGER;
} else {
/* Type will be error or status. */
if (r->fn && r->fn->createString)
obj = r->fn->createString(cur,p,len);
else
obj = (void*)(size_t)(cur->type);
}
if (obj == NULL) {
__redisReaderSetErrorOOM(r);
return REDIS_ERR;
}
/* Set reply if this is the root object. */
if (r->ridx == 0) r->reply = obj;
moveToNextTask(r);
return REDIS_OK;
}
return REDIS_ERR;
}
static int processBulkItem(redisReader *r) {
redisReadTask *cur = &(r->rstack[r->ridx]);
void *obj = NULL;
char *p, *s;
long len;
unsigned long bytelen;
int success = 0;
p = r->buf+r->pos;
s = seekNewline(p,r->len-r->pos);
if (s != NULL) {
p = r->buf+r->pos;
bytelen = s-(r->buf+r->pos)+2; /* include \r\n */
len = readLongLong(p);
if (len < 0) {
/* The nil object can always be created. */
if (r->fn && r->fn->createNil)
obj = r->fn->createNil(cur);
else
obj = (void*)REDIS_REPLY_NIL;
success = 1;
} else {
/* Only continue when the buffer contains the entire bulk item. */
bytelen += len+2; /* include \r\n */
if (r->pos+bytelen <= r->len) {
if (r->fn && r->fn->createString)
obj = r->fn->createString(cur,s+2,len);
else
obj = (void*)REDIS_REPLY_STRING;
success = 1;
}
}
/* Proceed when obj was created. */
if (success) {
if (obj == NULL) {
__redisReaderSetErrorOOM(r);
return REDIS_ERR;
}
r->pos += bytelen;
/* Set reply if this is the root object. */
if (r->ridx == 0) r->reply = obj;
moveToNextTask(r);
return REDIS_OK;
}
}
return REDIS_ERR;
}
static int processMultiBulkItem(redisReader *r) {
redisReadTask *cur = &(r->rstack[r->ridx]);
void *obj;
char *p;
long elements;
int root = 0;
/* Set error for nested multi bulks with depth > 7 */
if (r->ridx == 8) {
__redisReaderSetError(r,REDIS_ERR_PROTOCOL,
"No support for nested multi bulk replies with depth > 7");
return REDIS_ERR;
}
if ((p = readLine(r,NULL)) != NULL) {
elements = readLongLong(p);
root = (r->ridx == 0);
if (elements == -1) {
if (r->fn && r->fn->createNil)
obj = r->fn->createNil(cur);
else
obj = (void*)REDIS_REPLY_NIL;
if (obj == NULL) {
__redisReaderSetErrorOOM(r);
return REDIS_ERR;
}
moveToNextTask(r);
} else {
if (r->fn && r->fn->createArray)
obj = r->fn->createArray(cur,elements);
else
obj = (void*)REDIS_REPLY_ARRAY;
if (obj == NULL) {
__redisReaderSetErrorOOM(r);
return REDIS_ERR;
}
/* Modify task stack when there are more than 0 elements. */
if (elements > 0) {
cur->elements = elements;
cur->obj = obj;
r->ridx++;
r->rstack[r->ridx].type = -1;
r->rstack[r->ridx].elements = -1;
r->rstack[r->ridx].idx = 0;
r->rstack[r->ridx].obj = NULL;
r->rstack[r->ridx].parent = cur;
r->rstack[r->ridx].privdata = r->privdata;
} else {
moveToNextTask(r);
}
}
/* Set reply if this is the root object. */
if (root) r->reply = obj;
return REDIS_OK;
}
return REDIS_ERR;
}
static int processItem(redisReader *r) {
redisReadTask *cur = &(r->rstack[r->ridx]);
char *p;
/* check if we need to read type */
if (cur->type < 0) {
if ((p = readBytes(r,1)) != NULL) {
switch (p[0]) {
case '-':
cur->type = REDIS_REPLY_ERROR;
break;
case '+':
cur->type = REDIS_REPLY_STATUS;
break;
case ':':
cur->type = REDIS_REPLY_INTEGER;
break;
case '$':
cur->type = REDIS_REPLY_STRING;
break;
case '*':
cur->type = REDIS_REPLY_ARRAY;
break;
default:
__redisReaderSetErrorProtocolByte(r,*p);
return REDIS_ERR;
}
} else {
/* could not consume 1 byte */
return REDIS_ERR;
}
}
/* process typed item */
switch(cur->type) {
case REDIS_REPLY_ERROR:
case REDIS_REPLY_STATUS:
case REDIS_REPLY_INTEGER:
return processLineItem(r);
case REDIS_REPLY_STRING:
return processBulkItem(r);
case REDIS_REPLY_ARRAY:
return processMultiBulkItem(r);
default:
assert(NULL);
return REDIS_ERR; /* Avoid warning. */
}
}
redisReader *redisReaderCreateWithFunctions(redisReplyObjectFunctions *fn) {
redisReader *r;
r = calloc(sizeof(redisReader),1);
if (r == NULL)
return NULL;
r->err = 0;
r->errstr[0] = '\0';
r->fn = fn;
r->buf = sdsempty();
r->maxbuf = REDIS_READER_MAX_BUF;
if (r->buf == NULL) {
free(r);
return NULL;
}
r->ridx = -1;
return r;
}
void redisReaderFree(redisReader *r) {
if (r->reply != NULL && r->fn && r->fn->freeObject)
r->fn->freeObject(r->reply);
if (r->buf != NULL)
sdsfree(r->buf);
free(r);
}
int redisReaderFeed(redisReader *r, const char *buf, size_t len) {
sds newbuf;
/* Return early when this reader is in an erroneous state. */
if (r->err)
return REDIS_ERR;
/* Copy the provided buffer. */
if (buf != NULL && len >= 1) {
/* Destroy internal buffer when it is empty and is quite large. */
if (r->len == 0 && r->maxbuf != 0 && sdsavail(r->buf) > r->maxbuf) {
sdsfree(r->buf);
r->buf = sdsempty();
r->pos = 0;
/* r->buf should not be NULL since we just free'd a larger one. */
assert(r->buf != NULL);
}
newbuf = sdscatlen(r->buf,buf,len);
if (newbuf == NULL) {
__redisReaderSetErrorOOM(r);
return REDIS_ERR;
}
r->buf = newbuf;
r->len = sdslen(r->buf);
}
return REDIS_OK;
}
int redisReaderGetReply(redisReader *r, void **reply) {
/* Default target pointer to NULL. */
if (reply != NULL)
*reply = NULL;
/* Return early when this reader is in an erroneous state. */
if (r->err)
return REDIS_ERR;
/* When the buffer is empty, there will never be a reply. */
if (r->len == 0)
return REDIS_OK;
/* Set first item to process when the stack is empty. */
if (r->ridx == -1) {
r->rstack[0].type = -1;
r->rstack[0].elements = -1;
r->rstack[0].idx = -1;
r->rstack[0].obj = NULL;
r->rstack[0].parent = NULL;
r->rstack[0].privdata = r->privdata;
r->ridx = 0;
}
/* Process items in reply. */
while (r->ridx >= 0)
if (processItem(r) != REDIS_OK)
break;
/* Return ASAP when an error occurred. */
if (r->err)
return REDIS_ERR;
/* Discard part of the buffer when we've consumed at least 1k, to avoid
* doing unnecessary calls to memmove() in sds.c. */
if (r->pos >= 1024) {
sdsrange(r->buf,r->pos,-1);
r->pos = 0;
r->len = sdslen(r->buf);
}
/* Emit a reply when there is one. */
if (r->ridx == -1) {
if (reply != NULL)
*reply = r->reply;
r->reply = NULL;
}
return REDIS_OK;
}
-111
View File
@@ -1,111 +0,0 @@
/*
* Copyright (c) 2009-2011, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2010-2011, Pieter Noordhuis <pcnoordhuis at gmail dot com>
*
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Redis nor the names of its contributors may be used
* to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#ifndef __HIREDIS_READ_H
#define __HIREDIS_READ_H
#include <stdio.h> /* for size_t */
#define REDIS_ERR -1
#define REDIS_OK 0
/* When an error occurs, the err flag in a context is set to hold the type of
* error that occurred. REDIS_ERR_IO means there was an I/O error and you
* should use the "errno" variable to find out what is wrong.
* For other values, the "errstr" field will hold a description. */
#define REDIS_ERR_IO 1 /* Error in read or write */
#define REDIS_ERR_EOF 3 /* End of file */
#define REDIS_ERR_PROTOCOL 4 /* Protocol error */
#define REDIS_ERR_OOM 5 /* Out of memory */
#define REDIS_ERR_OTHER 2 /* Everything else... */
#define REDIS_REPLY_STRING 1
#define REDIS_REPLY_ARRAY 2
#define REDIS_REPLY_INTEGER 3
#define REDIS_REPLY_NIL 4
#define REDIS_REPLY_STATUS 5
#define REDIS_REPLY_ERROR 6
#define REDIS_READER_MAX_BUF (1024*16) /* Default max unused reader buffer. */
#ifdef __cplusplus
extern "C" {
#endif
typedef struct redisReadTask {
int type;
int elements; /* number of elements in multibulk container */
int idx; /* index in parent (array) object */
void *obj; /* holds user-generated value for a read task */
struct redisReadTask *parent; /* parent task */
void *privdata; /* user-settable arbitrary field */
} redisReadTask;
typedef struct redisReplyObjectFunctions {
void *(*createString)(const redisReadTask*, char*, size_t);
void *(*createArray)(const redisReadTask*, int);
void *(*createInteger)(const redisReadTask*, long long);
void *(*createNil)(const redisReadTask*);
void (*freeObject)(void*);
} redisReplyObjectFunctions;
typedef struct redisReader {
int err; /* Error flags, 0 when there is no error */
char errstr[128]; /* String representation of error when applicable */
char *buf; /* Read buffer */
size_t pos; /* Buffer cursor */
size_t len; /* Buffer length */
size_t maxbuf; /* Max length of unused buffer */
redisReadTask rstack[9];
int ridx; /* Index of current read task */
void *reply; /* Temporary reply pointer */
redisReplyObjectFunctions *fn;
void *privdata;
} redisReader;
/* Public API for the protocol parser. */
redisReader *redisReaderCreateWithFunctions(redisReplyObjectFunctions *fn);
void redisReaderFree(redisReader *r);
int redisReaderFeed(redisReader *r, const char *buf, size_t len);
int redisReaderGetReply(redisReader *r, void **reply);
#define redisReaderSetPrivdata(_r, _p) (int)(((redisReader*)(_r))->privdata = (_p))
#define redisReaderGetObject(_r) (((redisReader*)(_r))->reply)
#define redisReaderGetError(_r) (((redisReader*)(_r))->errstr)
#ifdef __cplusplus
}
#endif
#endif
+135 -304
View File
@@ -1,8 +1,6 @@
/* SDSLib 2.0 -- A C dynamic strings library
/* SDSLib, A C dynamic strings library
*
* Copyright (c) 2006-2015, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2015, Oran Agra
* Copyright (c) 2015, Redis Labs, Inc
* Copyright (c) 2006-2012, Salvatore Sanfilippo <antirez at gmail dot com>
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -36,35 +34,7 @@
#include <ctype.h>
#include <assert.h>
#include "sds.h"
#include "sdsalloc.h"
static inline int sdsHdrSize(char type) {
switch(type&SDS_TYPE_MASK) {
case SDS_TYPE_5:
return sizeof(struct sdshdr5);
case SDS_TYPE_8:
return sizeof(struct sdshdr8);
case SDS_TYPE_16:
return sizeof(struct sdshdr16);
case SDS_TYPE_32:
return sizeof(struct sdshdr32);
case SDS_TYPE_64:
return sizeof(struct sdshdr64);
}
return 0;
}
static inline char sdsReqType(size_t string_size) {
if (string_size < 32)
return SDS_TYPE_5;
if (string_size < 0xff)
return SDS_TYPE_8;
if (string_size < 0xffff)
return SDS_TYPE_16;
if (string_size < 0xffffffff)
return SDS_TYPE_32;
return SDS_TYPE_64;
}
#include "zmalloc.h"
/* Create a new sds string with the content specified by the 'init' pointer
* and 'initlen'.
@@ -73,65 +43,26 @@ static inline char sdsReqType(size_t string_size) {
* The string is always null-termined (all the sds strings are, always) so
* even if you create an sds string with:
*
* mystring = sdsnewlen("abc",3);
* mystring = sdsnewlen("abc",3");
*
* You can print the string with printf() as there is an implicit \0 at the
* end of the string. However the string is binary safe and can contain
* \0 characters in the middle, as the length is stored in the sds header. */
sds sdsnewlen(const void *init, size_t initlen) {
void *sh;
sds s;
char type = sdsReqType(initlen);
/* Empty strings are usually created in order to append. Use type 8
* since type 5 is not good at this. */
if (type == SDS_TYPE_5 && initlen == 0) type = SDS_TYPE_8;
int hdrlen = sdsHdrSize(type);
unsigned char *fp; /* flags pointer. */
struct sdshdr *sh;
sh = s_malloc(hdrlen+initlen+1);
if (sh == NULL) return NULL;
if (!init)
memset(sh, 0, hdrlen+initlen+1);
s = (char*)sh+hdrlen;
fp = ((unsigned char*)s)-1;
switch(type) {
case SDS_TYPE_5: {
*fp = type | (initlen << SDS_TYPE_BITS);
break;
}
case SDS_TYPE_8: {
SDS_HDR_VAR(8,s);
sh->len = initlen;
sh->alloc = initlen;
*fp = type;
break;
}
case SDS_TYPE_16: {
SDS_HDR_VAR(16,s);
sh->len = initlen;
sh->alloc = initlen;
*fp = type;
break;
}
case SDS_TYPE_32: {
SDS_HDR_VAR(32,s);
sh->len = initlen;
sh->alloc = initlen;
*fp = type;
break;
}
case SDS_TYPE_64: {
SDS_HDR_VAR(64,s);
sh->len = initlen;
sh->alloc = initlen;
*fp = type;
break;
}
if (init) {
sh = zmalloc(sizeof(struct sdshdr)+initlen+1);
} else {
sh = zcalloc(sizeof(struct sdshdr)+initlen+1);
}
if (sh == NULL) return NULL;
sh->len = initlen;
sh->free = 0;
if (initlen && init)
memcpy(s, init, initlen);
s[initlen] = '\0';
return s;
memcpy(sh->buf, init, initlen);
sh->buf[initlen] = '\0';
return (char*)sh->buf;
}
/* Create an empty (zero length) sds string. Even in this case the string
@@ -140,7 +71,7 @@ sds sdsempty(void) {
return sdsnewlen("",0);
}
/* Create a new sds string starting from a null terminated C string. */
/* Create a new sds string starting from a null termined C string. */
sds sdsnew(const char *init) {
size_t initlen = (init == NULL) ? 0 : strlen(init);
return sdsnewlen(init, initlen);
@@ -154,7 +85,7 @@ sds sdsdup(const sds s) {
/* Free an sds string. No operation is performed if 's' is NULL. */
void sdsfree(sds s) {
if (s == NULL) return;
s_free((char*)s-sdsHdrSize(s[-1]));
zfree(s-sizeof(struct sdshdr));
}
/* Set the sds string length to the length as obtained with strlen(), so
@@ -172,17 +103,21 @@ void sdsfree(sds s) {
* the output will be "6" as the string was modified but the logical length
* remains 6 bytes. */
void sdsupdatelen(sds s) {
struct sdshdr *sh = (void*) (s-(sizeof(struct sdshdr)));
int reallen = strlen(s);
sdssetlen(s, reallen);
sh->free += (sh->len-reallen);
sh->len = reallen;
}
/* Modify an sds string in-place to make it empty (zero length).
/* Modify an sds string on-place to make it empty (zero length).
* However all the existing buffer is not discarded but set as free space
* so that next append operations will not require allocations up to the
* number of bytes previously available. */
void sdsclear(sds s) {
sdssetlen(s, 0);
s[0] = '\0';
struct sdshdr *sh = (void*) (s-(sizeof(struct sdshdr)));
sh->free += sh->len;
sh->len = 0;
sh->buf[0] = '\0';
}
/* Enlarge the free space at the end of the sds string so that the caller
@@ -192,48 +127,23 @@ void sdsclear(sds s) {
* Note: this does not change the *length* of the sds string as returned
* by sdslen(), but only the free buffer space we have. */
sds sdsMakeRoomFor(sds s, size_t addlen) {
void *sh, *newsh;
size_t avail = sdsavail(s);
struct sdshdr *sh, *newsh;
size_t free = sdsavail(s);
size_t len, newlen;
char type, oldtype = s[-1] & SDS_TYPE_MASK;
int hdrlen;
/* Return ASAP if there is enough space left. */
if (avail >= addlen) return s;
if (free >= addlen) return s;
len = sdslen(s);
sh = (char*)s-sdsHdrSize(oldtype);
sh = (void*) (s-(sizeof(struct sdshdr)));
newlen = (len+addlen);
if (newlen < SDS_MAX_PREALLOC)
newlen *= 2;
else
newlen += SDS_MAX_PREALLOC;
newsh = zrealloc(sh, sizeof(struct sdshdr)+newlen+1);
if (newsh == NULL) return NULL;
type = sdsReqType(newlen);
/* Don't use type 5: the user is appending to the string and type 5 is
* not able to remember empty space, so sdsMakeRoomFor() must be called
* at every appending operation. */
if (type == SDS_TYPE_5) type = SDS_TYPE_8;
hdrlen = sdsHdrSize(type);
if (oldtype==type) {
newsh = s_realloc(sh, hdrlen+newlen+1);
if (newsh == NULL) return NULL;
s = (char*)newsh+hdrlen;
} else {
/* Since the header size changes, need to move the string forward,
* and can't use realloc */
newsh = s_malloc(hdrlen+newlen+1);
if (newsh == NULL) return NULL;
memcpy((char*)newsh+hdrlen, s, len+1);
s_free(sh);
s = (char*)newsh+hdrlen;
s[-1] = type;
sdssetlen(s, len);
}
sdssetalloc(s, newlen);
return s;
newsh->free = newlen - len;
return newsh->buf;
}
/* Reallocate the sds string so that it has no free space at the end. The
@@ -243,29 +153,12 @@ sds sdsMakeRoomFor(sds s, size_t addlen) {
* After the call, the passed sds string is no longer valid and all the
* references must be substituted with the new pointer returned by the call. */
sds sdsRemoveFreeSpace(sds s) {
void *sh, *newsh;
char type, oldtype = s[-1] & SDS_TYPE_MASK;
int hdrlen;
size_t len = sdslen(s);
sh = (char*)s-sdsHdrSize(oldtype);
struct sdshdr *sh;
type = sdsReqType(len);
hdrlen = sdsHdrSize(type);
if (oldtype==type) {
newsh = s_realloc(sh, hdrlen+len+1);
if (newsh == NULL) return NULL;
s = (char*)newsh+hdrlen;
} else {
newsh = s_malloc(hdrlen+len+1);
if (newsh == NULL) return NULL;
memcpy((char*)newsh+hdrlen, s, len+1);
s_free(sh);
s = (char*)newsh+hdrlen;
s[-1] = type;
sdssetlen(s, len);
}
sdssetalloc(s, len);
return s;
sh = (void*) (s-(sizeof(struct sdshdr)));
sh = zrealloc(sh, sizeof(struct sdshdr)+sh->len+1);
sh->free = 0;
return sh->buf;
}
/* Return the total size of the allocation of the specifed sds string,
@@ -276,14 +169,9 @@ sds sdsRemoveFreeSpace(sds s) {
* 4) The implicit null term.
*/
size_t sdsAllocSize(sds s) {
size_t alloc = sdsalloc(s);
return sdsHdrSize(s[-1])+alloc+1;
}
struct sdshdr *sh = (void*) (s-(sizeof(struct sdshdr)));
/* Return the pointer of the actual SDS allocation (normally SDS strings
* are referenced by the start of the string buffer). */
void *sdsAllocPtr(sds s) {
return (void*) (s-sdsHdrSize(s[-1]));
return sizeof(*sh)+sh->len+sh->free+1;
}
/* Increment the sds length and decrements the left free space at the
@@ -310,44 +198,15 @@ void *sdsAllocPtr(sds s) {
* sdsIncrLen(s, nread);
*/
void sdsIncrLen(sds s, int incr) {
unsigned char flags = s[-1];
size_t len;
switch(flags&SDS_TYPE_MASK) {
case SDS_TYPE_5: {
unsigned char *fp = ((unsigned char*)s)-1;
unsigned char oldlen = SDS_TYPE_5_LEN(flags);
assert((incr > 0 && oldlen+incr < 32) || (incr < 0 && oldlen >= (unsigned int)(-incr)));
*fp = SDS_TYPE_5 | ((oldlen+incr) << SDS_TYPE_BITS);
len = oldlen+incr;
break;
}
case SDS_TYPE_8: {
SDS_HDR_VAR(8,s);
assert((incr >= 0 && sh->alloc-sh->len >= incr) || (incr < 0 && sh->len >= (unsigned int)(-incr)));
len = (sh->len += incr);
break;
}
case SDS_TYPE_16: {
SDS_HDR_VAR(16,s);
assert((incr >= 0 && sh->alloc-sh->len >= incr) || (incr < 0 && sh->len >= (unsigned int)(-incr)));
len = (sh->len += incr);
break;
}
case SDS_TYPE_32: {
SDS_HDR_VAR(32,s);
assert((incr >= 0 && sh->alloc-sh->len >= (unsigned int)incr) || (incr < 0 && sh->len >= (unsigned int)(-incr)));
len = (sh->len += incr);
break;
}
case SDS_TYPE_64: {
SDS_HDR_VAR(64,s);
assert((incr >= 0 && sh->alloc-sh->len >= (uint64_t)incr) || (incr < 0 && sh->len >= (uint64_t)(-incr)));
len = (sh->len += incr);
break;
}
default: len = 0; /* Just to avoid compilation warnings. */
}
s[len] = '\0';
struct sdshdr *sh = (void*) (s-(sizeof(struct sdshdr)));
if (incr >= 0)
assert(sh->free >= (unsigned int)incr);
else
assert(sh->len >= (unsigned int)(-incr));
sh->len += incr;
sh->free -= incr;
s[sh->len] = '\0';
}
/* Grow the sds to have the specified length. Bytes that were not part of
@@ -356,15 +215,19 @@ void sdsIncrLen(sds s, int incr) {
* if the specified length is smaller than the current length, no operation
* is performed. */
sds sdsgrowzero(sds s, size_t len) {
size_t curlen = sdslen(s);
struct sdshdr *sh = (void*)(s-(sizeof(struct sdshdr)));
size_t totlen, curlen = sh->len;
if (len <= curlen) return s;
s = sdsMakeRoomFor(s,len-curlen);
if (s == NULL) return NULL;
/* Make sure added region doesn't contain garbage */
sh = (void*)(s-(sizeof(struct sdshdr)));
memset(s+curlen,0,(len-curlen+1)); /* also set trailing \0 byte */
sdssetlen(s, len);
totlen = sh->len+sh->free;
sh->len = len;
sh->free = totlen-sh->len;
return s;
}
@@ -374,12 +237,15 @@ sds sdsgrowzero(sds s, size_t len) {
* After the call, the passed sds string is no longer valid and all the
* references must be substituted with the new pointer returned by the call. */
sds sdscatlen(sds s, const void *t, size_t len) {
struct sdshdr *sh;
size_t curlen = sdslen(s);
s = sdsMakeRoomFor(s,len);
if (s == NULL) return NULL;
sh = (void*) (s-(sizeof(struct sdshdr)));
memcpy(s+curlen, t, len);
sdssetlen(s, curlen+len);
sh->len = curlen+len;
sh->free = sh->free-len;
s[curlen+len] = '\0';
return s;
}
@@ -403,13 +269,19 @@ sds sdscatsds(sds s, const sds t) {
/* Destructively modify the sds string 's' to hold the specified binary
* safe string pointed by 't' of length 'len' bytes. */
sds sdscpylen(sds s, const char *t, size_t len) {
if (sdsalloc(s) < len) {
s = sdsMakeRoomFor(s,len-sdslen(s));
struct sdshdr *sh = (void*) (s-(sizeof(struct sdshdr)));
size_t totlen = sh->free+sh->len;
if (totlen < len) {
s = sdsMakeRoomFor(s,len-sh->len);
if (s == NULL) return NULL;
sh = (void*) (s-(sizeof(struct sdshdr)));
totlen = sh->free+sh->len;
}
memcpy(s, t, len);
s[len] = '\0';
sdssetlen(s, len);
sh->len = len;
sh->free = totlen-len;
return s;
}
@@ -423,7 +295,7 @@ sds sdscpy(sds s, const char *t) {
* conversion. 's' must point to a string with room for at least
* SDS_LLSTR_SIZE bytes.
*
* The function returns the length of the null-terminated string
* The function returns the lenght of the null-terminated string
* representation stored at 's'. */
#define SDS_LLSTR_SIZE 21
int sdsll2str(char *s, long long value) {
@@ -497,7 +369,7 @@ sds sdsfromlonglong(long long value) {
return sdsnewlen(buf,len);
}
/* Like sdscatprintf() but gets va_list instead of being variadic. */
/* Like sdscatpritf() but gets va_list instead of being variadic. */
sds sdscatvprintf(sds s, const char *fmt, va_list ap) {
va_list cpy;
char staticbuf[1024], *buf = staticbuf, *t;
@@ -506,7 +378,7 @@ sds sdscatvprintf(sds s, const char *fmt, va_list ap) {
/* We try to start using a static buffer for speed.
* If not possible we revert to heap allocation. */
if (buflen > sizeof(staticbuf)) {
buf = s_malloc(buflen);
buf = zmalloc(buflen);
if (buf == NULL) return NULL;
} else {
buflen = sizeof(staticbuf);
@@ -518,11 +390,11 @@ sds sdscatvprintf(sds s, const char *fmt, va_list ap) {
buf[buflen-2] = '\0';
va_copy(cpy,ap);
vsnprintf(buf, buflen, fmt, cpy);
va_end(cpy);
va_end(ap);
if (buf[buflen-2] != '\0') {
if (buf != staticbuf) s_free(buf);
if (buf != staticbuf) zfree(buf);
buflen *= 2;
buf = s_malloc(buflen);
buf = zmalloc(buflen);
if (buf == NULL) return NULL;
continue;
}
@@ -531,7 +403,7 @@ sds sdscatvprintf(sds s, const char *fmt, va_list ap) {
/* Finally concat the obtained string to the SDS string and return it. */
t = sdscat(s, buf);
if (buf != staticbuf) s_free(buf);
if (buf != staticbuf) zfree(buf);
return t;
}
@@ -543,7 +415,7 @@ sds sdscatvprintf(sds s, const char *fmt, va_list ap) {
*
* Example:
*
* s = sdsnew("Sum is: ");
* s = sdsempty("Sum is: ");
* s = sdscatprintf(s,"%d+%d = %d",a,b,a+b).
*
* Often you need to create a string from scratch with the printf-alike
@@ -577,21 +449,25 @@ sds sdscatprintf(sds s, const char *fmt, ...) {
* %% - Verbatim "%" character.
*/
sds sdscatfmt(sds s, char const *fmt, ...) {
struct sdshdr *sh = (void*) (s-(sizeof(struct sdshdr)));
size_t initlen = sdslen(s);
const char *f = fmt;
int i;
va_list ap;
va_start(ap,fmt);
i = sdslen(s); /* Position of the next byte to write to dest str. */
f = fmt; /* Next format specifier byte to process. */
i = initlen; /* Position of the next byte to write to dest str. */
while(*f) {
char next, *str;
size_t l;
unsigned int l;
long long num;
unsigned long long unum;
/* Make sure there is always space for at least 1 char. */
if (sdsavail(s)==0) {
if (sh->free == 0) {
s = sdsMakeRoomFor(s,1);
sh = (void*) (s-(sizeof(struct sdshdr)));
}
switch(*f) {
@@ -603,11 +479,13 @@ sds sdscatfmt(sds s, char const *fmt, ...) {
case 'S':
str = va_arg(ap,char*);
l = (next == 's') ? strlen(str) : sdslen(str);
if (sdsavail(s) < l) {
if (sh->free < l) {
s = sdsMakeRoomFor(s,l);
sh = (void*) (s-(sizeof(struct sdshdr)));
}
memcpy(s+i,str,l);
sdsinclen(s,l);
sh->len += l;
sh->free -= l;
i += l;
break;
case 'i':
@@ -619,11 +497,13 @@ sds sdscatfmt(sds s, char const *fmt, ...) {
{
char buf[SDS_LLSTR_SIZE];
l = sdsll2str(buf,num);
if (sdsavail(s) < l) {
if (sh->free < l) {
s = sdsMakeRoomFor(s,l);
sh = (void*) (s-(sizeof(struct sdshdr)));
}
memcpy(s+i,buf,l);
sdsinclen(s,l);
sh->len += l;
sh->free -= l;
i += l;
}
break;
@@ -636,23 +516,27 @@ sds sdscatfmt(sds s, char const *fmt, ...) {
{
char buf[SDS_LLSTR_SIZE];
l = sdsull2str(buf,unum);
if (sdsavail(s) < l) {
if (sh->free < l) {
s = sdsMakeRoomFor(s,l);
sh = (void*) (s-(sizeof(struct sdshdr)));
}
memcpy(s+i,buf,l);
sdsinclen(s,l);
sh->len += l;
sh->free -= l;
i += l;
}
break;
default: /* Handle %% and generally %<unknown>. */
s[i++] = next;
sdsinclen(s,1);
sh->len += 1;
sh->free -= 1;
break;
}
break;
default:
s[i++] = *f;
sdsinclen(s,1);
sh->len += 1;
sh->free -= 1;
break;
}
f++;
@@ -673,23 +557,25 @@ sds sdscatfmt(sds s, char const *fmt, ...) {
* Example:
*
* s = sdsnew("AA...AA.a.aa.aHelloWorld :::");
* s = sdstrim(s,"Aa. :");
* s = sdstrim(s,"A. :");
* printf("%s\n", s);
*
* Output will be just "Hello World".
*/
sds sdstrim(sds s, const char *cset) {
struct sdshdr *sh = (void*) (s-(sizeof(struct sdshdr)));
char *start, *end, *sp, *ep;
size_t len;
sp = start = s;
ep = end = s+sdslen(s)-1;
while(sp <= end && strchr(cset, *sp)) sp++;
while(ep > sp && strchr(cset, *ep)) ep--;
while(ep > start && strchr(cset, *ep)) ep--;
len = (sp > ep) ? 0 : ((ep-sp)+1);
if (s != sp) memmove(s, sp, len);
s[len] = '\0';
sdssetlen(s,len);
if (sh->buf != sp) memmove(sh->buf, sp, len);
sh->buf[len] = '\0';
sh->free = sh->free+(sh->len-len);
sh->len = len;
return s;
}
@@ -710,6 +596,7 @@ sds sdstrim(sds s, const char *cset) {
* sdsrange(s,1,-1); => "ello World"
*/
void sdsrange(sds s, int start, int end) {
struct sdshdr *sh = (void*) (s-(sizeof(struct sdshdr)));
size_t newlen, len = sdslen(s);
if (len == 0) return;
@@ -732,9 +619,10 @@ void sdsrange(sds s, int start, int end) {
} else {
start = 0;
}
if (start && newlen) memmove(s, s+start, newlen);
s[newlen] = 0;
sdssetlen(s,newlen);
if (start && newlen) memmove(sh->buf, sh->buf+start, newlen);
sh->buf[newlen] = 0;
sh->free = sh->free+(sh->len-newlen);
sh->len = newlen;
}
/* Apply tolower() to every character of the sds string 's'. */
@@ -755,8 +643,8 @@ void sdstoupper(sds s) {
*
* Return value:
*
* positive if s1 > s2.
* negative if s1 < s2.
* 1 if s1 > s2.
* -1 if s1 < s2.
* 0 if s1 and s2 are exactly the same binary string.
*
* If two strings share exactly the same prefix, but one of the two has
@@ -796,7 +684,7 @@ sds *sdssplitlen(const char *s, int len, const char *sep, int seplen, int *count
if (seplen < 1 || len < 0) return NULL;
tokens = s_malloc(sizeof(sds)*slots);
tokens = zmalloc(sizeof(sds)*slots);
if (tokens == NULL) return NULL;
if (len == 0) {
@@ -809,7 +697,7 @@ sds *sdssplitlen(const char *s, int len, const char *sep, int seplen, int *count
sds *newtokens;
slots *= 2;
newtokens = s_realloc(tokens,sizeof(sds)*slots);
newtokens = zrealloc(tokens,sizeof(sds)*slots);
if (newtokens == NULL) goto cleanup;
tokens = newtokens;
}
@@ -833,7 +721,7 @@ cleanup:
{
int i;
for (i = 0; i < elements; i++) sdsfree(tokens[i]);
s_free(tokens);
zfree(tokens);
*count = 0;
return NULL;
}
@@ -844,7 +732,7 @@ void sdsfreesplitres(sds *tokens, int count) {
if (!tokens) return;
while(count--)
sdsfree(tokens[count]);
s_free(tokens);
zfree(tokens);
}
/* Append to the sds string "s" an escaped string representation where
@@ -1018,13 +906,13 @@ sds *sdssplitargs(const char *line, int *argc) {
if (*p) p++;
}
/* add the token to the vector */
vector = s_realloc(vector,((*argc)+1)*sizeof(char*));
vector = zrealloc(vector,((*argc)+1)*sizeof(char*));
vector[*argc] = current;
(*argc)++;
current = NULL;
} else {
/* Even on empty input string return something not NULL. */
if (vector == NULL) vector = s_malloc(sizeof(void*));
if (vector == NULL) vector = zmalloc(sizeof(void*));
return vector;
}
}
@@ -1032,7 +920,7 @@ sds *sdssplitargs(const char *line, int *argc) {
err:
while((*argc)--)
sdsfree(vector[*argc]);
s_free(vector);
zfree(vector);
if (current) sdsfree(current);
*argc = 0;
return NULL;
@@ -1074,35 +962,14 @@ sds sdsjoin(char **argv, int argc, char *sep) {
return join;
}
/* Like sdsjoin, but joins an array of SDS strings. */
sds sdsjoinsds(sds *argv, int argc, const char *sep, size_t seplen) {
sds join = sdsempty();
int j;
for (j = 0; j < argc; j++) {
join = sdscatsds(join, argv[j]);
if (j != argc-1) join = sdscatlen(join,sep,seplen);
}
return join;
}
/* Wrappers to the allocators used by SDS. Note that SDS will actually
* just use the macros defined into sdsalloc.h in order to avoid to pay
* the overhead of function calls. Here we define these wrappers only for
* the programs SDS is linked to, if they want to touch the SDS internals
* even if they use a different allocator. */
void *sds_malloc(size_t size) { return s_malloc(size); }
void *sds_realloc(void *ptr, size_t size) { return s_realloc(ptr,size); }
void sds_free(void *ptr) { s_free(ptr); }
#if defined(SDS_TEST_MAIN)
#ifdef SDS_TEST_MAIN
#include <stdio.h>
#include "testhelp.h"
#include "limits.h"
#define UNUSED(x) (void)(x)
int sdsTest(void) {
int main(void) {
{
struct sdshdr *sh;
sds x = sdsnew("foo"), y;
test_cond("Create a string and obtain the length",
@@ -1138,7 +1005,6 @@ int sdsTest(void) {
sdslen(x) == 60 &&
memcmp(x,"--Hello Hi! World -9223372036854775808,"
"9223372036854775807--",60) == 0)
printf("[%s]\n",x);
sdsfree(x);
x = sdsnew("--");
@@ -1147,18 +1013,6 @@ int sdsTest(void) {
sdslen(x) == 35 &&
memcmp(x,"--4294967295,18446744073709551615--",35) == 0)
sdsfree(x);
x = sdsnew(" x ");
sdstrim(x," x");
test_cond("sdstrim() works when all chars match",
sdslen(x) == 0)
sdsfree(x);
x = sdsnew(" x ");
sdstrim(x," ");
test_cond("sdstrim() works when a single char remains",
sdslen(x) == 1 && x[0] == 'x')
sdsfree(x);
x = sdsnew("xxciaoyyy");
sdstrim(x,"xy");
@@ -1226,47 +1080,24 @@ int sdsTest(void) {
memcmp(y,"\"\\a\\n\\x00foo\\r\"",15) == 0)
{
unsigned int oldfree;
char *p;
int step = 10, j, i;
int oldfree;
sdsfree(x);
sdsfree(y);
x = sdsnew("0");
test_cond("sdsnew() free/len buffers", sdslen(x) == 1 && sdsavail(x) == 0);
/* Run the test a few times in order to hit the first two
* SDS header types. */
for (i = 0; i < 10; i++) {
int oldlen = sdslen(x);
x = sdsMakeRoomFor(x,step);
int type = x[-1]&SDS_TYPE_MASK;
test_cond("sdsMakeRoomFor() len", sdslen(x) == oldlen);
if (type != SDS_TYPE_5) {
test_cond("sdsMakeRoomFor() free", sdsavail(x) >= step);
oldfree = sdsavail(x);
}
p = x+oldlen;
for (j = 0; j < step; j++) {
p[j] = 'A'+j;
}
sdsIncrLen(x,step);
}
test_cond("sdsMakeRoomFor() content",
memcmp("0ABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJ",x,101) == 0);
test_cond("sdsMakeRoomFor() final length",sdslen(x)==101);
sdsfree(x);
sh = (void*) (x-(sizeof(struct sdshdr)));
test_cond("sdsnew() free/len buffers", sh->len == 1 && sh->free == 0);
x = sdsMakeRoomFor(x,1);
sh = (void*) (x-(sizeof(struct sdshdr)));
test_cond("sdsMakeRoomFor()", sh->len == 1 && sh->free > 0);
oldfree = sh->free;
x[1] = '1';
sdsIncrLen(x,1);
test_cond("sdsIncrLen() -- content", x[0] == '0' && x[1] == '1');
test_cond("sdsIncrLen() -- len", sh->len == 2);
test_cond("sdsIncrLen() -- free", sh->free == oldfree-1);
}
}
test_report()
return 0;
}
#endif
#ifdef SDS_TEST_MAIN
int main(void) {
return sdsTest();
}
#endif
+11 -183
View File
@@ -1,8 +1,6 @@
/* SDSLib 2.0 -- A C dynamic strings library
/* SDSLib, A C dynamic strings library
*
* Copyright (c) 2006-2015, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2015, Oran Agra
* Copyright (c) 2015, Redis Labs, Inc
* Copyright (c) 2006-2010, Salvatore Sanfilippo <antirez at gmail dot com>
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -37,188 +35,32 @@
#include <sys/types.h>
#include <stdarg.h>
#include <stdint.h>
typedef char *sds;
/* Note: sdshdr5 is never used, we just access the flags byte directly.
* However is here to document the layout of type 5 SDS strings. */
struct __attribute__ ((__packed__)) sdshdr5 {
unsigned char flags; /* 3 lsb of type, and 5 msb of string length */
struct sdshdr {
unsigned int len;
unsigned int free;
char buf[];
};
struct __attribute__ ((__packed__)) sdshdr8 {
uint8_t len; /* used */
uint8_t alloc; /* excluding the header and null terminator */
unsigned char flags; /* 3 lsb of type, 5 unused bits */
char buf[];
};
struct __attribute__ ((__packed__)) sdshdr16 {
uint16_t len; /* used */
uint16_t alloc; /* excluding the header and null terminator */
unsigned char flags; /* 3 lsb of type, 5 unused bits */
char buf[];
};
struct __attribute__ ((__packed__)) sdshdr32 {
uint32_t len; /* used */
uint32_t alloc; /* excluding the header and null terminator */
unsigned char flags; /* 3 lsb of type, 5 unused bits */
char buf[];
};
struct __attribute__ ((__packed__)) sdshdr64 {
uint64_t len; /* used */
uint64_t alloc; /* excluding the header and null terminator */
unsigned char flags; /* 3 lsb of type, 5 unused bits */
char buf[];
};
#define SDS_TYPE_5 0
#define SDS_TYPE_8 1
#define SDS_TYPE_16 2
#define SDS_TYPE_32 3
#define SDS_TYPE_64 4
#define SDS_TYPE_MASK 7
#define SDS_TYPE_BITS 3
#define SDS_HDR_VAR(T,s) struct sdshdr##T *sh = (struct sdshdr##T *)((s)-(sizeof(struct sdshdr##T)));
#define SDS_HDR(T,s) ((struct sdshdr##T *)((s)-(sizeof(struct sdshdr##T))))
#define SDS_TYPE_5_LEN(f) ((f)>>SDS_TYPE_BITS)
static inline size_t sdslen(const sds s) {
unsigned char flags = s[-1];
switch(flags&SDS_TYPE_MASK) {
case SDS_TYPE_5:
return SDS_TYPE_5_LEN(flags);
case SDS_TYPE_8:
return SDS_HDR(8,s)->len;
case SDS_TYPE_16:
return SDS_HDR(16,s)->len;
case SDS_TYPE_32:
return SDS_HDR(32,s)->len;
case SDS_TYPE_64:
return SDS_HDR(64,s)->len;
}
return 0;
struct sdshdr *sh = (void*)(s-(sizeof(struct sdshdr)));
return sh->len;
}
static inline size_t sdsavail(const sds s) {
unsigned char flags = s[-1];
switch(flags&SDS_TYPE_MASK) {
case SDS_TYPE_5: {
return 0;
}
case SDS_TYPE_8: {
SDS_HDR_VAR(8,s);
return sh->alloc - sh->len;
}
case SDS_TYPE_16: {
SDS_HDR_VAR(16,s);
return sh->alloc - sh->len;
}
case SDS_TYPE_32: {
SDS_HDR_VAR(32,s);
return sh->alloc - sh->len;
}
case SDS_TYPE_64: {
SDS_HDR_VAR(64,s);
return sh->alloc - sh->len;
}
}
return 0;
}
static inline void sdssetlen(sds s, size_t newlen) {
unsigned char flags = s[-1];
switch(flags&SDS_TYPE_MASK) {
case SDS_TYPE_5:
{
unsigned char *fp = ((unsigned char*)s)-1;
*fp = SDS_TYPE_5 | (newlen << SDS_TYPE_BITS);
}
break;
case SDS_TYPE_8:
SDS_HDR(8,s)->len = newlen;
break;
case SDS_TYPE_16:
SDS_HDR(16,s)->len = newlen;
break;
case SDS_TYPE_32:
SDS_HDR(32,s)->len = newlen;
break;
case SDS_TYPE_64:
SDS_HDR(64,s)->len = newlen;
break;
}
}
static inline void sdsinclen(sds s, size_t inc) {
unsigned char flags = s[-1];
switch(flags&SDS_TYPE_MASK) {
case SDS_TYPE_5:
{
unsigned char *fp = ((unsigned char*)s)-1;
unsigned char newlen = SDS_TYPE_5_LEN(flags)+inc;
*fp = SDS_TYPE_5 | (newlen << SDS_TYPE_BITS);
}
break;
case SDS_TYPE_8:
SDS_HDR(8,s)->len += inc;
break;
case SDS_TYPE_16:
SDS_HDR(16,s)->len += inc;
break;
case SDS_TYPE_32:
SDS_HDR(32,s)->len += inc;
break;
case SDS_TYPE_64:
SDS_HDR(64,s)->len += inc;
break;
}
}
/* sdsalloc() = sdsavail() + sdslen() */
static inline size_t sdsalloc(const sds s) {
unsigned char flags = s[-1];
switch(flags&SDS_TYPE_MASK) {
case SDS_TYPE_5:
return SDS_TYPE_5_LEN(flags);
case SDS_TYPE_8:
return SDS_HDR(8,s)->alloc;
case SDS_TYPE_16:
return SDS_HDR(16,s)->alloc;
case SDS_TYPE_32:
return SDS_HDR(32,s)->alloc;
case SDS_TYPE_64:
return SDS_HDR(64,s)->alloc;
}
return 0;
}
static inline void sdssetalloc(sds s, size_t newlen) {
unsigned char flags = s[-1];
switch(flags&SDS_TYPE_MASK) {
case SDS_TYPE_5:
/* Nothing to do, this type has no total allocation info. */
break;
case SDS_TYPE_8:
SDS_HDR(8,s)->alloc = newlen;
break;
case SDS_TYPE_16:
SDS_HDR(16,s)->alloc = newlen;
break;
case SDS_TYPE_32:
SDS_HDR(32,s)->alloc = newlen;
break;
case SDS_TYPE_64:
SDS_HDR(64,s)->alloc = newlen;
break;
}
struct sdshdr *sh = (void*)(s-(sizeof(struct sdshdr)));
return sh->free;
}
sds sdsnewlen(const void *init, size_t initlen);
sds sdsnew(const char *init);
sds sdsempty(void);
size_t sdslen(const sds s);
sds sdsdup(const sds s);
void sdsfree(sds s);
size_t sdsavail(const sds s);
sds sdsgrowzero(sds s, size_t len);
sds sdscatlen(sds s, const void *t, size_t len);
sds sdscat(sds s, const char *t);
@@ -249,25 +91,11 @@ sds sdscatrepr(sds s, const char *p, size_t len);
sds *sdssplitargs(const char *line, int *argc);
sds sdsmapchars(sds s, const char *from, const char *to, size_t setlen);
sds sdsjoin(char **argv, int argc, char *sep);
sds sdsjoinsds(sds *argv, int argc, const char *sep, size_t seplen);
/* Low level functions exposed to the user API */
sds sdsMakeRoomFor(sds s, size_t addlen);
void sdsIncrLen(sds s, int incr);
sds sdsRemoveFreeSpace(sds s);
size_t sdsAllocSize(sds s);
void *sdsAllocPtr(sds s);
/* Export the allocator used by SDS to the program using SDS.
* Sometimes the program SDS is linked to, may use a different set of
* allocators, but may want to allocate or free things that SDS will
* respectively free or allocate. */
void *sds_malloc(size_t size);
void *sds_realloc(void *ptr, size_t size);
void sds_free(void *ptr);
#ifdef REDIS_TEST
int sdsTest(int argc, char *argv[]);
#endif
#endif
-42
View File
@@ -1,42 +0,0 @@
/* SDSLib 2.0 -- A C dynamic strings library
*
* Copyright (c) 2006-2015, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2015, Oran Agra
* Copyright (c) 2015, Redis Labs, Inc
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Redis nor the names of its contributors may be used
* to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
/* SDS allocator selection.
*
* This file is used in order to change the SDS allocator at compile time.
* Just define the following defines to what you want to use. Also add
* the include of your alternate allocator if needed (not needed in order
* to use the default libc allocator). */
#define s_malloc malloc
#define s_realloc realloc
#define s_free free
+15 -105
View File
@@ -11,7 +11,6 @@
#include <limits.h>
#include "hiredis.h"
#include "net.h"
enum connection_type {
CONN_TCP,
@@ -30,7 +29,7 @@ struct config {
struct {
const char *path;
} unix_sock;
} unix;
};
/* The following lines make up our testing "framework" :) */
@@ -44,13 +43,6 @@ static long long usec(void) {
return (((long long)tv.tv_sec)*1000000)+tv.tv_usec;
}
/* The assert() calls below have side effects, so we need assert()
* even if we are compiling without asserts (-DNDEBUG). */
#ifdef NDEBUG
#undef assert
#define assert(e) (void)(e)
#endif
static redisContext *select_database(redisContext *c) {
redisReply *reply;
@@ -59,7 +51,7 @@ static redisContext *select_database(redisContext *c) {
assert(reply != NULL);
freeReplyObject(reply);
/* Make sure the DB is emtpy */
/* Make sure the DB is empty */
reply = redisCommand(c,"DBSIZE");
assert(reply != NULL);
if (reply->type == REDIS_REPLY_INTEGER && reply->integer == 0) {
@@ -97,10 +89,10 @@ static redisContext *connect(struct config config) {
if (config.type == CONN_TCP) {
c = redisConnect(config.tcp.host, config.tcp.port);
} else if (config.type == CONN_UNIX) {
c = redisConnectUnix(config.unix_sock.path);
c = redisConnectUnix(config.unix.path);
} else if (config.type == CONN_FD) {
/* Create a dummy connection just to get an fd to inherit */
redisContext *dummy_ctx = redisConnectUnix(config.unix_sock.path);
redisContext *dummy_ctx = redisConnectUnix(config.unix.path);
if (dummy_ctx) {
int fd = disconnect(dummy_ctx, 1);
printf("Connecting to inherited fd %d\n", fd);
@@ -115,7 +107,6 @@ static redisContext *connect(struct config config) {
exit(1);
} else if (c->err) {
printf("Connection error: %s\n", c->errstr);
redisFree(c);
exit(1);
}
@@ -224,22 +215,6 @@ static void test_format_commands(void) {
test_cond(strncmp(cmd,"*3\r\n$3\r\nSET\r\n$7\r\nfoo\0xxx\r\n$3\r\nbar\r\n",len) == 0 &&
len == 4+4+(3+2)+4+(7+2)+4+(3+2));
free(cmd);
sds sds_cmd;
sds_cmd = sdsempty();
test("Format command into sds by passing argc/argv without lengths: ");
len = redisFormatSdsCommandArgv(&sds_cmd,argc,argv,NULL);
test_cond(strncmp(sds_cmd,"*3\r\n$3\r\nSET\r\n$3\r\nfoo\r\n$3\r\nbar\r\n",len) == 0 &&
len == 4+4+(3+2)+4+(3+2)+4+(3+2));
sdsfree(sds_cmd);
sds_cmd = sdsempty();
test("Format command into sds by passing argc/argv with lengths: ");
len = redisFormatSdsCommandArgv(&sds_cmd,argc,argv,lens);
test_cond(strncmp(sds_cmd,"*3\r\n$3\r\nSET\r\n$7\r\nfoo\0xxx\r\n$3\r\nbar\r\n",len) == 0 &&
len == 4+4+(3+2)+4+(7+2)+4+(3+2));
sdsfree(sds_cmd);
}
static void test_append_formatted_commands(struct config config) {
@@ -343,31 +318,16 @@ static void test_reply_reader(void) {
redisReaderFree(reader);
}
static void test_free_null(void) {
void *redisCtx = NULL;
void *reply = NULL;
test("Don't fail when redisFree is passed a NULL value: ");
redisFree(redisCtx);
test_cond(redisCtx == NULL);
test("Don't fail when freeReplyObject is passed a NULL value: ");
freeReplyObject(reply);
test_cond(reply == NULL);
}
static void test_blocking_connection_errors(void) {
redisContext *c;
test("Returns error when host cannot be resolved: ");
c = redisConnect((char*)"idontexist.test", 6379);
c = redisConnect((char*)"idontexist.local", 6379);
test_cond(c->err == REDIS_ERR_OTHER &&
(strcmp(c->errstr,"Name or service not known") == 0 ||
strcmp(c->errstr,"Can't resolve: idontexist.test") == 0 ||
strcmp(c->errstr,"Can't resolve: idontexist.local") == 0 ||
strcmp(c->errstr,"nodename nor servname provided, or not known") == 0 ||
strcmp(c->errstr,"No address associated with hostname") == 0 ||
strcmp(c->errstr,"Temporary failure in name resolution") == 0 ||
strcmp(c->errstr,"hostname nor servname provided, or not known") == 0 ||
strcmp(c->errstr,"no address associated with name") == 0));
redisFree(c);
@@ -377,7 +337,7 @@ static void test_blocking_connection_errors(void) {
strcmp(c->errstr,"Connection refused") == 0);
redisFree(c);
test("Returns error when the unix_sock socket path doesn't accept connections: ");
test("Returns error when the unix socket path doesn't accept connections: ");
c = redisConnectUnix((char*)"/tmp/idontexist.sock");
test_cond(c->err == REDIS_ERR_IO); /* Don't care about the message... */
redisFree(c);
@@ -461,52 +421,6 @@ static void test_blocking_connection(struct config config) {
disconnect(c, 0);
}
static void test_blocking_connection_timeouts(struct config config) {
redisContext *c;
redisReply *reply;
ssize_t s;
const char *cmd = "DEBUG SLEEP 3\r\n";
struct timeval tv;
c = connect(config);
test("Successfully completes a command when the timeout is not exceeded: ");
reply = redisCommand(c,"SET foo fast");
freeReplyObject(reply);
tv.tv_sec = 0;
tv.tv_usec = 10000;
redisSetTimeout(c, tv);
reply = redisCommand(c, "GET foo");
test_cond(reply != NULL && reply->type == REDIS_REPLY_STRING && memcmp(reply->str, "fast", 4) == 0);
freeReplyObject(reply);
disconnect(c, 0);
c = connect(config);
test("Does not return a reply when the command times out: ");
s = write(c->fd, cmd, strlen(cmd));
tv.tv_sec = 0;
tv.tv_usec = 10000;
redisSetTimeout(c, tv);
reply = redisCommand(c, "GET foo");
test_cond(s > 0 && reply == NULL && c->err == REDIS_ERR_IO && strcmp(c->errstr, "Resource temporarily unavailable") == 0);
freeReplyObject(reply);
test("Reconnect properly reconnects after a timeout: ");
redisReconnect(c);
reply = redisCommand(c, "PING");
test_cond(reply != NULL && reply->type == REDIS_REPLY_STATUS && strcmp(reply->str, "PONG") == 0);
freeReplyObject(reply);
test("Reconnect properly uses owned parameters: ");
config.tcp.host = "foo";
config.unix_sock.path = "foo";
redisReconnect(c);
reply = redisCommand(c, "PING");
test_cond(reply != NULL && reply->type == REDIS_REPLY_STATUS && strcmp(reply->str, "PONG") == 0);
freeReplyObject(reply);
disconnect(c, 0);
}
static void test_blocking_io_errors(struct config config) {
redisContext *c;
redisReply *reply;
@@ -530,7 +444,7 @@ static void test_blocking_io_errors(struct config config) {
test("Returns I/O error when the connection is lost: ");
reply = redisCommand(c,"QUIT");
if (major > 2 || (major == 2 && minor > 0)) {
if (major >= 2 && minor > 0) {
/* > 2.0 returns OK on QUIT and read() should be issued once more
* to know the descriptor is at EOF. */
test_cond(strcasecmp(reply->str,"OK") == 0 &&
@@ -568,8 +482,7 @@ static void test_invalid_timeout_errors(struct config config) {
c = redisConnectWithTimeout(config.tcp.host, config.tcp.port, config.tcp.timeout);
test_cond(c->err == REDIS_ERR_IO && strcmp(c->errstr, "Invalid timeout specified") == 0);
redisFree(c);
test_cond(c->err == REDIS_ERR_IO);
test("Set error when an invalid timeout sec value is given to redisConnectWithTimeout: ");
@@ -578,7 +491,8 @@ static void test_invalid_timeout_errors(struct config config) {
c = redisConnectWithTimeout(config.tcp.host, config.tcp.port, config.tcp.timeout);
test_cond(c->err == REDIS_ERR_IO && strcmp(c->errstr, "Invalid timeout specified") == 0);
test_cond(c->err == REDIS_ERR_IO);
redisFree(c);
}
@@ -752,7 +666,7 @@ int main(int argc, char **argv) {
.host = "127.0.0.1",
.port = 6379
},
.unix_sock = {
.unix = {
.path = "/tmp/redis.sock"
}
};
@@ -773,7 +687,7 @@ int main(int argc, char **argv) {
cfg.tcp.port = atoi(argv[0]);
} else if (argc >= 2 && !strcmp(argv[0],"-s")) {
argv++; argc--;
cfg.unix_sock.path = argv[0];
cfg.unix.path = argv[0];
} else if (argc >= 1 && !strcmp(argv[0],"--skip-throughput")) {
throughput = 0;
} else if (argc >= 1 && !strcmp(argv[0],"--skip-inherit-fd")) {
@@ -788,31 +702,27 @@ int main(int argc, char **argv) {
test_format_commands();
test_reply_reader();
test_blocking_connection_errors();
test_free_null();
printf("\nTesting against TCP connection (%s:%d):\n", cfg.tcp.host, cfg.tcp.port);
cfg.type = CONN_TCP;
test_blocking_connection(cfg);
test_blocking_connection_timeouts(cfg);
test_blocking_io_errors(cfg);
test_invalid_timeout_errors(cfg);
test_append_formatted_commands(cfg);
if (throughput) test_throughput(cfg);
printf("\nTesting against Unix socket connection (%s):\n", cfg.unix_sock.path);
printf("\nTesting against Unix socket connection (%s):\n", cfg.unix.path);
cfg.type = CONN_UNIX;
test_blocking_connection(cfg);
test_blocking_connection_timeouts(cfg);
test_blocking_io_errors(cfg);
if (throughput) test_throughput(cfg);
if (test_inherit_fd) {
printf("\nTesting against inherited fd (%s):\n", cfg.unix_sock.path);
printf("\nTesting against inherited fd (%s):\n", cfg.unix.path);
cfg.type = CONN_FD;
test_blocking_connection(cfg);
}
if (fails) {
printf("*** %d TESTS FAILED ***\n", fails);
return 1;
-42
View File
@@ -1,42 +0,0 @@
#ifndef _WIN32_HELPER_INCLUDE
#define _WIN32_HELPER_INCLUDE
#ifdef _MSC_VER
#ifndef inline
#define inline __inline
#endif
#ifndef va_copy
#define va_copy(d,s) ((d) = (s))
#endif
#ifndef snprintf
#define snprintf c99_snprintf
__inline int c99_vsnprintf(char* str, size_t size, const char* format, va_list ap)
{
int count = -1;
if (size != 0)
count = _vsnprintf_s(str, size, _TRUNCATE, format, ap);
if (count == -1)
count = _vscprintf(format, ap);
return count;
}
__inline int c99_snprintf(char* str, size_t size, const char* format, ...)
{
int count;
va_list ap;
va_start(ap, format);
count = c99_vsnprintf(str, size, format, ap);
va_end(ap);
return count;
}
#endif
#endif
#endif
+13
View File
@@ -0,0 +1,13 @@
/* Drop in replacement for zmalloc.h in order to just use libc malloc without
* any wrappering. */
#ifndef ZMALLOC_H
#define ZMALLOC_H
#define zmalloc malloc
#define zrealloc realloc
#define zcalloc(x) calloc(x,1)
#define zfree free
#define zstrdup strdup
#endif
-42
View File
@@ -1,42 +0,0 @@
version: '{build}'
environment:
matrix:
- MSYSTEM: MINGW64
CPU: x86_64
MSVC: amd64
- MSYSTEM: MINGW32
CPU: i686
MSVC: x86
- MSYSTEM: MINGW64
CPU: x86_64
- MSYSTEM: MINGW32
CPU: i686
- MSYSTEM: MINGW64
CPU: x86_64
MSVC: amd64
CONFIG_FLAGS: --enable-debug
- MSYSTEM: MINGW32
CPU: i686
MSVC: x86
CONFIG_FLAGS: --enable-debug
- MSYSTEM: MINGW64
CPU: x86_64
CONFIG_FLAGS: --enable-debug
- MSYSTEM: MINGW32
CPU: i686
CONFIG_FLAGS: --enable-debug
install:
- set PATH=c:\msys64\%MSYSTEM%\bin;c:\msys64\usr\bin;%PATH%
- if defined MSVC call "c:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\vcvarsall.bat" %MSVC%
- if defined MSVC pacman --noconfirm -Rsc mingw-w64-%CPU%-gcc gcc
- pacman --noconfirm -Suy mingw-w64-%CPU%-make
build_script:
- bash -c "autoconf"
- bash -c "./configure $CONFIG_FLAGS"
- mingw32-make
- file lib/jemalloc.dll
- mingw32-make tests
- mingw32-make -k check
-3
View File
@@ -1,3 +0,0 @@
begin-language: "Autoconf-without-aclocal-m4"
args: --no-cache
end-language: "Autoconf-without-aclocal-m4"
-1
View File
@@ -1 +0,0 @@
* text=auto eol=lf
+16 -37
View File
@@ -1,6 +1,8 @@
/bin/jemalloc-config
/*.gcov.*
/autom4te.cache/
/bin/jemalloc.sh
/bin/jeprof
/config.stamp
/config.log
@@ -13,20 +15,14 @@
/doc/jemalloc.html
/doc/jemalloc.3
/jemalloc.pc
/lib/
/Makefile
/include/jemalloc/internal/jemalloc_preamble.h
/include/jemalloc/internal/jemalloc_internal.h
/include/jemalloc/internal/jemalloc_internal_defs.h
/include/jemalloc/internal/private_namespace.gen.h
/include/jemalloc/internal/private_namespace.h
/include/jemalloc/internal/private_namespace_jet.gen.h
/include/jemalloc/internal/private_namespace_jet.h
/include/jemalloc/internal/private_symbols.awk
/include/jemalloc/internal/private_symbols_jet.awk
/include/jemalloc/internal/private_unnamespace.h
/include/jemalloc/internal/public_namespace.h
/include/jemalloc/internal/public_symbols.txt
/include/jemalloc/internal/public_unnamespace.h
@@ -39,12 +35,10 @@
/include/jemalloc/jemalloc_protos.h
/include/jemalloc/jemalloc_protos_jet.h
/include/jemalloc/jemalloc_rename.h
/include/jemalloc/jemalloc_typedefs.h
/src/*.[od]
/src/*.sym
/run_tests.out/
/src/*.gcda
/src/*.gcno
/test/test.sh
test/include/test/jemalloc_test.h
@@ -53,41 +47,26 @@ test/include/test/jemalloc_test_defs.h
/test/integration/[A-Za-z]*
!/test/integration/[A-Za-z]*.*
/test/integration/*.[od]
/test/integration/*.gcda
/test/integration/*.gcno
/test/integration/*.out
/test/integration/cpp/[A-Za-z]*
!/test/integration/cpp/[A-Za-z]*.*
/test/integration/cpp/*.[od]
/test/integration/cpp/*.out
/test/src/*.[od]
/test/src/*.gcda
/test/src/*.gcno
/test/stress/[A-Za-z]*
!/test/stress/[A-Za-z]*.*
/test/stress/*.[od]
/test/stress/*.gcda
/test/stress/*.gcno
/test/stress/*.out
/test/unit/[A-Za-z]*
!/test/unit/[A-Za-z]*.*
/test/unit/*.[od]
/test/unit/*.gcda
/test/unit/*.gcno
/test/unit/*.out
/VERSION
*.pdb
*.sdf
*.opendb
*.VC.db
*.opensdf
*.cachefile
*.suo
*.user
*.sln.docstates
*.tmp
.vs/
/msvc/Win32/
/msvc/x64/
/msvc/projects/*/*/Debug*/
/msvc/projects/*/*/Release*/
/msvc/projects/*/*/Win32/
/msvc/projects/*/*/x64/
-156
View File
@@ -1,156 +0,0 @@
language: generic
dist: precise
matrix:
include:
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: osx
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=clang CXX=clang++ COMPILER_FLAGS="" CONFIGURE_FLAGS="" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="-m32" CONFIGURE_FLAGS="" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
addons:
apt:
packages:
- gcc-multilib
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--enable-debug" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--enable-prof" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--disable-stats" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--with-malloc-conf=tcache:false" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--with-malloc-conf=dss:primary" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--with-malloc-conf=percpu_arena:percpu" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--with-malloc-conf=background_thread:true" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: osx
env: CC=clang CXX=clang++ COMPILER_FLAGS="" CONFIGURE_FLAGS="" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: osx
env: CC=gcc CXX=g++ COMPILER_FLAGS="-m32" CONFIGURE_FLAGS="" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: osx
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--enable-debug" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: osx
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--disable-stats" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: osx
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--with-malloc-conf=tcache:false" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=clang CXX=clang++ COMPILER_FLAGS="-m32" CONFIGURE_FLAGS="" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
addons:
apt:
packages:
- gcc-multilib
- os: linux
env: CC=clang CXX=clang++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--enable-debug" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=clang CXX=clang++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--enable-prof" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=clang CXX=clang++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--disable-stats" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=clang CXX=clang++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--with-malloc-conf=tcache:false" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=clang CXX=clang++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--with-malloc-conf=dss:primary" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=clang CXX=clang++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--with-malloc-conf=percpu_arena:percpu" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=clang CXX=clang++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--with-malloc-conf=background_thread:true" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="-m32" CONFIGURE_FLAGS="--enable-debug" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
addons:
apt:
packages:
- gcc-multilib
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="-m32" CONFIGURE_FLAGS="--enable-prof" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
addons:
apt:
packages:
- gcc-multilib
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="-m32" CONFIGURE_FLAGS="--disable-stats" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
addons:
apt:
packages:
- gcc-multilib
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="-m32" CONFIGURE_FLAGS="--with-malloc-conf=tcache:false" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
addons:
apt:
packages:
- gcc-multilib
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="-m32" CONFIGURE_FLAGS="--with-malloc-conf=dss:primary" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
addons:
apt:
packages:
- gcc-multilib
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="-m32" CONFIGURE_FLAGS="--with-malloc-conf=percpu_arena:percpu" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
addons:
apt:
packages:
- gcc-multilib
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="-m32" CONFIGURE_FLAGS="--with-malloc-conf=background_thread:true" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
addons:
apt:
packages:
- gcc-multilib
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--enable-debug --enable-prof" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--enable-debug --disable-stats" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--enable-debug --with-malloc-conf=tcache:false" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--enable-debug --with-malloc-conf=dss:primary" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--enable-debug --with-malloc-conf=percpu_arena:percpu" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--enable-debug --with-malloc-conf=background_thread:true" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--enable-prof --disable-stats" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--enable-prof --with-malloc-conf=tcache:false" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--enable-prof --with-malloc-conf=dss:primary" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--enable-prof --with-malloc-conf=percpu_arena:percpu" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--enable-prof --with-malloc-conf=background_thread:true" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--disable-stats --with-malloc-conf=tcache:false" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--disable-stats --with-malloc-conf=dss:primary" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--disable-stats --with-malloc-conf=percpu_arena:percpu" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--disable-stats --with-malloc-conf=background_thread:true" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--with-malloc-conf=tcache:false,dss:primary" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--with-malloc-conf=tcache:false,percpu_arena:percpu" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--with-malloc-conf=tcache:false,background_thread:true" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--with-malloc-conf=dss:primary,percpu_arena:percpu" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--with-malloc-conf=dss:primary,background_thread:true" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--with-malloc-conf=percpu_arena:percpu,background_thread:true" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
before_script:
- autoconf
- ./configure ${COMPILER_FLAGS:+ CC="$CC $COMPILER_FLAGS" CXX="$CXX $COMPILER_FLAGS" } $CONFIGURE_FLAGS
- make -j3
- make -j3 tests
script:
- make check
+2 -2
View File
@@ -1,10 +1,10 @@
Unless otherwise specified, files in the jemalloc source distribution are
subject to the following license:
--------------------------------------------------------------------------------
Copyright (C) 2002-2018 Jason Evans <jasone@canonware.com>.
Copyright (C) 2002-2014 Jason Evans <jasone@canonware.com>.
All rights reserved.
Copyright (C) 2007-2012 Mozilla Foundation. All rights reserved.
Copyright (C) 2009-2018 Facebook, Inc. All rights reserved.
Copyright (C) 2009-2014 Facebook, Inc. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
+5 -839
View File
@@ -1,844 +1,10 @@
Following are change highlights associated with official releases. Important
bug fixes are all mentioned, but some internal enhancements are omitted here for
brevity. Much more detail can be found in the git revision history:
bug fixes are all mentioned, but internal enhancements are omitted here for
brevity (even though they are more fun to write about). Much more detail can be
found in the git revision history:
https://github.com/jemalloc/jemalloc
* 5.1.0 (May 4th, 2018)
This release is primarily about fine-tuning, ranging from several new features
to numerous notable performance and portability enhancements. The release and
prior dev versions have been running in multiple large scale applications for
months, and the cumulative improvements are substantial in many cases.
Given the long and successful production runs, this release is likely a good
candidate for applications to upgrade, from both jemalloc 5.0 and before. For
performance-critical applications, the newly added TUNING.md provides
guidelines on jemalloc tuning.
New features:
- Implement transparent huge page support for internal metadata. (@interwq)
- Add opt.thp to allow enabling / disabling transparent huge pages for all
mappings. (@interwq)
- Add maximum background thread count option. (@djwatson)
- Allow prof_active to control opt.lg_prof_interval and prof.gdump.
(@interwq)
- Allow arena index lookup based on allocation addresses via mallctl.
(@lionkov)
- Allow disabling initial-exec TLS model. (@davidtgoldblatt, @KenMacD)
- Add opt.lg_extent_max_active_fit to set the max ratio between the size of
the active extent selected (to split off from) and the size of the requested
allocation. (@interwq, @davidtgoldblatt)
- Add retain_grow_limit to set the max size when growing virtual address
space. (@interwq)
- Add mallctl interfaces:
+ arena.<i>.retain_grow_limit (@interwq)
+ arenas.lookup (@lionkov)
+ max_background_threads (@djwatson)
+ opt.lg_extent_max_active_fit (@interwq)
+ opt.max_background_threads (@djwatson)
+ opt.metadata_thp (@interwq)
+ opt.thp (@interwq)
+ stats.metadata_thp (@interwq)
Portability improvements:
- Support GNU/kFreeBSD configuration. (@paravoid)
- Support m68k, nios2 and SH3 architectures. (@paravoid)
- Fall back to FD_CLOEXEC when O_CLOEXEC is unavailable. (@zonyitoo)
- Fix symbol listing for cross-compiling. (@tamird)
- Fix high bits computation on ARM. (@davidtgoldblatt, @paravoid)
- Disable the CPU_SPINWAIT macro for Power. (@davidtgoldblatt, @marxin)
- Fix MSVC 2015 & 2017 builds. (@rustyx)
- Improve RISC-V support. (@EdSchouten)
- Set name mangling script in strict mode. (@nicolov)
- Avoid MADV_HUGEPAGE on ARM. (@marxin)
- Modify configure to determine return value of strerror_r.
(@davidtgoldblatt, @cferris1000)
- Make sure CXXFLAGS is tested with CPP compiler. (@nehaljwani)
- Fix 32-bit build on MSVC. (@rustyx)
- Fix external symbol on MSVC. (@maksqwe)
- Avoid a printf format specifier warning. (@jasone)
- Add configure option --disable-initial-exec-tls which can allow jemalloc to
be dynamically loaded after program startup. (@davidtgoldblatt, @KenMacD)
- AArch64: Add ILP32 support. (@cmuellner)
- Add --with-lg-vaddr configure option to support cross compiling.
(@cmuellner, @davidtgoldblatt)
Optimizations and refactors:
- Improve active extent fit with extent_max_active_fit. This considerably
reduces fragmentation over time and improves virtual memory and metadata
usage. (@davidtgoldblatt, @interwq)
- Eagerly coalesce large extents to reduce fragmentation. (@interwq)
- sdallocx: only read size info when page aligned (i.e. possibly sampled),
which speeds up the sized deallocation path significantly. (@interwq)
- Avoid attempting new mappings for in place expansion with retain, since
it rarely succeeds in practice and causes high overhead. (@interwq)
- Refactor OOM handling in newImpl. (@wqfish)
- Add internal fine-grained logging functionality for debugging use.
(@davidtgoldblatt)
- Refactor arena / tcache interactions. (@davidtgoldblatt)
- Refactor extent management with dumpable flag. (@davidtgoldblatt)
- Add runtime detection of lazy purging. (@interwq)
- Use pairing heap instead of red-black tree for extents_avail. (@djwatson)
- Use sysctl on startup in FreeBSD. (@trasz)
- Use thread local prng state instead of atomic. (@djwatson)
- Make decay to always purge one more extent than before, because in
practice large extents are usually the ones that cross the decay threshold.
Purging the additional extent helps save memory as well as reduce VM
fragmentation. (@interwq)
- Fast division by dynamic values. (@davidtgoldblatt)
- Improve the fit for aligned allocation. (@interwq, @edwinsmith)
- Refactor extent_t bitpacking. (@rkmisra)
- Optimize the generated assembly for ticker operations. (@davidtgoldblatt)
- Convert stats printing to use a structured text emitter. (@davidtgoldblatt)
- Remove preserve_lru feature for extents management. (@djwatson)
- Consolidate two memory loads into one on the fast deallocation path.
(@davidtgoldblatt, @interwq)
Bug fixes (most of the issues are only relevant to jemalloc 5.0):
- Fix deadlock with multithreaded fork in OS X. (@davidtgoldblatt)
- Validate returned file descriptor before use. (@zonyitoo)
- Fix a few background thread initialization and shutdown issues. (@interwq)
- Fix an extent coalesce + decay race by taking both coalescing extents off
the LRU list. (@interwq)
- Fix potentially unbound increase during decay, caused by one thread keep
stashing memory to purge while other threads generating new pages. The
number of pages to purge is checked to prevent this. (@interwq)
- Fix a FreeBSD bootstrap assertion. (@strejda, @interwq)
- Handle 32 bit mutex counters. (@rkmisra)
- Fix a indexing bug when creating background threads. (@davidtgoldblatt,
@binliu19)
- Fix arguments passed to extent_init. (@yuleniwo, @interwq)
- Fix addresses used for ordering mutexes. (@rkmisra)
- Fix abort_conf processing during bootstrap. (@interwq)
- Fix include path order for out-of-tree builds. (@cmuellner)
Incompatible changes:
- Remove --disable-thp. (@interwq)
- Remove mallctl interfaces:
+ config.thp (@interwq)
Documentation:
- Add TUNING.md. (@interwq, @davidtgoldblatt, @djwatson)
* 5.0.1 (July 1, 2017)
This bugfix release fixes several issues, most of which are obscure enough
that typical applications are not impacted.
Bug fixes:
- Update decay->nunpurged before purging, in order to avoid potential update
races and subsequent incorrect purging volume. (@interwq)
- Only abort on dlsym(3) error if the failure impacts an enabled feature (lazy
locking and/or background threads). This mitigates an initialization
failure bug for which we still do not have a clear reproduction test case.
(@interwq)
- Modify tsd management so that it neither crashes nor leaks if a thread's
only allocation activity is to call free() after TLS destructors have been
executed. This behavior was observed when operating with GNU libc, and is
unlikely to be an issue with other libc implementations. (@interwq)
- Mask signals during background thread creation. This prevents signals from
being inadvertently delivered to background threads. (@jasone,
@davidtgoldblatt, @interwq)
- Avoid inactivity checks within background threads, in order to prevent
recursive mutex acquisition. (@interwq)
- Fix extent_grow_retained() to use the specified hooks when the
arena.<i>.extent_hooks mallctl is used to override the default hooks.
(@interwq)
- Add missing reentrancy support for custom extent hooks which allocate.
(@interwq)
- Post-fork(2), re-initialize the list of tcaches associated with each arena
to contain no tcaches except the forking thread's. (@interwq)
- Add missing post-fork(2) mutex reinitialization for extent_grow_mtx. This
fixes potential deadlocks after fork(2). (@interwq)
- Enforce minimum autoconf version (currently 2.68), since 2.63 is known to
generate corrupt configure scripts. (@jasone)
- Ensure that the configured page size (--with-lg-page) is no larger than the
configured huge page size (--with-lg-hugepage). (@jasone)
* 5.0.0 (June 13, 2017)
Unlike all previous jemalloc releases, this release does not use naturally
aligned "chunks" for virtual memory management, and instead uses page-aligned
"extents". This change has few externally visible effects, but the internal
impacts are... extensive. Many other internal changes combine to make this
the most cohesively designed version of jemalloc so far, with ample
opportunity for further enhancements.
Continuous integration is now an integral aspect of development thanks to the
efforts of @davidtgoldblatt, and the dev branch tends to remain reasonably
stable on the tested platforms (Linux, FreeBSD, macOS, and Windows). As a
side effect the official release frequency may decrease over time.
New features:
- Implement optional per-CPU arena support; threads choose which arena to use
based on current CPU rather than on fixed thread-->arena associations.
(@interwq)
- Implement two-phase decay of unused dirty pages. Pages transition from
dirty-->muzzy-->clean, where the first phase transition relies on
madvise(... MADV_FREE) semantics, and the second phase transition discards
pages such that they are replaced with demand-zeroed pages on next access.
(@jasone)
- Increase decay time resolution from seconds to milliseconds. (@jasone)
- Implement opt-in per CPU background threads, and use them for asynchronous
decay-driven unused dirty page purging. (@interwq)
- Add mutex profiling, which collects a variety of statistics useful for
diagnosing overhead/contention issues. (@interwq)
- Add C++ new/delete operator bindings. (@djwatson)
- Support manually created arena destruction, such that all data and metadata
are discarded. Add MALLCTL_ARENAS_DESTROYED for accessing merged stats
associated with destroyed arenas. (@jasone)
- Add MALLCTL_ARENAS_ALL as a fixed index for use in accessing
merged/destroyed arena statistics via mallctl. (@jasone)
- Add opt.abort_conf to optionally abort if invalid configuration options are
detected during initialization. (@interwq)
- Add opt.stats_print_opts, so that e.g. JSON output can be selected for the
stats dumped during exit if opt.stats_print is true. (@jasone)
- Add --with-version=VERSION for use when embedding jemalloc into another
project's git repository. (@jasone)
- Add --disable-thp to support cross compiling. (@jasone)
- Add --with-lg-hugepage to support cross compiling. (@jasone)
- Add mallctl interfaces (various authors):
+ background_thread
+ opt.abort_conf
+ opt.retain
+ opt.percpu_arena
+ opt.background_thread
+ opt.{dirty,muzzy}_decay_ms
+ opt.stats_print_opts
+ arena.<i>.initialized
+ arena.<i>.destroy
+ arena.<i>.{dirty,muzzy}_decay_ms
+ arena.<i>.extent_hooks
+ arenas.{dirty,muzzy}_decay_ms
+ arenas.bin.<i>.slab_size
+ arenas.nlextents
+ arenas.lextent.<i>.size
+ arenas.create
+ stats.background_thread.{num_threads,num_runs,run_interval}
+ stats.mutexes.{ctl,background_thread,prof,reset}.
{num_ops,num_spin_acq,num_wait,max_wait_time,total_wait_time,max_num_thds,
num_owner_switch}
+ stats.arenas.<i>.{dirty,muzzy}_decay_ms
+ stats.arenas.<i>.uptime
+ stats.arenas.<i>.{pmuzzy,base,internal,resident}
+ stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged}
+ stats.arenas.<i>.bins.<j>.{nslabs,reslabs,curslabs}
+ stats.arenas.<i>.bins.<j>.mutex.
{num_ops,num_spin_acq,num_wait,max_wait_time,total_wait_time,max_num_thds,
num_owner_switch}
+ stats.arenas.<i>.lextents.<j>.{nmalloc,ndalloc,nrequests,curlextents}
+ stats.arenas.i.mutexes.{large,extent_avail,extents_dirty,extents_muzzy,
extents_retained,decay_dirty,decay_muzzy,base,tcache_list}.
{num_ops,num_spin_acq,num_wait,max_wait_time,total_wait_time,max_num_thds,
num_owner_switch}
Portability improvements:
- Improve reentrant allocation support, such that deadlock is less likely if
e.g. a system library call in turn allocates memory. (@davidtgoldblatt,
@interwq)
- Support static linking of jemalloc with glibc. (@djwatson)
Optimizations and refactors:
- Organize virtual memory as "extents" of virtual memory pages, rather than as
naturally aligned "chunks", and store all metadata in arbitrarily distant
locations. This reduces virtual memory external fragmentation, and will
interact better with huge pages (not yet explicitly supported). (@jasone)
- Fold large and huge size classes together; only small and large size classes
remain. (@jasone)
- Unify the allocation paths, and merge most fast-path branching decisions.
(@davidtgoldblatt, @interwq)
- Embed per thread automatic tcache into thread-specific data, which reduces
conditional branches and dereferences. Also reorganize tcache to increase
fast-path data locality. (@interwq)
- Rewrite atomics to closely model the C11 API, convert various
synchronization from mutex-based to atomic, and use the explicit memory
ordering control to resolve various hypothetical races without increasing
synchronization overhead. (@davidtgoldblatt)
- Extensively optimize rtree via various methods:
+ Add multiple layers of rtree lookup caching, since rtree lookups are now
part of fast-path deallocation. (@interwq)
+ Determine rtree layout at compile time. (@jasone)
+ Make the tree shallower for common configurations. (@jasone)
+ Embed the root node in the top-level rtree data structure, thus avoiding
one level of indirection. (@jasone)
+ Further specialize leaf elements as compared to internal node elements,
and directly embed extent metadata needed for fast-path deallocation.
(@jasone)
+ Ignore leading always-zero address bits (architecture-specific).
(@jasone)
- Reorganize headers (ongoing work) to make them hermetic, and disentangle
various module dependencies. (@davidtgoldblatt)
- Convert various internal data structures such as size class metadata from
boot-time-initialized to compile-time-initialized. Propagate resulting data
structure simplifications, such as making arena metadata fixed-size.
(@jasone)
- Simplify size class lookups when constrained to size classes that are
multiples of the page size. This speeds lookups, but the primary benefit is
complexity reduction in code that was the source of numerous regressions.
(@jasone)
- Lock individual extents when possible for localized extent operations,
rather than relying on a top-level arena lock. (@davidtgoldblatt, @jasone)
- Use first fit layout policy instead of best fit, in order to improve
packing. (@jasone)
- If munmap(2) is not in use, use an exponential series to grow each arena's
virtual memory, so that the number of disjoint virtual memory mappings
remains low. (@jasone)
- Implement per arena base allocators, so that arenas never share any virtual
memory pages. (@jasone)
- Automatically generate private symbol name mangling macros. (@jasone)
Incompatible changes:
- Replace chunk hooks with an expanded/normalized set of extent hooks.
(@jasone)
- Remove ratio-based purging. (@jasone)
- Remove --disable-tcache. (@jasone)
- Remove --disable-tls. (@jasone)
- Remove --enable-ivsalloc. (@jasone)
- Remove --with-lg-size-class-group. (@jasone)
- Remove --with-lg-tiny-min. (@jasone)
- Remove --disable-cc-silence. (@jasone)
- Remove --enable-code-coverage. (@jasone)
- Remove --disable-munmap (replaced by opt.retain). (@jasone)
- Remove Valgrind support. (@jasone)
- Remove quarantine support. (@jasone)
- Remove redzone support. (@jasone)
- Remove mallctl interfaces (various authors):
+ config.munmap
+ config.tcache
+ config.tls
+ config.valgrind
+ opt.lg_chunk
+ opt.purge
+ opt.lg_dirty_mult
+ opt.decay_time
+ opt.quarantine
+ opt.redzone
+ opt.thp
+ arena.<i>.lg_dirty_mult
+ arena.<i>.decay_time
+ arena.<i>.chunk_hooks
+ arenas.initialized
+ arenas.lg_dirty_mult
+ arenas.decay_time
+ arenas.bin.<i>.run_size
+ arenas.nlruns
+ arenas.lrun.<i>.size
+ arenas.nhchunks
+ arenas.hchunk.<i>.size
+ arenas.extend
+ stats.cactive
+ stats.arenas.<i>.lg_dirty_mult
+ stats.arenas.<i>.decay_time
+ stats.arenas.<i>.metadata.{mapped,allocated}
+ stats.arenas.<i>.{npurge,nmadvise,purged}
+ stats.arenas.<i>.huge.{allocated,nmalloc,ndalloc,nrequests}
+ stats.arenas.<i>.bins.<j>.{nruns,reruns,curruns}
+ stats.arenas.<i>.lruns.<j>.{nmalloc,ndalloc,nrequests,curruns}
+ stats.arenas.<i>.hchunks.<j>.{nmalloc,ndalloc,nrequests,curhchunks}
Bug fixes:
- Improve interval-based profile dump triggering to dump only one profile when
a single allocation's size exceeds the interval. (@jasone)
- Use prefixed function names (as controlled by --with-jemalloc-prefix) when
pruning backtrace frames in jeprof. (@jasone)
* 4.5.0 (February 28, 2017)
This is the first release to benefit from much broader continuous integration
testing, thanks to @davidtgoldblatt. Had we had this testing infrastructure
in place for prior releases, it would have caught all of the most serious
regressions fixed by this release.
New features:
- Add --disable-thp and the opt.thp mallctl to provide opt-out mechanisms for
transparent huge page integration. (@jasone)
- Update zone allocator integration to work with macOS 10.12. (@glandium)
- Restructure *CFLAGS configuration, so that CFLAGS behaves typically, and
EXTRA_CFLAGS provides a way to specify e.g. -Werror during building, but not
during configuration. (@jasone, @ronawho)
Bug fixes:
- Fix DSS (sbrk(2)-based) allocation. This regression was first released in
4.3.0. (@jasone)
- Handle race in per size class utilization computation. This functionality
was first released in 4.0.0. (@interwq)
- Fix lock order reversal during gdump. (@jasone)
- Fix/refactor tcache synchronization. This regression was first released in
4.0.0. (@jasone)
- Fix various JSON-formatted malloc_stats_print() bugs. This functionality
was first released in 4.3.0. (@jasone)
- Fix huge-aligned allocation. This regression was first released in 4.4.0.
(@jasone)
- When transparent huge page integration is enabled, detect what state pages
start in according to the kernel's current operating mode, and only convert
arena chunks to non-huge during purging if that is not their initial state.
This functionality was first released in 4.4.0. (@jasone)
- Fix lg_chunk clamping for the --enable-cache-oblivious --disable-fill case.
This regression was first released in 4.0.0. (@jasone, @428desmo)
- Properly detect sparc64 when building for Linux. (@glaubitz)
* 4.4.0 (December 3, 2016)
New features:
- Add configure support for *-*-linux-android. (@cferris1000, @jasone)
- Add the --disable-syscall configure option, for use on systems that place
security-motivated limitations on syscall(2). (@jasone)
- Add support for Debian GNU/kFreeBSD. (@thesam)
Optimizations:
- Add extent serial numbers and use them where appropriate as a sort key that
is higher priority than address, so that the allocation policy prefers older
extents. This tends to improve locality (decrease fragmentation) when
memory grows downward. (@jasone)
- Refactor madvise(2) configuration so that MADV_FREE is detected and utilized
on Linux 4.5 and newer. (@jasone)
- Mark partially purged arena chunks as non-huge-page. This improves
interaction with Linux's transparent huge page functionality. (@jasone)
Bug fixes:
- Fix size class computations for edge conditions involving extremely large
allocations. This regression was first released in 4.0.0. (@jasone,
@ingvarha)
- Remove overly restrictive assertions related to the cactive statistic. This
regression was first released in 4.1.0. (@jasone)
- Implement a more reliable detection scheme for os_unfair_lock on macOS.
(@jszakmeister)
* 4.3.1 (November 7, 2016)
Bug fixes:
- Fix a severe virtual memory leak. This regression was first released in
4.3.0. (@interwq, @jasone)
- Refactor atomic and prng APIs to restore support for 32-bit platforms that
use pre-C11 toolchains, e.g. FreeBSD's mips. (@jasone)
* 4.3.0 (November 4, 2016)
This is the first release that passes the test suite for multiple Windows
configurations, thanks in large part to @glandium setting up continuous
integration via AppVeyor (and Travis CI for Linux and OS X).
New features:
- Add "J" (JSON) support to malloc_stats_print(). (@jasone)
- Add Cray compiler support. (@ronawho)
Optimizations:
- Add/use adaptive spinning for bootstrapping and radix tree node
initialization. (@jasone)
Bug fixes:
- Fix large allocation to search starting in the optimal size class heap,
which can substantially reduce virtual memory churn and fragmentation. This
regression was first released in 4.0.0. (@mjp41, @jasone)
- Fix stats.arenas.<i>.nthreads accounting. (@interwq)
- Fix and simplify decay-based purging. (@jasone)
- Make DSS (sbrk(2)-related) operations lockless, which resolves potential
deadlocks during thread exit. (@jasone)
- Fix over-sized allocation of radix tree leaf nodes. (@mjp41, @ogaun,
@jasone)
- Fix over-sized allocation of arena_t (plus associated stats) data
structures. (@jasone, @interwq)
- Fix EXTRA_CFLAGS to not affect configuration. (@jasone)
- Fix a Valgrind integration bug. (@ronawho)
- Disallow 0x5a junk filling when running in Valgrind. (@jasone)
- Fix a file descriptor leak on Linux. This regression was first released in
4.2.0. (@vsarunas, @jasone)
- Fix static linking of jemalloc with glibc. (@djwatson)
- Use syscall(2) rather than {open,read,close}(2) during boot on Linux. This
works around other libraries' system call wrappers performing reentrant
allocation. (@kspinka, @Whissi, @jasone)
- Fix OS X default zone replacement to work with OS X 10.12. (@glandium,
@jasone)
- Fix cached memory management to avoid needless commit/decommit operations
during purging, which resolves permanent virtual memory map fragmentation
issues on Windows. (@mjp41, @jasone)
- Fix TSD fetches to avoid (recursive) allocation. This is relevant to
non-TLS and Windows configurations. (@jasone)
- Fix malloc_conf overriding to work on Windows. (@jasone)
- Forcibly disable lazy-lock on Windows (was forcibly *enabled*). (@jasone)
* 4.2.1 (June 8, 2016)
Bug fixes:
- Fix bootstrapping issues for configurations that require allocation during
tsd initialization (e.g. --disable-tls). (@cferris1000, @jasone)
- Fix gettimeofday() version of nstime_update(). (@ronawho)
- Fix Valgrind regressions in calloc() and chunk_alloc_wrapper(). (@ronawho)
- Fix potential VM map fragmentation regression. (@jasone)
- Fix opt_zero-triggered in-place huge reallocation zeroing. (@jasone)
- Fix heap profiling context leaks in reallocation edge cases. (@jasone)
* 4.2.0 (May 12, 2016)
New features:
- Add the arena.<i>.reset mallctl, which makes it possible to discard all of
an arena's allocations in a single operation. (@jasone)
- Add the stats.retained and stats.arenas.<i>.retained statistics. (@jasone)
- Add the --with-version configure option. (@jasone)
- Support --with-lg-page values larger than actual page size. (@jasone)
Optimizations:
- Use pairing heaps rather than red-black trees for various hot data
structures. (@djwatson, @jasone)
- Streamline fast paths of rtree operations. (@jasone)
- Optimize the fast paths of calloc() and [m,d,sd]allocx(). (@jasone)
- Decommit unused virtual memory if the OS does not overcommit. (@jasone)
- Specify MAP_NORESERVE on Linux if [heuristic] overcommit is active, in order
to avoid unfortunate interactions during fork(2). (@jasone)
Bug fixes:
- Fix chunk accounting related to triggering gdump profiles. (@jasone)
- Link against librt for clock_gettime(2) if glibc < 2.17. (@jasone)
- Scale leak report summary according to sampling probability. (@jasone)
* 4.1.1 (May 3, 2016)
This bugfix release resolves a variety of mostly minor issues, though the
bitmap fix is critical for 64-bit Windows.
Bug fixes:
- Fix the linear scan version of bitmap_sfu() to shift by the proper amount
even when sizeof(long) is not the same as sizeof(void *), as on 64-bit
Windows. (@jasone)
- Fix hashing functions to avoid unaligned memory accesses (and resulting
crashes). This is relevant at least to some ARM-based platforms.
(@rkmisra)
- Fix fork()-related lock rank ordering reversals. These reversals were
unlikely to cause deadlocks in practice except when heap profiling was
enabled and active. (@jasone)
- Fix various chunk leaks in OOM code paths. (@jasone)
- Fix malloc_stats_print() to print opt.narenas correctly. (@jasone)
- Fix MSVC-specific build/test issues. (@rustyx, @yuslepukhin)
- Fix a variety of test failures that were due to test fragility rather than
core bugs. (@jasone)
* 4.1.0 (February 28, 2016)
This release is primarily about optimizations, but it also incorporates a lot
of portability-motivated refactoring and enhancements. Many people worked on
this release, to an extent that even with the omission here of minor changes
(see git revision history), and of the people who reported and diagnosed
issues, so much of the work was contributed that starting with this release,
changes are annotated with author credits to help reflect the collaborative
effort involved.
New features:
- Implement decay-based unused dirty page purging, a major optimization with
mallctl API impact. This is an alternative to the existing ratio-based
unused dirty page purging, and is intended to eventually become the sole
purging mechanism. New mallctls:
+ opt.purge
+ opt.decay_time
+ arena.<i>.decay
+ arena.<i>.decay_time
+ arenas.decay_time
+ stats.arenas.<i>.decay_time
(@jasone, @cevans87)
- Add --with-malloc-conf, which makes it possible to embed a default
options string during configuration. This was motivated by the desire to
specify --with-malloc-conf=purge:decay , since the default must remain
purge:ratio until the 5.0.0 release. (@jasone)
- Add MS Visual Studio 2015 support. (@rustyx, @yuslepukhin)
- Make *allocx() size class overflow behavior defined. The maximum
size class is now less than PTRDIFF_MAX to protect applications against
numerical overflow, and all allocation functions are guaranteed to indicate
errors rather than potentially crashing if the request size exceeds the
maximum size class. (@jasone)
- jeprof:
+ Add raw heap profile support. (@jasone)
+ Add --retain and --exclude for backtrace symbol filtering. (@jasone)
Optimizations:
- Optimize the fast path to combine various bootstrapping and configuration
checks and execute more streamlined code in the common case. (@interwq)
- Use linear scan for small bitmaps (used for small object tracking). In
addition to speeding up bitmap operations on 64-bit systems, this reduces
allocator metadata overhead by approximately 0.2%. (@djwatson)
- Separate arena_avail trees, which substantially speeds up run tree
operations. (@djwatson)
- Use memoization (boot-time-computed table) for run quantization. Separate
arena_avail trees reduced the importance of this optimization. (@jasone)
- Attempt mmap-based in-place huge reallocation. This can dramatically speed
up incremental huge reallocation. (@jasone)
Incompatible changes:
- Make opt.narenas unsigned rather than size_t. (@jasone)
Bug fixes:
- Fix stats.cactive accounting regression. (@rustyx, @jasone)
- Handle unaligned keys in hash(). This caused problems for some ARM systems.
(@jasone, @cferris1000)
- Refactor arenas array. In addition to fixing a fork-related deadlock, this
makes arena lookups faster and simpler. (@jasone)
- Move retained memory allocation out of the default chunk allocation
function, to a location that gets executed even if the application installs
a custom chunk allocation function. This resolves a virtual memory leak.
(@buchgr)
- Fix a potential tsd cleanup leak. (@cferris1000, @jasone)
- Fix run quantization. In practice this bug had no impact unless
applications requested memory with alignment exceeding one page.
(@jasone, @djwatson)
- Fix LinuxThreads-specific bootstrapping deadlock. (Cosmin Paraschiv)
- jeprof:
+ Don't discard curl options if timeout is not defined. (@djwatson)
+ Detect failed profile fetches. (@djwatson)
- Fix stats.arenas.<i>.{dss,lg_dirty_mult,decay_time,pactive,pdirty} for
--disable-stats case. (@jasone)
* 4.0.4 (October 24, 2015)
This bugfix release fixes another xallocx() regression. No other regressions
have come to light in over a month, so this is likely a good starting point
for people who prefer to wait for "dot one" releases with all the major issues
shaken out.
Bug fixes:
- Fix xallocx(..., MALLOCX_ZERO to zero the last full trailing page of large
allocations that have been randomly assigned an offset of 0 when
--enable-cache-oblivious configure option is enabled.
* 4.0.3 (September 24, 2015)
This bugfix release continues the trend of xallocx() and heap profiling fixes.
Bug fixes:
- Fix xallocx(..., MALLOCX_ZERO) to zero all trailing bytes of large
allocations when --enable-cache-oblivious configure option is enabled.
- Fix xallocx(..., MALLOCX_ZERO) to zero trailing bytes of huge allocations
when resizing from/to a size class that is not a multiple of the chunk size.
- Fix prof_tctx_dump_iter() to filter out nodes that were created after heap
profile dumping started.
- Work around a potentially bad thread-specific data initialization
interaction with NPTL (glibc's pthreads implementation).
* 4.0.2 (September 21, 2015)
This bugfix release addresses a few bugs specific to heap profiling.
Bug fixes:
- Fix ixallocx_prof_sample() to never modify nor create sampled small
allocations. xallocx() is in general incapable of moving small allocations,
so this fix removes buggy code without loss of generality.
- Fix irallocx_prof_sample() to always allocate large regions, even when
alignment is non-zero.
- Fix prof_alloc_rollback() to read tdata from thread-specific data rather
than dereferencing a potentially invalid tctx.
* 4.0.1 (September 15, 2015)
This is a bugfix release that is somewhat high risk due to the amount of
refactoring required to address deep xallocx() problems. As a side effect of
these fixes, xallocx() now tries harder to partially fulfill requests for
optional extra space. Note that a couple of minor heap profiling
optimizations are included, but these are better thought of as performance
fixes that were integral to discovering most of the other bugs.
Optimizations:
- Avoid a chunk metadata read in arena_prof_tctx_set(), since it is in the
fast path when heap profiling is enabled. Additionally, split a special
case out into arena_prof_tctx_reset(), which also avoids chunk metadata
reads.
- Optimize irallocx_prof() to optimistically update the sampler state. The
prior implementation appears to have been a holdover from when
rallocx()/xallocx() functionality was combined as rallocm().
Bug fixes:
- Fix TLS configuration such that it is enabled by default for platforms on
which it works correctly.
- Fix arenas_cache_cleanup() and arena_get_hard() to handle
allocation/deallocation within the application's thread-specific data
cleanup functions even after arenas_cache is torn down.
- Fix xallocx() bugs related to size+extra exceeding HUGE_MAXCLASS.
- Fix chunk purge hook calls for in-place huge shrinking reallocation to
specify the old chunk size rather than the new chunk size. This bug caused
no correctness issues for the default chunk purge function, but was
visible to custom functions set via the "arena.<i>.chunk_hooks" mallctl.
- Fix heap profiling bugs:
+ Fix heap profiling to distinguish among otherwise identical sample sites
with interposed resets (triggered via the "prof.reset" mallctl). This bug
could cause data structure corruption that would most likely result in a
segfault.
+ Fix irealloc_prof() to prof_alloc_rollback() on OOM.
+ Make one call to prof_active_get_unlocked() per allocation event, and use
the result throughout the relevant functions that handle an allocation
event. Also add a missing check in prof_realloc(). These fixes protect
allocation events against concurrent prof_active changes.
+ Fix ixallocx_prof() to pass usize_max and zero to ixallocx_prof_sample()
in the correct order.
+ Fix prof_realloc() to call prof_free_sampled_object() after calling
prof_malloc_sample_object(). Prior to this fix, if tctx and old_tctx were
the same, the tctx could have been prematurely destroyed.
- Fix portability bugs:
+ Don't bitshift by negative amounts when encoding/decoding run sizes in
chunk header maps. This affected systems with page sizes greater than 8
KiB.
+ Rename index_t to szind_t to avoid an existing type on Solaris.
+ Add JEMALLOC_CXX_THROW to the memalign() function prototype, in order to
match glibc and avoid compilation errors when including both
jemalloc/jemalloc.h and malloc.h in C++ code.
+ Don't assume that /bin/sh is appropriate when running size_classes.sh
during configuration.
+ Consider __sparcv9 a synonym for __sparc64__ when defining LG_QUANTUM.
+ Link tests to librt if it contains clock_gettime(2).
* 4.0.0 (August 17, 2015)
This version contains many speed and space optimizations, both minor and
major. The major themes are generalization, unification, and simplification.
Although many of these optimizations cause no visible behavior change, their
cumulative effect is substantial.
New features:
- Normalize size class spacing to be consistent across the complete size
range. By default there are four size classes per size doubling, but this
is now configurable via the --with-lg-size-class-group option. Also add the
--with-lg-page, --with-lg-page-sizes, --with-lg-quantum, and
--with-lg-tiny-min options, which can be used to tweak page and size class
settings. Impacts:
+ Worst case performance for incrementally growing/shrinking reallocation
is improved because there are far fewer size classes, and therefore
copying happens less often.
+ Internal fragmentation is limited to 20% for all but the smallest size
classes (those less than four times the quantum). (1B + 4 KiB)
and (1B + 4 MiB) previously suffered nearly 50% internal fragmentation.
+ Chunk fragmentation tends to be lower because there are fewer distinct run
sizes to pack.
- Add support for explicit tcaches. The "tcache.create", "tcache.flush", and
"tcache.destroy" mallctls control tcache lifetime and flushing, and the
MALLOCX_TCACHE(tc) and MALLOCX_TCACHE_NONE flags to the *allocx() API
control which tcache is used for each operation.
- Implement per thread heap profiling, as well as the ability to
enable/disable heap profiling on a per thread basis. Add the "prof.reset",
"prof.lg_sample", "thread.prof.name", "thread.prof.active",
"opt.prof_thread_active_init", "prof.thread_active_init", and
"thread.prof.active" mallctls.
- Add support for per arena application-specified chunk allocators, configured
via the "arena.<i>.chunk_hooks" mallctl.
- Refactor huge allocation to be managed by arenas, so that arenas now
function as general purpose independent allocators. This is important in
the context of user-specified chunk allocators, aside from the scalability
benefits. Related new statistics:
+ The "stats.arenas.<i>.huge.allocated", "stats.arenas.<i>.huge.nmalloc",
"stats.arenas.<i>.huge.ndalloc", and "stats.arenas.<i>.huge.nrequests"
mallctls provide high level per arena huge allocation statistics.
+ The "arenas.nhchunks", "arenas.hchunk.<i>.size",
"stats.arenas.<i>.hchunks.<j>.nmalloc",
"stats.arenas.<i>.hchunks.<j>.ndalloc",
"stats.arenas.<i>.hchunks.<j>.nrequests", and
"stats.arenas.<i>.hchunks.<j>.curhchunks" mallctls provide per size class
statistics.
- Add the 'util' column to malloc_stats_print() output, which reports the
proportion of available regions that are currently in use for each small
size class.
- Add "alloc" and "free" modes for for junk filling (see the "opt.junk"
mallctl), so that it is possible to separately enable junk filling for
allocation versus deallocation.
- Add the jemalloc-config script, which provides information about how
jemalloc was configured, and how to integrate it into application builds.
- Add metadata statistics, which are accessible via the "stats.metadata",
"stats.arenas.<i>.metadata.mapped", and
"stats.arenas.<i>.metadata.allocated" mallctls.
- Add the "stats.resident" mallctl, which reports the upper limit of
physically resident memory mapped by the allocator.
- Add per arena control over unused dirty page purging, via the
"arenas.lg_dirty_mult", "arena.<i>.lg_dirty_mult", and
"stats.arenas.<i>.lg_dirty_mult" mallctls.
- Add the "prof.gdump" mallctl, which makes it possible to toggle the gdump
feature on/off during program execution.
- Add sdallocx(), which implements sized deallocation. The primary
optimization over dallocx() is the removal of a metadata read, which often
suffers an L1 cache miss.
- Add missing header includes in jemalloc/jemalloc.h, so that applications
only have to #include <jemalloc/jemalloc.h>.
- Add support for additional platforms:
+ Bitrig
+ Cygwin
+ DragonFlyBSD
+ iOS
+ OpenBSD
+ OpenRISC/or1k
Optimizations:
- Maintain dirty runs in per arena LRUs rather than in per arena trees of
dirty-run-containing chunks. In practice this change significantly reduces
dirty page purging volume.
- Integrate whole chunks into the unused dirty page purging machinery. This
reduces the cost of repeated huge allocation/deallocation, because it
effectively introduces a cache of chunks.
- Split the arena chunk map into two separate arrays, in order to increase
cache locality for the frequently accessed bits.
- Move small run metadata out of runs, into arena chunk headers. This reduces
run fragmentation, smaller runs reduce external fragmentation for small size
classes, and packed (less uniformly aligned) metadata layout improves CPU
cache set distribution.
- Randomly distribute large allocation base pointer alignment relative to page
boundaries in order to more uniformly utilize CPU cache sets. This can be
disabled via the --disable-cache-oblivious configure option, and queried via
the "config.cache_oblivious" mallctl.
- Micro-optimize the fast paths for the public API functions.
- Refactor thread-specific data to reside in a single structure. This assures
that only a single TLS read is necessary per call into the public API.
- Implement in-place huge allocation growing and shrinking.
- Refactor rtree (radix tree for chunk lookups) to be lock-free, and make
additional optimizations that reduce maximum lookup depth to one or two
levels. This resolves what was a concurrency bottleneck for per arena huge
allocation, because a global data structure is critical for determining
which arenas own which huge allocations.
Incompatible changes:
- Replace --enable-cc-silence with --disable-cc-silence to suppress spurious
warnings by default.
- Assure that the constness of malloc_usable_size()'s return type matches that
of the system implementation.
- Change the heap profile dump format to support per thread heap profiling,
rename pprof to jeprof, and enhance it with the --thread=<n> option. As a
result, the bundled jeprof must now be used rather than the upstream
(gperftools) pprof.
- Disable "opt.prof_final" by default, in order to avoid atexit(3), which can
internally deadlock on some platforms.
- Change the "arenas.nlruns" mallctl type from size_t to unsigned.
- Replace the "stats.arenas.<i>.bins.<j>.allocated" mallctl with
"stats.arenas.<i>.bins.<j>.curregs".
- Ignore MALLOC_CONF in set{uid,gid,cap} binaries.
- Ignore MALLOCX_ARENA(a) in dallocx(), in favor of using the
MALLOCX_TCACHE(tc) and MALLOCX_TCACHE_NONE flags to control tcache usage.
Removed features:
- Remove the *allocm() API, which is superseded by the *allocx() API.
- Remove the --enable-dss options, and make dss non-optional on all platforms
which support sbrk(2).
- Remove the "arenas.purge" mallctl, which was obsoleted by the
"arena.<i>.purge" mallctl in 3.1.0.
- Remove the unnecessary "opt.valgrind" mallctl; jemalloc automatically
detects whether it is running inside Valgrind.
- Remove the "stats.huge.allocated", "stats.huge.nmalloc", and
"stats.huge.ndalloc" mallctls.
- Remove the --enable-mremap option.
- Remove the "stats.chunks.current", "stats.chunks.total", and
"stats.chunks.high" mallctls.
Bug fixes:
- Fix the cactive statistic to decrease (rather than increase) when active
memory decreases. This regression was first released in 3.5.0.
- Fix OOM handling in memalign() and valloc(). A variant of this bug existed
in all releases since 2.0.0, which introduced these functions.
- Fix an OOM-related regression in arena_tcache_fill_small(), which could
cause cache corruption on OOM. This regression was present in all releases
from 2.2.0 through 3.6.0.
- Fix size class overflow handling for malloc(), posix_memalign(), memalign(),
calloc(), and realloc() when profiling is enabled.
- Fix the "arena.<i>.dss" mallctl to return an error if "primary" or
"secondary" precedence is specified, but sbrk(2) is not supported.
- Fix fallback lg_floor() implementations to handle extremely large inputs.
- Ensure the default purgeable zone is after the default zone on OS X.
- Fix latent bugs in atomic_*().
- Fix the "arena.<i>.dss" mallctl to handle read-only calls.
- Fix tls_model configuration to enable the initial-exec model when possible.
- Mark malloc_conf as a weak symbol so that the application can override it.
- Correctly detect glibc's adaptive pthread mutexes.
- Fix the --without-export configure option.
* 3.6.0 (March 31, 2014)
This version contains a critical bug fix for a regression present in 3.5.0 and
@@ -855,7 +21,7 @@ brevity. Much more detail can be found in the git revision history:
backtracing to be reliable.
- Use dss allocation precedence for huge allocations as well as small/large
allocations.
- Fix test assertion failure message formatting. This bug did not manifest on
- Fix test assertion failure message formatting. This bug did not manifect on
x86_64 systems because of implementation subtleties in va_list.
- Fix inconsequential test failures for hash and SFMT code.
@@ -1350,7 +516,7 @@ brevity. Much more detail can be found in the git revision history:
- Make it possible for the application to manually flush a thread's cache, via
the "tcache.flush" mallctl.
- Base maximum dirty page count on proportion of active memory.
- Compute various additional run-time statistics, including per size class
- Compute various addtional run-time statistics, including per size class
statistics for large objects.
- Expose malloc_stats_print(), which can be called repeatedly by the
application.
+306
View File
@@ -0,0 +1,306 @@
Building and installing jemalloc can be as simple as typing the following while
in the root directory of the source tree:
./configure
make
make install
=== Advanced configuration =====================================================
The 'configure' script supports numerous options that allow control of which
functionality is enabled, where jemalloc is installed, etc. Optionally, pass
any of the following arguments (not a definitive list) to 'configure':
--help
Print a definitive list of options.
--prefix=<install-root-dir>
Set the base directory in which to install. For example:
./configure --prefix=/usr/local
will cause files to be installed into /usr/local/include, /usr/local/lib,
and /usr/local/man.
--with-rpath=<colon-separated-rpath>
Embed one or more library paths, so that libjemalloc can find the libraries
it is linked to. This works only on ELF-based systems.
--with-mangling=<map>
Mangle public symbols specified in <map> which is a comma-separated list of
name:mangled pairs.
For example, to use ld's --wrap option as an alternative method for
overriding libc's malloc implementation, specify something like:
--with-mangling=malloc:__wrap_malloc,free:__wrap_free[...]
Note that mangling happens prior to application of the prefix specified by
--with-jemalloc-prefix, and mangled symbols are then ignored when applying
the prefix.
--with-jemalloc-prefix=<prefix>
Prefix all public APIs with <prefix>. For example, if <prefix> is
"prefix_", API changes like the following occur:
malloc() --> prefix_malloc()
malloc_conf --> prefix_malloc_conf
/etc/malloc.conf --> /etc/prefix_malloc.conf
MALLOC_CONF --> PREFIX_MALLOC_CONF
This makes it possible to use jemalloc at the same time as the system
allocator, or even to use multiple copies of jemalloc simultaneously.
By default, the prefix is "", except on OS X, where it is "je_". On OS X,
jemalloc overlays the default malloc zone, but makes no attempt to actually
replace the "malloc", "calloc", etc. symbols.
--without-export
Don't export public APIs. This can be useful when building jemalloc as a
static library, or to avoid exporting public APIs when using the zone
allocator on OSX.
--with-private-namespace=<prefix>
Prefix all library-private APIs with <prefix>je_. For shared libraries,
symbol visibility mechanisms prevent these symbols from being exported, but
for static libraries, naming collisions are a real possibility. By
default, <prefix> is empty, which results in a symbol prefix of je_ .
--with-install-suffix=<suffix>
Append <suffix> to the base name of all installed files, such that multiple
versions of jemalloc can coexist in the same installation directory. For
example, libjemalloc.so.0 becomes libjemalloc<suffix>.so.0.
--enable-cc-silence
Enable code that silences non-useful compiler warnings. This is helpful
when trying to tell serious warnings from those due to compiler
limitations, but it potentially incurs a performance penalty.
--enable-debug
Enable assertions and validation code. This incurs a substantial
performance hit, but is very useful during application development.
Implies --enable-ivsalloc.
--enable-code-coverage
Enable code coverage support, for use during jemalloc test development.
Additional testing targets are available if this option is enabled:
coverage
coverage_unit
coverage_integration
coverage_stress
These targets do not clear code coverage results from previous runs, and
there are interactions between the various coverage targets, so it is
usually advisable to run 'make clean' between repeated code coverage runs.
--enable-ivsalloc
Enable validation code, which verifies that pointers reside within
jemalloc-owned chunks before dereferencing them. This incurs a substantial
performance hit.
--disable-stats
Disable statistics gathering functionality. See the "opt.stats_print"
option documentation for usage details.
--enable-prof
Enable heap profiling and leak detection functionality. See the "opt.prof"
option documentation for usage details. When enabled, there are several
approaches to backtracing, and the configure script chooses the first one
in the following list that appears to function correctly:
+ libunwind (requires --enable-prof-libunwind)
+ libgcc (unless --disable-prof-libgcc)
+ gcc intrinsics (unless --disable-prof-gcc)
--enable-prof-libunwind
Use the libunwind library (http://www.nongnu.org/libunwind/) for stack
backtracing.
--disable-prof-libgcc
Disable the use of libgcc's backtracing functionality.
--disable-prof-gcc
Disable the use of gcc intrinsics for backtracing.
--with-static-libunwind=<libunwind.a>
Statically link against the specified libunwind.a rather than dynamically
linking with -lunwind.
--disable-tcache
Disable thread-specific caches for small objects. Objects are cached and
released in bulk, thus reducing the total number of mutex operations. See
the "opt.tcache" option for usage details.
--enable-mremap
Enable huge realloc() via mremap(2). mremap() is disabled by default
because the flavor used is specific to Linux, which has a quirk in its
virtual memory allocation algorithm that causes semi-permanent VM map holes
under normal jemalloc operation.
--disable-munmap
Disable virtual memory deallocation via munmap(2); instead keep track of
the virtual memory for later use. munmap() is disabled by default (i.e.
--disable-munmap is implied) on Linux, which has a quirk in its virtual
memory allocation algorithm that causes semi-permanent VM map holes under
normal jemalloc operation.
--enable-dss
Enable support for page allocation/deallocation via sbrk(2), in addition to
mmap(2).
--disable-fill
Disable support for junk/zero filling of memory, quarantine, and redzones.
See the "opt.junk", "opt.zero", "opt.quarantine", and "opt.redzone" option
documentation for usage details.
--disable-valgrind
Disable support for Valgrind.
--disable-experimental
Disable support for the experimental API (*allocm()).
--disable-zone-allocator
Disable zone allocator for Darwin. This means jemalloc won't be hooked as
the default allocator on OSX/iOS.
--enable-utrace
Enable utrace(2)-based allocation tracing. This feature is not broadly
portable (FreeBSD has it, but Linux and OS X do not).
--enable-xmalloc
Enable support for optional immediate termination due to out-of-memory
errors, as is commonly implemented by "xmalloc" wrapper function for malloc.
See the "opt.xmalloc" option documentation for usage details.
--enable-lazy-lock
Enable code that wraps pthread_create() to detect when an application
switches from single-threaded to multi-threaded mode, so that it can avoid
mutex locking/unlocking operations while in single-threaded mode. In
practice, this feature usually has little impact on performance unless
thread-specific caching is disabled.
--disable-tls
Disable thread-local storage (TLS), which allows for fast access to
thread-local variables via the __thread keyword. If TLS is available,
jemalloc uses it for several purposes.
--with-xslroot=<path>
Specify where to find DocBook XSL stylesheets when building the
documentation.
The following environment variables (not a definitive list) impact configure's
behavior:
CFLAGS="?"
Pass these flags to the compiler. You probably shouldn't define this unless
you know what you are doing. (Use EXTRA_CFLAGS instead.)
EXTRA_CFLAGS="?"
Append these flags to CFLAGS. This makes it possible to add flags such as
-Werror, while allowing the configure script to determine what other flags
are appropriate for the specified configuration.
The configure script specifically checks whether an optimization flag (-O*)
is specified in EXTRA_CFLAGS, and refrains from specifying an optimization
level if it finds that one has already been specified.
CPPFLAGS="?"
Pass these flags to the C preprocessor. Note that CFLAGS is not passed to
'cpp' when 'configure' is looking for include files, so you must use
CPPFLAGS instead if you need to help 'configure' find header files.
LD_LIBRARY_PATH="?"
'ld' uses this colon-separated list to find libraries.
LDFLAGS="?"
Pass these flags when linking.
PATH="?"
'configure' uses this to find programs.
=== Advanced compilation =======================================================
To build only parts of jemalloc, use the following targets:
build_lib_shared
build_lib_static
build_lib
build_doc_html
build_doc_man
build_doc
To install only parts of jemalloc, use the following targets:
install_bin
install_include
install_lib_shared
install_lib_static
install_lib
install_doc_html
install_doc_man
install_doc
To clean up build results to varying degrees, use the following make targets:
clean
distclean
relclean
=== Advanced installation ======================================================
Optionally, define make variables when invoking make, including (not
exclusively):
INCLUDEDIR="?"
Use this as the installation prefix for header files.
LIBDIR="?"
Use this as the installation prefix for libraries.
MANDIR="?"
Use this as the installation prefix for man pages.
DESTDIR="?"
Prepend DESTDIR to INCLUDEDIR, LIBDIR, DATADIR, and MANDIR. This is useful
when installing to a different path than was specified via --prefix.
CC="?"
Use this to invoke the C compiler.
CFLAGS="?"
Pass these flags to the compiler.
CPPFLAGS="?"
Pass these flags to the C preprocessor.
LDFLAGS="?"
Pass these flags when linking.
PATH="?"
Use this to search for programs used during configuration and building.
=== Development ================================================================
If you intend to make non-trivial changes to jemalloc, use the 'autogen.sh'
script rather than 'configure'. This re-generates 'configure', enables
configuration dependency rules, and enables re-generation of automatically
generated source files.
The build system supports using an object directory separate from the source
tree. For example, you can create an 'obj' directory, and from within that
directory, issue configuration and build commands:
autoconf
mkdir obj
cd obj
../configure --enable-autogen
make
=== Documentation ==============================================================
The manual page is generated in both html and roff formats. Any web browser
can be used to view the html manual. The roff manual page can be formatted
prior to installation via the following command:
nroff -man -t doc/jemalloc.3
-423
View File
@@ -1,423 +0,0 @@
Building and installing a packaged release of jemalloc can be as simple as
typing the following while in the root directory of the source tree:
./configure
make
make install
If building from unpackaged developer sources, the simplest command sequence
that might work is:
./autogen.sh
make dist
make
make install
Note that documentation is not built by the default target because doing so
would create a dependency on xsltproc in packaged releases, hence the
requirement to either run 'make dist' or avoid installing docs via the various
install_* targets documented below.
## Advanced configuration
The 'configure' script supports numerous options that allow control of which
functionality is enabled, where jemalloc is installed, etc. Optionally, pass
any of the following arguments (not a definitive list) to 'configure':
* `--help`
Print a definitive list of options.
* `--prefix=<install-root-dir>`
Set the base directory in which to install. For example:
./configure --prefix=/usr/local
will cause files to be installed into /usr/local/include, /usr/local/lib,
and /usr/local/man.
* `--with-version=(<major>.<minor>.<bugfix>-<nrev>-g<gid>|VERSION)`
The VERSION file is mandatory for successful configuration, and the
following steps are taken to assure its presence:
1) If --with-version=<major>.<minor>.<bugfix>-<nrev>-g<gid> is specified,
generate VERSION using the specified value.
2) If --with-version is not specified in either form and the source
directory is inside a git repository, try to generate VERSION via 'git
describe' invocations that pattern-match release tags.
3) If VERSION is missing, generate it with a bogus version:
0.0.0-0-g0000000000000000000000000000000000000000
Note that --with-version=VERSION bypasses (1) and (2), which simplifies
VERSION configuration when embedding a jemalloc release into another
project's git repository.
* `--with-rpath=<colon-separated-rpath>`
Embed one or more library paths, so that libjemalloc can find the libraries
it is linked to. This works only on ELF-based systems.
* `--with-mangling=<map>`
Mangle public symbols specified in <map> which is a comma-separated list of
name:mangled pairs.
For example, to use ld's --wrap option as an alternative method for
overriding libc's malloc implementation, specify something like:
--with-mangling=malloc:__wrap_malloc,free:__wrap_free[...]
Note that mangling happens prior to application of the prefix specified by
--with-jemalloc-prefix, and mangled symbols are then ignored when applying
the prefix.
* `--with-jemalloc-prefix=<prefix>`
Prefix all public APIs with <prefix>. For example, if <prefix> is
"prefix_", API changes like the following occur:
malloc() --> prefix_malloc()
malloc_conf --> prefix_malloc_conf
/etc/malloc.conf --> /etc/prefix_malloc.conf
MALLOC_CONF --> PREFIX_MALLOC_CONF
This makes it possible to use jemalloc at the same time as the system
allocator, or even to use multiple copies of jemalloc simultaneously.
By default, the prefix is "", except on OS X, where it is "je_". On OS X,
jemalloc overlays the default malloc zone, but makes no attempt to actually
replace the "malloc", "calloc", etc. symbols.
* `--without-export`
Don't export public APIs. This can be useful when building jemalloc as a
static library, or to avoid exporting public APIs when using the zone
allocator on OSX.
* `--with-private-namespace=<prefix>`
Prefix all library-private APIs with <prefix>je_. For shared libraries,
symbol visibility mechanisms prevent these symbols from being exported, but
for static libraries, naming collisions are a real possibility. By
default, <prefix> is empty, which results in a symbol prefix of je_ .
* `--with-install-suffix=<suffix>`
Append <suffix> to the base name of all installed files, such that multiple
versions of jemalloc can coexist in the same installation directory. For
example, libjemalloc.so.0 becomes libjemalloc<suffix>.so.0.
* `--with-malloc-conf=<malloc_conf>`
Embed `<malloc_conf>` as a run-time options string that is processed prior to
the malloc_conf global variable, the /etc/malloc.conf symlink, and the
MALLOC_CONF environment variable. For example, to change the default decay
time to 30 seconds:
--with-malloc-conf=decay_ms:30000
* `--enable-debug`
Enable assertions and validation code. This incurs a substantial
performance hit, but is very useful during application development.
* `--disable-stats`
Disable statistics gathering functionality. See the "opt.stats_print"
option documentation for usage details.
* `--enable-prof`
Enable heap profiling and leak detection functionality. See the "opt.prof"
option documentation for usage details. When enabled, there are several
approaches to backtracing, and the configure script chooses the first one
in the following list that appears to function correctly:
+ libunwind (requires --enable-prof-libunwind)
+ libgcc (unless --disable-prof-libgcc)
+ gcc intrinsics (unless --disable-prof-gcc)
* `--enable-prof-libunwind`
Use the libunwind library (http://www.nongnu.org/libunwind/) for stack
backtracing.
* `--disable-prof-libgcc`
Disable the use of libgcc's backtracing functionality.
* `--disable-prof-gcc`
Disable the use of gcc intrinsics for backtracing.
* `--with-static-libunwind=<libunwind.a>`
Statically link against the specified libunwind.a rather than dynamically
linking with -lunwind.
* `--disable-fill`
Disable support for junk/zero filling of memory. See the "opt.junk" and
"opt.zero" option documentation for usage details.
* `--disable-zone-allocator`
Disable zone allocator for Darwin. This means jemalloc won't be hooked as
the default allocator on OSX/iOS.
* `--enable-utrace`
Enable utrace(2)-based allocation tracing. This feature is not broadly
portable (FreeBSD has it, but Linux and OS X do not).
* `--enable-xmalloc`
Enable support for optional immediate termination due to out-of-memory
errors, as is commonly implemented by "xmalloc" wrapper function for malloc.
See the "opt.xmalloc" option documentation for usage details.
* `--enable-lazy-lock`
Enable code that wraps pthread_create() to detect when an application
switches from single-threaded to multi-threaded mode, so that it can avoid
mutex locking/unlocking operations while in single-threaded mode. In
practice, this feature usually has little impact on performance unless
thread-specific caching is disabled.
* `--disable-cache-oblivious`
Disable cache-oblivious large allocation alignment for large allocation
requests with no alignment constraints. If this feature is disabled, all
large allocations are page-aligned as an implementation artifact, which can
severely harm CPU cache utilization. However, the cache-oblivious layout
comes at the cost of one extra page per large allocation, which in the
most extreme case increases physical memory usage for the 16 KiB size class
to 20 KiB.
* `--disable-syscall`
Disable use of syscall(2) rather than {open,read,write,close}(2). This is
intended as a workaround for systems that place security limitations on
syscall(2).
* `--disable-cxx`
Disable C++ integration. This will cause new and delete operator
implementations to be omitted.
* `--with-xslroot=<path>`
Specify where to find DocBook XSL stylesheets when building the
documentation.
* `--with-lg-page=<lg-page>`
Specify the base 2 log of the allocator page size, which must in turn be at
least as large as the system page size. By default the configure script
determines the host's page size and sets the allocator page size equal to
the system page size, so this option need not be specified unless the
system page size may change between configuration and execution, e.g. when
cross compiling.
* `--with-lg-page-sizes=<lg-page-sizes>`
Specify the comma-separated base 2 logs of the page sizes to support. This
option may be useful when cross compiling in combination with
`--with-lg-page`, but its primary use case is for integration with FreeBSD's
libc, wherein jemalloc is embedded.
* `--with-lg-hugepage=<lg-hugepage>`
Specify the base 2 log of the system huge page size. This option is useful
when cross compiling, or when overriding the default for systems that do
not explicitly support huge pages.
* `--with-lg-quantum=<lg-quantum>`
Specify the base 2 log of the minimum allocation alignment. jemalloc needs
to know the minimum alignment that meets the following C standard
requirement (quoted from the April 12, 2011 draft of the C11 standard):
> The pointer returned if the allocation succeeds is suitably aligned so
that it may be assigned to a pointer to any type of object with a
fundamental alignment requirement and then used to access such an object
or an array of such objects in the space allocated [...]
This setting is architecture-specific, and although jemalloc includes known
safe values for the most commonly used modern architectures, there is a
wrinkle related to GNU libc (glibc) that may impact your choice of
<lg-quantum>. On most modern architectures, this mandates 16-byte
alignment (<lg-quantum>=4), but the glibc developers chose not to meet this
requirement for performance reasons. An old discussion can be found at
<https://sourceware.org/bugzilla/show_bug.cgi?id=206> . Unlike glibc,
jemalloc does follow the C standard by default (caveat: jemalloc
technically cheats for size classes smaller than the quantum), but the fact
that Linux systems already work around this allocator noncompliance means
that it is generally safe in practice to let jemalloc's minimum alignment
follow glibc's lead. If you specify `--with-lg-quantum=3` during
configuration, jemalloc will provide additional size classes that are not
16-byte-aligned (24, 40, and 56).
* `--with-lg-vaddr=<lg-vaddr>`
Specify the number of significant virtual address bits. By default, the
configure script attempts to detect virtual address size on those platforms
where it knows how, and picks a default otherwise. This option may be
useful when cross-compiling.
* `--disable-initial-exec-tls`
Disable the initial-exec TLS model for jemalloc's internal thread-local
storage (on those platforms that support explicit settings). This can allow
jemalloc to be dynamically loaded after program startup (e.g. using dlopen).
Note that in this case, there will be two malloc implementations operating
in the same process, which will almost certainly result in confusing runtime
crashes if pointers leak from one implementation to the other.
The following environment variables (not a definitive list) impact configure's
behavior:
* `CFLAGS="?"`
* `CXXFLAGS="?"`
Pass these flags to the C/C++ compiler. Any flags set by the configure
script are prepended, which means explicitly set flags generally take
precedence. Take care when specifying flags such as -Werror, because
configure tests may be affected in undesirable ways.
* `EXTRA_CFLAGS="?"`
* `EXTRA_CXXFLAGS="?"`
Append these flags to CFLAGS/CXXFLAGS, without passing them to the
compiler(s) during configuration. This makes it possible to add flags such
as -Werror, while allowing the configure script to determine what other
flags are appropriate for the specified configuration.
* `CPPFLAGS="?"`
Pass these flags to the C preprocessor. Note that CFLAGS is not passed to
'cpp' when 'configure' is looking for include files, so you must use
CPPFLAGS instead if you need to help 'configure' find header files.
* `LD_LIBRARY_PATH="?"`
'ld' uses this colon-separated list to find libraries.
* `LDFLAGS="?"`
Pass these flags when linking.
* `PATH="?"`
'configure' uses this to find programs.
In some cases it may be necessary to work around configuration results that do
not match reality. For example, Linux 4.5 added support for the MADV_FREE flag
to madvise(2), which can cause problems if building on a host with MADV_FREE
support and deploying to a target without. To work around this, use a cache
file to override the relevant configuration variable defined in configure.ac,
e.g.:
echo "je_cv_madv_free=no" > config.cache && ./configure -C
## Advanced compilation
To build only parts of jemalloc, use the following targets:
build_lib_shared
build_lib_static
build_lib
build_doc_html
build_doc_man
build_doc
To install only parts of jemalloc, use the following targets:
install_bin
install_include
install_lib_shared
install_lib_static
install_lib_pc
install_lib
install_doc_html
install_doc_man
install_doc
To clean up build results to varying degrees, use the following make targets:
clean
distclean
relclean
## Advanced installation
Optionally, define make variables when invoking make, including (not
exclusively):
* `INCLUDEDIR="?"`
Use this as the installation prefix for header files.
* `LIBDIR="?"`
Use this as the installation prefix for libraries.
* `MANDIR="?"`
Use this as the installation prefix for man pages.
* `DESTDIR="?"`
Prepend DESTDIR to INCLUDEDIR, LIBDIR, DATADIR, and MANDIR. This is useful
when installing to a different path than was specified via --prefix.
* `CC="?"`
Use this to invoke the C compiler.
* `CFLAGS="?"`
Pass these flags to the compiler.
* `CPPFLAGS="?"`
Pass these flags to the C preprocessor.
* `LDFLAGS="?"`
Pass these flags when linking.
* `PATH="?"`
Use this to search for programs used during configuration and building.
## Development
If you intend to make non-trivial changes to jemalloc, use the 'autogen.sh'
script rather than 'configure'. This re-generates 'configure', enables
configuration dependency rules, and enables re-generation of automatically
generated source files.
The build system supports using an object directory separate from the source
tree. For example, you can create an 'obj' directory, and from within that
directory, issue configuration and build commands:
autoconf
mkdir obj
cd obj
../configure --enable-autogen
make
## Documentation
The manual page is generated in both html and roff formats. Any web browser
can be used to view the html manual. The roff manual page can be formatted
prior to installation via the following command:
nroff -man -t doc/jemalloc.3
+126 -257
View File
@@ -9,7 +9,6 @@ vpath % .
SHELL := /bin/sh
CC := @CC@
CXX := @CXX@
# Configuration parameters.
DESTDIR =
@@ -24,15 +23,8 @@ abs_srcroot := @abs_srcroot@
abs_objroot := @abs_objroot@
# Build parameters.
CPPFLAGS := @CPPFLAGS@ -I$(objroot)include -I$(srcroot)include
CONFIGURE_CFLAGS := @CONFIGURE_CFLAGS@
SPECIFIED_CFLAGS := @SPECIFIED_CFLAGS@
EXTRA_CFLAGS := @EXTRA_CFLAGS@
CFLAGS := $(strip $(CONFIGURE_CFLAGS) $(SPECIFIED_CFLAGS) $(EXTRA_CFLAGS))
CONFIGURE_CXXFLAGS := @CONFIGURE_CXXFLAGS@
SPECIFIED_CXXFLAGS := @SPECIFIED_CXXFLAGS@
EXTRA_CXXFLAGS := @EXTRA_CXXFLAGS@
CXXFLAGS := $(strip $(CONFIGURE_CXXFLAGS) $(SPECIFIED_CXXFLAGS) $(EXTRA_CXXFLAGS))
CPPFLAGS := @CPPFLAGS@ -I$(srcroot)include -I$(objroot)include
CFLAGS := @CFLAGS@
LDFLAGS := @LDFLAGS@
EXTRA_LDFLAGS := @EXTRA_LDFLAGS@
LIBS := @LIBS@
@@ -50,29 +42,23 @@ XSLTPROC := @XSLTPROC@
AUTOCONF := @AUTOCONF@
_RPATH = @RPATH@
RPATH = $(if $(1),$(call _RPATH,$(1)))
cfghdrs_in := $(addprefix $(srcroot),@cfghdrs_in@)
cfghdrs_in := @cfghdrs_in@
cfghdrs_out := @cfghdrs_out@
cfgoutputs_in := $(addprefix $(srcroot),@cfgoutputs_in@)
cfgoutputs_in := @cfgoutputs_in@
cfgoutputs_out := @cfgoutputs_out@
enable_autogen := @enable_autogen@
enable_prof := @enable_prof@
enable_code_coverage := @enable_code_coverage@
enable_experimental := @enable_experimental@
enable_zone_allocator := @enable_zone_allocator@
MALLOC_CONF := @JEMALLOC_CPREFIX@MALLOC_CONF
link_whole_archive := @link_whole_archive@
DSO_LDFLAGS = @DSO_LDFLAGS@
SOREV = @SOREV@
PIC_CFLAGS = @PIC_CFLAGS@
CTARGET = @CTARGET@
LDTARGET = @LDTARGET@
TEST_LD_MODE = @TEST_LD_MODE@
MKLIB = @MKLIB@
AR = @AR@
ARFLAGS = @ARFLAGS@
DUMP_SYMS = @DUMP_SYMS@
AWK := @AWK@
CC_MM = @CC_MM@
LM := @LM@
INSTALL = @INSTALL@
ifeq (macho, $(ABI))
TEST_LIBRARY_PATH := DYLD_FALLBACK_LIBRARY_PATH="$(objroot)lib"
@@ -87,38 +73,16 @@ endif
LIBJEMALLOC := $(LIBPREFIX)jemalloc$(install_suffix)
# Lists of files.
BINS := $(objroot)bin/jemalloc-config $(objroot)bin/jemalloc.sh $(objroot)bin/jeprof
BINS := $(srcroot)bin/pprof $(objroot)bin/jemalloc.sh
C_HDRS := $(objroot)include/jemalloc/jemalloc$(install_suffix).h
C_SRCS := $(srcroot)src/jemalloc.c \
$(srcroot)src/arena.c \
$(srcroot)src/background_thread.c \
$(srcroot)src/base.c \
$(srcroot)src/bin.c \
$(srcroot)src/bitmap.c \
$(srcroot)src/ckh.c \
$(srcroot)src/ctl.c \
$(srcroot)src/div.c \
$(srcroot)src/extent.c \
$(srcroot)src/extent_dss.c \
$(srcroot)src/extent_mmap.c \
$(srcroot)src/hash.c \
$(srcroot)src/hooks.c \
$(srcroot)src/large.c \
$(srcroot)src/log.c \
$(srcroot)src/malloc_io.c \
$(srcroot)src/mutex.c \
$(srcroot)src/mutex_pool.c \
$(srcroot)src/nstime.c \
$(srcroot)src/pages.c \
$(srcroot)src/prng.c \
$(srcroot)src/prof.c \
$(srcroot)src/rtree.c \
$(srcroot)src/stats.c \
$(srcroot)src/sz.c \
$(srcroot)src/tcache.c \
$(srcroot)src/ticker.c \
$(srcroot)src/tsd.c \
$(srcroot)src/witness.c
C_SRCS := $(srcroot)src/jemalloc.c $(srcroot)src/arena.c \
$(srcroot)src/atomic.c $(srcroot)src/base.c $(srcroot)src/bitmap.c \
$(srcroot)src/chunk.c $(srcroot)src/chunk_dss.c \
$(srcroot)src/chunk_mmap.c $(srcroot)src/ckh.c $(srcroot)src/ctl.c \
$(srcroot)src/extent.c $(srcroot)src/hash.c $(srcroot)src/huge.c \
$(srcroot)src/mb.c $(srcroot)src/mutex.c $(srcroot)src/prof.c \
$(srcroot)src/quarantine.c $(srcroot)src/rtree.c $(srcroot)src/stats.c \
$(srcroot)src/tcache.c $(srcroot)src/util.c $(srcroot)src/tsd.c
ifeq ($(enable_zone_allocator), 1)
C_SRCS += $(srcroot)src/zone.c
endif
@@ -134,119 +98,57 @@ DSOS := $(objroot)lib/$(LIBJEMALLOC).$(SOREV)
ifneq ($(SOREV),$(SO))
DSOS += $(objroot)lib/$(LIBJEMALLOC).$(SO)
endif
ifeq (1, $(link_whole_archive))
LJEMALLOC := -Wl,--whole-archive -L$(objroot)lib -l$(LIBJEMALLOC) -Wl,--no-whole-archive
else
LJEMALLOC := $(objroot)lib/$(LIBJEMALLOC).$(IMPORTLIB)
endif
PC := $(objroot)jemalloc.pc
MAN3 := $(objroot)doc/jemalloc$(install_suffix).3
DOCS_XML := $(objroot)doc/jemalloc$(install_suffix).xml
DOCS_HTML := $(DOCS_XML:$(objroot)%.xml=$(objroot)%.html)
DOCS_MAN3 := $(DOCS_XML:$(objroot)%.xml=$(objroot)%.3)
DOCS_HTML := $(DOCS_XML:$(objroot)%.xml=$(srcroot)%.html)
DOCS_MAN3 := $(DOCS_XML:$(objroot)%.xml=$(srcroot)%.3)
DOCS := $(DOCS_HTML) $(DOCS_MAN3)
C_TESTLIB_SRCS := $(srcroot)test/src/btalloc.c $(srcroot)test/src/btalloc_0.c \
$(srcroot)test/src/btalloc_1.c $(srcroot)test/src/math.c \
$(srcroot)test/src/mtx.c $(srcroot)test/src/mq.c \
C_TESTLIB_SRCS := $(srcroot)test/src/math.c $(srcroot)test/src/mtx.c \
$(srcroot)test/src/SFMT.c $(srcroot)test/src/test.c \
$(srcroot)test/src/thd.c $(srcroot)test/src/timer.c
ifeq (1, $(link_whole_archive))
C_UTIL_INTEGRATION_SRCS :=
C_UTIL_CPP_SRCS :=
else
C_UTIL_INTEGRATION_SRCS := $(srcroot)src/nstime.c $(srcroot)src/malloc_io.c
C_UTIL_CPP_SRCS := $(srcroot)src/nstime.c $(srcroot)src/malloc_io.c
endif
TESTS_UNIT := \
$(srcroot)test/unit/a0.c \
$(srcroot)test/unit/arena_reset.c \
$(srcroot)test/unit/atomic.c \
$(srcroot)test/unit/background_thread.c \
$(srcroot)test/unit/background_thread_enable.c \
$(srcroot)test/unit/base.c \
$(srcroot)test/unit/bitmap.c \
$(srcroot)test/src/thd.c
C_UTIL_INTEGRATION_SRCS := $(srcroot)src/util.c
TESTS_UNIT := $(srcroot)test/unit/bitmap.c \
$(srcroot)test/unit/ckh.c \
$(srcroot)test/unit/decay.c \
$(srcroot)test/unit/div.c \
$(srcroot)test/unit/emitter.c \
$(srcroot)test/unit/extent_quantize.c \
$(srcroot)test/unit/fork.c \
$(srcroot)test/unit/hash.c \
$(srcroot)test/unit/hooks.c \
$(srcroot)test/unit/junk.c \
$(srcroot)test/unit/junk_alloc.c \
$(srcroot)test/unit/junk_free.c \
$(srcroot)test/unit/log.c \
$(srcroot)test/unit/mallctl.c \
$(srcroot)test/unit/malloc_io.c \
$(srcroot)test/unit/math.c \
$(srcroot)test/unit/mq.c \
$(srcroot)test/unit/mtx.c \
$(srcroot)test/unit/pack.c \
$(srcroot)test/unit/pages.c \
$(srcroot)test/unit/ph.c \
$(srcroot)test/unit/prng.c \
$(srcroot)test/unit/prof_accum.c \
$(srcroot)test/unit/prof_active.c \
$(srcroot)test/unit/prof_gdump.c \
$(srcroot)test/unit/prof_idump.c \
$(srcroot)test/unit/prof_reset.c \
$(srcroot)test/unit/prof_tctx.c \
$(srcroot)test/unit/prof_thread_name.c \
$(srcroot)test/unit/ql.c \
$(srcroot)test/unit/qr.c \
$(srcroot)test/unit/quarantine.c \
$(srcroot)test/unit/rb.c \
$(srcroot)test/unit/retained.c \
$(srcroot)test/unit/rtree.c \
$(srcroot)test/unit/SFMT.c \
$(srcroot)test/unit/size_classes.c \
$(srcroot)test/unit/slab.c \
$(srcroot)test/unit/smoothstep.c \
$(srcroot)test/unit/spin.c \
$(srcroot)test/unit/stats.c \
$(srcroot)test/unit/stats_print.c \
$(srcroot)test/unit/ticker.c \
$(srcroot)test/unit/nstime.c \
$(srcroot)test/unit/tsd.c \
$(srcroot)test/unit/witness.c \
$(srcroot)test/unit/util.c \
$(srcroot)test/unit/zero.c
ifeq (@enable_prof@, 1)
TESTS_UNIT += \
$(srcroot)test/unit/arena_reset_prof.c
endif
TESTS_UNIT_AUX := $(srcroot)test/unit/prof_accum_a.c \
$(srcroot)test/unit/prof_accum_b.c
TESTS_INTEGRATION := $(srcroot)test/integration/aligned_alloc.c \
$(srcroot)test/integration/allocated.c \
$(srcroot)test/integration/extent.c \
$(srcroot)test/integration/mallocx.c \
$(srcroot)test/integration/MALLOCX_ARENA.c \
$(srcroot)test/integration/overflow.c \
$(srcroot)test/integration/mremap.c \
$(srcroot)test/integration/posix_memalign.c \
$(srcroot)test/integration/rallocx.c \
$(srcroot)test/integration/sdallocx.c \
$(srcroot)test/integration/thread_arena.c \
$(srcroot)test/integration/thread_tcache_enabled.c \
$(srcroot)test/integration/xallocx.c
ifeq (@enable_cxx@, 1)
CPP_SRCS := $(srcroot)src/jemalloc_cpp.cpp
TESTS_INTEGRATION_CPP := $(srcroot)test/integration/cpp/basic.cpp
else
CPP_SRCS :=
TESTS_INTEGRATION_CPP :=
ifeq ($(enable_experimental), 1)
TESTS_INTEGRATION += $(srcroot)test/integration/allocm.c \
$(srcroot)test/integration/MALLOCX_ARENA.c \
$(srcroot)test/integration/rallocm.c
endif
TESTS_STRESS := $(srcroot)test/stress/microbench.c
TESTS_STRESS :=
TESTS := $(TESTS_UNIT) $(TESTS_INTEGRATION) $(TESTS_STRESS)
TESTS := $(TESTS_UNIT) $(TESTS_INTEGRATION) $(TESTS_INTEGRATION_CPP) $(TESTS_STRESS)
PRIVATE_NAMESPACE_HDRS := $(objroot)include/jemalloc/internal/private_namespace.h $(objroot)include/jemalloc/internal/private_namespace_jet.h
PRIVATE_NAMESPACE_GEN_HDRS := $(PRIVATE_NAMESPACE_HDRS:%.h=%.gen.h)
C_SYM_OBJS := $(C_SRCS:$(srcroot)%.c=$(objroot)%.sym.$(O))
C_SYMS := $(C_SRCS:$(srcroot)%.c=$(objroot)%.sym)
C_OBJS := $(C_SRCS:$(srcroot)%.c=$(objroot)%.$(O))
CPP_OBJS := $(CPP_SRCS:$(srcroot)%.cpp=$(objroot)%.$(O))
C_PIC_OBJS := $(C_SRCS:$(srcroot)%.c=$(objroot)%.pic.$(O))
CPP_PIC_OBJS := $(CPP_SRCS:$(srcroot)%.cpp=$(objroot)%.pic.$(O))
C_JET_SYM_OBJS := $(C_SRCS:$(srcroot)%.c=$(objroot)%.jet.sym.$(O))
C_JET_SYMS := $(C_SRCS:$(srcroot)%.c=$(objroot)%.jet.sym)
C_JET_OBJS := $(C_SRCS:$(srcroot)%.c=$(objroot)%.jet.$(O))
C_TESTLIB_UNIT_OBJS := $(C_TESTLIB_SRCS:$(srcroot)%.c=$(objroot)%.unit.$(O))
C_TESTLIB_INTEGRATION_OBJS := $(C_TESTLIB_SRCS:$(srcroot)%.c=$(objroot)%.integration.$(O))
@@ -255,28 +157,27 @@ C_TESTLIB_STRESS_OBJS := $(C_TESTLIB_SRCS:$(srcroot)%.c=$(objroot)%.stress.$(O))
C_TESTLIB_OBJS := $(C_TESTLIB_UNIT_OBJS) $(C_TESTLIB_INTEGRATION_OBJS) $(C_UTIL_INTEGRATION_OBJS) $(C_TESTLIB_STRESS_OBJS)
TESTS_UNIT_OBJS := $(TESTS_UNIT:$(srcroot)%.c=$(objroot)%.$(O))
TESTS_UNIT_AUX_OBJS := $(TESTS_UNIT_AUX:$(srcroot)%.c=$(objroot)%.$(O))
TESTS_INTEGRATION_OBJS := $(TESTS_INTEGRATION:$(srcroot)%.c=$(objroot)%.$(O))
TESTS_INTEGRATION_CPP_OBJS := $(TESTS_INTEGRATION_CPP:$(srcroot)%.cpp=$(objroot)%.$(O))
TESTS_STRESS_OBJS := $(TESTS_STRESS:$(srcroot)%.c=$(objroot)%.$(O))
TESTS_OBJS := $(TESTS_UNIT_OBJS) $(TESTS_INTEGRATION_OBJS) $(TESTS_STRESS_OBJS)
TESTS_CPP_OBJS := $(TESTS_INTEGRATION_CPP_OBJS)
TESTS_OBJS := $(TESTS_UNIT_OBJS) $(TESTS_UNIT_AUX_OBJS) $(TESTS_INTEGRATION_OBJS) $(TESTS_STRESS_OBJS)
.PHONY: all dist build_doc_html build_doc_man build_doc
.PHONY: install_bin install_include install_lib
.PHONY: install_doc_html install_doc_man install_doc install
.PHONY: tests check clean distclean relclean
.SECONDARY : $(PRIVATE_NAMESPACE_GEN_HDRS) $(TESTS_OBJS) $(TESTS_CPP_OBJS)
.SECONDARY : $(TESTS_OBJS)
# Default target.
all: build_lib
dist: build_doc
$(objroot)doc/%.html : $(objroot)doc/%.xml $(srcroot)doc/stylesheet.xsl $(objroot)doc/html.xsl
$(srcroot)doc/%.html : $(objroot)doc/%.xml $(srcroot)doc/stylesheet.xsl $(objroot)doc/html.xsl
$(XSLTPROC) -o $@ $(objroot)doc/html.xsl $<
$(objroot)doc/%.3 : $(objroot)doc/%.xml $(srcroot)doc/stylesheet.xsl $(objroot)doc/manpages.xsl
$(srcroot)doc/%.3 : $(objroot)doc/%.xml $(srcroot)doc/stylesheet.xsl $(objroot)doc/manpages.xsl
$(XSLTPROC) -o $@ $(objroot)doc/manpages.xsl $<
build_doc_html: $(DOCS_HTML)
@@ -287,32 +188,18 @@ build_doc: $(DOCS)
# Include generated dependency files.
#
ifdef CC_MM
-include $(C_SYM_OBJS:%.$(O)=%.d)
-include $(C_OBJS:%.$(O)=%.d)
-include $(CPP_OBJS:%.$(O)=%.d)
-include $(C_PIC_OBJS:%.$(O)=%.d)
-include $(CPP_PIC_OBJS:%.$(O)=%.d)
-include $(C_JET_SYM_OBJS:%.$(O)=%.d)
-include $(C_JET_OBJS:%.$(O)=%.d)
-include $(C_TESTLIB_OBJS:%.$(O)=%.d)
-include $(TESTS_OBJS:%.$(O)=%.d)
-include $(TESTS_CPP_OBJS:%.$(O)=%.d)
endif
$(C_SYM_OBJS): $(objroot)src/%.sym.$(O): $(srcroot)src/%.c
$(C_SYM_OBJS): CPPFLAGS += -DJEMALLOC_NO_PRIVATE_NAMESPACE
$(C_SYMS): $(objroot)src/%.sym: $(objroot)src/%.sym.$(O)
$(C_OBJS): $(objroot)src/%.$(O): $(srcroot)src/%.c
$(CPP_OBJS): $(objroot)src/%.$(O): $(srcroot)src/%.cpp
$(C_PIC_OBJS): $(objroot)src/%.pic.$(O): $(srcroot)src/%.c
$(C_PIC_OBJS): CFLAGS += $(PIC_CFLAGS)
$(CPP_PIC_OBJS): $(objroot)src/%.pic.$(O): $(srcroot)src/%.cpp
$(CPP_PIC_OBJS): CXXFLAGS += $(PIC_CFLAGS)
$(C_JET_SYM_OBJS): $(objroot)src/%.jet.sym.$(O): $(srcroot)src/%.c
$(C_JET_SYM_OBJS): CPPFLAGS += -DJEMALLOC_JET -DJEMALLOC_NO_PRIVATE_NAMESPACE
$(C_JET_SYMS): $(objroot)src/%.jet.sym: $(objroot)src/%.jet.sym.$(O)
$(C_JET_OBJS): $(objroot)src/%.jet.$(O): $(srcroot)src/%.c
$(C_JET_OBJS): CPPFLAGS += -DJEMALLOC_JET
$(C_JET_OBJS): CFLAGS += -DJEMALLOC_JET
$(C_TESTLIB_UNIT_OBJS): $(objroot)test/src/%.unit.$(O): $(srcroot)test/src/%.c
$(C_TESTLIB_UNIT_OBJS): CPPFLAGS += -DJEMALLOC_UNIT_TEST
$(C_TESTLIB_INTEGRATION_OBJS): $(objroot)test/src/%.integration.$(O): $(srcroot)test/src/%.c
@@ -322,147 +209,112 @@ $(C_TESTLIB_STRESS_OBJS): $(objroot)test/src/%.stress.$(O): $(srcroot)test/src/%
$(C_TESTLIB_STRESS_OBJS): CPPFLAGS += -DJEMALLOC_STRESS_TEST -DJEMALLOC_STRESS_TESTLIB
$(C_TESTLIB_OBJS): CPPFLAGS += -I$(srcroot)test/include -I$(objroot)test/include
$(TESTS_UNIT_OBJS): CPPFLAGS += -DJEMALLOC_UNIT_TEST
$(TESTS_UNIT_AUX_OBJS): CPPFLAGS += -DJEMALLOC_UNIT_TEST
define make-unit-link-dep
$(1): TESTS_UNIT_LINK_OBJS += $(2)
$(1): $(2)
endef
$(foreach test, $(TESTS_UNIT:$(srcroot)test/unit/%.c=$(objroot)test/unit/%$(EXE)), $(eval $(call make-unit-link-dep,$(test),$(filter $(test:%=%_a.$(O)) $(test:%=%_b.$(O)),$(TESTS_UNIT_AUX_OBJS)))))
$(TESTS_INTEGRATION_OBJS): CPPFLAGS += -DJEMALLOC_INTEGRATION_TEST
$(TESTS_INTEGRATION_CPP_OBJS): CPPFLAGS += -DJEMALLOC_INTEGRATION_CPP_TEST
$(TESTS_STRESS_OBJS): CPPFLAGS += -DJEMALLOC_STRESS_TEST
$(TESTS_OBJS): $(objroot)test/%.$(O): $(srcroot)test/%.c
$(TESTS_CPP_OBJS): $(objroot)test/%.$(O): $(srcroot)test/%.cpp
$(TESTS_OBJS): CPPFLAGS += -I$(srcroot)test/include -I$(objroot)test/include
$(TESTS_CPP_OBJS): CPPFLAGS += -I$(srcroot)test/include -I$(objroot)test/include
ifneq ($(IMPORTLIB),$(SO))
$(CPP_OBJS) $(C_SYM_OBJS) $(C_OBJS) $(C_JET_SYM_OBJS) $(C_JET_OBJS): CPPFLAGS += -DDLLEXPORT
$(C_OBJS): CPPFLAGS += -DDLLEXPORT
endif
# Dependencies.
ifndef CC_MM
# Dependencies.
HEADER_DIRS = $(srcroot)include/jemalloc/internal \
$(objroot)include/jemalloc $(objroot)include/jemalloc/internal
HEADERS = $(filter-out $(PRIVATE_NAMESPACE_HDRS),$(wildcard $(foreach dir,$(HEADER_DIRS),$(dir)/*.h)))
$(C_SYM_OBJS) $(C_OBJS) $(CPP_OBJS) $(C_PIC_OBJS) $(CPP_PIC_OBJS) $(C_JET_SYM_OBJS) $(C_JET_OBJS) $(C_TESTLIB_OBJS) $(TESTS_OBJS) $(TESTS_CPP_OBJS): $(HEADERS)
$(TESTS_OBJS) $(TESTS_CPP_OBJS): $(objroot)test/include/test/jemalloc_test.h
HEADERS = $(wildcard $(foreach dir,$(HEADER_DIRS),$(dir)/*.h))
$(C_OBJS) $(C_PIC_OBJS) $(C_JET_OBJS) $(C_TESTLIB_OBJS) $(TESTS_OBJS): $(HEADERS)
$(TESTS_OBJS): $(objroot)test/unit/jemalloc_test.h
endif
$(C_OBJS) $(CPP_OBJS) $(C_PIC_OBJS) $(CPP_PIC_OBJS) $(C_TESTLIB_INTEGRATION_OBJS) $(C_UTIL_INTEGRATION_OBJS) $(TESTS_INTEGRATION_OBJS) $(TESTS_INTEGRATION_CPP_OBJS): $(objroot)include/jemalloc/internal/private_namespace.h
$(C_JET_OBJS) $(C_TESTLIB_UNIT_OBJS) $(C_TESTLIB_STRESS_OBJS) $(TESTS_UNIT_OBJS) $(TESTS_STRESS_OBJS): $(objroot)include/jemalloc/internal/private_namespace_jet.h
$(C_SYM_OBJS) $(C_OBJS) $(C_PIC_OBJS) $(C_JET_SYM_OBJS) $(C_JET_OBJS) $(C_TESTLIB_OBJS) $(TESTS_OBJS): %.$(O):
$(C_OBJS) $(C_PIC_OBJS) $(C_JET_OBJS) $(C_TESTLIB_OBJS) $(TESTS_OBJS): %.$(O):
@mkdir -p $(@D)
$(CC) $(CFLAGS) -c $(CPPFLAGS) $(CTARGET) $<
ifdef CC_MM
@$(CC) -MM $(CPPFLAGS) -MT $@ -o $(@:%.$(O)=%.d) $<
endif
$(C_SYMS): %.sym:
@mkdir -p $(@D)
$(DUMP_SYMS) $< | $(AWK) -f $(objroot)include/jemalloc/internal/private_symbols.awk > $@
$(C_JET_SYMS): %.sym:
@mkdir -p $(@D)
$(DUMP_SYMS) $< | $(AWK) -f $(objroot)include/jemalloc/internal/private_symbols_jet.awk > $@
$(objroot)include/jemalloc/internal/private_namespace.gen.h: $(C_SYMS)
$(SHELL) $(srcroot)include/jemalloc/internal/private_namespace.sh $^ > $@
$(objroot)include/jemalloc/internal/private_namespace_jet.gen.h: $(C_JET_SYMS)
$(SHELL) $(srcroot)include/jemalloc/internal/private_namespace.sh $^ > $@
%.h: %.gen.h
@if ! `cmp -s $< $@` ; then echo "cp $< $<"; cp $< $@ ; fi
$(CPP_OBJS) $(CPP_PIC_OBJS) $(TESTS_CPP_OBJS): %.$(O):
@mkdir -p $(@D)
$(CXX) $(CXXFLAGS) -c $(CPPFLAGS) $(CTARGET) $<
ifdef CC_MM
@$(CXX) -MM $(CPPFLAGS) -MT $@ -o $(@:%.$(O)=%.d) $<
endif
ifneq ($(SOREV),$(SO))
%.$(SO) : %.$(SOREV)
@mkdir -p $(@D)
ln -sf $(<F) $@
endif
$(objroot)lib/$(LIBJEMALLOC).$(SOREV) : $(if $(PIC_CFLAGS),$(C_PIC_OBJS),$(C_OBJS)) $(if $(PIC_CFLAGS),$(CPP_PIC_OBJS),$(CPP_OBJS))
$(objroot)lib/$(LIBJEMALLOC).$(SOREV) : $(if $(PIC_CFLAGS),$(C_PIC_OBJS),$(C_OBJS))
@mkdir -p $(@D)
$(CC) $(DSO_LDFLAGS) $(call RPATH,$(RPATH_EXTRA)) $(LDTARGET) $+ $(LDFLAGS) $(LIBS) $(EXTRA_LDFLAGS)
$(objroot)lib/$(LIBJEMALLOC)_pic.$(A) : $(C_PIC_OBJS) $(CPP_PIC_OBJS)
$(objroot)lib/$(LIBJEMALLOC).$(A) : $(C_OBJS) $(CPP_OBJS)
$(objroot)lib/$(LIBJEMALLOC)_s.$(A) : $(C_OBJS) $(CPP_OBJS)
$(objroot)lib/$(LIBJEMALLOC)_pic.$(A) : $(C_PIC_OBJS)
$(objroot)lib/$(LIBJEMALLOC).$(A) : $(C_OBJS)
$(objroot)lib/$(LIBJEMALLOC)_s.$(A) : $(C_OBJS)
$(STATIC_LIBS):
@mkdir -p $(@D)
$(AR) $(ARFLAGS)@AROUT@ $+
$(objroot)test/unit/%$(EXE): $(objroot)test/unit/%.$(O) $(C_JET_OBJS) $(C_TESTLIB_UNIT_OBJS)
$(objroot)test/unit/%$(EXE): $(objroot)test/unit/%.$(O) $(TESTS_UNIT_LINK_OBJS) $(C_JET_OBJS) $(C_TESTLIB_UNIT_OBJS)
@mkdir -p $(@D)
$(CC) $(LDTARGET) $(filter %.$(O),$^) $(call RPATH,$(objroot)lib) $(LDFLAGS) $(filter-out -lm,$(LIBS)) $(LM) $(EXTRA_LDFLAGS)
$(CC) $(LDTARGET) $(filter %.$(O),$^) $(call RPATH,$(objroot)lib) $(LDFLAGS) $(filter-out -lm,$(LIBS)) -lm $(EXTRA_LDFLAGS)
$(objroot)test/integration/%$(EXE): $(objroot)test/integration/%.$(O) $(C_TESTLIB_INTEGRATION_OBJS) $(C_UTIL_INTEGRATION_OBJS) $(objroot)lib/$(LIBJEMALLOC).$(IMPORTLIB)
@mkdir -p $(@D)
$(CC) $(TEST_LD_MODE) $(LDTARGET) $(filter %.$(O),$^) $(call RPATH,$(objroot)lib) $(LJEMALLOC) $(LDFLAGS) $(filter-out -lm,$(filter -lrt -lpthread -lstdc++,$(LIBS))) $(LM) $(EXTRA_LDFLAGS)
$(objroot)test/integration/cpp/%$(EXE): $(objroot)test/integration/cpp/%.$(O) $(C_TESTLIB_INTEGRATION_OBJS) $(C_UTIL_INTEGRATION_OBJS) $(objroot)lib/$(LIBJEMALLOC).$(IMPORTLIB)
@mkdir -p $(@D)
$(CXX) $(LDTARGET) $(filter %.$(O),$^) $(call RPATH,$(objroot)lib) $(objroot)lib/$(LIBJEMALLOC).$(IMPORTLIB) $(LDFLAGS) $(filter-out -lm,$(LIBS)) -lm $(EXTRA_LDFLAGS)
$(CC) $(LDTARGET) $(filter %.$(O),$^) $(call RPATH,$(objroot)lib) $(objroot)lib/$(LIBJEMALLOC).$(IMPORTLIB) $(LDFLAGS) $(filter-out -lm,$(filter -lpthread,$(LIBS))) -lm $(EXTRA_LDFLAGS)
$(objroot)test/stress/%$(EXE): $(objroot)test/stress/%.$(O) $(C_JET_OBJS) $(C_TESTLIB_STRESS_OBJS) $(objroot)lib/$(LIBJEMALLOC).$(IMPORTLIB)
@mkdir -p $(@D)
$(CC) $(TEST_LD_MODE) $(LDTARGET) $(filter %.$(O),$^) $(call RPATH,$(objroot)lib) $(objroot)lib/$(LIBJEMALLOC).$(IMPORTLIB) $(LDFLAGS) $(filter-out -lm,$(LIBS)) $(LM) $(EXTRA_LDFLAGS)
$(CC) $(LDTARGET) $(filter %.$(O),$^) $(call RPATH,$(objroot)lib) $(objroot)lib/$(LIBJEMALLOC).$(IMPORTLIB) $(LDFLAGS) $(filter-out -lm,$(LIBS)) -lm $(EXTRA_LDFLAGS)
build_lib_shared: $(DSOS)
build_lib_static: $(STATIC_LIBS)
build_lib: build_lib_shared build_lib_static
install_bin:
$(INSTALL) -d $(BINDIR)
install -d $(BINDIR)
@for b in $(BINS); do \
echo "$(INSTALL) -m 755 $$b $(BINDIR)"; \
$(INSTALL) -m 755 $$b $(BINDIR); \
echo "install -m 755 $$b $(BINDIR)"; \
install -m 755 $$b $(BINDIR); \
done
install_include:
$(INSTALL) -d $(INCLUDEDIR)/jemalloc
install -d $(INCLUDEDIR)/jemalloc
@for h in $(C_HDRS); do \
echo "$(INSTALL) -m 644 $$h $(INCLUDEDIR)/jemalloc"; \
$(INSTALL) -m 644 $$h $(INCLUDEDIR)/jemalloc; \
echo "install -m 644 $$h $(INCLUDEDIR)/jemalloc"; \
install -m 644 $$h $(INCLUDEDIR)/jemalloc; \
done
install_lib_shared: $(DSOS)
$(INSTALL) -d $(LIBDIR)
$(INSTALL) -m 755 $(objroot)lib/$(LIBJEMALLOC).$(SOREV) $(LIBDIR)
install -d $(LIBDIR)
install -m 755 $(objroot)lib/$(LIBJEMALLOC).$(SOREV) $(LIBDIR)
ifneq ($(SOREV),$(SO))
ln -sf $(LIBJEMALLOC).$(SOREV) $(LIBDIR)/$(LIBJEMALLOC).$(SO)
endif
install_lib_static: $(STATIC_LIBS)
$(INSTALL) -d $(LIBDIR)
install -d $(LIBDIR)
@for l in $(STATIC_LIBS); do \
echo "$(INSTALL) -m 755 $$l $(LIBDIR)"; \
$(INSTALL) -m 755 $$l $(LIBDIR); \
echo "install -m 755 $$l $(LIBDIR)"; \
install -m 755 $$l $(LIBDIR); \
done
install_lib_pc: $(PC)
$(INSTALL) -d $(LIBDIR)/pkgconfig
@for l in $(PC); do \
echo "$(INSTALL) -m 644 $$l $(LIBDIR)/pkgconfig"; \
$(INSTALL) -m 644 $$l $(LIBDIR)/pkgconfig; \
done
install_lib: install_lib_shared install_lib_static install_lib_pc
install_lib: install_lib_shared install_lib_static
install_doc_html:
$(INSTALL) -d $(DATADIR)/doc/jemalloc$(install_suffix)
install -d $(DATADIR)/doc/jemalloc$(install_suffix)
@for d in $(DOCS_HTML); do \
echo "$(INSTALL) -m 644 $$d $(DATADIR)/doc/jemalloc$(install_suffix)"; \
$(INSTALL) -m 644 $$d $(DATADIR)/doc/jemalloc$(install_suffix); \
echo "install -m 644 $$d $(DATADIR)/doc/jemalloc$(install_suffix)"; \
install -m 644 $$d $(DATADIR)/doc/jemalloc$(install_suffix); \
done
install_doc_man:
$(INSTALL) -d $(MANDIR)/man3
install -d $(MANDIR)/man3
@for d in $(DOCS_MAN3); do \
echo "$(INSTALL) -m 644 $$d $(MANDIR)/man3"; \
$(INSTALL) -m 644 $$d $(MANDIR)/man3; \
echo "install -m 644 $$d $(MANDIR)/man3"; \
install -m 644 $$d $(MANDIR)/man3; \
done
install_doc: install_doc_html install_doc_man
@@ -470,7 +322,7 @@ install_doc: install_doc_html install_doc_man
install: install_bin install_include install_lib install_doc
tests_unit: $(TESTS_UNIT:$(srcroot)%.c=$(objroot)%$(EXE))
tests_integration: $(TESTS_INTEGRATION:$(srcroot)%.c=$(objroot)%$(EXE)) $(TESTS_INTEGRATION_CPP:$(srcroot)%.cpp=$(objroot)%$(EXE))
tests_integration: $(TESTS_INTEGRATION:$(srcroot)%.c=$(objroot)%$(EXE))
tests_stress: $(TESTS_STRESS:$(srcroot)%.c=$(objroot)%$(EXE))
tests: tests_unit tests_integration tests_stress
@@ -478,61 +330,78 @@ check_unit_dir:
@mkdir -p $(objroot)test/unit
check_integration_dir:
@mkdir -p $(objroot)test/integration
stress_dir:
check_stress_dir:
@mkdir -p $(objroot)test/stress
check_dir: check_unit_dir check_integration_dir
check_dir: check_unit_dir check_integration_dir check_stress_dir
check_unit: tests_unit check_unit_dir
$(SHELL) $(objroot)test/test.sh $(TESTS_UNIT:$(srcroot)%.c=$(objroot)%)
check_integration_prof: tests_integration check_integration_dir
ifeq ($(enable_prof), 1)
$(MALLOC_CONF)="prof:true" $(SHELL) $(objroot)test/test.sh $(TESTS_INTEGRATION:$(srcroot)%.c=$(objroot)%) $(TESTS_INTEGRATION_CPP:$(srcroot)%.cpp=$(objroot)%)
$(MALLOC_CONF)="prof:true,prof_active:false" $(SHELL) $(objroot)test/test.sh $(TESTS_INTEGRATION:$(srcroot)%.c=$(objroot)%) $(TESTS_INTEGRATION_CPP:$(srcroot)%.cpp=$(objroot)%)
endif
check_integration_decay: tests_integration check_integration_dir
$(MALLOC_CONF)="dirty_decay_ms:-1,muzzy_decay_ms:-1" $(SHELL) $(objroot)test/test.sh $(TESTS_INTEGRATION:$(srcroot)%.c=$(objroot)%) $(TESTS_INTEGRATION_CPP:$(srcroot)%.cpp=$(objroot)%)
$(MALLOC_CONF)="dirty_decay_ms:0,muzzy_decay_ms:0" $(SHELL) $(objroot)test/test.sh $(TESTS_INTEGRATION:$(srcroot)%.c=$(objroot)%) $(TESTS_INTEGRATION_CPP:$(srcroot)%.cpp=$(objroot)%)
check_integration: tests_integration check_integration_dir
$(SHELL) $(objroot)test/test.sh $(TESTS_INTEGRATION:$(srcroot)%.c=$(objroot)%) $(TESTS_INTEGRATION_CPP:$(srcroot)%.cpp=$(objroot)%)
stress: tests_stress stress_dir
$(SHELL) $(objroot)test/test.sh $(TESTS_INTEGRATION:$(srcroot)%.c=$(objroot)%)
check_stress: tests_stress check_stress_dir
$(SHELL) $(objroot)test/test.sh $(TESTS_STRESS:$(srcroot)%.c=$(objroot)%)
check: check_unit check_integration check_integration_decay check_integration_prof
check: tests check_dir
$(SHELL) $(objroot)test/test.sh $(TESTS:$(srcroot)%.c=$(objroot)%)
ifeq ($(enable_code_coverage), 1)
coverage_unit: check_unit
$(SHELL) $(srcroot)coverage.sh $(srcroot)src jet $(C_JET_OBJS)
$(SHELL) $(srcroot)coverage.sh $(srcroot)test/src unit $(C_TESTLIB_UNIT_OBJS)
$(SHELL) $(srcroot)coverage.sh $(srcroot)test/unit unit $(TESTS_UNIT_OBJS)
coverage_integration: check_integration
$(SHELL) $(srcroot)coverage.sh $(srcroot)src pic $(C_PIC_OBJS)
$(SHELL) $(srcroot)coverage.sh $(srcroot)src integration $(C_UTIL_INTEGRATION_OBJS)
$(SHELL) $(srcroot)coverage.sh $(srcroot)test/src integration $(C_TESTLIB_INTEGRATION_OBJS)
$(SHELL) $(srcroot)coverage.sh $(srcroot)test/integration integration $(TESTS_INTEGRATION_OBJS)
coverage_stress: check_stress
$(SHELL) $(srcroot)coverage.sh $(srcroot)src pic $(C_PIC_OBJS)
$(SHELL) $(srcroot)coverage.sh $(srcroot)src jet $(C_JET_OBJS)
$(SHELL) $(srcroot)coverage.sh $(srcroot)test/src stress $(C_TESTLIB_STRESS_OBJS)
$(SHELL) $(srcroot)coverage.sh $(srcroot)test/stress stress $(TESTS_STRESS_OBJS)
coverage: check
$(SHELL) $(srcroot)coverage.sh $(srcroot)src pic $(C_PIC_OBJS)
$(SHELL) $(srcroot)coverage.sh $(srcroot)src jet $(C_JET_OBJS)
$(SHELL) $(srcroot)coverage.sh $(srcroot)src integration $(C_UTIL_INTEGRATION_OBJS)
$(SHELL) $(srcroot)coverage.sh $(srcroot)test/src unit $(C_TESTLIB_UNIT_OBJS)
$(SHELL) $(srcroot)coverage.sh $(srcroot)test/src integration $(C_TESTLIB_INTEGRATION_OBJS)
$(SHELL) $(srcroot)coverage.sh $(srcroot)test/src stress $(C_TESTLIB_STRESS_OBJS)
$(SHELL) $(srcroot)coverage.sh $(srcroot)test/unit unit $(TESTS_UNIT_OBJS) $(TESTS_UNIT_AUX_OBJS)
$(SHELL) $(srcroot)coverage.sh $(srcroot)test/integration integration $(TESTS_INTEGRATION_OBJS)
$(SHELL) $(srcroot)coverage.sh $(srcroot)test/stress integration $(TESTS_STRESS_OBJS)
endif
clean:
rm -f $(PRIVATE_NAMESPACE_HDRS)
rm -f $(PRIVATE_NAMESPACE_GEN_HDRS)
rm -f $(C_SYM_OBJS)
rm -f $(C_SYMS)
rm -f $(C_OBJS)
rm -f $(CPP_OBJS)
rm -f $(C_PIC_OBJS)
rm -f $(CPP_PIC_OBJS)
rm -f $(C_JET_SYM_OBJS)
rm -f $(C_JET_SYMS)
rm -f $(C_JET_OBJS)
rm -f $(C_TESTLIB_OBJS)
rm -f $(C_SYM_OBJS:%.$(O)=%.d)
rm -f $(C_OBJS:%.$(O)=%.d)
rm -f $(CPP_OBJS:%.$(O)=%.d)
rm -f $(C_OBJS:%.$(O)=%.gcda)
rm -f $(C_OBJS:%.$(O)=%.gcno)
rm -f $(C_PIC_OBJS:%.$(O)=%.d)
rm -f $(CPP_PIC_OBJS:%.$(O)=%.d)
rm -f $(C_JET_SYM_OBJS:%.$(O)=%.d)
rm -f $(C_PIC_OBJS:%.$(O)=%.gcda)
rm -f $(C_PIC_OBJS:%.$(O)=%.gcno)
rm -f $(C_JET_OBJS:%.$(O)=%.d)
rm -f $(C_JET_OBJS:%.$(O)=%.gcda)
rm -f $(C_JET_OBJS:%.$(O)=%.gcno)
rm -f $(C_TESTLIB_OBJS:%.$(O)=%.d)
rm -f $(C_TESTLIB_OBJS:%.$(O)=%.gcda)
rm -f $(C_TESTLIB_OBJS:%.$(O)=%.gcno)
rm -f $(TESTS_OBJS:%.$(O)=%$(EXE))
rm -f $(TESTS_OBJS)
rm -f $(TESTS_OBJS:%.$(O)=%.d)
rm -f $(TESTS_OBJS:%.$(O)=%.gcda)
rm -f $(TESTS_OBJS:%.$(O)=%.gcno)
rm -f $(TESTS_OBJS:%.$(O)=%.out)
rm -f $(TESTS_CPP_OBJS:%.$(O)=%$(EXE))
rm -f $(TESTS_CPP_OBJS)
rm -f $(TESTS_CPP_OBJS:%.$(O)=%.d)
rm -f $(TESTS_CPP_OBJS:%.$(O)=%.out)
rm -f $(DSOS) $(STATIC_LIBS)
rm -f $(objroot)*.gcov.*
distclean: clean
rm -f $(objroot)bin/jemalloc-config
rm -rf $(objroot)autom4te.cache
rm -f $(objroot)bin/jemalloc.sh
rm -f $(objroot)bin/jeprof
rm -f $(objroot)config.log
rm -f $(objroot)config.status
rm -f $(objroot)config.stamp
@@ -541,7 +410,7 @@ distclean: clean
relclean: distclean
rm -f $(objroot)configure
rm -f $(objroot)VERSION
rm -f $(srcroot)VERSION
rm -f $(DOCS_HTML)
rm -f $(DOCS_MAN3)
+7 -7
View File
@@ -3,12 +3,12 @@ fragmentation avoidance and scalable concurrency support. jemalloc first came
into use as the FreeBSD libc allocator in 2005, and since then it has found its
way into numerous applications that rely on its predictable behavior. In 2010
jemalloc development efforts broadened to include developer support features
such as heap profiling and extensive monitoring/tuning hooks. Modern jemalloc
releases continue to be integrated back into FreeBSD, and therefore versatility
remains critical. Ongoing development efforts trend toward making jemalloc
among the best allocators for a broad range of demanding applications, and
eliminating/mitigating weaknesses that have practical repercussions for real
world applications.
such as heap profiling, Valgrind integration, and extensive monitoring/tuning
hooks. Modern jemalloc releases continue to be integrated back into FreeBSD,
and therefore versatility remains critical. Ongoing development efforts trend
toward making jemalloc among the best allocators for a broad range of demanding
applications, and eliminating/mitigating weaknesses that have practical
repercussions for real world applications.
The COPYING file contains copyright and licensing information.
@@ -17,4 +17,4 @@ jemalloc.
The ChangeLog file contains a brief summary of changes for each release.
URL: http://jemalloc.net/
URL: http://www.canonware.com/jemalloc/
-129
View File
@@ -1,129 +0,0 @@
This document summarizes the common approaches for performance fine tuning with
jemalloc (as of 5.1.0). The default configuration of jemalloc tends to work
reasonably well in practice, and most applications should not have to tune any
options. However, in order to cover a wide range of applications and avoid
pathological cases, the default setting is sometimes kept conservative and
suboptimal, even for many common workloads. When jemalloc is properly tuned for
a specific application / workload, it is common to improve system level metrics
by a few percent, or make favorable trade-offs.
## Notable runtime options for performance tuning
Runtime options can be set via
[malloc_conf](http://jemalloc.net/jemalloc.3.html#tuning).
* [background_thread](http://jemalloc.net/jemalloc.3.html#background_thread)
Enabling jemalloc background threads generally improves the tail latency for
application threads, since unused memory purging is shifted to the dedicated
background threads. In addition, unintended purging delay caused by
application inactivity is avoided with background threads.
Suggested: `background_thread:true` when jemalloc managed threads can be
allowed.
* [metadata_thp](http://jemalloc.net/jemalloc.3.html#opt.metadata_thp)
Allowing jemalloc to utilize transparent huge pages for its internal
metadata usually reduces TLB misses significantly, especially for programs
with large memory footprint and frequent allocation / deallocation
activities. Metadata memory usage may increase due to the use of huge
pages.
Suggested for allocation intensive programs: `metadata_thp:auto` or
`metadata_thp:always`, which is expected to improve CPU utilization at a
small memory cost.
* [dirty_decay_ms](http://jemalloc.net/jemalloc.3.html#opt.dirty_decay_ms) and
[muzzy_decay_ms](http://jemalloc.net/jemalloc.3.html#opt.muzzy_decay_ms)
Decay time determines how fast jemalloc returns unused pages back to the
operating system, and therefore provides a fairly straightforward trade-off
between CPU and memory usage. Shorter decay time purges unused pages faster
to reduces memory usage (usually at the cost of more CPU cycles spent on
purging), and vice versa.
Suggested: tune the values based on the desired trade-offs.
* [narenas](http://jemalloc.net/jemalloc.3.html#opt.narenas)
By default jemalloc uses multiple arenas to reduce internal lock contention.
However high arena count may also increase overall memory fragmentation,
since arenas manage memory independently. When high degree of parallelism
is not expected at the allocator level, lower number of arenas often
improves memory usage.
Suggested: if low parallelism is expected, try lower arena count while
monitoring CPU and memory usage.
* [percpu_arena](http://jemalloc.net/jemalloc.3.html#opt.percpu_arena)
Enable dynamic thread to arena association based on running CPU. This has
the potential to improve locality, e.g. when thread to CPU affinity is
present.
Suggested: try `percpu_arena:percpu` or `percpu_arena:phycpu` if
thread migration between processors is expected to be infrequent.
Examples:
* High resource consumption application, prioritizing CPU utilization:
`background_thread:true,metadata_thp:auto` combined with relaxed decay time
(increased `dirty_decay_ms` and / or `muzzy_decay_ms`,
e.g. `dirty_decay_ms:30000,muzzy_decay_ms:30000`).
* High resource consumption application, prioritizing memory usage:
`background_thread:true` combined with shorter decay time (decreased
`dirty_decay_ms` and / or `muzzy_decay_ms`,
e.g. `dirty_decay_ms:5000,muzzy_decay_ms:5000`), and lower arena count
(e.g. number of CPUs).
* Low resource consumption application:
`narenas:1,lg_tcache_max:13` combined with shorter decay time (decreased
`dirty_decay_ms` and / or `muzzy_decay_ms`,e.g.
`dirty_decay_ms:1000,muzzy_decay_ms:0`).
* Extremely conservative -- minimize memory usage at all costs, only suitable when
allocation activity is very rare:
`narenas:1,tcache:false,dirty_decay_ms:0,muzzy_decay_ms:0`
Note that it is recommended to combine the options with `abort_conf:true` which
aborts immediately on illegal options.
## Beyond runtime options
In addition to the runtime options, there are a number of programmatic ways to
improve application performance with jemalloc.
* [Explicit arenas](http://jemalloc.net/jemalloc.3.html#arenas.create)
Manually created arenas can help performance in various ways, e.g. by
managing locality and contention for specific usages. For example,
applications can explicitly allocate frequently accessed objects from a
dedicated arena with
[mallocx()](http://jemalloc.net/jemalloc.3.html#MALLOCX_ARENA) to improve
locality. In addition, explicit arenas often benefit from individually
tuned options, e.g. relaxed [decay
time](http://jemalloc.net/jemalloc.3.html#arena.i.dirty_decay_ms) if
frequent reuse is expected.
* [Extent hooks](http://jemalloc.net/jemalloc.3.html#arena.i.extent_hooks)
Extent hooks allow customization for managing underlying memory. One use
case for performance purpose is to utilize huge pages -- for example,
[HHVM](https://github.com/facebook/hhvm/blob/master/hphp/util/alloc.cpp)
uses explicit arenas with customized extent hooks to manage 1GB huge pages
for frequently accessed data, which reduces TLB misses significantly.
* [Explicit thread-to-arena
binding](http://jemalloc.net/jemalloc.3.html#thread.arena)
It is common for some threads in an application to have different memory
access / allocation patterns. Threads with heavy workloads often benefit
from explicit binding, e.g. binding very active threads to dedicated arenas
may reduce contention at the allocator level.
+1 -1
View File
@@ -1 +1 @@
5.1.0-0-g0
3.6.0-0-g46c0af68bd248b04df75e4f92d5fb804c3d75340
-83
View File
@@ -1,83 +0,0 @@
#!/bin/sh
usage() {
cat <<EOF
Usage:
@BINDIR@/jemalloc-config <option>
Options:
--help | -h : Print usage.
--version : Print jemalloc version.
--revision : Print shared library revision number.
--config : Print configure options used to build jemalloc.
--prefix : Print installation directory prefix.
--bindir : Print binary installation directory.
--datadir : Print data installation directory.
--includedir : Print include installation directory.
--libdir : Print library installation directory.
--mandir : Print manual page installation directory.
--cc : Print compiler used to build jemalloc.
--cflags : Print compiler flags used to build jemalloc.
--cppflags : Print preprocessor flags used to build jemalloc.
--cxxflags : Print C++ compiler flags used to build jemalloc.
--ldflags : Print library flags used to build jemalloc.
--libs : Print libraries jemalloc was linked against.
EOF
}
prefix="@prefix@"
exec_prefix="@exec_prefix@"
case "$1" in
--help | -h)
usage
exit 0
;;
--version)
echo "@jemalloc_version@"
;;
--revision)
echo "@rev@"
;;
--config)
echo "@CONFIG@"
;;
--prefix)
echo "@PREFIX@"
;;
--bindir)
echo "@BINDIR@"
;;
--datadir)
echo "@DATADIR@"
;;
--includedir)
echo "@INCLUDEDIR@"
;;
--libdir)
echo "@LIBDIR@"
;;
--mandir)
echo "@MANDIR@"
;;
--cc)
echo "@CC@"
;;
--cflags)
echo "@CFLAGS@"
;;
--cppflags)
echo "@CPPFLAGS@"
;;
--cxxflags)
echo "@CXXFLAGS@"
;;
--ldflags)
echo "@LDFLAGS@ @EXTRA_LDFLAGS@"
;;
--libs)
echo "@LIBS@"
;;
*)
usage
exit 1
esac
+219 -471
View File
File diff suppressed because it is too large Load Diff
@@ -1,8 +1,8 @@
#! /bin/sh
# Attempt to guess a canonical system name.
# Copyright 1992-2016 Free Software Foundation, Inc.
# Copyright 1992-2013 Free Software Foundation, Inc.
timestamp='2016-10-02'
timestamp='2013-06-10'
# This file is free software; you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by
@@ -24,12 +24,12 @@ timestamp='2016-10-02'
# program. This Exception is an additional permission under section 7
# of the GNU General Public License, version 3 ("GPLv3").
#
# Originally written by Per Bothner; maintained since 2000 by Ben Elliston.
# Originally written by Per Bothner.
#
# You can get the latest version of this script from:
# http://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.guess
# http://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.guess;hb=HEAD
#
# Please send patches to <config-patches@gnu.org>.
# Please send patches with a ChangeLog entry to config-patches@gnu.org.
me=`echo "$0" | sed -e 's,.*/,,'`
@@ -50,7 +50,7 @@ version="\
GNU config.guess ($timestamp)
Originally written by Per Bothner.
Copyright 1992-2016 Free Software Foundation, Inc.
Copyright 1992-2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE."
@@ -149,7 +149,7 @@ Linux|GNU|GNU/*)
LIBC=gnu
#endif
EOF
eval `$CC_FOR_BUILD -E $dummy.c 2>/dev/null | grep '^LIBC' | sed 's, ,,g'`
eval `$CC_FOR_BUILD -E $dummy.c 2>/dev/null | grep '^LIBC'`
;;
esac
@@ -168,29 +168,19 @@ case "${UNAME_MACHINE}:${UNAME_SYSTEM}:${UNAME_RELEASE}:${UNAME_VERSION}" in
# Note: NetBSD doesn't particularly care about the vendor
# portion of the name. We always set it to "unknown".
sysctl="sysctl -n hw.machine_arch"
UNAME_MACHINE_ARCH=`(uname -p 2>/dev/null || \
/sbin/$sysctl 2>/dev/null || \
/usr/sbin/$sysctl 2>/dev/null || \
echo unknown)`
UNAME_MACHINE_ARCH=`(/sbin/$sysctl 2>/dev/null || \
/usr/sbin/$sysctl 2>/dev/null || echo unknown)`
case "${UNAME_MACHINE_ARCH}" in
armeb) machine=armeb-unknown ;;
arm*) machine=arm-unknown ;;
sh3el) machine=shl-unknown ;;
sh3eb) machine=sh-unknown ;;
sh5el) machine=sh5le-unknown ;;
earmv*)
arch=`echo ${UNAME_MACHINE_ARCH} | sed -e 's,^e\(armv[0-9]\).*$,\1,'`
endian=`echo ${UNAME_MACHINE_ARCH} | sed -ne 's,^.*\(eb\)$,\1,p'`
machine=${arch}${endian}-unknown
;;
*) machine=${UNAME_MACHINE_ARCH}-unknown ;;
esac
# The Operating System including object format, if it has switched
# to ELF recently (or will in the future) and ABI.
# to ELF recently, or will in the future.
case "${UNAME_MACHINE_ARCH}" in
earm*)
os=netbsdelf
;;
arm*|i386|m68k|ns32k|sh3*|sparc|vax)
eval $set_cc_for_build
if echo __ELF__ | $CC_FOR_BUILD -E - 2>/dev/null \
@@ -207,13 +197,6 @@ case "${UNAME_MACHINE}:${UNAME_SYSTEM}:${UNAME_RELEASE}:${UNAME_VERSION}" in
os=netbsd
;;
esac
# Determine ABI tags.
case "${UNAME_MACHINE_ARCH}" in
earm*)
expr='s/^earmv[0-9]/-eabi/;s/eb$//'
abi=`echo ${UNAME_MACHINE_ARCH} | sed -e "$expr"`
;;
esac
# The OS release
# Debian GNU/NetBSD machines have a different userland, and
# thus, need a distinct triplet. However, they do not need
@@ -224,13 +207,13 @@ case "${UNAME_MACHINE}:${UNAME_SYSTEM}:${UNAME_RELEASE}:${UNAME_VERSION}" in
release='-gnu'
;;
*)
release=`echo ${UNAME_RELEASE} | sed -e 's/[-_].*//' | cut -d. -f1,2`
release=`echo ${UNAME_RELEASE}|sed -e 's/[-_].*/\./'`
;;
esac
# Since CPU_TYPE-MANUFACTURER-KERNEL-OPERATING_SYSTEM:
# contains redundant information, the shorter form:
# CPU_TYPE-MANUFACTURER-OPERATING_SYSTEM is used.
echo "${machine}-${os}${release}${abi}"
echo "${machine}-${os}${release}"
exit ;;
*:Bitrig:*:*)
UNAME_MACHINE_ARCH=`arch | sed 's/Bitrig.//'`
@@ -240,10 +223,6 @@ case "${UNAME_MACHINE}:${UNAME_SYSTEM}:${UNAME_RELEASE}:${UNAME_VERSION}" in
UNAME_MACHINE_ARCH=`arch | sed 's/OpenBSD.//'`
echo ${UNAME_MACHINE_ARCH}-unknown-openbsd${UNAME_RELEASE}
exit ;;
*:LibertyBSD:*:*)
UNAME_MACHINE_ARCH=`arch | sed 's/^.*BSD\.//'`
echo ${UNAME_MACHINE_ARCH}-unknown-libertybsd${UNAME_RELEASE}
exit ;;
*:ekkoBSD:*:*)
echo ${UNAME_MACHINE}-unknown-ekkobsd${UNAME_RELEASE}
exit ;;
@@ -256,9 +235,6 @@ case "${UNAME_MACHINE}:${UNAME_SYSTEM}:${UNAME_RELEASE}:${UNAME_VERSION}" in
*:MirBSD:*:*)
echo ${UNAME_MACHINE}-unknown-mirbsd${UNAME_RELEASE}
exit ;;
*:Sortix:*:*)
echo ${UNAME_MACHINE}-unknown-sortix
exit ;;
alpha:OSF1:*:*)
case $UNAME_RELEASE in
*4.0)
@@ -275,42 +251,42 @@ case "${UNAME_MACHINE}:${UNAME_SYSTEM}:${UNAME_RELEASE}:${UNAME_VERSION}" in
ALPHA_CPU_TYPE=`/usr/sbin/psrinfo -v | sed -n -e 's/^ The alpha \(.*\) processor.*$/\1/p' | head -n 1`
case "$ALPHA_CPU_TYPE" in
"EV4 (21064)")
UNAME_MACHINE=alpha ;;
UNAME_MACHINE="alpha" ;;
"EV4.5 (21064)")
UNAME_MACHINE=alpha ;;
UNAME_MACHINE="alpha" ;;
"LCA4 (21066/21068)")
UNAME_MACHINE=alpha ;;
UNAME_MACHINE="alpha" ;;
"EV5 (21164)")
UNAME_MACHINE=alphaev5 ;;
UNAME_MACHINE="alphaev5" ;;
"EV5.6 (21164A)")
UNAME_MACHINE=alphaev56 ;;
UNAME_MACHINE="alphaev56" ;;
"EV5.6 (21164PC)")
UNAME_MACHINE=alphapca56 ;;
UNAME_MACHINE="alphapca56" ;;
"EV5.7 (21164PC)")
UNAME_MACHINE=alphapca57 ;;
UNAME_MACHINE="alphapca57" ;;
"EV6 (21264)")
UNAME_MACHINE=alphaev6 ;;
UNAME_MACHINE="alphaev6" ;;
"EV6.7 (21264A)")
UNAME_MACHINE=alphaev67 ;;
UNAME_MACHINE="alphaev67" ;;
"EV6.8CB (21264C)")
UNAME_MACHINE=alphaev68 ;;
UNAME_MACHINE="alphaev68" ;;
"EV6.8AL (21264B)")
UNAME_MACHINE=alphaev68 ;;
UNAME_MACHINE="alphaev68" ;;
"EV6.8CX (21264D)")
UNAME_MACHINE=alphaev68 ;;
UNAME_MACHINE="alphaev68" ;;
"EV6.9A (21264/EV69A)")
UNAME_MACHINE=alphaev69 ;;
UNAME_MACHINE="alphaev69" ;;
"EV7 (21364)")
UNAME_MACHINE=alphaev7 ;;
UNAME_MACHINE="alphaev7" ;;
"EV7.9 (21364A)")
UNAME_MACHINE=alphaev79 ;;
UNAME_MACHINE="alphaev79" ;;
esac
# A Pn.n version is a patched version.
# A Vn.n version is a released version.
# A Tn.n version is a released field test version.
# A Xn.n version is an unreleased experimental baselevel.
# 1.2 uses "1.2" for uname -r.
echo ${UNAME_MACHINE}-dec-osf`echo ${UNAME_RELEASE} | sed -e 's/^[PVTX]//' | tr ABCDEFGHIJKLMNOPQRSTUVWXYZ abcdefghijklmnopqrstuvwxyz`
echo ${UNAME_MACHINE}-dec-osf`echo ${UNAME_RELEASE} | sed -e 's/^[PVTX]//' | tr 'ABCDEFGHIJKLMNOPQRSTUVWXYZ' 'abcdefghijklmnopqrstuvwxyz'`
# Reset EXIT trap before exiting to avoid spurious non-zero exit code.
exitcode=$?
trap '' 0
@@ -383,16 +359,16 @@ case "${UNAME_MACHINE}:${UNAME_SYSTEM}:${UNAME_RELEASE}:${UNAME_VERSION}" in
exit ;;
i86pc:SunOS:5.*:* | i86xen:SunOS:5.*:*)
eval $set_cc_for_build
SUN_ARCH=i386
SUN_ARCH="i386"
# If there is a compiler, see if it is configured for 64-bit objects.
# Note that the Sun cc does not turn __LP64__ into 1 like gcc does.
# This test works for both compilers.
if [ "$CC_FOR_BUILD" != no_compiler_found ]; then
if [ "$CC_FOR_BUILD" != 'no_compiler_found' ]; then
if (echo '#ifdef __amd64'; echo IS_64BIT_ARCH; echo '#endif') | \
(CCOPTS="" $CC_FOR_BUILD -E - 2>/dev/null) | \
(CCOPTS= $CC_FOR_BUILD -E - 2>/dev/null) | \
grep IS_64BIT_ARCH >/dev/null
then
SUN_ARCH=x86_64
SUN_ARCH="x86_64"
fi
fi
echo ${SUN_ARCH}-pc-solaris2`echo ${UNAME_RELEASE}|sed -e 's/[^.]*//'`
@@ -417,7 +393,7 @@ case "${UNAME_MACHINE}:${UNAME_SYSTEM}:${UNAME_RELEASE}:${UNAME_VERSION}" in
exit ;;
sun*:*:4.2BSD:*)
UNAME_RELEASE=`(sed 1q /etc/motd | awk '{print substr($5,1,3)}') 2>/dev/null`
test "x${UNAME_RELEASE}" = x && UNAME_RELEASE=3
test "x${UNAME_RELEASE}" = "x" && UNAME_RELEASE=3
case "`/bin/arch`" in
sun3)
echo m68k-sun-sunos${UNAME_RELEASE}
@@ -603,9 +579,8 @@ EOF
else
IBM_ARCH=powerpc
fi
if [ -x /usr/bin/lslpp ] ; then
IBM_REV=`/usr/bin/lslpp -Lqc bos.rte.libc |
awk -F: '{ print $3 }' | sed s/[0-9]*$/0/`
if [ -x /usr/bin/oslevel ] ; then
IBM_REV=`/usr/bin/oslevel`
else
IBM_REV=${UNAME_VERSION}.${UNAME_RELEASE}
fi
@@ -642,13 +617,13 @@ EOF
sc_cpu_version=`/usr/bin/getconf SC_CPU_VERSION 2>/dev/null`
sc_kernel_bits=`/usr/bin/getconf SC_KERNEL_BITS 2>/dev/null`
case "${sc_cpu_version}" in
523) HP_ARCH=hppa1.0 ;; # CPU_PA_RISC1_0
528) HP_ARCH=hppa1.1 ;; # CPU_PA_RISC1_1
523) HP_ARCH="hppa1.0" ;; # CPU_PA_RISC1_0
528) HP_ARCH="hppa1.1" ;; # CPU_PA_RISC1_1
532) # CPU_PA_RISC2_0
case "${sc_kernel_bits}" in
32) HP_ARCH=hppa2.0n ;;
64) HP_ARCH=hppa2.0w ;;
'') HP_ARCH=hppa2.0 ;; # HP-UX 10.20
32) HP_ARCH="hppa2.0n" ;;
64) HP_ARCH="hppa2.0w" ;;
'') HP_ARCH="hppa2.0" ;; # HP-UX 10.20
esac ;;
esac
fi
@@ -687,11 +662,11 @@ EOF
exit (0);
}
EOF
(CCOPTS="" $CC_FOR_BUILD -o $dummy $dummy.c 2>/dev/null) && HP_ARCH=`$dummy`
(CCOPTS= $CC_FOR_BUILD -o $dummy $dummy.c 2>/dev/null) && HP_ARCH=`$dummy`
test -z "$HP_ARCH" && HP_ARCH=hppa
fi ;;
esac
if [ ${HP_ARCH} = hppa2.0w ]
if [ ${HP_ARCH} = "hppa2.0w" ]
then
eval $set_cc_for_build
@@ -704,12 +679,12 @@ EOF
# $ CC_FOR_BUILD="cc +DA2.0w" ./config.guess
# => hppa64-hp-hpux11.23
if echo __LP64__ | (CCOPTS="" $CC_FOR_BUILD -E - 2>/dev/null) |
if echo __LP64__ | (CCOPTS= $CC_FOR_BUILD -E - 2>/dev/null) |
grep -q __LP64__
then
HP_ARCH=hppa2.0w
HP_ARCH="hppa2.0w"
else
HP_ARCH=hppa64
HP_ARCH="hppa64"
fi
fi
echo ${HP_ARCH}-hp-hpux${HPUX_REV}
@@ -814,14 +789,14 @@ EOF
echo craynv-cray-unicosmp${UNAME_RELEASE} | sed -e 's/\.[^.]*$/.X/'
exit ;;
F30[01]:UNIX_System_V:*:* | F700:UNIX_System_V:*:*)
FUJITSU_PROC=`uname -m | tr ABCDEFGHIJKLMNOPQRSTUVWXYZ abcdefghijklmnopqrstuvwxyz`
FUJITSU_SYS=`uname -p | tr ABCDEFGHIJKLMNOPQRSTUVWXYZ abcdefghijklmnopqrstuvwxyz | sed -e 's/\///'`
FUJITSU_PROC=`uname -m | tr 'ABCDEFGHIJKLMNOPQRSTUVWXYZ' 'abcdefghijklmnopqrstuvwxyz'`
FUJITSU_SYS=`uname -p | tr 'ABCDEFGHIJKLMNOPQRSTUVWXYZ' 'abcdefghijklmnopqrstuvwxyz' | sed -e 's/\///'`
FUJITSU_REL=`echo ${UNAME_RELEASE} | sed -e 's/ /_/'`
echo "${FUJITSU_PROC}-fujitsu-${FUJITSU_SYS}${FUJITSU_REL}"
exit ;;
5000:UNIX_System_V:4.*:*)
FUJITSU_SYS=`uname -p | tr ABCDEFGHIJKLMNOPQRSTUVWXYZ abcdefghijklmnopqrstuvwxyz | sed -e 's/\///'`
FUJITSU_REL=`echo ${UNAME_RELEASE} | tr ABCDEFGHIJKLMNOPQRSTUVWXYZ abcdefghijklmnopqrstuvwxyz | sed -e 's/ /_/'`
FUJITSU_SYS=`uname -p | tr 'ABCDEFGHIJKLMNOPQRSTUVWXYZ' 'abcdefghijklmnopqrstuvwxyz' | sed -e 's/\///'`
FUJITSU_REL=`echo ${UNAME_RELEASE} | tr 'ABCDEFGHIJKLMNOPQRSTUVWXYZ' 'abcdefghijklmnopqrstuvwxyz' | sed -e 's/ /_/'`
echo "sparc-fujitsu-${FUJITSU_SYS}${FUJITSU_REL}"
exit ;;
i*86:BSD/386:*:* | i*86:BSD/OS:*:* | *:Ascend\ Embedded/OS:*:*)
@@ -851,7 +826,7 @@ EOF
*:MINGW*:*)
echo ${UNAME_MACHINE}-pc-mingw32
exit ;;
*:MSYS*:*)
i*:MSYS*:*)
echo ${UNAME_MACHINE}-pc-msys
exit ;;
i*:windows32*:*)
@@ -903,7 +878,7 @@ EOF
exit ;;
*:GNU/*:*:*)
# other systems with GNU libc and userland
echo ${UNAME_MACHINE}-unknown-`echo ${UNAME_SYSTEM} | sed 's,^[^/]*/,,' | tr "[:upper:]" "[:lower:]"``echo ${UNAME_RELEASE}|sed -e 's/[-(].*//'`-${LIBC}
echo ${UNAME_MACHINE}-unknown-`echo ${UNAME_SYSTEM} | sed 's,^[^/]*/,,' | tr '[A-Z]' '[a-z]'``echo ${UNAME_RELEASE}|sed -e 's/[-(].*//'`-${LIBC}
exit ;;
i*86:Minix:*:*)
echo ${UNAME_MACHINE}-pc-minix
@@ -926,7 +901,7 @@ EOF
EV68*) UNAME_MACHINE=alphaev68 ;;
esac
objdump --private-headers /bin/sh | grep -q ld.so.1
if test "$?" = 0 ; then LIBC=gnulibc1 ; fi
if test "$?" = 0 ; then LIBC="gnulibc1" ; fi
echo ${UNAME_MACHINE}-unknown-linux-${LIBC}
exit ;;
arc:Linux:*:* | arceb:Linux:*:*)
@@ -957,9 +932,6 @@ EOF
crisv32:Linux:*:*)
echo ${UNAME_MACHINE}-axis-linux-${LIBC}
exit ;;
e2k:Linux:*:*)
echo ${UNAME_MACHINE}-unknown-linux-${LIBC}
exit ;;
frv:Linux:*:*)
echo ${UNAME_MACHINE}-unknown-linux-${LIBC}
exit ;;
@@ -972,9 +944,6 @@ EOF
ia64:Linux:*:*)
echo ${UNAME_MACHINE}-unknown-linux-${LIBC}
exit ;;
k1om:Linux:*:*)
echo ${UNAME_MACHINE}-unknown-linux-${LIBC}
exit ;;
m32r*:Linux:*:*)
echo ${UNAME_MACHINE}-unknown-linux-${LIBC}
exit ;;
@@ -1000,13 +969,10 @@ EOF
eval `$CC_FOR_BUILD -E $dummy.c 2>/dev/null | grep '^CPU'`
test x"${CPU}" != x && { echo "${CPU}-unknown-linux-${LIBC}"; exit; }
;;
mips64el:Linux:*:*)
or1k:Linux:*:*)
echo ${UNAME_MACHINE}-unknown-linux-${LIBC}
exit ;;
openrisc*:Linux:*:*)
echo or1k-unknown-linux-${LIBC}
exit ;;
or32:Linux:*:* | or1k*:Linux:*:*)
or32:Linux:*:*)
echo ${UNAME_MACHINE}-unknown-linux-${LIBC}
exit ;;
padre:Linux:*:*)
@@ -1035,9 +1001,6 @@ EOF
ppcle:Linux:*:*)
echo powerpcle-unknown-linux-${LIBC}
exit ;;
riscv32:Linux:*:* | riscv64:Linux:*:*)
echo ${UNAME_MACHINE}-unknown-linux-${LIBC}
exit ;;
s390:Linux:*:* | s390x:Linux:*:*)
echo ${UNAME_MACHINE}-ibm-linux-${LIBC}
exit ;;
@@ -1057,7 +1020,7 @@ EOF
echo ${UNAME_MACHINE}-dec-linux-${LIBC}
exit ;;
x86_64:Linux:*:*)
echo ${UNAME_MACHINE}-pc-linux-${LIBC}
echo ${UNAME_MACHINE}-unknown-linux-${LIBC}
exit ;;
xtensa*:Linux:*:*)
echo ${UNAME_MACHINE}-unknown-linux-${LIBC}
@@ -1136,7 +1099,7 @@ EOF
# uname -m prints for DJGPP always 'pc', but it prints nothing about
# the processor, so we play safe by assuming i586.
# Note: whatever this is, it MUST be the same as what config.sub
# prints for the "djgpp" host, or else GDB configure will decide that
# prints for the "djgpp" host, or else GDB configury will decide that
# this is a cross-build.
echo i586-pc-msdosdjgpp
exit ;;
@@ -1285,9 +1248,6 @@ EOF
SX-8R:SUPER-UX:*:*)
echo sx8r-nec-superux${UNAME_RELEASE}
exit ;;
SX-ACE:SUPER-UX:*:*)
echo sxace-nec-superux${UNAME_RELEASE}
exit ;;
Power*:Rhapsody:*:*)
echo powerpc-apple-rhapsody${UNAME_RELEASE}
exit ;;
@@ -1300,32 +1260,22 @@ EOF
if test "$UNAME_PROCESSOR" = unknown ; then
UNAME_PROCESSOR=powerpc
fi
if test `echo "$UNAME_RELEASE" | sed -e 's/\..*//'` -le 10 ; then
if [ "$CC_FOR_BUILD" != no_compiler_found ]; then
if (echo '#ifdef __LP64__'; echo IS_64BIT_ARCH; echo '#endif') | \
(CCOPTS="" $CC_FOR_BUILD -E - 2>/dev/null) | \
grep IS_64BIT_ARCH >/dev/null
then
case $UNAME_PROCESSOR in
i386) UNAME_PROCESSOR=x86_64 ;;
powerpc) UNAME_PROCESSOR=powerpc64 ;;
esac
fi
if [ "$CC_FOR_BUILD" != 'no_compiler_found' ]; then
if (echo '#ifdef __LP64__'; echo IS_64BIT_ARCH; echo '#endif') | \
(CCOPTS= $CC_FOR_BUILD -E - 2>/dev/null) | \
grep IS_64BIT_ARCH >/dev/null
then
case $UNAME_PROCESSOR in
i386) UNAME_PROCESSOR=x86_64 ;;
powerpc) UNAME_PROCESSOR=powerpc64 ;;
esac
fi
elif test "$UNAME_PROCESSOR" = i386 ; then
# Avoid executing cc on OS X 10.9, as it ships with a stub
# that puts up a graphical alert prompting to install
# developer tools. Any system running Mac OS X 10.7 or
# later (Darwin 11 and later) is required to have a 64-bit
# processor. This is not true of the ARM version of Darwin
# that Apple uses in portable devices.
UNAME_PROCESSOR=x86_64
fi
echo ${UNAME_PROCESSOR}-apple-darwin${UNAME_RELEASE}
exit ;;
*:procnto*:*:* | *:QNX:[0123456789]*:*)
UNAME_PROCESSOR=`uname -p`
if test "$UNAME_PROCESSOR" = x86; then
if test "$UNAME_PROCESSOR" = "x86"; then
UNAME_PROCESSOR=i386
UNAME_MACHINE=pc
fi
@@ -1356,7 +1306,7 @@ EOF
# "uname -m" is not consistent, so use $cputype instead. 386
# is converted to i386 for consistency with other x86
# operating systems.
if test "$cputype" = 386; then
if test "$cputype" = "386"; then
UNAME_MACHINE=i386
else
UNAME_MACHINE="$cputype"
@@ -1398,7 +1348,7 @@ EOF
echo i386-pc-xenix
exit ;;
i*86:skyos:*:*)
echo ${UNAME_MACHINE}-pc-skyos`echo ${UNAME_RELEASE} | sed -e 's/ .*$//'`
echo ${UNAME_MACHINE}-pc-skyos`echo ${UNAME_RELEASE}` | sed -e 's/ .*$//'
exit ;;
i*86:rdos:*:*)
echo ${UNAME_MACHINE}-pc-rdos
@@ -1409,25 +1359,171 @@ EOF
x86_64:VMkernel:*:*)
echo ${UNAME_MACHINE}-unknown-esx
exit ;;
amd64:Isilon\ OneFS:*:*)
echo x86_64-unknown-onefs
exit ;;
esac
eval $set_cc_for_build
cat >$dummy.c <<EOF
#ifdef _SEQUENT_
# include <sys/types.h>
# include <sys/utsname.h>
#endif
main ()
{
#if defined (sony)
#if defined (MIPSEB)
/* BFD wants "bsd" instead of "newsos". Perhaps BFD should be changed,
I don't know.... */
printf ("mips-sony-bsd\n"); exit (0);
#else
#include <sys/param.h>
printf ("m68k-sony-newsos%s\n",
#ifdef NEWSOS4
"4"
#else
""
#endif
); exit (0);
#endif
#endif
#if defined (__arm) && defined (__acorn) && defined (__unix)
printf ("arm-acorn-riscix\n"); exit (0);
#endif
#if defined (hp300) && !defined (hpux)
printf ("m68k-hp-bsd\n"); exit (0);
#endif
#if defined (NeXT)
#if !defined (__ARCHITECTURE__)
#define __ARCHITECTURE__ "m68k"
#endif
int version;
version=`(hostinfo | sed -n 's/.*NeXT Mach \([0-9]*\).*/\1/p') 2>/dev/null`;
if (version < 4)
printf ("%s-next-nextstep%d\n", __ARCHITECTURE__, version);
else
printf ("%s-next-openstep%d\n", __ARCHITECTURE__, version);
exit (0);
#endif
#if defined (MULTIMAX) || defined (n16)
#if defined (UMAXV)
printf ("ns32k-encore-sysv\n"); exit (0);
#else
#if defined (CMU)
printf ("ns32k-encore-mach\n"); exit (0);
#else
printf ("ns32k-encore-bsd\n"); exit (0);
#endif
#endif
#endif
#if defined (__386BSD__)
printf ("i386-pc-bsd\n"); exit (0);
#endif
#if defined (sequent)
#if defined (i386)
printf ("i386-sequent-dynix\n"); exit (0);
#endif
#if defined (ns32000)
printf ("ns32k-sequent-dynix\n"); exit (0);
#endif
#endif
#if defined (_SEQUENT_)
struct utsname un;
uname(&un);
if (strncmp(un.version, "V2", 2) == 0) {
printf ("i386-sequent-ptx2\n"); exit (0);
}
if (strncmp(un.version, "V1", 2) == 0) { /* XXX is V1 correct? */
printf ("i386-sequent-ptx1\n"); exit (0);
}
printf ("i386-sequent-ptx\n"); exit (0);
#endif
#if defined (vax)
# if !defined (ultrix)
# include <sys/param.h>
# if defined (BSD)
# if BSD == 43
printf ("vax-dec-bsd4.3\n"); exit (0);
# else
# if BSD == 199006
printf ("vax-dec-bsd4.3reno\n"); exit (0);
# else
printf ("vax-dec-bsd\n"); exit (0);
# endif
# endif
# else
printf ("vax-dec-bsd\n"); exit (0);
# endif
# else
printf ("vax-dec-ultrix\n"); exit (0);
# endif
#endif
#if defined (alliant) && defined (i860)
printf ("i860-alliant-bsd\n"); exit (0);
#endif
exit (1);
}
EOF
$CC_FOR_BUILD -o $dummy $dummy.c 2>/dev/null && SYSTEM_NAME=`$dummy` &&
{ echo "$SYSTEM_NAME"; exit; }
# Apollos put the system type in the environment.
test -d /usr/apollo && { echo ${ISP}-apollo-${SYSTYPE}; exit; }
# Convex versions that predate uname can use getsysinfo(1)
if [ -x /usr/convex/getsysinfo ]
then
case `getsysinfo -f cpu_type` in
c1*)
echo c1-convex-bsd
exit ;;
c2*)
if getsysinfo -f scalar_acc
then echo c32-convex-bsd
else echo c2-convex-bsd
fi
exit ;;
c34*)
echo c34-convex-bsd
exit ;;
c38*)
echo c38-convex-bsd
exit ;;
c4*)
echo c4-convex-bsd
exit ;;
esac
fi
cat >&2 <<EOF
$0: unable to guess system type
This script (version $timestamp), has failed to recognize the
operating system you are using. If your script is old, overwrite
config.guess and config.sub with the latest versions from:
This script, last modified $timestamp, has failed to recognize
the operating system you are using. It is advised that you
download the most up to date version of the config scripts from
http://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.guess
http://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.guess;hb=HEAD
and
http://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.sub
http://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.sub;hb=HEAD
If $0 has already been updated, send the following data and any
information you think might be pertinent to config-patches@gnu.org to
provide the necessary information to handle your system.
If the version you run ($0) is already up to date, please
send the following data and any information you think might be
pertinent to <config-patches@gnu.org> in order to provide the needed
information to handle your system.
config.guess timestamp = $timestamp
@@ -1,8 +1,8 @@
#! /bin/sh
# Configuration validation subroutine script.
# Copyright 1992-2016 Free Software Foundation, Inc.
# Copyright 1992-2013 Free Software Foundation, Inc.
timestamp='2016-11-04'
timestamp='2013-10-01'
# This file is free software; you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by
@@ -25,7 +25,7 @@ timestamp='2016-11-04'
# of the GNU General Public License, version 3 ("GPLv3").
# Please send patches to <config-patches@gnu.org>.
# Please send patches with a ChangeLog entry to config-patches@gnu.org.
#
# Configuration subroutine to validate and canonicalize a configuration type.
# Supply the specified configuration type as an argument.
@@ -33,7 +33,7 @@ timestamp='2016-11-04'
# Otherwise, we print the canonical config type on stdout and succeed.
# You can get the latest version of this script from:
# http://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.sub
# http://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.sub;hb=HEAD
# This file is supposed to be the same for all GNU packages
# and recognize all the CPU types, system types and aliases
@@ -53,7 +53,8 @@ timestamp='2016-11-04'
me=`echo "$0" | sed -e 's,.*/,,'`
usage="\
Usage: $0 [OPTION] CPU-MFR-OPSYS or ALIAS
Usage: $0 [OPTION] CPU-MFR-OPSYS
$0 [OPTION] ALIAS
Canonicalize a configuration name.
@@ -67,7 +68,7 @@ Report bugs and patches to <config-patches@gnu.org>."
version="\
GNU config.sub ($timestamp)
Copyright 1992-2016 Free Software Foundation, Inc.
Copyright 1992-2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE."
@@ -116,8 +117,8 @@ maybe_os=`echo $1 | sed 's/^\(.*\)-\([^-]*-[^-]*\)$/\2/'`
case $maybe_os in
nto-qnx* | linux-gnu* | linux-android* | linux-dietlibc | linux-newlib* | \
linux-musl* | linux-uclibc* | uclinux-uclibc* | uclinux-gnu* | kfreebsd*-gnu* | \
knetbsd*-gnu* | netbsd*-gnu* | netbsd*-eabi* | \
kopensolaris*-gnu* | cloudabi*-eabi* | \
knetbsd*-gnu* | netbsd*-gnu* | \
kopensolaris*-gnu* | \
storm-chaos* | os2-emx* | rtmk-nova*)
os=-$maybe_os
basic_machine=`echo $1 | sed 's/^\(.*\)-\([^-]*-[^-]*\)$/\1/'`
@@ -254,13 +255,12 @@ case $basic_machine in
| arc | arceb \
| arm | arm[bl]e | arme[lb] | armv[2-8] | armv[3-8][lb] | armv7[arm] \
| avr | avr32 \
| ba \
| be32 | be64 \
| bfin \
| c4x | c8051 | clipper \
| d10v | d30v | dlx | dsp16xx \
| e2k | epiphany \
| fido | fr30 | frv | ft32 \
| epiphany \
| fido | fr30 | frv \
| h8300 | h8500 | hppa | hppa1.[01] | hppa2.0 | hppa2.0[nw] | hppa64 \
| hexagon \
| i370 | i860 | i960 | ia64 \
@@ -283,10 +283,8 @@ case $basic_machine in
| mips64vr5900 | mips64vr5900el \
| mipsisa32 | mipsisa32el \
| mipsisa32r2 | mipsisa32r2el \
| mipsisa32r6 | mipsisa32r6el \
| mipsisa64 | mipsisa64el \
| mipsisa64r2 | mipsisa64r2el \
| mipsisa64r6 | mipsisa64r6el \
| mipsisa64sb1 | mipsisa64sb1el \
| mipsisa64sr71k | mipsisa64sr71kel \
| mipsr5900 | mipsr5900el \
@@ -298,15 +296,14 @@ case $basic_machine in
| nds32 | nds32le | nds32be \
| nios | nios2 | nios2eb | nios2el \
| ns16k | ns32k \
| open8 | or1k | or1knd | or32 \
| open8 \
| or1k | or32 \
| pdp10 | pdp11 | pj | pjl \
| powerpc | powerpc64 | powerpc64le | powerpcle \
| pru \
| pyramid \
| riscv32 | riscv64 \
| rl78 | rx \
| score \
| sh | sh[1234] | sh[24]a | sh[24]aeb | sh[23]e | sh[234]eb | sheb | shbe | shle | sh[1234]le | sh3ele \
| sh | sh[1234] | sh[24]a | sh[24]aeb | sh[23]e | sh[34]eb | sheb | shbe | shle | sh[1234]le | sh3ele \
| sh64 | sh64le \
| sparc | sparc64 | sparc64b | sparc64v | sparc86x | sparclet | sparclite \
| sparcv8 | sparcv9 | sparcv9b | sparcv9v \
@@ -314,7 +311,6 @@ case $basic_machine in
| tahoe | tic4x | tic54x | tic55x | tic6x | tic80 | tron \
| ubicom32 \
| v850 | v850e | v850e1 | v850e2 | v850es | v850e2v3 \
| visium \
| we32k \
| x86 | xc16x | xstormy16 | xtensa \
| z8k | z80)
@@ -329,9 +325,6 @@ case $basic_machine in
c6x)
basic_machine=tic6x-unknown
;;
leon|leon[3-9])
basic_machine=sparc-$basic_machine
;;
m6811 | m68hc11 | m6812 | m68hc12 | m68hcs12x | nvptx | picochip)
basic_machine=$basic_machine-unknown
os=-none
@@ -377,13 +370,12 @@ case $basic_machine in
| alphapca5[67]-* | alpha64pca5[67]-* | arc-* | arceb-* \
| arm-* | armbe-* | armle-* | armeb-* | armv*-* \
| avr-* | avr32-* \
| ba-* \
| be32-* | be64-* \
| bfin-* | bs2000-* \
| c[123]* | c30-* | [cjt]90-* | c4x-* \
| c8051-* | clipper-* | craynv-* | cydra-* \
| d10v-* | d30v-* | dlx-* \
| e2k-* | elxsi-* \
| elxsi-* \
| f30[01]-* | f700-* | fido-* | fr30-* | frv-* | fx80-* \
| h8300-* | h8500-* \
| hppa-* | hppa1.[01]-* | hppa2.0-* | hppa2.0[nw]-* | hppa64-* \
@@ -410,10 +402,8 @@ case $basic_machine in
| mips64vr5900-* | mips64vr5900el-* \
| mipsisa32-* | mipsisa32el-* \
| mipsisa32r2-* | mipsisa32r2el-* \
| mipsisa32r6-* | mipsisa32r6el-* \
| mipsisa64-* | mipsisa64el-* \
| mipsisa64r2-* | mipsisa64r2el-* \
| mipsisa64r6-* | mipsisa64r6el-* \
| mipsisa64sb1-* | mipsisa64sb1el-* \
| mipsisa64sr71k-* | mipsisa64sr71kel-* \
| mipsr5900-* | mipsr5900el-* \
@@ -425,19 +415,16 @@ case $basic_machine in
| nios-* | nios2-* | nios2eb-* | nios2el-* \
| none-* | np1-* | ns16k-* | ns32k-* \
| open8-* \
| or1k*-* \
| orion-* \
| pdp10-* | pdp11-* | pj-* | pjl-* | pn-* | power-* \
| powerpc-* | powerpc64-* | powerpc64le-* | powerpcle-* \
| pru-* \
| pyramid-* \
| riscv32-* | riscv64-* \
| rl78-* | romp-* | rs6000-* | rx-* \
| sh-* | sh[1234]-* | sh[24]a-* | sh[24]aeb-* | sh[23]e-* | sh[34]eb-* | sheb-* | shbe-* \
| shle-* | sh[1234]le-* | sh3ele-* | sh64-* | sh64le-* \
| sparc-* | sparc64-* | sparc64b-* | sparc64v-* | sparc86x-* | sparclet-* \
| sparclite-* \
| sparcv8-* | sparcv9-* | sparcv9b-* | sparcv9v-* | sv1-* | sx*-* \
| sparcv8-* | sparcv9-* | sparcv9b-* | sparcv9v-* | sv1-* | sx?-* \
| tahoe-* \
| tic30-* | tic4x-* | tic54x-* | tic55x-* | tic6x-* | tic80-* \
| tile*-* \
@@ -445,7 +432,6 @@ case $basic_machine in
| ubicom32-* \
| v850-* | v850e-* | v850e1-* | v850es-* | v850e2-* | v850e2v3-* \
| vax-* \
| visium-* \
| we32k-* \
| x86-* | x86_64-* | xc16x-* | xps100-* \
| xstormy16-* | xtensa*-* \
@@ -522,9 +508,6 @@ case $basic_machine in
basic_machine=i386-pc
os=-aros
;;
asmjs)
basic_machine=asmjs-unknown
;;
aux)
basic_machine=m68k-apple
os=-aux
@@ -645,14 +628,6 @@ case $basic_machine in
basic_machine=m68k-bull
os=-sysv3
;;
e500v[12])
basic_machine=powerpc-unknown
os=$os"spe"
;;
e500v[12]-*)
basic_machine=powerpc-`echo $basic_machine | sed 's/^[^-]*-//'`
os=$os"spe"
;;
ebmon29k)
basic_machine=a29k-amd
os=-ebmon
@@ -794,9 +769,6 @@ case $basic_machine in
basic_machine=m68k-isi
os=-sysv
;;
leon-*|leon[3-9]-*)
basic_machine=sparc-`echo $basic_machine | sed 's/-.*//'`
;;
m68knommu)
basic_machine=m68k-unknown
os=-linux
@@ -852,10 +824,6 @@ case $basic_machine in
basic_machine=powerpc-unknown
os=-morphos
;;
moxiebox)
basic_machine=moxie-unknown
os=-moxiebox
;;
msdos)
basic_machine=i386-pc
os=-msdos
@@ -1032,7 +1000,7 @@ case $basic_machine in
ppc-* | ppcbe-*)
basic_machine=powerpc-`echo $basic_machine | sed 's/^[^-]*-//'`
;;
ppcle | powerpclittle)
ppcle | powerpclittle | ppc-le | powerpc-little)
basic_machine=powerpcle-unknown
;;
ppcle-* | powerpclittle-*)
@@ -1042,7 +1010,7 @@ case $basic_machine in
;;
ppc64-*) basic_machine=powerpc64-`echo $basic_machine | sed 's/^[^-]*-//'`
;;
ppc64le | powerpc64little)
ppc64le | powerpc64little | ppc64-le | powerpc64-little)
basic_machine=powerpc64le-unknown
;;
ppc64le-* | powerpc64little-*)
@@ -1388,28 +1356,27 @@ case $os in
| -hpux* | -unos* | -osf* | -luna* | -dgux* | -auroraux* | -solaris* \
| -sym* | -kopensolaris* | -plan9* \
| -amigaos* | -amigados* | -msdos* | -newsos* | -unicos* | -aof* \
| -aos* | -aros* | -cloudabi* | -sortix* \
| -aos* | -aros* \
| -nindy* | -vxsim* | -vxworks* | -ebmon* | -hms* | -mvs* \
| -clix* | -riscos* | -uniplus* | -iris* | -rtu* | -xenix* \
| -hiux* | -386bsd* | -knetbsd* | -mirbsd* | -netbsd* \
| -bitrig* | -openbsd* | -solidbsd* | -libertybsd* \
| -bitrig* | -openbsd* | -solidbsd* \
| -ekkobsd* | -kfreebsd* | -freebsd* | -riscix* | -lynxos* \
| -bosx* | -nextstep* | -cxux* | -aout* | -elf* | -oabi* \
| -ptx* | -coff* | -ecoff* | -winnt* | -domain* | -vsta* \
| -udi* | -eabi* | -lites* | -ieee* | -go32* | -aux* \
| -chorusos* | -chorusrdb* | -cegcc* \
| -cygwin* | -msys* | -pe* | -psos* | -moss* | -proelf* | -rtems* \
| -midipix* | -mingw32* | -mingw64* | -linux-gnu* | -linux-android* \
| -mingw32* | -mingw64* | -linux-gnu* | -linux-android* \
| -linux-newlib* | -linux-musl* | -linux-uclibc* \
| -uxpv* | -beos* | -mpeix* | -udk* | -moxiebox* \
| -uxpv* | -beos* | -mpeix* | -udk* \
| -interix* | -uwin* | -mks* | -rhapsody* | -darwin* | -opened* \
| -openstep* | -oskit* | -conix* | -pw32* | -nonstopux* \
| -storm-chaos* | -tops10* | -tenex* | -tops20* | -its* \
| -os2* | -vos* | -palmos* | -uclinux* | -nucleus* \
| -morphos* | -superux* | -rtmk* | -rtmk-nova* | -windiss* \
| -powermax* | -dnix* | -nx6 | -nx7 | -sei* | -dragonfly* \
| -skyos* | -haiku* | -rdos* | -toppers* | -drops* | -es* \
| -onefs* | -tirtos* | -phoenix* | -fuchsia*)
| -skyos* | -haiku* | -rdos* | -toppers* | -drops* | -es*)
# Remember, each alternative MUST END IN *, to match a version number.
;;
-qnx*)
@@ -1541,8 +1508,6 @@ case $os in
;;
-nacl*)
;;
-ios)
;;
-none)
;;
*)
@@ -1629,6 +1594,9 @@ case $basic_machine in
mips*-*)
os=-elf
;;
or1k-*)
os=-elf
;;
or32-*)
os=-coff
;;
+1073 -5831
View File
File diff suppressed because it is too large Load Diff
+463 -1270
View File
File diff suppressed because it is too large Load Diff
Vendored Executable
+16
View File
@@ -0,0 +1,16 @@
#!/bin/sh
set -e
objdir=$1
suffix=$2
shift 2
objs=$@
gcov -b -p -f -o "${objdir}" ${objs}
# Move gcov outputs so that subsequent gcov invocations won't clobber results
# for the same sources with different compilation flags.
for f in `find . -maxdepth 1 -type f -name '*.gcov'` ; do
mv "${f}" "${f}.${suffix}"
done
-1
View File
@@ -1,5 +1,4 @@
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
<xsl:import href="@XSLROOT@/html/docbook.xsl"/>
<xsl:import href="@abs_srcroot@doc/stylesheet.xsl"/>
<xsl:output method="xml" encoding="utf-8"/>
</xsl:stylesheet>
+1630
View File
File diff suppressed because it is too large Load Diff
+1508
View File
File diff suppressed because one or more lines are too long
+732 -1766
View File
File diff suppressed because it is too large Load Diff
+2 -5
View File
@@ -1,10 +1,7 @@
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
<xsl:param name="funcsynopsis.style">ansi</xsl:param>
<xsl:param name="function.parens" select="0"/>
<xsl:template match="function">
<xsl:call-template name="inline.monoseq"/>
</xsl:template>
<xsl:param name="function.parens" select="1"/>
<xsl:template match="mallctl">
<quote><xsl:call-template name="inline.monoseq"/></quote>
"<xsl:call-template name="inline.monoseq"/>"
</xsl:template>
</xsl:stylesheet>
File diff suppressed because it is too large Load Diff
-94
View File
@@ -1,94 +0,0 @@
#ifndef JEMALLOC_INTERNAL_ARENA_EXTERNS_H
#define JEMALLOC_INTERNAL_ARENA_EXTERNS_H
#include "jemalloc/internal/bin.h"
#include "jemalloc/internal/extent_dss.h"
#include "jemalloc/internal/pages.h"
#include "jemalloc/internal/size_classes.h"
#include "jemalloc/internal/stats.h"
extern ssize_t opt_dirty_decay_ms;
extern ssize_t opt_muzzy_decay_ms;
extern percpu_arena_mode_t opt_percpu_arena;
extern const char *percpu_arena_mode_names[];
extern const uint64_t h_steps[SMOOTHSTEP_NSTEPS];
extern malloc_mutex_t arenas_lock;
void arena_basic_stats_merge(tsdn_t *tsdn, arena_t *arena,
unsigned *nthreads, const char **dss, ssize_t *dirty_decay_ms,
ssize_t *muzzy_decay_ms, size_t *nactive, size_t *ndirty, size_t *nmuzzy);
void arena_stats_merge(tsdn_t *tsdn, arena_t *arena, unsigned *nthreads,
const char **dss, ssize_t *dirty_decay_ms, ssize_t *muzzy_decay_ms,
size_t *nactive, size_t *ndirty, size_t *nmuzzy, arena_stats_t *astats,
bin_stats_t *bstats, arena_stats_large_t *lstats);
void arena_extents_dirty_dalloc(tsdn_t *tsdn, arena_t *arena,
extent_hooks_t **r_extent_hooks, extent_t *extent);
#ifdef JEMALLOC_JET
size_t arena_slab_regind(extent_t *slab, szind_t binind, const void *ptr);
#endif
extent_t *arena_extent_alloc_large(tsdn_t *tsdn, arena_t *arena,
size_t usize, size_t alignment, bool *zero);
void arena_extent_dalloc_large_prep(tsdn_t *tsdn, arena_t *arena,
extent_t *extent);
void arena_extent_ralloc_large_shrink(tsdn_t *tsdn, arena_t *arena,
extent_t *extent, size_t oldsize);
void arena_extent_ralloc_large_expand(tsdn_t *tsdn, arena_t *arena,
extent_t *extent, size_t oldsize);
ssize_t arena_dirty_decay_ms_get(arena_t *arena);
bool arena_dirty_decay_ms_set(tsdn_t *tsdn, arena_t *arena, ssize_t decay_ms);
ssize_t arena_muzzy_decay_ms_get(arena_t *arena);
bool arena_muzzy_decay_ms_set(tsdn_t *tsdn, arena_t *arena, ssize_t decay_ms);
void arena_decay(tsdn_t *tsdn, arena_t *arena, bool is_background_thread,
bool all);
void arena_reset(tsd_t *tsd, arena_t *arena);
void arena_destroy(tsd_t *tsd, arena_t *arena);
void arena_tcache_fill_small(tsdn_t *tsdn, arena_t *arena, tcache_t *tcache,
cache_bin_t *tbin, szind_t binind, uint64_t prof_accumbytes);
void arena_alloc_junk_small(void *ptr, const bin_info_t *bin_info,
bool zero);
typedef void (arena_dalloc_junk_small_t)(void *, const bin_info_t *);
extern arena_dalloc_junk_small_t *JET_MUTABLE arena_dalloc_junk_small;
void *arena_malloc_hard(tsdn_t *tsdn, arena_t *arena, size_t size,
szind_t ind, bool zero);
void *arena_palloc(tsdn_t *tsdn, arena_t *arena, size_t usize,
size_t alignment, bool zero, tcache_t *tcache);
void arena_prof_promote(tsdn_t *tsdn, const void *ptr, size_t usize);
void arena_dalloc_promoted(tsdn_t *tsdn, void *ptr, tcache_t *tcache,
bool slow_path);
void arena_dalloc_bin_junked_locked(tsdn_t *tsdn, arena_t *arena,
extent_t *extent, void *ptr);
void arena_dalloc_small(tsdn_t *tsdn, void *ptr);
bool arena_ralloc_no_move(tsdn_t *tsdn, void *ptr, size_t oldsize, size_t size,
size_t extra, bool zero);
void *arena_ralloc(tsdn_t *tsdn, arena_t *arena, void *ptr, size_t oldsize,
size_t size, size_t alignment, bool zero, tcache_t *tcache);
dss_prec_t arena_dss_prec_get(arena_t *arena);
bool arena_dss_prec_set(arena_t *arena, dss_prec_t dss_prec);
ssize_t arena_dirty_decay_ms_default_get(void);
bool arena_dirty_decay_ms_default_set(ssize_t decay_ms);
ssize_t arena_muzzy_decay_ms_default_get(void);
bool arena_muzzy_decay_ms_default_set(ssize_t decay_ms);
bool arena_retain_grow_limit_get_set(tsd_t *tsd, arena_t *arena,
size_t *old_limit, size_t *new_limit);
unsigned arena_nthreads_get(arena_t *arena, bool internal);
void arena_nthreads_inc(arena_t *arena, bool internal);
void arena_nthreads_dec(arena_t *arena, bool internal);
size_t arena_extent_sn_next(arena_t *arena);
arena_t *arena_new(tsdn_t *tsdn, unsigned ind, extent_hooks_t *extent_hooks);
void arena_boot(void);
void arena_prefork0(tsdn_t *tsdn, arena_t *arena);
void arena_prefork1(tsdn_t *tsdn, arena_t *arena);
void arena_prefork2(tsdn_t *tsdn, arena_t *arena);
void arena_prefork3(tsdn_t *tsdn, arena_t *arena);
void arena_prefork4(tsdn_t *tsdn, arena_t *arena);
void arena_prefork5(tsdn_t *tsdn, arena_t *arena);
void arena_prefork6(tsdn_t *tsdn, arena_t *arena);
void arena_prefork7(tsdn_t *tsdn, arena_t *arena);
void arena_postfork_parent(tsdn_t *tsdn, arena_t *arena);
void arena_postfork_child(tsdn_t *tsdn, arena_t *arena);
#endif /* JEMALLOC_INTERNAL_ARENA_EXTERNS_H */
@@ -1,57 +0,0 @@
#ifndef JEMALLOC_INTERNAL_ARENA_INLINES_A_H
#define JEMALLOC_INTERNAL_ARENA_INLINES_A_H
static inline unsigned
arena_ind_get(const arena_t *arena) {
return base_ind_get(arena->base);
}
static inline void
arena_internal_add(arena_t *arena, size_t size) {
atomic_fetch_add_zu(&arena->stats.internal, size, ATOMIC_RELAXED);
}
static inline void
arena_internal_sub(arena_t *arena, size_t size) {
atomic_fetch_sub_zu(&arena->stats.internal, size, ATOMIC_RELAXED);
}
static inline size_t
arena_internal_get(arena_t *arena) {
return atomic_load_zu(&arena->stats.internal, ATOMIC_RELAXED);
}
static inline bool
arena_prof_accum(tsdn_t *tsdn, arena_t *arena, uint64_t accumbytes) {
cassert(config_prof);
if (likely(prof_interval == 0 || !prof_active_get_unlocked())) {
return false;
}
return prof_accum_add(tsdn, &arena->prof_accum, accumbytes);
}
static inline void
percpu_arena_update(tsd_t *tsd, unsigned cpu) {
assert(have_percpu_arena);
arena_t *oldarena = tsd_arena_get(tsd);
assert(oldarena != NULL);
unsigned oldind = arena_ind_get(oldarena);
if (oldind != cpu) {
unsigned newind = cpu;
arena_t *newarena = arena_get(tsd_tsdn(tsd), newind, true);
assert(newarena != NULL);
/* Set new arena/tcache associations. */
arena_migrate(tsd, oldind, newind);
tcache_t *tcache = tcache_get(tsd);
if (tcache != NULL) {
tcache_arena_reassociate(tsd_tsdn(tsd), tcache,
newarena);
}
}
}
#endif /* JEMALLOC_INTERNAL_ARENA_INLINES_A_H */
@@ -1,354 +0,0 @@
#ifndef JEMALLOC_INTERNAL_ARENA_INLINES_B_H
#define JEMALLOC_INTERNAL_ARENA_INLINES_B_H
#include "jemalloc/internal/jemalloc_internal_types.h"
#include "jemalloc/internal/mutex.h"
#include "jemalloc/internal/rtree.h"
#include "jemalloc/internal/size_classes.h"
#include "jemalloc/internal/sz.h"
#include "jemalloc/internal/ticker.h"
JEMALLOC_ALWAYS_INLINE prof_tctx_t *
arena_prof_tctx_get(tsdn_t *tsdn, const void *ptr, alloc_ctx_t *alloc_ctx) {
cassert(config_prof);
assert(ptr != NULL);
/* Static check. */
if (alloc_ctx == NULL) {
const extent_t *extent = iealloc(tsdn, ptr);
if (unlikely(!extent_slab_get(extent))) {
return large_prof_tctx_get(tsdn, extent);
}
} else {
if (unlikely(!alloc_ctx->slab)) {
return large_prof_tctx_get(tsdn, iealloc(tsdn, ptr));
}
}
return (prof_tctx_t *)(uintptr_t)1U;
}
JEMALLOC_ALWAYS_INLINE void
arena_prof_tctx_set(tsdn_t *tsdn, const void *ptr, UNUSED size_t usize,
alloc_ctx_t *alloc_ctx, prof_tctx_t *tctx) {
cassert(config_prof);
assert(ptr != NULL);
/* Static check. */
if (alloc_ctx == NULL) {
extent_t *extent = iealloc(tsdn, ptr);
if (unlikely(!extent_slab_get(extent))) {
large_prof_tctx_set(tsdn, extent, tctx);
}
} else {
if (unlikely(!alloc_ctx->slab)) {
large_prof_tctx_set(tsdn, iealloc(tsdn, ptr), tctx);
}
}
}
static inline void
arena_prof_tctx_reset(tsdn_t *tsdn, const void *ptr, UNUSED prof_tctx_t *tctx) {
cassert(config_prof);
assert(ptr != NULL);
extent_t *extent = iealloc(tsdn, ptr);
assert(!extent_slab_get(extent));
large_prof_tctx_reset(tsdn, extent);
}
JEMALLOC_ALWAYS_INLINE void
arena_decay_ticks(tsdn_t *tsdn, arena_t *arena, unsigned nticks) {
tsd_t *tsd;
ticker_t *decay_ticker;
if (unlikely(tsdn_null(tsdn))) {
return;
}
tsd = tsdn_tsd(tsdn);
decay_ticker = decay_ticker_get(tsd, arena_ind_get(arena));
if (unlikely(decay_ticker == NULL)) {
return;
}
if (unlikely(ticker_ticks(decay_ticker, nticks))) {
arena_decay(tsdn, arena, false, false);
}
}
JEMALLOC_ALWAYS_INLINE void
arena_decay_tick(tsdn_t *tsdn, arena_t *arena) {
malloc_mutex_assert_not_owner(tsdn, &arena->decay_dirty.mtx);
malloc_mutex_assert_not_owner(tsdn, &arena->decay_muzzy.mtx);
arena_decay_ticks(tsdn, arena, 1);
}
JEMALLOC_ALWAYS_INLINE void *
arena_malloc(tsdn_t *tsdn, arena_t *arena, size_t size, szind_t ind, bool zero,
tcache_t *tcache, bool slow_path) {
assert(!tsdn_null(tsdn) || tcache == NULL);
assert(size != 0);
if (likely(tcache != NULL)) {
if (likely(size <= SMALL_MAXCLASS)) {
return tcache_alloc_small(tsdn_tsd(tsdn), arena,
tcache, size, ind, zero, slow_path);
}
if (likely(size <= tcache_maxclass)) {
return tcache_alloc_large(tsdn_tsd(tsdn), arena,
tcache, size, ind, zero, slow_path);
}
/* (size > tcache_maxclass) case falls through. */
assert(size > tcache_maxclass);
}
return arena_malloc_hard(tsdn, arena, size, ind, zero);
}
JEMALLOC_ALWAYS_INLINE arena_t *
arena_aalloc(tsdn_t *tsdn, const void *ptr) {
return extent_arena_get(iealloc(tsdn, ptr));
}
JEMALLOC_ALWAYS_INLINE size_t
arena_salloc(tsdn_t *tsdn, const void *ptr) {
assert(ptr != NULL);
rtree_ctx_t rtree_ctx_fallback;
rtree_ctx_t *rtree_ctx = tsdn_rtree_ctx(tsdn, &rtree_ctx_fallback);
szind_t szind = rtree_szind_read(tsdn, &extents_rtree, rtree_ctx,
(uintptr_t)ptr, true);
assert(szind != NSIZES);
return sz_index2size(szind);
}
JEMALLOC_ALWAYS_INLINE size_t
arena_vsalloc(tsdn_t *tsdn, const void *ptr) {
/*
* Return 0 if ptr is not within an extent managed by jemalloc. This
* function has two extra costs relative to isalloc():
* - The rtree calls cannot claim to be dependent lookups, which induces
* rtree lookup load dependencies.
* - The lookup may fail, so there is an extra branch to check for
* failure.
*/
rtree_ctx_t rtree_ctx_fallback;
rtree_ctx_t *rtree_ctx = tsdn_rtree_ctx(tsdn, &rtree_ctx_fallback);
extent_t *extent;
szind_t szind;
if (rtree_extent_szind_read(tsdn, &extents_rtree, rtree_ctx,
(uintptr_t)ptr, false, &extent, &szind)) {
return 0;
}
if (extent == NULL) {
return 0;
}
assert(extent_state_get(extent) == extent_state_active);
/* Only slab members should be looked up via interior pointers. */
assert(extent_addr_get(extent) == ptr || extent_slab_get(extent));
assert(szind != NSIZES);
return sz_index2size(szind);
}
static inline void
arena_dalloc_no_tcache(tsdn_t *tsdn, void *ptr) {
assert(ptr != NULL);
rtree_ctx_t rtree_ctx_fallback;
rtree_ctx_t *rtree_ctx = tsdn_rtree_ctx(tsdn, &rtree_ctx_fallback);
szind_t szind;
bool slab;
rtree_szind_slab_read(tsdn, &extents_rtree, rtree_ctx, (uintptr_t)ptr,
true, &szind, &slab);
if (config_debug) {
extent_t *extent = rtree_extent_read(tsdn, &extents_rtree,
rtree_ctx, (uintptr_t)ptr, true);
assert(szind == extent_szind_get(extent));
assert(szind < NSIZES);
assert(slab == extent_slab_get(extent));
}
if (likely(slab)) {
/* Small allocation. */
arena_dalloc_small(tsdn, ptr);
} else {
extent_t *extent = iealloc(tsdn, ptr);
large_dalloc(tsdn, extent);
}
}
JEMALLOC_ALWAYS_INLINE void
arena_dalloc(tsdn_t *tsdn, void *ptr, tcache_t *tcache,
alloc_ctx_t *alloc_ctx, bool slow_path) {
assert(!tsdn_null(tsdn) || tcache == NULL);
assert(ptr != NULL);
if (unlikely(tcache == NULL)) {
arena_dalloc_no_tcache(tsdn, ptr);
return;
}
szind_t szind;
bool slab;
rtree_ctx_t *rtree_ctx;
if (alloc_ctx != NULL) {
szind = alloc_ctx->szind;
slab = alloc_ctx->slab;
assert(szind != NSIZES);
} else {
rtree_ctx = tsd_rtree_ctx(tsdn_tsd(tsdn));
rtree_szind_slab_read(tsdn, &extents_rtree, rtree_ctx,
(uintptr_t)ptr, true, &szind, &slab);
}
if (config_debug) {
rtree_ctx = tsd_rtree_ctx(tsdn_tsd(tsdn));
extent_t *extent = rtree_extent_read(tsdn, &extents_rtree,
rtree_ctx, (uintptr_t)ptr, true);
assert(szind == extent_szind_get(extent));
assert(szind < NSIZES);
assert(slab == extent_slab_get(extent));
}
if (likely(slab)) {
/* Small allocation. */
tcache_dalloc_small(tsdn_tsd(tsdn), tcache, ptr, szind,
slow_path);
} else {
if (szind < nhbins) {
if (config_prof && unlikely(szind < NBINS)) {
arena_dalloc_promoted(tsdn, ptr, tcache,
slow_path);
} else {
tcache_dalloc_large(tsdn_tsd(tsdn), tcache, ptr,
szind, slow_path);
}
} else {
extent_t *extent = iealloc(tsdn, ptr);
large_dalloc(tsdn, extent);
}
}
}
static inline void
arena_sdalloc_no_tcache(tsdn_t *tsdn, void *ptr, size_t size) {
assert(ptr != NULL);
assert(size <= LARGE_MAXCLASS);
szind_t szind;
bool slab;
if (!config_prof || !opt_prof) {
/*
* There is no risk of being confused by a promoted sampled
* object, so base szind and slab on the given size.
*/
szind = sz_size2index(size);
slab = (szind < NBINS);
}
if ((config_prof && opt_prof) || config_debug) {
rtree_ctx_t rtree_ctx_fallback;
rtree_ctx_t *rtree_ctx = tsdn_rtree_ctx(tsdn,
&rtree_ctx_fallback);
rtree_szind_slab_read(tsdn, &extents_rtree, rtree_ctx,
(uintptr_t)ptr, true, &szind, &slab);
assert(szind == sz_size2index(size));
assert((config_prof && opt_prof) || slab == (szind < NBINS));
if (config_debug) {
extent_t *extent = rtree_extent_read(tsdn,
&extents_rtree, rtree_ctx, (uintptr_t)ptr, true);
assert(szind == extent_szind_get(extent));
assert(slab == extent_slab_get(extent));
}
}
if (likely(slab)) {
/* Small allocation. */
arena_dalloc_small(tsdn, ptr);
} else {
extent_t *extent = iealloc(tsdn, ptr);
large_dalloc(tsdn, extent);
}
}
JEMALLOC_ALWAYS_INLINE void
arena_sdalloc(tsdn_t *tsdn, void *ptr, size_t size, tcache_t *tcache,
alloc_ctx_t *alloc_ctx, bool slow_path) {
assert(!tsdn_null(tsdn) || tcache == NULL);
assert(ptr != NULL);
assert(size <= LARGE_MAXCLASS);
if (unlikely(tcache == NULL)) {
arena_sdalloc_no_tcache(tsdn, ptr, size);
return;
}
szind_t szind;
bool slab;
UNUSED alloc_ctx_t local_ctx;
if (config_prof && opt_prof) {
if (alloc_ctx == NULL) {
/* Uncommon case and should be a static check. */
rtree_ctx_t rtree_ctx_fallback;
rtree_ctx_t *rtree_ctx = tsdn_rtree_ctx(tsdn,
&rtree_ctx_fallback);
rtree_szind_slab_read(tsdn, &extents_rtree, rtree_ctx,
(uintptr_t)ptr, true, &local_ctx.szind,
&local_ctx.slab);
assert(local_ctx.szind == sz_size2index(size));
alloc_ctx = &local_ctx;
}
slab = alloc_ctx->slab;
szind = alloc_ctx->szind;
} else {
/*
* There is no risk of being confused by a promoted sampled
* object, so base szind and slab on the given size.
*/
szind = sz_size2index(size);
slab = (szind < NBINS);
}
if (config_debug) {
rtree_ctx_t *rtree_ctx = tsd_rtree_ctx(tsdn_tsd(tsdn));
rtree_szind_slab_read(tsdn, &extents_rtree, rtree_ctx,
(uintptr_t)ptr, true, &szind, &slab);
extent_t *extent = rtree_extent_read(tsdn,
&extents_rtree, rtree_ctx, (uintptr_t)ptr, true);
assert(szind == extent_szind_get(extent));
assert(slab == extent_slab_get(extent));
}
if (likely(slab)) {
/* Small allocation. */
tcache_dalloc_small(tsdn_tsd(tsdn), tcache, ptr, szind,
slow_path);
} else {
if (szind < nhbins) {
if (config_prof && unlikely(szind < NBINS)) {
arena_dalloc_promoted(tsdn, ptr, tcache,
slow_path);
} else {
tcache_dalloc_large(tsdn_tsd(tsdn),
tcache, ptr, szind, slow_path);
}
} else {
extent_t *extent = iealloc(tsdn, ptr);
large_dalloc(tsdn, extent);
}
}
}
#endif /* JEMALLOC_INTERNAL_ARENA_INLINES_B_H */
-237
View File
@@ -1,237 +0,0 @@
#ifndef JEMALLOC_INTERNAL_ARENA_STATS_H
#define JEMALLOC_INTERNAL_ARENA_STATS_H
#include "jemalloc/internal/atomic.h"
#include "jemalloc/internal/mutex.h"
#include "jemalloc/internal/mutex_prof.h"
#include "jemalloc/internal/size_classes.h"
/*
* In those architectures that support 64-bit atomics, we use atomic updates for
* our 64-bit values. Otherwise, we use a plain uint64_t and synchronize
* externally.
*/
#ifdef JEMALLOC_ATOMIC_U64
typedef atomic_u64_t arena_stats_u64_t;
#else
/* Must hold the arena stats mutex while reading atomically. */
typedef uint64_t arena_stats_u64_t;
#endif
typedef struct arena_stats_large_s arena_stats_large_t;
struct arena_stats_large_s {
/*
* Total number of allocation/deallocation requests served directly by
* the arena.
*/
arena_stats_u64_t nmalloc;
arena_stats_u64_t ndalloc;
/*
* Number of allocation requests that correspond to this size class.
* This includes requests served by tcache, though tcache only
* periodically merges into this counter.
*/
arena_stats_u64_t nrequests; /* Partially derived. */
/* Current number of allocations of this size class. */
size_t curlextents; /* Derived. */
};
typedef struct arena_stats_decay_s arena_stats_decay_t;
struct arena_stats_decay_s {
/* Total number of purge sweeps. */
arena_stats_u64_t npurge;
/* Total number of madvise calls made. */
arena_stats_u64_t nmadvise;
/* Total number of pages purged. */
arena_stats_u64_t purged;
};
/*
* Arena stats. Note that fields marked "derived" are not directly maintained
* within the arena code; rather their values are derived during stats merge
* requests.
*/
typedef struct arena_stats_s arena_stats_t;
struct arena_stats_s {
#ifndef JEMALLOC_ATOMIC_U64
malloc_mutex_t mtx;
#endif
/* Number of bytes currently mapped, excluding retained memory. */
atomic_zu_t mapped; /* Partially derived. */
/*
* Number of unused virtual memory bytes currently retained. Retained
* bytes are technically mapped (though always decommitted or purged),
* but they are excluded from the mapped statistic (above).
*/
atomic_zu_t retained; /* Derived. */
arena_stats_decay_t decay_dirty;
arena_stats_decay_t decay_muzzy;
atomic_zu_t base; /* Derived. */
atomic_zu_t internal;
atomic_zu_t resident; /* Derived. */
atomic_zu_t metadata_thp;
atomic_zu_t allocated_large; /* Derived. */
arena_stats_u64_t nmalloc_large; /* Derived. */
arena_stats_u64_t ndalloc_large; /* Derived. */
arena_stats_u64_t nrequests_large; /* Derived. */
/* Number of bytes cached in tcache associated with this arena. */
atomic_zu_t tcache_bytes; /* Derived. */
mutex_prof_data_t mutex_prof_data[mutex_prof_num_arena_mutexes];
/* One element for each large size class. */
arena_stats_large_t lstats[NSIZES - NBINS];
/* Arena uptime. */
nstime_t uptime;
};
static inline bool
arena_stats_init(UNUSED tsdn_t *tsdn, arena_stats_t *arena_stats) {
if (config_debug) {
for (size_t i = 0; i < sizeof(arena_stats_t); i++) {
assert(((char *)arena_stats)[i] == 0);
}
}
#ifndef JEMALLOC_ATOMIC_U64
if (malloc_mutex_init(&arena_stats->mtx, "arena_stats",
WITNESS_RANK_ARENA_STATS, malloc_mutex_rank_exclusive)) {
return true;
}
#endif
/* Memory is zeroed, so there is no need to clear stats. */
return false;
}
static inline void
arena_stats_lock(tsdn_t *tsdn, arena_stats_t *arena_stats) {
#ifndef JEMALLOC_ATOMIC_U64
malloc_mutex_lock(tsdn, &arena_stats->mtx);
#endif
}
static inline void
arena_stats_unlock(tsdn_t *tsdn, arena_stats_t *arena_stats) {
#ifndef JEMALLOC_ATOMIC_U64
malloc_mutex_unlock(tsdn, &arena_stats->mtx);
#endif
}
static inline uint64_t
arena_stats_read_u64(tsdn_t *tsdn, arena_stats_t *arena_stats,
arena_stats_u64_t *p) {
#ifdef JEMALLOC_ATOMIC_U64
return atomic_load_u64(p, ATOMIC_RELAXED);
#else
malloc_mutex_assert_owner(tsdn, &arena_stats->mtx);
return *p;
#endif
}
static inline void
arena_stats_add_u64(tsdn_t *tsdn, arena_stats_t *arena_stats,
arena_stats_u64_t *p, uint64_t x) {
#ifdef JEMALLOC_ATOMIC_U64
atomic_fetch_add_u64(p, x, ATOMIC_RELAXED);
#else
malloc_mutex_assert_owner(tsdn, &arena_stats->mtx);
*p += x;
#endif
}
UNUSED static inline void
arena_stats_sub_u64(tsdn_t *tsdn, arena_stats_t *arena_stats,
arena_stats_u64_t *p, uint64_t x) {
#ifdef JEMALLOC_ATOMIC_U64
UNUSED uint64_t r = atomic_fetch_sub_u64(p, x, ATOMIC_RELAXED);
assert(r - x <= r);
#else
malloc_mutex_assert_owner(tsdn, &arena_stats->mtx);
*p -= x;
assert(*p + x >= *p);
#endif
}
/*
* Non-atomically sets *dst += src. *dst needs external synchronization.
* This lets us avoid the cost of a fetch_add when its unnecessary (note that
* the types here are atomic).
*/
static inline void
arena_stats_accum_u64(arena_stats_u64_t *dst, uint64_t src) {
#ifdef JEMALLOC_ATOMIC_U64
uint64_t cur_dst = atomic_load_u64(dst, ATOMIC_RELAXED);
atomic_store_u64(dst, src + cur_dst, ATOMIC_RELAXED);
#else
*dst += src;
#endif
}
static inline size_t
arena_stats_read_zu(tsdn_t *tsdn, arena_stats_t *arena_stats, atomic_zu_t *p) {
#ifdef JEMALLOC_ATOMIC_U64
return atomic_load_zu(p, ATOMIC_RELAXED);
#else
malloc_mutex_assert_owner(tsdn, &arena_stats->mtx);
return atomic_load_zu(p, ATOMIC_RELAXED);
#endif
}
static inline void
arena_stats_add_zu(tsdn_t *tsdn, arena_stats_t *arena_stats, atomic_zu_t *p,
size_t x) {
#ifdef JEMALLOC_ATOMIC_U64
atomic_fetch_add_zu(p, x, ATOMIC_RELAXED);
#else
malloc_mutex_assert_owner(tsdn, &arena_stats->mtx);
size_t cur = atomic_load_zu(p, ATOMIC_RELAXED);
atomic_store_zu(p, cur + x, ATOMIC_RELAXED);
#endif
}
static inline void
arena_stats_sub_zu(tsdn_t *tsdn, arena_stats_t *arena_stats, atomic_zu_t *p,
size_t x) {
#ifdef JEMALLOC_ATOMIC_U64
UNUSED size_t r = atomic_fetch_sub_zu(p, x, ATOMIC_RELAXED);
assert(r - x <= r);
#else
malloc_mutex_assert_owner(tsdn, &arena_stats->mtx);
size_t cur = atomic_load_zu(p, ATOMIC_RELAXED);
atomic_store_zu(p, cur - x, ATOMIC_RELAXED);
#endif
}
/* Like the _u64 variant, needs an externally synchronized *dst. */
static inline void
arena_stats_accum_zu(atomic_zu_t *dst, size_t src) {
size_t cur_dst = atomic_load_zu(dst, ATOMIC_RELAXED);
atomic_store_zu(dst, src + cur_dst, ATOMIC_RELAXED);
}
static inline void
arena_stats_large_nrequests_add(tsdn_t *tsdn, arena_stats_t *arena_stats,
szind_t szind, uint64_t nrequests) {
arena_stats_lock(tsdn, arena_stats);
arena_stats_add_u64(tsdn, arena_stats, &arena_stats->lstats[szind -
NBINS].nrequests, nrequests);
arena_stats_unlock(tsdn, arena_stats);
}
static inline void
arena_stats_mapped_add(tsdn_t *tsdn, arena_stats_t *arena_stats, size_t size) {
arena_stats_lock(tsdn, arena_stats);
arena_stats_add_zu(tsdn, arena_stats, &arena_stats->mapped, size);
arena_stats_unlock(tsdn, arena_stats);
}
#endif /* JEMALLOC_INTERNAL_ARENA_STATS_H */
@@ -1,11 +0,0 @@
#ifndef JEMALLOC_INTERNAL_ARENA_STRUCTS_A_H
#define JEMALLOC_INTERNAL_ARENA_STRUCTS_A_H
#include "jemalloc/internal/bitmap.h"
struct arena_slab_data_s {
/* Per region allocated/deallocated bitmap. */
bitmap_t bitmap[BITMAP_GROUPS_MAX];
};
#endif /* JEMALLOC_INTERNAL_ARENA_STRUCTS_A_H */
@@ -1,229 +0,0 @@
#ifndef JEMALLOC_INTERNAL_ARENA_STRUCTS_B_H
#define JEMALLOC_INTERNAL_ARENA_STRUCTS_B_H
#include "jemalloc/internal/arena_stats.h"
#include "jemalloc/internal/atomic.h"
#include "jemalloc/internal/bin.h"
#include "jemalloc/internal/bitmap.h"
#include "jemalloc/internal/extent_dss.h"
#include "jemalloc/internal/jemalloc_internal_types.h"
#include "jemalloc/internal/mutex.h"
#include "jemalloc/internal/nstime.h"
#include "jemalloc/internal/ql.h"
#include "jemalloc/internal/size_classes.h"
#include "jemalloc/internal/smoothstep.h"
#include "jemalloc/internal/ticker.h"
struct arena_decay_s {
/* Synchronizes all non-atomic fields. */
malloc_mutex_t mtx;
/*
* True if a thread is currently purging the extents associated with
* this decay structure.
*/
bool purging;
/*
* Approximate time in milliseconds from the creation of a set of unused
* dirty pages until an equivalent set of unused dirty pages is purged
* and/or reused.
*/
atomic_zd_t time_ms;
/* time / SMOOTHSTEP_NSTEPS. */
nstime_t interval;
/*
* Time at which the current decay interval logically started. We do
* not actually advance to a new epoch until sometime after it starts
* because of scheduling and computation delays, and it is even possible
* to completely skip epochs. In all cases, during epoch advancement we
* merge all relevant activity into the most recently recorded epoch.
*/
nstime_t epoch;
/* Deadline randomness generator. */
uint64_t jitter_state;
/*
* Deadline for current epoch. This is the sum of interval and per
* epoch jitter which is a uniform random variable in [0..interval).
* Epochs always advance by precise multiples of interval, but we
* randomize the deadline to reduce the likelihood of arenas purging in
* lockstep.
*/
nstime_t deadline;
/*
* Number of unpurged pages at beginning of current epoch. During epoch
* advancement we use the delta between arena->decay_*.nunpurged and
* extents_npages_get(&arena->extents_*) to determine how many dirty
* pages, if any, were generated.
*/
size_t nunpurged;
/*
* Trailing log of how many unused dirty pages were generated during
* each of the past SMOOTHSTEP_NSTEPS decay epochs, where the last
* element is the most recent epoch. Corresponding epoch times are
* relative to epoch.
*/
size_t backlog[SMOOTHSTEP_NSTEPS];
/*
* Pointer to associated stats. These stats are embedded directly in
* the arena's stats due to how stats structures are shared between the
* arena and ctl code.
*
* Synchronization: Same as associated arena's stats field. */
arena_stats_decay_t *stats;
/* Peak number of pages in associated extents. Used for debug only. */
uint64_t ceil_npages;
};
struct arena_s {
/*
* Number of threads currently assigned to this arena. Each thread has
* two distinct assignments, one for application-serving allocation, and
* the other for internal metadata allocation. Internal metadata must
* not be allocated from arenas explicitly created via the arenas.create
* mallctl, because the arena.<i>.reset mallctl indiscriminately
* discards all allocations for the affected arena.
*
* 0: Application allocation.
* 1: Internal metadata allocation.
*
* Synchronization: atomic.
*/
atomic_u_t nthreads[2];
/*
* When percpu_arena is enabled, to amortize the cost of reading /
* updating the current CPU id, track the most recent thread accessing
* this arena, and only read CPU if there is a mismatch.
*/
tsdn_t *last_thd;
/* Synchronization: internal. */
arena_stats_t stats;
/*
* Lists of tcaches and cache_bin_array_descriptors for extant threads
* associated with this arena. Stats from these are merged
* incrementally, and at exit if opt_stats_print is enabled.
*
* Synchronization: tcache_ql_mtx.
*/
ql_head(tcache_t) tcache_ql;
ql_head(cache_bin_array_descriptor_t) cache_bin_array_descriptor_ql;
malloc_mutex_t tcache_ql_mtx;
/* Synchronization: internal. */
prof_accum_t prof_accum;
uint64_t prof_accumbytes;
/*
* PRNG state for cache index randomization of large allocation base
* pointers.
*
* Synchronization: atomic.
*/
atomic_zu_t offset_state;
/*
* Extent serial number generator state.
*
* Synchronization: atomic.
*/
atomic_zu_t extent_sn_next;
/*
* Represents a dss_prec_t, but atomically.
*
* Synchronization: atomic.
*/
atomic_u_t dss_prec;
/*
* Number of pages in active extents.
*
* Synchronization: atomic.
*/
atomic_zu_t nactive;
/*
* Extant large allocations.
*
* Synchronization: large_mtx.
*/
extent_list_t large;
/* Synchronizes all large allocation/update/deallocation. */
malloc_mutex_t large_mtx;
/*
* Collections of extents that were previously allocated. These are
* used when allocating extents, in an attempt to re-use address space.
*
* Synchronization: internal.
*/
extents_t extents_dirty;
extents_t extents_muzzy;
extents_t extents_retained;
/*
* Decay-based purging state, responsible for scheduling extent state
* transitions.
*
* Synchronization: internal.
*/
arena_decay_t decay_dirty; /* dirty --> muzzy */
arena_decay_t decay_muzzy; /* muzzy --> retained */
/*
* Next extent size class in a growing series to use when satisfying a
* request via the extent hooks (only if opt_retain). This limits the
* number of disjoint virtual memory ranges so that extent merging can
* be effective even if multiple arenas' extent allocation requests are
* highly interleaved.
*
* retain_grow_limit is the max allowed size ind to expand (unless the
* required size is greater). Default is no limit, and controlled
* through mallctl only.
*
* Synchronization: extent_grow_mtx
*/
pszind_t extent_grow_next;
pszind_t retain_grow_limit;
malloc_mutex_t extent_grow_mtx;
/*
* Available extent structures that were allocated via
* base_alloc_extent().
*
* Synchronization: extent_avail_mtx.
*/
extent_tree_t extent_avail;
malloc_mutex_t extent_avail_mtx;
/*
* bins is used to store heaps of free regions.
*
* Synchronization: internal.
*/
bin_t bins[NBINS];
/*
* Base allocator, from which arena metadata are allocated.
*
* Synchronization: internal.
*/
base_t *base;
/* Used to determine uptime. Read-only after initialization. */
nstime_t create_time;
};
/* Used in conjunction with tsd for fast arena-related context lookup. */
struct arena_tdata_s {
ticker_t decay_ticker;
};
/* Used to pass rtree lookup context down the path. */
struct alloc_ctx_s {
szind_t szind;
bool slab;
};
#endif /* JEMALLOC_INTERNAL_ARENA_STRUCTS_B_H */
-43
View File
@@ -1,43 +0,0 @@
#ifndef JEMALLOC_INTERNAL_ARENA_TYPES_H
#define JEMALLOC_INTERNAL_ARENA_TYPES_H
/* Maximum number of regions in one slab. */
#define LG_SLAB_MAXREGS (LG_PAGE - LG_TINY_MIN)
#define SLAB_MAXREGS (1U << LG_SLAB_MAXREGS)
/* Default decay times in milliseconds. */
#define DIRTY_DECAY_MS_DEFAULT ZD(10 * 1000)
#define MUZZY_DECAY_MS_DEFAULT ZD(10 * 1000)
/* Number of event ticks between time checks. */
#define DECAY_NTICKS_PER_UPDATE 1000
typedef struct arena_slab_data_s arena_slab_data_t;
typedef struct arena_decay_s arena_decay_t;
typedef struct arena_s arena_t;
typedef struct arena_tdata_s arena_tdata_t;
typedef struct alloc_ctx_s alloc_ctx_t;
typedef enum {
percpu_arena_mode_names_base = 0, /* Used for options processing. */
/*
* *_uninit are used only during bootstrapping, and must correspond
* to initialized variant plus percpu_arena_mode_enabled_base.
*/
percpu_arena_uninit = 0,
per_phycpu_arena_uninit = 1,
/* All non-disabled modes must come after percpu_arena_disabled. */
percpu_arena_disabled = 2,
percpu_arena_mode_names_limit = 3, /* Used for options processing. */
percpu_arena_mode_enabled_base = 3,
percpu_arena = 3,
per_phycpu_arena = 4 /* Hyper threads share arena. */
} percpu_arena_mode_t;
#define PERCPU_ARENA_ENABLED(m) ((m) >= percpu_arena_mode_enabled_base)
#define PERCPU_ARENA_DEFAULT percpu_arena_disabled
#endif /* JEMALLOC_INTERNAL_ARENA_TYPES_H */
-56
View File
@@ -1,56 +0,0 @@
#include "jemalloc/internal/malloc_io.h"
#include "jemalloc/internal/util.h"
/*
* Define a custom assert() in order to reduce the chances of deadlock during
* assertion failure.
*/
#ifndef assert
#define assert(e) do { \
if (unlikely(config_debug && !(e))) { \
malloc_printf( \
"<jemalloc>: %s:%d: Failed assertion: \"%s\"\n", \
__FILE__, __LINE__, #e); \
abort(); \
} \
} while (0)
#endif
#ifndef not_reached
#define not_reached() do { \
if (config_debug) { \
malloc_printf( \
"<jemalloc>: %s:%d: Unreachable code reached\n", \
__FILE__, __LINE__); \
abort(); \
} \
unreachable(); \
} while (0)
#endif
#ifndef not_implemented
#define not_implemented() do { \
if (config_debug) { \
malloc_printf("<jemalloc>: %s:%d: Not implemented\n", \
__FILE__, __LINE__); \
abort(); \
} \
} while (0)
#endif
#ifndef assert_not_implemented
#define assert_not_implemented(e) do { \
if (unlikely(config_debug && !(e))) { \
not_implemented(); \
} \
} while (0)
#endif
/* Use to assert a particular configuration, e.g., cassert(config_debug). */
#ifndef cassert
#define cassert(c) do { \
if (unlikely(!(c))) { \
not_reached(); \
} \
} while (0)
#endif
+288 -61
View File
@@ -1,77 +1,304 @@
#ifndef JEMALLOC_INTERNAL_ATOMIC_H
#define JEMALLOC_INTERNAL_ATOMIC_H
/******************************************************************************/
#ifdef JEMALLOC_H_TYPES
#define ATOMIC_INLINE static inline
#endif /* JEMALLOC_H_TYPES */
/******************************************************************************/
#ifdef JEMALLOC_H_STRUCTS
#if defined(JEMALLOC_GCC_ATOMIC_ATOMICS)
# include "jemalloc/internal/atomic_gcc_atomic.h"
#elif defined(JEMALLOC_GCC_SYNC_ATOMICS)
# include "jemalloc/internal/atomic_gcc_sync.h"
#elif defined(_MSC_VER)
# include "jemalloc/internal/atomic_msvc.h"
#elif defined(JEMALLOC_C11_ATOMICS)
# include "jemalloc/internal/atomic_c11.h"
#else
# error "Don't have atomics implemented on this platform."
#endif /* JEMALLOC_H_STRUCTS */
/******************************************************************************/
#ifdef JEMALLOC_H_EXTERNS
#define atomic_read_uint64(p) atomic_add_uint64(p, 0)
#define atomic_read_uint32(p) atomic_add_uint32(p, 0)
#define atomic_read_z(p) atomic_add_z(p, 0)
#define atomic_read_u(p) atomic_add_u(p, 0)
#endif /* JEMALLOC_H_EXTERNS */
/******************************************************************************/
#ifdef JEMALLOC_H_INLINES
#ifndef JEMALLOC_ENABLE_INLINE
uint64_t atomic_add_uint64(uint64_t *p, uint64_t x);
uint64_t atomic_sub_uint64(uint64_t *p, uint64_t x);
uint32_t atomic_add_uint32(uint32_t *p, uint32_t x);
uint32_t atomic_sub_uint32(uint32_t *p, uint32_t x);
size_t atomic_add_z(size_t *p, size_t x);
size_t atomic_sub_z(size_t *p, size_t x);
unsigned atomic_add_u(unsigned *p, unsigned x);
unsigned atomic_sub_u(unsigned *p, unsigned x);
#endif
/*
* This header gives more or less a backport of C11 atomics. The user can write
* JEMALLOC_GENERATE_ATOMICS(type, short_type, lg_sizeof_type); to generate
* counterparts of the C11 atomic functions for type, as so:
* JEMALLOC_GENERATE_ATOMICS(int *, pi, 3);
* and then write things like:
* int *some_ptr;
* atomic_pi_t atomic_ptr_to_int;
* atomic_store_pi(&atomic_ptr_to_int, some_ptr, ATOMIC_RELAXED);
* int *prev_value = atomic_exchange_pi(&ptr_to_int, NULL, ATOMIC_ACQ_REL);
* assert(some_ptr == prev_value);
* and expect things to work in the obvious way.
*
* Also included (with naming differences to avoid conflicts with the standard
* library):
* atomic_fence(atomic_memory_order_t) (mimics C11's atomic_thread_fence).
* ATOMIC_INIT (mimics C11's ATOMIC_VAR_INIT).
*/
/*
* Pure convenience, so that we don't have to type "atomic_memory_order_"
* quite so often.
*/
#define ATOMIC_RELAXED atomic_memory_order_relaxed
#define ATOMIC_ACQUIRE atomic_memory_order_acquire
#define ATOMIC_RELEASE atomic_memory_order_release
#define ATOMIC_ACQ_REL atomic_memory_order_acq_rel
#define ATOMIC_SEQ_CST atomic_memory_order_seq_cst
/*
* Not all platforms have 64-bit atomics. If we do, this #define exposes that
* fact.
*/
#if (defined(JEMALLOC_ENABLE_INLINE) || defined(JEMALLOC_ATOMIC_C_))
/******************************************************************************/
/* 64-bit operations. */
#if (LG_SIZEOF_PTR == 3 || LG_SIZEOF_INT == 3)
# define JEMALLOC_ATOMIC_U64
# ifdef __GCC_HAVE_SYNC_COMPARE_AND_SWAP_8
JEMALLOC_INLINE uint64_t
atomic_add_uint64(uint64_t *p, uint64_t x)
{
return (__sync_add_and_fetch(p, x));
}
JEMALLOC_INLINE uint64_t
atomic_sub_uint64(uint64_t *p, uint64_t x)
{
return (__sync_sub_and_fetch(p, x));
}
#elif (defined(_MSC_VER))
JEMALLOC_INLINE uint64_t
atomic_add_uint64(uint64_t *p, uint64_t x)
{
return (InterlockedExchangeAdd64(p, x));
}
JEMALLOC_INLINE uint64_t
atomic_sub_uint64(uint64_t *p, uint64_t x)
{
return (InterlockedExchangeAdd64(p, -((int64_t)x)));
}
#elif (defined(JEMALLOC_OSATOMIC))
JEMALLOC_INLINE uint64_t
atomic_add_uint64(uint64_t *p, uint64_t x)
{
return (OSAtomicAdd64((int64_t)x, (int64_t *)p));
}
JEMALLOC_INLINE uint64_t
atomic_sub_uint64(uint64_t *p, uint64_t x)
{
return (OSAtomicAdd64(-((int64_t)x), (int64_t *)p));
}
# elif (defined(__amd64__) || defined(__x86_64__))
JEMALLOC_INLINE uint64_t
atomic_add_uint64(uint64_t *p, uint64_t x)
{
asm volatile (
"lock; xaddq %0, %1;"
: "+r" (x), "=m" (*p) /* Outputs. */
: "m" (*p) /* Inputs. */
);
return (x);
}
JEMALLOC_INLINE uint64_t
atomic_sub_uint64(uint64_t *p, uint64_t x)
{
x = (uint64_t)(-(int64_t)x);
asm volatile (
"lock; xaddq %0, %1;"
: "+r" (x), "=m" (*p) /* Outputs. */
: "m" (*p) /* Inputs. */
);
return (x);
}
# elif (defined(JEMALLOC_ATOMIC9))
JEMALLOC_INLINE uint64_t
atomic_add_uint64(uint64_t *p, uint64_t x)
{
/*
* atomic_fetchadd_64() doesn't exist, but we only ever use this
* function on LP64 systems, so atomic_fetchadd_long() will do.
*/
assert(sizeof(uint64_t) == sizeof(unsigned long));
return (atomic_fetchadd_long(p, (unsigned long)x) + x);
}
JEMALLOC_INLINE uint64_t
atomic_sub_uint64(uint64_t *p, uint64_t x)
{
assert(sizeof(uint64_t) == sizeof(unsigned long));
return (atomic_fetchadd_long(p, (unsigned long)(-(long)x)) - x);
}
# elif (defined(JE_FORCE_SYNC_COMPARE_AND_SWAP_8))
JEMALLOC_INLINE uint64_t
atomic_add_uint64(uint64_t *p, uint64_t x)
{
return (__sync_add_and_fetch(p, x));
}
JEMALLOC_INLINE uint64_t
atomic_sub_uint64(uint64_t *p, uint64_t x)
{
return (__sync_sub_and_fetch(p, x));
}
# else
# error "Missing implementation for 64-bit atomic operations"
# endif
#endif
JEMALLOC_GENERATE_ATOMICS(void *, p, LG_SIZEOF_PTR)
/******************************************************************************/
/* 32-bit operations. */
#ifdef __GCC_HAVE_SYNC_COMPARE_AND_SWAP_4
JEMALLOC_INLINE uint32_t
atomic_add_uint32(uint32_t *p, uint32_t x)
{
/*
* There's no actual guarantee that sizeof(bool) == 1, but it's true on the only
* platform that actually needs to know the size, MSVC.
*/
JEMALLOC_GENERATE_ATOMICS(bool, b, 0)
return (__sync_add_and_fetch(p, x));
}
JEMALLOC_GENERATE_INT_ATOMICS(unsigned, u, LG_SIZEOF_INT)
JEMALLOC_INLINE uint32_t
atomic_sub_uint32(uint32_t *p, uint32_t x)
{
JEMALLOC_GENERATE_INT_ATOMICS(size_t, zu, LG_SIZEOF_PTR)
return (__sync_sub_and_fetch(p, x));
}
#elif (defined(_MSC_VER))
JEMALLOC_INLINE uint32_t
atomic_add_uint32(uint32_t *p, uint32_t x)
{
JEMALLOC_GENERATE_INT_ATOMICS(ssize_t, zd, LG_SIZEOF_PTR)
return (InterlockedExchangeAdd(p, x));
}
JEMALLOC_GENERATE_INT_ATOMICS(uint32_t, u32, 2)
JEMALLOC_INLINE uint32_t
atomic_sub_uint32(uint32_t *p, uint32_t x)
{
#ifdef JEMALLOC_ATOMIC_U64
JEMALLOC_GENERATE_INT_ATOMICS(uint64_t, u64, 3)
return (InterlockedExchangeAdd(p, -((int32_t)x)));
}
#elif (defined(JEMALLOC_OSATOMIC))
JEMALLOC_INLINE uint32_t
atomic_add_uint32(uint32_t *p, uint32_t x)
{
return (OSAtomicAdd32((int32_t)x, (int32_t *)p));
}
JEMALLOC_INLINE uint32_t
atomic_sub_uint32(uint32_t *p, uint32_t x)
{
return (OSAtomicAdd32(-((int32_t)x), (int32_t *)p));
}
#elif (defined(__i386__) || defined(__amd64__) || defined(__x86_64__))
JEMALLOC_INLINE uint32_t
atomic_add_uint32(uint32_t *p, uint32_t x)
{
asm volatile (
"lock; xaddl %0, %1;"
: "+r" (x), "=m" (*p) /* Outputs. */
: "m" (*p) /* Inputs. */
);
return (x);
}
JEMALLOC_INLINE uint32_t
atomic_sub_uint32(uint32_t *p, uint32_t x)
{
x = (uint32_t)(-(int32_t)x);
asm volatile (
"lock; xaddl %0, %1;"
: "+r" (x), "=m" (*p) /* Outputs. */
: "m" (*p) /* Inputs. */
);
return (x);
}
#elif (defined(JEMALLOC_ATOMIC9))
JEMALLOC_INLINE uint32_t
atomic_add_uint32(uint32_t *p, uint32_t x)
{
return (atomic_fetchadd_32(p, x) + x);
}
JEMALLOC_INLINE uint32_t
atomic_sub_uint32(uint32_t *p, uint32_t x)
{
return (atomic_fetchadd_32(p, (uint32_t)(-(int32_t)x)) - x);
}
#elif (defined(JE_FORCE_SYNC_COMPARE_AND_SWAP_4))
JEMALLOC_INLINE uint32_t
atomic_add_uint32(uint32_t *p, uint32_t x)
{
return (__sync_add_and_fetch(p, x));
}
JEMALLOC_INLINE uint32_t
atomic_sub_uint32(uint32_t *p, uint32_t x)
{
return (__sync_sub_and_fetch(p, x));
}
#else
# error "Missing implementation for 32-bit atomic operations"
#endif
#undef ATOMIC_INLINE
/******************************************************************************/
/* size_t operations. */
JEMALLOC_INLINE size_t
atomic_add_z(size_t *p, size_t x)
{
#endif /* JEMALLOC_INTERNAL_ATOMIC_H */
#if (LG_SIZEOF_PTR == 3)
return ((size_t)atomic_add_uint64((uint64_t *)p, (uint64_t)x));
#elif (LG_SIZEOF_PTR == 2)
return ((size_t)atomic_add_uint32((uint32_t *)p, (uint32_t)x));
#endif
}
JEMALLOC_INLINE size_t
atomic_sub_z(size_t *p, size_t x)
{
#if (LG_SIZEOF_PTR == 3)
return ((size_t)atomic_add_uint64((uint64_t *)p,
(uint64_t)-((int64_t)x)));
#elif (LG_SIZEOF_PTR == 2)
return ((size_t)atomic_add_uint32((uint32_t *)p,
(uint32_t)-((int32_t)x)));
#endif
}
/******************************************************************************/
/* unsigned operations. */
JEMALLOC_INLINE unsigned
atomic_add_u(unsigned *p, unsigned x)
{
#if (LG_SIZEOF_INT == 3)
return ((unsigned)atomic_add_uint64((uint64_t *)p, (uint64_t)x));
#elif (LG_SIZEOF_INT == 2)
return ((unsigned)atomic_add_uint32((uint32_t *)p, (uint32_t)x));
#endif
}
JEMALLOC_INLINE unsigned
atomic_sub_u(unsigned *p, unsigned x)
{
#if (LG_SIZEOF_INT == 3)
return ((unsigned)atomic_add_uint64((uint64_t *)p,
(uint64_t)-((int64_t)x)));
#elif (LG_SIZEOF_INT == 2)
return ((unsigned)atomic_add_uint32((uint32_t *)p,
(uint32_t)-((int32_t)x)));
#endif
}
/******************************************************************************/
#endif
#endif /* JEMALLOC_H_INLINES */
/******************************************************************************/
-97
View File
@@ -1,97 +0,0 @@
#ifndef JEMALLOC_INTERNAL_ATOMIC_C11_H
#define JEMALLOC_INTERNAL_ATOMIC_C11_H
#include <stdatomic.h>
#define ATOMIC_INIT(...) ATOMIC_VAR_INIT(__VA_ARGS__)
#define atomic_memory_order_t memory_order
#define atomic_memory_order_relaxed memory_order_relaxed
#define atomic_memory_order_acquire memory_order_acquire
#define atomic_memory_order_release memory_order_release
#define atomic_memory_order_acq_rel memory_order_acq_rel
#define atomic_memory_order_seq_cst memory_order_seq_cst
#define atomic_fence atomic_thread_fence
#define JEMALLOC_GENERATE_ATOMICS(type, short_type, \
/* unused */ lg_size) \
typedef _Atomic(type) atomic_##short_type##_t; \
\
ATOMIC_INLINE type \
atomic_load_##short_type(const atomic_##short_type##_t *a, \
atomic_memory_order_t mo) { \
/* \
* A strict interpretation of the C standard prevents \
* atomic_load from taking a const argument, but it's \
* convenient for our purposes. This cast is a workaround. \
*/ \
atomic_##short_type##_t* a_nonconst = \
(atomic_##short_type##_t*)a; \
return atomic_load_explicit(a_nonconst, mo); \
} \
\
ATOMIC_INLINE void \
atomic_store_##short_type(atomic_##short_type##_t *a, \
type val, atomic_memory_order_t mo) { \
atomic_store_explicit(a, val, mo); \
} \
\
ATOMIC_INLINE type \
atomic_exchange_##short_type(atomic_##short_type##_t *a, type val, \
atomic_memory_order_t mo) { \
return atomic_exchange_explicit(a, val, mo); \
} \
\
ATOMIC_INLINE bool \
atomic_compare_exchange_weak_##short_type(atomic_##short_type##_t *a, \
type *expected, type desired, atomic_memory_order_t success_mo, \
atomic_memory_order_t failure_mo) { \
return atomic_compare_exchange_weak_explicit(a, expected, \
desired, success_mo, failure_mo); \
} \
\
ATOMIC_INLINE bool \
atomic_compare_exchange_strong_##short_type(atomic_##short_type##_t *a, \
type *expected, type desired, atomic_memory_order_t success_mo, \
atomic_memory_order_t failure_mo) { \
return atomic_compare_exchange_strong_explicit(a, expected, \
desired, success_mo, failure_mo); \
}
/*
* Integral types have some special operations available that non-integral ones
* lack.
*/
#define JEMALLOC_GENERATE_INT_ATOMICS(type, short_type, \
/* unused */ lg_size) \
JEMALLOC_GENERATE_ATOMICS(type, short_type, /* unused */ lg_size) \
\
ATOMIC_INLINE type \
atomic_fetch_add_##short_type(atomic_##short_type##_t *a, \
type val, atomic_memory_order_t mo) { \
return atomic_fetch_add_explicit(a, val, mo); \
} \
\
ATOMIC_INLINE type \
atomic_fetch_sub_##short_type(atomic_##short_type##_t *a, \
type val, atomic_memory_order_t mo) { \
return atomic_fetch_sub_explicit(a, val, mo); \
} \
ATOMIC_INLINE type \
atomic_fetch_and_##short_type(atomic_##short_type##_t *a, \
type val, atomic_memory_order_t mo) { \
return atomic_fetch_and_explicit(a, val, mo); \
} \
ATOMIC_INLINE type \
atomic_fetch_or_##short_type(atomic_##short_type##_t *a, \
type val, atomic_memory_order_t mo) { \
return atomic_fetch_or_explicit(a, val, mo); \
} \
ATOMIC_INLINE type \
atomic_fetch_xor_##short_type(atomic_##short_type##_t *a, \
type val, atomic_memory_order_t mo) { \
return atomic_fetch_xor_explicit(a, val, mo); \
}
#endif /* JEMALLOC_INTERNAL_ATOMIC_C11_H */
@@ -1,127 +0,0 @@
#ifndef JEMALLOC_INTERNAL_ATOMIC_GCC_ATOMIC_H
#define JEMALLOC_INTERNAL_ATOMIC_GCC_ATOMIC_H
#include "jemalloc/internal/assert.h"
#define ATOMIC_INIT(...) {__VA_ARGS__}
typedef enum {
atomic_memory_order_relaxed,
atomic_memory_order_acquire,
atomic_memory_order_release,
atomic_memory_order_acq_rel,
atomic_memory_order_seq_cst
} atomic_memory_order_t;
ATOMIC_INLINE int
atomic_enum_to_builtin(atomic_memory_order_t mo) {
switch (mo) {
case atomic_memory_order_relaxed:
return __ATOMIC_RELAXED;
case atomic_memory_order_acquire:
return __ATOMIC_ACQUIRE;
case atomic_memory_order_release:
return __ATOMIC_RELEASE;
case atomic_memory_order_acq_rel:
return __ATOMIC_ACQ_REL;
case atomic_memory_order_seq_cst:
return __ATOMIC_SEQ_CST;
}
/* Can't happen; the switch is exhaustive. */
not_reached();
}
ATOMIC_INLINE void
atomic_fence(atomic_memory_order_t mo) {
__atomic_thread_fence(atomic_enum_to_builtin(mo));
}
#define JEMALLOC_GENERATE_ATOMICS(type, short_type, \
/* unused */ lg_size) \
typedef struct { \
type repr; \
} atomic_##short_type##_t; \
\
ATOMIC_INLINE type \
atomic_load_##short_type(const atomic_##short_type##_t *a, \
atomic_memory_order_t mo) { \
type result; \
__atomic_load(&a->repr, &result, atomic_enum_to_builtin(mo)); \
return result; \
} \
\
ATOMIC_INLINE void \
atomic_store_##short_type(atomic_##short_type##_t *a, type val, \
atomic_memory_order_t mo) { \
__atomic_store(&a->repr, &val, atomic_enum_to_builtin(mo)); \
} \
\
ATOMIC_INLINE type \
atomic_exchange_##short_type(atomic_##short_type##_t *a, type val, \
atomic_memory_order_t mo) { \
type result; \
__atomic_exchange(&a->repr, &val, &result, \
atomic_enum_to_builtin(mo)); \
return result; \
} \
\
ATOMIC_INLINE bool \
atomic_compare_exchange_weak_##short_type(atomic_##short_type##_t *a, \
type *expected, type desired, atomic_memory_order_t success_mo, \
atomic_memory_order_t failure_mo) { \
return __atomic_compare_exchange(&a->repr, expected, &desired, \
true, atomic_enum_to_builtin(success_mo), \
atomic_enum_to_builtin(failure_mo)); \
} \
\
ATOMIC_INLINE bool \
atomic_compare_exchange_strong_##short_type(atomic_##short_type##_t *a, \
type *expected, type desired, atomic_memory_order_t success_mo, \
atomic_memory_order_t failure_mo) { \
return __atomic_compare_exchange(&a->repr, expected, &desired, \
false, \
atomic_enum_to_builtin(success_mo), \
atomic_enum_to_builtin(failure_mo)); \
}
#define JEMALLOC_GENERATE_INT_ATOMICS(type, short_type, \
/* unused */ lg_size) \
JEMALLOC_GENERATE_ATOMICS(type, short_type, /* unused */ lg_size) \
\
ATOMIC_INLINE type \
atomic_fetch_add_##short_type(atomic_##short_type##_t *a, type val, \
atomic_memory_order_t mo) { \
return __atomic_fetch_add(&a->repr, val, \
atomic_enum_to_builtin(mo)); \
} \
\
ATOMIC_INLINE type \
atomic_fetch_sub_##short_type(atomic_##short_type##_t *a, type val, \
atomic_memory_order_t mo) { \
return __atomic_fetch_sub(&a->repr, val, \
atomic_enum_to_builtin(mo)); \
} \
\
ATOMIC_INLINE type \
atomic_fetch_and_##short_type(atomic_##short_type##_t *a, type val, \
atomic_memory_order_t mo) { \
return __atomic_fetch_and(&a->repr, val, \
atomic_enum_to_builtin(mo)); \
} \
\
ATOMIC_INLINE type \
atomic_fetch_or_##short_type(atomic_##short_type##_t *a, type val, \
atomic_memory_order_t mo) { \
return __atomic_fetch_or(&a->repr, val, \
atomic_enum_to_builtin(mo)); \
} \
\
ATOMIC_INLINE type \
atomic_fetch_xor_##short_type(atomic_##short_type##_t *a, type val, \
atomic_memory_order_t mo) { \
return __atomic_fetch_xor(&a->repr, val, \
atomic_enum_to_builtin(mo)); \
}
#endif /* JEMALLOC_INTERNAL_ATOMIC_GCC_ATOMIC_H */
@@ -1,191 +0,0 @@
#ifndef JEMALLOC_INTERNAL_ATOMIC_GCC_SYNC_H
#define JEMALLOC_INTERNAL_ATOMIC_GCC_SYNC_H
#define ATOMIC_INIT(...) {__VA_ARGS__}
typedef enum {
atomic_memory_order_relaxed,
atomic_memory_order_acquire,
atomic_memory_order_release,
atomic_memory_order_acq_rel,
atomic_memory_order_seq_cst
} atomic_memory_order_t;
ATOMIC_INLINE void
atomic_fence(atomic_memory_order_t mo) {
/* Easy cases first: no barrier, and full barrier. */
if (mo == atomic_memory_order_relaxed) {
asm volatile("" ::: "memory");
return;
}
if (mo == atomic_memory_order_seq_cst) {
asm volatile("" ::: "memory");
__sync_synchronize();
asm volatile("" ::: "memory");
return;
}
asm volatile("" ::: "memory");
# if defined(__i386__) || defined(__x86_64__)
/* This is implicit on x86. */
# elif defined(__ppc__)
asm volatile("lwsync");
# elif defined(__sparc__) && defined(__arch64__)
if (mo == atomic_memory_order_acquire) {
asm volatile("membar #LoadLoad | #LoadStore");
} else if (mo == atomic_memory_order_release) {
asm volatile("membar #LoadStore | #StoreStore");
} else {
asm volatile("membar #LoadLoad | #LoadStore | #StoreStore");
}
# else
__sync_synchronize();
# endif
asm volatile("" ::: "memory");
}
/*
* A correct implementation of seq_cst loads and stores on weakly ordered
* architectures could do either of the following:
* 1. store() is weak-fence -> store -> strong fence, load() is load ->
* strong-fence.
* 2. store() is strong-fence -> store, load() is strong-fence -> load ->
* weak-fence.
* The tricky thing is, load() and store() above can be the load or store
* portions of a gcc __sync builtin, so we have to follow GCC's lead, which
* means going with strategy 2.
* On strongly ordered architectures, the natural strategy is to stick a strong
* fence after seq_cst stores, and have naked loads. So we want the strong
* fences in different places on different architectures.
* atomic_pre_sc_load_fence and atomic_post_sc_store_fence allow us to
* accomplish this.
*/
ATOMIC_INLINE void
atomic_pre_sc_load_fence() {
# if defined(__i386__) || defined(__x86_64__) || \
(defined(__sparc__) && defined(__arch64__))
atomic_fence(atomic_memory_order_relaxed);
# else
atomic_fence(atomic_memory_order_seq_cst);
# endif
}
ATOMIC_INLINE void
atomic_post_sc_store_fence() {
# if defined(__i386__) || defined(__x86_64__) || \
(defined(__sparc__) && defined(__arch64__))
atomic_fence(atomic_memory_order_seq_cst);
# else
atomic_fence(atomic_memory_order_relaxed);
# endif
}
#define JEMALLOC_GENERATE_ATOMICS(type, short_type, \
/* unused */ lg_size) \
typedef struct { \
type volatile repr; \
} atomic_##short_type##_t; \
\
ATOMIC_INLINE type \
atomic_load_##short_type(const atomic_##short_type##_t *a, \
atomic_memory_order_t mo) { \
if (mo == atomic_memory_order_seq_cst) { \
atomic_pre_sc_load_fence(); \
} \
type result = a->repr; \
if (mo != atomic_memory_order_relaxed) { \
atomic_fence(atomic_memory_order_acquire); \
} \
return result; \
} \
\
ATOMIC_INLINE void \
atomic_store_##short_type(atomic_##short_type##_t *a, \
type val, atomic_memory_order_t mo) { \
if (mo != atomic_memory_order_relaxed) { \
atomic_fence(atomic_memory_order_release); \
} \
a->repr = val; \
if (mo == atomic_memory_order_seq_cst) { \
atomic_post_sc_store_fence(); \
} \
} \
\
ATOMIC_INLINE type \
atomic_exchange_##short_type(atomic_##short_type##_t *a, type val, \
atomic_memory_order_t mo) { \
/* \
* Because of FreeBSD, we care about gcc 4.2, which doesn't have\
* an atomic exchange builtin. We fake it with a CAS loop. \
*/ \
while (true) { \
type old = a->repr; \
if (__sync_bool_compare_and_swap(&a->repr, old, val)) { \
return old; \
} \
} \
} \
\
ATOMIC_INLINE bool \
atomic_compare_exchange_weak_##short_type(atomic_##short_type##_t *a, \
type *expected, type desired, atomic_memory_order_t success_mo, \
atomic_memory_order_t failure_mo) { \
type prev = __sync_val_compare_and_swap(&a->repr, *expected, \
desired); \
if (prev == *expected) { \
return true; \
} else { \
*expected = prev; \
return false; \
} \
} \
ATOMIC_INLINE bool \
atomic_compare_exchange_strong_##short_type(atomic_##short_type##_t *a, \
type *expected, type desired, atomic_memory_order_t success_mo, \
atomic_memory_order_t failure_mo) { \
type prev = __sync_val_compare_and_swap(&a->repr, *expected, \
desired); \
if (prev == *expected) { \
return true; \
} else { \
*expected = prev; \
return false; \
} \
}
#define JEMALLOC_GENERATE_INT_ATOMICS(type, short_type, \
/* unused */ lg_size) \
JEMALLOC_GENERATE_ATOMICS(type, short_type, /* unused */ lg_size) \
\
ATOMIC_INLINE type \
atomic_fetch_add_##short_type(atomic_##short_type##_t *a, type val, \
atomic_memory_order_t mo) { \
return __sync_fetch_and_add(&a->repr, val); \
} \
\
ATOMIC_INLINE type \
atomic_fetch_sub_##short_type(atomic_##short_type##_t *a, type val, \
atomic_memory_order_t mo) { \
return __sync_fetch_and_sub(&a->repr, val); \
} \
\
ATOMIC_INLINE type \
atomic_fetch_and_##short_type(atomic_##short_type##_t *a, type val, \
atomic_memory_order_t mo) { \
return __sync_fetch_and_and(&a->repr, val); \
} \
\
ATOMIC_INLINE type \
atomic_fetch_or_##short_type(atomic_##short_type##_t *a, type val, \
atomic_memory_order_t mo) { \
return __sync_fetch_and_or(&a->repr, val); \
} \
\
ATOMIC_INLINE type \
atomic_fetch_xor_##short_type(atomic_##short_type##_t *a, type val, \
atomic_memory_order_t mo) { \
return __sync_fetch_and_xor(&a->repr, val); \
}
#endif /* JEMALLOC_INTERNAL_ATOMIC_GCC_SYNC_H */
-158
View File
@@ -1,158 +0,0 @@
#ifndef JEMALLOC_INTERNAL_ATOMIC_MSVC_H
#define JEMALLOC_INTERNAL_ATOMIC_MSVC_H
#define ATOMIC_INIT(...) {__VA_ARGS__}
typedef enum {
atomic_memory_order_relaxed,
atomic_memory_order_acquire,
atomic_memory_order_release,
atomic_memory_order_acq_rel,
atomic_memory_order_seq_cst
} atomic_memory_order_t;
typedef char atomic_repr_0_t;
typedef short atomic_repr_1_t;
typedef long atomic_repr_2_t;
typedef __int64 atomic_repr_3_t;
ATOMIC_INLINE void
atomic_fence(atomic_memory_order_t mo) {
_ReadWriteBarrier();
# if defined(_M_ARM) || defined(_M_ARM64)
/* ARM needs a barrier for everything but relaxed. */
if (mo != atomic_memory_order_relaxed) {
MemoryBarrier();
}
# elif defined(_M_IX86) || defined (_M_X64)
/* x86 needs a barrier only for seq_cst. */
if (mo == atomic_memory_order_seq_cst) {
MemoryBarrier();
}
# else
# error "Don't know how to create atomics for this platform for MSVC."
# endif
_ReadWriteBarrier();
}
#define ATOMIC_INTERLOCKED_REPR(lg_size) atomic_repr_ ## lg_size ## _t
#define ATOMIC_CONCAT(a, b) ATOMIC_RAW_CONCAT(a, b)
#define ATOMIC_RAW_CONCAT(a, b) a ## b
#define ATOMIC_INTERLOCKED_NAME(base_name, lg_size) ATOMIC_CONCAT( \
base_name, ATOMIC_INTERLOCKED_SUFFIX(lg_size))
#define ATOMIC_INTERLOCKED_SUFFIX(lg_size) \
ATOMIC_CONCAT(ATOMIC_INTERLOCKED_SUFFIX_, lg_size)
#define ATOMIC_INTERLOCKED_SUFFIX_0 8
#define ATOMIC_INTERLOCKED_SUFFIX_1 16
#define ATOMIC_INTERLOCKED_SUFFIX_2
#define ATOMIC_INTERLOCKED_SUFFIX_3 64
#define JEMALLOC_GENERATE_ATOMICS(type, short_type, lg_size) \
typedef struct { \
ATOMIC_INTERLOCKED_REPR(lg_size) repr; \
} atomic_##short_type##_t; \
\
ATOMIC_INLINE type \
atomic_load_##short_type(const atomic_##short_type##_t *a, \
atomic_memory_order_t mo) { \
ATOMIC_INTERLOCKED_REPR(lg_size) ret = a->repr; \
if (mo != atomic_memory_order_relaxed) { \
atomic_fence(atomic_memory_order_acquire); \
} \
return (type) ret; \
} \
\
ATOMIC_INLINE void \
atomic_store_##short_type(atomic_##short_type##_t *a, \
type val, atomic_memory_order_t mo) { \
if (mo != atomic_memory_order_relaxed) { \
atomic_fence(atomic_memory_order_release); \
} \
a->repr = (ATOMIC_INTERLOCKED_REPR(lg_size)) val; \
if (mo == atomic_memory_order_seq_cst) { \
atomic_fence(atomic_memory_order_seq_cst); \
} \
} \
\
ATOMIC_INLINE type \
atomic_exchange_##short_type(atomic_##short_type##_t *a, type val, \
atomic_memory_order_t mo) { \
return (type)ATOMIC_INTERLOCKED_NAME(_InterlockedExchange, \
lg_size)(&a->repr, (ATOMIC_INTERLOCKED_REPR(lg_size))val); \
} \
\
ATOMIC_INLINE bool \
atomic_compare_exchange_weak_##short_type(atomic_##short_type##_t *a, \
type *expected, type desired, atomic_memory_order_t success_mo, \
atomic_memory_order_t failure_mo) { \
ATOMIC_INTERLOCKED_REPR(lg_size) e = \
(ATOMIC_INTERLOCKED_REPR(lg_size))*expected; \
ATOMIC_INTERLOCKED_REPR(lg_size) d = \
(ATOMIC_INTERLOCKED_REPR(lg_size))desired; \
ATOMIC_INTERLOCKED_REPR(lg_size) old = \
ATOMIC_INTERLOCKED_NAME(_InterlockedCompareExchange, \
lg_size)(&a->repr, d, e); \
if (old == e) { \
return true; \
} else { \
*expected = (type)old; \
return false; \
} \
} \
\
ATOMIC_INLINE bool \
atomic_compare_exchange_strong_##short_type(atomic_##short_type##_t *a, \
type *expected, type desired, atomic_memory_order_t success_mo, \
atomic_memory_order_t failure_mo) { \
/* We implement the weak version with strong semantics. */ \
return atomic_compare_exchange_weak_##short_type(a, expected, \
desired, success_mo, failure_mo); \
}
#define JEMALLOC_GENERATE_INT_ATOMICS(type, short_type, lg_size) \
JEMALLOC_GENERATE_ATOMICS(type, short_type, lg_size) \
\
ATOMIC_INLINE type \
atomic_fetch_add_##short_type(atomic_##short_type##_t *a, \
type val, atomic_memory_order_t mo) { \
return (type)ATOMIC_INTERLOCKED_NAME(_InterlockedExchangeAdd, \
lg_size)(&a->repr, (ATOMIC_INTERLOCKED_REPR(lg_size))val); \
} \
\
ATOMIC_INLINE type \
atomic_fetch_sub_##short_type(atomic_##short_type##_t *a, \
type val, atomic_memory_order_t mo) { \
/* \
* MSVC warns on negation of unsigned operands, but for us it \
* gives exactly the right semantics (MAX_TYPE + 1 - operand). \
*/ \
__pragma(warning(push)) \
__pragma(warning(disable: 4146)) \
return atomic_fetch_add_##short_type(a, -val, mo); \
__pragma(warning(pop)) \
} \
ATOMIC_INLINE type \
atomic_fetch_and_##short_type(atomic_##short_type##_t *a, \
type val, atomic_memory_order_t mo) { \
return (type)ATOMIC_INTERLOCKED_NAME(_InterlockedAnd, lg_size)( \
&a->repr, (ATOMIC_INTERLOCKED_REPR(lg_size))val); \
} \
ATOMIC_INLINE type \
atomic_fetch_or_##short_type(atomic_##short_type##_t *a, \
type val, atomic_memory_order_t mo) { \
return (type)ATOMIC_INTERLOCKED_NAME(_InterlockedOr, lg_size)( \
&a->repr, (ATOMIC_INTERLOCKED_REPR(lg_size))val); \
} \
ATOMIC_INLINE type \
atomic_fetch_xor_##short_type(atomic_##short_type##_t *a, \
type val, atomic_memory_order_t mo) { \
return (type)ATOMIC_INTERLOCKED_NAME(_InterlockedXor, lg_size)( \
&a->repr, (ATOMIC_INTERLOCKED_REPR(lg_size))val); \
}
#endif /* JEMALLOC_INTERNAL_ATOMIC_MSVC_H */
@@ -1,33 +0,0 @@
#ifndef JEMALLOC_INTERNAL_BACKGROUND_THREAD_EXTERNS_H
#define JEMALLOC_INTERNAL_BACKGROUND_THREAD_EXTERNS_H
extern bool opt_background_thread;
extern size_t opt_max_background_threads;
extern malloc_mutex_t background_thread_lock;
extern atomic_b_t background_thread_enabled_state;
extern size_t n_background_threads;
extern size_t max_background_threads;
extern background_thread_info_t *background_thread_info;
extern bool can_enable_background_thread;
bool background_thread_create(tsd_t *tsd, unsigned arena_ind);
bool background_threads_enable(tsd_t *tsd);
bool background_threads_disable(tsd_t *tsd);
void background_thread_interval_check(tsdn_t *tsdn, arena_t *arena,
arena_decay_t *decay, size_t npages_new);
void background_thread_prefork0(tsdn_t *tsdn);
void background_thread_prefork1(tsdn_t *tsdn);
void background_thread_postfork_parent(tsdn_t *tsdn);
void background_thread_postfork_child(tsdn_t *tsdn);
bool background_thread_stats_read(tsdn_t *tsdn,
background_thread_stats_t *stats);
void background_thread_ctl_init(tsdn_t *tsdn);
#ifdef JEMALLOC_PTHREAD_CREATE_WRAPPER
extern int pthread_create_wrapper(pthread_t *__restrict, const pthread_attr_t *,
void *(*)(void *), void *__restrict);
#endif
bool background_thread_boot0(void);
bool background_thread_boot1(tsdn_t *tsdn);
#endif /* JEMALLOC_INTERNAL_BACKGROUND_THREAD_EXTERNS_H */
@@ -1,57 +0,0 @@
#ifndef JEMALLOC_INTERNAL_BACKGROUND_THREAD_INLINES_H
#define JEMALLOC_INTERNAL_BACKGROUND_THREAD_INLINES_H
JEMALLOC_ALWAYS_INLINE bool
background_thread_enabled(void) {
return atomic_load_b(&background_thread_enabled_state, ATOMIC_RELAXED);
}
JEMALLOC_ALWAYS_INLINE void
background_thread_enabled_set(tsdn_t *tsdn, bool state) {
malloc_mutex_assert_owner(tsdn, &background_thread_lock);
atomic_store_b(&background_thread_enabled_state, state, ATOMIC_RELAXED);
}
JEMALLOC_ALWAYS_INLINE background_thread_info_t *
arena_background_thread_info_get(arena_t *arena) {
unsigned arena_ind = arena_ind_get(arena);
return &background_thread_info[arena_ind % ncpus];
}
JEMALLOC_ALWAYS_INLINE uint64_t
background_thread_wakeup_time_get(background_thread_info_t *info) {
uint64_t next_wakeup = nstime_ns(&info->next_wakeup);
assert(atomic_load_b(&info->indefinite_sleep, ATOMIC_ACQUIRE) ==
(next_wakeup == BACKGROUND_THREAD_INDEFINITE_SLEEP));
return next_wakeup;
}
JEMALLOC_ALWAYS_INLINE void
background_thread_wakeup_time_set(tsdn_t *tsdn, background_thread_info_t *info,
uint64_t wakeup_time) {
malloc_mutex_assert_owner(tsdn, &info->mtx);
atomic_store_b(&info->indefinite_sleep,
wakeup_time == BACKGROUND_THREAD_INDEFINITE_SLEEP, ATOMIC_RELEASE);
nstime_init(&info->next_wakeup, wakeup_time);
}
JEMALLOC_ALWAYS_INLINE bool
background_thread_indefinite_sleep(background_thread_info_t *info) {
return atomic_load_b(&info->indefinite_sleep, ATOMIC_ACQUIRE);
}
JEMALLOC_ALWAYS_INLINE void
arena_background_thread_inactivity_check(tsdn_t *tsdn, arena_t *arena,
bool is_background_thread) {
if (!background_thread_enabled() || is_background_thread) {
return;
}
background_thread_info_t *info =
arena_background_thread_info_get(arena);
if (background_thread_indefinite_sleep(info)) {
background_thread_interval_check(tsdn, arena,
&arena->decay_dirty, 0);
}
}
#endif /* JEMALLOC_INTERNAL_BACKGROUND_THREAD_INLINES_H */
@@ -1,53 +0,0 @@
#ifndef JEMALLOC_INTERNAL_BACKGROUND_THREAD_STRUCTS_H
#define JEMALLOC_INTERNAL_BACKGROUND_THREAD_STRUCTS_H
/* This file really combines "structs" and "types", but only transitionally. */
#if defined(JEMALLOC_BACKGROUND_THREAD) || defined(JEMALLOC_LAZY_LOCK)
# define JEMALLOC_PTHREAD_CREATE_WRAPPER
#endif
#define BACKGROUND_THREAD_INDEFINITE_SLEEP UINT64_MAX
#define MAX_BACKGROUND_THREAD_LIMIT MALLOCX_ARENA_LIMIT
typedef enum {
background_thread_stopped,
background_thread_started,
/* Thread waits on the global lock when paused (for arena_reset). */
background_thread_paused,
} background_thread_state_t;
struct background_thread_info_s {
#ifdef JEMALLOC_BACKGROUND_THREAD
/* Background thread is pthread specific. */
pthread_t thread;
pthread_cond_t cond;
#endif
malloc_mutex_t mtx;
background_thread_state_t state;
/* When true, it means no wakeup scheduled. */
atomic_b_t indefinite_sleep;
/* Next scheduled wakeup time (absolute time in ns). */
nstime_t next_wakeup;
/*
* Since the last background thread run, newly added number of pages
* that need to be purged by the next wakeup. This is adjusted on
* epoch advance, and is used to determine whether we should signal the
* background thread to wake up earlier.
*/
size_t npages_to_purge_new;
/* Stats: total number of runs since started. */
uint64_t tot_n_runs;
/* Stats: total sleep time since started. */
nstime_t tot_sleep_time;
};
typedef struct background_thread_info_s background_thread_info_t;
struct background_thread_stats_s {
size_t num_threads;
uint64_t num_runs;
nstime_t run_interval;
};
typedef struct background_thread_stats_s background_thread_stats_t;
#endif /* JEMALLOC_INTERNAL_BACKGROUND_THREAD_STRUCTS_H */
+26
View File
@@ -0,0 +1,26 @@
/******************************************************************************/
#ifdef JEMALLOC_H_TYPES
#endif /* JEMALLOC_H_TYPES */
/******************************************************************************/
#ifdef JEMALLOC_H_STRUCTS
#endif /* JEMALLOC_H_STRUCTS */
/******************************************************************************/
#ifdef JEMALLOC_H_EXTERNS
void *base_alloc(size_t size);
void *base_calloc(size_t number, size_t size);
extent_node_t *base_node_alloc(void);
void base_node_dealloc(extent_node_t *node);
bool base_boot(void);
void base_prefork(void);
void base_postfork_parent(void);
void base_postfork_child(void);
#endif /* JEMALLOC_H_EXTERNS */
/******************************************************************************/
#ifdef JEMALLOC_H_INLINES
#endif /* JEMALLOC_H_INLINES */
/******************************************************************************/
-22
View File
@@ -1,22 +0,0 @@
#ifndef JEMALLOC_INTERNAL_BASE_EXTERNS_H
#define JEMALLOC_INTERNAL_BASE_EXTERNS_H
extern metadata_thp_mode_t opt_metadata_thp;
extern const char *metadata_thp_mode_names[];
base_t *b0get(void);
base_t *base_new(tsdn_t *tsdn, unsigned ind, extent_hooks_t *extent_hooks);
void base_delete(tsdn_t *tsdn, base_t *base);
extent_hooks_t *base_extent_hooks_get(base_t *base);
extent_hooks_t *base_extent_hooks_set(base_t *base,
extent_hooks_t *extent_hooks);
void *base_alloc(tsdn_t *tsdn, base_t *base, size_t size, size_t alignment);
extent_t *base_alloc_extent(tsdn_t *tsdn, base_t *base);
void base_stats_get(tsdn_t *tsdn, base_t *base, size_t *allocated,
size_t *resident, size_t *mapped, size_t *n_thp);
void base_prefork(tsdn_t *tsdn, base_t *base);
void base_postfork_parent(tsdn_t *tsdn, base_t *base);
void base_postfork_child(tsdn_t *tsdn, base_t *base);
bool base_boot(tsdn_t *tsdn);
#endif /* JEMALLOC_INTERNAL_BASE_EXTERNS_H */
-13
View File
@@ -1,13 +0,0 @@
#ifndef JEMALLOC_INTERNAL_BASE_INLINES_H
#define JEMALLOC_INTERNAL_BASE_INLINES_H
static inline unsigned
base_ind_get(const base_t *base) {
return base->ind;
}
static inline bool
metadata_thp_enabled(void) {
return (opt_metadata_thp != metadata_thp_disabled);
}
#endif /* JEMALLOC_INTERNAL_BASE_INLINES_H */
-59
View File
@@ -1,59 +0,0 @@
#ifndef JEMALLOC_INTERNAL_BASE_STRUCTS_H
#define JEMALLOC_INTERNAL_BASE_STRUCTS_H
#include "jemalloc/internal/jemalloc_internal_types.h"
#include "jemalloc/internal/mutex.h"
#include "jemalloc/internal/size_classes.h"
/* Embedded at the beginning of every block of base-managed virtual memory. */
struct base_block_s {
/* Total size of block's virtual memory mapping. */
size_t size;
/* Next block in list of base's blocks. */
base_block_t *next;
/* Tracks unused trailing space. */
extent_t extent;
};
struct base_s {
/* Associated arena's index within the arenas array. */
unsigned ind;
/*
* User-configurable extent hook functions. Points to an
* extent_hooks_t.
*/
atomic_p_t extent_hooks;
/* Protects base_alloc() and base_stats_get() operations. */
malloc_mutex_t mtx;
/* Using THP when true (metadata_thp auto mode). */
bool auto_thp_switched;
/*
* Most recent size class in the series of increasingly large base
* extents. Logarithmic spacing between subsequent allocations ensures
* that the total number of distinct mappings remains small.
*/
pszind_t pind_last;
/* Serial number generation state. */
size_t extent_sn_next;
/* Chain of all blocks associated with base. */
base_block_t *blocks;
/* Heap of extents that track unused trailing space within blocks. */
extent_heap_t avail[NSIZES];
/* Stats, only maintained if config_stats. */
size_t allocated;
size_t resident;
size_t mapped;
/* Number of THP regions touched. */
size_t n_thp;
};
#endif /* JEMALLOC_INTERNAL_BASE_STRUCTS_H */
-33
View File
@@ -1,33 +0,0 @@
#ifndef JEMALLOC_INTERNAL_BASE_TYPES_H
#define JEMALLOC_INTERNAL_BASE_TYPES_H
typedef struct base_block_s base_block_t;
typedef struct base_s base_t;
#define METADATA_THP_DEFAULT metadata_thp_disabled
/*
* In auto mode, arenas switch to huge pages for the base allocator on the
* second base block. a0 switches to thp on the 5th block (after 20 megabytes
* of metadata), since more metadata (e.g. rtree nodes) come from a0's base.
*/
#define BASE_AUTO_THP_THRESHOLD 2
#define BASE_AUTO_THP_THRESHOLD_A0 5
typedef enum {
metadata_thp_disabled = 0,
/*
* Lazily enable hugepage for metadata. To avoid high RSS caused by THP
* + low usage arena (i.e. THP becomes a significant percentage), the
* "auto" option only starts using THP after a base allocator used up
* the first THP region. Starting from the second hugepage (in a single
* arena), "auto" behaves the same as "always", i.e. madvise hugepage
* right away.
*/
metadata_thp_auto = 1,
metadata_thp_always = 2,
metadata_thp_mode_limit = 3
} metadata_thp_mode_t;
#endif /* JEMALLOC_INTERNAL_BASE_TYPES_H */
-106
View File
@@ -1,106 +0,0 @@
#ifndef JEMALLOC_INTERNAL_BIN_H
#define JEMALLOC_INTERNAL_BIN_H
#include "jemalloc/internal/extent_types.h"
#include "jemalloc/internal/extent_structs.h"
#include "jemalloc/internal/mutex.h"
#include "jemalloc/internal/bin_stats.h"
/*
* A bin contains a set of extents that are currently being used for slab
* allocations.
*/
/*
* Read-only information associated with each element of arena_t's bins array
* is stored separately, partly to reduce memory usage (only one copy, rather
* than one per arena), but mainly to avoid false cacheline sharing.
*
* Each slab has the following layout:
*
* /--------------------\
* | region 0 |
* |--------------------|
* | region 1 |
* |--------------------|
* | ... |
* | ... |
* | ... |
* |--------------------|
* | region nregs-1 |
* \--------------------/
*/
typedef struct bin_info_s bin_info_t;
struct bin_info_s {
/* Size of regions in a slab for this bin's size class. */
size_t reg_size;
/* Total size of a slab for this bin's size class. */
size_t slab_size;
/* Total number of regions in a slab for this bin's size class. */
uint32_t nregs;
/*
* Metadata used to manipulate bitmaps for slabs associated with this
* bin.
*/
bitmap_info_t bitmap_info;
};
extern const bin_info_t bin_infos[NBINS];
typedef struct bin_s bin_t;
struct bin_s {
/* All operations on bin_t fields require lock ownership. */
malloc_mutex_t lock;
/*
* Current slab being used to service allocations of this bin's size
* class. slabcur is independent of slabs_{nonfull,full}; whenever
* slabcur is reassigned, the previous slab must be deallocated or
* inserted into slabs_{nonfull,full}.
*/
extent_t *slabcur;
/*
* Heap of non-full slabs. This heap is used to assure that new
* allocations come from the non-full slab that is oldest/lowest in
* memory.
*/
extent_heap_t slabs_nonfull;
/* List used to track full slabs. */
extent_list_t slabs_full;
/* Bin statistics. */
bin_stats_t stats;
};
/* Initializes a bin to empty. Returns true on error. */
bool bin_init(bin_t *bin);
/* Forking. */
void bin_prefork(tsdn_t *tsdn, bin_t *bin);
void bin_postfork_parent(tsdn_t *tsdn, bin_t *bin);
void bin_postfork_child(tsdn_t *tsdn, bin_t *bin);
/* Stats. */
static inline void
bin_stats_merge(tsdn_t *tsdn, bin_stats_t *dst_bin_stats, bin_t *bin) {
malloc_mutex_lock(tsdn, &bin->lock);
malloc_mutex_prof_read(tsdn, &dst_bin_stats->mutex_data, &bin->lock);
dst_bin_stats->nmalloc += bin->stats.nmalloc;
dst_bin_stats->ndalloc += bin->stats.ndalloc;
dst_bin_stats->nrequests += bin->stats.nrequests;
dst_bin_stats->curregs += bin->stats.curregs;
dst_bin_stats->nfills += bin->stats.nfills;
dst_bin_stats->nflushes += bin->stats.nflushes;
dst_bin_stats->nslabs += bin->stats.nslabs;
dst_bin_stats->reslabs += bin->stats.reslabs;
dst_bin_stats->curslabs += bin->stats.curslabs;
malloc_mutex_unlock(tsdn, &bin->lock);
}
#endif /* JEMALLOC_INTERNAL_BIN_H */
-51
View File
@@ -1,51 +0,0 @@
#ifndef JEMALLOC_INTERNAL_BIN_STATS_H
#define JEMALLOC_INTERNAL_BIN_STATS_H
#include "jemalloc/internal/mutex_prof.h"
typedef struct bin_stats_s bin_stats_t;
struct bin_stats_s {
/*
* Total number of allocation/deallocation requests served directly by
* the bin. Note that tcache may allocate an object, then recycle it
* many times, resulting many increments to nrequests, but only one
* each to nmalloc and ndalloc.
*/
uint64_t nmalloc;
uint64_t ndalloc;
/*
* Number of allocation requests that correspond to the size of this
* bin. This includes requests served by tcache, though tcache only
* periodically merges into this counter.
*/
uint64_t nrequests;
/*
* Current number of regions of this size class, including regions
* currently cached by tcache.
*/
size_t curregs;
/* Number of tcache fills from this bin. */
uint64_t nfills;
/* Number of tcache flushes to this bin. */
uint64_t nflushes;
/* Total number of slabs created for this bin's size class. */
uint64_t nslabs;
/*
* Total number of slabs reused by extracting them from the slabs heap
* for this bin's size class.
*/
uint64_t reslabs;
/* Current number of slabs in this bin. */
size_t curslabs;
mutex_prof_data_t mutex_data;
};
#endif /* JEMALLOC_INTERNAL_BIN_STATS_H */
-165
View File
@@ -1,165 +0,0 @@
#ifndef JEMALLOC_INTERNAL_BIT_UTIL_H
#define JEMALLOC_INTERNAL_BIT_UTIL_H
#include "jemalloc/internal/assert.h"
#define BIT_UTIL_INLINE static inline
/* Sanity check. */
#if !defined(JEMALLOC_INTERNAL_FFSLL) || !defined(JEMALLOC_INTERNAL_FFSL) \
|| !defined(JEMALLOC_INTERNAL_FFS)
# error JEMALLOC_INTERNAL_FFS{,L,LL} should have been defined by configure
#endif
BIT_UTIL_INLINE unsigned
ffs_llu(unsigned long long bitmap) {
return JEMALLOC_INTERNAL_FFSLL(bitmap);
}
BIT_UTIL_INLINE unsigned
ffs_lu(unsigned long bitmap) {
return JEMALLOC_INTERNAL_FFSL(bitmap);
}
BIT_UTIL_INLINE unsigned
ffs_u(unsigned bitmap) {
return JEMALLOC_INTERNAL_FFS(bitmap);
}
BIT_UTIL_INLINE unsigned
ffs_zu(size_t bitmap) {
#if LG_SIZEOF_PTR == LG_SIZEOF_INT
return ffs_u(bitmap);
#elif LG_SIZEOF_PTR == LG_SIZEOF_LONG
return ffs_lu(bitmap);
#elif LG_SIZEOF_PTR == LG_SIZEOF_LONG_LONG
return ffs_llu(bitmap);
#else
#error No implementation for size_t ffs()
#endif
}
BIT_UTIL_INLINE unsigned
ffs_u64(uint64_t bitmap) {
#if LG_SIZEOF_LONG == 3
return ffs_lu(bitmap);
#elif LG_SIZEOF_LONG_LONG == 3
return ffs_llu(bitmap);
#else
#error No implementation for 64-bit ffs()
#endif
}
BIT_UTIL_INLINE unsigned
ffs_u32(uint32_t bitmap) {
#if LG_SIZEOF_INT == 2
return ffs_u(bitmap);
#else
#error No implementation for 32-bit ffs()
#endif
return ffs_u(bitmap);
}
BIT_UTIL_INLINE uint64_t
pow2_ceil_u64(uint64_t x) {
x--;
x |= x >> 1;
x |= x >> 2;
x |= x >> 4;
x |= x >> 8;
x |= x >> 16;
x |= x >> 32;
x++;
return x;
}
BIT_UTIL_INLINE uint32_t
pow2_ceil_u32(uint32_t x) {
x--;
x |= x >> 1;
x |= x >> 2;
x |= x >> 4;
x |= x >> 8;
x |= x >> 16;
x++;
return x;
}
/* Compute the smallest power of 2 that is >= x. */
BIT_UTIL_INLINE size_t
pow2_ceil_zu(size_t x) {
#if (LG_SIZEOF_PTR == 3)
return pow2_ceil_u64(x);
#else
return pow2_ceil_u32(x);
#endif
}
#if (defined(__i386__) || defined(__amd64__) || defined(__x86_64__))
BIT_UTIL_INLINE unsigned
lg_floor(size_t x) {
size_t ret;
assert(x != 0);
asm ("bsr %1, %0"
: "=r"(ret) // Outputs.
: "r"(x) // Inputs.
);
assert(ret < UINT_MAX);
return (unsigned)ret;
}
#elif (defined(_MSC_VER))
BIT_UTIL_INLINE unsigned
lg_floor(size_t x) {
unsigned long ret;
assert(x != 0);
#if (LG_SIZEOF_PTR == 3)
_BitScanReverse64(&ret, x);
#elif (LG_SIZEOF_PTR == 2)
_BitScanReverse(&ret, x);
#else
# error "Unsupported type size for lg_floor()"
#endif
assert(ret < UINT_MAX);
return (unsigned)ret;
}
#elif (defined(JEMALLOC_HAVE_BUILTIN_CLZ))
BIT_UTIL_INLINE unsigned
lg_floor(size_t x) {
assert(x != 0);
#if (LG_SIZEOF_PTR == LG_SIZEOF_INT)
return ((8 << LG_SIZEOF_PTR) - 1) - __builtin_clz(x);
#elif (LG_SIZEOF_PTR == LG_SIZEOF_LONG)
return ((8 << LG_SIZEOF_PTR) - 1) - __builtin_clzl(x);
#else
# error "Unsupported type size for lg_floor()"
#endif
}
#else
BIT_UTIL_INLINE unsigned
lg_floor(size_t x) {
assert(x != 0);
x |= (x >> 1);
x |= (x >> 2);
x |= (x >> 4);
x |= (x >> 8);
x |= (x >> 16);
#if (LG_SIZEOF_PTR == 3)
x |= (x >> 32);
#endif
if (x == SIZE_T_MAX) {
return (8 << LG_SIZEOF_PTR) - 1;
}
x++;
return ffs_zu(x) - 2;
}
#endif
#undef BIT_UTIL_INLINE
#endif /* JEMALLOC_INTERNAL_BIT_UTIL_H */
+82 -267
View File
@@ -1,159 +1,37 @@
#ifndef JEMALLOC_INTERNAL_BITMAP_H
#define JEMALLOC_INTERNAL_BITMAP_H
#include "jemalloc/internal/arena_types.h"
#include "jemalloc/internal/bit_util.h"
#include "jemalloc/internal/size_classes.h"
typedef unsigned long bitmap_t;
#define LG_SIZEOF_BITMAP LG_SIZEOF_LONG
/******************************************************************************/
#ifdef JEMALLOC_H_TYPES
/* Maximum bitmap bit count is 2^LG_BITMAP_MAXBITS. */
#if LG_SLAB_MAXREGS > LG_CEIL_NSIZES
/* Maximum bitmap bit count is determined by maximum regions per slab. */
# define LG_BITMAP_MAXBITS LG_SLAB_MAXREGS
#else
/* Maximum bitmap bit count is determined by number of extent size classes. */
# define LG_BITMAP_MAXBITS LG_CEIL_NSIZES
#endif
#define BITMAP_MAXBITS (ZU(1) << LG_BITMAP_MAXBITS)
#define LG_BITMAP_MAXBITS LG_RUN_MAXREGS
typedef struct bitmap_level_s bitmap_level_t;
typedef struct bitmap_info_s bitmap_info_t;
typedef unsigned long bitmap_t;
#define LG_SIZEOF_BITMAP LG_SIZEOF_LONG
/* Number of bits per group. */
#define LG_BITMAP_GROUP_NBITS (LG_SIZEOF_BITMAP + 3)
#define BITMAP_GROUP_NBITS (1U << LG_BITMAP_GROUP_NBITS)
#define BITMAP_GROUP_NBITS_MASK (BITMAP_GROUP_NBITS-1)
#define LG_BITMAP_GROUP_NBITS (LG_SIZEOF_BITMAP + 3)
#define BITMAP_GROUP_NBITS (ZU(1) << LG_BITMAP_GROUP_NBITS)
#define BITMAP_GROUP_NBITS_MASK (BITMAP_GROUP_NBITS-1)
/*
* Do some analysis on how big the bitmap is before we use a tree. For a brute
* force linear search, if we would have to call ffs_lu() more than 2^3 times,
* use a tree instead.
*/
#if LG_BITMAP_MAXBITS - LG_BITMAP_GROUP_NBITS > 3
# define BITMAP_USE_TREE
#endif
/* Maximum number of levels possible. */
#define BITMAP_MAX_LEVELS \
(LG_BITMAP_MAXBITS / LG_SIZEOF_BITMAP) \
+ !!(LG_BITMAP_MAXBITS % LG_SIZEOF_BITMAP)
/* Number of groups required to store a given number of bits. */
#define BITMAP_BITS2GROUPS(nbits) \
(((nbits) + BITMAP_GROUP_NBITS_MASK) >> LG_BITMAP_GROUP_NBITS)
#endif /* JEMALLOC_H_TYPES */
/******************************************************************************/
#ifdef JEMALLOC_H_STRUCTS
/*
* Number of groups required at a particular level for a given number of bits.
*/
#define BITMAP_GROUPS_L0(nbits) \
BITMAP_BITS2GROUPS(nbits)
#define BITMAP_GROUPS_L1(nbits) \
BITMAP_BITS2GROUPS(BITMAP_BITS2GROUPS(nbits))
#define BITMAP_GROUPS_L2(nbits) \
BITMAP_BITS2GROUPS(BITMAP_BITS2GROUPS(BITMAP_BITS2GROUPS((nbits))))
#define BITMAP_GROUPS_L3(nbits) \
BITMAP_BITS2GROUPS(BITMAP_BITS2GROUPS(BITMAP_BITS2GROUPS( \
BITMAP_BITS2GROUPS((nbits)))))
#define BITMAP_GROUPS_L4(nbits) \
BITMAP_BITS2GROUPS(BITMAP_BITS2GROUPS(BITMAP_BITS2GROUPS( \
BITMAP_BITS2GROUPS(BITMAP_BITS2GROUPS((nbits))))))
/*
* Assuming the number of levels, number of groups required for a given number
* of bits.
*/
#define BITMAP_GROUPS_1_LEVEL(nbits) \
BITMAP_GROUPS_L0(nbits)
#define BITMAP_GROUPS_2_LEVEL(nbits) \
(BITMAP_GROUPS_1_LEVEL(nbits) + BITMAP_GROUPS_L1(nbits))
#define BITMAP_GROUPS_3_LEVEL(nbits) \
(BITMAP_GROUPS_2_LEVEL(nbits) + BITMAP_GROUPS_L2(nbits))
#define BITMAP_GROUPS_4_LEVEL(nbits) \
(BITMAP_GROUPS_3_LEVEL(nbits) + BITMAP_GROUPS_L3(nbits))
#define BITMAP_GROUPS_5_LEVEL(nbits) \
(BITMAP_GROUPS_4_LEVEL(nbits) + BITMAP_GROUPS_L4(nbits))
/*
* Maximum number of groups required to support LG_BITMAP_MAXBITS.
*/
#ifdef BITMAP_USE_TREE
#if LG_BITMAP_MAXBITS <= LG_BITMAP_GROUP_NBITS
# define BITMAP_GROUPS(nbits) BITMAP_GROUPS_1_LEVEL(nbits)
# define BITMAP_GROUPS_MAX BITMAP_GROUPS_1_LEVEL(BITMAP_MAXBITS)
#elif LG_BITMAP_MAXBITS <= LG_BITMAP_GROUP_NBITS * 2
# define BITMAP_GROUPS(nbits) BITMAP_GROUPS_2_LEVEL(nbits)
# define BITMAP_GROUPS_MAX BITMAP_GROUPS_2_LEVEL(BITMAP_MAXBITS)
#elif LG_BITMAP_MAXBITS <= LG_BITMAP_GROUP_NBITS * 3
# define BITMAP_GROUPS(nbits) BITMAP_GROUPS_3_LEVEL(nbits)
# define BITMAP_GROUPS_MAX BITMAP_GROUPS_3_LEVEL(BITMAP_MAXBITS)
#elif LG_BITMAP_MAXBITS <= LG_BITMAP_GROUP_NBITS * 4
# define BITMAP_GROUPS(nbits) BITMAP_GROUPS_4_LEVEL(nbits)
# define BITMAP_GROUPS_MAX BITMAP_GROUPS_4_LEVEL(BITMAP_MAXBITS)
#elif LG_BITMAP_MAXBITS <= LG_BITMAP_GROUP_NBITS * 5
# define BITMAP_GROUPS(nbits) BITMAP_GROUPS_5_LEVEL(nbits)
# define BITMAP_GROUPS_MAX BITMAP_GROUPS_5_LEVEL(BITMAP_MAXBITS)
#else
# error "Unsupported bitmap size"
#endif
/*
* Maximum number of levels possible. This could be statically computed based
* on LG_BITMAP_MAXBITS:
*
* #define BITMAP_MAX_LEVELS \
* (LG_BITMAP_MAXBITS / LG_SIZEOF_BITMAP) \
* + !!(LG_BITMAP_MAXBITS % LG_SIZEOF_BITMAP)
*
* However, that would not allow the generic BITMAP_INFO_INITIALIZER() macro, so
* instead hardcode BITMAP_MAX_LEVELS to the largest number supported by the
* various cascading macros. The only additional cost this incurs is some
* unused trailing entries in bitmap_info_t structures; the bitmaps themselves
* are not impacted.
*/
#define BITMAP_MAX_LEVELS 5
#define BITMAP_INFO_INITIALIZER(nbits) { \
/* nbits. */ \
nbits, \
/* nlevels. */ \
(BITMAP_GROUPS_L0(nbits) > BITMAP_GROUPS_L1(nbits)) + \
(BITMAP_GROUPS_L1(nbits) > BITMAP_GROUPS_L2(nbits)) + \
(BITMAP_GROUPS_L2(nbits) > BITMAP_GROUPS_L3(nbits)) + \
(BITMAP_GROUPS_L3(nbits) > BITMAP_GROUPS_L4(nbits)) + 1, \
/* levels. */ \
{ \
{0}, \
{BITMAP_GROUPS_L0(nbits)}, \
{BITMAP_GROUPS_L1(nbits) + BITMAP_GROUPS_L0(nbits)}, \
{BITMAP_GROUPS_L2(nbits) + BITMAP_GROUPS_L1(nbits) + \
BITMAP_GROUPS_L0(nbits)}, \
{BITMAP_GROUPS_L3(nbits) + BITMAP_GROUPS_L2(nbits) + \
BITMAP_GROUPS_L1(nbits) + BITMAP_GROUPS_L0(nbits)}, \
{BITMAP_GROUPS_L4(nbits) + BITMAP_GROUPS_L3(nbits) + \
BITMAP_GROUPS_L2(nbits) + BITMAP_GROUPS_L1(nbits) \
+ BITMAP_GROUPS_L0(nbits)} \
} \
}
#else /* BITMAP_USE_TREE */
#define BITMAP_GROUPS(nbits) BITMAP_BITS2GROUPS(nbits)
#define BITMAP_GROUPS_MAX BITMAP_BITS2GROUPS(BITMAP_MAXBITS)
#define BITMAP_INFO_INITIALIZER(nbits) { \
/* nbits. */ \
nbits, \
/* ngroups. */ \
BITMAP_BITS2GROUPS(nbits) \
}
#endif /* BITMAP_USE_TREE */
typedef struct bitmap_level_s {
struct bitmap_level_s {
/* Offset of this level's groups within the array of groups. */
size_t group_offset;
} bitmap_level_t;
};
typedef struct bitmap_info_s {
struct bitmap_info_s {
/* Logical number of bits in bitmap (stored at bottom level). */
size_t nbits;
#ifdef BITMAP_USE_TREE
/* Number of levels necessary for nbits. */
unsigned nlevels;
@@ -162,62 +40,67 @@ typedef struct bitmap_info_s {
* bottom to top (e.g. the bottom level is stored in levels[0]).
*/
bitmap_level_t levels[BITMAP_MAX_LEVELS+1];
#else /* BITMAP_USE_TREE */
/* Number of groups necessary for nbits. */
size_t ngroups;
#endif /* BITMAP_USE_TREE */
} bitmap_info_t;
};
void bitmap_info_init(bitmap_info_t *binfo, size_t nbits);
void bitmap_init(bitmap_t *bitmap, const bitmap_info_t *binfo, bool fill);
size_t bitmap_size(const bitmap_info_t *binfo);
#endif /* JEMALLOC_H_STRUCTS */
/******************************************************************************/
#ifdef JEMALLOC_H_EXTERNS
static inline bool
bitmap_full(bitmap_t *bitmap, const bitmap_info_t *binfo) {
#ifdef BITMAP_USE_TREE
size_t rgoff = binfo->levels[binfo->nlevels].group_offset - 1;
void bitmap_info_init(bitmap_info_t *binfo, size_t nbits);
size_t bitmap_info_ngroups(const bitmap_info_t *binfo);
size_t bitmap_size(size_t nbits);
void bitmap_init(bitmap_t *bitmap, const bitmap_info_t *binfo);
#endif /* JEMALLOC_H_EXTERNS */
/******************************************************************************/
#ifdef JEMALLOC_H_INLINES
#ifndef JEMALLOC_ENABLE_INLINE
bool bitmap_full(bitmap_t *bitmap, const bitmap_info_t *binfo);
bool bitmap_get(bitmap_t *bitmap, const bitmap_info_t *binfo, size_t bit);
void bitmap_set(bitmap_t *bitmap, const bitmap_info_t *binfo, size_t bit);
size_t bitmap_sfu(bitmap_t *bitmap, const bitmap_info_t *binfo);
void bitmap_unset(bitmap_t *bitmap, const bitmap_info_t *binfo, size_t bit);
#endif
#if (defined(JEMALLOC_ENABLE_INLINE) || defined(JEMALLOC_BITMAP_C_))
JEMALLOC_INLINE bool
bitmap_full(bitmap_t *bitmap, const bitmap_info_t *binfo)
{
unsigned rgoff = binfo->levels[binfo->nlevels].group_offset - 1;
bitmap_t rg = bitmap[rgoff];
/* The bitmap is full iff the root group is 0. */
return (rg == 0);
#else
size_t i;
for (i = 0; i < binfo->ngroups; i++) {
if (bitmap[i] != 0) {
return false;
}
}
return true;
#endif
}
static inline bool
bitmap_get(bitmap_t *bitmap, const bitmap_info_t *binfo, size_t bit) {
JEMALLOC_INLINE bool
bitmap_get(bitmap_t *bitmap, const bitmap_info_t *binfo, size_t bit)
{
size_t goff;
bitmap_t g;
assert(bit < binfo->nbits);
goff = bit >> LG_BITMAP_GROUP_NBITS;
g = bitmap[goff];
return !(g & (ZU(1) << (bit & BITMAP_GROUP_NBITS_MASK)));
return (!(g & (1LU << (bit & BITMAP_GROUP_NBITS_MASK))));
}
static inline void
bitmap_set(bitmap_t *bitmap, const bitmap_info_t *binfo, size_t bit) {
JEMALLOC_INLINE void
bitmap_set(bitmap_t *bitmap, const bitmap_info_t *binfo, size_t bit)
{
size_t goff;
bitmap_t *gp;
bitmap_t g;
assert(bit < binfo->nbits);
assert(!bitmap_get(bitmap, binfo, bit));
assert(bitmap_get(bitmap, binfo, bit) == false);
goff = bit >> LG_BITMAP_GROUP_NBITS;
gp = &bitmap[goff];
g = *gp;
assert(g & (ZU(1) << (bit & BITMAP_GROUP_NBITS_MASK)));
g ^= ZU(1) << (bit & BITMAP_GROUP_NBITS_MASK);
assert(g & (1LU << (bit & BITMAP_GROUP_NBITS_MASK)));
g ^= 1LU << (bit & BITMAP_GROUP_NBITS_MASK);
*gp = g;
assert(bitmap_get(bitmap, binfo, bit));
#ifdef BITMAP_USE_TREE
/* Propagate group state transitions up the tree. */
if (g == 0) {
unsigned i;
@@ -226,113 +109,45 @@ bitmap_set(bitmap_t *bitmap, const bitmap_info_t *binfo, size_t bit) {
goff = bit >> LG_BITMAP_GROUP_NBITS;
gp = &bitmap[binfo->levels[i].group_offset + goff];
g = *gp;
assert(g & (ZU(1) << (bit & BITMAP_GROUP_NBITS_MASK)));
g ^= ZU(1) << (bit & BITMAP_GROUP_NBITS_MASK);
assert(g & (1LU << (bit & BITMAP_GROUP_NBITS_MASK)));
g ^= 1LU << (bit & BITMAP_GROUP_NBITS_MASK);
*gp = g;
if (g != 0) {
if (g != 0)
break;
}
}
}
#endif
}
/* ffu: find first unset >= bit. */
static inline size_t
bitmap_ffu(const bitmap_t *bitmap, const bitmap_info_t *binfo, size_t min_bit) {
assert(min_bit < binfo->nbits);
#ifdef BITMAP_USE_TREE
size_t bit = 0;
for (unsigned level = binfo->nlevels; level--;) {
size_t lg_bits_per_group = (LG_BITMAP_GROUP_NBITS * (level +
1));
bitmap_t group = bitmap[binfo->levels[level].group_offset + (bit
>> lg_bits_per_group)];
unsigned group_nmask = (unsigned)(((min_bit > bit) ? (min_bit -
bit) : 0) >> (lg_bits_per_group - LG_BITMAP_GROUP_NBITS));
assert(group_nmask <= BITMAP_GROUP_NBITS);
bitmap_t group_mask = ~((1LU << group_nmask) - 1);
bitmap_t group_masked = group & group_mask;
if (group_masked == 0LU) {
if (group == 0LU) {
return binfo->nbits;
}
/*
* min_bit was preceded by one or more unset bits in
* this group, but there are no other unset bits in this
* group. Try again starting at the first bit of the
* next sibling. This will recurse at most once per
* non-root level.
*/
size_t sib_base = bit + (ZU(1) << lg_bits_per_group);
assert(sib_base > min_bit);
assert(sib_base > bit);
if (sib_base >= binfo->nbits) {
return binfo->nbits;
}
return bitmap_ffu(bitmap, binfo, sib_base);
}
bit += ((size_t)(ffs_lu(group_masked) - 1)) <<
(lg_bits_per_group - LG_BITMAP_GROUP_NBITS);
}
assert(bit >= min_bit);
assert(bit < binfo->nbits);
return bit;
#else
size_t i = min_bit >> LG_BITMAP_GROUP_NBITS;
bitmap_t g = bitmap[i] & ~((1LU << (min_bit & BITMAP_GROUP_NBITS_MASK))
- 1);
size_t bit;
do {
bit = ffs_lu(g);
if (bit != 0) {
return (i << LG_BITMAP_GROUP_NBITS) + (bit - 1);
}
i++;
g = bitmap[i];
} while (i < binfo->ngroups);
return binfo->nbits;
#endif
}
/* sfu: set first unset. */
static inline size_t
bitmap_sfu(bitmap_t *bitmap, const bitmap_info_t *binfo) {
JEMALLOC_INLINE size_t
bitmap_sfu(bitmap_t *bitmap, const bitmap_info_t *binfo)
{
size_t bit;
bitmap_t g;
unsigned i;
assert(!bitmap_full(bitmap, binfo));
assert(bitmap_full(bitmap, binfo) == false);
#ifdef BITMAP_USE_TREE
i = binfo->nlevels - 1;
g = bitmap[binfo->levels[i].group_offset];
bit = ffs_lu(g) - 1;
bit = ffsl(g) - 1;
while (i > 0) {
i--;
g = bitmap[binfo->levels[i].group_offset + bit];
bit = (bit << LG_BITMAP_GROUP_NBITS) + (ffs_lu(g) - 1);
bit = (bit << LG_BITMAP_GROUP_NBITS) + (ffsl(g) - 1);
}
#else
i = 0;
g = bitmap[0];
while ((bit = ffs_lu(g)) == 0) {
i++;
g = bitmap[i];
}
bit = (i << LG_BITMAP_GROUP_NBITS) + (bit - 1);
#endif
bitmap_set(bitmap, binfo, bit);
return bit;
return (bit);
}
static inline void
bitmap_unset(bitmap_t *bitmap, const bitmap_info_t *binfo, size_t bit) {
JEMALLOC_INLINE void
bitmap_unset(bitmap_t *bitmap, const bitmap_info_t *binfo, size_t bit)
{
size_t goff;
bitmap_t *gp;
bitmap_t g;
UNUSED bool propagate;
bool propagate;
assert(bit < binfo->nbits);
assert(bitmap_get(bitmap, binfo, bit));
@@ -340,11 +155,10 @@ bitmap_unset(bitmap_t *bitmap, const bitmap_info_t *binfo, size_t bit) {
gp = &bitmap[goff];
g = *gp;
propagate = (g == 0);
assert((g & (ZU(1) << (bit & BITMAP_GROUP_NBITS_MASK))) == 0);
g ^= ZU(1) << (bit & BITMAP_GROUP_NBITS_MASK);
assert((g & (1LU << (bit & BITMAP_GROUP_NBITS_MASK))) == 0);
g ^= 1LU << (bit & BITMAP_GROUP_NBITS_MASK);
*gp = g;
assert(!bitmap_get(bitmap, binfo, bit));
#ifdef BITMAP_USE_TREE
assert(bitmap_get(bitmap, binfo, bit) == false);
/* Propagate group state transitions up the tree. */
if (propagate) {
unsigned i;
@@ -354,16 +168,17 @@ bitmap_unset(bitmap_t *bitmap, const bitmap_info_t *binfo, size_t bit) {
gp = &bitmap[binfo->levels[i].group_offset + goff];
g = *gp;
propagate = (g == 0);
assert((g & (ZU(1) << (bit & BITMAP_GROUP_NBITS_MASK)))
assert((g & (1LU << (bit & BITMAP_GROUP_NBITS_MASK)))
== 0);
g ^= ZU(1) << (bit & BITMAP_GROUP_NBITS_MASK);
g ^= 1LU << (bit & BITMAP_GROUP_NBITS_MASK);
*gp = g;
if (!propagate) {
if (propagate == false)
break;
}
}
}
#endif /* BITMAP_USE_TREE */
}
#endif /* JEMALLOC_INTERNAL_BITMAP_H */
#endif
#endif /* JEMALLOC_H_INLINES */
/******************************************************************************/
-114
View File
@@ -1,114 +0,0 @@
#ifndef JEMALLOC_INTERNAL_CACHE_BIN_H
#define JEMALLOC_INTERNAL_CACHE_BIN_H
#include "jemalloc/internal/ql.h"
/*
* The cache_bins are the mechanism that the tcache and the arena use to
* communicate. The tcache fills from and flushes to the arena by passing a
* cache_bin_t to fill/flush. When the arena needs to pull stats from the
* tcaches associated with it, it does so by iterating over its
* cache_bin_array_descriptor_t objects and reading out per-bin stats it
* contains. This makes it so that the arena need not know about the existence
* of the tcache at all.
*/
/*
* The count of the number of cached allocations in a bin. We make this signed
* so that negative numbers can encode "invalid" states (e.g. a low water mark
* of -1 for a cache that has been depleted).
*/
typedef int32_t cache_bin_sz_t;
typedef struct cache_bin_stats_s cache_bin_stats_t;
struct cache_bin_stats_s {
/*
* Number of allocation requests that corresponded to the size of this
* bin.
*/
uint64_t nrequests;
};
/*
* Read-only information associated with each element of tcache_t's tbins array
* is stored separately, mainly to reduce memory usage.
*/
typedef struct cache_bin_info_s cache_bin_info_t;
struct cache_bin_info_s {
/* Upper limit on ncached. */
cache_bin_sz_t ncached_max;
};
typedef struct cache_bin_s cache_bin_t;
struct cache_bin_s {
/* Min # cached since last GC. */
cache_bin_sz_t low_water;
/* # of cached objects. */
cache_bin_sz_t ncached;
/*
* ncached and stats are both modified frequently. Let's keep them
* close so that they have a higher chance of being on the same
* cacheline, thus less write-backs.
*/
cache_bin_stats_t tstats;
/*
* Stack of available objects.
*
* To make use of adjacent cacheline prefetch, the items in the avail
* stack goes to higher address for newer allocations. avail points
* just above the available space, which means that
* avail[-ncached, ... -1] are available items and the lowest item will
* be allocated first.
*/
void **avail;
};
typedef struct cache_bin_array_descriptor_s cache_bin_array_descriptor_t;
struct cache_bin_array_descriptor_s {
/*
* The arena keeps a list of the cache bins associated with it, for
* stats collection.
*/
ql_elm(cache_bin_array_descriptor_t) link;
/* Pointers to the tcache bins. */
cache_bin_t *bins_small;
cache_bin_t *bins_large;
};
static inline void
cache_bin_array_descriptor_init(cache_bin_array_descriptor_t *descriptor,
cache_bin_t *bins_small, cache_bin_t *bins_large) {
ql_elm_new(descriptor, link);
descriptor->bins_small = bins_small;
descriptor->bins_large = bins_large;
}
JEMALLOC_ALWAYS_INLINE void *
cache_bin_alloc_easy(cache_bin_t *bin, bool *success) {
void *ret;
if (unlikely(bin->ncached == 0)) {
bin->low_water = -1;
*success = false;
return NULL;
}
/*
* success (instead of ret) should be checked upon the return of this
* function. We avoid checking (ret == NULL) because there is never a
* null stored on the avail stack (which is unknown to the compiler),
* and eagerly checking ret would cause pipeline stall (waiting for the
* cacheline).
*/
*success = true;
ret = *(bin->avail - bin->ncached);
bin->ncached--;
if (unlikely(bin->ncached < bin->low_water)) {
bin->low_water = bin->ncached;
}
return ret;
}
#endif /* JEMALLOC_INTERNAL_CACHE_BIN_H */
+63
View File
@@ -0,0 +1,63 @@
/******************************************************************************/
#ifdef JEMALLOC_H_TYPES
/*
* Size and alignment of memory chunks that are allocated by the OS's virtual
* memory system.
*/
#define LG_CHUNK_DEFAULT 22
/* Return the chunk address for allocation address a. */
#define CHUNK_ADDR2BASE(a) \
((void *)((uintptr_t)(a) & ~chunksize_mask))
/* Return the chunk offset of address a. */
#define CHUNK_ADDR2OFFSET(a) \
((size_t)((uintptr_t)(a) & chunksize_mask))
/* Return the smallest chunk multiple that is >= s. */
#define CHUNK_CEILING(s) \
(((s) + chunksize_mask) & ~chunksize_mask)
#endif /* JEMALLOC_H_TYPES */
/******************************************************************************/
#ifdef JEMALLOC_H_STRUCTS
#endif /* JEMALLOC_H_STRUCTS */
/******************************************************************************/
#ifdef JEMALLOC_H_EXTERNS
extern size_t opt_lg_chunk;
extern const char *opt_dss;
/* Protects stats_chunks; currently not used for any other purpose. */
extern malloc_mutex_t chunks_mtx;
/* Chunk statistics. */
extern chunk_stats_t stats_chunks;
extern rtree_t *chunks_rtree;
extern size_t chunksize;
extern size_t chunksize_mask; /* (chunksize - 1). */
extern size_t chunk_npages;
extern size_t map_bias; /* Number of arena chunk header pages. */
extern size_t arena_maxclass; /* Max size class for arenas. */
void *chunk_alloc(size_t size, size_t alignment, bool base, bool *zero,
dss_prec_t dss_prec);
void chunk_unmap(void *chunk, size_t size);
void chunk_dealloc(void *chunk, size_t size, bool unmap);
bool chunk_boot(void);
void chunk_prefork(void);
void chunk_postfork_parent(void);
void chunk_postfork_child(void);
#endif /* JEMALLOC_H_EXTERNS */
/******************************************************************************/
#ifdef JEMALLOC_H_INLINES
#endif /* JEMALLOC_H_INLINES */
/******************************************************************************/
#include "jemalloc/internal/chunk_dss.h"
#include "jemalloc/internal/chunk_mmap.h"
+38
View File
@@ -0,0 +1,38 @@
/******************************************************************************/
#ifdef JEMALLOC_H_TYPES
typedef enum {
dss_prec_disabled = 0,
dss_prec_primary = 1,
dss_prec_secondary = 2,
dss_prec_limit = 3
} dss_prec_t;
#define DSS_PREC_DEFAULT dss_prec_secondary
#define DSS_DEFAULT "secondary"
#endif /* JEMALLOC_H_TYPES */
/******************************************************************************/
#ifdef JEMALLOC_H_STRUCTS
extern const char *dss_prec_names[];
#endif /* JEMALLOC_H_STRUCTS */
/******************************************************************************/
#ifdef JEMALLOC_H_EXTERNS
dss_prec_t chunk_dss_prec_get(void);
bool chunk_dss_prec_set(dss_prec_t dss_prec);
void *chunk_alloc_dss(size_t size, size_t alignment, bool *zero);
bool chunk_in_dss(void *chunk);
bool chunk_dss_boot(void);
void chunk_dss_prefork(void);
void chunk_dss_postfork_parent(void);
void chunk_dss_postfork_child(void);
#endif /* JEMALLOC_H_EXTERNS */
/******************************************************************************/
#ifdef JEMALLOC_H_INLINES
#endif /* JEMALLOC_H_INLINES */
/******************************************************************************/
+22
View File
@@ -0,0 +1,22 @@
/******************************************************************************/
#ifdef JEMALLOC_H_TYPES
#endif /* JEMALLOC_H_TYPES */
/******************************************************************************/
#ifdef JEMALLOC_H_STRUCTS
#endif /* JEMALLOC_H_STRUCTS */
/******************************************************************************/
#ifdef JEMALLOC_H_EXTERNS
bool pages_purge(void *addr, size_t length);
void *chunk_alloc_mmap(size_t size, size_t alignment, bool *zero);
bool chunk_dealloc_mmap(void *chunk, size_t size);
#endif /* JEMALLOC_H_EXTERNS */
/******************************************************************************/
#ifdef JEMALLOC_H_INLINES
#endif /* JEMALLOC_H_INLINES */
/******************************************************************************/
+57 -70
View File
@@ -1,101 +1,88 @@
#ifndef JEMALLOC_INTERNAL_CKH_H
#define JEMALLOC_INTERNAL_CKH_H
#include "jemalloc/internal/tsd.h"
/* Cuckoo hashing implementation. Skip to the end for the interface. */
/******************************************************************************/
/* INTERNAL DEFINITIONS -- IGNORE */
/******************************************************************************/
#ifdef JEMALLOC_H_TYPES
/* Maintain counters used to get an idea of performance. */
/* #define CKH_COUNT */
/* Print counter values in ckh_delete() (requires CKH_COUNT). */
/* #define CKH_VERBOSE */
/*
* There are 2^LG_CKH_BUCKET_CELLS cells in each hash table bucket. Try to fit
* one bucket per L1 cache line.
*/
#define LG_CKH_BUCKET_CELLS (LG_CACHELINE - LG_SIZEOF_PTR - 1)
typedef struct ckh_s ckh_t;
typedef struct ckhc_s ckhc_t;
/* Typedefs to allow easy function pointer passing. */
typedef void ckh_hash_t (const void *, size_t[2]);
typedef bool ckh_keycomp_t (const void *, const void *);
/* Hash table cell. */
typedef struct {
const void *key;
const void *data;
} ckhc_t;
/* Maintain counters used to get an idea of performance. */
/* #define CKH_COUNT */
/* Print counter values in ckh_delete() (requires CKH_COUNT). */
/* #define CKH_VERBOSE */
/* The hash table itself. */
typedef struct {
/*
* There are 2^LG_CKH_BUCKET_CELLS cells in each hash table bucket. Try to fit
* one bucket per L1 cache line.
*/
#define LG_CKH_BUCKET_CELLS (LG_CACHELINE - LG_SIZEOF_PTR - 1)
#endif /* JEMALLOC_H_TYPES */
/******************************************************************************/
#ifdef JEMALLOC_H_STRUCTS
/* Hash table cell. */
struct ckhc_s {
const void *key;
const void *data;
};
struct ckh_s {
#ifdef CKH_COUNT
/* Counters used to get an idea of performance. */
uint64_t ngrows;
uint64_t nshrinks;
uint64_t nshrinkfails;
uint64_t ninserts;
uint64_t nrelocs;
uint64_t ngrows;
uint64_t nshrinks;
uint64_t nshrinkfails;
uint64_t ninserts;
uint64_t nrelocs;
#endif
/* Used for pseudo-random number generation. */
uint64_t prng_state;
#define CKH_A 1103515241
#define CKH_C 12347
uint32_t prng_state;
/* Total number of items. */
size_t count;
size_t count;
/*
* Minimum and current number of hash table buckets. There are
* 2^LG_CKH_BUCKET_CELLS cells per bucket.
*/
unsigned lg_minbuckets;
unsigned lg_curbuckets;
unsigned lg_minbuckets;
unsigned lg_curbuckets;
/* Hash and comparison functions. */
ckh_hash_t *hash;
ckh_keycomp_t *keycomp;
ckh_hash_t *hash;
ckh_keycomp_t *keycomp;
/* Hash table with 2^lg_curbuckets buckets. */
ckhc_t *tab;
} ckh_t;
ckhc_t *tab;
};
#endif /* JEMALLOC_H_STRUCTS */
/******************************************************************************/
/* BEGIN PUBLIC API */
/******************************************************************************/
#ifdef JEMALLOC_H_EXTERNS
/* Lifetime management. Minitems is the initial capacity. */
bool ckh_new(tsd_t *tsd, ckh_t *ckh, size_t minitems, ckh_hash_t *hash,
bool ckh_new(ckh_t *ckh, size_t minitems, ckh_hash_t *hash,
ckh_keycomp_t *keycomp);
void ckh_delete(tsd_t *tsd, ckh_t *ckh);
/* Get the number of elements in the set. */
size_t ckh_count(ckh_t *ckh);
/*
* To iterate over the elements in the table, initialize *tabind to 0 and call
* this function until it returns true. Each call that returns false will
* update *key and *data to the next element in the table, assuming the pointers
* are non-NULL.
*/
bool ckh_iter(ckh_t *ckh, size_t *tabind, void **key, void **data);
/*
* Basic hash table operations -- insert, removal, lookup. For ckh_remove and
* ckh_search, key or data can be NULL. The hash-table only stores pointers to
* the key and value, and doesn't do any lifetime management.
*/
bool ckh_insert(tsd_t *tsd, ckh_t *ckh, const void *key, const void *data);
bool ckh_remove(tsd_t *tsd, ckh_t *ckh, const void *searchkey, void **key,
void ckh_delete(ckh_t *ckh);
size_t ckh_count(ckh_t *ckh);
bool ckh_iter(ckh_t *ckh, size_t *tabind, void **key, void **data);
bool ckh_insert(ckh_t *ckh, const void *key, const void *data);
bool ckh_remove(ckh_t *ckh, const void *searchkey, void **key,
void **data);
bool ckh_search(ckh_t *ckh, const void *searchkey, void **key, void **data);
bool ckh_search(ckh_t *ckh, const void *seachkey, void **key, void **data);
void ckh_string_hash(const void *key, size_t r_hash[2]);
bool ckh_string_keycomp(const void *k1, const void *k2);
void ckh_pointer_hash(const void *key, size_t r_hash[2]);
bool ckh_pointer_keycomp(const void *k1, const void *k2);
/* Some useful hash and comparison functions for strings and pointers. */
void ckh_string_hash(const void *key, size_t r_hash[2]);
bool ckh_string_keycomp(const void *k1, const void *k2);
void ckh_pointer_hash(const void *key, size_t r_hash[2]);
bool ckh_pointer_keycomp(const void *k1, const void *k2);
#endif /* JEMALLOC_H_EXTERNS */
/******************************************************************************/
#ifdef JEMALLOC_H_INLINES
#endif /* JEMALLOC_INTERNAL_CKH_H */
#endif /* JEMALLOC_H_INLINES */
/******************************************************************************/
+84 -98
View File
@@ -1,107 +1,87 @@
#ifndef JEMALLOC_INTERNAL_CTL_H
#define JEMALLOC_INTERNAL_CTL_H
/******************************************************************************/
#ifdef JEMALLOC_H_TYPES
#include "jemalloc/internal/jemalloc_internal_types.h"
#include "jemalloc/internal/malloc_io.h"
#include "jemalloc/internal/mutex_prof.h"
#include "jemalloc/internal/ql.h"
#include "jemalloc/internal/size_classes.h"
#include "jemalloc/internal/stats.h"
typedef struct ctl_node_s ctl_node_t;
typedef struct ctl_named_node_s ctl_named_node_t;
typedef struct ctl_indexed_node_s ctl_indexed_node_t;
typedef struct ctl_arena_stats_s ctl_arena_stats_t;
typedef struct ctl_stats_s ctl_stats_t;
/* Maximum ctl tree depth. */
#define CTL_MAX_DEPTH 7
#endif /* JEMALLOC_H_TYPES */
/******************************************************************************/
#ifdef JEMALLOC_H_STRUCTS
typedef struct ctl_node_s {
bool named;
} ctl_node_t;
typedef struct ctl_named_node_s {
ctl_node_t node;
const char *name;
/* If (nchildren == 0), this is a terminal node. */
size_t nchildren;
const ctl_node_t *children;
int (*ctl)(tsd_t *, const size_t *, size_t, void *, size_t *, void *,
size_t);
} ctl_named_node_t;
typedef struct ctl_indexed_node_s {
struct ctl_node_s node;
const ctl_named_node_t *(*index)(tsdn_t *, const size_t *, size_t,
size_t);
} ctl_indexed_node_t;
typedef struct ctl_arena_stats_s {
arena_stats_t astats;
/* Aggregate stats for small size classes, based on bin stats. */
size_t allocated_small;
uint64_t nmalloc_small;
uint64_t ndalloc_small;
uint64_t nrequests_small;
bin_stats_t bstats[NBINS];
arena_stats_large_t lstats[NSIZES - NBINS];
} ctl_arena_stats_t;
typedef struct ctl_stats_s {
size_t allocated;
size_t active;
size_t metadata;
size_t metadata_thp;
size_t resident;
size_t mapped;
size_t retained;
background_thread_stats_t background_thread;
mutex_prof_data_t mutex_prof_data[mutex_prof_num_global_mutexes];
} ctl_stats_t;
typedef struct ctl_arena_s ctl_arena_t;
struct ctl_arena_s {
unsigned arena_ind;
bool initialized;
ql_elm(ctl_arena_t) destroyed_link;
/* Basic stats, supported even if !config_stats. */
unsigned nthreads;
const char *dss;
ssize_t dirty_decay_ms;
ssize_t muzzy_decay_ms;
size_t pactive;
size_t pdirty;
size_t pmuzzy;
/* NULL if !config_stats. */
ctl_arena_stats_t *astats;
struct ctl_node_s {
bool named;
};
typedef struct ctl_arenas_s {
uint64_t epoch;
unsigned narenas;
ql_head(ctl_arena_t) destroyed;
struct ctl_named_node_s {
struct ctl_node_s node;
const char *name;
/* If (nchildren == 0), this is a terminal node. */
unsigned nchildren;
const ctl_node_t *children;
int (*ctl)(const size_t *, size_t, void *, size_t *,
void *, size_t);
};
/*
* Element 0 corresponds to merged stats for extant arenas (accessed via
* MALLCTL_ARENAS_ALL), element 1 corresponds to merged stats for
* destroyed arenas (accessed via MALLCTL_ARENAS_DESTROYED), and the
* remaining MALLOCX_ARENA_LIMIT elements correspond to arenas.
*/
ctl_arena_t *arenas[2 + MALLOCX_ARENA_LIMIT];
} ctl_arenas_t;
struct ctl_indexed_node_s {
struct ctl_node_s node;
const ctl_named_node_t *(*index)(const size_t *, size_t, size_t);
};
int ctl_byname(tsd_t *tsd, const char *name, void *oldp, size_t *oldlenp,
struct ctl_arena_stats_s {
bool initialized;
unsigned nthreads;
const char *dss;
size_t pactive;
size_t pdirty;
arena_stats_t astats;
/* Aggregate stats for small size classes, based on bin stats. */
size_t allocated_small;
uint64_t nmalloc_small;
uint64_t ndalloc_small;
uint64_t nrequests_small;
malloc_bin_stats_t bstats[NBINS];
malloc_large_stats_t *lstats; /* nlclasses elements. */
};
struct ctl_stats_s {
size_t allocated;
size_t active;
size_t mapped;
struct {
size_t current; /* stats_chunks.curchunks */
uint64_t total; /* stats_chunks.nchunks */
size_t high; /* stats_chunks.highchunks */
} chunks;
struct {
size_t allocated; /* huge_allocated */
uint64_t nmalloc; /* huge_nmalloc */
uint64_t ndalloc; /* huge_ndalloc */
} huge;
unsigned narenas;
ctl_arena_stats_t *arenas; /* (narenas + 1) elements. */
};
#endif /* JEMALLOC_H_STRUCTS */
/******************************************************************************/
#ifdef JEMALLOC_H_EXTERNS
int ctl_byname(const char *name, void *oldp, size_t *oldlenp, void *newp,
size_t newlen);
int ctl_nametomib(const char *name, size_t *mibp, size_t *miblenp);
int ctl_bymib(const size_t *mib, size_t miblen, void *oldp, size_t *oldlenp,
void *newp, size_t newlen);
int ctl_nametomib(tsd_t *tsd, const char *name, size_t *mibp, size_t *miblenp);
bool ctl_boot(void);
void ctl_prefork(void);
void ctl_postfork_parent(void);
void ctl_postfork_child(void);
int ctl_bymib(tsd_t *tsd, const size_t *mib, size_t miblen, void *oldp,
size_t *oldlenp, void *newp, size_t newlen);
bool ctl_boot(void);
void ctl_prefork(tsdn_t *tsdn);
void ctl_postfork_parent(tsdn_t *tsdn);
void ctl_postfork_child(tsdn_t *tsdn);
#define xmallctl(name, oldp, oldlenp, newp, newlen) do { \
#define xmallctl(name, oldp, oldlenp, newp, newlen) do { \
if (je_mallctl(name, oldp, oldlenp, newp, newlen) \
!= 0) { \
malloc_printf( \
@@ -111,7 +91,7 @@ void ctl_postfork_child(tsdn_t *tsdn);
} \
} while (0)
#define xmallctlnametomib(name, mibp, miblenp) do { \
#define xmallctlnametomib(name, mibp, miblenp) do { \
if (je_mallctlnametomib(name, mibp, miblenp) != 0) { \
malloc_printf("<jemalloc>: Failure in " \
"xmallctlnametomib(\"%s\", ...)\n", name); \
@@ -119,7 +99,7 @@ void ctl_postfork_child(tsdn_t *tsdn);
} \
} while (0)
#define xmallctlbymib(mib, miblen, oldp, oldlenp, newp, newlen) do { \
#define xmallctlbymib(mib, miblen, oldp, oldlenp, newp, newlen) do { \
if (je_mallctlbymib(mib, miblen, oldp, oldlenp, newp, \
newlen) != 0) { \
malloc_write( \
@@ -128,4 +108,10 @@ void ctl_postfork_child(tsdn_t *tsdn);
} \
} while (0)
#endif /* JEMALLOC_INTERNAL_CTL_H */
#endif /* JEMALLOC_H_EXTERNS */
/******************************************************************************/
#ifdef JEMALLOC_H_INLINES
#endif /* JEMALLOC_H_INLINES */
/******************************************************************************/
-41
View File
@@ -1,41 +0,0 @@
#ifndef JEMALLOC_INTERNAL_DIV_H
#define JEMALLOC_INTERNAL_DIV_H
#include "jemalloc/internal/assert.h"
/*
* This module does the division that computes the index of a region in a slab,
* given its offset relative to the base.
* That is, given a divisor d, an n = i * d (all integers), we'll return i.
* We do some pre-computation to do this more quickly than a CPU division
* instruction.
* We bound n < 2^32, and don't support dividing by one.
*/
typedef struct div_info_s div_info_t;
struct div_info_s {
uint32_t magic;
#ifdef JEMALLOC_DEBUG
size_t d;
#endif
};
void div_init(div_info_t *div_info, size_t divisor);
static inline size_t
div_compute(div_info_t *div_info, size_t n) {
assert(n <= (uint32_t)-1);
/*
* This generates, e.g. mov; imul; shr on x86-64. On a 32-bit machine,
* the compilers I tried were all smart enough to turn this into the
* appropriate "get the high 32 bits of the result of a multiply" (e.g.
* mul; mov edx eax; on x86, umull on arm, etc.).
*/
size_t i = ((uint64_t)n * (uint64_t)div_info->magic) >> 32;
#ifdef JEMALLOC_DEBUG
assert(i * div_info->d == n);
#endif
return i;
}
#endif /* JEMALLOC_INTERNAL_DIV_H */

Some files were not shown because too many files have changed in this diff Show More