# Redis Array For years, Redis has been missing a real indexed data structure for the use cases where the index and the spatial relationship of elements are semantic. Hashes give you random lookups, but you have to store an index as a key, and have no range visibility. Lists give you appending and trimming, but what is in the middle remains hard to access. Streams give you append-only events, which is another (useful, indeed) beast. None of these is what you want when the *position itself* has business meaning — slot 37, step 4, row 18552, day from 2934 to 2949, file line 11, 12, 15 and so forth. And, all those types, for different reasons, are all suboptimal when you want a **ring buffer** able to store the latest N observed samples of something. Up to now, users found ways (they always do \o/) using the fact that the data structures that are obvious in this universe are also extremely powerful, if well implemented. But this forces compromises. Arrays handle these index-first requirements natively, and usually with much better memory and CPU usage than the workarounds. If the use case is the right one, Arrays often provide much better space, time and usability at the same time. ## Internal encoding 1. When dense, an Array is essentially a more fancy C array. You don't pay anything for storing the index. 2. Yet, instead of going really flat, arrays are sliced into 4096-element slices, and each slice, when it contains just a few elements, uses a special sparse encoding. When a slice is empty it's just a `NULL` stored in the directory. 3. Small ints, floats, and short strings are pointer-tagged, so they cost zero additional memory beyond the pointer slot itself. 4. When very sparse, a super-directory of windowed directories is used. This allows the data type to be safe, instead of exhibiting pathological space or time behavior. This representation is only triggered when there are more than 8 million elements or very high indexes set. ## Use cases Arrays are mostly stateless if not for the fact that each array remembers the index of the latest added item, allowing `ARINSERT` and `ARRING` to work properly. Otherwise it is a set/get at this index game, with solid support for both setting / getting ranges, server-side scanning, returning only populated elements in a time which is proportional not to the range size, but to the population size. A few concrete examples, that may work as mental models for the set of problems that are similar to them (from the POV of the data modeling). **Thermometer.** A sensor reporting once per minute, with gaps: ``` ARSET temp:room12:day7 123 22.3 ARGETRANGE temp:room12:day7 600 660 # the 10:00–11:00 window, with NULLs ARSCAN temp:room12:day7 600 660 # only populated elements AROP temp:room12:day7 0 1439 MAX # peak of the day, server-side ``` Missing minutes cost little to nothing. Numeric aggregation runs inside Redis. Telemetry, IoT, meter readings, KPI rollups. **Calendar.** A clinic with 96 fifteen-minute slots per day: ``` ARSET sched:room12:day 32 booking:991 ARSCAN sched:room12:day 0 95 # only occupied slots ARGETRANGE sched:room12:day 48 63 # the afternoon full view to render ``` The slot number is the business key in this case. Room booking, parking spaces, warehouse bins, lockers, ... **Ring buffer.** ARRING replaces the classic LPUSH+LTRIM pattern. Imagine remote `dmesg`. ``` ARRING machine:123 200 "[141087.430123]: arm_cpu_init(): cpu 14 online" # Capped to 200 entries ARLASTITEMS machine:123 50 REV # 50 newest first ``` Faster than LPUSH+LTRIM, keep indexed access to past elements. Last-N alarms, recent fraud scores, access history, remote logs, device events. Ok here the use cases are mainly the ones of the old pattern: it is just a better fit and allows to access random items in the middle, aggregate server-side, and so forth. **Workflow.** Step number is the index, value is the status. Gaps are meaningful: ``` ARSET claim:99172 0 received ARSET claim:99172 3 waiting:reviewer42 ARSET claim:99172 5 approved ARGETRANGE claim:99172 0 5 # full workflow view, with NULLs for missing steps ARSCAN claim:99172 0 5 # only steps that have a state ARCOUNT claim:99172 # number of recorded steps ARLEN claim:99172 # highest reached step + 1 ``` **Skills knowledge base for agents.** Arrays are good at representing / grepping into Markdown files: ``` ARSET skill:metal_gpu 0 "...." ARSET skill:metal_gpu 1 "...." ARSET skill:metal_gpu 2 "...." ARGREP skill:metal_gpu - + RE "M3|M4" WITHVALUES ``` ARGREP has EXACT, MATCH, GLOB, RE, you can have multiple predicates, can select AND or OR behavior. **Bulk import results.** Sparse row annotations over millions of rows / CSV / ...: ``` ARSET import:job551 18552 ERR:bad_email ARSCAN import:job551 0 1000000 # Provides only rows that have something ``` ## TLDR If the position is part of the meaning, use an Array. If you want to aggregate or grep remotely, use an Array. Feedback welcome :) --------- Co-authored-by: debing.sun <debing.sun@redis.com> Co-authored-by: Shubham S Taple <155555100+ShubhamTaple@users.noreply.github.com> Co-authored-by: Yuan Wang <yuan.wang@redis.com> Co-authored-by: Marc Gravell <marc.gravell@gmail.com>
189 lines
4.1 KiB
C
189 lines
4.1 KiB
C
/*
|
|
tre_regcomp.c - TRE POSIX compatible regex compilation functions.
|
|
|
|
This software is released under a BSD-style license.
|
|
See the file LICENSE for details and copyright.
|
|
|
|
*/
|
|
|
|
#ifdef HAVE_CONFIG_H
|
|
#include <config.h>
|
|
#endif /* HAVE_CONFIG_H */
|
|
|
|
#include <string.h>
|
|
#include <errno.h>
|
|
#include <stdlib.h>
|
|
|
|
#include "tre-internal.h"
|
|
#include "xmalloc.h"
|
|
|
|
int
|
|
tre_regncomp(regex_t *preg, const char *regex, size_t n, int cflags)
|
|
{
|
|
int ret;
|
|
if (n > TRE_MAX_RE)
|
|
return REG_ESPACE;
|
|
#if TRE_WCHAR
|
|
tre_char_t *wregex;
|
|
size_t wlen;
|
|
|
|
wregex = xmalloc(sizeof(tre_char_t) * (n + 1));
|
|
if (wregex == NULL)
|
|
return REG_ESPACE;
|
|
|
|
/* If the current locale uses the standard single byte encoding of
|
|
characters, we don't do a multibyte string conversion. If we did,
|
|
many applications which use the default locale would break since
|
|
the default "C" locale uses the 7-bit ASCII character set, and
|
|
all characters with the eighth bit set would be considered invalid. */
|
|
#if TRE_MULTIBYTE
|
|
if (TRE_MB_CUR_MAX == 1)
|
|
#endif /* TRE_MULTIBYTE */
|
|
{
|
|
size_t i;
|
|
const unsigned char *str = (const unsigned char *)regex;
|
|
tre_char_t *wstr = wregex;
|
|
|
|
for (i = 0; i < n; i++)
|
|
*(wstr++) = *(str++);
|
|
wlen = n;
|
|
}
|
|
#if TRE_MULTIBYTE
|
|
else
|
|
{
|
|
size_t consumed;
|
|
tre_char_t *wcptr = wregex;
|
|
#ifdef HAVE_MBSTATE_T
|
|
mbstate_t state;
|
|
memset(&state, '\0', sizeof(state));
|
|
#endif /* HAVE_MBSTATE_T */
|
|
while (n > 0)
|
|
{
|
|
consumed = tre_mbrtowc(wcptr, regex, n, &state);
|
|
|
|
switch (consumed)
|
|
{
|
|
case 0:
|
|
if (*regex == '\0')
|
|
consumed = 1;
|
|
else
|
|
{
|
|
xfree(wregex);
|
|
return REG_BADPAT;
|
|
}
|
|
break;
|
|
case -1:
|
|
DPRINT(("mbrtowc: error %d: %s.\n", errno, strerror(errno)));
|
|
xfree(wregex);
|
|
return REG_BADPAT;
|
|
case -2:
|
|
/* The last character wasn't complete. Let's not call it a
|
|
fatal error. */
|
|
consumed = n;
|
|
break;
|
|
}
|
|
regex += consumed;
|
|
n -= consumed;
|
|
wcptr++;
|
|
}
|
|
wlen = wcptr - wregex;
|
|
}
|
|
#endif /* TRE_MULTIBYTE */
|
|
|
|
wregex[wlen] = L'\0';
|
|
ret = tre_compile(preg, wregex, wlen, cflags);
|
|
xfree(wregex);
|
|
#else /* !TRE_WCHAR */
|
|
ret = tre_compile(preg, (const tre_char_t *)regex, n, cflags);
|
|
#endif /* !TRE_WCHAR */
|
|
|
|
return ret;
|
|
}
|
|
|
|
/* this version takes bytes literally, to be used with raw vectors */
|
|
int
|
|
tre_regncompb(regex_t *preg, const char *regex, size_t n, int cflags)
|
|
{
|
|
int ret;
|
|
if (n > TRE_MAX_RE)
|
|
return REG_ESPACE;
|
|
#if TRE_WCHAR /* wide chars = we need to convert it all to the wide format */
|
|
tre_char_t *wregex;
|
|
size_t i;
|
|
|
|
wregex = xmalloc(sizeof(tre_char_t) * n);
|
|
if (wregex == NULL)
|
|
return REG_ESPACE;
|
|
|
|
for (i = 0; i < n; i++)
|
|
wregex[i] = (tre_char_t) ((unsigned char) regex[i]);
|
|
|
|
ret = tre_compile(preg, wregex, n, cflags | REG_USEBYTES);
|
|
xfree(wregex);
|
|
#else /* !TRE_WCHAR */
|
|
ret = tre_compile(preg, (const tre_char_t *)regex, n, cflags | REG_USEBYTES);
|
|
#endif /* !TRE_WCHAR */
|
|
|
|
return ret;
|
|
}
|
|
|
|
int
|
|
tre_regcomp(regex_t *preg, const char *regex, int cflags)
|
|
{
|
|
size_t n = regex ? strlen(regex) : 0;
|
|
if (n > TRE_MAX_RE)
|
|
return REG_ESPACE;
|
|
return tre_regncomp(preg, regex, n, cflags);
|
|
}
|
|
|
|
int
|
|
tre_regcompb(regex_t *preg, const char *regex, int cflags)
|
|
{
|
|
int ret;
|
|
tre_char_t *wregex;
|
|
size_t i, n = regex ? strlen(regex) : 0;
|
|
const unsigned char *str = (const unsigned char *)regex;
|
|
tre_char_t *wstr;
|
|
|
|
if (n > TRE_MAX_RE)
|
|
return REG_ESPACE;
|
|
wregex = xmalloc(sizeof(tre_char_t) * (n + 1));
|
|
if (wregex == NULL) return REG_ESPACE;
|
|
wstr = wregex;
|
|
|
|
for (i = 0; i < n; i++)
|
|
*(wstr++) = *(str++);
|
|
wregex[n] = L'\0';
|
|
ret = tre_compile(preg, wregex, n, cflags | REG_USEBYTES);
|
|
xfree(wregex);
|
|
return ret;
|
|
}
|
|
|
|
|
|
#ifdef TRE_WCHAR
|
|
int
|
|
tre_regwncomp(regex_t *preg, const wchar_t *regex, size_t n, int cflags)
|
|
{
|
|
if (n > TRE_MAX_RE)
|
|
return REG_ESPACE;
|
|
return tre_compile(preg, regex, n, cflags);
|
|
}
|
|
|
|
int
|
|
tre_regwcomp(regex_t *preg, const wchar_t *regex, int cflags)
|
|
{
|
|
size_t n = regex ? wcslen(regex) : 0;
|
|
if (n > TRE_MAX_RE)
|
|
return REG_ESPACE;
|
|
return tre_compile(preg, regex, n, cflags);
|
|
}
|
|
#endif /* TRE_WCHAR */
|
|
|
|
void
|
|
tre_regfree(regex_t *preg)
|
|
{
|
|
tre_free(preg);
|
|
}
|
|
|
|
/* EOF */
|