Files
redis/deps/tre/lib/regcomp.c
T
0d9576435f Implement the new Redis Array type (#15162)
# Redis Array

For years, Redis has been missing a real indexed data structure for the
use cases where the index and the spatial relationship of elements are
semantic. Hashes give you random lookups, but you have to store an index
as a key, and have no range visibility. Lists give you appending and
trimming, but what is in the middle remains hard to access. Streams give
you append-only events, which is another (useful, indeed) beast. None of
these is what you want when the *position itself* has business meaning —
slot 37, step 4, row 18552, day from 2934 to 2949, file line 11, 12, 15
and so forth. And, all those types, for different reasons, are all
suboptimal when you want a **ring buffer** able to store the latest N
observed samples of something.

Up to now, users found ways (they always do \o/) using the fact that the
data structures that are obvious in this universe are also extremely
powerful, if well implemented. But this forces compromises. Arrays
handle these index-first requirements natively, and usually with much
better memory and CPU usage than the workarounds. If the use case is the
right one, Arrays often provide much better space, time and usability at
the same time.

## Internal encoding

1. When dense, an Array is essentially a more fancy C array. You don't
pay anything for storing the index.
2. Yet, instead of going really flat, arrays are sliced into
4096-element slices, and each slice, when it contains just a few
elements, uses a special sparse encoding. When a slice is empty it's
just a `NULL` stored in the directory.
3. Small ints, floats, and short strings are pointer-tagged, so they
cost zero additional memory beyond the pointer slot itself.
4. When very sparse, a super-directory of windowed directories is used.
This allows the data type to be safe, instead of exhibiting pathological
space or time behavior. This representation is only triggered when there
are more than 8 million elements or very high indexes set.

## Use cases

Arrays are mostly stateless if not for the fact that each array
remembers the index of the latest added item, allowing `ARINSERT` and
`ARRING` to work properly. Otherwise it is a set/get at this index game,
with solid support for both setting / getting ranges, server-side
scanning, returning only populated elements in a time which is
proportional not to the range size, but to the population size.

A few concrete examples, that may work as mental models for the set of
problems that are similar to them (from the POV of the data modeling).

**Thermometer.** A sensor reporting once per minute, with gaps:

```
ARSET       temp:room12:day7 123 22.3
ARGETRANGE  temp:room12:day7 600 660     # the 10:00–11:00 window, with NULLs
ARSCAN      temp:room12:day7 600 660     # only populated elements
AROP        temp:room12:day7 0 1439 MAX  # peak of the day, server-side
```

Missing minutes cost little to nothing. Numeric aggregation runs inside
Redis. Telemetry, IoT, meter readings, KPI rollups.

**Calendar.** A clinic with 96 fifteen-minute slots per day:

```
ARSET       sched:room12:day 32 booking:991
ARSCAN      sched:room12:day 0 95      # only occupied slots
ARGETRANGE  sched:room12:day 48 63     # the afternoon full view to render
```

The slot number is the business key in this case. Room booking, parking
spaces, warehouse bins, lockers, ...

**Ring buffer.** ARRING replaces the classic LPUSH+LTRIM pattern.
Imagine remote `dmesg`.

```
ARRING       machine:123 200 "[141087.430123]: arm_cpu_init(): cpu 14 online" # Capped to 200 entries
ARLASTITEMS  machine:123 50 REV       # 50 newest first
```

Faster than LPUSH+LTRIM, keep indexed access to past elements. Last-N
alarms, recent fraud scores, access history, remote logs, device events.
Ok here the use cases are mainly the ones of the old pattern: it is just
a better fit and allows to access random items in the middle, aggregate
server-side, and so forth.

**Workflow.** Step number is the index, value is the status. Gaps are
meaningful:

```
ARSET       claim:99172 0 received
ARSET       claim:99172 3 waiting:reviewer42
ARSET       claim:99172 5 approved
ARGETRANGE  claim:99172 0 5     # full workflow view, with NULLs for missing steps
ARSCAN      claim:99172 0 5     # only steps that have a state
ARCOUNT     claim:99172         # number of recorded steps
ARLEN       claim:99172         # highest reached step + 1
```

**Skills knowledge base for agents.** Arrays are good at representing /
grepping into Markdown files:

```
ARSET   skill:metal_gpu 0 "...."
ARSET   skill:metal_gpu 1 "...."
ARSET   skill:metal_gpu 2 "...."
ARGREP  skill:metal_gpu - + RE "M3|M4" WITHVALUES
```

ARGREP has EXACT, MATCH, GLOB, RE, you can have multiple predicates, can
select AND or OR behavior.

**Bulk import results.** Sparse row annotations over millions of rows /
CSV / ...:

```
ARSET  import:job551 18552 ERR:bad_email
ARSCAN import:job551 0 1000000        # Provides only rows that have something
```

## TLDR

If the position is part of the meaning, use an Array. If you want to
aggregate or grep remotely, use an Array. Feedback welcome :)

---------

Co-authored-by: debing.sun <debing.sun@redis.com>
Co-authored-by: Shubham S Taple <155555100+ShubhamTaple@users.noreply.github.com>
Co-authored-by: Yuan Wang <yuan.wang@redis.com>
Co-authored-by: Marc Gravell <marc.gravell@gmail.com>
2026-05-14 00:56:44 +08:00

189 lines
4.1 KiB
C

/*
tre_regcomp.c - TRE POSIX compatible regex compilation functions.
This software is released under a BSD-style license.
See the file LICENSE for details and copyright.
*/
#ifdef HAVE_CONFIG_H
#include <config.h>
#endif /* HAVE_CONFIG_H */
#include <string.h>
#include <errno.h>
#include <stdlib.h>
#include "tre-internal.h"
#include "xmalloc.h"
int
tre_regncomp(regex_t *preg, const char *regex, size_t n, int cflags)
{
int ret;
if (n > TRE_MAX_RE)
return REG_ESPACE;
#if TRE_WCHAR
tre_char_t *wregex;
size_t wlen;
wregex = xmalloc(sizeof(tre_char_t) * (n + 1));
if (wregex == NULL)
return REG_ESPACE;
/* If the current locale uses the standard single byte encoding of
characters, we don't do a multibyte string conversion. If we did,
many applications which use the default locale would break since
the default "C" locale uses the 7-bit ASCII character set, and
all characters with the eighth bit set would be considered invalid. */
#if TRE_MULTIBYTE
if (TRE_MB_CUR_MAX == 1)
#endif /* TRE_MULTIBYTE */
{
size_t i;
const unsigned char *str = (const unsigned char *)regex;
tre_char_t *wstr = wregex;
for (i = 0; i < n; i++)
*(wstr++) = *(str++);
wlen = n;
}
#if TRE_MULTIBYTE
else
{
size_t consumed;
tre_char_t *wcptr = wregex;
#ifdef HAVE_MBSTATE_T
mbstate_t state;
memset(&state, '\0', sizeof(state));
#endif /* HAVE_MBSTATE_T */
while (n > 0)
{
consumed = tre_mbrtowc(wcptr, regex, n, &state);
switch (consumed)
{
case 0:
if (*regex == '\0')
consumed = 1;
else
{
xfree(wregex);
return REG_BADPAT;
}
break;
case -1:
DPRINT(("mbrtowc: error %d: %s.\n", errno, strerror(errno)));
xfree(wregex);
return REG_BADPAT;
case -2:
/* The last character wasn't complete. Let's not call it a
fatal error. */
consumed = n;
break;
}
regex += consumed;
n -= consumed;
wcptr++;
}
wlen = wcptr - wregex;
}
#endif /* TRE_MULTIBYTE */
wregex[wlen] = L'\0';
ret = tre_compile(preg, wregex, wlen, cflags);
xfree(wregex);
#else /* !TRE_WCHAR */
ret = tre_compile(preg, (const tre_char_t *)regex, n, cflags);
#endif /* !TRE_WCHAR */
return ret;
}
/* this version takes bytes literally, to be used with raw vectors */
int
tre_regncompb(regex_t *preg, const char *regex, size_t n, int cflags)
{
int ret;
if (n > TRE_MAX_RE)
return REG_ESPACE;
#if TRE_WCHAR /* wide chars = we need to convert it all to the wide format */
tre_char_t *wregex;
size_t i;
wregex = xmalloc(sizeof(tre_char_t) * n);
if (wregex == NULL)
return REG_ESPACE;
for (i = 0; i < n; i++)
wregex[i] = (tre_char_t) ((unsigned char) regex[i]);
ret = tre_compile(preg, wregex, n, cflags | REG_USEBYTES);
xfree(wregex);
#else /* !TRE_WCHAR */
ret = tre_compile(preg, (const tre_char_t *)regex, n, cflags | REG_USEBYTES);
#endif /* !TRE_WCHAR */
return ret;
}
int
tre_regcomp(regex_t *preg, const char *regex, int cflags)
{
size_t n = regex ? strlen(regex) : 0;
if (n > TRE_MAX_RE)
return REG_ESPACE;
return tre_regncomp(preg, regex, n, cflags);
}
int
tre_regcompb(regex_t *preg, const char *regex, int cflags)
{
int ret;
tre_char_t *wregex;
size_t i, n = regex ? strlen(regex) : 0;
const unsigned char *str = (const unsigned char *)regex;
tre_char_t *wstr;
if (n > TRE_MAX_RE)
return REG_ESPACE;
wregex = xmalloc(sizeof(tre_char_t) * (n + 1));
if (wregex == NULL) return REG_ESPACE;
wstr = wregex;
for (i = 0; i < n; i++)
*(wstr++) = *(str++);
wregex[n] = L'\0';
ret = tre_compile(preg, wregex, n, cflags | REG_USEBYTES);
xfree(wregex);
return ret;
}
#ifdef TRE_WCHAR
int
tre_regwncomp(regex_t *preg, const wchar_t *regex, size_t n, int cflags)
{
if (n > TRE_MAX_RE)
return REG_ESPACE;
return tre_compile(preg, regex, n, cflags);
}
int
tre_regwcomp(regex_t *preg, const wchar_t *regex, int cflags)
{
size_t n = regex ? wcslen(regex) : 0;
if (n > TRE_MAX_RE)
return REG_ESPACE;
return tre_compile(preg, regex, n, cflags);
}
#endif /* TRE_WCHAR */
void
tre_regfree(regex_t *preg)
{
tre_free(preg);
}
/* EOF */