Compare commits
8
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
faf94027e7 | ||
|
|
40c3277ea0 | ||
|
|
ceadf083e3 | ||
|
|
f9c0832100 | ||
|
|
fb28914c5c | ||
|
|
ab92d6e177 | ||
|
|
eba8e99e5f | ||
|
|
295f37eb65 |
@@ -46,7 +46,8 @@ npm run format # Prettier
|
||||
```
|
||||
api/ # Express backend (port 3080)
|
||||
server/ # Entry point, routes, controllers, middleware
|
||||
models/ # Mongoose models (MongoDB)
|
||||
db/base/ # Hanzo Base data-layer adapter (replaces MongoDB) — see below
|
||||
models/ # Model method wrappers (persist to Base via the adapter)
|
||||
client/ # React frontend (Vite)
|
||||
src/components/ # UI components
|
||||
src/routes/ # Client-side routing
|
||||
@@ -60,6 +61,121 @@ packages/
|
||||
mcp/ # MCP server integration
|
||||
```
|
||||
|
||||
## Data Layer — Hanzo Base (MongoDB dropped)
|
||||
|
||||
Hanzo Chat does **not** use MongoDB. Persistence runs on **Hanzo Base** (the Go
|
||||
PocketBase-lineage server; SQLite embedded for dev, Postgres for prod). The
|
||||
"drop mongo completely" directive (#48) is realised by a thin adapter under
|
||||
`api/db/base/` that presents a Mongoose-Model-compatible surface backed by the
|
||||
`@hanzo/base` JS client. There is **one** adapter; every model comes along.
|
||||
|
||||
### How it works (one adapter, all models)
|
||||
`@librechat/data-schemas` builds its schemas with real `mongoose` purely as a
|
||||
schema DSL, then asks a mongoose instance to turn them into models
|
||||
(`createModels`/`createMethods`). We hand it a **facade** instead of real
|
||||
mongoose:
|
||||
|
||||
- `api/db/base/mongoose.js` — a `mongoose`-compatible facade. Delegates
|
||||
schema/ObjectId/type concerns to real mongoose (never connected) but
|
||||
overrides `model()`/`models`/`connect`/`connection`. `model(name, schema)`
|
||||
returns a Mongoose-style **constructor** (statics + `new Model(data)` docs).
|
||||
- `api/db/base/model.js` — `BaseModel`: the Mongoose Model static surface
|
||||
(`find/findOne/findOneAndUpdate/create/insertMany/updateOne/updateMany/
|
||||
deleteOne/deleteMany/countDocuments/bulkWrite/aggregate/distinct/exists`),
|
||||
a chainable/thenable `Query` (`select/sort/limit/skip/lean/populate` +
|
||||
`find(A).deleteMany(B)`), and lightweight documents (`toObject/save`).
|
||||
- `api/db/base/query.js` — a correct in-JS Mongo query/update engine
|
||||
(`$or/$and/$in/$nin/$gt../$exists/$regex/$elemMatch`, update ops
|
||||
`$set/$unset/$inc/$push/$addToSet/$pull/$setOnInsert`, projection, sort).
|
||||
This is the correctness backstop — every predicate is evaluated here.
|
||||
- `api/db/base/store.js` — the Base backend via `@hanzo/base` `BaseClient`.
|
||||
Each model → one Base collection whose records hold the **full document as a
|
||||
JSON `data` field** plus promoted scalar columns (from indexed/unique schema
|
||||
paths) used only for **filter pushdown** (a guaranteed superset; the JS
|
||||
engine then narrows). `@hanzo/base` is ESM-only → loaded via dynamic
|
||||
`import()` from this CommonJS code.
|
||||
- `api/db/base/schema.js` — introspects a Mongoose schema for defaults,
|
||||
timestamps, and promotable columns.
|
||||
- `api/db/base/index.js` — `connectDb()` inits the Base client, health-checks,
|
||||
and provisions every registered model's collection (idempotent).
|
||||
|
||||
### Wiring (all app code off mongoose)
|
||||
`api/db/{connect,models,index}.js`, `api/models/index.js`, and every remaining
|
||||
app file (`Agent`, `inviteUser`, `PermissionService`, `PermissionsController`,
|
||||
`initializeMCPs`, `migration`) import the facade (`~/db/base`) — not `mongoose`.
|
||||
The facade itself carries **no `mongoose` runtime dependency** (own ObjectId,
|
||||
Schema.Types shim, no-op session/connection). `grep "require('mongoose')"` over
|
||||
non-test `api/` is **zero**; **nothing calls `mongoose.connect`**. `_id` is a
|
||||
real 24-hex ObjectId string stored inside the document; conversations/messages
|
||||
remain keyed by their `conversationId`/`messageId` UUIDs.
|
||||
|
||||
### Search (FTS on Base/SQLite)
|
||||
MeiliSearch is dropped. `BaseModel.meiliSearch(query, {filter})` runs a
|
||||
user-scoped, case-insensitive substring search over content fields flagged
|
||||
`meiliIndex` (title, text), returning Meili-shaped `{ hits }`. `indexSync.js`
|
||||
is a no-op (no external index); `/search/enable` reports availability from Base
|
||||
(set `SEARCH=false` to disable). A future optimisation is a SQLite FTS5 index
|
||||
instead of the per-user scan.
|
||||
|
||||
### Security invariants (do not regress — from blue+red review)
|
||||
- **Prototype pollution:** `query.js` path helpers reject `__proto__` /
|
||||
`constructor` / `prototype` segments (attacker update keys flow in via
|
||||
`saveConvo` / conversation import). Keep the guard.
|
||||
- **`select:false`:** secrets (password, totpSecret, backupCodes, keyHash) are
|
||||
excluded from reads by default; only `+field` returns them. `save()` merges
|
||||
onto the stored record so a projected load never erases them.
|
||||
- **Pushdown = SUPERSET:** `store.js buildFilter` must only ever broaden; the JS
|
||||
matcher narrows. Never push a predicate that could wrongly exclude — Date
|
||||
columns and non-finite numbers are intentionally NOT pushed (Base date-column
|
||||
normalization vs ISO would silently drop records).
|
||||
- **DSL injection:** `quote()` escapes `\` and `'`; verified a crafted
|
||||
`conversationId` cannot inject `||`/operators.
|
||||
|
||||
### Uniqueness / concurrency
|
||||
Collections carry a Base **UNIQUE** index on always-present unique keys (`_id`,
|
||||
`conversationId`, `messageId`, `email`); optional/sparse unique fields
|
||||
(`googleId`, `openidId`, …) stay plain-indexed to avoid null conflicts, with
|
||||
logical uniqueness enforced by the adapter upsert. The unique index is the
|
||||
race-safety net (verified: a duplicate `conversationId` insert is rejected).
|
||||
|
||||
### Env / running
|
||||
```
|
||||
HANZO_BASE_URL=http://… # Hanzo Base instance (replaces MONGO_URI)
|
||||
HANZO_BASE_TOKEN=… # Base superuser/service token (IAM in prod)
|
||||
```
|
||||
Local Base for dev: build a pure-SQLite launcher from `~/work/hanzo/base`
|
||||
(`base.New()` without the IAM platform plugin), `serve --http=127.0.0.1:8090`;
|
||||
the first serve prints a 30-min superuser installer token. API prefix is `/v1`.
|
||||
|
||||
### Tests
|
||||
- `api/db/base/query.spec.js`, `api/db/base/model.spec.js` — jest, in-memory
|
||||
store, no Base/Mongo needed (CI). `cd api && npx jest db/base`.
|
||||
- `api/db/base/scripts/live-check.js` — real data-schemas factories vs a live
|
||||
Base: createUser + balance grant + findUser + bcrypt + saveConvo + saveMessage,
|
||||
verified via raw Base REST. Env-gated (`HANZO_BASE_URL`/`HANZO_BASE_TOKEN`).
|
||||
- `api/db/base/scripts/boot-smoke.js` — drives the wired `~/db` + `~/models`
|
||||
boot path (`connectDb` + `seedDatabase`) against live Base.
|
||||
|
||||
### Phased status
|
||||
- **Phase 1 (done):** adapter + core hot path verified on Base (User, Balance,
|
||||
Conversation, Message, Agent, plus Role/Session/Token/Category via the same
|
||||
adapter — 29 collections provisioned). Nothing connects to Mongo.
|
||||
- **Phase 2 (done):** Base/SQLite full-text search replaces MeiliSearch;
|
||||
`indexSync` no-op; dead Meili helpers removed. Live-verified real hits.
|
||||
- **Phase 3 (done):** all app code off `mongoose` (self-contained facade);
|
||||
Base UNIQUE indexes on natural keys (race-safety, live-verified);
|
||||
`select:false` secrets honored; all Date fields hydrated.
|
||||
- **Remaining tail (test migration):** ~28 legacy `*.spec.js` still build models
|
||||
on real `mongoose` + `mongodb-memory-server` (e.g. `api/models/*.spec.js`).
|
||||
They're incompatible with the port (they expect mongoose-backed models) and
|
||||
must be migrated to the adapter (in-memory store harness) — that is what gates
|
||||
dropping the `mongoose`/`mongodb-memory-server` devDeps and a literal
|
||||
`grep -riE 'mongoose|mongodb' api/` = 0. The adapter's own suites
|
||||
(`api/db/base/*.spec.js`) are the migration template.
|
||||
- **Deploy:** build image via CI → `registry.hanzo.ai`, deploy to DOKS with a
|
||||
Base instance + IAM service token (`HANZO_BASE_URL`/`HANZO_BASE_TOKEN`), then
|
||||
Playwright-verify login+chat+agent+search in the browser. Gated on cluster access.
|
||||
|
||||
## Configuration
|
||||
|
||||
- `librechat.yaml` (or ConfigMap `chat-config` -> `/app/librechat.yaml`)
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* Hanzo Base data-layer adapter — entry point.
|
||||
*
|
||||
* Exposes:
|
||||
* - `mongoose` : the mongoose-compatible facade (feed to createModels/createMethods)
|
||||
* - `connectDb()`: initialise the Base client and provision collections
|
||||
* - `store` : the underlying DocumentStore (for tests/introspection)
|
||||
*/
|
||||
|
||||
const { logger } = require('@librechat/data-schemas');
|
||||
const mongoose = require('./mongoose');
|
||||
const store = require('./store');
|
||||
|
||||
function resolveConfig() {
|
||||
const url = process.env.HANZO_BASE_URL || process.env.BASE_URL || null;
|
||||
const token = process.env.HANZO_BASE_TOKEN || process.env.BASE_TOKEN || null;
|
||||
return { url, token };
|
||||
}
|
||||
|
||||
let ready = null;
|
||||
|
||||
/**
|
||||
* Connect to Hanzo Base and ensure every registered model's collection exists.
|
||||
* Idempotent: repeated calls return the same in-flight/settled promise.
|
||||
*/
|
||||
async function connectDb() {
|
||||
if (ready) {
|
||||
return ready;
|
||||
}
|
||||
ready = (async () => {
|
||||
const { url, token } = resolveConfig();
|
||||
if (!url) {
|
||||
throw new Error(
|
||||
'[BaseAdapter] Please define HANZO_BASE_URL (Hanzo Base instance URL) for the data layer',
|
||||
);
|
||||
}
|
||||
await store.init({ url, token });
|
||||
try {
|
||||
await store.health();
|
||||
logger.info(`[BaseAdapter] Connected to Hanzo Base at ${url}`);
|
||||
} catch (err) {
|
||||
logger.error('[BaseAdapter] Base health check failed', err);
|
||||
throw err;
|
||||
}
|
||||
await ensureCollections();
|
||||
return mongoose;
|
||||
})();
|
||||
return ready;
|
||||
}
|
||||
|
||||
/** Provision the Base collection for every model registered on the facade. */
|
||||
async function ensureCollections() {
|
||||
const registered = mongoose.models;
|
||||
const names = Object.keys(registered);
|
||||
for (const name of names) {
|
||||
await registered[name].ensure();
|
||||
}
|
||||
logger.info(`[BaseAdapter] Provisioned ${names.length} Base collections`);
|
||||
}
|
||||
|
||||
module.exports = { mongoose, store, connectDb, ensureCollections };
|
||||
@@ -0,0 +1,828 @@
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* BaseModel — a Mongoose-Model-compatible facade backed by Hanzo Base.
|
||||
*
|
||||
* One instance per collection. It presents the subset of the Mongoose Model
|
||||
* static API that the LibreChat data layer actually uses (find / findOne /
|
||||
* findOneAndUpdate / create / updateOne / deleteMany / countDocuments /
|
||||
* bulkWrite / aggregate …) and executes it against a document `store`
|
||||
* (see store.js) whose records hold the full Mongo document as JSON.
|
||||
*
|
||||
* Correctness comes from query.js: candidate records are fetched from Base
|
||||
* (with best-effort filter pushdown) and every predicate, update operator,
|
||||
* projection and sort is then evaluated in JS.
|
||||
*/
|
||||
|
||||
const {
|
||||
matches,
|
||||
applyUpdate,
|
||||
parseProjection,
|
||||
applyProjection,
|
||||
sortDocs,
|
||||
getPath,
|
||||
setPath,
|
||||
idString,
|
||||
} = require('./query');
|
||||
const { describeSchema, resolveDefault } = require('./schema');
|
||||
const { generateObjectId } = require('./objectId');
|
||||
|
||||
function now() {
|
||||
return new Date();
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse a MeiliSearch-style filter string (e.g. `user = "abc"`) into a Mongo
|
||||
* equality filter for scoping. Supports the `field = "value"` / `field = 'value'`
|
||||
* clauses the chat app emits; unknown syntax is ignored (no scoping).
|
||||
*/
|
||||
function parseMeiliFilter(filter) {
|
||||
const scope = {};
|
||||
if (typeof filter !== 'string') {
|
||||
return scope;
|
||||
}
|
||||
const re = /(\w+)\s*=\s*(?:"([^"]*)"|'([^']*)')/g;
|
||||
let m;
|
||||
while ((m = re.exec(filter)) !== null) {
|
||||
scope[m[1]] = m[2] !== undefined ? m[2] : m[3];
|
||||
}
|
||||
return scope;
|
||||
}
|
||||
|
||||
/** Extract top-level equality conditions from a filter (for upsert seeding). */
|
||||
function equalitySeed(filter) {
|
||||
const seed = {};
|
||||
if (!filter || typeof filter !== 'object') {
|
||||
return seed;
|
||||
}
|
||||
for (const [k, v] of Object.entries(filter)) {
|
||||
if (k.startsWith('$')) {
|
||||
continue;
|
||||
}
|
||||
if (v === null || typeof v !== 'object' || v instanceof Date || Array.isArray(v)) {
|
||||
seed[k] = v;
|
||||
}
|
||||
}
|
||||
return seed;
|
||||
}
|
||||
|
||||
class BaseModel {
|
||||
/**
|
||||
* @param {string} name - Mongoose model name (e.g. "Conversation")
|
||||
* @param {import('mongoose').Schema} schema
|
||||
* @param {import('./store').DocumentStore} store
|
||||
*/
|
||||
constructor(name, schema, store) {
|
||||
this.modelName = name;
|
||||
this.schema = schema;
|
||||
this.store = store;
|
||||
this.collection = { collectionName: name.toLowerCase() };
|
||||
this.collectionName = name.toLowerCase();
|
||||
|
||||
const desc = describeSchema(schema);
|
||||
this._defaults = desc.defaults;
|
||||
this._timestamps = desc.timestamps;
|
||||
this._promoted = desc.promoted;
|
||||
this._promotedNames = desc.promoted.map((p) => p.name);
|
||||
this._deselected = desc.deselected;
|
||||
this._dateFields = desc.dateFields;
|
||||
this._searchable = desc.searchable;
|
||||
}
|
||||
|
||||
/** Register/verify this collection's schema in Base. Idempotent. */
|
||||
async ensure() {
|
||||
await this.store.ensureCollection(this.collectionName, this._promoted);
|
||||
}
|
||||
|
||||
// ---- internal helpers -------------------------------------------------
|
||||
|
||||
/** Build the persisted record { promoted columns…, data } from a Mongo doc. */
|
||||
_toRecord(doc) {
|
||||
const record = { data: doc };
|
||||
for (const col of this._promoted) {
|
||||
let v = idString(getPath(doc, col.name));
|
||||
if (v === undefined) {
|
||||
v = null;
|
||||
}
|
||||
if (v instanceof Date) {
|
||||
v = v.toISOString();
|
||||
}
|
||||
record[col.name] = v;
|
||||
}
|
||||
return record;
|
||||
}
|
||||
|
||||
/** Materialize a stored Mongo doc: all Date-typed fields become Date objects. */
|
||||
_hydrate(doc) {
|
||||
for (const f of this._dateFields) {
|
||||
const v = getPath(doc, f);
|
||||
// Cast ISO strings and epoch numbers (e.g. `default: Date.now`) to Date.
|
||||
if (typeof v === 'string' || typeof v === 'number') {
|
||||
setPath(doc, f, new Date(v));
|
||||
}
|
||||
}
|
||||
return doc;
|
||||
}
|
||||
|
||||
/** Apply schema defaults for keys absent from doc. */
|
||||
_applyDefaults(doc) {
|
||||
for (const [name, def] of this._defaults) {
|
||||
if (doc[name] === undefined) {
|
||||
const value = resolveDefault(def);
|
||||
if (value !== undefined) {
|
||||
doc[name] = value;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** Fetch candidate {baseId, doc} records for a filter (pushdown + always safe). */
|
||||
async _candidates(filter) {
|
||||
const raw = await this.store.list(this.collectionName, filter, this._promotedNames);
|
||||
return raw;
|
||||
}
|
||||
|
||||
_newDoc(seed) {
|
||||
const doc = { ...seed };
|
||||
if (!doc._id) {
|
||||
doc._id = generateObjectId();
|
||||
}
|
||||
this._applyDefaults(doc);
|
||||
if (this._timestamps) {
|
||||
const ts = now();
|
||||
if (doc[this._timestamps.createdAt] === undefined) {
|
||||
doc[this._timestamps.createdAt] = ts;
|
||||
}
|
||||
doc[this._timestamps.updatedAt] = ts;
|
||||
}
|
||||
return doc;
|
||||
}
|
||||
|
||||
async _insert(doc) {
|
||||
const stored = { ...doc };
|
||||
// Persist Dates as ISO strings inside the JSON blob.
|
||||
if (this._timestamps) {
|
||||
for (const f of [this._timestamps.createdAt, this._timestamps.updatedAt]) {
|
||||
if (stored[f] instanceof Date) {
|
||||
stored[f] = stored[f].toISOString();
|
||||
}
|
||||
}
|
||||
}
|
||||
const rec = await this.store.create(this.collectionName, this._toRecord(stored));
|
||||
return rec.baseId;
|
||||
}
|
||||
|
||||
async _replace(baseId, doc, { touch = true } = {}) {
|
||||
if (this._timestamps && touch) {
|
||||
doc[this._timestamps.updatedAt] = now(); // reflect fresh updatedAt on the caller's doc
|
||||
}
|
||||
const stored = { ...doc };
|
||||
if (this._timestamps) {
|
||||
for (const f of [this._timestamps.createdAt, this._timestamps.updatedAt]) {
|
||||
if (stored[f] instanceof Date) {
|
||||
stored[f] = stored[f].toISOString();
|
||||
}
|
||||
}
|
||||
}
|
||||
await this.store.update(this.collectionName, baseId, this._toRecord(stored));
|
||||
}
|
||||
|
||||
// ---- write operations -------------------------------------------------
|
||||
|
||||
async _create(input) {
|
||||
const doc = this._newDoc(input);
|
||||
await this._insert(doc);
|
||||
return this._hydrate({ ...doc });
|
||||
}
|
||||
|
||||
/** `new Model(data)` — build an unsaved document (defaults + _id) with .save(). */
|
||||
_newDocument(data = {}) {
|
||||
return makeDocument(this, this._hydrate(this._newDoc(data)));
|
||||
}
|
||||
|
||||
async create(input) {
|
||||
if (Array.isArray(input)) {
|
||||
const out = [];
|
||||
for (const item of input) {
|
||||
out.push(makeDocument(this, await this._create(item)));
|
||||
}
|
||||
return out;
|
||||
}
|
||||
return makeDocument(this, await this._create(input));
|
||||
}
|
||||
|
||||
async insertMany(docs, options = {}) {
|
||||
const created = [];
|
||||
for (const d of docs) {
|
||||
created.push(await this._create(d));
|
||||
}
|
||||
if (options.lean) {
|
||||
return created;
|
||||
}
|
||||
return created.map((d) => makeDocument(this, d));
|
||||
}
|
||||
|
||||
/**
|
||||
* Core upsert/update primitive shared by findOneAndUpdate/updateOne/updateMany.
|
||||
* @returns {{ doc: object|null, matched: number, modified: number, upserted: boolean }}
|
||||
*/
|
||||
async _updateOneInternal(filter, update, options = {}) {
|
||||
const candidates = await this._candidates(filter);
|
||||
const hit = candidates.find((c) => matches(c.doc, filter));
|
||||
if (hit) {
|
||||
const before = this._hydrate({ ...hit.doc });
|
||||
const next = { ...hit.doc };
|
||||
applyUpdate(next, update, { isInsert: false });
|
||||
await this._replace(hit.baseId, next, { touch: options.timestamps !== false });
|
||||
return { doc: this._hydrate(next), before, matched: 1, modified: 1, upserted: false };
|
||||
}
|
||||
if (options.upsert) {
|
||||
const seed = equalitySeed(filter);
|
||||
const doc = { ...seed };
|
||||
applyUpdate(doc, update, { isInsert: true });
|
||||
const full = this._newDoc(doc);
|
||||
await this._insert(full);
|
||||
return { doc: this._hydrate(full), before: null, matched: 0, modified: 0, upserted: true };
|
||||
}
|
||||
return { doc: null, before: null, matched: 0, modified: 0, upserted: false };
|
||||
}
|
||||
|
||||
findOneAndUpdate(filter, update, options = {}) {
|
||||
return new Query(this, 'findOneAndUpdate', { filter, update, options });
|
||||
}
|
||||
|
||||
findByIdAndUpdate(id, update, options = {}) {
|
||||
return this.findOneAndUpdate({ _id: id }, update, options);
|
||||
}
|
||||
|
||||
findOneAndDelete(filter) {
|
||||
return new Query(this, 'findOneAndDelete', { filter });
|
||||
}
|
||||
|
||||
async updateOne(filter, update, options = {}) {
|
||||
const r = await this._updateOneInternalPublic(filter, update, options);
|
||||
return r;
|
||||
}
|
||||
|
||||
async _updateOneInternalPublic(filter, update, options) {
|
||||
const r = await this._updateOneInternal(filter, update, options);
|
||||
return {
|
||||
acknowledged: true,
|
||||
matchedCount: r.matched,
|
||||
modifiedCount: r.modified,
|
||||
upsertedCount: r.upserted ? 1 : 0,
|
||||
upsertedId: r.upserted ? r.doc._id : null,
|
||||
};
|
||||
}
|
||||
|
||||
async updateMany(filter, update, options = {}) {
|
||||
const candidates = await this._candidates(filter);
|
||||
const hits = candidates.filter((c) => matches(c.doc, filter));
|
||||
for (const hit of hits) {
|
||||
const next = { ...hit.doc };
|
||||
applyUpdate(next, update, { isInsert: false });
|
||||
await this._replace(hit.baseId, next, { touch: options.timestamps !== false });
|
||||
}
|
||||
if (!hits.length && options.upsert) {
|
||||
const seed = equalitySeed(filter);
|
||||
const doc = { ...seed };
|
||||
applyUpdate(doc, update, { isInsert: true });
|
||||
await this._insert(this._newDoc(doc));
|
||||
return { acknowledged: true, matchedCount: 0, modifiedCount: 0, upsertedCount: 1 };
|
||||
}
|
||||
return {
|
||||
acknowledged: true,
|
||||
matchedCount: hits.length,
|
||||
modifiedCount: hits.length,
|
||||
upsertedCount: 0,
|
||||
};
|
||||
}
|
||||
|
||||
async _deleteWhere(filter, limit) {
|
||||
const candidates = await this._candidates(filter);
|
||||
const hits = candidates.filter((c) => matches(c.doc, filter));
|
||||
const toDelete = limit ? hits.slice(0, limit) : hits;
|
||||
for (const hit of toDelete) {
|
||||
await this.store.delete(this.collectionName, hit.baseId);
|
||||
}
|
||||
return { acknowledged: true, deletedCount: toDelete.length };
|
||||
}
|
||||
|
||||
deleteOne(filter) {
|
||||
return this._deleteWhere(filter, 1);
|
||||
}
|
||||
|
||||
deleteMany(filter) {
|
||||
return this._deleteWhere(filter || {}, 0);
|
||||
}
|
||||
|
||||
// ---- read operations --------------------------------------------------
|
||||
|
||||
find(filter = {}, projection, options) {
|
||||
return new Query(this, 'find', { filter, projection, options });
|
||||
}
|
||||
|
||||
findOne(filter = {}, projection, options) {
|
||||
return new Query(this, 'findOne', { filter, projection, options });
|
||||
}
|
||||
|
||||
findById(id, projection, options) {
|
||||
return new Query(this, 'findOne', { filter: { _id: id }, projection, options });
|
||||
}
|
||||
|
||||
async countDocuments(filter = {}) {
|
||||
const candidates = await this._candidates(filter);
|
||||
return candidates.filter((c) => matches(c.doc, filter)).length;
|
||||
}
|
||||
|
||||
async estimatedDocumentCount() {
|
||||
const candidates = await this._candidates({});
|
||||
return candidates.length;
|
||||
}
|
||||
|
||||
async distinct(field, filter = {}) {
|
||||
const candidates = await this._candidates(filter);
|
||||
const set = new Set();
|
||||
for (const c of candidates) {
|
||||
if (!matches(c.doc, filter)) {
|
||||
continue;
|
||||
}
|
||||
const v = getPath(c.doc, field);
|
||||
if (Array.isArray(v)) {
|
||||
v.forEach((x) => set.add(x));
|
||||
} else if (v !== undefined) {
|
||||
set.add(v);
|
||||
}
|
||||
}
|
||||
return [...set];
|
||||
}
|
||||
|
||||
async exists(filter) {
|
||||
const candidates = await this._candidates(filter);
|
||||
const hit = candidates.find((c) => matches(c.doc, filter));
|
||||
return hit ? { _id: hit.doc._id } : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Full-text search over Base/SQLite (replaces MeiliSearch).
|
||||
* Case-insensitive substring match across the model's searchable content
|
||||
* fields (title, text, …), scoped by the Meili-style `opts.filter` string
|
||||
* (e.g. `user = "abc"`). Returns MeiliSearch-shaped `{ hits }`.
|
||||
*/
|
||||
async meiliSearch(query, opts = {}) {
|
||||
const terms = (query || '').trim().toLowerCase();
|
||||
if (!terms || !this._searchable.size) {
|
||||
return { hits: [] };
|
||||
}
|
||||
const scope = parseMeiliFilter(opts.filter);
|
||||
const candidates = await this._candidates(scope);
|
||||
const hits = [];
|
||||
for (const c of candidates) {
|
||||
if (!matches(c.doc, scope)) {
|
||||
continue;
|
||||
}
|
||||
for (const field of this._searchable) {
|
||||
const value = getPath(c.doc, field);
|
||||
if (typeof value === 'string' && value.toLowerCase().includes(terms)) {
|
||||
hits.push(applyProjection(this._hydrate({ ...c.doc }), null, this._deselected));
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
return { hits };
|
||||
}
|
||||
|
||||
async syncWithMeili() {
|
||||
/* no-op: MeiliSearch sync disabled under Base adapter (Phase 2 FTS). */
|
||||
}
|
||||
|
||||
async bulkWrite(operations = []) {
|
||||
const result = {
|
||||
insertedCount: 0,
|
||||
matchedCount: 0,
|
||||
modifiedCount: 0,
|
||||
deletedCount: 0,
|
||||
upsertedCount: 0,
|
||||
upsertedIds: {},
|
||||
insertedIds: {},
|
||||
};
|
||||
for (const op of operations) {
|
||||
if (op.insertOne) {
|
||||
const doc = await this._create(op.insertOne.document);
|
||||
result.insertedIds[result.insertedCount] = doc._id;
|
||||
result.insertedCount += 1;
|
||||
} else if (op.updateOne) {
|
||||
const { filter, update, upsert, timestamps } = op.updateOne;
|
||||
const r = await this._updateOneInternal(filter, update, { upsert, timestamps });
|
||||
result.matchedCount += r.matched;
|
||||
result.modifiedCount += r.modified;
|
||||
if (r.upserted) {
|
||||
result.upsertedIds[result.upsertedCount] = r.doc._id;
|
||||
result.upsertedCount += 1;
|
||||
}
|
||||
} else if (op.updateMany) {
|
||||
const { filter, update, upsert, timestamps } = op.updateMany;
|
||||
const r = await this.updateMany(filter, update, { upsert, timestamps });
|
||||
result.matchedCount += r.matchedCount;
|
||||
result.modifiedCount += r.modifiedCount;
|
||||
result.upsertedCount += r.upsertedCount || 0;
|
||||
} else if (op.deleteOne) {
|
||||
const r = await this._deleteWhere(op.deleteOne.filter, 1);
|
||||
result.deletedCount += r.deletedCount;
|
||||
} else if (op.deleteMany) {
|
||||
const r = await this._deleteWhere(op.deleteMany.filter, 0);
|
||||
result.deletedCount += r.deletedCount;
|
||||
}
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
/** Minimal aggregation engine (used by Prompt / agentCategory, not the hot path). */
|
||||
async aggregate(pipeline = []) {
|
||||
const candidates = await this._candidates({});
|
||||
let docs = candidates.map((c) => this._hydrate({ ...c.doc }));
|
||||
for (const stage of pipeline) {
|
||||
const op = Object.keys(stage)[0];
|
||||
const arg = stage[op];
|
||||
switch (op) {
|
||||
case '$match':
|
||||
docs = docs.filter((d) => matches(d, arg));
|
||||
break;
|
||||
case '$sort':
|
||||
docs = sortDocs(docs, arg);
|
||||
break;
|
||||
case '$limit':
|
||||
docs = docs.slice(0, arg);
|
||||
break;
|
||||
case '$skip':
|
||||
docs = docs.slice(arg);
|
||||
break;
|
||||
case '$count':
|
||||
docs = [{ [arg]: docs.length }];
|
||||
break;
|
||||
case '$project':
|
||||
docs = docs.map((d) => applyProjection(d, parseProjection(arg)));
|
||||
break;
|
||||
case '$unwind': {
|
||||
const field = (typeof arg === 'string' ? arg : arg.path).replace(/^\$/, '');
|
||||
const next = [];
|
||||
for (const d of docs) {
|
||||
const v = getPath(d, field);
|
||||
if (Array.isArray(v)) {
|
||||
for (const item of v) {
|
||||
next.push({ ...d, [field]: item });
|
||||
}
|
||||
} else if (v !== undefined) {
|
||||
next.push(d);
|
||||
}
|
||||
}
|
||||
docs = next;
|
||||
break;
|
||||
}
|
||||
case '$group':
|
||||
docs = groupStage(docs, arg);
|
||||
break;
|
||||
default:
|
||||
throw new Error(`[BaseModel.aggregate] Unsupported stage: ${op}`);
|
||||
}
|
||||
}
|
||||
return docs;
|
||||
}
|
||||
}
|
||||
|
||||
/** $group implementation for the aggregation engine. */
|
||||
function groupStage(docs, spec) {
|
||||
const { _id: idSpec, ...accs } = spec;
|
||||
const groups = new Map();
|
||||
const keyOf = (d) => {
|
||||
if (idSpec === null) {
|
||||
return '__null__';
|
||||
}
|
||||
if (typeof idSpec === 'string' && idSpec.startsWith('$')) {
|
||||
return JSON.stringify(getPath(d, idSpec.slice(1)) ?? null);
|
||||
}
|
||||
return JSON.stringify(idSpec);
|
||||
};
|
||||
const idValue = (d) => {
|
||||
if (idSpec === null) {
|
||||
return null;
|
||||
}
|
||||
if (typeof idSpec === 'string' && idSpec.startsWith('$')) {
|
||||
return getPath(d, idSpec.slice(1)) ?? null;
|
||||
}
|
||||
return idSpec;
|
||||
};
|
||||
for (const d of docs) {
|
||||
const k = keyOf(d);
|
||||
if (!groups.has(k)) {
|
||||
groups.set(k, { _id: idValue(d), _docs: [] });
|
||||
}
|
||||
groups.get(k)._docs.push(d);
|
||||
}
|
||||
const out = [];
|
||||
for (const g of groups.values()) {
|
||||
const row = { _id: g._id };
|
||||
for (const [field, acc] of Object.entries(accs)) {
|
||||
const accOp = Object.keys(acc)[0];
|
||||
const expr = acc[accOp];
|
||||
const values = g._docs.map((d) =>
|
||||
typeof expr === 'string' && expr.startsWith('$') ? getPath(d, expr.slice(1)) : expr,
|
||||
);
|
||||
switch (accOp) {
|
||||
case '$sum':
|
||||
row[field] = values.reduce((s, v) => s + (Number(v) || (expr === 1 ? 1 : 0)), 0);
|
||||
break;
|
||||
case '$count':
|
||||
row[field] = g._docs.length;
|
||||
break;
|
||||
case '$avg':
|
||||
row[field] = values.reduce((s, v) => s + (Number(v) || 0), 0) / (values.length || 1);
|
||||
break;
|
||||
case '$min':
|
||||
row[field] = values.reduce((m, v) => (m == null || v < m ? v : m), null);
|
||||
break;
|
||||
case '$max':
|
||||
row[field] = values.reduce((m, v) => (m == null || v > m ? v : m), null);
|
||||
break;
|
||||
case '$first':
|
||||
row[field] = values[0];
|
||||
break;
|
||||
case '$last':
|
||||
row[field] = values[values.length - 1];
|
||||
break;
|
||||
case '$push':
|
||||
row[field] = values;
|
||||
break;
|
||||
case '$addToSet':
|
||||
row[field] = [...new Set(values)];
|
||||
break;
|
||||
default:
|
||||
throw new Error(`[BaseModel.aggregate] Unsupported accumulator: ${accOp}`);
|
||||
}
|
||||
}
|
||||
out.push(row);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Query — a lazy, chainable, thenable wrapper mirroring the Mongoose Query API
|
||||
* subset in use: select / sort / limit / skip / lean / populate / collation,
|
||||
* plus terminal execution when awaited (or via .then/.exec).
|
||||
*/
|
||||
class Query {
|
||||
constructor(model, op, params) {
|
||||
this.model = model;
|
||||
this.op = op;
|
||||
this.params = params;
|
||||
this._projection = params.projection;
|
||||
this._sort = null;
|
||||
this._limit = null;
|
||||
this._skip = null;
|
||||
this._lean = false;
|
||||
}
|
||||
|
||||
select(projection) {
|
||||
this._projection = projection;
|
||||
return this;
|
||||
}
|
||||
|
||||
sort(spec) {
|
||||
this._sort = spec;
|
||||
return this;
|
||||
}
|
||||
|
||||
limit(n) {
|
||||
this._limit = n;
|
||||
return this;
|
||||
}
|
||||
|
||||
skip(n) {
|
||||
this._skip = n;
|
||||
return this;
|
||||
}
|
||||
|
||||
lean() {
|
||||
this._lean = true;
|
||||
return this;
|
||||
}
|
||||
|
||||
populate() {
|
||||
return this; // refs are stored inline as ids; no server-side population
|
||||
}
|
||||
|
||||
collation() {
|
||||
return this;
|
||||
}
|
||||
|
||||
session() {
|
||||
return this; // Base has no transactions; sessions are inert
|
||||
}
|
||||
|
||||
hint() {
|
||||
return this;
|
||||
}
|
||||
|
||||
read() {
|
||||
return this;
|
||||
}
|
||||
|
||||
/** Merge an additional filter (Mongoose `Model.find(A).find(B)` / `.deleteMany(B)`). */
|
||||
find(extra) {
|
||||
this.params.filter = mergeFilter(this.params.filter, extra);
|
||||
return this;
|
||||
}
|
||||
|
||||
/** `Model.find(A).deleteMany(B)` — delete with merged conditions. */
|
||||
deleteMany(extra) {
|
||||
const filter = mergeFilter(this.params.filter, extra);
|
||||
return this.model._deleteWhere(filter, 0);
|
||||
}
|
||||
|
||||
deleteOne(extra) {
|
||||
const filter = mergeFilter(this.params.filter, extra);
|
||||
return this.model._deleteWhere(filter, 1);
|
||||
}
|
||||
|
||||
countDocuments() {
|
||||
return this.model.countDocuments(this.params.filter);
|
||||
}
|
||||
|
||||
async _run() {
|
||||
const model = this.model;
|
||||
if (this.op === 'findOneAndUpdate') {
|
||||
const r = await model._updateOneInternal(
|
||||
this.params.filter,
|
||||
this.params.update,
|
||||
this.params.options,
|
||||
);
|
||||
// Mongoose returns the pre-update doc when `new` is not truthy (default),
|
||||
// and the post-update doc when `{ new: true }`. Upserts with `new:false`
|
||||
// return null (there was no prior document).
|
||||
const wantNew = !!(this.params.options && this.params.options.new);
|
||||
let doc = wantNew ? r.doc : r.upserted ? null : r.before;
|
||||
if (!doc) {
|
||||
return null;
|
||||
}
|
||||
doc = applyProjection(doc, parseProjection(this._projection), model._deselected);
|
||||
return this._lean ? doc : makeDocument(model, doc);
|
||||
}
|
||||
if (this.op === 'findOneAndDelete') {
|
||||
const candidates = await model._candidates(this.params.filter);
|
||||
const hit = candidates.find((c) => matches(c.doc, this.params.filter));
|
||||
if (!hit) {
|
||||
return null;
|
||||
}
|
||||
await model.store.delete(model.collectionName, hit.baseId);
|
||||
const doc = applyProjection(
|
||||
model._hydrate({ ...hit.doc }),
|
||||
parseProjection(this._projection),
|
||||
model._deselected,
|
||||
);
|
||||
return this._lean ? doc : makeDocument(model, doc);
|
||||
}
|
||||
|
||||
// read path (find / findOne)
|
||||
const candidates = await model._candidates(this.params.filter);
|
||||
let docs = candidates.filter((c) => matches(c.doc, this.params.filter)).map((c) => c.doc);
|
||||
if (this._sort) {
|
||||
docs = sortDocs(docs, this._sort);
|
||||
}
|
||||
if (this._skip) {
|
||||
docs = docs.slice(this._skip);
|
||||
}
|
||||
if (this.op === 'findOne') {
|
||||
const doc = docs[0];
|
||||
if (!doc) {
|
||||
return null;
|
||||
}
|
||||
const out = applyProjection(
|
||||
model._hydrate({ ...doc }),
|
||||
parseProjection(this._projection),
|
||||
model._deselected,
|
||||
);
|
||||
return this._lean ? out : makeDocument(model, out);
|
||||
}
|
||||
if (this._limit != null) {
|
||||
docs = docs.slice(0, this._limit);
|
||||
}
|
||||
const projection = parseProjection(this._projection);
|
||||
const out = docs.map((d) =>
|
||||
applyProjection(model._hydrate({ ...d }), projection, model._deselected),
|
||||
);
|
||||
return this._lean ? out : out.map((d) => makeDocument(model, d));
|
||||
}
|
||||
|
||||
exec() {
|
||||
return this._run();
|
||||
}
|
||||
|
||||
then(onFulfilled, onRejected) {
|
||||
return this._run().then(onFulfilled, onRejected);
|
||||
}
|
||||
|
||||
catch(onRejected) {
|
||||
return this._run().catch(onRejected);
|
||||
}
|
||||
|
||||
finally(onFinally) {
|
||||
return this._run().finally(onFinally);
|
||||
}
|
||||
}
|
||||
|
||||
function mergeFilter(a, b) {
|
||||
if (!b || !Object.keys(b).length) {
|
||||
return a;
|
||||
}
|
||||
if (!a || !Object.keys(a).length) {
|
||||
return b;
|
||||
}
|
||||
return { $and: [a, b] };
|
||||
}
|
||||
|
||||
/**
|
||||
* Wrap a plain doc as a lightweight "document": all data fields are enumerable
|
||||
* (so spreads, `.map`, JSON all behave), while toObject/save/etc. are hidden.
|
||||
*/
|
||||
function makeDocument(model, doc) {
|
||||
if (doc == null) {
|
||||
return doc;
|
||||
}
|
||||
const defineHidden = (name, value) =>
|
||||
Object.defineProperty(doc, name, { value, enumerable: false, writable: true, configurable: true });
|
||||
|
||||
defineHidden('toObject', function toObject() {
|
||||
const plain = {};
|
||||
for (const k of Object.keys(this)) {
|
||||
plain[k] = this[k];
|
||||
}
|
||||
return plain;
|
||||
});
|
||||
defineHidden('toJSON', doc.toObject);
|
||||
defineHidden('save', async function save() {
|
||||
const candidates = await model._candidates({ _id: this._id });
|
||||
const plain = this.toObject();
|
||||
const hit = candidates.find((c) => c.doc._id === this._id);
|
||||
if (hit) {
|
||||
// Merge over the stored record so a doc loaded with a narrowed projection
|
||||
// (e.g. without select:false secrets) never erases the unloaded fields.
|
||||
await model._replace(hit.baseId, { ...hit.doc, ...plain }, { touch: true });
|
||||
} else {
|
||||
await model._insert(model._newDoc(plain));
|
||||
}
|
||||
return this;
|
||||
});
|
||||
defineHidden('deleteOne', async function deleteOne() {
|
||||
return model._deleteWhere({ _id: this._id }, 1);
|
||||
});
|
||||
if (doc._id !== undefined) {
|
||||
defineHidden('id', String(doc._id));
|
||||
}
|
||||
return doc;
|
||||
}
|
||||
|
||||
/** Static methods exposed on the constructor returned by `mongoose.model()`. */
|
||||
const MODEL_STATICS = [
|
||||
'find',
|
||||
'findOne',
|
||||
'findById',
|
||||
'findOneAndUpdate',
|
||||
'findByIdAndUpdate',
|
||||
'findOneAndDelete',
|
||||
'create',
|
||||
'insertMany',
|
||||
'updateOne',
|
||||
'updateMany',
|
||||
'deleteOne',
|
||||
'deleteMany',
|
||||
'countDocuments',
|
||||
'estimatedDocumentCount',
|
||||
'distinct',
|
||||
'exists',
|
||||
'bulkWrite',
|
||||
'aggregate',
|
||||
'meiliSearch',
|
||||
'syncWithMeili',
|
||||
'ensure',
|
||||
];
|
||||
|
||||
/**
|
||||
* Wrap a BaseModel as a Mongoose-style model constructor:
|
||||
* - `new Model(data)` creates an unsaved document (with `.save()`)
|
||||
* - `Model.find(...)`, `Model.findOneAndUpdate(...)`, … are the statics
|
||||
*/
|
||||
function makeModelCtor(baseModel) {
|
||||
function ModelCtor(data) {
|
||||
return baseModel._newDocument(data);
|
||||
}
|
||||
for (const name of MODEL_STATICS) {
|
||||
ModelCtor[name] = baseModel[name].bind(baseModel);
|
||||
}
|
||||
ModelCtor.modelName = baseModel.modelName;
|
||||
ModelCtor.schema = baseModel.schema;
|
||||
ModelCtor.collection = baseModel.collection;
|
||||
ModelCtor.base = baseModel;
|
||||
return ModelCtor;
|
||||
}
|
||||
|
||||
module.exports = { BaseModel, Query, makeDocument, makeModelCtor };
|
||||
@@ -0,0 +1,284 @@
|
||||
'use strict';
|
||||
/*
|
||||
* BaseModel behaviour over an in-memory store (no Base/Mongo required).
|
||||
* Exercises the exact Mongoose call patterns the chat hot path relies on.
|
||||
*/
|
||||
const { Schema } = require('mongoose');
|
||||
const { BaseModel } = require('./model');
|
||||
|
||||
/** In-memory backend mirroring DocumentStore's interface. */
|
||||
function memStore() {
|
||||
const cols = {};
|
||||
let seq = 0;
|
||||
const clone = (x) => JSON.parse(JSON.stringify(x));
|
||||
return {
|
||||
async ensureCollection(name) {
|
||||
cols[name] = cols[name] || new Map();
|
||||
},
|
||||
async list(name) {
|
||||
cols[name] = cols[name] || new Map();
|
||||
return [...cols[name].entries()].map(([baseId, doc]) => ({ baseId, doc: clone(doc) }));
|
||||
},
|
||||
async create(name, record) {
|
||||
cols[name] = cols[name] || new Map();
|
||||
const baseId = 'b' + ++seq;
|
||||
cols[name].set(baseId, clone(record.data));
|
||||
return { baseId, doc: clone(record.data) };
|
||||
},
|
||||
async update(name, baseId, record) {
|
||||
cols[name].set(baseId, clone(record.data));
|
||||
return { baseId, doc: clone(record.data) };
|
||||
},
|
||||
async delete(name, baseId) {
|
||||
cols[name].delete(baseId);
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
const convoSchema = new Schema(
|
||||
{
|
||||
conversationId: { type: String, unique: true, required: true, index: true },
|
||||
title: { type: String, default: 'New Chat', meiliIndex: true },
|
||||
user: { type: String, index: true },
|
||||
tags: { type: [String], default: [] },
|
||||
expiredAt: { type: Date },
|
||||
},
|
||||
{ timestamps: true },
|
||||
);
|
||||
const messageSchema = new Schema(
|
||||
{
|
||||
messageId: { type: String, unique: true, required: true, index: true },
|
||||
conversationId: { type: String, index: true, required: true },
|
||||
user: { type: String, index: true, required: true, default: null },
|
||||
text: { type: String, meiliIndex: true },
|
||||
},
|
||||
{ timestamps: true },
|
||||
);
|
||||
const userSchema = new Schema(
|
||||
{
|
||||
email: { type: String, required: true, unique: true },
|
||||
role: { type: String, default: 'USER' },
|
||||
password: { type: String, select: false },
|
||||
totpSecret: { type: String, select: false },
|
||||
},
|
||||
{ timestamps: true },
|
||||
);
|
||||
const balanceSchema = new Schema({
|
||||
user: { type: Schema.Types.ObjectId, ref: 'User', index: true, required: true },
|
||||
tokenCredits: { type: Number, default: 0 },
|
||||
expiresAt: { type: Date, default: null },
|
||||
lastRefill: { type: Date, default: Date.now },
|
||||
});
|
||||
const sessionSchema = new Schema({
|
||||
refreshTokenHash: { type: String, index: true },
|
||||
user: { type: Schema.Types.ObjectId, ref: 'User', index: true },
|
||||
expiration: { type: Date, required: true, index: true },
|
||||
});
|
||||
|
||||
let store, Conversation, Message, User, Balance, Session;
|
||||
|
||||
beforeEach(async () => {
|
||||
store = memStore();
|
||||
Conversation = new BaseModel('Conversation', convoSchema, store);
|
||||
Message = new BaseModel('Message', messageSchema, store);
|
||||
User = new BaseModel('User', userSchema, store);
|
||||
Balance = new BaseModel('Balance', balanceSchema, store);
|
||||
Session = new BaseModel('Session', sessionSchema, store);
|
||||
for (const m of [Conversation, Message, User, Balance, Session]) await m.ensure();
|
||||
});
|
||||
|
||||
describe('schema introspection', () => {
|
||||
test('promotes indexed/unique/timestamp fields and _id', () => {
|
||||
expect(Conversation._promotedNames.sort()).toEqual(
|
||||
['_id', 'conversationId', 'createdAt', 'updatedAt', 'user'].sort(),
|
||||
);
|
||||
});
|
||||
test('timestamps detected only when configured', () => {
|
||||
expect(Conversation._timestamps.createdAt).toBe('createdAt');
|
||||
expect(Balance._timestamps).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('create + defaults + ids', () => {
|
||||
test('User.create applies defaults, ObjectId _id, Date timestamps', async () => {
|
||||
const u = await User.create({ email: 'z@zoo.ngo', password: 'hashed' });
|
||||
expect(u._id).toMatch(/^[0-9a-f]{24}$/);
|
||||
expect(u.role).toBe('USER');
|
||||
expect(u.createdAt).toBeInstanceOf(Date);
|
||||
});
|
||||
});
|
||||
|
||||
describe('login-path reads', () => {
|
||||
test('findOne + select projection + lean', async () => {
|
||||
await User.create({ email: 'z@zoo.ngo', password: 'hashed' });
|
||||
const found = await User.findOne({ email: 'z@zoo.ngo' }).select('email password').lean();
|
||||
expect(found.password).toBe('hashed');
|
||||
expect(found.role).toBeUndefined();
|
||||
expect(found._id).toBeTruthy();
|
||||
});
|
||||
|
||||
test('select:false secrets (password/totpSecret) hidden by default', async () => {
|
||||
await User.create({ email: 'z@zoo.ngo', password: 'hashed', totpSecret: 'seed' });
|
||||
const def = await User.findOne({ email: 'z@zoo.ngo' }).lean();
|
||||
expect(def.password).toBeUndefined();
|
||||
expect(def.totpSecret).toBeUndefined();
|
||||
expect(def.email).toBe('z@zoo.ngo');
|
||||
// login path explicitly requests +password
|
||||
const withPw = await User.findOne({ email: 'z@zoo.ngo' }, '+password').lean();
|
||||
expect(withPw.password).toBe('hashed');
|
||||
expect(withPw.totpSecret).toBeUndefined(); // still hidden
|
||||
});
|
||||
});
|
||||
|
||||
describe('date-equality reads (H1 — no broken date pushdown)', () => {
|
||||
test('find by an exact Date value returns the record', async () => {
|
||||
const exp = new Date('2026-07-04T12:34:56.789Z');
|
||||
await Conversation.findOneAndUpdate(
|
||||
{ conversationId: 'd1', user: '507f1f77bcf86cd799439011' },
|
||||
{ $set: { expiredAt: exp } },
|
||||
{ upsert: true, new: true },
|
||||
);
|
||||
const byDate = await Conversation.find({ expiredAt: exp }).lean();
|
||||
expect(byDate.map((c) => c.conversationId)).toEqual(['d1']);
|
||||
const gt = await Conversation.find({ expiredAt: { $gt: new Date('2026-07-01T00:00:00Z') } }).lean();
|
||||
expect(gt.some((c) => c.conversationId === 'd1')).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('save() preserves select:false fields after a projected load (M2)', () => {
|
||||
test('mutate a default-loaded user + save keeps password/totpSecret', async () => {
|
||||
await User.create({ email: 'z@zoo.ngo', password: 'hashed', totpSecret: 'seed' });
|
||||
const u = await User.findOne({ email: 'z@zoo.ngo' }); // default view: secrets hidden
|
||||
expect(u.password).toBeUndefined();
|
||||
u.name = 'Renamed';
|
||||
await u.save();
|
||||
const reloaded = await User.findOne({ email: 'z@zoo.ngo' }, '+password').lean();
|
||||
expect(reloaded.password).toBe('hashed'); // not erased by save()
|
||||
expect(reloaded.name).toBe('Renamed');
|
||||
});
|
||||
});
|
||||
|
||||
describe('date hydration (non-timestamp Date fields)', () => {
|
||||
test('session.expiration and balance dates hydrate to Date', async () => {
|
||||
const exp = new Date(Date.now() + 3600_000);
|
||||
await Session.create({ user: '507f1f77bcf86cd799439011', expiration: exp });
|
||||
const s = await Session.findOne({ user: '507f1f77bcf86cd799439011' }).lean();
|
||||
expect(s.expiration).toBeInstanceOf(Date);
|
||||
expect(typeof s.expiration.getTime()).toBe('number'); // would throw on a string
|
||||
const bal = await Balance.findOneAndUpdate(
|
||||
{ user: '507f1f77bcf86cd799439011' },
|
||||
{ $inc: { tokenCredits: 1 } },
|
||||
{ upsert: true, new: true },
|
||||
).lean();
|
||||
expect(bal.lastRefill).toBeInstanceOf(Date);
|
||||
});
|
||||
});
|
||||
|
||||
describe('balance upsert', () => {
|
||||
test('$inc + $set upsert then accumulate', async () => {
|
||||
const uid = '507f1f77bcf86cd799439011';
|
||||
const bal = await Balance.findOneAndUpdate(
|
||||
{ user: uid },
|
||||
{ $inc: { tokenCredits: 1000 } },
|
||||
{ upsert: true, new: true },
|
||||
).lean();
|
||||
expect(bal.tokenCredits).toBe(1000);
|
||||
expect(String(bal.user)).toBe(uid);
|
||||
const bal2 = await Balance.findOneAndUpdate(
|
||||
{ user: uid },
|
||||
{ $inc: { tokenCredits: 500 } },
|
||||
{ new: true },
|
||||
).lean();
|
||||
expect(bal2.tokenCredits).toBe(1500);
|
||||
});
|
||||
});
|
||||
|
||||
describe('chat hot path', () => {
|
||||
const uid = '507f1f77bcf86cd799439011';
|
||||
|
||||
test('saveConvo upsert + toObject', async () => {
|
||||
const c = await Conversation.findOneAndUpdate(
|
||||
{ conversationId: 'c1', user: uid },
|
||||
{ $set: { title: 'Hello', endpoint: 'openAI' } },
|
||||
{ new: true, upsert: true },
|
||||
);
|
||||
expect(c.conversationId).toBe('c1');
|
||||
expect(c.toObject().title).toBe('Hello');
|
||||
});
|
||||
|
||||
test('saveMessage upsert + partial merge, sorted read', async () => {
|
||||
const base = Date.now();
|
||||
for (let i = 0; i < 3; i++) {
|
||||
await Message.findOneAndUpdate(
|
||||
{ messageId: 'm' + i, user: uid },
|
||||
{ conversationId: 'c1', text: 'msg' + i, createdAt: new Date(base + i * 1000) },
|
||||
{ upsert: true, new: true },
|
||||
);
|
||||
}
|
||||
await Message.findOneAndUpdate({ messageId: 'm1', user: uid }, { text: 'edited' }, { new: true });
|
||||
const msgs = await Message.find({ conversationId: 'c1', user: uid }).sort({ createdAt: 1 }).lean();
|
||||
expect(msgs.map((m) => m.messageId)).toEqual(['m0', 'm1', 'm2']);
|
||||
expect(msgs[1].text).toBe('edited');
|
||||
expect(msgs[1].conversationId).toBe('c1'); // merge preserved
|
||||
});
|
||||
|
||||
test('find(A).deleteMany(B) merges conditions', async () => {
|
||||
const base = Date.now();
|
||||
for (let i = 0; i < 3; i++) {
|
||||
await Message.create({ messageId: 'm' + i, conversationId: 'c1', user: uid, createdAt: new Date(base + i * 1000) });
|
||||
}
|
||||
const res = await Message.find({ conversationId: 'c1', user: uid }).deleteMany({
|
||||
createdAt: { $gt: new Date(base + 500) },
|
||||
});
|
||||
expect(res.deletedCount).toBe(2);
|
||||
const left = await Message.find({ conversationId: 'c1' }).lean();
|
||||
expect(left).toHaveLength(1);
|
||||
expect(left[0].messageId).toBe('m0');
|
||||
});
|
||||
|
||||
test('deleteMany $or/$exists', async () => {
|
||||
await Message.create({ messageId: 'mx', conversationId: '', user: uid });
|
||||
const res = await Message.deleteMany({
|
||||
$or: [{ conversationId: '' }, { conversationId: { $exists: false } }],
|
||||
});
|
||||
expect(res.deletedCount).toBe(1);
|
||||
});
|
||||
|
||||
test('bulkWrite updateOne upsert', async () => {
|
||||
const res = await Conversation.bulkWrite([
|
||||
{ updateOne: { filter: { conversationId: 'c1', user: uid }, update: { title: 'A' }, upsert: true } },
|
||||
{ updateOne: { filter: { conversationId: 'c2', user: uid }, update: { title: 'B' }, upsert: true } },
|
||||
]);
|
||||
expect(res.upsertedCount).toBe(2);
|
||||
expect(await Conversation.countDocuments({ conversationId: { $in: ['c1', 'c2'] } })).toBe(2);
|
||||
});
|
||||
|
||||
test('meiliSearch (Base/SQLite FTS) — title + text, user-scoped', async () => {
|
||||
const other = '507f1f77bcf86cd799439099';
|
||||
await Conversation.findOneAndUpdate({ conversationId: 'c1', user: uid }, { $set: { title: 'Quantum physics notes' } }, { upsert: true, new: true });
|
||||
await Conversation.findOneAndUpdate({ conversationId: 'c2', user: uid }, { $set: { title: 'Grocery list' } }, { upsert: true, new: true });
|
||||
await Conversation.findOneAndUpdate({ conversationId: 'c3', user: other }, { $set: { title: 'Quantum leaps' } }, { upsert: true, new: true });
|
||||
|
||||
const conv = await Conversation.meiliSearch('quantum', { filter: `user = "${uid}"` });
|
||||
expect(conv.hits.map((h) => h.conversationId)).toEqual(['c1']); // case-insensitive, user-scoped
|
||||
|
||||
await Message.create({ messageId: 'm1', conversationId: 'c1', user: uid, text: 'the WAVEFUNCTION collapses' });
|
||||
await Message.create({ messageId: 'm2', conversationId: 'c1', user: uid, text: 'unrelated chatter' });
|
||||
const msg = await Message.meiliSearch('wavefunction', { filter: `user = "${uid}"` });
|
||||
expect(msg.hits.map((h) => h.messageId)).toEqual(['m1']);
|
||||
|
||||
expect((await Conversation.meiliSearch('', { filter: `user = "${uid}"` })).hits).toEqual([]);
|
||||
});
|
||||
|
||||
test('findOneAndUpdate new:false returns pre-update doc', async () => {
|
||||
await Conversation.findOneAndUpdate({ conversationId: 'c1', user: uid }, { title: 'first' }, { upsert: true, new: true });
|
||||
const before = await Conversation.findOneAndUpdate(
|
||||
{ conversationId: 'c1', user: uid },
|
||||
{ title: 'second' },
|
||||
{ new: false },
|
||||
);
|
||||
expect(before.title).toBe('first');
|
||||
const now = await Conversation.findOne({ conversationId: 'c1' }).lean();
|
||||
expect(now.title).toBe('second');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,122 @@
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* A self-contained, `mongoose`-shaped facade backed by Hanzo Base.
|
||||
*
|
||||
* `@librechat/data-schemas` builds its schemas with real `mongoose` (a pure
|
||||
* schema DSL inside that package) and then asks a mongoose instance to turn
|
||||
* those schemas into models (`createModels`/`createMethods`). We hand it THIS
|
||||
* facade instead: it registers every model as a BaseModel persisting to Base —
|
||||
* never MongoDB — and provides just the mongoose surface the data layer uses at
|
||||
* runtime (`model`, `models`, `Types.ObjectId`, `Schema.Types`, a no-op session
|
||||
* / connection). It carries **no `mongoose` runtime dependency**; schema objects
|
||||
* are passed in already-built, so this module never imports the driver.
|
||||
*
|
||||
* Result: one adapter, and the entire data-schemas model+method surface
|
||||
* (User, Session, Token, Role, Balance, Conversation, Message, Agent, …) runs
|
||||
* on Base with zero per-model porting.
|
||||
*/
|
||||
|
||||
const { BaseModel, makeModelCtor } = require('./model');
|
||||
const { ObjectId, isValidObjectId } = require('./objectId');
|
||||
const store = require('./store');
|
||||
|
||||
/** Shared model registry (mirrors `mongoose.models`) — values are model ctors. */
|
||||
const models = {};
|
||||
/** BaseModel instances keyed by name (for connectDb collection provisioning). */
|
||||
const baseModels = {};
|
||||
|
||||
function model(name, schema) {
|
||||
if (models[name]) {
|
||||
return models[name];
|
||||
}
|
||||
if (!schema) {
|
||||
return undefined;
|
||||
}
|
||||
const base = new BaseModel(name, schema, store);
|
||||
const ctor = makeModelCtor(base);
|
||||
baseModels[name] = base;
|
||||
models[name] = ctor;
|
||||
return ctor;
|
||||
}
|
||||
|
||||
/** No-op session: Base has no multi-document transactions (see adapter notes). */
|
||||
function makeSession() {
|
||||
return {
|
||||
startTransaction() {},
|
||||
async commitTransaction() {},
|
||||
async abortTransaction() {},
|
||||
async endSession() {},
|
||||
async withTransaction(fn) {
|
||||
return fn();
|
||||
},
|
||||
inTransaction() {
|
||||
return false;
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/** Connection stub — the real Base connection lives in ./index.js connectDb(). */
|
||||
const connection = {
|
||||
readyState: 1,
|
||||
_readyState: 1,
|
||||
on() {},
|
||||
once() {},
|
||||
model,
|
||||
models,
|
||||
collections: {},
|
||||
async dropDatabase() {},
|
||||
async close() {},
|
||||
db: {
|
||||
async dropDatabase() {},
|
||||
collection() {
|
||||
return {};
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
/** Minimal Schema.Types surface for the few `mongoose.Schema.Types.*` runtime uses. */
|
||||
class Mixed {}
|
||||
const SchemaTypes = {
|
||||
ObjectId,
|
||||
Mixed,
|
||||
String,
|
||||
Number,
|
||||
Boolean,
|
||||
Date,
|
||||
Array,
|
||||
Buffer,
|
||||
Map,
|
||||
};
|
||||
|
||||
const facade = {
|
||||
__isBaseFacade: true,
|
||||
model,
|
||||
models,
|
||||
connection,
|
||||
connections: [connection],
|
||||
Types: { ObjectId, Decimal128: Number, Mixed },
|
||||
Schema: { Types: SchemaTypes },
|
||||
Model: BaseModel,
|
||||
isValidObjectId,
|
||||
async connect() {
|
||||
return facade;
|
||||
},
|
||||
createConnection() {
|
||||
return connection;
|
||||
},
|
||||
async disconnect() {},
|
||||
startSession() {
|
||||
return makeSession();
|
||||
},
|
||||
set() {
|
||||
return facade;
|
||||
},
|
||||
get() {
|
||||
return undefined;
|
||||
},
|
||||
/** Access the underlying BaseModel instances (used by connectDb). */
|
||||
baseModels,
|
||||
};
|
||||
|
||||
module.exports = facade;
|
||||
@@ -0,0 +1,93 @@
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* Mongo-compatible ObjectId generation.
|
||||
*
|
||||
* Hanzo Base stores documents keyed by its own 15-char id, but LibreChat code
|
||||
* treats `_id` as a 24-hex Mongo ObjectId string (equality, `.toString()`,
|
||||
* refs). We generate real ObjectId-shaped values so that behaviour is
|
||||
* preserved without a live MongoDB.
|
||||
*
|
||||
* Layout (12 bytes -> 24 hex): 4-byte timestamp | 5-byte process random | 3-byte counter.
|
||||
*/
|
||||
|
||||
const crypto = require('crypto');
|
||||
|
||||
const PROCESS_RANDOM = crypto.randomBytes(5);
|
||||
let COUNTER = crypto.randomBytes(3).readUIntBE(0, 3);
|
||||
|
||||
/** @returns {string} 24-char lowercase hex ObjectId string. */
|
||||
function generateObjectId() {
|
||||
const buf = Buffer.allocUnsafe(12);
|
||||
buf.writeUInt32BE(Math.floor(Date.now() / 1000), 0);
|
||||
PROCESS_RANDOM.copy(buf, 4);
|
||||
COUNTER = (COUNTER + 1) % 0xffffff;
|
||||
buf.writeUIntBE(COUNTER, 9, 3);
|
||||
return buf.toString('hex');
|
||||
}
|
||||
|
||||
const HEX_24 = /^[0-9a-fA-F]{24}$/;
|
||||
|
||||
/** @param {unknown} value @returns {boolean} */
|
||||
function isValidObjectId(value) {
|
||||
if (value == null) {
|
||||
return false;
|
||||
}
|
||||
if (typeof value === 'string') {
|
||||
return HEX_24.test(value);
|
||||
}
|
||||
if (typeof value === 'object' && typeof value.toString === 'function') {
|
||||
return HEX_24.test(value.toString());
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
/**
|
||||
* A minimal, dependency-free ObjectId compatible with the surface the chat code
|
||||
* uses from `mongoose.Types.ObjectId`: `new ObjectId()`, `new ObjectId(hex)`,
|
||||
* `.isValid()`, `.createFromHexString()`, and instances that stringify / JSON /
|
||||
* compare as their 24-hex value. Lets us drop the `mongoose` runtime dependency.
|
||||
*/
|
||||
class ObjectId {
|
||||
constructor(id) {
|
||||
if (id == null) {
|
||||
this._id = generateObjectId();
|
||||
} else if (typeof id === 'string') {
|
||||
this._id = id;
|
||||
} else if (id instanceof ObjectId) {
|
||||
this._id = id._id;
|
||||
} else if (typeof id.toHexString === 'function') {
|
||||
this._id = id.toHexString();
|
||||
} else {
|
||||
this._id = String(id);
|
||||
}
|
||||
}
|
||||
|
||||
toString() {
|
||||
return this._id;
|
||||
}
|
||||
toHexString() {
|
||||
return this._id;
|
||||
}
|
||||
toJSON() {
|
||||
return this._id;
|
||||
}
|
||||
valueOf() {
|
||||
return this._id;
|
||||
}
|
||||
equals(other) {
|
||||
if (other == null) {
|
||||
return false;
|
||||
}
|
||||
return String(typeof other.toHexString === 'function' ? other.toHexString() : other) === this._id;
|
||||
}
|
||||
|
||||
static isValid(value) {
|
||||
return isValidObjectId(value);
|
||||
}
|
||||
static createFromHexString(hex) {
|
||||
return new ObjectId(hex);
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = { generateObjectId, isValidObjectId, HEX_24, ObjectId };
|
||||
@@ -0,0 +1,537 @@
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* A small, correct MongoDB query/update engine used by the Hanzo Base adapter.
|
||||
*
|
||||
* The Base document store keeps each document as a JSON blob; filtering,
|
||||
* update-operator application, projection and sorting all run here in JS.
|
||||
* This is the correctness backstop for the whole adapter: any Mongo filter the
|
||||
* LibreChat data layer produces is evaluated here, so translation to Base's
|
||||
* filter DSL (see store.js) is only ever a best-effort pushdown optimisation.
|
||||
*/
|
||||
|
||||
/**
|
||||
* Path segments that must never be traversed or written — guards against
|
||||
* prototype pollution via attacker-controlled dotted keys (e.g. an update or
|
||||
* imported document containing `__proto__.x`). Mongoose sanitized these; the
|
||||
* adapter must too.
|
||||
*/
|
||||
const FORBIDDEN_SEGMENTS = new Set(['__proto__', 'constructor', 'prototype']);
|
||||
|
||||
/** Resolve a possibly-dotted path against a document. */
|
||||
function getPath(doc, path) {
|
||||
if (doc == null) {
|
||||
return undefined;
|
||||
}
|
||||
if (!path.includes('.')) {
|
||||
return FORBIDDEN_SEGMENTS.has(path) ? undefined : doc[path];
|
||||
}
|
||||
let cur = doc;
|
||||
for (const part of path.split('.')) {
|
||||
if (cur == null || FORBIDDEN_SEGMENTS.has(part)) {
|
||||
return undefined;
|
||||
}
|
||||
cur = cur[part];
|
||||
}
|
||||
return cur;
|
||||
}
|
||||
|
||||
/** Set a possibly-dotted path on a document (mutates). No-op on forbidden segments. */
|
||||
function setPath(doc, path, value) {
|
||||
if (!path.includes('.')) {
|
||||
if (!FORBIDDEN_SEGMENTS.has(path)) {
|
||||
doc[path] = value;
|
||||
}
|
||||
return;
|
||||
}
|
||||
const parts = path.split('.');
|
||||
if (parts.some((p) => FORBIDDEN_SEGMENTS.has(p))) {
|
||||
return;
|
||||
}
|
||||
let cur = doc;
|
||||
for (let i = 0; i < parts.length - 1; i++) {
|
||||
if (cur[parts[i]] == null || typeof cur[parts[i]] !== 'object') {
|
||||
cur[parts[i]] = {};
|
||||
}
|
||||
cur = cur[parts[i]];
|
||||
}
|
||||
cur[parts[parts.length - 1]] = value;
|
||||
}
|
||||
|
||||
/** Delete a possibly-dotted path (mutates). No-op on forbidden segments. */
|
||||
function unsetPath(doc, path) {
|
||||
if (!path.includes('.')) {
|
||||
if (!FORBIDDEN_SEGMENTS.has(path)) {
|
||||
delete doc[path];
|
||||
}
|
||||
return;
|
||||
}
|
||||
const parts = path.split('.');
|
||||
if (parts.some((p) => FORBIDDEN_SEGMENTS.has(p))) {
|
||||
return;
|
||||
}
|
||||
let cur = doc;
|
||||
for (let i = 0; i < parts.length - 1; i++) {
|
||||
if (cur[parts[i]] == null) {
|
||||
return;
|
||||
}
|
||||
cur = cur[parts[i]];
|
||||
}
|
||||
delete cur[parts[parts.length - 1]];
|
||||
}
|
||||
|
||||
/** Coerce ObjectId-like objects (Mongo ObjectId, wrappers) to their hex string. */
|
||||
function idString(value) {
|
||||
if (value && typeof value === 'object' && typeof value.toHexString === 'function') {
|
||||
return value.toHexString();
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
/** Coerce a value to a comparable primitive (Date -> ms, ObjectId -> hex). */
|
||||
function comparable(value) {
|
||||
value = idString(value);
|
||||
if (value instanceof Date) {
|
||||
return value.getTime();
|
||||
}
|
||||
if (typeof value === 'string') {
|
||||
// ISO date strings sort correctly lexicographically, but to compare against
|
||||
// Date values we normalise anything that parses as a date to ms.
|
||||
const t = Date.parse(value);
|
||||
if (!Number.isNaN(t) && /\d{4}-\d{2}-\d{2}T/.test(value)) {
|
||||
return t;
|
||||
}
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
/** Deep-ish equality sufficient for query matching (primitives, dates, arrays, plain objects). */
|
||||
function valueEquals(a, b) {
|
||||
a = idString(a);
|
||||
b = idString(b);
|
||||
if (a === b) {
|
||||
return true;
|
||||
}
|
||||
if (a instanceof Date || b instanceof Date) {
|
||||
return comparable(a) === comparable(b);
|
||||
}
|
||||
if (a == null || b == null) {
|
||||
return a == null && b == null;
|
||||
}
|
||||
if (Array.isArray(a) && Array.isArray(b)) {
|
||||
return a.length === b.length && a.every((x, i) => valueEquals(x, b[i]));
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
function compare(a, b) {
|
||||
const ca = comparable(a);
|
||||
const cb = comparable(b);
|
||||
if (ca == null && cb == null) {
|
||||
return 0;
|
||||
}
|
||||
if (ca == null) {
|
||||
return -1;
|
||||
}
|
||||
if (cb == null) {
|
||||
return 1;
|
||||
}
|
||||
if (ca < cb) {
|
||||
return -1;
|
||||
}
|
||||
if (ca > cb) {
|
||||
return 1;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
function toRegExp(spec, options) {
|
||||
if (spec instanceof RegExp) {
|
||||
return spec;
|
||||
}
|
||||
return new RegExp(spec, options || '');
|
||||
}
|
||||
|
||||
/** Evaluate a single field's operator expression against a document value. */
|
||||
function matchOperators(fieldValue, expr) {
|
||||
// Array-aware equality helper: a field matches a scalar if it equals it, or
|
||||
// (when the field is an array) if the array contains it — Mongo semantics.
|
||||
const eq = (target) => {
|
||||
if (Array.isArray(fieldValue) && !Array.isArray(target)) {
|
||||
return fieldValue.some((v) => valueEquals(v, target));
|
||||
}
|
||||
return valueEquals(fieldValue, target);
|
||||
};
|
||||
|
||||
if (expr instanceof RegExp) {
|
||||
return typeof fieldValue === 'string' && expr.test(fieldValue);
|
||||
}
|
||||
if (expr === null || typeof expr !== 'object' || expr instanceof Date || Array.isArray(expr)) {
|
||||
return eq(expr);
|
||||
}
|
||||
|
||||
const keys = Object.keys(expr);
|
||||
const isOperatorExpr = keys.some((k) => k.startsWith('$'));
|
||||
if (!isOperatorExpr) {
|
||||
return eq(expr);
|
||||
}
|
||||
|
||||
for (const op of keys) {
|
||||
const operand = expr[op];
|
||||
switch (op) {
|
||||
case '$eq':
|
||||
if (!eq(operand)) return false;
|
||||
break;
|
||||
case '$ne':
|
||||
if (eq(operand)) return false;
|
||||
break;
|
||||
case '$in': {
|
||||
const arr = operand || [];
|
||||
const hit = Array.isArray(fieldValue)
|
||||
? fieldValue.some((v) => arr.some((o) => valueEquals(v, o)))
|
||||
: arr.some((o) => valueEquals(fieldValue, o));
|
||||
if (!hit) return false;
|
||||
break;
|
||||
}
|
||||
case '$nin': {
|
||||
const arr = operand || [];
|
||||
const hit = Array.isArray(fieldValue)
|
||||
? fieldValue.some((v) => arr.some((o) => valueEquals(v, o)))
|
||||
: arr.some((o) => valueEquals(fieldValue, o));
|
||||
if (hit) return false;
|
||||
break;
|
||||
}
|
||||
case '$gt':
|
||||
if (fieldValue === undefined || compare(fieldValue, operand) <= 0) return false;
|
||||
break;
|
||||
case '$gte':
|
||||
if (fieldValue === undefined || compare(fieldValue, operand) < 0) return false;
|
||||
break;
|
||||
case '$lt':
|
||||
if (fieldValue === undefined || compare(fieldValue, operand) >= 0) return false;
|
||||
break;
|
||||
case '$lte':
|
||||
if (fieldValue === undefined || compare(fieldValue, operand) > 0) return false;
|
||||
break;
|
||||
case '$exists':
|
||||
if ((fieldValue !== undefined) !== !!operand) return false;
|
||||
break;
|
||||
case '$regex': {
|
||||
const re = toRegExp(operand, expr.$options);
|
||||
if (typeof fieldValue !== 'string' || !re.test(fieldValue)) return false;
|
||||
break;
|
||||
}
|
||||
case '$options':
|
||||
break; // handled with $regex
|
||||
case '$not':
|
||||
if (matchOperators(fieldValue, operand)) return false;
|
||||
break;
|
||||
case '$size':
|
||||
if (!Array.isArray(fieldValue) || fieldValue.length !== operand) return false;
|
||||
break;
|
||||
case '$all': {
|
||||
if (!Array.isArray(fieldValue)) return false;
|
||||
const ok = (operand || []).every((o) => fieldValue.some((v) => valueEquals(v, o)));
|
||||
if (!ok) return false;
|
||||
break;
|
||||
}
|
||||
case '$elemMatch': {
|
||||
if (!Array.isArray(fieldValue)) return false;
|
||||
if (!fieldValue.some((v) => matches(v, operand))) return false;
|
||||
break;
|
||||
}
|
||||
default:
|
||||
// Unknown operator: treat the whole expression as an equality target.
|
||||
return eq(expr);
|
||||
}
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
/**
|
||||
* Does `doc` match Mongo filter `query`?
|
||||
* @param {Record<string, unknown>} doc
|
||||
* @param {Record<string, unknown>} query
|
||||
* @returns {boolean}
|
||||
*/
|
||||
function matches(doc, query) {
|
||||
if (!query || typeof query !== 'object') {
|
||||
return true;
|
||||
}
|
||||
for (const key of Object.keys(query)) {
|
||||
const val = query[key];
|
||||
if (key === '$and') {
|
||||
if (!(val || []).every((sub) => matches(doc, sub))) return false;
|
||||
continue;
|
||||
}
|
||||
if (key === '$or') {
|
||||
if (!(val || []).some((sub) => matches(doc, sub))) return false;
|
||||
continue;
|
||||
}
|
||||
if (key === '$nor') {
|
||||
if ((val || []).some((sub) => matches(doc, sub))) return false;
|
||||
continue;
|
||||
}
|
||||
if (key === '$not') {
|
||||
if (matches(doc, val)) return false;
|
||||
continue;
|
||||
}
|
||||
if (key.startsWith('$')) {
|
||||
continue; // unsupported top-level operator -> ignore (permissive)
|
||||
}
|
||||
if (!matchOperators(getPath(doc, key), val)) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
const UPDATE_OPERATORS = new Set([
|
||||
'$set',
|
||||
'$unset',
|
||||
'$inc',
|
||||
'$push',
|
||||
'$addToSet',
|
||||
'$pull',
|
||||
'$setOnInsert',
|
||||
'$min',
|
||||
'$max',
|
||||
'$mul',
|
||||
'$rename',
|
||||
]);
|
||||
|
||||
/** Split a Mongoose-style update into operator groups, wrapping bare fields as $set. */
|
||||
function normalizeUpdate(update) {
|
||||
if (!update || typeof update !== 'object') {
|
||||
return { $set: {} };
|
||||
}
|
||||
const keys = Object.keys(update);
|
||||
const hasOperator = keys.some((k) => UPDATE_OPERATORS.has(k));
|
||||
if (!hasOperator) {
|
||||
return { $set: { ...update } };
|
||||
}
|
||||
const out = {};
|
||||
const bareSet = {};
|
||||
for (const k of keys) {
|
||||
if (UPDATE_OPERATORS.has(k)) {
|
||||
out[k] = update[k];
|
||||
} else {
|
||||
bareSet[k] = update[k];
|
||||
}
|
||||
}
|
||||
if (Object.keys(bareSet).length) {
|
||||
out.$set = { ...bareSet, ...(out.$set || {}) };
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Apply a normalized/raw update to a document in place and return it.
|
||||
* @param {Record<string, unknown>} doc - target document (mutated)
|
||||
* @param {Record<string, unknown>} update
|
||||
* @param {{ isInsert?: boolean }} [opts]
|
||||
*/
|
||||
function applyUpdate(doc, update, opts = {}) {
|
||||
const ops = normalizeUpdate(update);
|
||||
|
||||
if (ops.$set) {
|
||||
for (const [k, v] of Object.entries(ops.$set)) {
|
||||
setPath(doc, k, v);
|
||||
}
|
||||
}
|
||||
if (ops.$setOnInsert && opts.isInsert) {
|
||||
for (const [k, v] of Object.entries(ops.$setOnInsert)) {
|
||||
setPath(doc, k, v);
|
||||
}
|
||||
}
|
||||
if (ops.$unset) {
|
||||
for (const k of Object.keys(ops.$unset)) {
|
||||
unsetPath(doc, k);
|
||||
}
|
||||
}
|
||||
if (ops.$inc) {
|
||||
for (const [k, v] of Object.entries(ops.$inc)) {
|
||||
setPath(doc, k, (Number(getPath(doc, k)) || 0) + Number(v));
|
||||
}
|
||||
}
|
||||
if (ops.$mul) {
|
||||
for (const [k, v] of Object.entries(ops.$mul)) {
|
||||
setPath(doc, k, (Number(getPath(doc, k)) || 0) * Number(v));
|
||||
}
|
||||
}
|
||||
if (ops.$min) {
|
||||
for (const [k, v] of Object.entries(ops.$min)) {
|
||||
const cur = getPath(doc, k);
|
||||
if (cur === undefined || compare(v, cur) < 0) setPath(doc, k, v);
|
||||
}
|
||||
}
|
||||
if (ops.$max) {
|
||||
for (const [k, v] of Object.entries(ops.$max)) {
|
||||
const cur = getPath(doc, k);
|
||||
if (cur === undefined || compare(v, cur) > 0) setPath(doc, k, v);
|
||||
}
|
||||
}
|
||||
if (ops.$push) {
|
||||
for (const [k, v] of Object.entries(ops.$push)) {
|
||||
const arr = Array.isArray(getPath(doc, k)) ? getPath(doc, k) : [];
|
||||
if (v && typeof v === 'object' && Array.isArray(v.$each)) {
|
||||
arr.push(...v.$each);
|
||||
} else {
|
||||
arr.push(v);
|
||||
}
|
||||
setPath(doc, k, arr);
|
||||
}
|
||||
}
|
||||
if (ops.$addToSet) {
|
||||
for (const [k, v] of Object.entries(ops.$addToSet)) {
|
||||
const arr = Array.isArray(getPath(doc, k)) ? getPath(doc, k) : [];
|
||||
const items = v && typeof v === 'object' && Array.isArray(v.$each) ? v.$each : [v];
|
||||
for (const item of items) {
|
||||
if (!arr.some((x) => valueEquals(x, item))) arr.push(item);
|
||||
}
|
||||
setPath(doc, k, arr);
|
||||
}
|
||||
}
|
||||
if (ops.$pull) {
|
||||
for (const [k, cond] of Object.entries(ops.$pull)) {
|
||||
const arr = getPath(doc, k);
|
||||
if (Array.isArray(arr)) {
|
||||
setPath(
|
||||
doc,
|
||||
k,
|
||||
arr.filter((x) =>
|
||||
cond && typeof cond === 'object' && !Array.isArray(cond)
|
||||
? !matchOperators(x, cond)
|
||||
: !valueEquals(x, cond),
|
||||
),
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
if (ops.$rename) {
|
||||
for (const [from, to] of Object.entries(ops.$rename)) {
|
||||
const v = getPath(doc, from);
|
||||
unsetPath(doc, from);
|
||||
setPath(doc, to, v);
|
||||
}
|
||||
}
|
||||
return doc;
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse a projection spec (space string or object) into
|
||||
* { plain:Set, plus:Set, exclude:Set } or null for no projection.
|
||||
* - `field` -> plain inclusion
|
||||
* - `+field` -> force-include a `select:false` field on top of the default set
|
||||
* - `-field` -> exclusion
|
||||
*/
|
||||
function parseProjection(spec) {
|
||||
if (!spec) {
|
||||
return null;
|
||||
}
|
||||
const plain = new Set();
|
||||
const plus = new Set();
|
||||
const exclude = new Set();
|
||||
if (typeof spec === 'string') {
|
||||
for (const token of spec.split(/\s+/).filter(Boolean)) {
|
||||
if (token.startsWith('-')) {
|
||||
exclude.add(token.slice(1));
|
||||
} else if (token.startsWith('+')) {
|
||||
plus.add(token.slice(1));
|
||||
} else {
|
||||
plain.add(token);
|
||||
}
|
||||
}
|
||||
} else if (typeof spec === 'object') {
|
||||
for (const [k, v] of Object.entries(spec)) {
|
||||
if (v) {
|
||||
plain.add(k);
|
||||
} else {
|
||||
exclude.add(k);
|
||||
}
|
||||
}
|
||||
}
|
||||
if (!plain.size && !plus.size && !exclude.size) {
|
||||
return null;
|
||||
}
|
||||
return { plain, plus, exclude };
|
||||
}
|
||||
|
||||
/**
|
||||
* Apply a parsed projection with Mongoose `select:false` semantics, returning a
|
||||
* new object. `deselected` are fields excluded from reads by default (secrets).
|
||||
*/
|
||||
function applyProjection(doc, projection, deselected) {
|
||||
const hidden = deselected || EMPTY_SET;
|
||||
// No projection: default view = everything minus deselected (secrets).
|
||||
if (!projection) {
|
||||
if (!hidden.size) {
|
||||
return doc;
|
||||
}
|
||||
const out = { ...doc };
|
||||
for (const f of hidden) {
|
||||
unsetPath(out, f);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
// Inclusion mode: return only the named fields (+plus), _id unless -_id.
|
||||
if (projection.plain.size) {
|
||||
const fields = new Set([...projection.plain, ...projection.plus]);
|
||||
if (!projection.exclude.has('_id')) {
|
||||
fields.add('_id');
|
||||
}
|
||||
const out = {};
|
||||
for (const f of fields) {
|
||||
const v = getPath(doc, f);
|
||||
if (v !== undefined) {
|
||||
setPath(out, f, v);
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
// Default + adjustments: base minus deselected (unless +included) minus -excluded.
|
||||
const out = { ...doc };
|
||||
for (const f of hidden) {
|
||||
if (!projection.plus.has(f)) {
|
||||
unsetPath(out, f);
|
||||
}
|
||||
}
|
||||
for (const f of projection.exclude) {
|
||||
unsetPath(out, f);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
const EMPTY_SET = new Set();
|
||||
|
||||
/** Sort documents by a Mongo sort spec ({ field: 1|-1 }). Returns a new array. */
|
||||
function sortDocs(docs, sortSpec) {
|
||||
if (!sortSpec || !Object.keys(sortSpec).length) {
|
||||
return docs;
|
||||
}
|
||||
const keys = Object.entries(sortSpec).map(([k, v]) => [k, v === -1 || v === 'desc' ? -1 : 1]);
|
||||
return [...docs].sort((a, b) => {
|
||||
for (const [k, dir] of keys) {
|
||||
const c = compare(getPath(a, k), getPath(b, k));
|
||||
if (c !== 0) {
|
||||
return c * dir;
|
||||
}
|
||||
}
|
||||
return 0;
|
||||
});
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
matches,
|
||||
applyUpdate,
|
||||
normalizeUpdate,
|
||||
parseProjection,
|
||||
applyProjection,
|
||||
sortDocs,
|
||||
getPath,
|
||||
setPath,
|
||||
unsetPath,
|
||||
valueEquals,
|
||||
compare,
|
||||
idString,
|
||||
};
|
||||
@@ -0,0 +1,124 @@
|
||||
'use strict';
|
||||
const { matches, applyUpdate, parseProjection, applyProjection, sortDocs } = require('./query');
|
||||
|
||||
describe('base/query matcher', () => {
|
||||
const doc = {
|
||||
_id: 'a1',
|
||||
user: 'u1',
|
||||
conversationId: 'c1',
|
||||
tags: ['x', 'y'],
|
||||
tokenCount: 10,
|
||||
createdAt: '2026-07-04T00:00:00.000Z',
|
||||
};
|
||||
|
||||
test('equality + implicit array-contains', () => {
|
||||
expect(matches(doc, { user: 'u1' })).toBe(true);
|
||||
expect(matches(doc, { user: 'u2' })).toBe(false);
|
||||
expect(matches(doc, { tags: 'x' })).toBe(true); // array contains
|
||||
expect(matches(doc, { tags: 'z' })).toBe(false);
|
||||
});
|
||||
|
||||
test('operators $in/$nin/$ne/$exists', () => {
|
||||
expect(matches(doc, { conversationId: { $in: ['c1', 'c2'] } })).toBe(true);
|
||||
expect(matches(doc, { conversationId: { $nin: ['c2'] } })).toBe(true);
|
||||
expect(matches(doc, { user: { $ne: 'u2' } })).toBe(true);
|
||||
expect(matches(doc, { missing: { $exists: false } })).toBe(true);
|
||||
expect(matches(doc, { user: { $exists: true } })).toBe(true);
|
||||
});
|
||||
|
||||
test('comparison operators with numbers and dates', () => {
|
||||
expect(matches(doc, { tokenCount: { $gt: 5, $lte: 10 } })).toBe(true);
|
||||
expect(matches(doc, { tokenCount: { $gte: 11 } })).toBe(false);
|
||||
expect(matches(doc, { createdAt: { $gt: new Date('2026-07-03T00:00:00Z') } })).toBe(true);
|
||||
expect(matches(doc, { createdAt: { $lt: new Date('2026-07-03T00:00:00Z') } })).toBe(false);
|
||||
});
|
||||
|
||||
test('logical $or / $and / $nor', () => {
|
||||
expect(matches(doc, { $or: [{ user: 'nope' }, { conversationId: 'c1' }] })).toBe(true);
|
||||
expect(matches(doc, { $and: [{ user: 'u1' }, { conversationId: 'c1' }] })).toBe(true);
|
||||
expect(matches(doc, { $nor: [{ user: 'u1' }] })).toBe(false);
|
||||
});
|
||||
|
||||
test('$exists false / null equivalence for absent fields', () => {
|
||||
expect(matches({ a: 1 }, { $or: [{ b: null }, { b: { $exists: false } }] })).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('base/query applyUpdate', () => {
|
||||
test('$set / implicit set / $unset', () => {
|
||||
expect(applyUpdate({ a: 1 }, { $set: { b: 2 } })).toEqual({ a: 1, b: 2 });
|
||||
expect(applyUpdate({ a: 1 }, { b: 2 })).toEqual({ a: 1, b: 2 }); // implicit $set (merge)
|
||||
expect(applyUpdate({ a: 1, b: 2 }, { $unset: { b: 1 } })).toEqual({ a: 1 });
|
||||
});
|
||||
|
||||
test('$inc accumulates', () => {
|
||||
expect(applyUpdate({ n: 5 }, { $inc: { n: 3 } })).toEqual({ n: 8 });
|
||||
expect(applyUpdate({}, { $inc: { n: 2 } })).toEqual({ n: 2 });
|
||||
});
|
||||
|
||||
test('$push / $addToSet / $pull', () => {
|
||||
expect(applyUpdate({ a: [1] }, { $push: { a: 2 } })).toEqual({ a: [1, 2] });
|
||||
expect(applyUpdate({ a: [1] }, { $addToSet: { a: 1 } })).toEqual({ a: [1] });
|
||||
expect(applyUpdate({ a: [1, 2, 3] }, { $pull: { a: 2 } })).toEqual({ a: [1, 3] });
|
||||
});
|
||||
|
||||
test('$setOnInsert only applies on insert', () => {
|
||||
expect(applyUpdate({}, { $setOnInsert: { a: 1 } }, { isInsert: true })).toEqual({ a: 1 });
|
||||
expect(applyUpdate({}, { $setOnInsert: { a: 1 } }, { isInsert: false })).toEqual({});
|
||||
});
|
||||
|
||||
test('prototype pollution is blocked (dotted + bare keys, every operator)', () => {
|
||||
applyUpdate({}, { $set: { '__proto__.polluted': 'x' } });
|
||||
applyUpdate({}, { '__proto__.polluted2': 'y' }); // implicit $set
|
||||
applyUpdate({}, { $inc: { '__proto__.count': 5 } });
|
||||
applyUpdate({}, { $set: { 'constructor.prototype.polluted3': 'z' } });
|
||||
applyUpdate({}, { $set: { __proto__: { polluted4: 'w' } } });
|
||||
applyUpdate({}, { $rename: { a: '__proto__.x' } });
|
||||
expect({}.polluted).toBeUndefined();
|
||||
expect({}.polluted2).toBeUndefined();
|
||||
expect({}.count).toBeUndefined();
|
||||
expect({}.polluted3).toBeUndefined();
|
||||
expect({}.polluted4).toBeUndefined();
|
||||
expect({}.x).toBeUndefined();
|
||||
expect(Object.prototype.polluted).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe('base/query projection + sort', () => {
|
||||
const d = { _id: '1', a: 1, b: 2, c: 3, password: 'secret' };
|
||||
const hidden = new Set(['password']);
|
||||
test('inclusion projection keeps _id', () => {
|
||||
expect(applyProjection(d, parseProjection('a b'), hidden)).toEqual({ _id: '1', a: 1, b: 2 });
|
||||
});
|
||||
test('exclusion projection', () => {
|
||||
expect(applyProjection(d, parseProjection('-b -c -password'), hidden)).toEqual({ _id: '1', a: 1 });
|
||||
});
|
||||
test('select:false field hidden by default', () => {
|
||||
expect(applyProjection(d, null, hidden)).toEqual({ _id: '1', a: 1, b: 2, c: 3 });
|
||||
});
|
||||
test('+field re-includes a select:false field on top of default', () => {
|
||||
expect(applyProjection(d, parseProjection('+password'), hidden)).toEqual({
|
||||
_id: '1',
|
||||
a: 1,
|
||||
b: 2,
|
||||
c: 3,
|
||||
password: 'secret',
|
||||
});
|
||||
});
|
||||
test('explicit inclusion can return a select:false field', () => {
|
||||
expect(applyProjection(d, parseProjection('a password'), hidden)).toEqual({
|
||||
_id: '1',
|
||||
a: 1,
|
||||
password: 'secret',
|
||||
});
|
||||
});
|
||||
test('sort by multiple keys / directions', () => {
|
||||
const rows = [
|
||||
{ k: 2, t: 'b' },
|
||||
{ k: 1, t: 'a' },
|
||||
{ k: 2, t: 'a' },
|
||||
];
|
||||
expect(sortDocs(rows, { k: 1, t: 1 }).map((r) => `${r.k}${r.t}`)).toEqual(['1a', '2a', '2b']);
|
||||
expect(sortDocs(rows, { k: -1 }).map((r) => r.k)).toEqual([2, 2, 1]);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,167 @@
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* Introspects a real Mongoose Schema (built by @librechat/data-schemas as a
|
||||
* pure schema DSL — never connected to Mongo) and distills exactly what the
|
||||
* Base adapter needs:
|
||||
* - default values to apply on insert
|
||||
* - whether timestamps (createdAt/updatedAt) are managed
|
||||
* - which primitive fields to promote to real Base columns for filter/sort
|
||||
*
|
||||
* Everything else about the Mongoose schema (validators, casting, refs) is
|
||||
* intentionally ignored: the adapter trusts the application layer and stores
|
||||
* the full document as JSON.
|
||||
*/
|
||||
|
||||
const PRIMITIVE_INSTANCES = new Set(['String', 'Number', 'Date', 'Boolean', 'ObjectID', 'ObjectId']);
|
||||
|
||||
/** Identifier / scoping fields that are `meiliIndex` but not useful free-text search targets. */
|
||||
const SEARCH_STOPLIST = new Set([
|
||||
'_id',
|
||||
'_meiliIndex',
|
||||
'conversationId',
|
||||
'messageId',
|
||||
'user',
|
||||
'organization',
|
||||
'endpoint',
|
||||
'model',
|
||||
'sender',
|
||||
]);
|
||||
|
||||
/** Map a Mongoose SchemaType instance to a Base column type. */
|
||||
function baseColumnType(instance) {
|
||||
switch (instance) {
|
||||
case 'Number':
|
||||
return 'number';
|
||||
case 'Boolean':
|
||||
return 'bool';
|
||||
case 'Date':
|
||||
return 'date';
|
||||
default:
|
||||
return 'text';
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {import('mongoose').Schema} schema
|
||||
* @returns {{
|
||||
* defaults: Array<[string, unknown]>,
|
||||
* timestamps: { createdAt: string, updatedAt: string } | null,
|
||||
* promoted: Array<{ name: string, type: string, unique: boolean }>,
|
||||
* deselected: Set<string>,
|
||||
* dateFields: Set<string>,
|
||||
* }}
|
||||
*/
|
||||
function describeSchema(schema) {
|
||||
const defaults = [];
|
||||
const promoted = new Map();
|
||||
const deselected = new Set();
|
||||
const dateFields = new Set();
|
||||
const searchable = new Set();
|
||||
|
||||
// Always promote _id so document lookup by id is a real, unique Base column.
|
||||
promoted.set('_id', { name: '_id', type: 'text', unique: true });
|
||||
|
||||
const paths = schema.paths || {};
|
||||
for (const name of Object.keys(paths)) {
|
||||
if (name === '_id' || name === '__v') {
|
||||
continue;
|
||||
}
|
||||
const type = paths[name];
|
||||
const options = type.options || {};
|
||||
|
||||
// Fields marked `select: false` are excluded from reads by default
|
||||
// (Mongoose semantics) — critical for secrets: password, totpSecret,
|
||||
// backupCodes, keyHash. Only returned when explicitly `+selected`.
|
||||
if (options.select === false) {
|
||||
deselected.add(name);
|
||||
}
|
||||
|
||||
// All Date-typed paths must be re-hydrated to Date objects on read so
|
||||
// callers can safely call `.getTime()` / date methods (e.g. session.expiration).
|
||||
if (type.instance === 'Date') {
|
||||
dateFields.add(name);
|
||||
}
|
||||
|
||||
// Free-text search targets: fields flagged for MeiliSearch that are actual
|
||||
// content (title, text, …), not identifiers. Used by Base/SQLite FTS.
|
||||
if (options.meiliIndex && type.instance === 'String' && !SEARCH_STOPLIST.has(name)) {
|
||||
searchable.add(name);
|
||||
}
|
||||
|
||||
// Collect declared defaults for primitive top-level paths (no dots).
|
||||
if (!name.includes('.') && Object.prototype.hasOwnProperty.call(options, 'default')) {
|
||||
defaults.push([name, options.default]);
|
||||
}
|
||||
|
||||
// Promote indexed / unique primitive fields to Base columns for filtering.
|
||||
// A DB-level UNIQUE index is only safe for always-present unique keys
|
||||
// (unique && required) — e.g. conversationId, messageId — to avoid the
|
||||
// sparse-null conflicts of optional unique fields (googleId, openidId, …),
|
||||
// whose logical uniqueness stays enforced by the adapter-level upsert.
|
||||
if (!name.includes('.') && PRIMITIVE_INSTANCES.has(type.instance)) {
|
||||
if ((options.index || options.unique) && !promoted.has(name)) {
|
||||
const columnType = baseColumnType(type.instance);
|
||||
promoted.set(name, {
|
||||
name,
|
||||
type: columnType,
|
||||
// Full UNIQUE for always-present keys (conversationId, messageId, email).
|
||||
unique: !!(options.unique && options.required),
|
||||
// Partial UNIQUE (WHERE != '') for optional unique text keys
|
||||
// (googleId, openidId, username, …): DB-level race safety without the
|
||||
// sparse-null conflict.
|
||||
sparseUnique: !!(options.unique && !options.required && columnType === 'text'),
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Single-field indexes declared via schema.index({ field: 1 }).
|
||||
try {
|
||||
for (const [spec] of schema.indexes()) {
|
||||
const fields = Object.keys(spec || {});
|
||||
if (fields.length === 1) {
|
||||
const f = fields[0];
|
||||
if (f !== '_id' && !f.includes('.') && paths[f] && PRIMITIVE_INSTANCES.has(paths[f].instance) && !promoted.has(f)) {
|
||||
promoted.set(f, { name: f, type: baseColumnType(paths[f].instance), unique: false });
|
||||
}
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
/* schema.indexes() unavailable — ignore */
|
||||
}
|
||||
|
||||
let timestamps = null;
|
||||
const tsOpt = schema.options && schema.options.timestamps;
|
||||
if (tsOpt) {
|
||||
timestamps =
|
||||
tsOpt === true
|
||||
? { createdAt: 'createdAt', updatedAt: 'updatedAt' }
|
||||
: {
|
||||
createdAt: tsOpt.createdAt || 'createdAt',
|
||||
updatedAt: tsOpt.updatedAt || 'updatedAt',
|
||||
};
|
||||
for (const f of [timestamps.createdAt, timestamps.updatedAt]) {
|
||||
dateFields.add(f);
|
||||
if (!promoted.has(f)) {
|
||||
promoted.set(f, { name: f, type: 'date', unique: false });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
defaults,
|
||||
timestamps,
|
||||
promoted: [...promoted.values()],
|
||||
deselected,
|
||||
dateFields,
|
||||
searchable,
|
||||
};
|
||||
}
|
||||
|
||||
/** Resolve a schema default (calling it if it is a function). */
|
||||
function resolveDefault(def) {
|
||||
return typeof def === 'function' ? def() : def;
|
||||
}
|
||||
|
||||
module.exports = { describeSchema, resolveDefault, baseColumnType };
|
||||
@@ -0,0 +1,44 @@
|
||||
'use strict';
|
||||
/*
|
||||
* Boot smoke: drive the REAL wired boot entry points against Hanzo Base, the
|
||||
* same way api/server/index.js does:
|
||||
* require('~/db').connectDb() -> Base facade + collection provisioning
|
||||
* require('~/models').seedDatabase() -> roles + default categories
|
||||
*
|
||||
* Proves the wired data layer boots on Base and that non-core models
|
||||
* (Role, AgentCategory) come along through the same adapter.
|
||||
*
|
||||
* Run: HANZO_BASE_URL=... HANZO_BASE_TOKEN=... node api/db/base/__tests__/boot-smoke.js
|
||||
*/
|
||||
const path = require('path');
|
||||
require('module-alias')({ base: path.resolve(__dirname, '../../..') }); // base = api/
|
||||
|
||||
const assert = require('assert');
|
||||
|
||||
async function main() {
|
||||
const { connectDb } = require('~/db');
|
||||
const { seedDatabase } = require('~/models');
|
||||
const { Role } = require('~/db/models');
|
||||
|
||||
await connectDb();
|
||||
console.log('[boot] connectDb() OK — Base connected + collections provisioned');
|
||||
|
||||
await seedDatabase();
|
||||
console.log('[boot] seedDatabase() OK — roles + categories seeded');
|
||||
|
||||
const adminRole = await Role.findOne({ name: 'ADMIN' }).lean();
|
||||
assert(adminRole && adminRole.name === 'ADMIN', 'ADMIN role seeded to Base');
|
||||
console.log(' ok - ADMIN role present on Base:', adminRole.name);
|
||||
|
||||
const userRole = await Role.findOne({ name: 'USER' }).lean();
|
||||
assert(userRole && userRole.name === 'USER', 'USER role seeded to Base');
|
||||
console.log(' ok - USER role present on Base:', userRole.name);
|
||||
|
||||
console.log('\nBOOT SMOKE PASSED — wired data layer boots on Hanzo Base.');
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
main().catch((e) => {
|
||||
console.error('\nBOOT SMOKE FAILED:', e && e.stack ? e.stack : e);
|
||||
process.exit(1);
|
||||
});
|
||||
@@ -0,0 +1,141 @@
|
||||
'use strict';
|
||||
/*
|
||||
* LIVE proof: the REAL @librechat/data-schemas model+method factories, driven
|
||||
* through the Hanzo Base facade, persisting to a running Base instance.
|
||||
*
|
||||
* Exercises the login path (createUser + balance grant + findUser +
|
||||
* bcrypt.compare) and the chat hot path (saveConvo + saveMessage shapes),
|
||||
* then reads the documents straight back out of Base to prove they landed
|
||||
* there — not in MongoDB.
|
||||
*
|
||||
* Run: HANZO_BASE_URL=... HANZO_BASE_TOKEN=... node api/db/base/__tests__/live.js
|
||||
*/
|
||||
const assert = require('assert');
|
||||
const bcrypt = require('bcryptjs');
|
||||
const { createModels, createMethods } = require('@librechat/data-schemas');
|
||||
const { mongoose, connectDb, store } = require('..');
|
||||
|
||||
async function main() {
|
||||
const models = createModels(mongoose);
|
||||
const methods = createMethods(mongoose);
|
||||
await connectDb();
|
||||
console.log('[live] connected + collections provisioned');
|
||||
|
||||
const ok = (m) => console.log(' ok -', m);
|
||||
const email = `z+${Date.now()}@zoo.ngo`;
|
||||
const plain = 'IloveChat2026!!';
|
||||
const password = await bcrypt.hash(plain, 10);
|
||||
|
||||
// ---- LOGIN PATH ----------------------------------------------------------
|
||||
const userId = await methods.createUser(
|
||||
{ email, password, name: 'Z', username: 'z', provider: 'local' },
|
||||
{ enabled: true, startBalance: 100000 },
|
||||
true,
|
||||
false,
|
||||
);
|
||||
assert(userId, 'createUser returned an id');
|
||||
ok('createUser (User + Balance grant)');
|
||||
|
||||
// Default read hides select:false secrets (password/totpSecret) — regression guard.
|
||||
const safeUser = await methods.findUser({ email: email.toUpperCase() }); // email normalized
|
||||
assert(safeUser && safeUser.email === email, 'findUser by (normalized) email');
|
||||
assert.strictEqual(safeUser.password, undefined, 'password NOT returned by default (select:false)');
|
||||
ok('findUser default view hides the password hash (select:false honored)');
|
||||
|
||||
// Real login path selects +password explicitly (see api/strategies/localStrategy.js).
|
||||
const user = await methods.findUser({ email }, '+password');
|
||||
const match = await bcrypt.compare(plain, user.password);
|
||||
assert.strictEqual(match, true, 'bcrypt.compare succeeds — password stored hashed');
|
||||
assert.notStrictEqual(user.password, plain, 'password is NOT plaintext');
|
||||
ok('findUser(+password) + bcrypt verification (hashed, never plaintext)');
|
||||
|
||||
const balance = await models.Balance.findOne({ user: user._id }).lean();
|
||||
assert.strictEqual(balance.tokenCredits, 100000, 'balance granted via $inc upsert');
|
||||
ok('Balance record created on Base with $inc grant');
|
||||
|
||||
// ---- CHAT HOT PATH -------------------------------------------------------
|
||||
const uid = String(user._id);
|
||||
const convo = await models.Conversation.findOneAndUpdate(
|
||||
{ conversationId: 'conv-live-1', user: uid },
|
||||
{ $set: { title: 'Hello Base', endpoint: 'openAI', model: 'gpt-4o', messages: [] } },
|
||||
{ new: true, upsert: true },
|
||||
);
|
||||
assert.strictEqual(convo.conversationId, 'conv-live-1');
|
||||
assert.strictEqual(convo.title, 'Hello Base');
|
||||
ok('saveConvo — conversation upserted to Base');
|
||||
|
||||
const msg = await models.Message.findOneAndUpdate(
|
||||
{ messageId: 'msg-live-1', user: uid },
|
||||
{
|
||||
conversationId: 'conv-live-1',
|
||||
text: 'Persisted to Hanzo Base (SQLite), not Mongo.',
|
||||
sender: 'User',
|
||||
isCreatedByUser: true,
|
||||
},
|
||||
{ new: true, upsert: true },
|
||||
);
|
||||
assert.strictEqual(msg.messageId, 'msg-live-1');
|
||||
ok('saveMessage — message upserted to Base');
|
||||
|
||||
const msgs = await models.Message.find({ conversationId: 'conv-live-1', user: uid })
|
||||
.sort({ createdAt: 1 })
|
||||
.lean();
|
||||
assert.strictEqual(msgs.length, 1);
|
||||
assert.ok(msgs[0].text.includes('Hanzo Base'));
|
||||
ok('getMessages — read back from Base, sorted');
|
||||
|
||||
// ---- VERIFY THE BYTES ARE IN BASE (raw REST, bypassing the adapter) ------
|
||||
const raw = await store.client.send(
|
||||
`/v1/collections/message/records?filter=${encodeURIComponent(`messageId='msg-live-1' && user='${uid}'`)}`,
|
||||
);
|
||||
assert.ok(raw.items && raw.items.length === 1, 'message present in Base via raw REST');
|
||||
const storedDoc = raw.items[0].data;
|
||||
const parsed = typeof storedDoc === 'string' ? JSON.parse(storedDoc) : storedDoc;
|
||||
assert.strictEqual(parsed.messageId, 'msg-live-1', 'Base record.data holds the full document');
|
||||
assert.strictEqual(raw.items[0].conversationId, 'conv-live-1', 'promoted column mirrored for filter');
|
||||
ok('RAW Base REST confirms the document physically persisted to Base/SQLite');
|
||||
|
||||
// ---- FTS (Base/SQLite search replaces MeiliSearch) -----------------------
|
||||
const convoHits = await models.Conversation.meiliSearch('hello', { filter: `user = "${uid}"` });
|
||||
assert.ok(
|
||||
convoHits.hits.some((h) => h.conversationId === 'conv-live-1'),
|
||||
'conversation search returns the matching convo by title',
|
||||
);
|
||||
ok('Conversation.meiliSearch — real title hit from Base/SQLite');
|
||||
|
||||
const msgHits = await models.Message.meiliSearch('hanzo base', { filter: `user = "${uid}"` });
|
||||
assert.ok(
|
||||
msgHits.hits.some((h) => h.messageId === 'msg-live-1'),
|
||||
'message search returns the matching message by text',
|
||||
);
|
||||
assert.strictEqual(
|
||||
(await models.Conversation.meiliSearch('zzz-no-such-term', { filter: `user = "${uid}"` })).hits.length,
|
||||
0,
|
||||
'no false positives',
|
||||
);
|
||||
ok('Message.meiliSearch — real text hit from Base/SQLite (Meili dropped)');
|
||||
|
||||
// ---- Base UNIQUE index on natural key rejects a duplicate (race-safety) ----
|
||||
let rejected = false;
|
||||
try {
|
||||
await store.create('conversation', {
|
||||
_id: '0'.repeat(24),
|
||||
conversationId: 'conv-live-1', // duplicate of the one saved above
|
||||
user: uid,
|
||||
data: { _id: '0'.repeat(24), conversationId: 'conv-live-1', user: uid },
|
||||
});
|
||||
} catch (err) {
|
||||
rejected = true;
|
||||
}
|
||||
assert.ok(rejected, 'Base UNIQUE index on conversationId rejects a duplicate insert');
|
||||
ok('Base UNIQUE index enforces conversationId uniqueness (race-safety)');
|
||||
|
||||
console.log('\nLIVE proof PASSED — login + conversation + message + SEARCH on Hanzo Base.');
|
||||
console.log(` user collection id space, email=${email}`);
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
main().catch((e) => {
|
||||
console.error('\nLIVE proof FAILED:', e && e.stack ? e.stack : e);
|
||||
process.exit(1);
|
||||
});
|
||||
@@ -0,0 +1,233 @@
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* DocumentStore — the Hanzo Base backend for the Mongoose adapter.
|
||||
*
|
||||
* Each Mongo model maps to one Base collection whose records carry the full
|
||||
* document as a JSON `data` field plus a handful of promoted scalar columns
|
||||
* (derived from indexed/unique schema paths) used only for filter pushdown.
|
||||
*
|
||||
* Filter pushdown builds a Base filter-DSL string that is guaranteed to be a
|
||||
* SUPERSET of the true result; query.js then narrows it in JS, so pushdown can
|
||||
* be as conservative as needed without ever being wrong.
|
||||
*/
|
||||
|
||||
const { idString } = require('./query');
|
||||
|
||||
// `@hanzo/base` ships as an ESM-only package (exports expose only the `import`
|
||||
// condition), so it must be loaded from this CommonJS module via dynamic import.
|
||||
let _baseModulePromise = null;
|
||||
function loadBase() {
|
||||
if (!_baseModulePromise) {
|
||||
_baseModulePromise = import('@hanzo/base');
|
||||
}
|
||||
return _baseModulePromise;
|
||||
}
|
||||
|
||||
const DATA_FIELD_MAX_SIZE = 8 * 1024 * 1024; // 8 MiB per document
|
||||
|
||||
/** Escape a string value for the Base filter DSL (single-quoted). */
|
||||
function quote(value) {
|
||||
return `'${String(value).replace(/\\/g, '\\\\').replace(/'/g, "\\'")}'`;
|
||||
}
|
||||
|
||||
/** Format a scalar for the filter DSL, or return undefined if not pushable. */
|
||||
function literal(value) {
|
||||
const v = idString(value);
|
||||
if (v === null || v === undefined) {
|
||||
return undefined;
|
||||
}
|
||||
if (typeof v === 'number') {
|
||||
return Number.isFinite(v) ? String(v) : undefined; // never emit `= NaN/Infinity`
|
||||
}
|
||||
if (typeof v === 'boolean') {
|
||||
return String(v);
|
||||
}
|
||||
// Dates are intentionally NOT pushed down: Base normalizes its date columns to
|
||||
// a `YYYY-MM-DD HH:MM:SS.sssZ` (space) form that never string-matches an ISO
|
||||
// `T` literal, which would wrongly EXCLUDE the record (superset invariant
|
||||
// violation). The JS matcher (query.js) handles all date predicates.
|
||||
if (v instanceof Date) {
|
||||
return undefined;
|
||||
}
|
||||
if (typeof v === 'string') {
|
||||
return quote(v);
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* Collect conjunctive (AND-ed) constraints on promoted fields into a Base DSL
|
||||
* filter. Disjunctions and non-promoted / operator predicates are skipped —
|
||||
* the JS matcher enforces them — keeping the result a safe superset.
|
||||
*/
|
||||
function buildFilter(query, promoted) {
|
||||
const promotedSet = new Set(promoted);
|
||||
const clauses = [];
|
||||
|
||||
const walk = (q) => {
|
||||
if (!q || typeof q !== 'object') {
|
||||
return;
|
||||
}
|
||||
for (const [key, val] of Object.entries(q)) {
|
||||
if (key === '$and' && Array.isArray(val)) {
|
||||
val.forEach(walk);
|
||||
continue;
|
||||
}
|
||||
if (key.startsWith('$')) {
|
||||
continue; // $or/$nor/etc — not safe to push
|
||||
}
|
||||
if (!promotedSet.has(key)) {
|
||||
continue;
|
||||
}
|
||||
if (val === null || typeof val !== 'object' || val instanceof Date) {
|
||||
const lit = literal(val);
|
||||
if (lit !== undefined) {
|
||||
clauses.push(`${key} = ${lit}`);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (Array.isArray(val)) {
|
||||
continue;
|
||||
}
|
||||
// operator expression on a promoted field
|
||||
if (Array.isArray(val.$in) && val.$in.length) {
|
||||
const parts = val.$in.map((v) => literal(v)).filter((l) => l !== undefined);
|
||||
if (parts.length === val.$in.length) {
|
||||
clauses.push(`(${parts.map((l) => `${key} = ${l}`).join(' || ')})`);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (val.$eq !== undefined) {
|
||||
const lit = literal(val.$eq);
|
||||
if (lit !== undefined) {
|
||||
clauses.push(`${key} = ${lit}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
walk(query);
|
||||
return clauses.length ? clauses.join(' && ') : undefined;
|
||||
}
|
||||
|
||||
class DocumentStore {
|
||||
constructor() {
|
||||
this.client = null;
|
||||
this.url = null;
|
||||
this._ensured = new Set();
|
||||
}
|
||||
|
||||
/** @param {{ url: string, token?: string }} config */
|
||||
async init({ url, token }) {
|
||||
if (!url) {
|
||||
throw new Error('[BaseStore] Base URL is required (HANZO_BASE_URL / BASE_URL)');
|
||||
}
|
||||
const { BaseClient } = await loadBase();
|
||||
this.url = url.replace(/\/$/, '');
|
||||
this.client = new BaseClient(this.url);
|
||||
if (token) {
|
||||
this.client.authStore.save(token, null);
|
||||
}
|
||||
return this;
|
||||
}
|
||||
|
||||
async health() {
|
||||
return this.client.send('/v1/health');
|
||||
}
|
||||
|
||||
/** Idempotently ensure the Base collection exists with the required schema. */
|
||||
async ensureCollection(name, promoted) {
|
||||
if (this._ensured.has(name)) {
|
||||
return;
|
||||
}
|
||||
try {
|
||||
await this.client.send(`/v1/collections/${name}`, { method: 'GET' });
|
||||
this._ensured.add(name);
|
||||
return;
|
||||
} catch (err) {
|
||||
if (!isNotFound(err)) {
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
|
||||
const fields = [{ name: 'data', type: 'json', maxSize: DATA_FIELD_MAX_SIZE }];
|
||||
const indexes = [];
|
||||
for (const col of promoted) {
|
||||
fields.push({ name: col.name, type: col.type });
|
||||
const safe = col.name.replace(/[^A-Za-z0-9_]/g, '_');
|
||||
const idx = `\`idx_${name}_${safe}\` ON \`${name}\` (\`${col.name}\`)`;
|
||||
if (col.unique) {
|
||||
indexes.push(`CREATE UNIQUE INDEX ${idx}`);
|
||||
} else if (col.sparseUnique) {
|
||||
// Partial unique: allow many empty values, enforce uniqueness on real ones.
|
||||
indexes.push(`CREATE UNIQUE INDEX ${idx} WHERE \`${col.name}\` != ''`);
|
||||
} else {
|
||||
indexes.push(`CREATE INDEX ${idx}`);
|
||||
}
|
||||
}
|
||||
const body = { name, type: 'base', fields, indexes };
|
||||
try {
|
||||
await this.client.send('/v1/collections', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify(body),
|
||||
});
|
||||
} catch (err) {
|
||||
// Tolerate a concurrent creator (unique-name violation).
|
||||
if (!isConflict(err) && !isNotFound(err)) {
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
this._ensured.add(name);
|
||||
}
|
||||
|
||||
_parse(record) {
|
||||
let doc = record.data;
|
||||
if (typeof doc === 'string') {
|
||||
try {
|
||||
doc = JSON.parse(doc);
|
||||
} catch {
|
||||
doc = {};
|
||||
}
|
||||
}
|
||||
return { baseId: record.id, doc: doc || {} };
|
||||
}
|
||||
|
||||
/** List candidate docs for a Mongo filter (superset via pushdown). */
|
||||
async list(collectionName, mongoFilter, promotedNames) {
|
||||
const filter = buildFilter(mongoFilter, promotedNames || []);
|
||||
const items = await this.client
|
||||
.collection(collectionName)
|
||||
.getFullList(filter ? { filter } : {});
|
||||
return items.map((r) => this._parse(r));
|
||||
}
|
||||
|
||||
async create(collectionName, record) {
|
||||
const created = await this.client.collection(collectionName).create(record);
|
||||
return this._parse(created);
|
||||
}
|
||||
|
||||
async update(collectionName, baseId, record) {
|
||||
const updated = await this.client.collection(collectionName).update(baseId, record);
|
||||
return this._parse(updated);
|
||||
}
|
||||
|
||||
async delete(collectionName, baseId) {
|
||||
await this.client.collection(collectionName).delete(baseId);
|
||||
}
|
||||
}
|
||||
|
||||
function statusOf(err) {
|
||||
return (err && (err.status || (err.data && err.data.status))) || 0;
|
||||
}
|
||||
function isNotFound(err) {
|
||||
return statusOf(err) === 404;
|
||||
}
|
||||
function isConflict(err) {
|
||||
return statusOf(err) === 400 || statusOf(err) === 409;
|
||||
}
|
||||
|
||||
module.exports = new DocumentStore();
|
||||
module.exports.DocumentStore = DocumentStore;
|
||||
module.exports.buildFilter = buildFilter;
|
||||
@@ -0,0 +1,43 @@
|
||||
'use strict';
|
||||
/* buildFilter — the Base filter-DSL pushdown. Must only ever emit a SUPERSET. */
|
||||
const { buildFilter } = require('./store');
|
||||
|
||||
const promoted = ['_id', 'conversationId', 'user', 'expiredAt', 'tokenCount'];
|
||||
|
||||
describe('store/buildFilter pushdown', () => {
|
||||
test('promoted equality + $in', () => {
|
||||
expect(buildFilter({ conversationId: 'c1', user: 'u1' }, promoted)).toBe(
|
||||
"conversationId = 'c1' && user = 'u1'",
|
||||
);
|
||||
expect(buildFilter({ conversationId: { $in: ['a', 'b'] } }, promoted)).toBe(
|
||||
"(conversationId = 'a' || conversationId = 'b')",
|
||||
);
|
||||
});
|
||||
|
||||
test('DSL injection cannot break out of the quoted literal', () => {
|
||||
const f = buildFilter({ conversationId: "x' || user='victim" }, promoted);
|
||||
expect(f).toBe("conversationId = 'x\\' || user=\\'victim'");
|
||||
expect(f).not.toMatch(/ \|\| user='victim/); // no un-escaped injected clause
|
||||
});
|
||||
|
||||
test('Date predicates are NOT pushed down (superset invariant — H1)', () => {
|
||||
// Base date columns normalize to a space separator; pushing an ISO literal
|
||||
// would wrongly EXCLUDE the record. Dates must fall through to the JS matcher.
|
||||
expect(buildFilter({ expiredAt: new Date('2026-07-04T12:00:00Z') }, promoted)).toBeUndefined();
|
||||
expect(buildFilter({ expiredAt: { $eq: new Date() } }, promoted)).toBeUndefined();
|
||||
// a date predicate must never partially push while dropping the date clause
|
||||
expect(buildFilter({ user: 'u1', expiredAt: new Date() }, promoted)).toBe("user = 'u1'");
|
||||
});
|
||||
|
||||
test('non-finite numbers are not pushed (invalid DSL guard — L1)', () => {
|
||||
expect(buildFilter({ tokenCount: NaN }, promoted)).toBeUndefined();
|
||||
expect(buildFilter({ tokenCount: Infinity }, promoted)).toBeUndefined();
|
||||
expect(buildFilter({ tokenCount: 5 }, promoted)).toBe('tokenCount = 5');
|
||||
});
|
||||
|
||||
test('non-promoted / $or / empty-$in are not pushed (JS matcher handles them)', () => {
|
||||
expect(buildFilter({ title: 'x' }, promoted)).toBeUndefined();
|
||||
expect(buildFilter({ $or: [{ user: 'a' }, { user: 'b' }] }, promoted)).toBeUndefined();
|
||||
expect(buildFilter({ conversationId: { $in: [] } }, promoted)).toBeUndefined();
|
||||
});
|
||||
});
|
||||
+7
-80
@@ -1,83 +1,10 @@
|
||||
require('dotenv').config();
|
||||
const { isEnabled } = require('@hanzochat/api');
|
||||
const { logger } = require('@librechat/data-schemas');
|
||||
|
||||
const mongoose = require('mongoose');
|
||||
const MONGO_URI = process.env.MONGO_URI;
|
||||
|
||||
if (!MONGO_URI) {
|
||||
throw new Error('Please define the MONGO_URI environment variable');
|
||||
}
|
||||
/** The maximum number of connections in the connection pool. */
|
||||
const maxPoolSize = parseInt(process.env.MONGO_MAX_POOL_SIZE) || undefined;
|
||||
/** The minimum number of connections in the connection pool. */
|
||||
const minPoolSize = parseInt(process.env.MONGO_MIN_POOL_SIZE) || undefined;
|
||||
/** The maximum number of connections that may be in the process of being established concurrently by the connection pool. */
|
||||
const maxConnecting = parseInt(process.env.MONGO_MAX_CONNECTING) || undefined;
|
||||
/** The maximum number of milliseconds that a connection can remain idle in the pool before being removed and closed. */
|
||||
const maxIdleTimeMS = parseInt(process.env.MONGO_MAX_IDLE_TIME_MS) || undefined;
|
||||
/** The maximum time in milliseconds that a thread can wait for a connection to become available. */
|
||||
const waitQueueTimeoutMS = parseInt(process.env.MONGO_WAIT_QUEUE_TIMEOUT_MS) || undefined;
|
||||
/** Set to false to disable automatic index creation for all models associated with this connection. */
|
||||
const autoIndex =
|
||||
process.env.MONGO_AUTO_INDEX != undefined
|
||||
? isEnabled(process.env.MONGO_AUTO_INDEX) || false
|
||||
: undefined;
|
||||
|
||||
/** Set to `false` to disable Mongoose automatically calling `createCollection()` on every model created on this connection. */
|
||||
const autoCreate =
|
||||
process.env.MONGO_AUTO_CREATE != undefined
|
||||
? isEnabled(process.env.MONGO_AUTO_CREATE) || false
|
||||
: undefined;
|
||||
/**
|
||||
* Global is used here to maintain a cached connection across hot reloads
|
||||
* in development. This prevents connections growing exponentially
|
||||
* during API Route usage.
|
||||
* Database connection.
|
||||
*
|
||||
* MongoDB has been dropped from Hanzo Chat — the data layer runs on Hanzo Base
|
||||
* (SQLite embedded / Postgres for prod multi-instance). `connectDb` initialises
|
||||
* the Base client and provisions every registered model's collection. See ./base.
|
||||
*/
|
||||
let cached = global.mongoose;
|
||||
const { connectDb } = require('./base');
|
||||
|
||||
if (!cached) {
|
||||
cached = global.mongoose = { conn: null, promise: null };
|
||||
}
|
||||
|
||||
mongoose.connection.on('error', (err) => {
|
||||
logger.error('[connectDb] MongoDB connection error:', err);
|
||||
});
|
||||
|
||||
async function connectDb() {
|
||||
if (cached.conn && cached.conn?._readyState === 1) {
|
||||
return cached.conn;
|
||||
}
|
||||
|
||||
const disconnected = cached.conn && cached.conn?._readyState !== 1;
|
||||
if (!cached.promise || disconnected) {
|
||||
const opts = {
|
||||
bufferCommands: false,
|
||||
...(maxPoolSize ? { maxPoolSize } : {}),
|
||||
...(minPoolSize ? { minPoolSize } : {}),
|
||||
...(maxConnecting ? { maxConnecting } : {}),
|
||||
...(maxIdleTimeMS ? { maxIdleTimeMS } : {}),
|
||||
...(waitQueueTimeoutMS ? { waitQueueTimeoutMS } : {}),
|
||||
...(autoIndex != undefined ? { autoIndex } : {}),
|
||||
...(autoCreate != undefined ? { autoCreate } : {}),
|
||||
// useNewUrlParser: true,
|
||||
// useUnifiedTopology: true,
|
||||
// bufferMaxEntries: 0,
|
||||
// useFindAndModify: true,
|
||||
// useCreateIndex: true
|
||||
};
|
||||
logger.info('Mongo Connection options');
|
||||
logger.info(JSON.stringify(opts, null, 2));
|
||||
mongoose.set('strictQuery', true);
|
||||
cached.promise = mongoose.connect(MONGO_URI, opts).then((mongoose) => {
|
||||
return mongoose;
|
||||
});
|
||||
}
|
||||
cached.conn = await cached.promise;
|
||||
|
||||
return cached.conn;
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
connectDb,
|
||||
};
|
||||
module.exports = { connectDb };
|
||||
|
||||
+2
-2
@@ -1,8 +1,8 @@
|
||||
const mongoose = require('mongoose');
|
||||
const { mongoose, connectDb } = require('./base');
|
||||
const { createModels } = require('@librechat/data-schemas');
|
||||
const { connectDb } = require('./connect');
|
||||
const indexSync = require('./indexSync');
|
||||
|
||||
// Register every model on the Base-backed mongoose facade.
|
||||
createModels(mongoose);
|
||||
|
||||
module.exports = { connectDb, indexSync };
|
||||
|
||||
+8
-355
@@ -1,363 +1,16 @@
|
||||
const mongoose = require('mongoose');
|
||||
const { MeiliSearch } = require('meilisearch');
|
||||
const { logger } = require('@librechat/data-schemas');
|
||||
const { CacheKeys } = require('librechat-data-provider');
|
||||
const { isEnabled, FlowStateManager } = require('@hanzochat/api');
|
||||
const { getLogStores } = require('~/cache');
|
||||
const { batchResetMeiliFlags } = require('./utils');
|
||||
|
||||
const Conversation = mongoose.models.Conversation;
|
||||
const Message = mongoose.models.Message;
|
||||
|
||||
const searchEnabled = isEnabled(process.env.SEARCH);
|
||||
const indexingDisabled = isEnabled(process.env.MEILI_NO_SYNC);
|
||||
let currentTimeout = null;
|
||||
|
||||
const defaultSyncThreshold = 1000;
|
||||
const syncThreshold = process.env.MEILI_SYNC_THRESHOLD
|
||||
? parseInt(process.env.MEILI_SYNC_THRESHOLD, 10)
|
||||
: defaultSyncThreshold;
|
||||
|
||||
class MeiliSearchClient {
|
||||
static instance = null;
|
||||
|
||||
static getInstance() {
|
||||
if (!MeiliSearchClient.instance) {
|
||||
if (!process.env.MEILI_HOST || !process.env.MEILI_MASTER_KEY) {
|
||||
throw new Error('Meilisearch configuration is missing.');
|
||||
}
|
||||
MeiliSearchClient.instance = new MeiliSearch({
|
||||
host: process.env.MEILI_HOST,
|
||||
apiKey: process.env.MEILI_MASTER_KEY,
|
||||
});
|
||||
}
|
||||
return MeiliSearchClient.instance;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Deletes documents from MeiliSearch index that are missing the user field
|
||||
* @param {import('meilisearch').Index} index - MeiliSearch index instance
|
||||
* @param {string} indexName - Name of the index for logging
|
||||
* @returns {Promise<number>} - Number of documents deleted
|
||||
*/
|
||||
async function deleteDocumentsWithoutUserField(index, indexName) {
|
||||
let deletedCount = 0;
|
||||
let offset = 0;
|
||||
const batchSize = 1000;
|
||||
|
||||
try {
|
||||
while (true) {
|
||||
const searchResult = await index.search('', {
|
||||
limit: batchSize,
|
||||
offset: offset,
|
||||
});
|
||||
|
||||
if (searchResult.hits.length === 0) {
|
||||
break;
|
||||
}
|
||||
|
||||
const idsToDelete = searchResult.hits.filter((hit) => !hit.user).map((hit) => hit.id);
|
||||
|
||||
if (idsToDelete.length > 0) {
|
||||
logger.info(
|
||||
`[indexSync] Deleting ${idsToDelete.length} documents without user field from ${indexName} index`,
|
||||
);
|
||||
await index.deleteDocuments(idsToDelete);
|
||||
deletedCount += idsToDelete.length;
|
||||
}
|
||||
|
||||
if (searchResult.hits.length < batchSize) {
|
||||
break;
|
||||
}
|
||||
|
||||
offset += batchSize;
|
||||
}
|
||||
|
||||
if (deletedCount > 0) {
|
||||
logger.info(`[indexSync] Deleted ${deletedCount} orphaned documents from ${indexName} index`);
|
||||
}
|
||||
} catch (error) {
|
||||
logger.error(`[indexSync] Error deleting documents from ${indexName}:`, error);
|
||||
}
|
||||
|
||||
return deletedCount;
|
||||
}
|
||||
|
||||
/**
|
||||
* Ensures indexes have proper filterable attributes configured and checks if documents have user field
|
||||
* @param {MeiliSearch} client - MeiliSearch client instance
|
||||
* @returns {Promise<{settingsUpdated: boolean, orphanedDocsFound: boolean}>} - Status of what was done
|
||||
*/
|
||||
async function ensureFilterableAttributes(client) {
|
||||
let settingsUpdated = false;
|
||||
let hasOrphanedDocs = false;
|
||||
|
||||
try {
|
||||
// Check and update messages index
|
||||
try {
|
||||
const messagesIndex = client.index('messages');
|
||||
const settings = await messagesIndex.getSettings();
|
||||
|
||||
if (!settings.filterableAttributes || !settings.filterableAttributes.includes('user')) {
|
||||
logger.info('[indexSync] Configuring messages index to filter by user...');
|
||||
await messagesIndex.updateSettings({
|
||||
filterableAttributes: ['user'],
|
||||
});
|
||||
logger.info('[indexSync] Messages index configured for user filtering');
|
||||
settingsUpdated = true;
|
||||
}
|
||||
|
||||
// Check if existing documents have user field indexed
|
||||
try {
|
||||
const searchResult = await messagesIndex.search('', { limit: 1 });
|
||||
if (searchResult.hits.length > 0 && !searchResult.hits[0].user) {
|
||||
logger.info(
|
||||
'[indexSync] Existing messages missing user field, will clean up orphaned documents...',
|
||||
);
|
||||
hasOrphanedDocs = true;
|
||||
}
|
||||
} catch (searchError) {
|
||||
logger.debug('[indexSync] Could not check message documents:', searchError.message);
|
||||
}
|
||||
} catch (error) {
|
||||
if (error.code !== 'index_not_found') {
|
||||
logger.warn('[indexSync] Could not check/update messages index settings:', error.message);
|
||||
}
|
||||
}
|
||||
|
||||
// Check and update conversations index
|
||||
try {
|
||||
const convosIndex = client.index('convos');
|
||||
const settings = await convosIndex.getSettings();
|
||||
|
||||
if (!settings.filterableAttributes || !settings.filterableAttributes.includes('user')) {
|
||||
logger.info('[indexSync] Configuring convos index to filter by user...');
|
||||
await convosIndex.updateSettings({
|
||||
filterableAttributes: ['user'],
|
||||
});
|
||||
logger.info('[indexSync] Convos index configured for user filtering');
|
||||
settingsUpdated = true;
|
||||
}
|
||||
|
||||
// Check if existing documents have user field indexed
|
||||
try {
|
||||
const searchResult = await convosIndex.search('', { limit: 1 });
|
||||
if (searchResult.hits.length > 0 && !searchResult.hits[0].user) {
|
||||
logger.info(
|
||||
'[indexSync] Existing conversations missing user field, will clean up orphaned documents...',
|
||||
);
|
||||
hasOrphanedDocs = true;
|
||||
}
|
||||
} catch (searchError) {
|
||||
logger.debug('[indexSync] Could not check conversation documents:', searchError.message);
|
||||
}
|
||||
} catch (error) {
|
||||
if (error.code !== 'index_not_found') {
|
||||
logger.warn('[indexSync] Could not check/update convos index settings:', error.message);
|
||||
}
|
||||
}
|
||||
|
||||
// If either index has orphaned documents, clean them up (but don't force resync)
|
||||
if (hasOrphanedDocs) {
|
||||
try {
|
||||
const messagesIndex = client.index('messages');
|
||||
await deleteDocumentsWithoutUserField(messagesIndex, 'messages');
|
||||
} catch (error) {
|
||||
logger.debug('[indexSync] Could not clean up messages:', error.message);
|
||||
}
|
||||
|
||||
try {
|
||||
const convosIndex = client.index('convos');
|
||||
await deleteDocumentsWithoutUserField(convosIndex, 'convos');
|
||||
} catch (error) {
|
||||
logger.debug('[indexSync] Could not clean up convos:', error.message);
|
||||
}
|
||||
|
||||
logger.info('[indexSync] Orphaned documents cleaned up without forcing resync.');
|
||||
}
|
||||
|
||||
if (settingsUpdated) {
|
||||
logger.info('[indexSync] Index settings updated. Full re-sync will be triggered.');
|
||||
}
|
||||
} catch (error) {
|
||||
logger.error('[indexSync] Error ensuring filterable attributes:', error);
|
||||
}
|
||||
|
||||
return { settingsUpdated, orphanedDocsFound: hasOrphanedDocs };
|
||||
}
|
||||
|
||||
/**
|
||||
* Performs the actual sync operations for messages and conversations
|
||||
* @param {FlowStateManager} flowManager - Flow state manager instance
|
||||
* @param {string} flowId - Flow identifier
|
||||
* @param {string} flowType - Flow type
|
||||
*/
|
||||
async function performSync(flowManager, flowId, flowType) {
|
||||
try {
|
||||
if (indexingDisabled === true) {
|
||||
logger.info('[indexSync] Indexing is disabled, skipping...');
|
||||
return { messagesSync: false, convosSync: false };
|
||||
}
|
||||
|
||||
const client = MeiliSearchClient.getInstance();
|
||||
|
||||
const { status } = await client.health();
|
||||
if (status !== 'available') {
|
||||
throw new Error('Meilisearch not available');
|
||||
}
|
||||
|
||||
/** Ensures indexes have proper filterable attributes configured */
|
||||
const { settingsUpdated, orphanedDocsFound: _orphanedDocsFound } =
|
||||
await ensureFilterableAttributes(client);
|
||||
|
||||
let messagesSync = false;
|
||||
let convosSync = false;
|
||||
|
||||
// Only reset flags if settings were actually updated (not just for orphaned doc cleanup)
|
||||
if (settingsUpdated) {
|
||||
logger.info(
|
||||
'[indexSync] Settings updated. Forcing full re-sync to reindex with new configuration...',
|
||||
);
|
||||
|
||||
// Reset sync flags to force full re-sync
|
||||
await batchResetMeiliFlags(Message.collection);
|
||||
await batchResetMeiliFlags(Conversation.collection);
|
||||
}
|
||||
|
||||
// Check if we need to sync messages
|
||||
logger.info('[indexSync] Requesting message sync progress...');
|
||||
const messageProgress = await Message.getSyncProgress();
|
||||
if (!messageProgress.isComplete || settingsUpdated) {
|
||||
logger.info(
|
||||
`[indexSync] Messages need syncing: ${messageProgress.totalProcessed}/${messageProgress.totalDocuments} indexed`,
|
||||
);
|
||||
|
||||
const messageCount = messageProgress.totalDocuments;
|
||||
const messagesIndexed = messageProgress.totalProcessed;
|
||||
const unindexedMessages = messageCount - messagesIndexed;
|
||||
|
||||
if (settingsUpdated || unindexedMessages > syncThreshold) {
|
||||
logger.info(`[indexSync] Starting message sync (${unindexedMessages} unindexed)`);
|
||||
await Message.syncWithMeili();
|
||||
messagesSync = true;
|
||||
} else if (unindexedMessages > 0) {
|
||||
logger.info(
|
||||
`[indexSync] ${unindexedMessages} messages unindexed (below threshold: ${syncThreshold}, skipping)`,
|
||||
);
|
||||
}
|
||||
} else {
|
||||
logger.info(
|
||||
`[indexSync] Messages are fully synced: ${messageProgress.totalProcessed}/${messageProgress.totalDocuments}`,
|
||||
);
|
||||
}
|
||||
|
||||
// Check if we need to sync conversations
|
||||
const convoProgress = await Conversation.getSyncProgress();
|
||||
if (!convoProgress.isComplete || settingsUpdated) {
|
||||
logger.info(
|
||||
`[indexSync] Conversations need syncing: ${convoProgress.totalProcessed}/${convoProgress.totalDocuments} indexed`,
|
||||
);
|
||||
|
||||
const convoCount = convoProgress.totalDocuments;
|
||||
const convosIndexed = convoProgress.totalProcessed;
|
||||
|
||||
const unindexedConvos = convoCount - convosIndexed;
|
||||
if (settingsUpdated || unindexedConvos > syncThreshold) {
|
||||
logger.info(`[indexSync] Starting convos sync (${unindexedConvos} unindexed)`);
|
||||
await Conversation.syncWithMeili();
|
||||
convosSync = true;
|
||||
} else if (unindexedConvos > 0) {
|
||||
logger.info(
|
||||
`[indexSync] ${unindexedConvos} convos unindexed (below threshold: ${syncThreshold}, skipping)`,
|
||||
);
|
||||
}
|
||||
} else {
|
||||
logger.info(
|
||||
`[indexSync] Conversations are fully synced: ${convoProgress.totalProcessed}/${convoProgress.totalDocuments}`,
|
||||
);
|
||||
}
|
||||
|
||||
return { messagesSync, convosSync };
|
||||
} finally {
|
||||
if (indexingDisabled === true) {
|
||||
logger.info('[indexSync] Indexing is disabled, skipping cleanup...');
|
||||
} else if (flowManager && flowId && flowType) {
|
||||
try {
|
||||
await flowManager.deleteFlow(flowId, flowType);
|
||||
logger.debug('[indexSync] Flow state cleaned up');
|
||||
} catch (cleanupErr) {
|
||||
logger.debug('[indexSync] Could not clean up flow state:', cleanupErr.message);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Main index sync function that uses FlowStateManager to prevent concurrent execution
|
||||
* Search index sync.
|
||||
*
|
||||
* MongoDB + MeiliSearch have been dropped. Full-text search now runs directly
|
||||
* on Hanzo Base/SQLite via the data-layer adapter (the model `meiliSearch()`
|
||||
* method in api/db/base searches content fields on Base). There is no separate
|
||||
* index to sync, so this is a no-op kept for boot-sequence compatibility
|
||||
* (api/server/index.js calls `indexSync()`).
|
||||
*/
|
||||
async function indexSync() {
|
||||
if (!searchEnabled) {
|
||||
return;
|
||||
}
|
||||
|
||||
logger.info('[indexSync] Starting index synchronization check...');
|
||||
|
||||
// Get or create FlowStateManager instance
|
||||
const flowsCache = getLogStores(CacheKeys.FLOWS);
|
||||
if (!flowsCache) {
|
||||
logger.warn('[indexSync] Flows cache not available, falling back to direct sync');
|
||||
return await performSync(null, null, null);
|
||||
}
|
||||
|
||||
const flowManager = new FlowStateManager(flowsCache, {
|
||||
ttl: 60000 * 10, // 10 minutes TTL for sync operations
|
||||
});
|
||||
|
||||
// Use a unique flow ID for the sync operation
|
||||
const flowId = 'meili-index-sync';
|
||||
const flowType = 'MEILI_SYNC';
|
||||
|
||||
try {
|
||||
// This will only execute the handler if no other instance is running the sync
|
||||
const result = await flowManager.createFlowWithHandler(flowId, flowType, () =>
|
||||
performSync(flowManager, flowId, flowType),
|
||||
);
|
||||
|
||||
if (result.messagesSync || result.convosSync) {
|
||||
logger.info('[indexSync] Sync completed successfully');
|
||||
} else {
|
||||
logger.debug('[indexSync] No sync was needed');
|
||||
}
|
||||
|
||||
return result;
|
||||
} catch (err) {
|
||||
if (err.message.includes('flow already exists')) {
|
||||
logger.info('[indexSync] Sync already running on another instance');
|
||||
return;
|
||||
}
|
||||
|
||||
if (err.message.includes('not found')) {
|
||||
logger.debug('[indexSync] Creating indices...');
|
||||
currentTimeout = setTimeout(async () => {
|
||||
try {
|
||||
await Message.syncWithMeili();
|
||||
await Conversation.syncWithMeili();
|
||||
} catch (err) {
|
||||
logger.error('[indexSync] Trouble creating indices, try restarting the server.', err);
|
||||
}
|
||||
}, 750);
|
||||
} else if (err.message.includes('Meilisearch not configured')) {
|
||||
logger.info('[indexSync] Meilisearch not configured, search will be disabled.');
|
||||
} else {
|
||||
logger.error('[indexSync] error', err);
|
||||
}
|
||||
}
|
||||
logger.info('[indexSync] Base/SQLite search active — no external index to sync');
|
||||
}
|
||||
|
||||
process.on('exit', () => {
|
||||
logger.debug('[indexSync] Clearing sync timeouts before exiting...');
|
||||
clearTimeout(currentTimeout);
|
||||
});
|
||||
|
||||
module.exports = indexSync;
|
||||
|
||||
@@ -1,465 +0,0 @@
|
||||
/**
|
||||
* Unit tests for performSync() function in indexSync.js
|
||||
*
|
||||
* Tests use real mongoose with mocked model methods, only mocking external calls.
|
||||
*/
|
||||
|
||||
const mongoose = require('mongoose');
|
||||
|
||||
// Mock only external dependencies (not internal classes/models)
|
||||
const mockLogger = {
|
||||
info: jest.fn(),
|
||||
warn: jest.fn(),
|
||||
error: jest.fn(),
|
||||
debug: jest.fn(),
|
||||
};
|
||||
|
||||
const mockMeiliHealth = jest.fn();
|
||||
const mockMeiliIndex = jest.fn();
|
||||
const mockBatchResetMeiliFlags = jest.fn();
|
||||
const mockIsEnabled = jest.fn();
|
||||
const mockGetLogStores = jest.fn();
|
||||
|
||||
// Create mock models that will be reused
|
||||
const createMockModel = (collectionName) => ({
|
||||
collection: { name: collectionName },
|
||||
getSyncProgress: jest.fn(),
|
||||
syncWithMeili: jest.fn(),
|
||||
countDocuments: jest.fn(),
|
||||
});
|
||||
|
||||
const originalMessageModel = mongoose.models.Message;
|
||||
const originalConversationModel = mongoose.models.Conversation;
|
||||
|
||||
// Mock external modules
|
||||
jest.mock('@librechat/data-schemas', () => ({
|
||||
logger: mockLogger,
|
||||
}));
|
||||
|
||||
jest.mock('meilisearch', () => ({
|
||||
MeiliSearch: jest.fn(() => ({
|
||||
health: mockMeiliHealth,
|
||||
index: mockMeiliIndex,
|
||||
})),
|
||||
}));
|
||||
|
||||
jest.mock('./utils', () => ({
|
||||
batchResetMeiliFlags: mockBatchResetMeiliFlags,
|
||||
}));
|
||||
|
||||
jest.mock('@hanzochat/api', () => ({
|
||||
isEnabled: mockIsEnabled,
|
||||
FlowStateManager: jest.fn(),
|
||||
}));
|
||||
|
||||
jest.mock('~/cache', () => ({
|
||||
getLogStores: mockGetLogStores,
|
||||
}));
|
||||
|
||||
// Set environment before module load
|
||||
process.env.MEILI_HOST = 'http://localhost:7700';
|
||||
process.env.MEILI_MASTER_KEY = 'test-key';
|
||||
process.env.SEARCH = 'true';
|
||||
process.env.MEILI_SYNC_THRESHOLD = '1000'; // Set threshold before module loads
|
||||
|
||||
describe('performSync() - syncThreshold logic', () => {
|
||||
const ORIGINAL_ENV = process.env;
|
||||
let Message;
|
||||
let Conversation;
|
||||
|
||||
beforeAll(() => {
|
||||
Message = createMockModel('messages');
|
||||
Conversation = createMockModel('conversations');
|
||||
|
||||
mongoose.models.Message = Message;
|
||||
mongoose.models.Conversation = Conversation;
|
||||
});
|
||||
|
||||
beforeEach(() => {
|
||||
// Reset all mocks
|
||||
jest.clearAllMocks();
|
||||
// Reset modules to ensure fresh load of indexSync.js and its top-level consts (like syncThreshold)
|
||||
jest.resetModules();
|
||||
|
||||
// Set up environment
|
||||
process.env = { ...ORIGINAL_ENV };
|
||||
process.env.MEILI_HOST = 'http://localhost:7700';
|
||||
process.env.MEILI_MASTER_KEY = 'test-key';
|
||||
process.env.SEARCH = 'true';
|
||||
delete process.env.MEILI_NO_SYNC;
|
||||
|
||||
// Re-ensure models are available in mongoose after resetModules
|
||||
// We must require mongoose again to get the fresh instance that indexSync will use
|
||||
const mongoose = require('mongoose');
|
||||
mongoose.models.Message = Message;
|
||||
mongoose.models.Conversation = Conversation;
|
||||
|
||||
// Mock isEnabled
|
||||
mockIsEnabled.mockImplementation((val) => val === 'true' || val === true);
|
||||
|
||||
// Mock MeiliSearch client responses
|
||||
mockMeiliHealth.mockResolvedValue({ status: 'available' });
|
||||
mockMeiliIndex.mockReturnValue({
|
||||
getSettings: jest.fn().mockResolvedValue({ filterableAttributes: ['user'] }),
|
||||
updateSettings: jest.fn().mockResolvedValue({}),
|
||||
search: jest.fn().mockResolvedValue({ hits: [] }),
|
||||
});
|
||||
|
||||
mockBatchResetMeiliFlags.mockResolvedValue(undefined);
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
process.env = ORIGINAL_ENV;
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
mongoose.models.Message = originalMessageModel;
|
||||
mongoose.models.Conversation = originalConversationModel;
|
||||
});
|
||||
|
||||
test('triggers sync when unindexed messages exceed syncThreshold', async () => {
|
||||
// Arrange: Set threshold before module load
|
||||
process.env.MEILI_SYNC_THRESHOLD = '1000';
|
||||
|
||||
// Arrange: 1050 unindexed messages > 1000 threshold
|
||||
Message.getSyncProgress.mockResolvedValue({
|
||||
totalProcessed: 100,
|
||||
totalDocuments: 1150, // 1050 unindexed
|
||||
isComplete: false,
|
||||
});
|
||||
|
||||
Conversation.getSyncProgress.mockResolvedValue({
|
||||
totalProcessed: 50,
|
||||
totalDocuments: 50,
|
||||
isComplete: true,
|
||||
});
|
||||
|
||||
Message.syncWithMeili.mockResolvedValue(undefined);
|
||||
|
||||
// Act
|
||||
const indexSync = require('./indexSync');
|
||||
await indexSync();
|
||||
|
||||
// Assert: No countDocuments calls
|
||||
expect(Message.countDocuments).not.toHaveBeenCalled();
|
||||
expect(Conversation.countDocuments).not.toHaveBeenCalled();
|
||||
|
||||
// Assert: Message sync triggered because 1050 > 1000
|
||||
expect(Message.syncWithMeili).toHaveBeenCalledTimes(1);
|
||||
expect(mockLogger.info).toHaveBeenCalledWith(
|
||||
'[indexSync] Messages need syncing: 100/1150 indexed',
|
||||
);
|
||||
expect(mockLogger.info).toHaveBeenCalledWith(
|
||||
'[indexSync] Starting message sync (1050 unindexed)',
|
||||
);
|
||||
|
||||
// Assert: Conversation sync NOT triggered (already complete)
|
||||
expect(Conversation.syncWithMeili).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('skips sync when unindexed messages are below syncThreshold', async () => {
|
||||
// Arrange: 50 unindexed messages < 1000 threshold
|
||||
Message.getSyncProgress.mockResolvedValue({
|
||||
totalProcessed: 100,
|
||||
totalDocuments: 150, // 50 unindexed
|
||||
isComplete: false,
|
||||
});
|
||||
|
||||
Conversation.getSyncProgress.mockResolvedValue({
|
||||
totalProcessed: 50,
|
||||
totalDocuments: 50,
|
||||
isComplete: true,
|
||||
});
|
||||
|
||||
process.env.MEILI_SYNC_THRESHOLD = '1000';
|
||||
|
||||
// Act
|
||||
const indexSync = require('./indexSync');
|
||||
await indexSync();
|
||||
|
||||
// Assert: No countDocuments calls
|
||||
expect(Message.countDocuments).not.toHaveBeenCalled();
|
||||
expect(Conversation.countDocuments).not.toHaveBeenCalled();
|
||||
|
||||
// Assert: Message sync NOT triggered because 50 < 1000
|
||||
expect(Message.syncWithMeili).not.toHaveBeenCalled();
|
||||
expect(mockLogger.info).toHaveBeenCalledWith(
|
||||
'[indexSync] Messages need syncing: 100/150 indexed',
|
||||
);
|
||||
expect(mockLogger.info).toHaveBeenCalledWith(
|
||||
'[indexSync] 50 messages unindexed (below threshold: 1000, skipping)',
|
||||
);
|
||||
|
||||
// Assert: Conversation sync NOT triggered (already complete)
|
||||
expect(Conversation.syncWithMeili).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('respects syncThreshold at boundary (exactly at threshold)', async () => {
|
||||
// Arrange: 1000 unindexed messages = 1000 threshold (NOT greater than)
|
||||
Message.getSyncProgress.mockResolvedValue({
|
||||
totalProcessed: 100,
|
||||
totalDocuments: 1100, // 1000 unindexed
|
||||
isComplete: false,
|
||||
});
|
||||
|
||||
Conversation.getSyncProgress.mockResolvedValue({
|
||||
totalProcessed: 0,
|
||||
totalDocuments: 0,
|
||||
isComplete: true,
|
||||
});
|
||||
|
||||
process.env.MEILI_SYNC_THRESHOLD = '1000';
|
||||
|
||||
// Act
|
||||
const indexSync = require('./indexSync');
|
||||
await indexSync();
|
||||
|
||||
// Assert: No countDocuments calls
|
||||
expect(Message.countDocuments).not.toHaveBeenCalled();
|
||||
|
||||
// Assert: Message sync NOT triggered because 1000 is NOT > 1000
|
||||
expect(Message.syncWithMeili).not.toHaveBeenCalled();
|
||||
expect(mockLogger.info).toHaveBeenCalledWith(
|
||||
'[indexSync] Messages need syncing: 100/1100 indexed',
|
||||
);
|
||||
expect(mockLogger.info).toHaveBeenCalledWith(
|
||||
'[indexSync] 1000 messages unindexed (below threshold: 1000, skipping)',
|
||||
);
|
||||
});
|
||||
|
||||
test('triggers sync when unindexed is threshold + 1', async () => {
|
||||
// Arrange: 1001 unindexed messages > 1000 threshold
|
||||
Message.getSyncProgress.mockResolvedValue({
|
||||
totalProcessed: 100,
|
||||
totalDocuments: 1101, // 1001 unindexed
|
||||
isComplete: false,
|
||||
});
|
||||
|
||||
Conversation.getSyncProgress.mockResolvedValue({
|
||||
totalProcessed: 0,
|
||||
totalDocuments: 0,
|
||||
isComplete: true,
|
||||
});
|
||||
|
||||
Message.syncWithMeili.mockResolvedValue(undefined);
|
||||
|
||||
process.env.MEILI_SYNC_THRESHOLD = '1000';
|
||||
|
||||
// Act
|
||||
const indexSync = require('./indexSync');
|
||||
await indexSync();
|
||||
|
||||
// Assert: No countDocuments calls
|
||||
expect(Message.countDocuments).not.toHaveBeenCalled();
|
||||
|
||||
// Assert: Message sync triggered because 1001 > 1000
|
||||
expect(Message.syncWithMeili).toHaveBeenCalledTimes(1);
|
||||
expect(mockLogger.info).toHaveBeenCalledWith(
|
||||
'[indexSync] Messages need syncing: 100/1101 indexed',
|
||||
);
|
||||
expect(mockLogger.info).toHaveBeenCalledWith(
|
||||
'[indexSync] Starting message sync (1001 unindexed)',
|
||||
);
|
||||
});
|
||||
|
||||
test('uses totalDocuments from convoProgress for conversation sync decisions', async () => {
|
||||
// Arrange: Messages complete, conversations need sync
|
||||
Message.getSyncProgress.mockResolvedValue({
|
||||
totalProcessed: 100,
|
||||
totalDocuments: 100,
|
||||
isComplete: true,
|
||||
});
|
||||
|
||||
Conversation.getSyncProgress.mockResolvedValue({
|
||||
totalProcessed: 50,
|
||||
totalDocuments: 1100, // 1050 unindexed > 1000 threshold
|
||||
isComplete: false,
|
||||
});
|
||||
|
||||
Conversation.syncWithMeili.mockResolvedValue(undefined);
|
||||
|
||||
process.env.MEILI_SYNC_THRESHOLD = '1000';
|
||||
|
||||
// Act
|
||||
const indexSync = require('./indexSync');
|
||||
await indexSync();
|
||||
|
||||
// Assert: No countDocuments calls (the optimization)
|
||||
expect(Message.countDocuments).not.toHaveBeenCalled();
|
||||
expect(Conversation.countDocuments).not.toHaveBeenCalled();
|
||||
|
||||
// Assert: Only conversation sync triggered
|
||||
expect(Message.syncWithMeili).not.toHaveBeenCalled();
|
||||
expect(Conversation.syncWithMeili).toHaveBeenCalledTimes(1);
|
||||
expect(mockLogger.info).toHaveBeenCalledWith(
|
||||
'[indexSync] Conversations need syncing: 50/1100 indexed',
|
||||
);
|
||||
expect(mockLogger.info).toHaveBeenCalledWith(
|
||||
'[indexSync] Starting convos sync (1050 unindexed)',
|
||||
);
|
||||
});
|
||||
|
||||
test('skips sync when collections are fully synced', async () => {
|
||||
// Arrange: Everything already synced
|
||||
Message.getSyncProgress.mockResolvedValue({
|
||||
totalProcessed: 100,
|
||||
totalDocuments: 100,
|
||||
isComplete: true,
|
||||
});
|
||||
|
||||
Conversation.getSyncProgress.mockResolvedValue({
|
||||
totalProcessed: 50,
|
||||
totalDocuments: 50,
|
||||
isComplete: true,
|
||||
});
|
||||
|
||||
// Act
|
||||
const indexSync = require('./indexSync');
|
||||
await indexSync();
|
||||
|
||||
// Assert: No countDocuments calls
|
||||
expect(Message.countDocuments).not.toHaveBeenCalled();
|
||||
expect(Conversation.countDocuments).not.toHaveBeenCalled();
|
||||
|
||||
// Assert: No sync triggered
|
||||
expect(Message.syncWithMeili).not.toHaveBeenCalled();
|
||||
expect(Conversation.syncWithMeili).not.toHaveBeenCalled();
|
||||
|
||||
// Assert: Correct logs
|
||||
expect(mockLogger.info).toHaveBeenCalledWith('[indexSync] Messages are fully synced: 100/100');
|
||||
expect(mockLogger.info).toHaveBeenCalledWith(
|
||||
'[indexSync] Conversations are fully synced: 50/50',
|
||||
);
|
||||
});
|
||||
|
||||
test('triggers message sync when settingsUpdated even if below syncThreshold', async () => {
|
||||
// Arrange: Only 50 unindexed messages (< 1000 threshold), but settings were updated
|
||||
Message.getSyncProgress.mockResolvedValue({
|
||||
totalProcessed: 100,
|
||||
totalDocuments: 150, // 50 unindexed
|
||||
isComplete: false,
|
||||
});
|
||||
|
||||
Conversation.getSyncProgress.mockResolvedValue({
|
||||
totalProcessed: 50,
|
||||
totalDocuments: 50,
|
||||
isComplete: true,
|
||||
});
|
||||
|
||||
Message.syncWithMeili.mockResolvedValue(undefined);
|
||||
|
||||
// Mock settings update scenario
|
||||
mockMeiliIndex.mockReturnValue({
|
||||
getSettings: jest.fn().mockResolvedValue({ filterableAttributes: [] }), // No user field
|
||||
updateSettings: jest.fn().mockResolvedValue({}),
|
||||
search: jest.fn().mockResolvedValue({ hits: [] }),
|
||||
});
|
||||
|
||||
process.env.MEILI_SYNC_THRESHOLD = '1000';
|
||||
|
||||
// Act
|
||||
const indexSync = require('./indexSync');
|
||||
await indexSync();
|
||||
|
||||
// Assert: Flags were reset due to settings update
|
||||
expect(mockBatchResetMeiliFlags).toHaveBeenCalledWith(Message.collection);
|
||||
expect(mockBatchResetMeiliFlags).toHaveBeenCalledWith(Conversation.collection);
|
||||
|
||||
// Assert: Message sync triggered despite being below threshold (50 < 1000)
|
||||
expect(Message.syncWithMeili).toHaveBeenCalledTimes(1);
|
||||
expect(mockLogger.info).toHaveBeenCalledWith(
|
||||
'[indexSync] Settings updated. Forcing full re-sync to reindex with new configuration...',
|
||||
);
|
||||
expect(mockLogger.info).toHaveBeenCalledWith(
|
||||
'[indexSync] Starting message sync (50 unindexed)',
|
||||
);
|
||||
});
|
||||
|
||||
test('triggers conversation sync when settingsUpdated even if below syncThreshold', async () => {
|
||||
// Arrange: Messages complete, conversations have 50 unindexed (< 1000 threshold), but settings were updated
|
||||
Message.getSyncProgress.mockResolvedValue({
|
||||
totalProcessed: 100,
|
||||
totalDocuments: 100,
|
||||
isComplete: true,
|
||||
});
|
||||
|
||||
Conversation.getSyncProgress.mockResolvedValue({
|
||||
totalProcessed: 50,
|
||||
totalDocuments: 100, // 50 unindexed
|
||||
isComplete: false,
|
||||
});
|
||||
|
||||
Conversation.syncWithMeili.mockResolvedValue(undefined);
|
||||
|
||||
// Mock settings update scenario
|
||||
mockMeiliIndex.mockReturnValue({
|
||||
getSettings: jest.fn().mockResolvedValue({ filterableAttributes: [] }), // No user field
|
||||
updateSettings: jest.fn().mockResolvedValue({}),
|
||||
search: jest.fn().mockResolvedValue({ hits: [] }),
|
||||
});
|
||||
|
||||
process.env.MEILI_SYNC_THRESHOLD = '1000';
|
||||
|
||||
// Act
|
||||
const indexSync = require('./indexSync');
|
||||
await indexSync();
|
||||
|
||||
// Assert: Flags were reset due to settings update
|
||||
expect(mockBatchResetMeiliFlags).toHaveBeenCalledWith(Message.collection);
|
||||
expect(mockBatchResetMeiliFlags).toHaveBeenCalledWith(Conversation.collection);
|
||||
|
||||
// Assert: Conversation sync triggered despite being below threshold (50 < 1000)
|
||||
expect(Conversation.syncWithMeili).toHaveBeenCalledTimes(1);
|
||||
expect(mockLogger.info).toHaveBeenCalledWith(
|
||||
'[indexSync] Settings updated. Forcing full re-sync to reindex with new configuration...',
|
||||
);
|
||||
expect(mockLogger.info).toHaveBeenCalledWith('[indexSync] Starting convos sync (50 unindexed)');
|
||||
});
|
||||
|
||||
test('triggers both message and conversation sync when settingsUpdated even if both below syncThreshold', async () => {
|
||||
// Arrange: Set threshold before module load
|
||||
process.env.MEILI_SYNC_THRESHOLD = '1000';
|
||||
|
||||
// Arrange: Both have documents below threshold (50 each), but settings were updated
|
||||
Message.getSyncProgress.mockResolvedValue({
|
||||
totalProcessed: 100,
|
||||
totalDocuments: 150, // 50 unindexed
|
||||
isComplete: false,
|
||||
});
|
||||
|
||||
Conversation.getSyncProgress.mockResolvedValue({
|
||||
totalProcessed: 50,
|
||||
totalDocuments: 100, // 50 unindexed
|
||||
isComplete: false,
|
||||
});
|
||||
|
||||
Message.syncWithMeili.mockResolvedValue(undefined);
|
||||
Conversation.syncWithMeili.mockResolvedValue(undefined);
|
||||
|
||||
// Mock settings update scenario
|
||||
mockMeiliIndex.mockReturnValue({
|
||||
getSettings: jest.fn().mockResolvedValue({ filterableAttributes: [] }), // No user field
|
||||
updateSettings: jest.fn().mockResolvedValue({}),
|
||||
search: jest.fn().mockResolvedValue({ hits: [] }),
|
||||
});
|
||||
|
||||
// Act
|
||||
const indexSync = require('./indexSync');
|
||||
await indexSync();
|
||||
|
||||
// Assert: Flags were reset due to settings update
|
||||
expect(mockBatchResetMeiliFlags).toHaveBeenCalledWith(Message.collection);
|
||||
expect(mockBatchResetMeiliFlags).toHaveBeenCalledWith(Conversation.collection);
|
||||
|
||||
// Assert: Both syncs triggered despite both being below threshold
|
||||
expect(Message.syncWithMeili).toHaveBeenCalledTimes(1);
|
||||
expect(Conversation.syncWithMeili).toHaveBeenCalledTimes(1);
|
||||
expect(mockLogger.info).toHaveBeenCalledWith(
|
||||
'[indexSync] Settings updated. Forcing full re-sync to reindex with new configuration...',
|
||||
);
|
||||
expect(mockLogger.info).toHaveBeenCalledWith(
|
||||
'[indexSync] Starting message sync (50 unindexed)',
|
||||
);
|
||||
expect(mockLogger.info).toHaveBeenCalledWith('[indexSync] Starting convos sync (50 unindexed)');
|
||||
});
|
||||
});
|
||||
+1
-1
@@ -1,4 +1,4 @@
|
||||
const mongoose = require('mongoose');
|
||||
const { mongoose } = require('./base');
|
||||
const { createModels } = require('@librechat/data-schemas');
|
||||
const models = createModels(mongoose);
|
||||
|
||||
|
||||
@@ -1,90 +0,0 @@
|
||||
const { logger } = require('@librechat/data-schemas');
|
||||
|
||||
const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
|
||||
|
||||
/**
|
||||
* Batch update documents in chunks to avoid timeouts on weak instances
|
||||
* @param {mongoose.Collection} collection - MongoDB collection
|
||||
* @returns {Promise<number>} - Total modified count
|
||||
* @throws {Error} - Throws if database operations fail (e.g., network issues, connection loss, permission problems)
|
||||
*/
|
||||
async function batchResetMeiliFlags(collection) {
|
||||
const DEFAULT_BATCH_SIZE = 1000;
|
||||
|
||||
let BATCH_SIZE = parseEnvInt('MEILI_SYNC_BATCH_SIZE', DEFAULT_BATCH_SIZE);
|
||||
if (BATCH_SIZE === 0) {
|
||||
logger.warn(
|
||||
`[batchResetMeiliFlags] MEILI_SYNC_BATCH_SIZE cannot be 0. Using default: ${DEFAULT_BATCH_SIZE}`,
|
||||
);
|
||||
BATCH_SIZE = DEFAULT_BATCH_SIZE;
|
||||
}
|
||||
|
||||
const BATCH_DELAY_MS = parseEnvInt('MEILI_SYNC_DELAY_MS', 100);
|
||||
let totalModified = 0;
|
||||
let hasMore = true;
|
||||
|
||||
try {
|
||||
while (hasMore) {
|
||||
const docs = await collection
|
||||
.find({ expiredAt: null, _meiliIndex: { $ne: false } }, { projection: { _id: 1 } })
|
||||
.limit(BATCH_SIZE)
|
||||
.toArray();
|
||||
|
||||
if (docs.length === 0) {
|
||||
break;
|
||||
}
|
||||
|
||||
const ids = docs.map((doc) => doc._id);
|
||||
const result = await collection.updateMany(
|
||||
{ _id: { $in: ids } },
|
||||
{ $set: { _meiliIndex: false } },
|
||||
);
|
||||
|
||||
totalModified += result.modifiedCount;
|
||||
process.stdout.write(
|
||||
`\r Updating ${collection.collectionName}: ${totalModified} documents...`,
|
||||
);
|
||||
|
||||
if (docs.length < BATCH_SIZE) {
|
||||
hasMore = false;
|
||||
}
|
||||
|
||||
if (hasMore && BATCH_DELAY_MS > 0) {
|
||||
await sleep(BATCH_DELAY_MS);
|
||||
}
|
||||
}
|
||||
|
||||
return totalModified;
|
||||
} catch (error) {
|
||||
throw new Error(
|
||||
`Failed to batch reset Meili flags for collection '${collection.collectionName}' after processing ${totalModified} documents: ${error.message}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse and validate an environment variable as a positive integer
|
||||
* @param {string} varName - Environment variable name
|
||||
* @param {number} defaultValue - Default value to use if invalid or missing
|
||||
* @returns {number} - Parsed value or default
|
||||
*/
|
||||
function parseEnvInt(varName, defaultValue) {
|
||||
const value = process.env[varName];
|
||||
if (!value) {
|
||||
return defaultValue;
|
||||
}
|
||||
|
||||
const parsed = parseInt(value, 10);
|
||||
if (isNaN(parsed) || parsed < 0) {
|
||||
logger.warn(
|
||||
`[batchResetMeiliFlags] Invalid value for ${varName}="${value}". Expected a positive integer. Using default: ${defaultValue}`,
|
||||
);
|
||||
return defaultValue;
|
||||
}
|
||||
|
||||
return parsed;
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
batchResetMeiliFlags,
|
||||
};
|
||||
@@ -1,523 +0,0 @@
|
||||
const mongoose = require('mongoose');
|
||||
const { MongoMemoryServer } = require('mongodb-memory-server');
|
||||
const { batchResetMeiliFlags } = require('./utils');
|
||||
|
||||
describe('batchResetMeiliFlags', () => {
|
||||
let mongoServer;
|
||||
let testCollection;
|
||||
const ORIGINAL_BATCH_SIZE = process.env.MEILI_SYNC_BATCH_SIZE;
|
||||
const ORIGINAL_BATCH_DELAY = process.env.MEILI_SYNC_DELAY_MS;
|
||||
|
||||
beforeAll(async () => {
|
||||
mongoServer = await MongoMemoryServer.create();
|
||||
const mongoUri = mongoServer.getUri();
|
||||
await mongoose.connect(mongoUri);
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await mongoose.disconnect();
|
||||
await mongoServer.stop();
|
||||
|
||||
// Restore original env variables
|
||||
if (ORIGINAL_BATCH_SIZE !== undefined) {
|
||||
process.env.MEILI_SYNC_BATCH_SIZE = ORIGINAL_BATCH_SIZE;
|
||||
} else {
|
||||
delete process.env.MEILI_SYNC_BATCH_SIZE;
|
||||
}
|
||||
|
||||
if (ORIGINAL_BATCH_DELAY !== undefined) {
|
||||
process.env.MEILI_SYNC_DELAY_MS = ORIGINAL_BATCH_DELAY;
|
||||
} else {
|
||||
delete process.env.MEILI_SYNC_DELAY_MS;
|
||||
}
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
// Create a fresh collection for each test
|
||||
testCollection = mongoose.connection.db.collection('test_meili_batch');
|
||||
await testCollection.deleteMany({});
|
||||
|
||||
// Reset env variables to defaults
|
||||
delete process.env.MEILI_SYNC_BATCH_SIZE;
|
||||
delete process.env.MEILI_SYNC_DELAY_MS;
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
if (testCollection) {
|
||||
await testCollection.deleteMany({});
|
||||
}
|
||||
});
|
||||
|
||||
describe('basic functionality', () => {
|
||||
it('should reset _meiliIndex flag for documents with expiredAt: null and _meiliIndex: true', async () => {
|
||||
// Insert test documents
|
||||
await testCollection.insertMany([
|
||||
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: true, name: 'doc1' },
|
||||
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: true, name: 'doc2' },
|
||||
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: true, name: 'doc3' },
|
||||
]);
|
||||
|
||||
const result = await batchResetMeiliFlags(testCollection);
|
||||
|
||||
expect(result).toBe(3);
|
||||
|
||||
const updatedDocs = await testCollection.find({ _meiliIndex: false }).toArray();
|
||||
expect(updatedDocs).toHaveLength(3);
|
||||
|
||||
const notUpdatedDocs = await testCollection.find({ _meiliIndex: true }).toArray();
|
||||
expect(notUpdatedDocs).toHaveLength(0);
|
||||
});
|
||||
|
||||
it('should not modify documents with expiredAt set', async () => {
|
||||
const expiredDate = new Date();
|
||||
await testCollection.insertMany([
|
||||
{ _id: new mongoose.Types.ObjectId(), expiredAt: expiredDate, _meiliIndex: true },
|
||||
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: true },
|
||||
]);
|
||||
|
||||
const result = await batchResetMeiliFlags(testCollection);
|
||||
|
||||
expect(result).toBe(1);
|
||||
|
||||
const expiredDoc = await testCollection.findOne({ expiredAt: expiredDate });
|
||||
expect(expiredDoc._meiliIndex).toBe(true);
|
||||
});
|
||||
|
||||
it('should not modify documents with _meiliIndex: false', async () => {
|
||||
await testCollection.insertMany([
|
||||
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: false },
|
||||
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: true },
|
||||
]);
|
||||
|
||||
const result = await batchResetMeiliFlags(testCollection);
|
||||
|
||||
expect(result).toBe(1);
|
||||
});
|
||||
|
||||
it('should return 0 when no documents match the criteria', async () => {
|
||||
await testCollection.insertMany([
|
||||
{ _id: new mongoose.Types.ObjectId(), expiredAt: new Date(), _meiliIndex: true },
|
||||
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: false },
|
||||
]);
|
||||
|
||||
const result = await batchResetMeiliFlags(testCollection);
|
||||
|
||||
expect(result).toBe(0);
|
||||
});
|
||||
|
||||
it('should return 0 when collection is empty', async () => {
|
||||
const result = await batchResetMeiliFlags(testCollection);
|
||||
|
||||
expect(result).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('batch processing', () => {
|
||||
it('should process documents in batches according to MEILI_SYNC_BATCH_SIZE', async () => {
|
||||
process.env.MEILI_SYNC_BATCH_SIZE = '2';
|
||||
|
||||
const docs = [];
|
||||
for (let i = 0; i < 5; i++) {
|
||||
docs.push({
|
||||
_id: new mongoose.Types.ObjectId(),
|
||||
expiredAt: null,
|
||||
_meiliIndex: true,
|
||||
name: `doc${i}`,
|
||||
});
|
||||
}
|
||||
await testCollection.insertMany(docs);
|
||||
|
||||
const result = await batchResetMeiliFlags(testCollection);
|
||||
|
||||
expect(result).toBe(5);
|
||||
|
||||
const updatedDocs = await testCollection.find({ _meiliIndex: false }).toArray();
|
||||
expect(updatedDocs).toHaveLength(5);
|
||||
});
|
||||
|
||||
it('should handle large datasets with small batch sizes', async () => {
|
||||
process.env.MEILI_SYNC_BATCH_SIZE = '10';
|
||||
|
||||
const docs = [];
|
||||
for (let i = 0; i < 25; i++) {
|
||||
docs.push({
|
||||
_id: new mongoose.Types.ObjectId(),
|
||||
expiredAt: null,
|
||||
_meiliIndex: true,
|
||||
});
|
||||
}
|
||||
await testCollection.insertMany(docs);
|
||||
|
||||
const result = await batchResetMeiliFlags(testCollection);
|
||||
|
||||
expect(result).toBe(25);
|
||||
});
|
||||
|
||||
it('should use default batch size of 1000 when env variable is not set', async () => {
|
||||
// Create exactly 1000 documents to verify default batch behavior
|
||||
const docs = [];
|
||||
for (let i = 0; i < 1000; i++) {
|
||||
docs.push({
|
||||
_id: new mongoose.Types.ObjectId(),
|
||||
expiredAt: null,
|
||||
_meiliIndex: true,
|
||||
});
|
||||
}
|
||||
await testCollection.insertMany(docs);
|
||||
|
||||
const result = await batchResetMeiliFlags(testCollection);
|
||||
|
||||
expect(result).toBe(1000);
|
||||
});
|
||||
});
|
||||
|
||||
describe('return value', () => {
|
||||
it('should return correct modified count', async () => {
|
||||
await testCollection.insertMany([
|
||||
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: true },
|
||||
]);
|
||||
|
||||
await expect(batchResetMeiliFlags(testCollection)).resolves.toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe('batch delay', () => {
|
||||
it('should respect MEILI_SYNC_DELAY_MS between batches', async () => {
|
||||
process.env.MEILI_SYNC_BATCH_SIZE = '2';
|
||||
process.env.MEILI_SYNC_DELAY_MS = '50';
|
||||
|
||||
const docs = [];
|
||||
for (let i = 0; i < 5; i++) {
|
||||
docs.push({
|
||||
_id: new mongoose.Types.ObjectId(),
|
||||
expiredAt: null,
|
||||
_meiliIndex: true,
|
||||
});
|
||||
}
|
||||
await testCollection.insertMany(docs);
|
||||
|
||||
const startTime = Date.now();
|
||||
await batchResetMeiliFlags(testCollection);
|
||||
const endTime = Date.now();
|
||||
|
||||
// With 5 documents and batch size 2, we need 3 batches
|
||||
// That means 2 delays between batches (not after the last one)
|
||||
// So minimum time should be around 100ms (2 * 50ms)
|
||||
// Using a slightly lower threshold to account for timing variations
|
||||
const elapsed = endTime - startTime;
|
||||
expect(elapsed).toBeGreaterThanOrEqual(80);
|
||||
});
|
||||
|
||||
it('should not delay when MEILI_SYNC_DELAY_MS is 0', async () => {
|
||||
process.env.MEILI_SYNC_BATCH_SIZE = '2';
|
||||
process.env.MEILI_SYNC_DELAY_MS = '0';
|
||||
|
||||
const docs = [];
|
||||
for (let i = 0; i < 5; i++) {
|
||||
docs.push({
|
||||
_id: new mongoose.Types.ObjectId(),
|
||||
expiredAt: null,
|
||||
_meiliIndex: true,
|
||||
});
|
||||
}
|
||||
await testCollection.insertMany(docs);
|
||||
|
||||
const startTime = Date.now();
|
||||
await batchResetMeiliFlags(testCollection);
|
||||
const endTime = Date.now();
|
||||
|
||||
const elapsed = endTime - startTime;
|
||||
// Should complete without intentional delays, but database operations still take time
|
||||
// Just verify it completes and returns the correct count
|
||||
expect(elapsed).toBeLessThan(1000); // More reasonable upper bound
|
||||
|
||||
const result = await testCollection.countDocuments({ _meiliIndex: false });
|
||||
expect(result).toBe(5);
|
||||
});
|
||||
|
||||
it('should not delay after the last batch', async () => {
|
||||
process.env.MEILI_SYNC_BATCH_SIZE = '3';
|
||||
process.env.MEILI_SYNC_DELAY_MS = '100';
|
||||
|
||||
// Exactly 3 documents - should fit in one batch, no delay
|
||||
await testCollection.insertMany([
|
||||
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: true },
|
||||
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: true },
|
||||
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: true },
|
||||
]);
|
||||
|
||||
const result = await batchResetMeiliFlags(testCollection);
|
||||
|
||||
// Verify all 3 documents were processed in a single batch
|
||||
expect(result).toBe(3);
|
||||
|
||||
const updatedDocs = await testCollection.countDocuments({ _meiliIndex: false });
|
||||
expect(updatedDocs).toBe(3);
|
||||
});
|
||||
});
|
||||
|
||||
describe('edge cases', () => {
|
||||
it('should handle documents without _meiliIndex field', async () => {
|
||||
await testCollection.insertMany([
|
||||
{ _id: new mongoose.Types.ObjectId(), expiredAt: null },
|
||||
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: true },
|
||||
]);
|
||||
|
||||
const result = await batchResetMeiliFlags(testCollection);
|
||||
|
||||
// both documents should be updated
|
||||
expect(result).toBe(2);
|
||||
});
|
||||
|
||||
it('should handle mixed document states correctly', async () => {
|
||||
await testCollection.insertMany([
|
||||
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: true },
|
||||
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: false },
|
||||
{ _id: new mongoose.Types.ObjectId(), expiredAt: new Date(), _meiliIndex: true },
|
||||
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: true },
|
||||
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: null },
|
||||
{ _id: new mongoose.Types.ObjectId(), expiredAt: null },
|
||||
]);
|
||||
|
||||
const result = await batchResetMeiliFlags(testCollection);
|
||||
|
||||
expect(result).toBe(4);
|
||||
|
||||
const flaggedDocs = await testCollection
|
||||
.find({ expiredAt: null, _meiliIndex: false })
|
||||
.toArray();
|
||||
expect(flaggedDocs).toHaveLength(5); // 4 were updated, 1 was already false
|
||||
});
|
||||
});
|
||||
|
||||
describe('error handling', () => {
|
||||
it('should throw error with context when find operation fails', async () => {
|
||||
const mockCollection = {
|
||||
collectionName: 'test_meili_batch',
|
||||
find: jest.fn().mockReturnValue({
|
||||
limit: jest.fn().mockReturnValue({
|
||||
toArray: jest.fn().mockRejectedValue(new Error('Network error')),
|
||||
}),
|
||||
}),
|
||||
};
|
||||
|
||||
await expect(batchResetMeiliFlags(mockCollection)).rejects.toThrow(
|
||||
"Failed to batch reset Meili flags for collection 'test_meili_batch' after processing 0 documents: Network error",
|
||||
);
|
||||
});
|
||||
|
||||
it('should throw error with context when updateMany operation fails', async () => {
|
||||
const mockCollection = {
|
||||
collectionName: 'test_meili_batch',
|
||||
find: jest.fn().mockReturnValue({
|
||||
limit: jest.fn().mockReturnValue({
|
||||
toArray: jest
|
||||
.fn()
|
||||
.mockResolvedValue([
|
||||
{ _id: new mongoose.Types.ObjectId() },
|
||||
{ _id: new mongoose.Types.ObjectId() },
|
||||
]),
|
||||
}),
|
||||
}),
|
||||
updateMany: jest.fn().mockRejectedValue(new Error('Connection lost')),
|
||||
};
|
||||
|
||||
await expect(batchResetMeiliFlags(mockCollection)).rejects.toThrow(
|
||||
"Failed to batch reset Meili flags for collection 'test_meili_batch' after processing 0 documents: Connection lost",
|
||||
);
|
||||
});
|
||||
|
||||
it('should include documents processed count in error when failure occurs mid-batch', async () => {
|
||||
// Set batch size to 2 to force multiple batches
|
||||
process.env.MEILI_SYNC_BATCH_SIZE = '2';
|
||||
process.env.MEILI_SYNC_DELAY_MS = '0'; // No delay for faster test
|
||||
|
||||
let findCallCount = 0;
|
||||
let updateCallCount = 0;
|
||||
|
||||
const mockCollection = {
|
||||
collectionName: 'test_meili_batch',
|
||||
find: jest.fn().mockReturnValue({
|
||||
limit: jest.fn().mockReturnValue({
|
||||
toArray: jest.fn().mockImplementation(() => {
|
||||
findCallCount++;
|
||||
// Return 2 documents for first two calls (to keep loop going)
|
||||
// Return 2 documents for third call (to trigger third update which will fail)
|
||||
if (findCallCount <= 3) {
|
||||
return Promise.resolve([
|
||||
{ _id: new mongoose.Types.ObjectId() },
|
||||
{ _id: new mongoose.Types.ObjectId() },
|
||||
]);
|
||||
}
|
||||
// Should not reach here due to error
|
||||
return Promise.resolve([]);
|
||||
}),
|
||||
}),
|
||||
}),
|
||||
updateMany: jest.fn().mockImplementation(() => {
|
||||
updateCallCount++;
|
||||
if (updateCallCount === 1) {
|
||||
return Promise.resolve({ modifiedCount: 2 });
|
||||
} else if (updateCallCount === 2) {
|
||||
return Promise.resolve({ modifiedCount: 2 });
|
||||
} else {
|
||||
return Promise.reject(new Error('Database timeout'));
|
||||
}
|
||||
}),
|
||||
};
|
||||
|
||||
await expect(batchResetMeiliFlags(mockCollection)).rejects.toThrow(
|
||||
"Failed to batch reset Meili flags for collection 'test_meili_batch' after processing 4 documents: Database timeout",
|
||||
);
|
||||
});
|
||||
|
||||
it('should use collection.collectionName in error messages', async () => {
|
||||
const mockCollection = {
|
||||
collectionName: 'messages',
|
||||
find: jest.fn().mockReturnValue({
|
||||
limit: jest.fn().mockReturnValue({
|
||||
toArray: jest.fn().mockRejectedValue(new Error('Permission denied')),
|
||||
}),
|
||||
}),
|
||||
};
|
||||
|
||||
await expect(batchResetMeiliFlags(mockCollection)).rejects.toThrow(
|
||||
"Failed to batch reset Meili flags for collection 'messages' after processing 0 documents: Permission denied",
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('environment variable validation', () => {
|
||||
let warnSpy;
|
||||
|
||||
beforeEach(() => {
|
||||
// Mock logger.warn to track warning calls
|
||||
const { logger } = require('@librechat/data-schemas');
|
||||
warnSpy = jest.spyOn(logger, 'warn').mockImplementation(() => {});
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
if (warnSpy) {
|
||||
warnSpy.mockRestore();
|
||||
}
|
||||
});
|
||||
|
||||
it('should log warning and use default when MEILI_SYNC_BATCH_SIZE is not a number', async () => {
|
||||
process.env.MEILI_SYNC_BATCH_SIZE = 'abc';
|
||||
|
||||
await testCollection.insertMany([
|
||||
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: true },
|
||||
]);
|
||||
|
||||
const result = await batchResetMeiliFlags(testCollection);
|
||||
|
||||
expect(result).toBe(1);
|
||||
expect(warnSpy).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Invalid value for MEILI_SYNC_BATCH_SIZE="abc"'),
|
||||
);
|
||||
expect(warnSpy).toHaveBeenCalledWith(expect.stringContaining('Using default: 1000'));
|
||||
});
|
||||
|
||||
it('should log warning and use default when MEILI_SYNC_DELAY_MS is not a number', async () => {
|
||||
process.env.MEILI_SYNC_DELAY_MS = 'xyz';
|
||||
|
||||
await testCollection.insertMany([
|
||||
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: true },
|
||||
]);
|
||||
|
||||
const result = await batchResetMeiliFlags(testCollection);
|
||||
|
||||
expect(result).toBe(1);
|
||||
expect(warnSpy).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Invalid value for MEILI_SYNC_DELAY_MS="xyz"'),
|
||||
);
|
||||
expect(warnSpy).toHaveBeenCalledWith(expect.stringContaining('Using default: 100'));
|
||||
});
|
||||
|
||||
it('should log warning and use default when MEILI_SYNC_BATCH_SIZE is negative', async () => {
|
||||
process.env.MEILI_SYNC_BATCH_SIZE = '-50';
|
||||
|
||||
await testCollection.insertMany([
|
||||
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: true },
|
||||
]);
|
||||
|
||||
const result = await batchResetMeiliFlags(testCollection);
|
||||
|
||||
expect(result).toBe(1);
|
||||
expect(warnSpy).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Invalid value for MEILI_SYNC_BATCH_SIZE="-50"'),
|
||||
);
|
||||
});
|
||||
|
||||
it('should log warning and use default when MEILI_SYNC_DELAY_MS is negative', async () => {
|
||||
process.env.MEILI_SYNC_DELAY_MS = '-100';
|
||||
|
||||
await testCollection.insertMany([
|
||||
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: true },
|
||||
]);
|
||||
|
||||
const result = await batchResetMeiliFlags(testCollection);
|
||||
|
||||
expect(result).toBe(1);
|
||||
expect(warnSpy).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Invalid value for MEILI_SYNC_DELAY_MS="-100"'),
|
||||
);
|
||||
});
|
||||
|
||||
it('should accept valid positive integer values without warnings', async () => {
|
||||
process.env.MEILI_SYNC_BATCH_SIZE = '500';
|
||||
process.env.MEILI_SYNC_DELAY_MS = '50';
|
||||
|
||||
await testCollection.insertMany([
|
||||
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: true },
|
||||
]);
|
||||
|
||||
const result = await batchResetMeiliFlags(testCollection);
|
||||
|
||||
expect(result).toBe(1);
|
||||
expect(warnSpy).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('should log warning and use default when MEILI_SYNC_BATCH_SIZE is zero', async () => {
|
||||
process.env.MEILI_SYNC_BATCH_SIZE = '0';
|
||||
|
||||
await testCollection.insertMany([
|
||||
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: true },
|
||||
]);
|
||||
|
||||
const result = await batchResetMeiliFlags(testCollection);
|
||||
|
||||
expect(result).toBe(1);
|
||||
expect(warnSpy).toHaveBeenCalledWith(
|
||||
expect.stringContaining('MEILI_SYNC_BATCH_SIZE cannot be 0. Using default: 1000'),
|
||||
);
|
||||
});
|
||||
|
||||
it('should accept zero as a valid value for MEILI_SYNC_DELAY_MS without warnings', async () => {
|
||||
process.env.MEILI_SYNC_DELAY_MS = '0';
|
||||
|
||||
await testCollection.insertMany([
|
||||
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: true },
|
||||
]);
|
||||
|
||||
const result = await batchResetMeiliFlags(testCollection);
|
||||
|
||||
expect(result).toBe(1);
|
||||
expect(warnSpy).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('should not log warnings when environment variables are not set', async () => {
|
||||
delete process.env.MEILI_SYNC_BATCH_SIZE;
|
||||
delete process.env.MEILI_SYNC_DELAY_MS;
|
||||
|
||||
await testCollection.insertMany([
|
||||
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: true },
|
||||
]);
|
||||
|
||||
const result = await batchResetMeiliFlags(testCollection);
|
||||
|
||||
expect(result).toBe(1);
|
||||
expect(warnSpy).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
});
|
||||
+1
-1
@@ -1,4 +1,4 @@
|
||||
const mongoose = require('mongoose');
|
||||
const { mongoose } = require('~/db/base');
|
||||
const crypto = require('node:crypto');
|
||||
const { logger } = require('@librechat/data-schemas');
|
||||
const { getCustomEndpointConfig } = require('@hanzochat/api');
|
||||
|
||||
+1
-1
@@ -1,4 +1,4 @@
|
||||
const mongoose = require('mongoose');
|
||||
const { mongoose } = require('~/db/base');
|
||||
const { createMethods } = require('@librechat/data-schemas');
|
||||
const methods = createMethods(mongoose);
|
||||
const { comparePassword } = require('./userMethods');
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
const mongoose = require('mongoose');
|
||||
const { mongoose } = require('~/db/base');
|
||||
const { logger, hashToken, getRandomValues } = require('@librechat/data-schemas');
|
||||
const { createToken, findToken } = require('~/models');
|
||||
|
||||
|
||||
@@ -52,6 +52,7 @@
|
||||
"@node-saml/passport-saml": "^5.1.0",
|
||||
"@smithy/node-http-handler": "^4.4.5",
|
||||
"axios": "^1.13.5",
|
||||
"@hanzo/base": "^0.2.1",
|
||||
"bcryptjs": "^2.4.3",
|
||||
"compression": "^1.8.1",
|
||||
"connect-redis": "^8.1.0",
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
* @import { TUpdateResourcePermissionsRequest, TUpdateResourcePermissionsResponse } from 'librechat-data-provider'
|
||||
*/
|
||||
|
||||
const mongoose = require('mongoose');
|
||||
const { mongoose } = require('~/db/base');
|
||||
const { logger } = require('@librechat/data-schemas');
|
||||
const { ResourceType, PrincipalType, PermissionBits } = require('librechat-data-provider');
|
||||
const { enrichRemoteAgentPrincipals, backfillRemoteAgentPermissions } = require('@hanzochat/api');
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
const mongoose = require('mongoose');
|
||||
const { mongoose } = require('~/db/base');
|
||||
const { logger } = require('@librechat/data-schemas');
|
||||
const {
|
||||
MAX_SKILL_STATES,
|
||||
|
||||
@@ -1,5 +1,4 @@
|
||||
const express = require('express');
|
||||
const { MeiliSearch } = require('meilisearch');
|
||||
const { isEnabled } = require('@hanzochat/api');
|
||||
const requireJwtAuth = require('~/server/middleware/requireJwtAuth');
|
||||
|
||||
@@ -7,22 +6,15 @@ const router = express.Router();
|
||||
|
||||
router.use(requireJwtAuth);
|
||||
|
||||
/**
|
||||
* Reports whether conversation/message search is available.
|
||||
*
|
||||
* Search now runs on Hanzo Base/SQLite (via the data-layer adapter), so it is
|
||||
* always available unless explicitly disabled with `SEARCH=false`.
|
||||
*/
|
||||
router.get('/enable', async function (req, res) {
|
||||
if (!isEnabled(process.env.SEARCH)) {
|
||||
return res.send(false);
|
||||
}
|
||||
|
||||
try {
|
||||
const client = new MeiliSearch({
|
||||
host: process.env.MEILI_HOST,
|
||||
apiKey: process.env.MEILI_MASTER_KEY,
|
||||
});
|
||||
|
||||
const { status } = await client.health();
|
||||
return res.send(status === 'available');
|
||||
} catch (error) {
|
||||
return res.send(false);
|
||||
}
|
||||
const disabled = 'SEARCH' in process.env && !isEnabled(process.env.SEARCH);
|
||||
return res.send(!disabled);
|
||||
});
|
||||
|
||||
module.exports = router;
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
const mongoose = require('mongoose');
|
||||
const { mongoose } = require('~/db/base');
|
||||
const { isEnabled } = require('@hanzochat/api');
|
||||
const { getTransactionSupport, logger } = require('@librechat/data-schemas');
|
||||
const { ResourceType, PrincipalType, PrincipalModel } = require('librechat-data-provider');
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
const mongoose = require('mongoose');
|
||||
const { mongoose } = require('~/db/base');
|
||||
const { logger } = require('@librechat/data-schemas');
|
||||
const { mergeAppTools, getAppConfig } = require('./Config');
|
||||
const { createMCPServersRegistry, createMCPManager } = require('~/config');
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
const mongoose = require('mongoose');
|
||||
const { mongoose } = require('~/db/base');
|
||||
const { logger } = require('@librechat/data-schemas');
|
||||
const {
|
||||
logAgentMigrationWarning,
|
||||
|
||||
Generated
+124
-10
@@ -70,7 +70,7 @@ importers:
|
||||
version: 2.32.0(@typescript-eslint/parser@8.57.1(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3))(eslint-import-resolver-typescript@3.10.1)(eslint@9.39.4(jiti@1.21.7))
|
||||
eslint-plugin-jest:
|
||||
specifier: ^29.1.0
|
||||
version: 29.15.0(@typescript-eslint/eslint-plugin@8.57.1(@typescript-eslint/parser@8.57.1(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3))(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3))(eslint@9.39.4(jiti@1.21.7))(jest@30.3.0(@types/node@25.5.0)(ts-node@10.9.2(@types/node@25.5.0)(typescript@5.9.3)))(typescript@5.9.3)
|
||||
version: 29.15.0(@typescript-eslint/eslint-plugin@8.57.1(@typescript-eslint/parser@8.57.1(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3))(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3))(eslint@9.39.4(jiti@1.21.7))(jest@30.3.0)(typescript@5.9.3)
|
||||
eslint-plugin-jsx-a11y:
|
||||
specifier: ^6.10.2
|
||||
version: 6.10.2(eslint@9.39.4(jiti@1.21.7))
|
||||
@@ -94,7 +94,7 @@ importers:
|
||||
version: 9.1.7
|
||||
jest:
|
||||
specifier: ^30.2.0
|
||||
version: 30.3.0(@types/node@25.5.0)(ts-node@10.9.2(@types/node@25.5.0)(typescript@5.9.3))
|
||||
version: 30.3.0
|
||||
lint-staged:
|
||||
specifier: ^15.4.3
|
||||
version: 15.5.2
|
||||
@@ -137,6 +137,9 @@ importers:
|
||||
'@google/genai':
|
||||
specifier: ^1.19.0
|
||||
version: 1.46.0(@modelcontextprotocol/sdk@1.22.0(@cfworker/json-schema@4.1.1))
|
||||
'@hanzo/base':
|
||||
specifier: ^0.2.1
|
||||
version: 0.2.1(react@19.2.4)
|
||||
'@hanzochat/api':
|
||||
specifier: workspace:*
|
||||
version: link:../packages/api
|
||||
@@ -347,7 +350,7 @@ importers:
|
||||
devDependencies:
|
||||
jest:
|
||||
specifier: ^30.2.0
|
||||
version: 30.3.0(@types/node@20.19.37)(ts-node@10.9.2(@types/node@20.19.37)(typescript@5.9.3))
|
||||
version: 30.3.0
|
||||
mongodb-memory-server:
|
||||
specifier: ^10.1.4
|
||||
version: 10.4.3
|
||||
@@ -693,7 +696,7 @@ importers:
|
||||
version: 1.0.3
|
||||
eslint-plugin-jest:
|
||||
specifier: ^29.1.0
|
||||
version: 29.15.0(@typescript-eslint/eslint-plugin@8.57.1(@typescript-eslint/parser@8.57.1(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3))(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3))(eslint@9.39.4(jiti@1.21.7))(jest@30.3.0(@types/node@20.19.37)(ts-node@10.9.2(@types/node@20.19.37)(typescript@5.9.3)))(typescript@5.9.3)
|
||||
version: 29.15.0(@typescript-eslint/eslint-plugin@8.57.1(@typescript-eslint/parser@8.57.1(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3))(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3))(eslint@9.39.4(jiti@1.21.7))(jest@30.3.0(@types/node@20.19.37))(typescript@5.9.3)
|
||||
fs-extra:
|
||||
specifier: ^11.3.2
|
||||
version: 11.3.4
|
||||
@@ -702,7 +705,7 @@ importers:
|
||||
version: 3.0.0
|
||||
jest:
|
||||
specifier: ^30.2.0
|
||||
version: 30.3.0(@types/node@20.19.37)(ts-node@10.9.2(@types/node@20.19.37)(typescript@5.9.3))
|
||||
version: 30.3.0(@types/node@20.19.37)
|
||||
jest-canvas-mock:
|
||||
specifier: ^2.5.2
|
||||
version: 2.5.2
|
||||
@@ -1215,7 +1218,7 @@ importers:
|
||||
version: 2.4.4
|
||||
jest:
|
||||
specifier: ^30.2.0
|
||||
version: 30.3.0(@types/node@20.19.37)(ts-node@10.9.2(@types/node@20.19.37)(typescript@5.9.3))
|
||||
version: 30.3.0(@types/node@20.19.37)
|
||||
jest-junit:
|
||||
specifier: ^16.0.0
|
||||
version: 16.0.0
|
||||
@@ -3706,6 +3709,14 @@ packages:
|
||||
engines: {node: '>=6'}
|
||||
hasBin: true
|
||||
|
||||
'@hanzo/base@0.2.1':
|
||||
resolution: {integrity: sha512-eHVQXYr5m5RIG3EqCsXCB1urvLLiGk8F/Gr6d4WWK8H5G5RTooYS5MdNxb/FCs2iE/+VszvPkWV2jWxXC3Ct+A==}
|
||||
peerDependencies:
|
||||
react: '>=18.0.0'
|
||||
peerDependenciesMeta:
|
||||
react:
|
||||
optional: true
|
||||
|
||||
'@hanzo/iam@0.13.1':
|
||||
resolution: {integrity: sha512-MAqXY7jL5pcPkaUE0Qb/ePL2WKvk3p6EQDLHGZENj+/gbNKs7e/9qG+nmRlPMFVcwXPk+suzkBAEyIMT2EiJ8w==}
|
||||
engines: {node: '>=18'}
|
||||
@@ -17268,6 +17279,10 @@ snapshots:
|
||||
protobufjs: 7.5.4
|
||||
yargs: 17.7.2
|
||||
|
||||
'@hanzo/base@0.2.1(react@19.2.4)':
|
||||
optionalDependencies:
|
||||
react: 19.2.4
|
||||
|
||||
'@hanzo/iam@0.13.1(react@18.3.1)':
|
||||
dependencies:
|
||||
jose: 6.2.2
|
||||
@@ -17592,6 +17607,41 @@ snapshots:
|
||||
jest-util: 30.3.0
|
||||
slash: 3.0.0
|
||||
|
||||
'@jest/core@30.3.0':
|
||||
dependencies:
|
||||
'@jest/console': 30.3.0
|
||||
'@jest/pattern': 30.0.1
|
||||
'@jest/reporters': 30.3.0
|
||||
'@jest/test-result': 30.3.0
|
||||
'@jest/transform': 30.3.0
|
||||
'@jest/types': 30.3.0
|
||||
'@types/node': 20.19.37
|
||||
ansi-escapes: 4.3.2
|
||||
chalk: 4.1.2
|
||||
ci-info: 4.4.0
|
||||
exit-x: 0.2.2
|
||||
graceful-fs: 4.2.11
|
||||
jest-changed-files: 30.3.0
|
||||
jest-config: 30.3.0(@types/node@20.19.37)(ts-node@10.9.2(@types/node@20.19.37)(typescript@5.9.3))
|
||||
jest-haste-map: 30.3.0
|
||||
jest-message-util: 30.3.0
|
||||
jest-regex-util: 30.0.1
|
||||
jest-resolve: 30.3.0
|
||||
jest-resolve-dependencies: 30.3.0
|
||||
jest-runner: 30.3.0
|
||||
jest-runtime: 30.3.0
|
||||
jest-snapshot: 30.3.0
|
||||
jest-util: 30.3.0
|
||||
jest-validate: 30.3.0
|
||||
jest-watcher: 30.3.0
|
||||
pretty-format: 30.3.0
|
||||
slash: 3.0.0
|
||||
transitivePeerDependencies:
|
||||
- babel-plugin-macros
|
||||
- esbuild-register
|
||||
- supports-color
|
||||
- ts-node
|
||||
|
||||
'@jest/core@30.3.0(ts-node@10.9.2(@types/node@20.19.37)(typescript@5.9.3))':
|
||||
dependencies:
|
||||
'@jest/console': 30.3.0
|
||||
@@ -23184,24 +23234,24 @@ snapshots:
|
||||
- eslint-import-resolver-webpack
|
||||
- supports-color
|
||||
|
||||
eslint-plugin-jest@29.15.0(@typescript-eslint/eslint-plugin@8.57.1(@typescript-eslint/parser@8.57.1(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3))(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3))(eslint@9.39.4(jiti@1.21.7))(jest@30.3.0(@types/node@20.19.37)(ts-node@10.9.2(@types/node@20.19.37)(typescript@5.9.3)))(typescript@5.9.3):
|
||||
eslint-plugin-jest@29.15.0(@typescript-eslint/eslint-plugin@8.57.1(@typescript-eslint/parser@8.57.1(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3))(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3))(eslint@9.39.4(jiti@1.21.7))(jest@30.3.0(@types/node@20.19.37))(typescript@5.9.3):
|
||||
dependencies:
|
||||
'@typescript-eslint/utils': 8.57.1(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3)
|
||||
eslint: 9.39.4(jiti@1.21.7)
|
||||
optionalDependencies:
|
||||
'@typescript-eslint/eslint-plugin': 8.57.1(@typescript-eslint/parser@8.57.1(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3))(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3)
|
||||
jest: 30.3.0(@types/node@20.19.37)(ts-node@10.9.2(@types/node@20.19.37)(typescript@5.9.3))
|
||||
jest: 30.3.0(@types/node@20.19.37)
|
||||
typescript: 5.9.3
|
||||
transitivePeerDependencies:
|
||||
- supports-color
|
||||
|
||||
eslint-plugin-jest@29.15.0(@typescript-eslint/eslint-plugin@8.57.1(@typescript-eslint/parser@8.57.1(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3))(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3))(eslint@9.39.4(jiti@1.21.7))(jest@30.3.0(@types/node@25.5.0)(ts-node@10.9.2(@types/node@25.5.0)(typescript@5.9.3)))(typescript@5.9.3):
|
||||
eslint-plugin-jest@29.15.0(@typescript-eslint/eslint-plugin@8.57.1(@typescript-eslint/parser@8.57.1(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3))(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3))(eslint@9.39.4(jiti@1.21.7))(jest@30.3.0)(typescript@5.9.3):
|
||||
dependencies:
|
||||
'@typescript-eslint/utils': 8.57.1(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3)
|
||||
eslint: 9.39.4(jiti@1.21.7)
|
||||
optionalDependencies:
|
||||
'@typescript-eslint/eslint-plugin': 8.57.1(@typescript-eslint/parser@8.57.1(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3))(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3)
|
||||
jest: 30.3.0(@types/node@25.5.0)(ts-node@10.9.2(@types/node@25.5.0)(typescript@5.9.3))
|
||||
jest: 30.3.0
|
||||
typescript: 5.9.3
|
||||
transitivePeerDependencies:
|
||||
- supports-color
|
||||
@@ -24609,6 +24659,44 @@ snapshots:
|
||||
- babel-plugin-macros
|
||||
- supports-color
|
||||
|
||||
jest-cli@30.3.0:
|
||||
dependencies:
|
||||
'@jest/core': 30.3.0
|
||||
'@jest/test-result': 30.3.0
|
||||
'@jest/types': 30.3.0
|
||||
chalk: 4.1.2
|
||||
exit-x: 0.2.2
|
||||
import-local: 3.2.0
|
||||
jest-config: 30.3.0(@types/node@25.5.0)(ts-node@10.9.2(@types/node@25.5.0)(typescript@5.9.3))
|
||||
jest-util: 30.3.0
|
||||
jest-validate: 30.3.0
|
||||
yargs: 17.7.2
|
||||
transitivePeerDependencies:
|
||||
- '@types/node'
|
||||
- babel-plugin-macros
|
||||
- esbuild-register
|
||||
- supports-color
|
||||
- ts-node
|
||||
|
||||
jest-cli@30.3.0(@types/node@20.19.37):
|
||||
dependencies:
|
||||
'@jest/core': 30.3.0
|
||||
'@jest/test-result': 30.3.0
|
||||
'@jest/types': 30.3.0
|
||||
chalk: 4.1.2
|
||||
exit-x: 0.2.2
|
||||
import-local: 3.2.0
|
||||
jest-config: 30.3.0(@types/node@20.19.37)(ts-node@10.9.2(@types/node@20.19.37)(typescript@5.9.3))
|
||||
jest-util: 30.3.0
|
||||
jest-validate: 30.3.0
|
||||
yargs: 17.7.2
|
||||
transitivePeerDependencies:
|
||||
- '@types/node'
|
||||
- babel-plugin-macros
|
||||
- esbuild-register
|
||||
- supports-color
|
||||
- ts-node
|
||||
|
||||
jest-cli@30.3.0(@types/node@20.19.37)(ts-node@10.9.2(@types/node@20.19.37)(typescript@5.9.3)):
|
||||
dependencies:
|
||||
'@jest/core': 30.3.0(ts-node@10.9.2(@types/node@20.19.37)(typescript@5.9.3))
|
||||
@@ -25037,6 +25125,32 @@ snapshots:
|
||||
merge-stream: 2.0.0
|
||||
supports-color: 8.1.1
|
||||
|
||||
jest@30.3.0:
|
||||
dependencies:
|
||||
'@jest/core': 30.3.0
|
||||
'@jest/types': 30.3.0
|
||||
import-local: 3.2.0
|
||||
jest-cli: 30.3.0
|
||||
transitivePeerDependencies:
|
||||
- '@types/node'
|
||||
- babel-plugin-macros
|
||||
- esbuild-register
|
||||
- supports-color
|
||||
- ts-node
|
||||
|
||||
jest@30.3.0(@types/node@20.19.37):
|
||||
dependencies:
|
||||
'@jest/core': 30.3.0
|
||||
'@jest/types': 30.3.0
|
||||
import-local: 3.2.0
|
||||
jest-cli: 30.3.0(@types/node@20.19.37)
|
||||
transitivePeerDependencies:
|
||||
- '@types/node'
|
||||
- babel-plugin-macros
|
||||
- esbuild-register
|
||||
- supports-color
|
||||
- ts-node
|
||||
|
||||
jest@30.3.0(@types/node@20.19.37)(ts-node@10.9.2(@types/node@20.19.37)(typescript@5.9.3)):
|
||||
dependencies:
|
||||
'@jest/core': 30.3.0(ts-node@10.9.2(@types/node@20.19.37)(typescript@5.9.3))
|
||||
|
||||
Reference in New Issue
Block a user