Compare commits

...
Author SHA1 Message Date
hanzo-dev faf94027e7 fix(db/base): route SkillStatesController off mongoose onto Base facade
New upstream file (skills feature) added after the de-Mongo port; the
re-synced main reintroduced a direct `require('mongoose')` here. Swap it
to the `~/db/base` facade like every other app file — `mongoose.models.Skill`
resolves via the shared model registry (undefined-safe fallback preserved).
Keeps production `api/` free of any direct mongoose runtime require.
2026-07-04 13:02:36 -07:00
hanzo-dev 40c3277ea0 docs(db/base): record security invariants from blue+red review
(cherry picked from commit 5324854c44)
2026-07-04 13:01:59 -07:00
hanzo-dev ceadf083e3 fix(db/base): address red adversarial review (C1/H1/M1/M2/L1)
- C1 CRITICAL: block prototype pollution — getPath/setPath/unsetPath reject
  __proto__/constructor/prototype segments (dotted + bare). Attacker update
  keys (saveConvo, conversation import) can no longer poison Object.prototype.
- H1 HIGH: stop pushing Date predicates to Base filter DSL. Base normalizes
  date columns to a space separator; the ISO literal never matched, silently
  excluding records (superset-invariant violation). JS matcher handles dates.
- M1: partial UNIQUE indexes (WHERE != '') on sparse unique keys
  (googleId/openidId/username/…) — DB-level OAuth race safety without null
  conflicts. Live-verified on a fresh Base (user collection).
- M2: document.save() merges onto the stored record instead of full-replace,
  so a projected load (secrets hidden) can't erase select:false fields.
- L1: literal() drops non-finite numbers (no `= NaN` invalid DSL).
- meiliSearch applies deselected projection for read-path consistency.
- +8 regression specs (37 green): pollution guard, date-no-pushdown,
  date-equality read, save() secret preservation, non-finite guard.

(cherry picked from commit 61192a55fe)
2026-07-04 13:01:59 -07:00
hanzo-dev f9c0832100 docs(db/base): update phased status — Phases 1-3 done, remaining tail documented
(cherry picked from commit 362ce9a26e)
2026-07-04 13:01:59 -07:00
hanzo-dev fb28914c5c feat(db/base): Phase 3 — self-contained facade, drop mongoose from app code
- Facade no longer requires the mongoose driver: own ObjectId (isValid/
  createFromHexString/equals/toJSON), Schema.Types shim, no-op session/
  connection. api non-spec code has ZERO require('mongoose').
- Convert the 6 remaining app files (Agent, inviteUser, PermissionService,
  PermissionsController, initializeMCPs, migration) to the Base facade.
- Base UNIQUE indexes on always-present unique keys (conversationId, messageId,
  email, _id) for race-safety; sparse unique fields stay plain-indexed.
  Verified live: duplicate conversationId insert is rejected by Base.
- Query.session() no-op (data-schemas userGroup passes sessions).
- 29 jest green; boot-smoke + live-check (login+convo+msg+search+unique) pass.

(cherry picked from commit c33aa4aaf4)
2026-07-04 13:01:59 -07:00
hanzo-dev ab92d6e177 feat(db/base): Phase 2 — Base/SQLite full-text search, drop MeiliSearch
- BaseModel.meiliSearch now performs real user-scoped substring search over
  content fields (title, text) flagged meiliIndex, returning Meili-shaped
  { hits }. Conversation + Message search work on Base/SQLite.
- schema.js captures `searchable` content fields (meiliIndex string, minus ids).
- indexSync.js → no-op (no external index; drops mongoose + meilisearch imports).
- Remove dead Meili helper api/db/utils.js (+specs) and indexSync.spec.js.
- /search/enable reports availability from Base (SEARCH=false to disable).
- +1 unit spec (29 green); live-check asserts real title+text hits on Base.

(cherry picked from commit 6dc3d9b6fe)
2026-07-04 13:01:41 -07:00
hanzo-dev eba8e99e5f fix(db/base): honor Mongoose select:false + hydrate all Date fields (blue review)
- select:false secrets (password, totpSecret, backupCodes, keyHash, _meiliIndex)
  are now excluded from reads by default; only returned via explicit +field.
  Login path (findUser +password) still works. Prevents secret-hash exposure.
- Hydrate every Date-typed schema path (not just timestamps) to Date objects so
  callers can safely call .getTime()/date methods (e.g. session.expiration);
  cast epoch-number defaults (Date.now) to Date too.
- Projection engine reworked to Mongoose semantics (plain/+plus/-exclude).
- +5 specs (28 total green); live-check asserts default view hides the hash.

(cherry picked from commit 2886fce18f)
2026-07-04 13:01:10 -07:00
hanzo-dev 295f37eb65 feat(db): port data layer off MongoDB to Hanzo Base (Phase 1, #48)
Drop-mongo directive: replace MongoDB with a Mongoose-compatible adapter
over @hanzo/base. One facade fed to @librechat/data-schemas' createModels/
createMethods makes every model Base-backed with zero per-model porting.

- api/db/base/{objectId,query,schema,model,store,mongoose,index}.js — adapter
- Wire api/db/{connect,models,index}.js + api/models/index.js off mongoose
- query.spec.js + model.spec.js (jest, in-memory store, 23 tests green)
- scripts/{live-check,boot-smoke}.js prove login+convo+message persist on Base

Nothing calls mongoose.connect; hot path off mongoose. FTS (Meili) + the 7
remaining ObjectId/migration mongoose imports are Phase 2/3 (see LLM.md).

(cherry picked from commit c1ef90baa2)
2026-07-04 13:01:10 -07:00
31 changed files with 2955 additions and 1551 deletions
+117 -1
View File
@@ -46,7 +46,8 @@ npm run format # Prettier
```
api/ # Express backend (port 3080)
server/ # Entry point, routes, controllers, middleware
models/ # Mongoose models (MongoDB)
db/base/ # Hanzo Base data-layer adapter (replaces MongoDB) — see below
models/ # Model method wrappers (persist to Base via the adapter)
client/ # React frontend (Vite)
src/components/ # UI components
src/routes/ # Client-side routing
@@ -60,6 +61,121 @@ packages/
mcp/ # MCP server integration
```
## Data Layer — Hanzo Base (MongoDB dropped)
Hanzo Chat does **not** use MongoDB. Persistence runs on **Hanzo Base** (the Go
PocketBase-lineage server; SQLite embedded for dev, Postgres for prod). The
"drop mongo completely" directive (#48) is realised by a thin adapter under
`api/db/base/` that presents a Mongoose-Model-compatible surface backed by the
`@hanzo/base` JS client. There is **one** adapter; every model comes along.
### How it works (one adapter, all models)
`@librechat/data-schemas` builds its schemas with real `mongoose` purely as a
schema DSL, then asks a mongoose instance to turn them into models
(`createModels`/`createMethods`). We hand it a **facade** instead of real
mongoose:
- `api/db/base/mongoose.js` — a `mongoose`-compatible facade. Delegates
schema/ObjectId/type concerns to real mongoose (never connected) but
overrides `model()`/`models`/`connect`/`connection`. `model(name, schema)`
returns a Mongoose-style **constructor** (statics + `new Model(data)` docs).
- `api/db/base/model.js``BaseModel`: the Mongoose Model static surface
(`find/findOne/findOneAndUpdate/create/insertMany/updateOne/updateMany/
deleteOne/deleteMany/countDocuments/bulkWrite/aggregate/distinct/exists`),
a chainable/thenable `Query` (`select/sort/limit/skip/lean/populate` +
`find(A).deleteMany(B)`), and lightweight documents (`toObject/save`).
- `api/db/base/query.js` — a correct in-JS Mongo query/update engine
(`$or/$and/$in/$nin/$gt../$exists/$regex/$elemMatch`, update ops
`$set/$unset/$inc/$push/$addToSet/$pull/$setOnInsert`, projection, sort).
This is the correctness backstop — every predicate is evaluated here.
- `api/db/base/store.js` — the Base backend via `@hanzo/base` `BaseClient`.
Each model → one Base collection whose records hold the **full document as a
JSON `data` field** plus promoted scalar columns (from indexed/unique schema
paths) used only for **filter pushdown** (a guaranteed superset; the JS
engine then narrows). `@hanzo/base` is ESM-only → loaded via dynamic
`import()` from this CommonJS code.
- `api/db/base/schema.js` — introspects a Mongoose schema for defaults,
timestamps, and promotable columns.
- `api/db/base/index.js``connectDb()` inits the Base client, health-checks,
and provisions every registered model's collection (idempotent).
### Wiring (all app code off mongoose)
`api/db/{connect,models,index}.js`, `api/models/index.js`, and every remaining
app file (`Agent`, `inviteUser`, `PermissionService`, `PermissionsController`,
`initializeMCPs`, `migration`) import the facade (`~/db/base`) — not `mongoose`.
The facade itself carries **no `mongoose` runtime dependency** (own ObjectId,
Schema.Types shim, no-op session/connection). `grep "require('mongoose')"` over
non-test `api/` is **zero**; **nothing calls `mongoose.connect`**. `_id` is a
real 24-hex ObjectId string stored inside the document; conversations/messages
remain keyed by their `conversationId`/`messageId` UUIDs.
### Search (FTS on Base/SQLite)
MeiliSearch is dropped. `BaseModel.meiliSearch(query, {filter})` runs a
user-scoped, case-insensitive substring search over content fields flagged
`meiliIndex` (title, text), returning Meili-shaped `{ hits }`. `indexSync.js`
is a no-op (no external index); `/search/enable` reports availability from Base
(set `SEARCH=false` to disable). A future optimisation is a SQLite FTS5 index
instead of the per-user scan.
### Security invariants (do not regress — from blue+red review)
- **Prototype pollution:** `query.js` path helpers reject `__proto__` /
`constructor` / `prototype` segments (attacker update keys flow in via
`saveConvo` / conversation import). Keep the guard.
- **`select:false`:** secrets (password, totpSecret, backupCodes, keyHash) are
excluded from reads by default; only `+field` returns them. `save()` merges
onto the stored record so a projected load never erases them.
- **Pushdown = SUPERSET:** `store.js buildFilter` must only ever broaden; the JS
matcher narrows. Never push a predicate that could wrongly exclude — Date
columns and non-finite numbers are intentionally NOT pushed (Base date-column
normalization vs ISO would silently drop records).
- **DSL injection:** `quote()` escapes `\` and `'`; verified a crafted
`conversationId` cannot inject `||`/operators.
### Uniqueness / concurrency
Collections carry a Base **UNIQUE** index on always-present unique keys (`_id`,
`conversationId`, `messageId`, `email`); optional/sparse unique fields
(`googleId`, `openidId`, …) stay plain-indexed to avoid null conflicts, with
logical uniqueness enforced by the adapter upsert. The unique index is the
race-safety net (verified: a duplicate `conversationId` insert is rejected).
### Env / running
```
HANZO_BASE_URL=http://… # Hanzo Base instance (replaces MONGO_URI)
HANZO_BASE_TOKEN=… # Base superuser/service token (IAM in prod)
```
Local Base for dev: build a pure-SQLite launcher from `~/work/hanzo/base`
(`base.New()` without the IAM platform plugin), `serve --http=127.0.0.1:8090`;
the first serve prints a 30-min superuser installer token. API prefix is `/v1`.
### Tests
- `api/db/base/query.spec.js`, `api/db/base/model.spec.js` — jest, in-memory
store, no Base/Mongo needed (CI). `cd api && npx jest db/base`.
- `api/db/base/scripts/live-check.js` — real data-schemas factories vs a live
Base: createUser + balance grant + findUser + bcrypt + saveConvo + saveMessage,
verified via raw Base REST. Env-gated (`HANZO_BASE_URL`/`HANZO_BASE_TOKEN`).
- `api/db/base/scripts/boot-smoke.js` — drives the wired `~/db` + `~/models`
boot path (`connectDb` + `seedDatabase`) against live Base.
### Phased status
- **Phase 1 (done):** adapter + core hot path verified on Base (User, Balance,
Conversation, Message, Agent, plus Role/Session/Token/Category via the same
adapter — 29 collections provisioned). Nothing connects to Mongo.
- **Phase 2 (done):** Base/SQLite full-text search replaces MeiliSearch;
`indexSync` no-op; dead Meili helpers removed. Live-verified real hits.
- **Phase 3 (done):** all app code off `mongoose` (self-contained facade);
Base UNIQUE indexes on natural keys (race-safety, live-verified);
`select:false` secrets honored; all Date fields hydrated.
- **Remaining tail (test migration):** ~28 legacy `*.spec.js` still build models
on real `mongoose` + `mongodb-memory-server` (e.g. `api/models/*.spec.js`).
They're incompatible with the port (they expect mongoose-backed models) and
must be migrated to the adapter (in-memory store harness) — that is what gates
dropping the `mongoose`/`mongodb-memory-server` devDeps and a literal
`grep -riE 'mongoose|mongodb' api/` = 0. The adapter's own suites
(`api/db/base/*.spec.js`) are the migration template.
- **Deploy:** build image via CI → `registry.hanzo.ai`, deploy to DOKS with a
Base instance + IAM service token (`HANZO_BASE_URL`/`HANZO_BASE_TOKEN`), then
Playwright-verify login+chat+agent+search in the browser. Gated on cluster access.
## Configuration
- `librechat.yaml` (or ConfigMap `chat-config` -> `/app/librechat.yaml`)
+63
View File
@@ -0,0 +1,63 @@
'use strict';
/**
* Hanzo Base data-layer adapter — entry point.
*
* Exposes:
* - `mongoose` : the mongoose-compatible facade (feed to createModels/createMethods)
* - `connectDb()`: initialise the Base client and provision collections
* - `store` : the underlying DocumentStore (for tests/introspection)
*/
const { logger } = require('@librechat/data-schemas');
const mongoose = require('./mongoose');
const store = require('./store');
function resolveConfig() {
const url = process.env.HANZO_BASE_URL || process.env.BASE_URL || null;
const token = process.env.HANZO_BASE_TOKEN || process.env.BASE_TOKEN || null;
return { url, token };
}
let ready = null;
/**
* Connect to Hanzo Base and ensure every registered model's collection exists.
* Idempotent: repeated calls return the same in-flight/settled promise.
*/
async function connectDb() {
if (ready) {
return ready;
}
ready = (async () => {
const { url, token } = resolveConfig();
if (!url) {
throw new Error(
'[BaseAdapter] Please define HANZO_BASE_URL (Hanzo Base instance URL) for the data layer',
);
}
await store.init({ url, token });
try {
await store.health();
logger.info(`[BaseAdapter] Connected to Hanzo Base at ${url}`);
} catch (err) {
logger.error('[BaseAdapter] Base health check failed', err);
throw err;
}
await ensureCollections();
return mongoose;
})();
return ready;
}
/** Provision the Base collection for every model registered on the facade. */
async function ensureCollections() {
const registered = mongoose.models;
const names = Object.keys(registered);
for (const name of names) {
await registered[name].ensure();
}
logger.info(`[BaseAdapter] Provisioned ${names.length} Base collections`);
}
module.exports = { mongoose, store, connectDb, ensureCollections };
+828
View File
@@ -0,0 +1,828 @@
'use strict';
/**
* BaseModel — a Mongoose-Model-compatible facade backed by Hanzo Base.
*
* One instance per collection. It presents the subset of the Mongoose Model
* static API that the LibreChat data layer actually uses (find / findOne /
* findOneAndUpdate / create / updateOne / deleteMany / countDocuments /
* bulkWrite / aggregate …) and executes it against a document `store`
* (see store.js) whose records hold the full Mongo document as JSON.
*
* Correctness comes from query.js: candidate records are fetched from Base
* (with best-effort filter pushdown) and every predicate, update operator,
* projection and sort is then evaluated in JS.
*/
const {
matches,
applyUpdate,
parseProjection,
applyProjection,
sortDocs,
getPath,
setPath,
idString,
} = require('./query');
const { describeSchema, resolveDefault } = require('./schema');
const { generateObjectId } = require('./objectId');
function now() {
return new Date();
}
/**
* Parse a MeiliSearch-style filter string (e.g. `user = "abc"`) into a Mongo
* equality filter for scoping. Supports the `field = "value"` / `field = 'value'`
* clauses the chat app emits; unknown syntax is ignored (no scoping).
*/
function parseMeiliFilter(filter) {
const scope = {};
if (typeof filter !== 'string') {
return scope;
}
const re = /(\w+)\s*=\s*(?:"([^"]*)"|'([^']*)')/g;
let m;
while ((m = re.exec(filter)) !== null) {
scope[m[1]] = m[2] !== undefined ? m[2] : m[3];
}
return scope;
}
/** Extract top-level equality conditions from a filter (for upsert seeding). */
function equalitySeed(filter) {
const seed = {};
if (!filter || typeof filter !== 'object') {
return seed;
}
for (const [k, v] of Object.entries(filter)) {
if (k.startsWith('$')) {
continue;
}
if (v === null || typeof v !== 'object' || v instanceof Date || Array.isArray(v)) {
seed[k] = v;
}
}
return seed;
}
class BaseModel {
/**
* @param {string} name - Mongoose model name (e.g. "Conversation")
* @param {import('mongoose').Schema} schema
* @param {import('./store').DocumentStore} store
*/
constructor(name, schema, store) {
this.modelName = name;
this.schema = schema;
this.store = store;
this.collection = { collectionName: name.toLowerCase() };
this.collectionName = name.toLowerCase();
const desc = describeSchema(schema);
this._defaults = desc.defaults;
this._timestamps = desc.timestamps;
this._promoted = desc.promoted;
this._promotedNames = desc.promoted.map((p) => p.name);
this._deselected = desc.deselected;
this._dateFields = desc.dateFields;
this._searchable = desc.searchable;
}
/** Register/verify this collection's schema in Base. Idempotent. */
async ensure() {
await this.store.ensureCollection(this.collectionName, this._promoted);
}
// ---- internal helpers -------------------------------------------------
/** Build the persisted record { promoted columns…, data } from a Mongo doc. */
_toRecord(doc) {
const record = { data: doc };
for (const col of this._promoted) {
let v = idString(getPath(doc, col.name));
if (v === undefined) {
v = null;
}
if (v instanceof Date) {
v = v.toISOString();
}
record[col.name] = v;
}
return record;
}
/** Materialize a stored Mongo doc: all Date-typed fields become Date objects. */
_hydrate(doc) {
for (const f of this._dateFields) {
const v = getPath(doc, f);
// Cast ISO strings and epoch numbers (e.g. `default: Date.now`) to Date.
if (typeof v === 'string' || typeof v === 'number') {
setPath(doc, f, new Date(v));
}
}
return doc;
}
/** Apply schema defaults for keys absent from doc. */
_applyDefaults(doc) {
for (const [name, def] of this._defaults) {
if (doc[name] === undefined) {
const value = resolveDefault(def);
if (value !== undefined) {
doc[name] = value;
}
}
}
}
/** Fetch candidate {baseId, doc} records for a filter (pushdown + always safe). */
async _candidates(filter) {
const raw = await this.store.list(this.collectionName, filter, this._promotedNames);
return raw;
}
_newDoc(seed) {
const doc = { ...seed };
if (!doc._id) {
doc._id = generateObjectId();
}
this._applyDefaults(doc);
if (this._timestamps) {
const ts = now();
if (doc[this._timestamps.createdAt] === undefined) {
doc[this._timestamps.createdAt] = ts;
}
doc[this._timestamps.updatedAt] = ts;
}
return doc;
}
async _insert(doc) {
const stored = { ...doc };
// Persist Dates as ISO strings inside the JSON blob.
if (this._timestamps) {
for (const f of [this._timestamps.createdAt, this._timestamps.updatedAt]) {
if (stored[f] instanceof Date) {
stored[f] = stored[f].toISOString();
}
}
}
const rec = await this.store.create(this.collectionName, this._toRecord(stored));
return rec.baseId;
}
async _replace(baseId, doc, { touch = true } = {}) {
if (this._timestamps && touch) {
doc[this._timestamps.updatedAt] = now(); // reflect fresh updatedAt on the caller's doc
}
const stored = { ...doc };
if (this._timestamps) {
for (const f of [this._timestamps.createdAt, this._timestamps.updatedAt]) {
if (stored[f] instanceof Date) {
stored[f] = stored[f].toISOString();
}
}
}
await this.store.update(this.collectionName, baseId, this._toRecord(stored));
}
// ---- write operations -------------------------------------------------
async _create(input) {
const doc = this._newDoc(input);
await this._insert(doc);
return this._hydrate({ ...doc });
}
/** `new Model(data)` — build an unsaved document (defaults + _id) with .save(). */
_newDocument(data = {}) {
return makeDocument(this, this._hydrate(this._newDoc(data)));
}
async create(input) {
if (Array.isArray(input)) {
const out = [];
for (const item of input) {
out.push(makeDocument(this, await this._create(item)));
}
return out;
}
return makeDocument(this, await this._create(input));
}
async insertMany(docs, options = {}) {
const created = [];
for (const d of docs) {
created.push(await this._create(d));
}
if (options.lean) {
return created;
}
return created.map((d) => makeDocument(this, d));
}
/**
* Core upsert/update primitive shared by findOneAndUpdate/updateOne/updateMany.
* @returns {{ doc: object|null, matched: number, modified: number, upserted: boolean }}
*/
async _updateOneInternal(filter, update, options = {}) {
const candidates = await this._candidates(filter);
const hit = candidates.find((c) => matches(c.doc, filter));
if (hit) {
const before = this._hydrate({ ...hit.doc });
const next = { ...hit.doc };
applyUpdate(next, update, { isInsert: false });
await this._replace(hit.baseId, next, { touch: options.timestamps !== false });
return { doc: this._hydrate(next), before, matched: 1, modified: 1, upserted: false };
}
if (options.upsert) {
const seed = equalitySeed(filter);
const doc = { ...seed };
applyUpdate(doc, update, { isInsert: true });
const full = this._newDoc(doc);
await this._insert(full);
return { doc: this._hydrate(full), before: null, matched: 0, modified: 0, upserted: true };
}
return { doc: null, before: null, matched: 0, modified: 0, upserted: false };
}
findOneAndUpdate(filter, update, options = {}) {
return new Query(this, 'findOneAndUpdate', { filter, update, options });
}
findByIdAndUpdate(id, update, options = {}) {
return this.findOneAndUpdate({ _id: id }, update, options);
}
findOneAndDelete(filter) {
return new Query(this, 'findOneAndDelete', { filter });
}
async updateOne(filter, update, options = {}) {
const r = await this._updateOneInternalPublic(filter, update, options);
return r;
}
async _updateOneInternalPublic(filter, update, options) {
const r = await this._updateOneInternal(filter, update, options);
return {
acknowledged: true,
matchedCount: r.matched,
modifiedCount: r.modified,
upsertedCount: r.upserted ? 1 : 0,
upsertedId: r.upserted ? r.doc._id : null,
};
}
async updateMany(filter, update, options = {}) {
const candidates = await this._candidates(filter);
const hits = candidates.filter((c) => matches(c.doc, filter));
for (const hit of hits) {
const next = { ...hit.doc };
applyUpdate(next, update, { isInsert: false });
await this._replace(hit.baseId, next, { touch: options.timestamps !== false });
}
if (!hits.length && options.upsert) {
const seed = equalitySeed(filter);
const doc = { ...seed };
applyUpdate(doc, update, { isInsert: true });
await this._insert(this._newDoc(doc));
return { acknowledged: true, matchedCount: 0, modifiedCount: 0, upsertedCount: 1 };
}
return {
acknowledged: true,
matchedCount: hits.length,
modifiedCount: hits.length,
upsertedCount: 0,
};
}
async _deleteWhere(filter, limit) {
const candidates = await this._candidates(filter);
const hits = candidates.filter((c) => matches(c.doc, filter));
const toDelete = limit ? hits.slice(0, limit) : hits;
for (const hit of toDelete) {
await this.store.delete(this.collectionName, hit.baseId);
}
return { acknowledged: true, deletedCount: toDelete.length };
}
deleteOne(filter) {
return this._deleteWhere(filter, 1);
}
deleteMany(filter) {
return this._deleteWhere(filter || {}, 0);
}
// ---- read operations --------------------------------------------------
find(filter = {}, projection, options) {
return new Query(this, 'find', { filter, projection, options });
}
findOne(filter = {}, projection, options) {
return new Query(this, 'findOne', { filter, projection, options });
}
findById(id, projection, options) {
return new Query(this, 'findOne', { filter: { _id: id }, projection, options });
}
async countDocuments(filter = {}) {
const candidates = await this._candidates(filter);
return candidates.filter((c) => matches(c.doc, filter)).length;
}
async estimatedDocumentCount() {
const candidates = await this._candidates({});
return candidates.length;
}
async distinct(field, filter = {}) {
const candidates = await this._candidates(filter);
const set = new Set();
for (const c of candidates) {
if (!matches(c.doc, filter)) {
continue;
}
const v = getPath(c.doc, field);
if (Array.isArray(v)) {
v.forEach((x) => set.add(x));
} else if (v !== undefined) {
set.add(v);
}
}
return [...set];
}
async exists(filter) {
const candidates = await this._candidates(filter);
const hit = candidates.find((c) => matches(c.doc, filter));
return hit ? { _id: hit.doc._id } : null;
}
/**
* Full-text search over Base/SQLite (replaces MeiliSearch).
* Case-insensitive substring match across the model's searchable content
* fields (title, text, …), scoped by the Meili-style `opts.filter` string
* (e.g. `user = "abc"`). Returns MeiliSearch-shaped `{ hits }`.
*/
async meiliSearch(query, opts = {}) {
const terms = (query || '').trim().toLowerCase();
if (!terms || !this._searchable.size) {
return { hits: [] };
}
const scope = parseMeiliFilter(opts.filter);
const candidates = await this._candidates(scope);
const hits = [];
for (const c of candidates) {
if (!matches(c.doc, scope)) {
continue;
}
for (const field of this._searchable) {
const value = getPath(c.doc, field);
if (typeof value === 'string' && value.toLowerCase().includes(terms)) {
hits.push(applyProjection(this._hydrate({ ...c.doc }), null, this._deselected));
break;
}
}
}
return { hits };
}
async syncWithMeili() {
/* no-op: MeiliSearch sync disabled under Base adapter (Phase 2 FTS). */
}
async bulkWrite(operations = []) {
const result = {
insertedCount: 0,
matchedCount: 0,
modifiedCount: 0,
deletedCount: 0,
upsertedCount: 0,
upsertedIds: {},
insertedIds: {},
};
for (const op of operations) {
if (op.insertOne) {
const doc = await this._create(op.insertOne.document);
result.insertedIds[result.insertedCount] = doc._id;
result.insertedCount += 1;
} else if (op.updateOne) {
const { filter, update, upsert, timestamps } = op.updateOne;
const r = await this._updateOneInternal(filter, update, { upsert, timestamps });
result.matchedCount += r.matched;
result.modifiedCount += r.modified;
if (r.upserted) {
result.upsertedIds[result.upsertedCount] = r.doc._id;
result.upsertedCount += 1;
}
} else if (op.updateMany) {
const { filter, update, upsert, timestamps } = op.updateMany;
const r = await this.updateMany(filter, update, { upsert, timestamps });
result.matchedCount += r.matchedCount;
result.modifiedCount += r.modifiedCount;
result.upsertedCount += r.upsertedCount || 0;
} else if (op.deleteOne) {
const r = await this._deleteWhere(op.deleteOne.filter, 1);
result.deletedCount += r.deletedCount;
} else if (op.deleteMany) {
const r = await this._deleteWhere(op.deleteMany.filter, 0);
result.deletedCount += r.deletedCount;
}
}
return result;
}
/** Minimal aggregation engine (used by Prompt / agentCategory, not the hot path). */
async aggregate(pipeline = []) {
const candidates = await this._candidates({});
let docs = candidates.map((c) => this._hydrate({ ...c.doc }));
for (const stage of pipeline) {
const op = Object.keys(stage)[0];
const arg = stage[op];
switch (op) {
case '$match':
docs = docs.filter((d) => matches(d, arg));
break;
case '$sort':
docs = sortDocs(docs, arg);
break;
case '$limit':
docs = docs.slice(0, arg);
break;
case '$skip':
docs = docs.slice(arg);
break;
case '$count':
docs = [{ [arg]: docs.length }];
break;
case '$project':
docs = docs.map((d) => applyProjection(d, parseProjection(arg)));
break;
case '$unwind': {
const field = (typeof arg === 'string' ? arg : arg.path).replace(/^\$/, '');
const next = [];
for (const d of docs) {
const v = getPath(d, field);
if (Array.isArray(v)) {
for (const item of v) {
next.push({ ...d, [field]: item });
}
} else if (v !== undefined) {
next.push(d);
}
}
docs = next;
break;
}
case '$group':
docs = groupStage(docs, arg);
break;
default:
throw new Error(`[BaseModel.aggregate] Unsupported stage: ${op}`);
}
}
return docs;
}
}
/** $group implementation for the aggregation engine. */
function groupStage(docs, spec) {
const { _id: idSpec, ...accs } = spec;
const groups = new Map();
const keyOf = (d) => {
if (idSpec === null) {
return '__null__';
}
if (typeof idSpec === 'string' && idSpec.startsWith('$')) {
return JSON.stringify(getPath(d, idSpec.slice(1)) ?? null);
}
return JSON.stringify(idSpec);
};
const idValue = (d) => {
if (idSpec === null) {
return null;
}
if (typeof idSpec === 'string' && idSpec.startsWith('$')) {
return getPath(d, idSpec.slice(1)) ?? null;
}
return idSpec;
};
for (const d of docs) {
const k = keyOf(d);
if (!groups.has(k)) {
groups.set(k, { _id: idValue(d), _docs: [] });
}
groups.get(k)._docs.push(d);
}
const out = [];
for (const g of groups.values()) {
const row = { _id: g._id };
for (const [field, acc] of Object.entries(accs)) {
const accOp = Object.keys(acc)[0];
const expr = acc[accOp];
const values = g._docs.map((d) =>
typeof expr === 'string' && expr.startsWith('$') ? getPath(d, expr.slice(1)) : expr,
);
switch (accOp) {
case '$sum':
row[field] = values.reduce((s, v) => s + (Number(v) || (expr === 1 ? 1 : 0)), 0);
break;
case '$count':
row[field] = g._docs.length;
break;
case '$avg':
row[field] = values.reduce((s, v) => s + (Number(v) || 0), 0) / (values.length || 1);
break;
case '$min':
row[field] = values.reduce((m, v) => (m == null || v < m ? v : m), null);
break;
case '$max':
row[field] = values.reduce((m, v) => (m == null || v > m ? v : m), null);
break;
case '$first':
row[field] = values[0];
break;
case '$last':
row[field] = values[values.length - 1];
break;
case '$push':
row[field] = values;
break;
case '$addToSet':
row[field] = [...new Set(values)];
break;
default:
throw new Error(`[BaseModel.aggregate] Unsupported accumulator: ${accOp}`);
}
}
out.push(row);
}
return out;
}
/**
* Query — a lazy, chainable, thenable wrapper mirroring the Mongoose Query API
* subset in use: select / sort / limit / skip / lean / populate / collation,
* plus terminal execution when awaited (or via .then/.exec).
*/
class Query {
constructor(model, op, params) {
this.model = model;
this.op = op;
this.params = params;
this._projection = params.projection;
this._sort = null;
this._limit = null;
this._skip = null;
this._lean = false;
}
select(projection) {
this._projection = projection;
return this;
}
sort(spec) {
this._sort = spec;
return this;
}
limit(n) {
this._limit = n;
return this;
}
skip(n) {
this._skip = n;
return this;
}
lean() {
this._lean = true;
return this;
}
populate() {
return this; // refs are stored inline as ids; no server-side population
}
collation() {
return this;
}
session() {
return this; // Base has no transactions; sessions are inert
}
hint() {
return this;
}
read() {
return this;
}
/** Merge an additional filter (Mongoose `Model.find(A).find(B)` / `.deleteMany(B)`). */
find(extra) {
this.params.filter = mergeFilter(this.params.filter, extra);
return this;
}
/** `Model.find(A).deleteMany(B)` — delete with merged conditions. */
deleteMany(extra) {
const filter = mergeFilter(this.params.filter, extra);
return this.model._deleteWhere(filter, 0);
}
deleteOne(extra) {
const filter = mergeFilter(this.params.filter, extra);
return this.model._deleteWhere(filter, 1);
}
countDocuments() {
return this.model.countDocuments(this.params.filter);
}
async _run() {
const model = this.model;
if (this.op === 'findOneAndUpdate') {
const r = await model._updateOneInternal(
this.params.filter,
this.params.update,
this.params.options,
);
// Mongoose returns the pre-update doc when `new` is not truthy (default),
// and the post-update doc when `{ new: true }`. Upserts with `new:false`
// return null (there was no prior document).
const wantNew = !!(this.params.options && this.params.options.new);
let doc = wantNew ? r.doc : r.upserted ? null : r.before;
if (!doc) {
return null;
}
doc = applyProjection(doc, parseProjection(this._projection), model._deselected);
return this._lean ? doc : makeDocument(model, doc);
}
if (this.op === 'findOneAndDelete') {
const candidates = await model._candidates(this.params.filter);
const hit = candidates.find((c) => matches(c.doc, this.params.filter));
if (!hit) {
return null;
}
await model.store.delete(model.collectionName, hit.baseId);
const doc = applyProjection(
model._hydrate({ ...hit.doc }),
parseProjection(this._projection),
model._deselected,
);
return this._lean ? doc : makeDocument(model, doc);
}
// read path (find / findOne)
const candidates = await model._candidates(this.params.filter);
let docs = candidates.filter((c) => matches(c.doc, this.params.filter)).map((c) => c.doc);
if (this._sort) {
docs = sortDocs(docs, this._sort);
}
if (this._skip) {
docs = docs.slice(this._skip);
}
if (this.op === 'findOne') {
const doc = docs[0];
if (!doc) {
return null;
}
const out = applyProjection(
model._hydrate({ ...doc }),
parseProjection(this._projection),
model._deselected,
);
return this._lean ? out : makeDocument(model, out);
}
if (this._limit != null) {
docs = docs.slice(0, this._limit);
}
const projection = parseProjection(this._projection);
const out = docs.map((d) =>
applyProjection(model._hydrate({ ...d }), projection, model._deselected),
);
return this._lean ? out : out.map((d) => makeDocument(model, d));
}
exec() {
return this._run();
}
then(onFulfilled, onRejected) {
return this._run().then(onFulfilled, onRejected);
}
catch(onRejected) {
return this._run().catch(onRejected);
}
finally(onFinally) {
return this._run().finally(onFinally);
}
}
function mergeFilter(a, b) {
if (!b || !Object.keys(b).length) {
return a;
}
if (!a || !Object.keys(a).length) {
return b;
}
return { $and: [a, b] };
}
/**
* Wrap a plain doc as a lightweight "document": all data fields are enumerable
* (so spreads, `.map`, JSON all behave), while toObject/save/etc. are hidden.
*/
function makeDocument(model, doc) {
if (doc == null) {
return doc;
}
const defineHidden = (name, value) =>
Object.defineProperty(doc, name, { value, enumerable: false, writable: true, configurable: true });
defineHidden('toObject', function toObject() {
const plain = {};
for (const k of Object.keys(this)) {
plain[k] = this[k];
}
return plain;
});
defineHidden('toJSON', doc.toObject);
defineHidden('save', async function save() {
const candidates = await model._candidates({ _id: this._id });
const plain = this.toObject();
const hit = candidates.find((c) => c.doc._id === this._id);
if (hit) {
// Merge over the stored record so a doc loaded with a narrowed projection
// (e.g. without select:false secrets) never erases the unloaded fields.
await model._replace(hit.baseId, { ...hit.doc, ...plain }, { touch: true });
} else {
await model._insert(model._newDoc(plain));
}
return this;
});
defineHidden('deleteOne', async function deleteOne() {
return model._deleteWhere({ _id: this._id }, 1);
});
if (doc._id !== undefined) {
defineHidden('id', String(doc._id));
}
return doc;
}
/** Static methods exposed on the constructor returned by `mongoose.model()`. */
const MODEL_STATICS = [
'find',
'findOne',
'findById',
'findOneAndUpdate',
'findByIdAndUpdate',
'findOneAndDelete',
'create',
'insertMany',
'updateOne',
'updateMany',
'deleteOne',
'deleteMany',
'countDocuments',
'estimatedDocumentCount',
'distinct',
'exists',
'bulkWrite',
'aggregate',
'meiliSearch',
'syncWithMeili',
'ensure',
];
/**
* Wrap a BaseModel as a Mongoose-style model constructor:
* - `new Model(data)` creates an unsaved document (with `.save()`)
* - `Model.find(...)`, `Model.findOneAndUpdate(...)`, … are the statics
*/
function makeModelCtor(baseModel) {
function ModelCtor(data) {
return baseModel._newDocument(data);
}
for (const name of MODEL_STATICS) {
ModelCtor[name] = baseModel[name].bind(baseModel);
}
ModelCtor.modelName = baseModel.modelName;
ModelCtor.schema = baseModel.schema;
ModelCtor.collection = baseModel.collection;
ModelCtor.base = baseModel;
return ModelCtor;
}
module.exports = { BaseModel, Query, makeDocument, makeModelCtor };
+284
View File
@@ -0,0 +1,284 @@
'use strict';
/*
* BaseModel behaviour over an in-memory store (no Base/Mongo required).
* Exercises the exact Mongoose call patterns the chat hot path relies on.
*/
const { Schema } = require('mongoose');
const { BaseModel } = require('./model');
/** In-memory backend mirroring DocumentStore's interface. */
function memStore() {
const cols = {};
let seq = 0;
const clone = (x) => JSON.parse(JSON.stringify(x));
return {
async ensureCollection(name) {
cols[name] = cols[name] || new Map();
},
async list(name) {
cols[name] = cols[name] || new Map();
return [...cols[name].entries()].map(([baseId, doc]) => ({ baseId, doc: clone(doc) }));
},
async create(name, record) {
cols[name] = cols[name] || new Map();
const baseId = 'b' + ++seq;
cols[name].set(baseId, clone(record.data));
return { baseId, doc: clone(record.data) };
},
async update(name, baseId, record) {
cols[name].set(baseId, clone(record.data));
return { baseId, doc: clone(record.data) };
},
async delete(name, baseId) {
cols[name].delete(baseId);
},
};
}
const convoSchema = new Schema(
{
conversationId: { type: String, unique: true, required: true, index: true },
title: { type: String, default: 'New Chat', meiliIndex: true },
user: { type: String, index: true },
tags: { type: [String], default: [] },
expiredAt: { type: Date },
},
{ timestamps: true },
);
const messageSchema = new Schema(
{
messageId: { type: String, unique: true, required: true, index: true },
conversationId: { type: String, index: true, required: true },
user: { type: String, index: true, required: true, default: null },
text: { type: String, meiliIndex: true },
},
{ timestamps: true },
);
const userSchema = new Schema(
{
email: { type: String, required: true, unique: true },
role: { type: String, default: 'USER' },
password: { type: String, select: false },
totpSecret: { type: String, select: false },
},
{ timestamps: true },
);
const balanceSchema = new Schema({
user: { type: Schema.Types.ObjectId, ref: 'User', index: true, required: true },
tokenCredits: { type: Number, default: 0 },
expiresAt: { type: Date, default: null },
lastRefill: { type: Date, default: Date.now },
});
const sessionSchema = new Schema({
refreshTokenHash: { type: String, index: true },
user: { type: Schema.Types.ObjectId, ref: 'User', index: true },
expiration: { type: Date, required: true, index: true },
});
let store, Conversation, Message, User, Balance, Session;
beforeEach(async () => {
store = memStore();
Conversation = new BaseModel('Conversation', convoSchema, store);
Message = new BaseModel('Message', messageSchema, store);
User = new BaseModel('User', userSchema, store);
Balance = new BaseModel('Balance', balanceSchema, store);
Session = new BaseModel('Session', sessionSchema, store);
for (const m of [Conversation, Message, User, Balance, Session]) await m.ensure();
});
describe('schema introspection', () => {
test('promotes indexed/unique/timestamp fields and _id', () => {
expect(Conversation._promotedNames.sort()).toEqual(
['_id', 'conversationId', 'createdAt', 'updatedAt', 'user'].sort(),
);
});
test('timestamps detected only when configured', () => {
expect(Conversation._timestamps.createdAt).toBe('createdAt');
expect(Balance._timestamps).toBeNull();
});
});
describe('create + defaults + ids', () => {
test('User.create applies defaults, ObjectId _id, Date timestamps', async () => {
const u = await User.create({ email: 'z@zoo.ngo', password: 'hashed' });
expect(u._id).toMatch(/^[0-9a-f]{24}$/);
expect(u.role).toBe('USER');
expect(u.createdAt).toBeInstanceOf(Date);
});
});
describe('login-path reads', () => {
test('findOne + select projection + lean', async () => {
await User.create({ email: 'z@zoo.ngo', password: 'hashed' });
const found = await User.findOne({ email: 'z@zoo.ngo' }).select('email password').lean();
expect(found.password).toBe('hashed');
expect(found.role).toBeUndefined();
expect(found._id).toBeTruthy();
});
test('select:false secrets (password/totpSecret) hidden by default', async () => {
await User.create({ email: 'z@zoo.ngo', password: 'hashed', totpSecret: 'seed' });
const def = await User.findOne({ email: 'z@zoo.ngo' }).lean();
expect(def.password).toBeUndefined();
expect(def.totpSecret).toBeUndefined();
expect(def.email).toBe('z@zoo.ngo');
// login path explicitly requests +password
const withPw = await User.findOne({ email: 'z@zoo.ngo' }, '+password').lean();
expect(withPw.password).toBe('hashed');
expect(withPw.totpSecret).toBeUndefined(); // still hidden
});
});
describe('date-equality reads (H1 — no broken date pushdown)', () => {
test('find by an exact Date value returns the record', async () => {
const exp = new Date('2026-07-04T12:34:56.789Z');
await Conversation.findOneAndUpdate(
{ conversationId: 'd1', user: '507f1f77bcf86cd799439011' },
{ $set: { expiredAt: exp } },
{ upsert: true, new: true },
);
const byDate = await Conversation.find({ expiredAt: exp }).lean();
expect(byDate.map((c) => c.conversationId)).toEqual(['d1']);
const gt = await Conversation.find({ expiredAt: { $gt: new Date('2026-07-01T00:00:00Z') } }).lean();
expect(gt.some((c) => c.conversationId === 'd1')).toBe(true);
});
});
describe('save() preserves select:false fields after a projected load (M2)', () => {
test('mutate a default-loaded user + save keeps password/totpSecret', async () => {
await User.create({ email: 'z@zoo.ngo', password: 'hashed', totpSecret: 'seed' });
const u = await User.findOne({ email: 'z@zoo.ngo' }); // default view: secrets hidden
expect(u.password).toBeUndefined();
u.name = 'Renamed';
await u.save();
const reloaded = await User.findOne({ email: 'z@zoo.ngo' }, '+password').lean();
expect(reloaded.password).toBe('hashed'); // not erased by save()
expect(reloaded.name).toBe('Renamed');
});
});
describe('date hydration (non-timestamp Date fields)', () => {
test('session.expiration and balance dates hydrate to Date', async () => {
const exp = new Date(Date.now() + 3600_000);
await Session.create({ user: '507f1f77bcf86cd799439011', expiration: exp });
const s = await Session.findOne({ user: '507f1f77bcf86cd799439011' }).lean();
expect(s.expiration).toBeInstanceOf(Date);
expect(typeof s.expiration.getTime()).toBe('number'); // would throw on a string
const bal = await Balance.findOneAndUpdate(
{ user: '507f1f77bcf86cd799439011' },
{ $inc: { tokenCredits: 1 } },
{ upsert: true, new: true },
).lean();
expect(bal.lastRefill).toBeInstanceOf(Date);
});
});
describe('balance upsert', () => {
test('$inc + $set upsert then accumulate', async () => {
const uid = '507f1f77bcf86cd799439011';
const bal = await Balance.findOneAndUpdate(
{ user: uid },
{ $inc: { tokenCredits: 1000 } },
{ upsert: true, new: true },
).lean();
expect(bal.tokenCredits).toBe(1000);
expect(String(bal.user)).toBe(uid);
const bal2 = await Balance.findOneAndUpdate(
{ user: uid },
{ $inc: { tokenCredits: 500 } },
{ new: true },
).lean();
expect(bal2.tokenCredits).toBe(1500);
});
});
describe('chat hot path', () => {
const uid = '507f1f77bcf86cd799439011';
test('saveConvo upsert + toObject', async () => {
const c = await Conversation.findOneAndUpdate(
{ conversationId: 'c1', user: uid },
{ $set: { title: 'Hello', endpoint: 'openAI' } },
{ new: true, upsert: true },
);
expect(c.conversationId).toBe('c1');
expect(c.toObject().title).toBe('Hello');
});
test('saveMessage upsert + partial merge, sorted read', async () => {
const base = Date.now();
for (let i = 0; i < 3; i++) {
await Message.findOneAndUpdate(
{ messageId: 'm' + i, user: uid },
{ conversationId: 'c1', text: 'msg' + i, createdAt: new Date(base + i * 1000) },
{ upsert: true, new: true },
);
}
await Message.findOneAndUpdate({ messageId: 'm1', user: uid }, { text: 'edited' }, { new: true });
const msgs = await Message.find({ conversationId: 'c1', user: uid }).sort({ createdAt: 1 }).lean();
expect(msgs.map((m) => m.messageId)).toEqual(['m0', 'm1', 'm2']);
expect(msgs[1].text).toBe('edited');
expect(msgs[1].conversationId).toBe('c1'); // merge preserved
});
test('find(A).deleteMany(B) merges conditions', async () => {
const base = Date.now();
for (let i = 0; i < 3; i++) {
await Message.create({ messageId: 'm' + i, conversationId: 'c1', user: uid, createdAt: new Date(base + i * 1000) });
}
const res = await Message.find({ conversationId: 'c1', user: uid }).deleteMany({
createdAt: { $gt: new Date(base + 500) },
});
expect(res.deletedCount).toBe(2);
const left = await Message.find({ conversationId: 'c1' }).lean();
expect(left).toHaveLength(1);
expect(left[0].messageId).toBe('m0');
});
test('deleteMany $or/$exists', async () => {
await Message.create({ messageId: 'mx', conversationId: '', user: uid });
const res = await Message.deleteMany({
$or: [{ conversationId: '' }, { conversationId: { $exists: false } }],
});
expect(res.deletedCount).toBe(1);
});
test('bulkWrite updateOne upsert', async () => {
const res = await Conversation.bulkWrite([
{ updateOne: { filter: { conversationId: 'c1', user: uid }, update: { title: 'A' }, upsert: true } },
{ updateOne: { filter: { conversationId: 'c2', user: uid }, update: { title: 'B' }, upsert: true } },
]);
expect(res.upsertedCount).toBe(2);
expect(await Conversation.countDocuments({ conversationId: { $in: ['c1', 'c2'] } })).toBe(2);
});
test('meiliSearch (Base/SQLite FTS) — title + text, user-scoped', async () => {
const other = '507f1f77bcf86cd799439099';
await Conversation.findOneAndUpdate({ conversationId: 'c1', user: uid }, { $set: { title: 'Quantum physics notes' } }, { upsert: true, new: true });
await Conversation.findOneAndUpdate({ conversationId: 'c2', user: uid }, { $set: { title: 'Grocery list' } }, { upsert: true, new: true });
await Conversation.findOneAndUpdate({ conversationId: 'c3', user: other }, { $set: { title: 'Quantum leaps' } }, { upsert: true, new: true });
const conv = await Conversation.meiliSearch('quantum', { filter: `user = "${uid}"` });
expect(conv.hits.map((h) => h.conversationId)).toEqual(['c1']); // case-insensitive, user-scoped
await Message.create({ messageId: 'm1', conversationId: 'c1', user: uid, text: 'the WAVEFUNCTION collapses' });
await Message.create({ messageId: 'm2', conversationId: 'c1', user: uid, text: 'unrelated chatter' });
const msg = await Message.meiliSearch('wavefunction', { filter: `user = "${uid}"` });
expect(msg.hits.map((h) => h.messageId)).toEqual(['m1']);
expect((await Conversation.meiliSearch('', { filter: `user = "${uid}"` })).hits).toEqual([]);
});
test('findOneAndUpdate new:false returns pre-update doc', async () => {
await Conversation.findOneAndUpdate({ conversationId: 'c1', user: uid }, { title: 'first' }, { upsert: true, new: true });
const before = await Conversation.findOneAndUpdate(
{ conversationId: 'c1', user: uid },
{ title: 'second' },
{ new: false },
);
expect(before.title).toBe('first');
const now = await Conversation.findOne({ conversationId: 'c1' }).lean();
expect(now.title).toBe('second');
});
});
+122
View File
@@ -0,0 +1,122 @@
'use strict';
/**
* A self-contained, `mongoose`-shaped facade backed by Hanzo Base.
*
* `@librechat/data-schemas` builds its schemas with real `mongoose` (a pure
* schema DSL inside that package) and then asks a mongoose instance to turn
* those schemas into models (`createModels`/`createMethods`). We hand it THIS
* facade instead: it registers every model as a BaseModel persisting to Base —
* never MongoDB — and provides just the mongoose surface the data layer uses at
* runtime (`model`, `models`, `Types.ObjectId`, `Schema.Types`, a no-op session
* / connection). It carries **no `mongoose` runtime dependency**; schema objects
* are passed in already-built, so this module never imports the driver.
*
* Result: one adapter, and the entire data-schemas model+method surface
* (User, Session, Token, Role, Balance, Conversation, Message, Agent, …) runs
* on Base with zero per-model porting.
*/
const { BaseModel, makeModelCtor } = require('./model');
const { ObjectId, isValidObjectId } = require('./objectId');
const store = require('./store');
/** Shared model registry (mirrors `mongoose.models`) — values are model ctors. */
const models = {};
/** BaseModel instances keyed by name (for connectDb collection provisioning). */
const baseModels = {};
function model(name, schema) {
if (models[name]) {
return models[name];
}
if (!schema) {
return undefined;
}
const base = new BaseModel(name, schema, store);
const ctor = makeModelCtor(base);
baseModels[name] = base;
models[name] = ctor;
return ctor;
}
/** No-op session: Base has no multi-document transactions (see adapter notes). */
function makeSession() {
return {
startTransaction() {},
async commitTransaction() {},
async abortTransaction() {},
async endSession() {},
async withTransaction(fn) {
return fn();
},
inTransaction() {
return false;
},
};
}
/** Connection stub — the real Base connection lives in ./index.js connectDb(). */
const connection = {
readyState: 1,
_readyState: 1,
on() {},
once() {},
model,
models,
collections: {},
async dropDatabase() {},
async close() {},
db: {
async dropDatabase() {},
collection() {
return {};
},
},
};
/** Minimal Schema.Types surface for the few `mongoose.Schema.Types.*` runtime uses. */
class Mixed {}
const SchemaTypes = {
ObjectId,
Mixed,
String,
Number,
Boolean,
Date,
Array,
Buffer,
Map,
};
const facade = {
__isBaseFacade: true,
model,
models,
connection,
connections: [connection],
Types: { ObjectId, Decimal128: Number, Mixed },
Schema: { Types: SchemaTypes },
Model: BaseModel,
isValidObjectId,
async connect() {
return facade;
},
createConnection() {
return connection;
},
async disconnect() {},
startSession() {
return makeSession();
},
set() {
return facade;
},
get() {
return undefined;
},
/** Access the underlying BaseModel instances (used by connectDb). */
baseModels,
};
module.exports = facade;
+93
View File
@@ -0,0 +1,93 @@
'use strict';
/**
* Mongo-compatible ObjectId generation.
*
* Hanzo Base stores documents keyed by its own 15-char id, but LibreChat code
* treats `_id` as a 24-hex Mongo ObjectId string (equality, `.toString()`,
* refs). We generate real ObjectId-shaped values so that behaviour is
* preserved without a live MongoDB.
*
* Layout (12 bytes -> 24 hex): 4-byte timestamp | 5-byte process random | 3-byte counter.
*/
const crypto = require('crypto');
const PROCESS_RANDOM = crypto.randomBytes(5);
let COUNTER = crypto.randomBytes(3).readUIntBE(0, 3);
/** @returns {string} 24-char lowercase hex ObjectId string. */
function generateObjectId() {
const buf = Buffer.allocUnsafe(12);
buf.writeUInt32BE(Math.floor(Date.now() / 1000), 0);
PROCESS_RANDOM.copy(buf, 4);
COUNTER = (COUNTER + 1) % 0xffffff;
buf.writeUIntBE(COUNTER, 9, 3);
return buf.toString('hex');
}
const HEX_24 = /^[0-9a-fA-F]{24}$/;
/** @param {unknown} value @returns {boolean} */
function isValidObjectId(value) {
if (value == null) {
return false;
}
if (typeof value === 'string') {
return HEX_24.test(value);
}
if (typeof value === 'object' && typeof value.toString === 'function') {
return HEX_24.test(value.toString());
}
return false;
}
/**
* A minimal, dependency-free ObjectId compatible with the surface the chat code
* uses from `mongoose.Types.ObjectId`: `new ObjectId()`, `new ObjectId(hex)`,
* `.isValid()`, `.createFromHexString()`, and instances that stringify / JSON /
* compare as their 24-hex value. Lets us drop the `mongoose` runtime dependency.
*/
class ObjectId {
constructor(id) {
if (id == null) {
this._id = generateObjectId();
} else if (typeof id === 'string') {
this._id = id;
} else if (id instanceof ObjectId) {
this._id = id._id;
} else if (typeof id.toHexString === 'function') {
this._id = id.toHexString();
} else {
this._id = String(id);
}
}
toString() {
return this._id;
}
toHexString() {
return this._id;
}
toJSON() {
return this._id;
}
valueOf() {
return this._id;
}
equals(other) {
if (other == null) {
return false;
}
return String(typeof other.toHexString === 'function' ? other.toHexString() : other) === this._id;
}
static isValid(value) {
return isValidObjectId(value);
}
static createFromHexString(hex) {
return new ObjectId(hex);
}
}
module.exports = { generateObjectId, isValidObjectId, HEX_24, ObjectId };
+537
View File
@@ -0,0 +1,537 @@
'use strict';
/**
* A small, correct MongoDB query/update engine used by the Hanzo Base adapter.
*
* The Base document store keeps each document as a JSON blob; filtering,
* update-operator application, projection and sorting all run here in JS.
* This is the correctness backstop for the whole adapter: any Mongo filter the
* LibreChat data layer produces is evaluated here, so translation to Base's
* filter DSL (see store.js) is only ever a best-effort pushdown optimisation.
*/
/**
* Path segments that must never be traversed or written — guards against
* prototype pollution via attacker-controlled dotted keys (e.g. an update or
* imported document containing `__proto__.x`). Mongoose sanitized these; the
* adapter must too.
*/
const FORBIDDEN_SEGMENTS = new Set(['__proto__', 'constructor', 'prototype']);
/** Resolve a possibly-dotted path against a document. */
function getPath(doc, path) {
if (doc == null) {
return undefined;
}
if (!path.includes('.')) {
return FORBIDDEN_SEGMENTS.has(path) ? undefined : doc[path];
}
let cur = doc;
for (const part of path.split('.')) {
if (cur == null || FORBIDDEN_SEGMENTS.has(part)) {
return undefined;
}
cur = cur[part];
}
return cur;
}
/** Set a possibly-dotted path on a document (mutates). No-op on forbidden segments. */
function setPath(doc, path, value) {
if (!path.includes('.')) {
if (!FORBIDDEN_SEGMENTS.has(path)) {
doc[path] = value;
}
return;
}
const parts = path.split('.');
if (parts.some((p) => FORBIDDEN_SEGMENTS.has(p))) {
return;
}
let cur = doc;
for (let i = 0; i < parts.length - 1; i++) {
if (cur[parts[i]] == null || typeof cur[parts[i]] !== 'object') {
cur[parts[i]] = {};
}
cur = cur[parts[i]];
}
cur[parts[parts.length - 1]] = value;
}
/** Delete a possibly-dotted path (mutates). No-op on forbidden segments. */
function unsetPath(doc, path) {
if (!path.includes('.')) {
if (!FORBIDDEN_SEGMENTS.has(path)) {
delete doc[path];
}
return;
}
const parts = path.split('.');
if (parts.some((p) => FORBIDDEN_SEGMENTS.has(p))) {
return;
}
let cur = doc;
for (let i = 0; i < parts.length - 1; i++) {
if (cur[parts[i]] == null) {
return;
}
cur = cur[parts[i]];
}
delete cur[parts[parts.length - 1]];
}
/** Coerce ObjectId-like objects (Mongo ObjectId, wrappers) to their hex string. */
function idString(value) {
if (value && typeof value === 'object' && typeof value.toHexString === 'function') {
return value.toHexString();
}
return value;
}
/** Coerce a value to a comparable primitive (Date -> ms, ObjectId -> hex). */
function comparable(value) {
value = idString(value);
if (value instanceof Date) {
return value.getTime();
}
if (typeof value === 'string') {
// ISO date strings sort correctly lexicographically, but to compare against
// Date values we normalise anything that parses as a date to ms.
const t = Date.parse(value);
if (!Number.isNaN(t) && /\d{4}-\d{2}-\d{2}T/.test(value)) {
return t;
}
}
return value;
}
/** Deep-ish equality sufficient for query matching (primitives, dates, arrays, plain objects). */
function valueEquals(a, b) {
a = idString(a);
b = idString(b);
if (a === b) {
return true;
}
if (a instanceof Date || b instanceof Date) {
return comparable(a) === comparable(b);
}
if (a == null || b == null) {
return a == null && b == null;
}
if (Array.isArray(a) && Array.isArray(b)) {
return a.length === b.length && a.every((x, i) => valueEquals(x, b[i]));
}
return false;
}
function compare(a, b) {
const ca = comparable(a);
const cb = comparable(b);
if (ca == null && cb == null) {
return 0;
}
if (ca == null) {
return -1;
}
if (cb == null) {
return 1;
}
if (ca < cb) {
return -1;
}
if (ca > cb) {
return 1;
}
return 0;
}
function toRegExp(spec, options) {
if (spec instanceof RegExp) {
return spec;
}
return new RegExp(spec, options || '');
}
/** Evaluate a single field's operator expression against a document value. */
function matchOperators(fieldValue, expr) {
// Array-aware equality helper: a field matches a scalar if it equals it, or
// (when the field is an array) if the array contains it — Mongo semantics.
const eq = (target) => {
if (Array.isArray(fieldValue) && !Array.isArray(target)) {
return fieldValue.some((v) => valueEquals(v, target));
}
return valueEquals(fieldValue, target);
};
if (expr instanceof RegExp) {
return typeof fieldValue === 'string' && expr.test(fieldValue);
}
if (expr === null || typeof expr !== 'object' || expr instanceof Date || Array.isArray(expr)) {
return eq(expr);
}
const keys = Object.keys(expr);
const isOperatorExpr = keys.some((k) => k.startsWith('$'));
if (!isOperatorExpr) {
return eq(expr);
}
for (const op of keys) {
const operand = expr[op];
switch (op) {
case '$eq':
if (!eq(operand)) return false;
break;
case '$ne':
if (eq(operand)) return false;
break;
case '$in': {
const arr = operand || [];
const hit = Array.isArray(fieldValue)
? fieldValue.some((v) => arr.some((o) => valueEquals(v, o)))
: arr.some((o) => valueEquals(fieldValue, o));
if (!hit) return false;
break;
}
case '$nin': {
const arr = operand || [];
const hit = Array.isArray(fieldValue)
? fieldValue.some((v) => arr.some((o) => valueEquals(v, o)))
: arr.some((o) => valueEquals(fieldValue, o));
if (hit) return false;
break;
}
case '$gt':
if (fieldValue === undefined || compare(fieldValue, operand) <= 0) return false;
break;
case '$gte':
if (fieldValue === undefined || compare(fieldValue, operand) < 0) return false;
break;
case '$lt':
if (fieldValue === undefined || compare(fieldValue, operand) >= 0) return false;
break;
case '$lte':
if (fieldValue === undefined || compare(fieldValue, operand) > 0) return false;
break;
case '$exists':
if ((fieldValue !== undefined) !== !!operand) return false;
break;
case '$regex': {
const re = toRegExp(operand, expr.$options);
if (typeof fieldValue !== 'string' || !re.test(fieldValue)) return false;
break;
}
case '$options':
break; // handled with $regex
case '$not':
if (matchOperators(fieldValue, operand)) return false;
break;
case '$size':
if (!Array.isArray(fieldValue) || fieldValue.length !== operand) return false;
break;
case '$all': {
if (!Array.isArray(fieldValue)) return false;
const ok = (operand || []).every((o) => fieldValue.some((v) => valueEquals(v, o)));
if (!ok) return false;
break;
}
case '$elemMatch': {
if (!Array.isArray(fieldValue)) return false;
if (!fieldValue.some((v) => matches(v, operand))) return false;
break;
}
default:
// Unknown operator: treat the whole expression as an equality target.
return eq(expr);
}
}
return true;
}
/**
* Does `doc` match Mongo filter `query`?
* @param {Record<string, unknown>} doc
* @param {Record<string, unknown>} query
* @returns {boolean}
*/
function matches(doc, query) {
if (!query || typeof query !== 'object') {
return true;
}
for (const key of Object.keys(query)) {
const val = query[key];
if (key === '$and') {
if (!(val || []).every((sub) => matches(doc, sub))) return false;
continue;
}
if (key === '$or') {
if (!(val || []).some((sub) => matches(doc, sub))) return false;
continue;
}
if (key === '$nor') {
if ((val || []).some((sub) => matches(doc, sub))) return false;
continue;
}
if (key === '$not') {
if (matches(doc, val)) return false;
continue;
}
if (key.startsWith('$')) {
continue; // unsupported top-level operator -> ignore (permissive)
}
if (!matchOperators(getPath(doc, key), val)) {
return false;
}
}
return true;
}
const UPDATE_OPERATORS = new Set([
'$set',
'$unset',
'$inc',
'$push',
'$addToSet',
'$pull',
'$setOnInsert',
'$min',
'$max',
'$mul',
'$rename',
]);
/** Split a Mongoose-style update into operator groups, wrapping bare fields as $set. */
function normalizeUpdate(update) {
if (!update || typeof update !== 'object') {
return { $set: {} };
}
const keys = Object.keys(update);
const hasOperator = keys.some((k) => UPDATE_OPERATORS.has(k));
if (!hasOperator) {
return { $set: { ...update } };
}
const out = {};
const bareSet = {};
for (const k of keys) {
if (UPDATE_OPERATORS.has(k)) {
out[k] = update[k];
} else {
bareSet[k] = update[k];
}
}
if (Object.keys(bareSet).length) {
out.$set = { ...bareSet, ...(out.$set || {}) };
}
return out;
}
/**
* Apply a normalized/raw update to a document in place and return it.
* @param {Record<string, unknown>} doc - target document (mutated)
* @param {Record<string, unknown>} update
* @param {{ isInsert?: boolean }} [opts]
*/
function applyUpdate(doc, update, opts = {}) {
const ops = normalizeUpdate(update);
if (ops.$set) {
for (const [k, v] of Object.entries(ops.$set)) {
setPath(doc, k, v);
}
}
if (ops.$setOnInsert && opts.isInsert) {
for (const [k, v] of Object.entries(ops.$setOnInsert)) {
setPath(doc, k, v);
}
}
if (ops.$unset) {
for (const k of Object.keys(ops.$unset)) {
unsetPath(doc, k);
}
}
if (ops.$inc) {
for (const [k, v] of Object.entries(ops.$inc)) {
setPath(doc, k, (Number(getPath(doc, k)) || 0) + Number(v));
}
}
if (ops.$mul) {
for (const [k, v] of Object.entries(ops.$mul)) {
setPath(doc, k, (Number(getPath(doc, k)) || 0) * Number(v));
}
}
if (ops.$min) {
for (const [k, v] of Object.entries(ops.$min)) {
const cur = getPath(doc, k);
if (cur === undefined || compare(v, cur) < 0) setPath(doc, k, v);
}
}
if (ops.$max) {
for (const [k, v] of Object.entries(ops.$max)) {
const cur = getPath(doc, k);
if (cur === undefined || compare(v, cur) > 0) setPath(doc, k, v);
}
}
if (ops.$push) {
for (const [k, v] of Object.entries(ops.$push)) {
const arr = Array.isArray(getPath(doc, k)) ? getPath(doc, k) : [];
if (v && typeof v === 'object' && Array.isArray(v.$each)) {
arr.push(...v.$each);
} else {
arr.push(v);
}
setPath(doc, k, arr);
}
}
if (ops.$addToSet) {
for (const [k, v] of Object.entries(ops.$addToSet)) {
const arr = Array.isArray(getPath(doc, k)) ? getPath(doc, k) : [];
const items = v && typeof v === 'object' && Array.isArray(v.$each) ? v.$each : [v];
for (const item of items) {
if (!arr.some((x) => valueEquals(x, item))) arr.push(item);
}
setPath(doc, k, arr);
}
}
if (ops.$pull) {
for (const [k, cond] of Object.entries(ops.$pull)) {
const arr = getPath(doc, k);
if (Array.isArray(arr)) {
setPath(
doc,
k,
arr.filter((x) =>
cond && typeof cond === 'object' && !Array.isArray(cond)
? !matchOperators(x, cond)
: !valueEquals(x, cond),
),
);
}
}
}
if (ops.$rename) {
for (const [from, to] of Object.entries(ops.$rename)) {
const v = getPath(doc, from);
unsetPath(doc, from);
setPath(doc, to, v);
}
}
return doc;
}
/**
* Parse a projection spec (space string or object) into
* { plain:Set, plus:Set, exclude:Set } or null for no projection.
* - `field` -> plain inclusion
* - `+field` -> force-include a `select:false` field on top of the default set
* - `-field` -> exclusion
*/
function parseProjection(spec) {
if (!spec) {
return null;
}
const plain = new Set();
const plus = new Set();
const exclude = new Set();
if (typeof spec === 'string') {
for (const token of spec.split(/\s+/).filter(Boolean)) {
if (token.startsWith('-')) {
exclude.add(token.slice(1));
} else if (token.startsWith('+')) {
plus.add(token.slice(1));
} else {
plain.add(token);
}
}
} else if (typeof spec === 'object') {
for (const [k, v] of Object.entries(spec)) {
if (v) {
plain.add(k);
} else {
exclude.add(k);
}
}
}
if (!plain.size && !plus.size && !exclude.size) {
return null;
}
return { plain, plus, exclude };
}
/**
* Apply a parsed projection with Mongoose `select:false` semantics, returning a
* new object. `deselected` are fields excluded from reads by default (secrets).
*/
function applyProjection(doc, projection, deselected) {
const hidden = deselected || EMPTY_SET;
// No projection: default view = everything minus deselected (secrets).
if (!projection) {
if (!hidden.size) {
return doc;
}
const out = { ...doc };
for (const f of hidden) {
unsetPath(out, f);
}
return out;
}
// Inclusion mode: return only the named fields (+plus), _id unless -_id.
if (projection.plain.size) {
const fields = new Set([...projection.plain, ...projection.plus]);
if (!projection.exclude.has('_id')) {
fields.add('_id');
}
const out = {};
for (const f of fields) {
const v = getPath(doc, f);
if (v !== undefined) {
setPath(out, f, v);
}
}
return out;
}
// Default + adjustments: base minus deselected (unless +included) minus -excluded.
const out = { ...doc };
for (const f of hidden) {
if (!projection.plus.has(f)) {
unsetPath(out, f);
}
}
for (const f of projection.exclude) {
unsetPath(out, f);
}
return out;
}
const EMPTY_SET = new Set();
/** Sort documents by a Mongo sort spec ({ field: 1|-1 }). Returns a new array. */
function sortDocs(docs, sortSpec) {
if (!sortSpec || !Object.keys(sortSpec).length) {
return docs;
}
const keys = Object.entries(sortSpec).map(([k, v]) => [k, v === -1 || v === 'desc' ? -1 : 1]);
return [...docs].sort((a, b) => {
for (const [k, dir] of keys) {
const c = compare(getPath(a, k), getPath(b, k));
if (c !== 0) {
return c * dir;
}
}
return 0;
});
}
module.exports = {
matches,
applyUpdate,
normalizeUpdate,
parseProjection,
applyProjection,
sortDocs,
getPath,
setPath,
unsetPath,
valueEquals,
compare,
idString,
};
+124
View File
@@ -0,0 +1,124 @@
'use strict';
const { matches, applyUpdate, parseProjection, applyProjection, sortDocs } = require('./query');
describe('base/query matcher', () => {
const doc = {
_id: 'a1',
user: 'u1',
conversationId: 'c1',
tags: ['x', 'y'],
tokenCount: 10,
createdAt: '2026-07-04T00:00:00.000Z',
};
test('equality + implicit array-contains', () => {
expect(matches(doc, { user: 'u1' })).toBe(true);
expect(matches(doc, { user: 'u2' })).toBe(false);
expect(matches(doc, { tags: 'x' })).toBe(true); // array contains
expect(matches(doc, { tags: 'z' })).toBe(false);
});
test('operators $in/$nin/$ne/$exists', () => {
expect(matches(doc, { conversationId: { $in: ['c1', 'c2'] } })).toBe(true);
expect(matches(doc, { conversationId: { $nin: ['c2'] } })).toBe(true);
expect(matches(doc, { user: { $ne: 'u2' } })).toBe(true);
expect(matches(doc, { missing: { $exists: false } })).toBe(true);
expect(matches(doc, { user: { $exists: true } })).toBe(true);
});
test('comparison operators with numbers and dates', () => {
expect(matches(doc, { tokenCount: { $gt: 5, $lte: 10 } })).toBe(true);
expect(matches(doc, { tokenCount: { $gte: 11 } })).toBe(false);
expect(matches(doc, { createdAt: { $gt: new Date('2026-07-03T00:00:00Z') } })).toBe(true);
expect(matches(doc, { createdAt: { $lt: new Date('2026-07-03T00:00:00Z') } })).toBe(false);
});
test('logical $or / $and / $nor', () => {
expect(matches(doc, { $or: [{ user: 'nope' }, { conversationId: 'c1' }] })).toBe(true);
expect(matches(doc, { $and: [{ user: 'u1' }, { conversationId: 'c1' }] })).toBe(true);
expect(matches(doc, { $nor: [{ user: 'u1' }] })).toBe(false);
});
test('$exists false / null equivalence for absent fields', () => {
expect(matches({ a: 1 }, { $or: [{ b: null }, { b: { $exists: false } }] })).toBe(true);
});
});
describe('base/query applyUpdate', () => {
test('$set / implicit set / $unset', () => {
expect(applyUpdate({ a: 1 }, { $set: { b: 2 } })).toEqual({ a: 1, b: 2 });
expect(applyUpdate({ a: 1 }, { b: 2 })).toEqual({ a: 1, b: 2 }); // implicit $set (merge)
expect(applyUpdate({ a: 1, b: 2 }, { $unset: { b: 1 } })).toEqual({ a: 1 });
});
test('$inc accumulates', () => {
expect(applyUpdate({ n: 5 }, { $inc: { n: 3 } })).toEqual({ n: 8 });
expect(applyUpdate({}, { $inc: { n: 2 } })).toEqual({ n: 2 });
});
test('$push / $addToSet / $pull', () => {
expect(applyUpdate({ a: [1] }, { $push: { a: 2 } })).toEqual({ a: [1, 2] });
expect(applyUpdate({ a: [1] }, { $addToSet: { a: 1 } })).toEqual({ a: [1] });
expect(applyUpdate({ a: [1, 2, 3] }, { $pull: { a: 2 } })).toEqual({ a: [1, 3] });
});
test('$setOnInsert only applies on insert', () => {
expect(applyUpdate({}, { $setOnInsert: { a: 1 } }, { isInsert: true })).toEqual({ a: 1 });
expect(applyUpdate({}, { $setOnInsert: { a: 1 } }, { isInsert: false })).toEqual({});
});
test('prototype pollution is blocked (dotted + bare keys, every operator)', () => {
applyUpdate({}, { $set: { '__proto__.polluted': 'x' } });
applyUpdate({}, { '__proto__.polluted2': 'y' }); // implicit $set
applyUpdate({}, { $inc: { '__proto__.count': 5 } });
applyUpdate({}, { $set: { 'constructor.prototype.polluted3': 'z' } });
applyUpdate({}, { $set: { __proto__: { polluted4: 'w' } } });
applyUpdate({}, { $rename: { a: '__proto__.x' } });
expect({}.polluted).toBeUndefined();
expect({}.polluted2).toBeUndefined();
expect({}.count).toBeUndefined();
expect({}.polluted3).toBeUndefined();
expect({}.polluted4).toBeUndefined();
expect({}.x).toBeUndefined();
expect(Object.prototype.polluted).toBeUndefined();
});
});
describe('base/query projection + sort', () => {
const d = { _id: '1', a: 1, b: 2, c: 3, password: 'secret' };
const hidden = new Set(['password']);
test('inclusion projection keeps _id', () => {
expect(applyProjection(d, parseProjection('a b'), hidden)).toEqual({ _id: '1', a: 1, b: 2 });
});
test('exclusion projection', () => {
expect(applyProjection(d, parseProjection('-b -c -password'), hidden)).toEqual({ _id: '1', a: 1 });
});
test('select:false field hidden by default', () => {
expect(applyProjection(d, null, hidden)).toEqual({ _id: '1', a: 1, b: 2, c: 3 });
});
test('+field re-includes a select:false field on top of default', () => {
expect(applyProjection(d, parseProjection('+password'), hidden)).toEqual({
_id: '1',
a: 1,
b: 2,
c: 3,
password: 'secret',
});
});
test('explicit inclusion can return a select:false field', () => {
expect(applyProjection(d, parseProjection('a password'), hidden)).toEqual({
_id: '1',
a: 1,
password: 'secret',
});
});
test('sort by multiple keys / directions', () => {
const rows = [
{ k: 2, t: 'b' },
{ k: 1, t: 'a' },
{ k: 2, t: 'a' },
];
expect(sortDocs(rows, { k: 1, t: 1 }).map((r) => `${r.k}${r.t}`)).toEqual(['1a', '2a', '2b']);
expect(sortDocs(rows, { k: -1 }).map((r) => r.k)).toEqual([2, 2, 1]);
});
});
+167
View File
@@ -0,0 +1,167 @@
'use strict';
/**
* Introspects a real Mongoose Schema (built by @librechat/data-schemas as a
* pure schema DSL — never connected to Mongo) and distills exactly what the
* Base adapter needs:
* - default values to apply on insert
* - whether timestamps (createdAt/updatedAt) are managed
* - which primitive fields to promote to real Base columns for filter/sort
*
* Everything else about the Mongoose schema (validators, casting, refs) is
* intentionally ignored: the adapter trusts the application layer and stores
* the full document as JSON.
*/
const PRIMITIVE_INSTANCES = new Set(['String', 'Number', 'Date', 'Boolean', 'ObjectID', 'ObjectId']);
/** Identifier / scoping fields that are `meiliIndex` but not useful free-text search targets. */
const SEARCH_STOPLIST = new Set([
'_id',
'_meiliIndex',
'conversationId',
'messageId',
'user',
'organization',
'endpoint',
'model',
'sender',
]);
/** Map a Mongoose SchemaType instance to a Base column type. */
function baseColumnType(instance) {
switch (instance) {
case 'Number':
return 'number';
case 'Boolean':
return 'bool';
case 'Date':
return 'date';
default:
return 'text';
}
}
/**
* @param {import('mongoose').Schema} schema
* @returns {{
* defaults: Array<[string, unknown]>,
* timestamps: { createdAt: string, updatedAt: string } | null,
* promoted: Array<{ name: string, type: string, unique: boolean }>,
* deselected: Set<string>,
* dateFields: Set<string>,
* }}
*/
function describeSchema(schema) {
const defaults = [];
const promoted = new Map();
const deselected = new Set();
const dateFields = new Set();
const searchable = new Set();
// Always promote _id so document lookup by id is a real, unique Base column.
promoted.set('_id', { name: '_id', type: 'text', unique: true });
const paths = schema.paths || {};
for (const name of Object.keys(paths)) {
if (name === '_id' || name === '__v') {
continue;
}
const type = paths[name];
const options = type.options || {};
// Fields marked `select: false` are excluded from reads by default
// (Mongoose semantics) — critical for secrets: password, totpSecret,
// backupCodes, keyHash. Only returned when explicitly `+selected`.
if (options.select === false) {
deselected.add(name);
}
// All Date-typed paths must be re-hydrated to Date objects on read so
// callers can safely call `.getTime()` / date methods (e.g. session.expiration).
if (type.instance === 'Date') {
dateFields.add(name);
}
// Free-text search targets: fields flagged for MeiliSearch that are actual
// content (title, text, …), not identifiers. Used by Base/SQLite FTS.
if (options.meiliIndex && type.instance === 'String' && !SEARCH_STOPLIST.has(name)) {
searchable.add(name);
}
// Collect declared defaults for primitive top-level paths (no dots).
if (!name.includes('.') && Object.prototype.hasOwnProperty.call(options, 'default')) {
defaults.push([name, options.default]);
}
// Promote indexed / unique primitive fields to Base columns for filtering.
// A DB-level UNIQUE index is only safe for always-present unique keys
// (unique && required) — e.g. conversationId, messageId — to avoid the
// sparse-null conflicts of optional unique fields (googleId, openidId, …),
// whose logical uniqueness stays enforced by the adapter-level upsert.
if (!name.includes('.') && PRIMITIVE_INSTANCES.has(type.instance)) {
if ((options.index || options.unique) && !promoted.has(name)) {
const columnType = baseColumnType(type.instance);
promoted.set(name, {
name,
type: columnType,
// Full UNIQUE for always-present keys (conversationId, messageId, email).
unique: !!(options.unique && options.required),
// Partial UNIQUE (WHERE != '') for optional unique text keys
// (googleId, openidId, username, …): DB-level race safety without the
// sparse-null conflict.
sparseUnique: !!(options.unique && !options.required && columnType === 'text'),
});
}
}
}
// Single-field indexes declared via schema.index({ field: 1 }).
try {
for (const [spec] of schema.indexes()) {
const fields = Object.keys(spec || {});
if (fields.length === 1) {
const f = fields[0];
if (f !== '_id' && !f.includes('.') && paths[f] && PRIMITIVE_INSTANCES.has(paths[f].instance) && !promoted.has(f)) {
promoted.set(f, { name: f, type: baseColumnType(paths[f].instance), unique: false });
}
}
}
} catch {
/* schema.indexes() unavailable — ignore */
}
let timestamps = null;
const tsOpt = schema.options && schema.options.timestamps;
if (tsOpt) {
timestamps =
tsOpt === true
? { createdAt: 'createdAt', updatedAt: 'updatedAt' }
: {
createdAt: tsOpt.createdAt || 'createdAt',
updatedAt: tsOpt.updatedAt || 'updatedAt',
};
for (const f of [timestamps.createdAt, timestamps.updatedAt]) {
dateFields.add(f);
if (!promoted.has(f)) {
promoted.set(f, { name: f, type: 'date', unique: false });
}
}
}
return {
defaults,
timestamps,
promoted: [...promoted.values()],
deselected,
dateFields,
searchable,
};
}
/** Resolve a schema default (calling it if it is a function). */
function resolveDefault(def) {
return typeof def === 'function' ? def() : def;
}
module.exports = { describeSchema, resolveDefault, baseColumnType };
+44
View File
@@ -0,0 +1,44 @@
'use strict';
/*
* Boot smoke: drive the REAL wired boot entry points against Hanzo Base, the
* same way api/server/index.js does:
* require('~/db').connectDb() -> Base facade + collection provisioning
* require('~/models').seedDatabase() -> roles + default categories
*
* Proves the wired data layer boots on Base and that non-core models
* (Role, AgentCategory) come along through the same adapter.
*
* Run: HANZO_BASE_URL=... HANZO_BASE_TOKEN=... node api/db/base/__tests__/boot-smoke.js
*/
const path = require('path');
require('module-alias')({ base: path.resolve(__dirname, '../../..') }); // base = api/
const assert = require('assert');
async function main() {
const { connectDb } = require('~/db');
const { seedDatabase } = require('~/models');
const { Role } = require('~/db/models');
await connectDb();
console.log('[boot] connectDb() OK — Base connected + collections provisioned');
await seedDatabase();
console.log('[boot] seedDatabase() OK — roles + categories seeded');
const adminRole = await Role.findOne({ name: 'ADMIN' }).lean();
assert(adminRole && adminRole.name === 'ADMIN', 'ADMIN role seeded to Base');
console.log(' ok - ADMIN role present on Base:', adminRole.name);
const userRole = await Role.findOne({ name: 'USER' }).lean();
assert(userRole && userRole.name === 'USER', 'USER role seeded to Base');
console.log(' ok - USER role present on Base:', userRole.name);
console.log('\nBOOT SMOKE PASSED — wired data layer boots on Hanzo Base.');
process.exit(0);
}
main().catch((e) => {
console.error('\nBOOT SMOKE FAILED:', e && e.stack ? e.stack : e);
process.exit(1);
});
+141
View File
@@ -0,0 +1,141 @@
'use strict';
/*
* LIVE proof: the REAL @librechat/data-schemas model+method factories, driven
* through the Hanzo Base facade, persisting to a running Base instance.
*
* Exercises the login path (createUser + balance grant + findUser +
* bcrypt.compare) and the chat hot path (saveConvo + saveMessage shapes),
* then reads the documents straight back out of Base to prove they landed
* there — not in MongoDB.
*
* Run: HANZO_BASE_URL=... HANZO_BASE_TOKEN=... node api/db/base/__tests__/live.js
*/
const assert = require('assert');
const bcrypt = require('bcryptjs');
const { createModels, createMethods } = require('@librechat/data-schemas');
const { mongoose, connectDb, store } = require('..');
async function main() {
const models = createModels(mongoose);
const methods = createMethods(mongoose);
await connectDb();
console.log('[live] connected + collections provisioned');
const ok = (m) => console.log(' ok -', m);
const email = `z+${Date.now()}@zoo.ngo`;
const plain = 'IloveChat2026!!';
const password = await bcrypt.hash(plain, 10);
// ---- LOGIN PATH ----------------------------------------------------------
const userId = await methods.createUser(
{ email, password, name: 'Z', username: 'z', provider: 'local' },
{ enabled: true, startBalance: 100000 },
true,
false,
);
assert(userId, 'createUser returned an id');
ok('createUser (User + Balance grant)');
// Default read hides select:false secrets (password/totpSecret) — regression guard.
const safeUser = await methods.findUser({ email: email.toUpperCase() }); // email normalized
assert(safeUser && safeUser.email === email, 'findUser by (normalized) email');
assert.strictEqual(safeUser.password, undefined, 'password NOT returned by default (select:false)');
ok('findUser default view hides the password hash (select:false honored)');
// Real login path selects +password explicitly (see api/strategies/localStrategy.js).
const user = await methods.findUser({ email }, '+password');
const match = await bcrypt.compare(plain, user.password);
assert.strictEqual(match, true, 'bcrypt.compare succeeds — password stored hashed');
assert.notStrictEqual(user.password, plain, 'password is NOT plaintext');
ok('findUser(+password) + bcrypt verification (hashed, never plaintext)');
const balance = await models.Balance.findOne({ user: user._id }).lean();
assert.strictEqual(balance.tokenCredits, 100000, 'balance granted via $inc upsert');
ok('Balance record created on Base with $inc grant');
// ---- CHAT HOT PATH -------------------------------------------------------
const uid = String(user._id);
const convo = await models.Conversation.findOneAndUpdate(
{ conversationId: 'conv-live-1', user: uid },
{ $set: { title: 'Hello Base', endpoint: 'openAI', model: 'gpt-4o', messages: [] } },
{ new: true, upsert: true },
);
assert.strictEqual(convo.conversationId, 'conv-live-1');
assert.strictEqual(convo.title, 'Hello Base');
ok('saveConvo — conversation upserted to Base');
const msg = await models.Message.findOneAndUpdate(
{ messageId: 'msg-live-1', user: uid },
{
conversationId: 'conv-live-1',
text: 'Persisted to Hanzo Base (SQLite), not Mongo.',
sender: 'User',
isCreatedByUser: true,
},
{ new: true, upsert: true },
);
assert.strictEqual(msg.messageId, 'msg-live-1');
ok('saveMessage — message upserted to Base');
const msgs = await models.Message.find({ conversationId: 'conv-live-1', user: uid })
.sort({ createdAt: 1 })
.lean();
assert.strictEqual(msgs.length, 1);
assert.ok(msgs[0].text.includes('Hanzo Base'));
ok('getMessages — read back from Base, sorted');
// ---- VERIFY THE BYTES ARE IN BASE (raw REST, bypassing the adapter) ------
const raw = await store.client.send(
`/v1/collections/message/records?filter=${encodeURIComponent(`messageId='msg-live-1' && user='${uid}'`)}`,
);
assert.ok(raw.items && raw.items.length === 1, 'message present in Base via raw REST');
const storedDoc = raw.items[0].data;
const parsed = typeof storedDoc === 'string' ? JSON.parse(storedDoc) : storedDoc;
assert.strictEqual(parsed.messageId, 'msg-live-1', 'Base record.data holds the full document');
assert.strictEqual(raw.items[0].conversationId, 'conv-live-1', 'promoted column mirrored for filter');
ok('RAW Base REST confirms the document physically persisted to Base/SQLite');
// ---- FTS (Base/SQLite search replaces MeiliSearch) -----------------------
const convoHits = await models.Conversation.meiliSearch('hello', { filter: `user = "${uid}"` });
assert.ok(
convoHits.hits.some((h) => h.conversationId === 'conv-live-1'),
'conversation search returns the matching convo by title',
);
ok('Conversation.meiliSearch — real title hit from Base/SQLite');
const msgHits = await models.Message.meiliSearch('hanzo base', { filter: `user = "${uid}"` });
assert.ok(
msgHits.hits.some((h) => h.messageId === 'msg-live-1'),
'message search returns the matching message by text',
);
assert.strictEqual(
(await models.Conversation.meiliSearch('zzz-no-such-term', { filter: `user = "${uid}"` })).hits.length,
0,
'no false positives',
);
ok('Message.meiliSearch — real text hit from Base/SQLite (Meili dropped)');
// ---- Base UNIQUE index on natural key rejects a duplicate (race-safety) ----
let rejected = false;
try {
await store.create('conversation', {
_id: '0'.repeat(24),
conversationId: 'conv-live-1', // duplicate of the one saved above
user: uid,
data: { _id: '0'.repeat(24), conversationId: 'conv-live-1', user: uid },
});
} catch (err) {
rejected = true;
}
assert.ok(rejected, 'Base UNIQUE index on conversationId rejects a duplicate insert');
ok('Base UNIQUE index enforces conversationId uniqueness (race-safety)');
console.log('\nLIVE proof PASSED — login + conversation + message + SEARCH on Hanzo Base.');
console.log(` user collection id space, email=${email}`);
process.exit(0);
}
main().catch((e) => {
console.error('\nLIVE proof FAILED:', e && e.stack ? e.stack : e);
process.exit(1);
});
+233
View File
@@ -0,0 +1,233 @@
'use strict';
/**
* DocumentStore — the Hanzo Base backend for the Mongoose adapter.
*
* Each Mongo model maps to one Base collection whose records carry the full
* document as a JSON `data` field plus a handful of promoted scalar columns
* (derived from indexed/unique schema paths) used only for filter pushdown.
*
* Filter pushdown builds a Base filter-DSL string that is guaranteed to be a
* SUPERSET of the true result; query.js then narrows it in JS, so pushdown can
* be as conservative as needed without ever being wrong.
*/
const { idString } = require('./query');
// `@hanzo/base` ships as an ESM-only package (exports expose only the `import`
// condition), so it must be loaded from this CommonJS module via dynamic import.
let _baseModulePromise = null;
function loadBase() {
if (!_baseModulePromise) {
_baseModulePromise = import('@hanzo/base');
}
return _baseModulePromise;
}
const DATA_FIELD_MAX_SIZE = 8 * 1024 * 1024; // 8 MiB per document
/** Escape a string value for the Base filter DSL (single-quoted). */
function quote(value) {
return `'${String(value).replace(/\\/g, '\\\\').replace(/'/g, "\\'")}'`;
}
/** Format a scalar for the filter DSL, or return undefined if not pushable. */
function literal(value) {
const v = idString(value);
if (v === null || v === undefined) {
return undefined;
}
if (typeof v === 'number') {
return Number.isFinite(v) ? String(v) : undefined; // never emit `= NaN/Infinity`
}
if (typeof v === 'boolean') {
return String(v);
}
// Dates are intentionally NOT pushed down: Base normalizes its date columns to
// a `YYYY-MM-DD HH:MM:SS.sssZ` (space) form that never string-matches an ISO
// `T` literal, which would wrongly EXCLUDE the record (superset invariant
// violation). The JS matcher (query.js) handles all date predicates.
if (v instanceof Date) {
return undefined;
}
if (typeof v === 'string') {
return quote(v);
}
return undefined;
}
/**
* Collect conjunctive (AND-ed) constraints on promoted fields into a Base DSL
* filter. Disjunctions and non-promoted / operator predicates are skipped —
* the JS matcher enforces them — keeping the result a safe superset.
*/
function buildFilter(query, promoted) {
const promotedSet = new Set(promoted);
const clauses = [];
const walk = (q) => {
if (!q || typeof q !== 'object') {
return;
}
for (const [key, val] of Object.entries(q)) {
if (key === '$and' && Array.isArray(val)) {
val.forEach(walk);
continue;
}
if (key.startsWith('$')) {
continue; // $or/$nor/etc — not safe to push
}
if (!promotedSet.has(key)) {
continue;
}
if (val === null || typeof val !== 'object' || val instanceof Date) {
const lit = literal(val);
if (lit !== undefined) {
clauses.push(`${key} = ${lit}`);
}
continue;
}
if (Array.isArray(val)) {
continue;
}
// operator expression on a promoted field
if (Array.isArray(val.$in) && val.$in.length) {
const parts = val.$in.map((v) => literal(v)).filter((l) => l !== undefined);
if (parts.length === val.$in.length) {
clauses.push(`(${parts.map((l) => `${key} = ${l}`).join(' || ')})`);
}
continue;
}
if (val.$eq !== undefined) {
const lit = literal(val.$eq);
if (lit !== undefined) {
clauses.push(`${key} = ${lit}`);
}
}
}
};
walk(query);
return clauses.length ? clauses.join(' && ') : undefined;
}
class DocumentStore {
constructor() {
this.client = null;
this.url = null;
this._ensured = new Set();
}
/** @param {{ url: string, token?: string }} config */
async init({ url, token }) {
if (!url) {
throw new Error('[BaseStore] Base URL is required (HANZO_BASE_URL / BASE_URL)');
}
const { BaseClient } = await loadBase();
this.url = url.replace(/\/$/, '');
this.client = new BaseClient(this.url);
if (token) {
this.client.authStore.save(token, null);
}
return this;
}
async health() {
return this.client.send('/v1/health');
}
/** Idempotently ensure the Base collection exists with the required schema. */
async ensureCollection(name, promoted) {
if (this._ensured.has(name)) {
return;
}
try {
await this.client.send(`/v1/collections/${name}`, { method: 'GET' });
this._ensured.add(name);
return;
} catch (err) {
if (!isNotFound(err)) {
throw err;
}
}
const fields = [{ name: 'data', type: 'json', maxSize: DATA_FIELD_MAX_SIZE }];
const indexes = [];
for (const col of promoted) {
fields.push({ name: col.name, type: col.type });
const safe = col.name.replace(/[^A-Za-z0-9_]/g, '_');
const idx = `\`idx_${name}_${safe}\` ON \`${name}\` (\`${col.name}\`)`;
if (col.unique) {
indexes.push(`CREATE UNIQUE INDEX ${idx}`);
} else if (col.sparseUnique) {
// Partial unique: allow many empty values, enforce uniqueness on real ones.
indexes.push(`CREATE UNIQUE INDEX ${idx} WHERE \`${col.name}\` != ''`);
} else {
indexes.push(`CREATE INDEX ${idx}`);
}
}
const body = { name, type: 'base', fields, indexes };
try {
await this.client.send('/v1/collections', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(body),
});
} catch (err) {
// Tolerate a concurrent creator (unique-name violation).
if (!isConflict(err) && !isNotFound(err)) {
throw err;
}
}
this._ensured.add(name);
}
_parse(record) {
let doc = record.data;
if (typeof doc === 'string') {
try {
doc = JSON.parse(doc);
} catch {
doc = {};
}
}
return { baseId: record.id, doc: doc || {} };
}
/** List candidate docs for a Mongo filter (superset via pushdown). */
async list(collectionName, mongoFilter, promotedNames) {
const filter = buildFilter(mongoFilter, promotedNames || []);
const items = await this.client
.collection(collectionName)
.getFullList(filter ? { filter } : {});
return items.map((r) => this._parse(r));
}
async create(collectionName, record) {
const created = await this.client.collection(collectionName).create(record);
return this._parse(created);
}
async update(collectionName, baseId, record) {
const updated = await this.client.collection(collectionName).update(baseId, record);
return this._parse(updated);
}
async delete(collectionName, baseId) {
await this.client.collection(collectionName).delete(baseId);
}
}
function statusOf(err) {
return (err && (err.status || (err.data && err.data.status))) || 0;
}
function isNotFound(err) {
return statusOf(err) === 404;
}
function isConflict(err) {
return statusOf(err) === 400 || statusOf(err) === 409;
}
module.exports = new DocumentStore();
module.exports.DocumentStore = DocumentStore;
module.exports.buildFilter = buildFilter;
+43
View File
@@ -0,0 +1,43 @@
'use strict';
/* buildFilter — the Base filter-DSL pushdown. Must only ever emit a SUPERSET. */
const { buildFilter } = require('./store');
const promoted = ['_id', 'conversationId', 'user', 'expiredAt', 'tokenCount'];
describe('store/buildFilter pushdown', () => {
test('promoted equality + $in', () => {
expect(buildFilter({ conversationId: 'c1', user: 'u1' }, promoted)).toBe(
"conversationId = 'c1' && user = 'u1'",
);
expect(buildFilter({ conversationId: { $in: ['a', 'b'] } }, promoted)).toBe(
"(conversationId = 'a' || conversationId = 'b')",
);
});
test('DSL injection cannot break out of the quoted literal', () => {
const f = buildFilter({ conversationId: "x' || user='victim" }, promoted);
expect(f).toBe("conversationId = 'x\\' || user=\\'victim'");
expect(f).not.toMatch(/ \|\| user='victim/); // no un-escaped injected clause
});
test('Date predicates are NOT pushed down (superset invariant — H1)', () => {
// Base date columns normalize to a space separator; pushing an ISO literal
// would wrongly EXCLUDE the record. Dates must fall through to the JS matcher.
expect(buildFilter({ expiredAt: new Date('2026-07-04T12:00:00Z') }, promoted)).toBeUndefined();
expect(buildFilter({ expiredAt: { $eq: new Date() } }, promoted)).toBeUndefined();
// a date predicate must never partially push while dropping the date clause
expect(buildFilter({ user: 'u1', expiredAt: new Date() }, promoted)).toBe("user = 'u1'");
});
test('non-finite numbers are not pushed (invalid DSL guard — L1)', () => {
expect(buildFilter({ tokenCount: NaN }, promoted)).toBeUndefined();
expect(buildFilter({ tokenCount: Infinity }, promoted)).toBeUndefined();
expect(buildFilter({ tokenCount: 5 }, promoted)).toBe('tokenCount = 5');
});
test('non-promoted / $or / empty-$in are not pushed (JS matcher handles them)', () => {
expect(buildFilter({ title: 'x' }, promoted)).toBeUndefined();
expect(buildFilter({ $or: [{ user: 'a' }, { user: 'b' }] }, promoted)).toBeUndefined();
expect(buildFilter({ conversationId: { $in: [] } }, promoted)).toBeUndefined();
});
});
+7 -80
View File
@@ -1,83 +1,10 @@
require('dotenv').config();
const { isEnabled } = require('@hanzochat/api');
const { logger } = require('@librechat/data-schemas');
const mongoose = require('mongoose');
const MONGO_URI = process.env.MONGO_URI;
if (!MONGO_URI) {
throw new Error('Please define the MONGO_URI environment variable');
}
/** The maximum number of connections in the connection pool. */
const maxPoolSize = parseInt(process.env.MONGO_MAX_POOL_SIZE) || undefined;
/** The minimum number of connections in the connection pool. */
const minPoolSize = parseInt(process.env.MONGO_MIN_POOL_SIZE) || undefined;
/** The maximum number of connections that may be in the process of being established concurrently by the connection pool. */
const maxConnecting = parseInt(process.env.MONGO_MAX_CONNECTING) || undefined;
/** The maximum number of milliseconds that a connection can remain idle in the pool before being removed and closed. */
const maxIdleTimeMS = parseInt(process.env.MONGO_MAX_IDLE_TIME_MS) || undefined;
/** The maximum time in milliseconds that a thread can wait for a connection to become available. */
const waitQueueTimeoutMS = parseInt(process.env.MONGO_WAIT_QUEUE_TIMEOUT_MS) || undefined;
/** Set to false to disable automatic index creation for all models associated with this connection. */
const autoIndex =
process.env.MONGO_AUTO_INDEX != undefined
? isEnabled(process.env.MONGO_AUTO_INDEX) || false
: undefined;
/** Set to `false` to disable Mongoose automatically calling `createCollection()` on every model created on this connection. */
const autoCreate =
process.env.MONGO_AUTO_CREATE != undefined
? isEnabled(process.env.MONGO_AUTO_CREATE) || false
: undefined;
/**
* Global is used here to maintain a cached connection across hot reloads
* in development. This prevents connections growing exponentially
* during API Route usage.
* Database connection.
*
* MongoDB has been dropped from Hanzo Chat — the data layer runs on Hanzo Base
* (SQLite embedded / Postgres for prod multi-instance). `connectDb` initialises
* the Base client and provisions every registered model's collection. See ./base.
*/
let cached = global.mongoose;
const { connectDb } = require('./base');
if (!cached) {
cached = global.mongoose = { conn: null, promise: null };
}
mongoose.connection.on('error', (err) => {
logger.error('[connectDb] MongoDB connection error:', err);
});
async function connectDb() {
if (cached.conn && cached.conn?._readyState === 1) {
return cached.conn;
}
const disconnected = cached.conn && cached.conn?._readyState !== 1;
if (!cached.promise || disconnected) {
const opts = {
bufferCommands: false,
...(maxPoolSize ? { maxPoolSize } : {}),
...(minPoolSize ? { minPoolSize } : {}),
...(maxConnecting ? { maxConnecting } : {}),
...(maxIdleTimeMS ? { maxIdleTimeMS } : {}),
...(waitQueueTimeoutMS ? { waitQueueTimeoutMS } : {}),
...(autoIndex != undefined ? { autoIndex } : {}),
...(autoCreate != undefined ? { autoCreate } : {}),
// useNewUrlParser: true,
// useUnifiedTopology: true,
// bufferMaxEntries: 0,
// useFindAndModify: true,
// useCreateIndex: true
};
logger.info('Mongo Connection options');
logger.info(JSON.stringify(opts, null, 2));
mongoose.set('strictQuery', true);
cached.promise = mongoose.connect(MONGO_URI, opts).then((mongoose) => {
return mongoose;
});
}
cached.conn = await cached.promise;
return cached.conn;
}
module.exports = {
connectDb,
};
module.exports = { connectDb };
+2 -2
View File
@@ -1,8 +1,8 @@
const mongoose = require('mongoose');
const { mongoose, connectDb } = require('./base');
const { createModels } = require('@librechat/data-schemas');
const { connectDb } = require('./connect');
const indexSync = require('./indexSync');
// Register every model on the Base-backed mongoose facade.
createModels(mongoose);
module.exports = { connectDb, indexSync };
+8 -355
View File
@@ -1,363 +1,16 @@
const mongoose = require('mongoose');
const { MeiliSearch } = require('meilisearch');
const { logger } = require('@librechat/data-schemas');
const { CacheKeys } = require('librechat-data-provider');
const { isEnabled, FlowStateManager } = require('@hanzochat/api');
const { getLogStores } = require('~/cache');
const { batchResetMeiliFlags } = require('./utils');
const Conversation = mongoose.models.Conversation;
const Message = mongoose.models.Message;
const searchEnabled = isEnabled(process.env.SEARCH);
const indexingDisabled = isEnabled(process.env.MEILI_NO_SYNC);
let currentTimeout = null;
const defaultSyncThreshold = 1000;
const syncThreshold = process.env.MEILI_SYNC_THRESHOLD
? parseInt(process.env.MEILI_SYNC_THRESHOLD, 10)
: defaultSyncThreshold;
class MeiliSearchClient {
static instance = null;
static getInstance() {
if (!MeiliSearchClient.instance) {
if (!process.env.MEILI_HOST || !process.env.MEILI_MASTER_KEY) {
throw new Error('Meilisearch configuration is missing.');
}
MeiliSearchClient.instance = new MeiliSearch({
host: process.env.MEILI_HOST,
apiKey: process.env.MEILI_MASTER_KEY,
});
}
return MeiliSearchClient.instance;
}
}
/**
* Deletes documents from MeiliSearch index that are missing the user field
* @param {import('meilisearch').Index} index - MeiliSearch index instance
* @param {string} indexName - Name of the index for logging
* @returns {Promise<number>} - Number of documents deleted
*/
async function deleteDocumentsWithoutUserField(index, indexName) {
let deletedCount = 0;
let offset = 0;
const batchSize = 1000;
try {
while (true) {
const searchResult = await index.search('', {
limit: batchSize,
offset: offset,
});
if (searchResult.hits.length === 0) {
break;
}
const idsToDelete = searchResult.hits.filter((hit) => !hit.user).map((hit) => hit.id);
if (idsToDelete.length > 0) {
logger.info(
`[indexSync] Deleting ${idsToDelete.length} documents without user field from ${indexName} index`,
);
await index.deleteDocuments(idsToDelete);
deletedCount += idsToDelete.length;
}
if (searchResult.hits.length < batchSize) {
break;
}
offset += batchSize;
}
if (deletedCount > 0) {
logger.info(`[indexSync] Deleted ${deletedCount} orphaned documents from ${indexName} index`);
}
} catch (error) {
logger.error(`[indexSync] Error deleting documents from ${indexName}:`, error);
}
return deletedCount;
}
/**
* Ensures indexes have proper filterable attributes configured and checks if documents have user field
* @param {MeiliSearch} client - MeiliSearch client instance
* @returns {Promise<{settingsUpdated: boolean, orphanedDocsFound: boolean}>} - Status of what was done
*/
async function ensureFilterableAttributes(client) {
let settingsUpdated = false;
let hasOrphanedDocs = false;
try {
// Check and update messages index
try {
const messagesIndex = client.index('messages');
const settings = await messagesIndex.getSettings();
if (!settings.filterableAttributes || !settings.filterableAttributes.includes('user')) {
logger.info('[indexSync] Configuring messages index to filter by user...');
await messagesIndex.updateSettings({
filterableAttributes: ['user'],
});
logger.info('[indexSync] Messages index configured for user filtering');
settingsUpdated = true;
}
// Check if existing documents have user field indexed
try {
const searchResult = await messagesIndex.search('', { limit: 1 });
if (searchResult.hits.length > 0 && !searchResult.hits[0].user) {
logger.info(
'[indexSync] Existing messages missing user field, will clean up orphaned documents...',
);
hasOrphanedDocs = true;
}
} catch (searchError) {
logger.debug('[indexSync] Could not check message documents:', searchError.message);
}
} catch (error) {
if (error.code !== 'index_not_found') {
logger.warn('[indexSync] Could not check/update messages index settings:', error.message);
}
}
// Check and update conversations index
try {
const convosIndex = client.index('convos');
const settings = await convosIndex.getSettings();
if (!settings.filterableAttributes || !settings.filterableAttributes.includes('user')) {
logger.info('[indexSync] Configuring convos index to filter by user...');
await convosIndex.updateSettings({
filterableAttributes: ['user'],
});
logger.info('[indexSync] Convos index configured for user filtering');
settingsUpdated = true;
}
// Check if existing documents have user field indexed
try {
const searchResult = await convosIndex.search('', { limit: 1 });
if (searchResult.hits.length > 0 && !searchResult.hits[0].user) {
logger.info(
'[indexSync] Existing conversations missing user field, will clean up orphaned documents...',
);
hasOrphanedDocs = true;
}
} catch (searchError) {
logger.debug('[indexSync] Could not check conversation documents:', searchError.message);
}
} catch (error) {
if (error.code !== 'index_not_found') {
logger.warn('[indexSync] Could not check/update convos index settings:', error.message);
}
}
// If either index has orphaned documents, clean them up (but don't force resync)
if (hasOrphanedDocs) {
try {
const messagesIndex = client.index('messages');
await deleteDocumentsWithoutUserField(messagesIndex, 'messages');
} catch (error) {
logger.debug('[indexSync] Could not clean up messages:', error.message);
}
try {
const convosIndex = client.index('convos');
await deleteDocumentsWithoutUserField(convosIndex, 'convos');
} catch (error) {
logger.debug('[indexSync] Could not clean up convos:', error.message);
}
logger.info('[indexSync] Orphaned documents cleaned up without forcing resync.');
}
if (settingsUpdated) {
logger.info('[indexSync] Index settings updated. Full re-sync will be triggered.');
}
} catch (error) {
logger.error('[indexSync] Error ensuring filterable attributes:', error);
}
return { settingsUpdated, orphanedDocsFound: hasOrphanedDocs };
}
/**
* Performs the actual sync operations for messages and conversations
* @param {FlowStateManager} flowManager - Flow state manager instance
* @param {string} flowId - Flow identifier
* @param {string} flowType - Flow type
*/
async function performSync(flowManager, flowId, flowType) {
try {
if (indexingDisabled === true) {
logger.info('[indexSync] Indexing is disabled, skipping...');
return { messagesSync: false, convosSync: false };
}
const client = MeiliSearchClient.getInstance();
const { status } = await client.health();
if (status !== 'available') {
throw new Error('Meilisearch not available');
}
/** Ensures indexes have proper filterable attributes configured */
const { settingsUpdated, orphanedDocsFound: _orphanedDocsFound } =
await ensureFilterableAttributes(client);
let messagesSync = false;
let convosSync = false;
// Only reset flags if settings were actually updated (not just for orphaned doc cleanup)
if (settingsUpdated) {
logger.info(
'[indexSync] Settings updated. Forcing full re-sync to reindex with new configuration...',
);
// Reset sync flags to force full re-sync
await batchResetMeiliFlags(Message.collection);
await batchResetMeiliFlags(Conversation.collection);
}
// Check if we need to sync messages
logger.info('[indexSync] Requesting message sync progress...');
const messageProgress = await Message.getSyncProgress();
if (!messageProgress.isComplete || settingsUpdated) {
logger.info(
`[indexSync] Messages need syncing: ${messageProgress.totalProcessed}/${messageProgress.totalDocuments} indexed`,
);
const messageCount = messageProgress.totalDocuments;
const messagesIndexed = messageProgress.totalProcessed;
const unindexedMessages = messageCount - messagesIndexed;
if (settingsUpdated || unindexedMessages > syncThreshold) {
logger.info(`[indexSync] Starting message sync (${unindexedMessages} unindexed)`);
await Message.syncWithMeili();
messagesSync = true;
} else if (unindexedMessages > 0) {
logger.info(
`[indexSync] ${unindexedMessages} messages unindexed (below threshold: ${syncThreshold}, skipping)`,
);
}
} else {
logger.info(
`[indexSync] Messages are fully synced: ${messageProgress.totalProcessed}/${messageProgress.totalDocuments}`,
);
}
// Check if we need to sync conversations
const convoProgress = await Conversation.getSyncProgress();
if (!convoProgress.isComplete || settingsUpdated) {
logger.info(
`[indexSync] Conversations need syncing: ${convoProgress.totalProcessed}/${convoProgress.totalDocuments} indexed`,
);
const convoCount = convoProgress.totalDocuments;
const convosIndexed = convoProgress.totalProcessed;
const unindexedConvos = convoCount - convosIndexed;
if (settingsUpdated || unindexedConvos > syncThreshold) {
logger.info(`[indexSync] Starting convos sync (${unindexedConvos} unindexed)`);
await Conversation.syncWithMeili();
convosSync = true;
} else if (unindexedConvos > 0) {
logger.info(
`[indexSync] ${unindexedConvos} convos unindexed (below threshold: ${syncThreshold}, skipping)`,
);
}
} else {
logger.info(
`[indexSync] Conversations are fully synced: ${convoProgress.totalProcessed}/${convoProgress.totalDocuments}`,
);
}
return { messagesSync, convosSync };
} finally {
if (indexingDisabled === true) {
logger.info('[indexSync] Indexing is disabled, skipping cleanup...');
} else if (flowManager && flowId && flowType) {
try {
await flowManager.deleteFlow(flowId, flowType);
logger.debug('[indexSync] Flow state cleaned up');
} catch (cleanupErr) {
logger.debug('[indexSync] Could not clean up flow state:', cleanupErr.message);
}
}
}
}
/**
* Main index sync function that uses FlowStateManager to prevent concurrent execution
* Search index sync.
*
* MongoDB + MeiliSearch have been dropped. Full-text search now runs directly
* on Hanzo Base/SQLite via the data-layer adapter (the model `meiliSearch()`
* method in api/db/base searches content fields on Base). There is no separate
* index to sync, so this is a no-op kept for boot-sequence compatibility
* (api/server/index.js calls `indexSync()`).
*/
async function indexSync() {
if (!searchEnabled) {
return;
}
logger.info('[indexSync] Starting index synchronization check...');
// Get or create FlowStateManager instance
const flowsCache = getLogStores(CacheKeys.FLOWS);
if (!flowsCache) {
logger.warn('[indexSync] Flows cache not available, falling back to direct sync');
return await performSync(null, null, null);
}
const flowManager = new FlowStateManager(flowsCache, {
ttl: 60000 * 10, // 10 minutes TTL for sync operations
});
// Use a unique flow ID for the sync operation
const flowId = 'meili-index-sync';
const flowType = 'MEILI_SYNC';
try {
// This will only execute the handler if no other instance is running the sync
const result = await flowManager.createFlowWithHandler(flowId, flowType, () =>
performSync(flowManager, flowId, flowType),
);
if (result.messagesSync || result.convosSync) {
logger.info('[indexSync] Sync completed successfully');
} else {
logger.debug('[indexSync] No sync was needed');
}
return result;
} catch (err) {
if (err.message.includes('flow already exists')) {
logger.info('[indexSync] Sync already running on another instance');
return;
}
if (err.message.includes('not found')) {
logger.debug('[indexSync] Creating indices...');
currentTimeout = setTimeout(async () => {
try {
await Message.syncWithMeili();
await Conversation.syncWithMeili();
} catch (err) {
logger.error('[indexSync] Trouble creating indices, try restarting the server.', err);
}
}, 750);
} else if (err.message.includes('Meilisearch not configured')) {
logger.info('[indexSync] Meilisearch not configured, search will be disabled.');
} else {
logger.error('[indexSync] error', err);
}
}
logger.info('[indexSync] Base/SQLite search active — no external index to sync');
}
process.on('exit', () => {
logger.debug('[indexSync] Clearing sync timeouts before exiting...');
clearTimeout(currentTimeout);
});
module.exports = indexSync;
-465
View File
@@ -1,465 +0,0 @@
/**
* Unit tests for performSync() function in indexSync.js
*
* Tests use real mongoose with mocked model methods, only mocking external calls.
*/
const mongoose = require('mongoose');
// Mock only external dependencies (not internal classes/models)
const mockLogger = {
info: jest.fn(),
warn: jest.fn(),
error: jest.fn(),
debug: jest.fn(),
};
const mockMeiliHealth = jest.fn();
const mockMeiliIndex = jest.fn();
const mockBatchResetMeiliFlags = jest.fn();
const mockIsEnabled = jest.fn();
const mockGetLogStores = jest.fn();
// Create mock models that will be reused
const createMockModel = (collectionName) => ({
collection: { name: collectionName },
getSyncProgress: jest.fn(),
syncWithMeili: jest.fn(),
countDocuments: jest.fn(),
});
const originalMessageModel = mongoose.models.Message;
const originalConversationModel = mongoose.models.Conversation;
// Mock external modules
jest.mock('@librechat/data-schemas', () => ({
logger: mockLogger,
}));
jest.mock('meilisearch', () => ({
MeiliSearch: jest.fn(() => ({
health: mockMeiliHealth,
index: mockMeiliIndex,
})),
}));
jest.mock('./utils', () => ({
batchResetMeiliFlags: mockBatchResetMeiliFlags,
}));
jest.mock('@hanzochat/api', () => ({
isEnabled: mockIsEnabled,
FlowStateManager: jest.fn(),
}));
jest.mock('~/cache', () => ({
getLogStores: mockGetLogStores,
}));
// Set environment before module load
process.env.MEILI_HOST = 'http://localhost:7700';
process.env.MEILI_MASTER_KEY = 'test-key';
process.env.SEARCH = 'true';
process.env.MEILI_SYNC_THRESHOLD = '1000'; // Set threshold before module loads
describe('performSync() - syncThreshold logic', () => {
const ORIGINAL_ENV = process.env;
let Message;
let Conversation;
beforeAll(() => {
Message = createMockModel('messages');
Conversation = createMockModel('conversations');
mongoose.models.Message = Message;
mongoose.models.Conversation = Conversation;
});
beforeEach(() => {
// Reset all mocks
jest.clearAllMocks();
// Reset modules to ensure fresh load of indexSync.js and its top-level consts (like syncThreshold)
jest.resetModules();
// Set up environment
process.env = { ...ORIGINAL_ENV };
process.env.MEILI_HOST = 'http://localhost:7700';
process.env.MEILI_MASTER_KEY = 'test-key';
process.env.SEARCH = 'true';
delete process.env.MEILI_NO_SYNC;
// Re-ensure models are available in mongoose after resetModules
// We must require mongoose again to get the fresh instance that indexSync will use
const mongoose = require('mongoose');
mongoose.models.Message = Message;
mongoose.models.Conversation = Conversation;
// Mock isEnabled
mockIsEnabled.mockImplementation((val) => val === 'true' || val === true);
// Mock MeiliSearch client responses
mockMeiliHealth.mockResolvedValue({ status: 'available' });
mockMeiliIndex.mockReturnValue({
getSettings: jest.fn().mockResolvedValue({ filterableAttributes: ['user'] }),
updateSettings: jest.fn().mockResolvedValue({}),
search: jest.fn().mockResolvedValue({ hits: [] }),
});
mockBatchResetMeiliFlags.mockResolvedValue(undefined);
});
afterEach(() => {
process.env = ORIGINAL_ENV;
});
afterAll(() => {
mongoose.models.Message = originalMessageModel;
mongoose.models.Conversation = originalConversationModel;
});
test('triggers sync when unindexed messages exceed syncThreshold', async () => {
// Arrange: Set threshold before module load
process.env.MEILI_SYNC_THRESHOLD = '1000';
// Arrange: 1050 unindexed messages > 1000 threshold
Message.getSyncProgress.mockResolvedValue({
totalProcessed: 100,
totalDocuments: 1150, // 1050 unindexed
isComplete: false,
});
Conversation.getSyncProgress.mockResolvedValue({
totalProcessed: 50,
totalDocuments: 50,
isComplete: true,
});
Message.syncWithMeili.mockResolvedValue(undefined);
// Act
const indexSync = require('./indexSync');
await indexSync();
// Assert: No countDocuments calls
expect(Message.countDocuments).not.toHaveBeenCalled();
expect(Conversation.countDocuments).not.toHaveBeenCalled();
// Assert: Message sync triggered because 1050 > 1000
expect(Message.syncWithMeili).toHaveBeenCalledTimes(1);
expect(mockLogger.info).toHaveBeenCalledWith(
'[indexSync] Messages need syncing: 100/1150 indexed',
);
expect(mockLogger.info).toHaveBeenCalledWith(
'[indexSync] Starting message sync (1050 unindexed)',
);
// Assert: Conversation sync NOT triggered (already complete)
expect(Conversation.syncWithMeili).not.toHaveBeenCalled();
});
test('skips sync when unindexed messages are below syncThreshold', async () => {
// Arrange: 50 unindexed messages < 1000 threshold
Message.getSyncProgress.mockResolvedValue({
totalProcessed: 100,
totalDocuments: 150, // 50 unindexed
isComplete: false,
});
Conversation.getSyncProgress.mockResolvedValue({
totalProcessed: 50,
totalDocuments: 50,
isComplete: true,
});
process.env.MEILI_SYNC_THRESHOLD = '1000';
// Act
const indexSync = require('./indexSync');
await indexSync();
// Assert: No countDocuments calls
expect(Message.countDocuments).not.toHaveBeenCalled();
expect(Conversation.countDocuments).not.toHaveBeenCalled();
// Assert: Message sync NOT triggered because 50 < 1000
expect(Message.syncWithMeili).not.toHaveBeenCalled();
expect(mockLogger.info).toHaveBeenCalledWith(
'[indexSync] Messages need syncing: 100/150 indexed',
);
expect(mockLogger.info).toHaveBeenCalledWith(
'[indexSync] 50 messages unindexed (below threshold: 1000, skipping)',
);
// Assert: Conversation sync NOT triggered (already complete)
expect(Conversation.syncWithMeili).not.toHaveBeenCalled();
});
test('respects syncThreshold at boundary (exactly at threshold)', async () => {
// Arrange: 1000 unindexed messages = 1000 threshold (NOT greater than)
Message.getSyncProgress.mockResolvedValue({
totalProcessed: 100,
totalDocuments: 1100, // 1000 unindexed
isComplete: false,
});
Conversation.getSyncProgress.mockResolvedValue({
totalProcessed: 0,
totalDocuments: 0,
isComplete: true,
});
process.env.MEILI_SYNC_THRESHOLD = '1000';
// Act
const indexSync = require('./indexSync');
await indexSync();
// Assert: No countDocuments calls
expect(Message.countDocuments).not.toHaveBeenCalled();
// Assert: Message sync NOT triggered because 1000 is NOT > 1000
expect(Message.syncWithMeili).not.toHaveBeenCalled();
expect(mockLogger.info).toHaveBeenCalledWith(
'[indexSync] Messages need syncing: 100/1100 indexed',
);
expect(mockLogger.info).toHaveBeenCalledWith(
'[indexSync] 1000 messages unindexed (below threshold: 1000, skipping)',
);
});
test('triggers sync when unindexed is threshold + 1', async () => {
// Arrange: 1001 unindexed messages > 1000 threshold
Message.getSyncProgress.mockResolvedValue({
totalProcessed: 100,
totalDocuments: 1101, // 1001 unindexed
isComplete: false,
});
Conversation.getSyncProgress.mockResolvedValue({
totalProcessed: 0,
totalDocuments: 0,
isComplete: true,
});
Message.syncWithMeili.mockResolvedValue(undefined);
process.env.MEILI_SYNC_THRESHOLD = '1000';
// Act
const indexSync = require('./indexSync');
await indexSync();
// Assert: No countDocuments calls
expect(Message.countDocuments).not.toHaveBeenCalled();
// Assert: Message sync triggered because 1001 > 1000
expect(Message.syncWithMeili).toHaveBeenCalledTimes(1);
expect(mockLogger.info).toHaveBeenCalledWith(
'[indexSync] Messages need syncing: 100/1101 indexed',
);
expect(mockLogger.info).toHaveBeenCalledWith(
'[indexSync] Starting message sync (1001 unindexed)',
);
});
test('uses totalDocuments from convoProgress for conversation sync decisions', async () => {
// Arrange: Messages complete, conversations need sync
Message.getSyncProgress.mockResolvedValue({
totalProcessed: 100,
totalDocuments: 100,
isComplete: true,
});
Conversation.getSyncProgress.mockResolvedValue({
totalProcessed: 50,
totalDocuments: 1100, // 1050 unindexed > 1000 threshold
isComplete: false,
});
Conversation.syncWithMeili.mockResolvedValue(undefined);
process.env.MEILI_SYNC_THRESHOLD = '1000';
// Act
const indexSync = require('./indexSync');
await indexSync();
// Assert: No countDocuments calls (the optimization)
expect(Message.countDocuments).not.toHaveBeenCalled();
expect(Conversation.countDocuments).not.toHaveBeenCalled();
// Assert: Only conversation sync triggered
expect(Message.syncWithMeili).not.toHaveBeenCalled();
expect(Conversation.syncWithMeili).toHaveBeenCalledTimes(1);
expect(mockLogger.info).toHaveBeenCalledWith(
'[indexSync] Conversations need syncing: 50/1100 indexed',
);
expect(mockLogger.info).toHaveBeenCalledWith(
'[indexSync] Starting convos sync (1050 unindexed)',
);
});
test('skips sync when collections are fully synced', async () => {
// Arrange: Everything already synced
Message.getSyncProgress.mockResolvedValue({
totalProcessed: 100,
totalDocuments: 100,
isComplete: true,
});
Conversation.getSyncProgress.mockResolvedValue({
totalProcessed: 50,
totalDocuments: 50,
isComplete: true,
});
// Act
const indexSync = require('./indexSync');
await indexSync();
// Assert: No countDocuments calls
expect(Message.countDocuments).not.toHaveBeenCalled();
expect(Conversation.countDocuments).not.toHaveBeenCalled();
// Assert: No sync triggered
expect(Message.syncWithMeili).not.toHaveBeenCalled();
expect(Conversation.syncWithMeili).not.toHaveBeenCalled();
// Assert: Correct logs
expect(mockLogger.info).toHaveBeenCalledWith('[indexSync] Messages are fully synced: 100/100');
expect(mockLogger.info).toHaveBeenCalledWith(
'[indexSync] Conversations are fully synced: 50/50',
);
});
test('triggers message sync when settingsUpdated even if below syncThreshold', async () => {
// Arrange: Only 50 unindexed messages (< 1000 threshold), but settings were updated
Message.getSyncProgress.mockResolvedValue({
totalProcessed: 100,
totalDocuments: 150, // 50 unindexed
isComplete: false,
});
Conversation.getSyncProgress.mockResolvedValue({
totalProcessed: 50,
totalDocuments: 50,
isComplete: true,
});
Message.syncWithMeili.mockResolvedValue(undefined);
// Mock settings update scenario
mockMeiliIndex.mockReturnValue({
getSettings: jest.fn().mockResolvedValue({ filterableAttributes: [] }), // No user field
updateSettings: jest.fn().mockResolvedValue({}),
search: jest.fn().mockResolvedValue({ hits: [] }),
});
process.env.MEILI_SYNC_THRESHOLD = '1000';
// Act
const indexSync = require('./indexSync');
await indexSync();
// Assert: Flags were reset due to settings update
expect(mockBatchResetMeiliFlags).toHaveBeenCalledWith(Message.collection);
expect(mockBatchResetMeiliFlags).toHaveBeenCalledWith(Conversation.collection);
// Assert: Message sync triggered despite being below threshold (50 < 1000)
expect(Message.syncWithMeili).toHaveBeenCalledTimes(1);
expect(mockLogger.info).toHaveBeenCalledWith(
'[indexSync] Settings updated. Forcing full re-sync to reindex with new configuration...',
);
expect(mockLogger.info).toHaveBeenCalledWith(
'[indexSync] Starting message sync (50 unindexed)',
);
});
test('triggers conversation sync when settingsUpdated even if below syncThreshold', async () => {
// Arrange: Messages complete, conversations have 50 unindexed (< 1000 threshold), but settings were updated
Message.getSyncProgress.mockResolvedValue({
totalProcessed: 100,
totalDocuments: 100,
isComplete: true,
});
Conversation.getSyncProgress.mockResolvedValue({
totalProcessed: 50,
totalDocuments: 100, // 50 unindexed
isComplete: false,
});
Conversation.syncWithMeili.mockResolvedValue(undefined);
// Mock settings update scenario
mockMeiliIndex.mockReturnValue({
getSettings: jest.fn().mockResolvedValue({ filterableAttributes: [] }), // No user field
updateSettings: jest.fn().mockResolvedValue({}),
search: jest.fn().mockResolvedValue({ hits: [] }),
});
process.env.MEILI_SYNC_THRESHOLD = '1000';
// Act
const indexSync = require('./indexSync');
await indexSync();
// Assert: Flags were reset due to settings update
expect(mockBatchResetMeiliFlags).toHaveBeenCalledWith(Message.collection);
expect(mockBatchResetMeiliFlags).toHaveBeenCalledWith(Conversation.collection);
// Assert: Conversation sync triggered despite being below threshold (50 < 1000)
expect(Conversation.syncWithMeili).toHaveBeenCalledTimes(1);
expect(mockLogger.info).toHaveBeenCalledWith(
'[indexSync] Settings updated. Forcing full re-sync to reindex with new configuration...',
);
expect(mockLogger.info).toHaveBeenCalledWith('[indexSync] Starting convos sync (50 unindexed)');
});
test('triggers both message and conversation sync when settingsUpdated even if both below syncThreshold', async () => {
// Arrange: Set threshold before module load
process.env.MEILI_SYNC_THRESHOLD = '1000';
// Arrange: Both have documents below threshold (50 each), but settings were updated
Message.getSyncProgress.mockResolvedValue({
totalProcessed: 100,
totalDocuments: 150, // 50 unindexed
isComplete: false,
});
Conversation.getSyncProgress.mockResolvedValue({
totalProcessed: 50,
totalDocuments: 100, // 50 unindexed
isComplete: false,
});
Message.syncWithMeili.mockResolvedValue(undefined);
Conversation.syncWithMeili.mockResolvedValue(undefined);
// Mock settings update scenario
mockMeiliIndex.mockReturnValue({
getSettings: jest.fn().mockResolvedValue({ filterableAttributes: [] }), // No user field
updateSettings: jest.fn().mockResolvedValue({}),
search: jest.fn().mockResolvedValue({ hits: [] }),
});
// Act
const indexSync = require('./indexSync');
await indexSync();
// Assert: Flags were reset due to settings update
expect(mockBatchResetMeiliFlags).toHaveBeenCalledWith(Message.collection);
expect(mockBatchResetMeiliFlags).toHaveBeenCalledWith(Conversation.collection);
// Assert: Both syncs triggered despite both being below threshold
expect(Message.syncWithMeili).toHaveBeenCalledTimes(1);
expect(Conversation.syncWithMeili).toHaveBeenCalledTimes(1);
expect(mockLogger.info).toHaveBeenCalledWith(
'[indexSync] Settings updated. Forcing full re-sync to reindex with new configuration...',
);
expect(mockLogger.info).toHaveBeenCalledWith(
'[indexSync] Starting message sync (50 unindexed)',
);
expect(mockLogger.info).toHaveBeenCalledWith('[indexSync] Starting convos sync (50 unindexed)');
});
});
+1 -1
View File
@@ -1,4 +1,4 @@
const mongoose = require('mongoose');
const { mongoose } = require('./base');
const { createModels } = require('@librechat/data-schemas');
const models = createModels(mongoose);
-90
View File
@@ -1,90 +0,0 @@
const { logger } = require('@librechat/data-schemas');
const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
/**
* Batch update documents in chunks to avoid timeouts on weak instances
* @param {mongoose.Collection} collection - MongoDB collection
* @returns {Promise<number>} - Total modified count
* @throws {Error} - Throws if database operations fail (e.g., network issues, connection loss, permission problems)
*/
async function batchResetMeiliFlags(collection) {
const DEFAULT_BATCH_SIZE = 1000;
let BATCH_SIZE = parseEnvInt('MEILI_SYNC_BATCH_SIZE', DEFAULT_BATCH_SIZE);
if (BATCH_SIZE === 0) {
logger.warn(
`[batchResetMeiliFlags] MEILI_SYNC_BATCH_SIZE cannot be 0. Using default: ${DEFAULT_BATCH_SIZE}`,
);
BATCH_SIZE = DEFAULT_BATCH_SIZE;
}
const BATCH_DELAY_MS = parseEnvInt('MEILI_SYNC_DELAY_MS', 100);
let totalModified = 0;
let hasMore = true;
try {
while (hasMore) {
const docs = await collection
.find({ expiredAt: null, _meiliIndex: { $ne: false } }, { projection: { _id: 1 } })
.limit(BATCH_SIZE)
.toArray();
if (docs.length === 0) {
break;
}
const ids = docs.map((doc) => doc._id);
const result = await collection.updateMany(
{ _id: { $in: ids } },
{ $set: { _meiliIndex: false } },
);
totalModified += result.modifiedCount;
process.stdout.write(
`\r Updating ${collection.collectionName}: ${totalModified} documents...`,
);
if (docs.length < BATCH_SIZE) {
hasMore = false;
}
if (hasMore && BATCH_DELAY_MS > 0) {
await sleep(BATCH_DELAY_MS);
}
}
return totalModified;
} catch (error) {
throw new Error(
`Failed to batch reset Meili flags for collection '${collection.collectionName}' after processing ${totalModified} documents: ${error.message}`,
);
}
}
/**
* Parse and validate an environment variable as a positive integer
* @param {string} varName - Environment variable name
* @param {number} defaultValue - Default value to use if invalid or missing
* @returns {number} - Parsed value or default
*/
function parseEnvInt(varName, defaultValue) {
const value = process.env[varName];
if (!value) {
return defaultValue;
}
const parsed = parseInt(value, 10);
if (isNaN(parsed) || parsed < 0) {
logger.warn(
`[batchResetMeiliFlags] Invalid value for ${varName}="${value}". Expected a positive integer. Using default: ${defaultValue}`,
);
return defaultValue;
}
return parsed;
}
module.exports = {
batchResetMeiliFlags,
};
-523
View File
@@ -1,523 +0,0 @@
const mongoose = require('mongoose');
const { MongoMemoryServer } = require('mongodb-memory-server');
const { batchResetMeiliFlags } = require('./utils');
describe('batchResetMeiliFlags', () => {
let mongoServer;
let testCollection;
const ORIGINAL_BATCH_SIZE = process.env.MEILI_SYNC_BATCH_SIZE;
const ORIGINAL_BATCH_DELAY = process.env.MEILI_SYNC_DELAY_MS;
beforeAll(async () => {
mongoServer = await MongoMemoryServer.create();
const mongoUri = mongoServer.getUri();
await mongoose.connect(mongoUri);
});
afterAll(async () => {
await mongoose.disconnect();
await mongoServer.stop();
// Restore original env variables
if (ORIGINAL_BATCH_SIZE !== undefined) {
process.env.MEILI_SYNC_BATCH_SIZE = ORIGINAL_BATCH_SIZE;
} else {
delete process.env.MEILI_SYNC_BATCH_SIZE;
}
if (ORIGINAL_BATCH_DELAY !== undefined) {
process.env.MEILI_SYNC_DELAY_MS = ORIGINAL_BATCH_DELAY;
} else {
delete process.env.MEILI_SYNC_DELAY_MS;
}
});
beforeEach(async () => {
// Create a fresh collection for each test
testCollection = mongoose.connection.db.collection('test_meili_batch');
await testCollection.deleteMany({});
// Reset env variables to defaults
delete process.env.MEILI_SYNC_BATCH_SIZE;
delete process.env.MEILI_SYNC_DELAY_MS;
});
afterEach(async () => {
if (testCollection) {
await testCollection.deleteMany({});
}
});
describe('basic functionality', () => {
it('should reset _meiliIndex flag for documents with expiredAt: null and _meiliIndex: true', async () => {
// Insert test documents
await testCollection.insertMany([
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: true, name: 'doc1' },
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: true, name: 'doc2' },
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: true, name: 'doc3' },
]);
const result = await batchResetMeiliFlags(testCollection);
expect(result).toBe(3);
const updatedDocs = await testCollection.find({ _meiliIndex: false }).toArray();
expect(updatedDocs).toHaveLength(3);
const notUpdatedDocs = await testCollection.find({ _meiliIndex: true }).toArray();
expect(notUpdatedDocs).toHaveLength(0);
});
it('should not modify documents with expiredAt set', async () => {
const expiredDate = new Date();
await testCollection.insertMany([
{ _id: new mongoose.Types.ObjectId(), expiredAt: expiredDate, _meiliIndex: true },
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: true },
]);
const result = await batchResetMeiliFlags(testCollection);
expect(result).toBe(1);
const expiredDoc = await testCollection.findOne({ expiredAt: expiredDate });
expect(expiredDoc._meiliIndex).toBe(true);
});
it('should not modify documents with _meiliIndex: false', async () => {
await testCollection.insertMany([
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: false },
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: true },
]);
const result = await batchResetMeiliFlags(testCollection);
expect(result).toBe(1);
});
it('should return 0 when no documents match the criteria', async () => {
await testCollection.insertMany([
{ _id: new mongoose.Types.ObjectId(), expiredAt: new Date(), _meiliIndex: true },
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: false },
]);
const result = await batchResetMeiliFlags(testCollection);
expect(result).toBe(0);
});
it('should return 0 when collection is empty', async () => {
const result = await batchResetMeiliFlags(testCollection);
expect(result).toBe(0);
});
});
describe('batch processing', () => {
it('should process documents in batches according to MEILI_SYNC_BATCH_SIZE', async () => {
process.env.MEILI_SYNC_BATCH_SIZE = '2';
const docs = [];
for (let i = 0; i < 5; i++) {
docs.push({
_id: new mongoose.Types.ObjectId(),
expiredAt: null,
_meiliIndex: true,
name: `doc${i}`,
});
}
await testCollection.insertMany(docs);
const result = await batchResetMeiliFlags(testCollection);
expect(result).toBe(5);
const updatedDocs = await testCollection.find({ _meiliIndex: false }).toArray();
expect(updatedDocs).toHaveLength(5);
});
it('should handle large datasets with small batch sizes', async () => {
process.env.MEILI_SYNC_BATCH_SIZE = '10';
const docs = [];
for (let i = 0; i < 25; i++) {
docs.push({
_id: new mongoose.Types.ObjectId(),
expiredAt: null,
_meiliIndex: true,
});
}
await testCollection.insertMany(docs);
const result = await batchResetMeiliFlags(testCollection);
expect(result).toBe(25);
});
it('should use default batch size of 1000 when env variable is not set', async () => {
// Create exactly 1000 documents to verify default batch behavior
const docs = [];
for (let i = 0; i < 1000; i++) {
docs.push({
_id: new mongoose.Types.ObjectId(),
expiredAt: null,
_meiliIndex: true,
});
}
await testCollection.insertMany(docs);
const result = await batchResetMeiliFlags(testCollection);
expect(result).toBe(1000);
});
});
describe('return value', () => {
it('should return correct modified count', async () => {
await testCollection.insertMany([
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: true },
]);
await expect(batchResetMeiliFlags(testCollection)).resolves.toBe(1);
});
});
describe('batch delay', () => {
it('should respect MEILI_SYNC_DELAY_MS between batches', async () => {
process.env.MEILI_SYNC_BATCH_SIZE = '2';
process.env.MEILI_SYNC_DELAY_MS = '50';
const docs = [];
for (let i = 0; i < 5; i++) {
docs.push({
_id: new mongoose.Types.ObjectId(),
expiredAt: null,
_meiliIndex: true,
});
}
await testCollection.insertMany(docs);
const startTime = Date.now();
await batchResetMeiliFlags(testCollection);
const endTime = Date.now();
// With 5 documents and batch size 2, we need 3 batches
// That means 2 delays between batches (not after the last one)
// So minimum time should be around 100ms (2 * 50ms)
// Using a slightly lower threshold to account for timing variations
const elapsed = endTime - startTime;
expect(elapsed).toBeGreaterThanOrEqual(80);
});
it('should not delay when MEILI_SYNC_DELAY_MS is 0', async () => {
process.env.MEILI_SYNC_BATCH_SIZE = '2';
process.env.MEILI_SYNC_DELAY_MS = '0';
const docs = [];
for (let i = 0; i < 5; i++) {
docs.push({
_id: new mongoose.Types.ObjectId(),
expiredAt: null,
_meiliIndex: true,
});
}
await testCollection.insertMany(docs);
const startTime = Date.now();
await batchResetMeiliFlags(testCollection);
const endTime = Date.now();
const elapsed = endTime - startTime;
// Should complete without intentional delays, but database operations still take time
// Just verify it completes and returns the correct count
expect(elapsed).toBeLessThan(1000); // More reasonable upper bound
const result = await testCollection.countDocuments({ _meiliIndex: false });
expect(result).toBe(5);
});
it('should not delay after the last batch', async () => {
process.env.MEILI_SYNC_BATCH_SIZE = '3';
process.env.MEILI_SYNC_DELAY_MS = '100';
// Exactly 3 documents - should fit in one batch, no delay
await testCollection.insertMany([
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: true },
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: true },
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: true },
]);
const result = await batchResetMeiliFlags(testCollection);
// Verify all 3 documents were processed in a single batch
expect(result).toBe(3);
const updatedDocs = await testCollection.countDocuments({ _meiliIndex: false });
expect(updatedDocs).toBe(3);
});
});
describe('edge cases', () => {
it('should handle documents without _meiliIndex field', async () => {
await testCollection.insertMany([
{ _id: new mongoose.Types.ObjectId(), expiredAt: null },
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: true },
]);
const result = await batchResetMeiliFlags(testCollection);
// both documents should be updated
expect(result).toBe(2);
});
it('should handle mixed document states correctly', async () => {
await testCollection.insertMany([
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: true },
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: false },
{ _id: new mongoose.Types.ObjectId(), expiredAt: new Date(), _meiliIndex: true },
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: true },
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: null },
{ _id: new mongoose.Types.ObjectId(), expiredAt: null },
]);
const result = await batchResetMeiliFlags(testCollection);
expect(result).toBe(4);
const flaggedDocs = await testCollection
.find({ expiredAt: null, _meiliIndex: false })
.toArray();
expect(flaggedDocs).toHaveLength(5); // 4 were updated, 1 was already false
});
});
describe('error handling', () => {
it('should throw error with context when find operation fails', async () => {
const mockCollection = {
collectionName: 'test_meili_batch',
find: jest.fn().mockReturnValue({
limit: jest.fn().mockReturnValue({
toArray: jest.fn().mockRejectedValue(new Error('Network error')),
}),
}),
};
await expect(batchResetMeiliFlags(mockCollection)).rejects.toThrow(
"Failed to batch reset Meili flags for collection 'test_meili_batch' after processing 0 documents: Network error",
);
});
it('should throw error with context when updateMany operation fails', async () => {
const mockCollection = {
collectionName: 'test_meili_batch',
find: jest.fn().mockReturnValue({
limit: jest.fn().mockReturnValue({
toArray: jest
.fn()
.mockResolvedValue([
{ _id: new mongoose.Types.ObjectId() },
{ _id: new mongoose.Types.ObjectId() },
]),
}),
}),
updateMany: jest.fn().mockRejectedValue(new Error('Connection lost')),
};
await expect(batchResetMeiliFlags(mockCollection)).rejects.toThrow(
"Failed to batch reset Meili flags for collection 'test_meili_batch' after processing 0 documents: Connection lost",
);
});
it('should include documents processed count in error when failure occurs mid-batch', async () => {
// Set batch size to 2 to force multiple batches
process.env.MEILI_SYNC_BATCH_SIZE = '2';
process.env.MEILI_SYNC_DELAY_MS = '0'; // No delay for faster test
let findCallCount = 0;
let updateCallCount = 0;
const mockCollection = {
collectionName: 'test_meili_batch',
find: jest.fn().mockReturnValue({
limit: jest.fn().mockReturnValue({
toArray: jest.fn().mockImplementation(() => {
findCallCount++;
// Return 2 documents for first two calls (to keep loop going)
// Return 2 documents for third call (to trigger third update which will fail)
if (findCallCount <= 3) {
return Promise.resolve([
{ _id: new mongoose.Types.ObjectId() },
{ _id: new mongoose.Types.ObjectId() },
]);
}
// Should not reach here due to error
return Promise.resolve([]);
}),
}),
}),
updateMany: jest.fn().mockImplementation(() => {
updateCallCount++;
if (updateCallCount === 1) {
return Promise.resolve({ modifiedCount: 2 });
} else if (updateCallCount === 2) {
return Promise.resolve({ modifiedCount: 2 });
} else {
return Promise.reject(new Error('Database timeout'));
}
}),
};
await expect(batchResetMeiliFlags(mockCollection)).rejects.toThrow(
"Failed to batch reset Meili flags for collection 'test_meili_batch' after processing 4 documents: Database timeout",
);
});
it('should use collection.collectionName in error messages', async () => {
const mockCollection = {
collectionName: 'messages',
find: jest.fn().mockReturnValue({
limit: jest.fn().mockReturnValue({
toArray: jest.fn().mockRejectedValue(new Error('Permission denied')),
}),
}),
};
await expect(batchResetMeiliFlags(mockCollection)).rejects.toThrow(
"Failed to batch reset Meili flags for collection 'messages' after processing 0 documents: Permission denied",
);
});
});
describe('environment variable validation', () => {
let warnSpy;
beforeEach(() => {
// Mock logger.warn to track warning calls
const { logger } = require('@librechat/data-schemas');
warnSpy = jest.spyOn(logger, 'warn').mockImplementation(() => {});
});
afterEach(() => {
if (warnSpy) {
warnSpy.mockRestore();
}
});
it('should log warning and use default when MEILI_SYNC_BATCH_SIZE is not a number', async () => {
process.env.MEILI_SYNC_BATCH_SIZE = 'abc';
await testCollection.insertMany([
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: true },
]);
const result = await batchResetMeiliFlags(testCollection);
expect(result).toBe(1);
expect(warnSpy).toHaveBeenCalledWith(
expect.stringContaining('Invalid value for MEILI_SYNC_BATCH_SIZE="abc"'),
);
expect(warnSpy).toHaveBeenCalledWith(expect.stringContaining('Using default: 1000'));
});
it('should log warning and use default when MEILI_SYNC_DELAY_MS is not a number', async () => {
process.env.MEILI_SYNC_DELAY_MS = 'xyz';
await testCollection.insertMany([
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: true },
]);
const result = await batchResetMeiliFlags(testCollection);
expect(result).toBe(1);
expect(warnSpy).toHaveBeenCalledWith(
expect.stringContaining('Invalid value for MEILI_SYNC_DELAY_MS="xyz"'),
);
expect(warnSpy).toHaveBeenCalledWith(expect.stringContaining('Using default: 100'));
});
it('should log warning and use default when MEILI_SYNC_BATCH_SIZE is negative', async () => {
process.env.MEILI_SYNC_BATCH_SIZE = '-50';
await testCollection.insertMany([
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: true },
]);
const result = await batchResetMeiliFlags(testCollection);
expect(result).toBe(1);
expect(warnSpy).toHaveBeenCalledWith(
expect.stringContaining('Invalid value for MEILI_SYNC_BATCH_SIZE="-50"'),
);
});
it('should log warning and use default when MEILI_SYNC_DELAY_MS is negative', async () => {
process.env.MEILI_SYNC_DELAY_MS = '-100';
await testCollection.insertMany([
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: true },
]);
const result = await batchResetMeiliFlags(testCollection);
expect(result).toBe(1);
expect(warnSpy).toHaveBeenCalledWith(
expect.stringContaining('Invalid value for MEILI_SYNC_DELAY_MS="-100"'),
);
});
it('should accept valid positive integer values without warnings', async () => {
process.env.MEILI_SYNC_BATCH_SIZE = '500';
process.env.MEILI_SYNC_DELAY_MS = '50';
await testCollection.insertMany([
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: true },
]);
const result = await batchResetMeiliFlags(testCollection);
expect(result).toBe(1);
expect(warnSpy).not.toHaveBeenCalled();
});
it('should log warning and use default when MEILI_SYNC_BATCH_SIZE is zero', async () => {
process.env.MEILI_SYNC_BATCH_SIZE = '0';
await testCollection.insertMany([
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: true },
]);
const result = await batchResetMeiliFlags(testCollection);
expect(result).toBe(1);
expect(warnSpy).toHaveBeenCalledWith(
expect.stringContaining('MEILI_SYNC_BATCH_SIZE cannot be 0. Using default: 1000'),
);
});
it('should accept zero as a valid value for MEILI_SYNC_DELAY_MS without warnings', async () => {
process.env.MEILI_SYNC_DELAY_MS = '0';
await testCollection.insertMany([
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: true },
]);
const result = await batchResetMeiliFlags(testCollection);
expect(result).toBe(1);
expect(warnSpy).not.toHaveBeenCalled();
});
it('should not log warnings when environment variables are not set', async () => {
delete process.env.MEILI_SYNC_BATCH_SIZE;
delete process.env.MEILI_SYNC_DELAY_MS;
await testCollection.insertMany([
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: true },
]);
const result = await batchResetMeiliFlags(testCollection);
expect(result).toBe(1);
expect(warnSpy).not.toHaveBeenCalled();
});
});
});
+1 -1
View File
@@ -1,4 +1,4 @@
const mongoose = require('mongoose');
const { mongoose } = require('~/db/base');
const crypto = require('node:crypto');
const { logger } = require('@librechat/data-schemas');
const { getCustomEndpointConfig } = require('@hanzochat/api');
+1 -1
View File
@@ -1,4 +1,4 @@
const mongoose = require('mongoose');
const { mongoose } = require('~/db/base');
const { createMethods } = require('@librechat/data-schemas');
const methods = createMethods(mongoose);
const { comparePassword } = require('./userMethods');
+1 -1
View File
@@ -1,4 +1,4 @@
const mongoose = require('mongoose');
const { mongoose } = require('~/db/base');
const { logger, hashToken, getRandomValues } = require('@librechat/data-schemas');
const { createToken, findToken } = require('~/models');
+1
View File
@@ -52,6 +52,7 @@
"@node-saml/passport-saml": "^5.1.0",
"@smithy/node-http-handler": "^4.4.5",
"axios": "^1.13.5",
"@hanzo/base": "^0.2.1",
"bcryptjs": "^2.4.3",
"compression": "^1.8.1",
"connect-redis": "^8.1.0",
@@ -2,7 +2,7 @@
* @import { TUpdateResourcePermissionsRequest, TUpdateResourcePermissionsResponse } from 'librechat-data-provider'
*/
const mongoose = require('mongoose');
const { mongoose } = require('~/db/base');
const { logger } = require('@librechat/data-schemas');
const { ResourceType, PrincipalType, PermissionBits } = require('librechat-data-provider');
const { enrichRemoteAgentPrincipals, backfillRemoteAgentPermissions } = require('@hanzochat/api');
@@ -1,4 +1,4 @@
const mongoose = require('mongoose');
const { mongoose } = require('~/db/base');
const { logger } = require('@librechat/data-schemas');
const {
MAX_SKILL_STATES,
+8 -16
View File
@@ -1,5 +1,4 @@
const express = require('express');
const { MeiliSearch } = require('meilisearch');
const { isEnabled } = require('@hanzochat/api');
const requireJwtAuth = require('~/server/middleware/requireJwtAuth');
@@ -7,22 +6,15 @@ const router = express.Router();
router.use(requireJwtAuth);
/**
* Reports whether conversation/message search is available.
*
* Search now runs on Hanzo Base/SQLite (via the data-layer adapter), so it is
* always available unless explicitly disabled with `SEARCH=false`.
*/
router.get('/enable', async function (req, res) {
if (!isEnabled(process.env.SEARCH)) {
return res.send(false);
}
try {
const client = new MeiliSearch({
host: process.env.MEILI_HOST,
apiKey: process.env.MEILI_MASTER_KEY,
});
const { status } = await client.health();
return res.send(status === 'available');
} catch (error) {
return res.send(false);
}
const disabled = 'SEARCH' in process.env && !isEnabled(process.env.SEARCH);
return res.send(!disabled);
});
module.exports = router;
+1 -1
View File
@@ -1,4 +1,4 @@
const mongoose = require('mongoose');
const { mongoose } = require('~/db/base');
const { isEnabled } = require('@hanzochat/api');
const { getTransactionSupport, logger } = require('@librechat/data-schemas');
const { ResourceType, PrincipalType, PrincipalModel } = require('librechat-data-provider');
+1 -1
View File
@@ -1,4 +1,4 @@
const mongoose = require('mongoose');
const { mongoose } = require('~/db/base');
const { logger } = require('@librechat/data-schemas');
const { mergeAppTools, getAppConfig } = require('./Config');
const { createMCPServersRegistry, createMCPManager } = require('~/config');
+1 -1
View File
@@ -1,4 +1,4 @@
const mongoose = require('mongoose');
const { mongoose } = require('~/db/base');
const { logger } = require('@librechat/data-schemas');
const {
logAgentMigrationWarning,
+124 -10
View File
@@ -70,7 +70,7 @@ importers:
version: 2.32.0(@typescript-eslint/parser@8.57.1(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3))(eslint-import-resolver-typescript@3.10.1)(eslint@9.39.4(jiti@1.21.7))
eslint-plugin-jest:
specifier: ^29.1.0
version: 29.15.0(@typescript-eslint/eslint-plugin@8.57.1(@typescript-eslint/parser@8.57.1(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3))(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3))(eslint@9.39.4(jiti@1.21.7))(jest@30.3.0(@types/node@25.5.0)(ts-node@10.9.2(@types/node@25.5.0)(typescript@5.9.3)))(typescript@5.9.3)
version: 29.15.0(@typescript-eslint/eslint-plugin@8.57.1(@typescript-eslint/parser@8.57.1(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3))(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3))(eslint@9.39.4(jiti@1.21.7))(jest@30.3.0)(typescript@5.9.3)
eslint-plugin-jsx-a11y:
specifier: ^6.10.2
version: 6.10.2(eslint@9.39.4(jiti@1.21.7))
@@ -94,7 +94,7 @@ importers:
version: 9.1.7
jest:
specifier: ^30.2.0
version: 30.3.0(@types/node@25.5.0)(ts-node@10.9.2(@types/node@25.5.0)(typescript@5.9.3))
version: 30.3.0
lint-staged:
specifier: ^15.4.3
version: 15.5.2
@@ -137,6 +137,9 @@ importers:
'@google/genai':
specifier: ^1.19.0
version: 1.46.0(@modelcontextprotocol/sdk@1.22.0(@cfworker/json-schema@4.1.1))
'@hanzo/base':
specifier: ^0.2.1
version: 0.2.1(react@19.2.4)
'@hanzochat/api':
specifier: workspace:*
version: link:../packages/api
@@ -347,7 +350,7 @@ importers:
devDependencies:
jest:
specifier: ^30.2.0
version: 30.3.0(@types/node@20.19.37)(ts-node@10.9.2(@types/node@20.19.37)(typescript@5.9.3))
version: 30.3.0
mongodb-memory-server:
specifier: ^10.1.4
version: 10.4.3
@@ -693,7 +696,7 @@ importers:
version: 1.0.3
eslint-plugin-jest:
specifier: ^29.1.0
version: 29.15.0(@typescript-eslint/eslint-plugin@8.57.1(@typescript-eslint/parser@8.57.1(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3))(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3))(eslint@9.39.4(jiti@1.21.7))(jest@30.3.0(@types/node@20.19.37)(ts-node@10.9.2(@types/node@20.19.37)(typescript@5.9.3)))(typescript@5.9.3)
version: 29.15.0(@typescript-eslint/eslint-plugin@8.57.1(@typescript-eslint/parser@8.57.1(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3))(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3))(eslint@9.39.4(jiti@1.21.7))(jest@30.3.0(@types/node@20.19.37))(typescript@5.9.3)
fs-extra:
specifier: ^11.3.2
version: 11.3.4
@@ -702,7 +705,7 @@ importers:
version: 3.0.0
jest:
specifier: ^30.2.0
version: 30.3.0(@types/node@20.19.37)(ts-node@10.9.2(@types/node@20.19.37)(typescript@5.9.3))
version: 30.3.0(@types/node@20.19.37)
jest-canvas-mock:
specifier: ^2.5.2
version: 2.5.2
@@ -1215,7 +1218,7 @@ importers:
version: 2.4.4
jest:
specifier: ^30.2.0
version: 30.3.0(@types/node@20.19.37)(ts-node@10.9.2(@types/node@20.19.37)(typescript@5.9.3))
version: 30.3.0(@types/node@20.19.37)
jest-junit:
specifier: ^16.0.0
version: 16.0.0
@@ -3706,6 +3709,14 @@ packages:
engines: {node: '>=6'}
hasBin: true
'@hanzo/base@0.2.1':
resolution: {integrity: sha512-eHVQXYr5m5RIG3EqCsXCB1urvLLiGk8F/Gr6d4WWK8H5G5RTooYS5MdNxb/FCs2iE/+VszvPkWV2jWxXC3Ct+A==}
peerDependencies:
react: '>=18.0.0'
peerDependenciesMeta:
react:
optional: true
'@hanzo/iam@0.13.1':
resolution: {integrity: sha512-MAqXY7jL5pcPkaUE0Qb/ePL2WKvk3p6EQDLHGZENj+/gbNKs7e/9qG+nmRlPMFVcwXPk+suzkBAEyIMT2EiJ8w==}
engines: {node: '>=18'}
@@ -17268,6 +17279,10 @@ snapshots:
protobufjs: 7.5.4
yargs: 17.7.2
'@hanzo/base@0.2.1(react@19.2.4)':
optionalDependencies:
react: 19.2.4
'@hanzo/iam@0.13.1(react@18.3.1)':
dependencies:
jose: 6.2.2
@@ -17592,6 +17607,41 @@ snapshots:
jest-util: 30.3.0
slash: 3.0.0
'@jest/core@30.3.0':
dependencies:
'@jest/console': 30.3.0
'@jest/pattern': 30.0.1
'@jest/reporters': 30.3.0
'@jest/test-result': 30.3.0
'@jest/transform': 30.3.0
'@jest/types': 30.3.0
'@types/node': 20.19.37
ansi-escapes: 4.3.2
chalk: 4.1.2
ci-info: 4.4.0
exit-x: 0.2.2
graceful-fs: 4.2.11
jest-changed-files: 30.3.0
jest-config: 30.3.0(@types/node@20.19.37)(ts-node@10.9.2(@types/node@20.19.37)(typescript@5.9.3))
jest-haste-map: 30.3.0
jest-message-util: 30.3.0
jest-regex-util: 30.0.1
jest-resolve: 30.3.0
jest-resolve-dependencies: 30.3.0
jest-runner: 30.3.0
jest-runtime: 30.3.0
jest-snapshot: 30.3.0
jest-util: 30.3.0
jest-validate: 30.3.0
jest-watcher: 30.3.0
pretty-format: 30.3.0
slash: 3.0.0
transitivePeerDependencies:
- babel-plugin-macros
- esbuild-register
- supports-color
- ts-node
'@jest/core@30.3.0(ts-node@10.9.2(@types/node@20.19.37)(typescript@5.9.3))':
dependencies:
'@jest/console': 30.3.0
@@ -23184,24 +23234,24 @@ snapshots:
- eslint-import-resolver-webpack
- supports-color
eslint-plugin-jest@29.15.0(@typescript-eslint/eslint-plugin@8.57.1(@typescript-eslint/parser@8.57.1(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3))(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3))(eslint@9.39.4(jiti@1.21.7))(jest@30.3.0(@types/node@20.19.37)(ts-node@10.9.2(@types/node@20.19.37)(typescript@5.9.3)))(typescript@5.9.3):
eslint-plugin-jest@29.15.0(@typescript-eslint/eslint-plugin@8.57.1(@typescript-eslint/parser@8.57.1(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3))(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3))(eslint@9.39.4(jiti@1.21.7))(jest@30.3.0(@types/node@20.19.37))(typescript@5.9.3):
dependencies:
'@typescript-eslint/utils': 8.57.1(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3)
eslint: 9.39.4(jiti@1.21.7)
optionalDependencies:
'@typescript-eslint/eslint-plugin': 8.57.1(@typescript-eslint/parser@8.57.1(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3))(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3)
jest: 30.3.0(@types/node@20.19.37)(ts-node@10.9.2(@types/node@20.19.37)(typescript@5.9.3))
jest: 30.3.0(@types/node@20.19.37)
typescript: 5.9.3
transitivePeerDependencies:
- supports-color
eslint-plugin-jest@29.15.0(@typescript-eslint/eslint-plugin@8.57.1(@typescript-eslint/parser@8.57.1(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3))(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3))(eslint@9.39.4(jiti@1.21.7))(jest@30.3.0(@types/node@25.5.0)(ts-node@10.9.2(@types/node@25.5.0)(typescript@5.9.3)))(typescript@5.9.3):
eslint-plugin-jest@29.15.0(@typescript-eslint/eslint-plugin@8.57.1(@typescript-eslint/parser@8.57.1(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3))(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3))(eslint@9.39.4(jiti@1.21.7))(jest@30.3.0)(typescript@5.9.3):
dependencies:
'@typescript-eslint/utils': 8.57.1(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3)
eslint: 9.39.4(jiti@1.21.7)
optionalDependencies:
'@typescript-eslint/eslint-plugin': 8.57.1(@typescript-eslint/parser@8.57.1(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3))(eslint@9.39.4(jiti@1.21.7))(typescript@5.9.3)
jest: 30.3.0(@types/node@25.5.0)(ts-node@10.9.2(@types/node@25.5.0)(typescript@5.9.3))
jest: 30.3.0
typescript: 5.9.3
transitivePeerDependencies:
- supports-color
@@ -24609,6 +24659,44 @@ snapshots:
- babel-plugin-macros
- supports-color
jest-cli@30.3.0:
dependencies:
'@jest/core': 30.3.0
'@jest/test-result': 30.3.0
'@jest/types': 30.3.0
chalk: 4.1.2
exit-x: 0.2.2
import-local: 3.2.0
jest-config: 30.3.0(@types/node@25.5.0)(ts-node@10.9.2(@types/node@25.5.0)(typescript@5.9.3))
jest-util: 30.3.0
jest-validate: 30.3.0
yargs: 17.7.2
transitivePeerDependencies:
- '@types/node'
- babel-plugin-macros
- esbuild-register
- supports-color
- ts-node
jest-cli@30.3.0(@types/node@20.19.37):
dependencies:
'@jest/core': 30.3.0
'@jest/test-result': 30.3.0
'@jest/types': 30.3.0
chalk: 4.1.2
exit-x: 0.2.2
import-local: 3.2.0
jest-config: 30.3.0(@types/node@20.19.37)(ts-node@10.9.2(@types/node@20.19.37)(typescript@5.9.3))
jest-util: 30.3.0
jest-validate: 30.3.0
yargs: 17.7.2
transitivePeerDependencies:
- '@types/node'
- babel-plugin-macros
- esbuild-register
- supports-color
- ts-node
jest-cli@30.3.0(@types/node@20.19.37)(ts-node@10.9.2(@types/node@20.19.37)(typescript@5.9.3)):
dependencies:
'@jest/core': 30.3.0(ts-node@10.9.2(@types/node@20.19.37)(typescript@5.9.3))
@@ -25037,6 +25125,32 @@ snapshots:
merge-stream: 2.0.0
supports-color: 8.1.1
jest@30.3.0:
dependencies:
'@jest/core': 30.3.0
'@jest/types': 30.3.0
import-local: 3.2.0
jest-cli: 30.3.0
transitivePeerDependencies:
- '@types/node'
- babel-plugin-macros
- esbuild-register
- supports-color
- ts-node
jest@30.3.0(@types/node@20.19.37):
dependencies:
'@jest/core': 30.3.0
'@jest/types': 30.3.0
import-local: 3.2.0
jest-cli: 30.3.0(@types/node@20.19.37)
transitivePeerDependencies:
- '@types/node'
- babel-plugin-macros
- esbuild-register
- supports-color
- ts-node
jest@30.3.0(@types/node@20.19.37)(ts-node@10.9.2(@types/node@20.19.37)(typescript@5.9.3)):
dependencies:
'@jest/core': 30.3.0(ts-node@10.9.2(@types/node@20.19.37)(typescript@5.9.3))