The build-manifest sidecar scanned only image/video/marketing extensions, so a
.glb was dropped on every rebuild — the ingested mesh vanished from the library
within a sweep cycle (same class as the earlier video/ingest drops). Add
MODEL3D_EXTS to the scan and the kind default.
The retry-cap + dispatch-dedup patches stop the two known triggers; this makes the
whole class impossible at every layer, independent of any upstream bug:
COMPUTE — don't render a duplicate:
* retry cap (maximumAttempts=1): a failed render never re-runs itself.
* dispatch dedup: identical (org, prefix, instruction, refs) submits within 60s
collapse to the in-flight render.
QUEUE — nothing lingers to replay:
* scheduleToStartTimeout=1800s: a job no worker picks up in 30 min EXPIRES, so a
worker outage can't leave a backlog that silently re-bills when a worker returns.
STORAGE — the gallery is CONTENT-ADDRESSED (the last line of defense):
* library_manifest (the one indexer that builds every library.json, sweeping each
org continuously) now enforces AT MOST ONE active asset per content hash. Any
byte-identical duplicate is auto-soft-deleted, keeper = a curated asset if one
exists else the earliest. Idempotent, recoverable, and O(new-files) per sweep
(sha reused when mtime is stable). So even 873 identical re-renders can NEVER
accumulate in the gallery — the invariant heals it within one sweep.
Verified: 3 identical + 1 distinct -> 2 active (one-per-hash), distinct untouched,
stable across sweeps.
LIBRARY.JSON CORRUPTION (Red V3, the curation-wipe mechanism): the build-manifest
sidecar wrote a FIXED '.library.json.tmp', shared by every pod's sidecar — two
overlapping during a roll interleaved into one inode → torn JSON → load_prior()/
reader hit JSONDecodeError → {} → every asset reset to status=draft, which mirror
then pushed to S3 and restore rehydrated. Fixed both writers: mkstemp gives each a
private tmp (kills the tear), and a shared flock (.library.lock, taken by the
sidecar across build+write and by the app's _save_library across its write)
serializes the two rebuild-then-replace paths so neither clobbers the other.
/READY LIVENESS (Red V1): the readiness probe checked ,
set once at boot and never cleared — constant-true, saying nothing about the render
path. A dead worker thread kept the pod in-rotation accepting jobs that never ran,
and a roll would promote a wedged pod. /ready now gates on the prompt_worker daemon
thread actually being alive.
Tests: unique-tmp, no fixed-tmp residue, concurrent-writers-never-tear, lock file.
226 pass.
The everything-indexes sweep pulled .thumbs/ cache files in as rows; fixing
one sent a thumb path into the dispatcher (unknown asset). One rule, both
writers: any path segment starting with a dot is excluded — forks and cache
trees alike.
Co-authored-by: Hanzo AI <ai@hanzo.ai>
The sweeper rebuilt library.json and DROPPED every image classify() could
not parse — erasing ingest-indexed renders on each cycle (rows appeared,
then vanished minutes later). Unclassifiable images now keep a minimal row,
prior fields preserved.
Co-authored-by: Hanzo AI <ai@hanzo.ai>
The indexer skips them; _safe_asset refuses them (one gate for file serving,
fix sources, and every other path lookup). AppleDouble forks carry image
extensions, serve resource-fork bytes, and rendered 0x0 across the library.
Co-authored-by: Hanzo AI <ai@hanzo.ai>
BertModelWarper now delegates to the real BertModel.forward instead of
reimplementing internals whose private signatures drift between releases;
checkpoint weights remap bert.* -> bert.bert_model.* at load. The installer
force-checkouts each pin (discarding prior patches) then applies any
scripts/patches/<pack>-*.patch, so re-runs stay idempotent and the image
build gets the same fix the GB10 venv had by hand.
karma_manifest.py regenerates the ONE canonical library.json index of the
karma org's rendered assets (storefront + socials queue both read it).
studio_watchdog.sh keeps the local render backend on :8188 alive unattended;
paths derive from the script location (STUDIO/STUDIO_PY overridable).
Add a comfy->studio import shim so upstream ComfyUI custom nodes run
unmodified against this fork's renamed core, then vendor the popular packs
(Impact-Pack/Subpack, IPAdapter_plus, controlnet_aux, Ultimate SD Upscale,
Frame-Interpolation, VideoHelperSuite, rgthree, Custom-Scripts, KJNodes,
cg-use-everywhere, SAM2, segment_anything) pinned to immutable commits and
reproducibly rebuilt in the image.
- studio_compat: one meta-path finder aliasing comfy*->studio* and mirroring
Comfy*->Studio* API classes (the deferred phase-4 of scripts/rename_modules.py,
done with the 3.12 find_spec API); activated by one import at top of main.py.
- nodes.py: honor the upstream `comfy_entrypoint` V3 extension name too.
- custom_nodes/hanzo-packs.txt: pinned pack manifest (SHAs + SPDX licenses).
- custom_nodes-requirements.txt + scripts/install_custom_nodes.sh: pack deps
with the cu130 torch / NumPy 2.x / Transformers 5.x stack locked (derived
from the live env, never downgraded); ultralytics installed --no-deps.
- Dockerfile/.dockerignore/.gitignore: image vendors the packs at build.
Live GB10 worker verified: node types 644 -> 1290 (+646), 0 import failures.
was-node-suite quarantined (hard numba dep incompatible with NumPy 2.5).
Root cause of the "Event loop stopped before Future completed" crash that
wiped the in-memory queue/history (seen 3x today): the fork's _graceful_shutdown
handler (added in a45a441) called event_loop.stop(). The server coroutine passed
to run_until_complete() never completes on its own, so stopping the loop makes
run_until_complete() raise RuntimeError; the surrounding try only caught
KeyboardInterrupt, so the process died non-zero with a traceback and discarded
the queue. The dirty exit also failed to release :8188 promptly, and a duplicate
transient/user `hanzo-studio.service` racing the canonical studio.service turned
that into an EADDRINUSE restart storm.
main.py:
- handler is now idempotent, flags the queue closed, snapshots it, then stops
the loop; the outer try treats RuntimeError('Event loop stopped before Future
completed.') as the expected graceful-exit path (logs "Server stopped by
shutdown signal", exits 0). Verified live in journald.
execution.py:
- opt-in crash-durable PromptQueue behind STUDIO_PERSIST_QUEUE=1. Pending +
in-flight prompts are snapshotted atomically to user/queue_snapshot.json on
every change and re-queued on boot. Best-effort: all I/O is guarded so it
never affects the render path; inert when the flag is unset.
ops/systemd/studio.service:
- enable STUDIO_PERSIST_QUEUE=1 on the single canonical unit.
scripts/studio-service-swap.sh:
- batch-safe reconcile+apply: waits for an empty queue, removes any stray
duplicate unit, reinstalls the canonical unit, restarts, health-checks.
docs/ops-crash-and-service.md:
- topology (one canonical unit), how to read crash history, the fix, persistence.
tests-unit/execution_test/prompt_queue_persist_test.py:
- round-trip, inert-when-disabled, and corrupt-row-skipping tests.
Hanzo AI