Make EVERY studio render flow through the org's visible gpu-jobs queue and
eliminate the direct-to-127.0.0.1:8188 bypass. Studio-side only (cloud CLI /
worker two-lane claim owned by the cloud agent).
- gpu_dispatch: per-GPU targeting. X-Target-GPU -> taskQueue "gpu:<identity>"
(namespace stays gpu-jobs); an explicit target always enqueues. Untargeted =
shared "gpu-jobs" lane.
- server.post_prompt: in --worker-mode refuse a direct /prompt POST without
X-Worker-Token (403) via worker_client.reject_untrusted_worker_submit (one
policy gate). The only accepted execution is a claimed job over
/v1/worker/execute. Legacy prompt_router push is OFF unless
STUDIO_LEGACY_PUSH_ROUTER=1 -> gpu-jobs enqueue is the ONE submit path.
- worker_client: /v1/worker/execute validates X-Worker-Token, runs the graph on
the local prompt_queue, scopes outputs to the job's org, returns {prompt_id}.
- studio_home: /v1/render-queue + /v1/nodes are AUTHORITATIVE — read the cloud
tasks engine's gpu-jobs activities (real claiming identity, status ->
queued/running/done/failed, lastHeartbeatTime, per-node depth). render_jobs.json
is only a ~2s pre-claim placeholder. X-Target-GPU forwarded through _queue_prompt.
- UI: "Run on <gpu>" chips (studio_home.html/shell.js) set X-Target-GPU; in-flight
rows show the REAL worker + queued/running state.
Tests: gpu_dispatch_test (targeting), worker_seam_test (gate 403 + claim seam).
378 passed. Follow-up (cloud agent): content/studio_render.go:86 convergence.
Two flywheel surfaces the gallery already had but the asset views did not.
1) Full-image (zoom) rating bar. Opening any asset full-size now shows the same
👍/👎/💬 controls as the cards, bound to the asset in view (ZPATH threaded
through every zoom entry — grid, queue list, stack view, openLibItem). 👍/👎
toggle + persist via the existing castVote; 💬 opens the existing critique
chat. Note-less; reuses the proven .vote classes. Verified on desktop + mobile
(Playwright): centred pill, correct up/down highlight toggle, critique dialog
opens, 44px tap targets, safe-area bottom inset, Comment label no-wrap.
2) Curation IS training signal. Archiving or flagging an output (the ONE
/v1/library/status route every archive path funnels through — queue Archive,
gallery delete, stack delete) now records it as a NEGATIVE example: vote -1 in
ratings.json + an event in feedback.jsonl carrying the deleted/flagged status
(negative-only downstream). Restoring clears that auto-negative. The downvote
is NOTE-LESS so it trains the reward dataset without polluting prompt lessons
(_mistakes gates on note text). A judgement the user typed is user-owned —
never overwritten or cleared by the auto-negative. Automatic dedup writes the
library directly (not via the route), so byte-identical twins are never
mislabelled bad. The queue button is now an explicit "🗑 Archive" and says it
teaches the model.
Decomplect: one atomic ratings writer (_rate_write, unique-temp — kills the
shared-name clobber the fixed ratings.json.tmp had, same hazard fixed for
library.json) shared by the rate route and the archive hook. Tests: archive →
vote -1 + feedback event + status=deleted + no injected lesson; restore clears
it; a user's downvote+note survives archive/restore and stays a lesson. 268
studio_home + middleware tests pass.
Trace of the whole Edit lifecycle (click -> /v1/library/fix -> dispatch_fix ->
_fix_graph -> gpu_dispatch -> worker LoadImage) confirmed the selected asset is
bound explicitly at every hop and the low-denoise Edit graph is guarded by
_assert_edit_graph (no EmptyLatentImage / text-to-image fallback). The one
residual state-isolation hole was the staged source FILENAME.
Every lane (fix/video/model3d/compose/amend) staged its source under
f"{kind}_src_{abs(hash((str(p), mtime))) % 10**10}.png"
— Python's process-salted hash, duplicated in 5 places. Two concurrent edits of
DIFFERENT sources whose names collided on a shared worker input dir would
overwrite each other's LoadImage file: the "Edit produced an unrelated picture /
mixed a different image" class of bug.
Decomplect to ONE helper, _stage_source(): the staged name is the sha256 of the
source bytes. The name IS a verifiable checksum of the exact selected image, so
the bytes the worker loads can never drift to another asset, and different
sources are cryptographically un-collidable (same source -> one idempotent
file). 5 call sites -> 1; the 3D downsize rides an optional max_px.
Observability the report asked for: a _trace() line correlated by job id at
build -> queue -> gpu.publish (studio.home + studio.gpu_dispatch). Logs ids,
content-address checksums, counts, lane/node ONLY — never signed URLs, tokens,
cookies or image bytes.
Tests (pure/fs, no torch/GPU/net): content-addressing (same bytes->same name,
one byte flips it, name==sha256); two concurrent edits share no staged file /
LoadImage ref / positive prompt / latent / output prefix; _collect_input_images
ships the selected org's exact bytes and is org-scoped; _assert_edit_graph
rejects a text-to-image graph. 188 middleware tests pass.
The CI 'Test (per hanzo.yml)' step failed because tests-unit/middleware_test/
studio_home_fix_test.py still asserted the OLD fix behavior (denoise 1.0, the
'same model, pose, framing' prompt, and the removed size=→EmptySD3LatentImage repose
branch). Updated to the new invariants: denoise == EDIT_DENOISE_DEFAULT (0.25), the
new preservation prompt, and — replacing the repose-Empty test — a
test_fix_graph_never_uses_empty_latent that proves Edit NEVER falls back to an empty
latent for any instruction (repose is now Regenerate, not Edit). Full CI set: 375 pass.
ROOT CAUSE: 'Fix' on a generated gallery image ran the Qwen edit at denoise 1.0 with a
RANDOM seed (full re-paint → color/composition/subject drift even for 'brighten very
slightly'), and a broad _REPOSE keyword regex silently swapped in EmptySD3LatentImage
(a literal text-to-image regenerate) on words like 'standing'/'looking'/'side view'.
THREE DISTINCT ACTIONS now, no ambiguity:
- Regenerate = Re-run (/v1/library/rerun: re-queue the recipe, new seed) — the only
path that may use EmptyLatentImage + denoise 1.0.
- Edit = low-denoise Qwen edit of the SOURCE: LoadImage → FluxKontextImageScale →
VAEEncode → KSampler(latent = the source, denoise from an edit-strength slider,
default 0.25) → VAEDecode → Save. Reuses the source's ORIGINAL seed (read from the
embedded graph via _provenance) so a locked seed is deterministic; a varied seed
still edits the source. NO _REPOSE→Empty branch. Full-res source (never the thumb).
- Edit selected area = masked inpaint: SetLatentNoiseMask restricts sampling to the
painted region + ImageCompositeMasked keeps pixels OUTSIDE the mask byte-identical.
INVARIANT (impossible-by-construction): _assert_edit_graph refuses any edit whose
KSampler.latent_image is an EmptyLatentImage or whose denoise leaves the edit band
[0.05,0.95] — an Edit can NEVER silently become a Regenerate; a bad edit surfaces an
inline error, never a fallback to random generation.
Frontend: Fix dialog → 'Edit image' with an edit-strength slider (10–95% → denoise,
default 25%) + a paint-the-area mask canvas (Edit area) + clear labels (gallery card:
Regenerate · Versions · Edit). sendFix passes {strength, seed?, mask?}. Output linked
as a child (parents=[source]); color-match post-process (v0.17.9) still runs.
Tests: 6 new edit-invariant tests (VAEEncode-not-Empty, denoise 0.25, regenerate
rejected, seed determinism, prompt-preservation-not-expansion, masked composite);
existing fix test updated 1.0→0.25. 42 pass. Model-compatible with the Qwen edit model
already on spark (kept per the user's choice); I did NOT submit any live render to
verify visuals (per the no-autonomous-image-creation rule) — the graph structure is
unit-proven and the user validates the visual output.
You can now delete any queue entry (a cancelled probe, a failed attempt, clutter) —
a 🗑 on every non-active queue row + a Delete button in the item detail dialog.
worklog.remove(org, ids) + POST /v1/worklog/delete drop ONLY the log rows; output
images live in the library and are archived/deleted there (unchanged). Optimistic +
idempotent. Pairs with the earlier Queue archive (output→Archive) and the retry-cap +
dispatch-dedup that stopped the workflow from re-running/duplicating a render in the
first place. Verified: remove by id + bulk + idempotent-missing.
The Assets gallery (and every design filter) now shows only status=published — a
curated view. Nothing is deleted: draft + approved items stay in the library,
reachable via the new ✎ Drafts chip (count shown), where each keeps its
Approve/Publish/Fix tools so you can publish it into the gallery. Empty published
view shows a 'open Drafts to publish' hint instead of looking like work vanished.
Filter predicate verified: all→published-only, design→published-of-design,
drafts→all unpublished (excl. deleted/flagged).
The WebP-thumbnail change served the .webp via FileResponse WITHOUT an explicit
Content-Type. aiohttp guesses from the extension via Python's mimetypes, which
does not know .webp, so every grid thumbnail came back application/octet-stream —
which the browser refuses to render as an <img>, so the whole gallery went blank.
The full-image path was unaffected (it sets Content-Type explicitly). Set
image/webp on the thumb response (and image/jpeg on the video poster for
consistency). Regression test pins it.
Root-caused 'can't see any assets': it's org resolution, not lost data. The active
org lives in the studio_active_org cookie; a fresh re-login clears it, dropping the
session to the token's home org — and if that resolves to 'default' (doesn't exist)
or an empty org (maxpower=0), the gallery is blank. The user's work is intact in
karma (416 visible) + hanzo (130). Verified the render path is sound at full scale.
Fix the confusion at the source: the empty gallery now NAMES the current workspace
and points at the org switcher (account pill, top-right) — so landing on the wrong
org reads as 'switch workspace', never 'my assets disappeared'. Also distinguishes a
no-search-match empty from a truly-empty workspace.
Verified: empty maxpower org shows 'No assets in maxpower yet — switch from the
account menu'.
Investigating 'can't see assets': verified the gallery renders all 416 karma cards at
full scale (1300 assets) with thumbnails painting on scroll, so the render path is
sound. Two real fragilities hardened + one cache trap removed:
- Thumb cache was 'immutable, max-age=1yr' (v0.17.17). Since the ?v=sha URL already
busts on change, immutable bought nothing but risked locking a single bad thumb
response in the browser for a YEAR (un-clearable without hard-refresh). → long but
REVALIDATABLE (max-age=1wk, stale-while-revalidate), so a bad response self-heals.
- runs() did (await fetch(x)).json().catch() — catches the json reject but NOT a
fetch reject, so a failed ratings/favorites fetch threw before rgrid rendered →
blank queue. Now each non-critical fetch is fully guarded.
- load() now shows 'Couldn't load your library — retry' on a transient /v1/library
failure instead of a silent blank gallery.
Happy path verified unchanged (416 cards, queue renders).
`ruff check .` (the Python Linting gate) had been red on main since the
3D-lane commit: E702 semicolon-joined statements in
studio_home_model3d_test.py and an unused `root` local (F841) in
studio_home_video_test.py. Split the statements onto their own lines and
drop the dead assignment. No behavior change; both suites stay green.
The IAM auth gate refused any unauthenticated request whose Accept lacked
`text/html` with a 401, so a plain browser/curl navigation to a protected
page (e.g. /studio) hit a dead 401 instead of the login flow.
Invert the rule and decomplect it: a top-level browser navigation is now
redirected (302) to the one server-owned login entry point,
`/login?next=<path>` — which starts the OIDC code flow and returns the user
where they were. Only a request that self-identifies as API/XHR/JSON
(Sec-Fetch-Mode != navigate, X-Requested-With, or Accept: application/json
without HTML) still gets 401, so the SPA's own fetch calls handle re-auth
instead of following a redirect into HTML. An ambiguous client (plain curl,
Accept: */*) counts as a navigation and is redirected — a human never sees a
raw 401. handle_login honors a sanitized ?next (open-redirect/loop-safe).
Bump 0.17.18.
/v1/gpu-status returns online reliably (verified live), but gpuOnline() cached
BOTH true and false for 15s. When a worker's heartbeat blipped for a few seconds
(spark flapped tonight on a local-network issue), the banner cached 'no GPU' and
held it — showing 'connect a GPU to generate' even with GPUs 3/3 in the chip.
Now only a POSITIVE is cached; a negative re-checks every call, so the banner
self-heals on the next 5s livebar tick.
Four wins across gallery + queue + stacks:
1. LAZY: cards no longer eager-load every thumbnail (a 1000-image gallery fired 1000
requests up front). Each .im carries its URL as data-bg; an IntersectionObserver
(rootMargin 600px) paints it only as the card nears the viewport → a screenful of
requests, not N. Falls back to eager where IO is unavailable.
2. WEBP: thumbnails are now WebP q80 instead of JPEG q82 — measured 45% smaller.
3. QUEUE was loading FULL-RES outputs (vURL) as row backgrounds — now uses the same
downsized thumbnail as the gallery (huge cut on the History tab).
4. IMMUTABLE CACHE: thumbURL is versioned by content sha (?v=, from the indexer) or
mtime, so a thumb is served Cache-Control: immutable, max-age=1yr — instant on
repeat loads; a re-render gets a new sha → new URL → never stale.
Verified: lazy observer paints in-view cards + defers off-screen; WebP 23KB vs JPEG
41KB on a sample; ?v= present on thumb URLs.
On Queue & History, per the ask:
- Click a photo → fullscreen (already opened zoom, now with navigation). Generalized
the lightbox: zoomList(list,i) carries the queue's outputs so the on-screen ‹ › AND
the keyboard ← / → step through them (wraps both ways); Esc closes. Stack view still
uses svStep — zStep() routes to whichever context is open.
- 👍/👎 + 💬 on each output — the SAME rate endpoint + feedback.jsonl training log as
the gallery, keyed by the output's library path (a queue vote == a gallery vote).
- 🗑 Reject → Archive — soft-deletes the output (⌘Z-undoable), dims the row + marks it
✓ Rejected. Reuses the one status/undo path; no new lifecycle.
Each queue output resolves to its library asset (path = subfolder/filename), so it
reuses the gallery's vote/archive machinery — one way to do everything, no duplication.
Verified headless: lightbox a→b→c + wrap via ←/→ and the arrows; 👍 logs training
feedback (stats up=1); reject archives + ⌘Z toast.
The build-manifest sidecar scanned only image/video/marketing extensions, so a
.glb was dropped on every rebuild — the ingested mesh vanished from the library
within a sweep cycle (same class as the earlier video/ingest drops). Add
MODEL3D_EXTS to the scan and the kind default.
A 4k catalog master (~20MB) inlined to the gpu-jobs enqueue overran the 15s
urlopen timeout, so dispatch_if_worker fell back and the model-less pod returned
'GPU worker momentarily unavailable' on a perfectly good render — for EVERY lane,
not just 3D. Two fixes: the 3D lane downscales its staged reference to 1536px
(Hunyuan3D works at ~518px internally, so the mesh is identical and the body is
small), and the enqueue upload timeout goes 15s→90s so large fix/compose/video
references actually finish uploading.
The queue/history detail showed the render as a 60px thumbnail — you open a job
to SEE the result. Now the result is a large hero: an image (zoom on click), a
video (inline player), or an interactive rotatable <model-viewer> for a .glb.
References move below as a labelled small row. On phone every dialog becomes a
full-width bottom sheet (safe-area padding, scrollable, stretched action
buttons), the hero caps at 48vh so it fits, and queue-row thumbs grew 60→74px.
The retry-cap + dispatch-dedup patches stop the two known triggers; this makes the
whole class impossible at every layer, independent of any upstream bug:
COMPUTE — don't render a duplicate:
* retry cap (maximumAttempts=1): a failed render never re-runs itself.
* dispatch dedup: identical (org, prefix, instruction, refs) submits within 60s
collapse to the in-flight render.
QUEUE — nothing lingers to replay:
* scheduleToStartTimeout=1800s: a job no worker picks up in 30 min EXPIRES, so a
worker outage can't leave a backlog that silently re-bills when a worker returns.
STORAGE — the gallery is CONTENT-ADDRESSED (the last line of defense):
* library_manifest (the one indexer that builds every library.json, sweeping each
org continuously) now enforces AT MOST ONE active asset per content hash. Any
byte-identical duplicate is auto-soft-deleted, keeper = a curated asset if one
exists else the earliest. Idempotent, recoverable, and O(new-files) per sweep
(sha reused when mtime is stable). So even 873 identical re-renders can NEVER
accumulate in the gallery — the invariant heals it within one sweep.
Verified: 3 identical + 1 distinct -> 2 active (one-per-hash), distinct untouched,
stable across sweeps.
Third suite content type. Proven node-for-node on spark from ComfyUI's own
3d_hunyuan3d-v2.1 template: ImageOnlyCheckpointLoader (the ONE bundled
model+CLIP-vision+VAE — separate loaders produced noise) → CLIPVisionEncode →
Hunyuan3Dv2Conditioning → ModelSamplingAuraFlow(shift 1, the flow-matching
wrapper WITHOUT which the sampler yields fragments) → KSampler → VAEDecodeHunyuan3D
→ VoxelToMesh(surface net) → SaveGLB. Verified: a clean 3D garment mesh from a
ghost-mannequin shot (341k faces, coherent).
Lane contract, same as video: _model3d_graph + POST /v1/library/model3d (ONE
reference image required, no text-to-3D, unique job tag); .glb ingest (glTF
magic bytes, model/gltf-binary CTYPE, kind=model3d); an interactive <model-viewer>
zoom card (grid shows a 🧊 3D badge, click to rotate); the 'Image → 3D' template.
No new infra — one library, one queue, one wallet, one GPU fleet.
Tests: proven graph, dispatch route (org-from-token, unique tag, one required
ref), .glb ingest. All studio suites green.
Adversarial review (contract lens) blocked on a real bug: the composer KIND flag
stayed 'video' after a video dispatch, so the next single-photo generate silently
POSTed /v1/library/video (image-to-video) instead of /v1/library/fix — a wrong,
billable GPU dispatch. clearLane() now resets after every video dispatch, and the
Template chip is clickable to return to the neutral edit/compose lane.
DRY (identity lens): the SaveImage/SaveVideo/SaveWEBM triple was hardcoded three
times. Consolidated the two Python copies into gpu_dispatch.SAVE_NODES (the base
module, no cycle) imported by studio_home; the HTML keeps its own JS-runtime copy.
244 pass; XSS esc() + job-identity _retag + library.json lock all intact post-rebase.
LIBRARY.JSON CORRUPTION (Red V3, the curation-wipe mechanism): the build-manifest
sidecar wrote a FIXED '.library.json.tmp', shared by every pod's sidecar — two
overlapping during a roll interleaved into one inode → torn JSON → load_prior()/
reader hit JSONDecodeError → {} → every asset reset to status=draft, which mirror
then pushed to S3 and restore rehydrated. Fixed both writers: mkstemp gives each a
private tmp (kills the tear), and a shared flock (.library.lock, taken by the
sidecar across build+write and by the app's _save_library across its write)
serializes the two rebuild-then-replace paths so neither clobbers the other.
/READY LIVENESS (Red V1): the readiness probe checked ,
set once at boot and never cleared — constant-true, saying nothing about the render
path. A dead worker thread kept the pod in-rotation accepting jobs that never ran,
and a roll would promote a wedged pod. /ready now gates on the prompt_worker daemon
thread actually being alive.
Tests: unique-tmp, no fixed-tmp residue, concurrent-writers-never-tear, lock file.
226 pass.
Two root causes of repeated/duplicate GPU renders (money + gallery clutter):
1. NO retry cap: the render activity had heartbeat/startToClose timeouts but no
retryPolicy, so the tasks framework re-ran a failed render (e.g. spark OOM mid-
sample) — the same reject re-rendered. Add retryPolicy maximumAttempts=1: one
attempt, a failure surfaces to the user, no auto re-burn.
2. NO dispatch idempotency: a double-fired/unguarded submit queued a SECOND render
of the same thing (logs showed one output dispatched 4x in a second). _dispatch
is now idempotent — an identical (org, output-prefix, instruction, refs) dispatch
within a 60s window returns the in-flight prompt_id instead of billing a new job.
A genuinely different instruction on the same source still dispatches.
Server-side, so it covers every submit path (composer, fix dialog, compose, chat).
Verified: 4 identical submits → 1 render; different instruction → allowed; retry
cap present.
Red-team, same phantom-image class the user hit: re-run replayed the source
graph's own SaveImage prefix, so its file landed in the SOURCE namespace and N
re-runs shared one prefix — the library then showed the newest sibling for every
row (a job displaying a photo it never made). _retag() now points every SaveImage
at a unique per-dispatch prefix for re-run and template-render, the guarantee
fix/compose/amend already had.
Composer create() (↑ button / Enter) had no double-submit guard — a fast double
fired two billable dispatches. Now wrapped in the same enqueue()/QBUSY latch as
the dialog buttons. 223 pass.
CRITIQUE CHAT (the ask: chat about a generated photo, to make it better and
train it): the 💬 on any render now opens a conversation ABOUT that image.
/v1/library/critique loads the actual photo so the vision model's reply is
grounded in what was rendered, and EVERY user turn is recorded as the note the
flywheel reads (_mistakes → _lessons), so critiquing here conditions the next
generation of that design. The old sidebar copilot stays graph-only; this is
the image-scoped path. Thread persists per image (reopen to continue).
ARCHIVE FROM QUEUE: done rows in Queue & History get a 🗄 button — archive the
render straight from history (soft, ⌘Z to undo), no need to find it in the grid.
STORED XSS (red-team, launch-blocking): a hostile upload filename like
'<img onerror=...>.png' rendered raw into gallery card HTML and executed for any
org member. Fixed at the boundary (upload name collapsed to [A-Za-z0-9._-]) and
in depth (esc() on the filename text in render(); template Render onclick now
carries only the safe slug, name looked up client-side). svCard already esc'd —
render() and the template card were the gaps.
Tests: critique thread persistence + lesson recording + tenancy + guards;
upload-name sanitization. 222 pass.
zen-vl became a reasoning model: max_tokens 8 was spent entirely on
reasoning_content, content came back null, the strict parse read nothing and
the gate failed open on EVERY render — flagged reached zero while corrupted
outputs sat in the grid, and the failure logged at an invisible level.
_reply_text now falls back to reasoning_content when content is empty;
max_tokens 768; the verdict is the LAST whole PASS/FAIL token (deliberation
mentions both — position is the conclusion); judge failures log at warning.
The Advanced-mode node editor is served from the upstream comfyui-frontend package,
whose index still shipped <title>ComfyUI</title> + 'Loading ComfyUI...' — the only
user-facing surface still showing the upstream name (the home + web/dist SPA are
already Hanzo Studio, and the running image is our fork ghcr.io/hanzoai/studio).
editor_index() already injects our shell into that index; add two idempotent swaps
there so the Editor tab title + loading status read 'Hanzo Studio'. The ComfyUI-format
compatibility stays internal, per rule — the name is out of the UI. (Deeper editor
menu/i18n strings live in the minified frontend bundle; a full frontend rebrand is a
separate build-side change.)
Usage tracking: Umami pageviews (analytics.hanzo.ai, pre-existing site id) and
six product events (generate, fix, compose, stack_create, rate, template_use)
via a zero-dependency sendBeacon to insights.hanzo.ai/v1/e — the /v1 prefix is
load-bearing (root /e/ is IAM-gated) and the key is the project's public write
token. Telemetry fails silent.
UX: body clears the floating shell dock (content was hidden under it at page
bottom on phone); stack name input flexes instead of truncating.
Tests: gpu_dispatch tests updated to the has_models_for(prompt) seam
(per-graph model check renamed the old has_local_models).
Reported again on compose (base + ref). Root cause the v0.17.5 guard missed: the
/prompt intercept fell back to LOCAL validate_prompt whenever dispatch didn't happen
AND has_local_models() was True — but has_local_models() is 'ANY model present', and
the coordinator pod carries a stray SD1.5 checkpoint. So a Qwen fix/compose graph the
pod CANNOT load (no qwen-image-edit unet) still validated locally → 'unet not in []'
→ 'Prompt outputs failed validation'. The stray checkpoint defeated the model-less
invariant.
Fix: replace has_local_models() with has_models_for(prompt) — a per-GRAPH check of
the UNET/checkpoint/CLIP/VAE loader inputs against folder_paths. A box renders locally
only a graph it can actually load; every other graph dispatches to the fleet (or a
retryable 503), regardless of unrelated models on disk. Used in BOTH the dispatch
decision (gpu_dispatch.dispatch_if_worker) and the intercept fallback (server.post_prompt).
Verified: stray-SD1.5 pod → False (enqueue, not local); real Qwen box → True (renders
locally, dev/worker unaffected); model-less pod → False.
Reported: adding a photo to the top 'What should we create?' box opened a SECOND
box at the bottom; only that bottom box worked; typing in the top box + clicking ↑
did nothing.
Root cause: the top ↑ (create()) was a STUB that only toast'd a HIP-0506 placeholder
and never generated, while the '+' funnelled photos into a separate fixed bottom
'Next generation' bar (#reftray/genFromTray) that was the only real generate path.
Two composers, only the second wired.
Fix — collapse to ONE composer (the top box):
- create() now really generates from the top prompt + the photos attached to it:
1 photo → edit (/v1/library/fix), 2-5 → compose — the same proven graphs the
gallery uses. Text-only (no photo) gives a clear 'attach a photo with +' hint.
- attached photos render INSIDE the top composer (a thumbnail strip with × to
remove), fed by the same TRAY (+ / drag / paste / a gallery card's +).
- the bottom #reftray bar + genFromTray are removed (dead once create() is real).
- Enter submits (Shift+Enter = newline).
Verified headless: attach → thumb in the top box, NO bottom bar; type + ↑ → POST
/v1/library/fix {upload, instruction} → Queued.
Scaling the reference against a differently-sized empty latent re-created the
size-mismatch artifact class (oversaturated drift + reinvented garment,
observed live). Raw ref + AuraFlow-only matches the graphs that delivered the
velvet-front and mesh-back renders. Known limit, verified across two families
and three prompts: a true 90-degree profile from a front-facing photo
reference is beyond the current model — back views and three-quarter turns
render correctly.
Rating an output (👍/👎/💬 on every card) already logs to the per-org training
dataset. Now the retrieval half: at fix/compose/amend dispatch, the mistakes the
user noted on the base and its lineage (👎 or ≤2★ notes, and bare notes) are
injected into BOTH prompt encoders — an avoid clause in the positive, the texts
in the negative — so the very next render on that design cannot repeat what was
pointed out. Praise notes are never injected. The same judgements keep
accumulating in feedback.jsonl (with curation status) for weight training.
The queue detail showed a prior (rejected) render as a queued job's result:
re-fixing the same source reused the same SaveImage prefix, and every view
prefix-globbed disk regardless of row status. Each dispatch now appends a
6-hex job tag to its prefix (fix/compose/amend), rows expose an output only
once THEY are done, and the workflow route runs the same finalize mechanism.
Also routes view/pose-change instructions (from the side, back view, turn
around…) to the free-latent edit variant: the anchored fix latent encodes
the base — deliberately structure-preserving — and its prompt tail pinned
'same pose, framing', which contradicted exactly these asks. Repose asks now
get EmptySD3LatentImage at the reference's aspect and a tail that frees the
pose while keeping model/outfit/hair/lighting.
Feedback records now carry the judged output's curation status, so a
flagged/archived output is negative-only signal downstream — never a
positive training exemplar.
There was no way to turn user judgement into a training signal — ratings.json fed
only the runtime copilot, on the runs view, as stars. This adds a clear 👍/👎 vote
+ a written note on EVERY render (gallery card + a feedback dialog), and logs each
judgement, immutable, to the org's append-only training dataset.
- 👍/👎 on each image card (one click; click again to clear) + 💬 opens a dialog
for the written critique ('what should the model do differently?').
- /v1/library/rate extended with a normalized vote (1|-1|0); every rate also
appends {ts,user,org,key,output,vote,stars,text,prov} to orgs/<org>/output/
feedback.jsonl — the flywheel the model trainer consumes. The output PNG each row
anchors to carries its own prompt+refs in its embedded graph, so the row stays
small and the dataset is training-ready (preference / reward / instruction-tuning).
- GET /v1/feedback/stats (👍/👎/with-text/total) + GET /v1/feedback/export (the JSONL,
as a download). Toolbar shows '⬇ Feedback (N)'.
ratings.json stays the current-state store the UI reads; feedback.jsonl is the
append-only event log — orthogonal concerns, one rate call writes both. Verified
end-to-end headless: vote registers + stats increment, dialog captures vote+text,
export emits training-ready JSONL; backend unit + full app reboot clean.
A Qwen-Image-Edit fix runs at denoise 1.0 → it re-paints EVERY pixel, so the
render's global white-balance/exposure/saturation drift off the source, and the
drift COMPOUNDS across fix-of-a-fix (each fix re-encodes the prior, already-shifted
output). That is why repeatedly-fixed images end up visibly mis-colored.
Fix: restore the render's global color statistics to its source (Reinhard mean/std
transfer, per RGB channel) — the local edit stays, the global cast goes; a render
that didn't drift matches to itself = untouched. Verified: 4 successive fixes hold
Δ→source ~1 with the match vs 6.5→16.5 (compounding) without.
Applied POD-SIDE at worker-render ingest (server.upload_output → colormatch_fix_ingest):
- needs NO node on the GPU worker and touches no render graph → can never make a
render fail (avoids the 'failed validation' class entirely)
- scoped to fixes/… outputs only (never compose/normal renders), keyed to the
staged source via the render-jobs record
- fail-safe throughout: any error / missing source / size mismatch leaves the raw
bytes exactly as uploaded (proven: a save-format bug left files untouched, never
corrupted)
- strength 0.85 default (STUDIO_COLORMATCH_STRENGTH), leaving headroom for an
intentional global recolor in the instruction
New middleware/color_fidelity.py (pure algorithm + atomic in-place match);
studio_home.colormatch_fix_ingest + _fix_source_ref resolve org+source.
The soft-delete holding area is now an Archive (recover any item, or the
whole history stays visible). Inside it, permanent removal:
- 🗑 Delete forever per card + bulk on a selection, confirmed via the
existing <dialog> ask() (irreversible — no ⌘Z can undo an unlinked file)
- 🗑 Empty Archive clears the lot in one confirmed step
- POST /v1/library/purge {paths[]|all}: unlink file + drop record + drop
cached thumb. Archive-only by design — refuses any asset whose status is
not 'deleted', so a gallery mis-click can only ever archive (recoverable),
never destroy. Verified end-to-end headless: per-card/bulk/empty purge,
the archive-only guard, and rename coverage (tab, count, chips, toasts).
Rename is display-only; the underlying status stays 'deleted', so the
1,259 already-archived items need no migration.
The /v1/library/fix flow degraded images and compounded the damage every pass —
the source of the noise-stipple "bad skin" epidemic across the library. Three
layers, all decomplected.
1. GRAPH. _fix_graph (and _compose_graph) dropped the two
FluxKontextMultiReferenceLatentMethod nodes — a Flux-Kontext
`index_timestep_zero` reference method grafted onto Qwen-Image-Edit
conditioning. It survived on in-distribution studio shots but on real photos
with large flat regions (sky/water) it baked latent-grid mosaic + skin
speckle, and each fix-of-a-fix amplified it. What remains is exactly the
render line's proven-clean family: TextEncodeQwenImageEditPlus reference edit
into an EMPTY latent at denoise 1.0, model names pinned to the exact files on
the worker (qwen-image-edit-2511 / qwen2.5-vl-7b / qwen-image-vae). Proven
e2e on spark: the same input+seed that produced severe mosaic renders clean.
2. HONESTY GUARD. A >=10-step render that lands within 5s of dispatch cannot
have sampled (a graph that errored into history, stale-history reuse, a
degenerate self-POST). _finalize_landed — the ONE place a landed render goes
terminal, shared by the queue poller and the work-log view — marks it FAILED
("engine did not sample") instead of delivering it, and skips a matched
output that predates the dispatch (a same-prefix sibling, not this job).
3. ITERATION HYGIENE. When the chosen fix base is itself corrupt (quality-gate
flagged, or deleted), _clean_base walks the lineage to the nearest clean
ancestor and fixes from THAT, and the dialog says so ("Fixing from the
original — this version is corrupted"). One mechanism in the dispatch path,
used by both the route and the chat create capability.
Tests: graphs name only worker-present models and carry no Flux-Kontext
reference method; the implausibly-fast guard flips status; _finalize_landed
does done/failed/skip-stale; the ancestor fallback picks the clean root.
Co-authored-by: Hanzo AI <ai@hanzo.ai>
The catalog seeder emits byte-identical copies (found 94 dups in one org's 268
visible assets). POST /v1/library/dedup hashes visible assets and soft-deletes
exact-content duplicates — keeps one per cluster, distinct re-renders (different
seeds) untouched — and returns the moved paths. Frontend "Dedupe" button beside
Select; the whole sweep pushes ONE undo entry so ⌘Z restores it all. no-store on
the response. Verified in-browser: dedup hides dups, Deleted count updates, ⌘Z
restores. v0.17.7
The org-scoped fleet read is org-blind: a BYO GPU registered under a sibling
org (karma's workers live under hanzo) is invisible to a karma user token, so
dispatch_if_worker saw "no GPU", fell through, and the GPU-less coordinator
self-POSTed its own ComfyUI — an instant "Prompt outputs failed validation"
(node "studio pod"). Composes for karma died on this every time.
Two halves, one mechanism:
1. A model-less cloud coordinator NEVER renders locally. dispatch_if_worker now
enqueues to the SHARED gpu-jobs lane whenever this box has no models of its
own, even when the org-scoped fleet read shows none — the job WAITS for a
worker (workers drain the shared queue regardless of which org they register
under). A box WITH its own models still renders locally when it sees no
worker. A queued job that waits beats an instant fake failure.
2. The capacity gate (/v1/gpu-status → the ⚠ warning) is advisory ONLY and now
reads the fleet with a SERVICE identity (STUDIO_WORKER_TOKEN, then
STUDIO_COMMERCE_TOKEN), user token as fallback — so every org sees true
worker availability. Job tenancy is untouched: dispatch still enqueues under
the user token; only the org-blind gate changed.
Tests: karma-org model-less dispatch enqueues (never local, never on POST
failure); a box with models still renders local; online worker keeps its label;
the advisory gate uses the service identity first and falls back to the user
token.
Co-authored-by: Hanzo AI <ai@hanzo.ai>
Deletes already persist server-side (verified live: soft-delete → status 'deleted'
holds, zero reverts); the "moved to deleted then restored" was the frontend serving
a cached /v1/library on refresh. Fixes:
- /v1/library → Cache-Control: no-store, so a delete is never masked by a stale copy.
- ⌘Z / Ctrl+Z undoes the last delete (and any status change) — soft-deletes are
reversible, so undo is just a status write-back. Dropped the bulk confirm dialog.
- 🗑 Deleted is now a top-level tab with a live count — trash is easy to find;
per-item Recover + bulk Recover (context-aware bulk bar).
- Bulk: Select → pick many → one-click "🗑 Delete selected". Discoverable.
Verified in-browser (Playwright): delete hides the card, ⌘Z restores it, Deleted tab
filters + Recover works. v0.17.6
Full Stacks spec on the /studio home surface: create by drag (image→image) or
multi-select; layered Stack cards (cover/name/count/updated); a detail view with
add/remove/reorder/move/set-cover/rename/describe/unstack/delete; the delete
dialog defaulting to preserve; generation 'Save outputs to' routing (fix/compose/
rerun/template) with per-org auto-stack-batch setting; keyboard + touch parity
(no drag-and-drop ever required); a11y labels + live region; loading/empty/missing
states.
Backend: middleware/stacks_store.py — per-org SQLite (WAL, BEGIN IMMEDIATE) with
the spec's stacks + stack_assets join, UNIQUE(asset_id) = one Stack per asset,
tenancy structural (one file per org). Replaces the JSON whole-set /v1/stacks
store with the full REST surface. Ingest assigns membership; remove != delete;
delete-images maps to the library soft-delete. Tests: tests-unit/studio_home_test/
test_stacks.py.
Co-authored-by: Hanzo AI <ai@hanzo.ai>
* studio-chat: Up next / History / Stacks queue surface with ratings + favorites
The studio-chat SPA's right rail becomes the full YouTube-watch-page surface,
shaped around generative flows (a card = a run in a flow):
- Up next: the live queue (GET /v1/queue/status joined to /v1/worklog) — pinned
"Next" card + vertical cards, thumb-left with an ETA/elapsed corner badge,
2-line prompt, node · time · status · position meta, per-card overflow menu.
Delete (POST /v1/queue/cancel) and Add references (file pick + drag-drop →
/upload/image → POST /v1/queue/amend) on each queued card.
- History: past runs newest-first (GET /v1/worklog +?q= search), output
thumbnail, "New" badge on just-landed renders, version pips for runs with
descendants → a walker over GET /v1/worklog/lineage.
- Rate 0-5 stars + a note ("What should the AI do differently next time?")
per run — the flywheel row {path, stars, note, ts} keyed by run id.
- Favorite (heart) → offer "turn this into a template" (name it → confirm what
it captures → POST /v1/templates with a captured flow).
- Stacks: drag one card and HOLD (~600ms) over another to form an iOS-style
folder; stacked-cards tile + count; click = folder-open reveal; unstack to
dissolve. Persisted per org.
- Filter chips (All · Queued · Done · Favorites · Stacks), resizable main|rail
and Up-next|History dividers, per-section collapse, active filter — all
persisted in localStorage keyed by org+user; "Reset layout" in the rail menu.
- Mobile (<=820px): single-scroll column, rail as the bottom section, composer
fixed to the viewport bottom with safe-area inset; example-prompt chips; a
44px account avatar; AA-contrast grays; real send-button state pair.
Middleware (all tenant-scoped via _org_of, one atomic per-org store each):
- GET/POST /v1/favorites (favorites.json) — the ONE favorites store; the legacy
studio_home.html favorite toggle migrated onto it too (was ratings.favorite).
- GET/POST /v1/stacks (stacks.json) — normalized, >=2 members, bounded.
- POST /v1/library/rate: carries `path` for the flywheel; favorite removed
(now its own store — one way).
- POST /v1/templates: accepts a `flow` payload, not only a node graph.
Tests: tests-unit/studio_home_test extended (favorites, stacks, rate stars+
notes+path, flow templates) — tenancy proven on each; added to the hanzo.yml
unit gate. web build green.
* studio: self-heal stale-asset dispatch — refresh + "pick it again", never strand
When the item a user clicked was deduped/renamed/cleaned since their page loaded,
a fix/compose/rerun dispatch 400s "unknown asset: <path>" and the raw error left
them stuck. One self-heal wherever a dispatch happens:
- Legacy studio_home.html: healStale(j) — on an unknown-asset error, close any
open dialog, drop the stale selection/tray, re-fetch the library (load()), and
toast "That item changed — library refreshed, pick it again." Wired into the
fix + compose (sendRef), compose (sendCompose), and rerun (bulkFork) branches.
- studio-chat SPA rail: heal(e) — a stale run (unknown asset / not found / no such
job / already finished) on cancel or amend refreshes the queue and shows "That
run changed — refreshed, pick it again." instead of the raw error.
* rail: bigger card thumbnails + a visible × on every card (cancel / soft-delete)
- Thumbnails grow to ~half the card width and a taller 4:5 crop (44% on mobile)
so the render is actually legible in Up next and History.
- Every card gets an always-visible × (fast path) beside the ⋯ overflow menu —
ONE shared handler, no second mechanism: a queued run cancels immediately
(POST /v1/queue/cancel); a finished result is a destructive soft-delete
(POST /v1/library/status status=deleted) and asks first with a small inline
confirm. History hides soft-deleted results (optimistic + reconciled from the
org's deleted set via /v1/library) so the delete persists across reloads.
---------
Co-authored-by: Hanzo AI <ai@hanzo.ai>
The /prompt intercept sends every render to the org's GPU worker, but a
transient fleet-check fault (/v1/machines 502 / TLS reset / timeout) made
dispatch_if_worker return False, so the render fell through to LOCAL
validation on the model-less cloud pod → the cryptic "Prompt outputs failed
validation" (unet_name '…' not in []). That's why Fix failed only *sometimes*.
- _fleet_machines: retry 3x with backoff so a transient hiccup isn't read as
"no GPU" and bounced onto the model-less pod.
- has_local_models() + intercept: on a model-less coordinator pod a failed
dispatch now returns a clear retryable 503 "GPU render worker is momentarily
unavailable — please retry", never the doomed local validation. A box WITH
models still renders locally (dev). v0.17.5
Every render lands in the library through the one ingest choke point,
POST /v1/library/upload -> _index_upload. The moment that creates a NEW
asset row, a quality judgment fires out-of-band (aiohttp task, never
blocks the upload response): the just-stored image is downscaled to a
~512px JPEG and posted to the cloud vision judge (zen-vl) for a strict
PASS/FAIL on the observed corruption classes -- mottled/blotchy/speckled
skin, ghosting/double-exposure, frame-within-frame borders, glyph garbage.
A clear FAIL moves the asset draft -> flagged (a new first-class status
with deleted's visibility semantics: excluded from every default grid,
recoverable) and annotates its work-log row. Everything else -- PASS,
inconclusive, 402/429/timeout/unreachable, unreadable image -- leaves the
render a normal draft. The gate fails open: it can only ever hide clearly
broken work, never good work, and never blocks a render from landing. A
late verdict only ever transitions draft, so it can't yank curated work.
- middleware/quality_gate.py: the mechanism (judge prompt, strict verdict
parse, downscale, fail-open run, fire-and-forget schedule). Same bearer
the upload carried -> tenant-isolated cloud billing, one auth path.
- studio_home: _STATUSES gains flagged; _index_upload reports new rows;
the upload route schedules the gate on a newly-surfaced render.
- worklog.add_event: annotate a completed row without touching its status.
- studio_home.html: flagged hidden from normal grids, a Flagged review
filter, Restore -> draft.
- /v1/library/status accepts flagged (flag + restore). Sweeper preserves
and tallies flagged like deleted (verified).
Config: STUDIO_GATE=off kills it; STUDIO_GATE_MODEL (default zen-vl);
STUDIO_GATE_URL override for local engines. No other knobs.
Co-authored-by: Hanzo AI <ai@hanzo.ai>
The everything-indexes sweep pulled .thumbs/ cache files in as rows; fixing
one sent a thumb path into the dispatcher (unknown asset). One rule, both
writers: any path segment starting with a dot is excluded — forks and cache
trees alike.
Co-authored-by: Hanzo AI <ai@hanzo.ai>
The sweeper rebuilt library.json and DROPPED every image classify() could
not parse — erasing ingest-indexed renders on each cycle (rows appeared,
then vanished minutes later). Unclassifiable images now keep a minimal row,
prior fields preserved.
Co-authored-by: Hanzo AI <ai@hanzo.ai>
The web/dist chat SPA shadowed studio_home.html at /studio, dropping the
per-image affordances: delx (remove → POST /v1/library/status deleted),
the Fix dialog (type-and-edit → /v1/library/fix), and the + reference tray
(→ /upload/image → /v1/library/compose, up to 5 refs). Serve the rich home
explicitly; drop the dead /studio/assets route + the node web-build stage.
web/login.html still ships (server.py serves /login). v0.17.4
An interrupted mirror leaves bytes on disk with no library or worklog row;
the dedup branch skipped indexing, so re-ingesting repaired nothing. The
library upserts by path; the worklog now upserts by output. Recovery is a
re-POST of the same bytes.
Co-authored-by: Hanzo AI <ai@hanzo.ai>
The indexer skips them; _safe_asset refuses them (one gate for file serving,
fix sources, and every other path lookup). AppleDouble forks carry image
extensions, serve resource-fork bytes, and rendered 0x0 across the library.
Co-authored-by: Hanzo AI <ai@hanzo.ai>
The SPA called the gateway cross-origin; the session cookie is host-only, so
every signed-in user got 403 on their first message (401 once the cookie was
widened — the gateway takes bearers, not cookies). The bridge forwards the
caller's bearer server-side, the wallet pattern. VITE_AGENT_API still selects
direct mode.
Co-authored-by: Hanzo AI <ai@hanzo.ai>
On a phone (<=820px) the library rail was display:none, so a user could chat and
dispatch renders but never see them. Now the body stacks (chat fills, library
drops under it) and the rail shows as a compact scrollable bottom strip — but only
when there is something to show (jobs or assets set the `active` class), so the
empty chat state stays clean. Verified with Playwright at iPhone 14 (390x844):
empty = clean chat; with renders = 4-col strip under the composer, no overflow.
* upload: reject dotfiles and sniff magic bytes — the name lies, the bytes never do
AppleDouble forks (._foo.png) passed the extension check; 700+ poisoned a
library within an hour of the mirror going live. PNG/JPEG/WebP signatures
required; hidden names rejected.
* upload: size cap precedes the sniff; test payloads carry the full PNG signature
---------
Co-authored-by: Hanzo AI <ai@hanzo.ai>
The studio-chat frontend (web/, Vite on @hanzo/ai talking to POST /v1/agent) was
repointed + committed but never served. Build it in a node stage of the Dockerfile
(-> /app/web/dist) and serve it at GET /studio, with GET /studio/assets/{name} for
the hashed bundle (immutable) and a traversal guard. Falls back to the legacy
studio_home.html when the build is absent so /studio is never a dark page. This is
the decided cutover: studio.hanzo.ai/studio becomes the /v1/agent chat app.
An ingested render row has an empty prompt, so filtering Queue & History by
filename found nothing. Widen the /v1/worklog ?q= filter to also match kind,
output_prefix, output (the landed path), and refs — so a render is findable by
its filename or source image, not only by a typed prompt. Test covers filename,
ref, and prompt matches.
Co-authored-by: Hanzo AI <ai@hanzo.ai>
* Library ingest: POST /v1/library/upload — every render lands in the library
Tenant-scoped ingest so EVERY render is stored in studio.hanzo.ai, including
renders produced on a GPU node outside the job path (the node's mirror loop
POSTs here). Org via _org_of like every route.
Body: multipart (field image/file) OR raw bytes + ?name=, optional ?subpath=
(default renders). Writes orgs/<org>/output/<subpath>/<name> atomically
(tmp+replace, same pattern as the work log). Rejects traversal (basename-only
name + allowlisted subpath chars), caps size at 30MB, and skips a byte-identical
existing file (name+size match → 200 {ok, existed:true}); else 200 {ok, path}.
The manifest sidecar already indexes output/ — no other wiring.
Tests (studio_home_test): round-trip via GET /v1/library/file, dedup, tenant
isolation (org A upload never visible to org B, no cross-tenant overwrite),
subpath-traversal rejected, traversal-y name neutralized to basename, non-image
rejected, oversize rejected. 259 passed.
* Ship scripts/ whole (unbreak manifest sidecar) + index renders at ingest
(A) Regression: .dockerignore `scripts/*` + `!install_custom_nodes.sh` dropped
scripts/library_manifest.py from the image, so the build-manifest sidecar's
`[ -f ]` guard silently no-op'd and NOTHING got indexed into any org library.
Ship scripts/ whole again.
(B) Make indexing event-driven at the ingest point: POST /v1/library/upload, after
writing the asset, atomically appends the org library.json entry ({path, design?,
kind, role?, status:"draft", tags:[], updatedAt}) AND a worklog row ({kind:"render",
node, prompt:"", refs:[], parents:[], output_prefix:<path>, status:"done"}) — so
every mirrored render is instantly findable in Assets AND Queue & History with its
source node, filterable by run/flow, the moment it exists. Optional ?design= ?kind=
?role= ?node= with safe defaults (node→"upload"). Upserts by path (no dup on
re-render); the existed/dedup path skips indexing. The manifest sidecar stays the
sweeper for files that appear outside the route. _row_output resolves an ingested
render's output (its stored path) so it shows a thumbnail + opens its graph.
Tests: upload → library.json entry + worklog row appear (node, done, resolved
output), tenancy holds, dedup adds no second entry. 260 passed.
---------
Co-authored-by: Hanzo AI <ai@hanzo.ai>
cursor:pointer + title=Account with no handler. Tap now opens a menu:
identity, Manage account (hanzo.id), Sign out (/logout — existing route).
Co-authored-by: Hanzo AI <ai@hanzo.ai>
The orchestrator lives at POST /v1/agent (github.com/hanzoai/agent in the cloud
binary), not /v1/chat (which ai owns as its completions route). Use the canonical
`preset` selector (not the legacy `capability` alias) and thread conversationId so
the per-org history the orchestrator persists actually continues across turns.
VITE_CHAT_API → VITE_AGENT_API.
One shared shell asset (middleware/shell.{js,css}, served at /studio/shell.*)
is loaded by BOTH views — the Studio home <script>-includes it, and the Editor
gets a single tag injected into the engine index.html by server.get_root. It
renders one chrome in both: a Studio/Editor toggle, wallet chip, GPUs badge +
panel, and a dockable right chat sidebar backed by the ONE chat path
(/v1/copilot/chat). Same-origin cookies carry the IAM session, so login/org are
inherited — no auth is rebuilt.
New tenant-scoped routes (all via _org_of):
GET /studio/shell.js|css
GET /v1/workflow?id=<worklog id> graph from the item's output PNG (404 if
not owned / not landed) — Open in Editor
POST /v1/templates save {name, graph} → orgs/<org>/templates
GET /v1/templates list
GET /v1/templates/{name} fetch
POST /v1/templates/render queue through the ONE dispatch funnel
GET /v1/wallet display-only proxy of the user token to the
cloud finance balance → {org, balance, currency}
Studio home: "Advanced mode" link replaced by the shell toggle; dead chatfab CSS
removed; Templates tab lists org templates (Open in Editor + Render); work-item
context gets Open in Editor; favorite→template now writes a REAL template.
Mobile (verified 390x844 + 768x1024): chat collapses to a full-width bottom
sheet; the pill bar stays one compact line (icon-only Chat/GPUs) so all controls
stay reachable and clear the content; zero horizontal overflow.
Wallet is display + Top up link (pay.hanzo.ai) only — no card fields, no minting.
Tests extend studio_home_test for the new routes (tenant isolation + shape).
Co-authored-by: Hanzo AI <ai@hanzo.ai>
Liveness is the 1200s heartbeat; the cap only bounds a live-but-stuck render.
Observed: activities reaped at ~68m mid-render, queue re-ran the same job six
times, results stranded on the node.
Co-authored-by: Hanzo AI <ai@hanzo.ai>
Makes the unified queue honest about WHERE and WHEN, all server-computed and
tenant-scoped (an org sees only its own jobs' positions and its own nodes).
WHERE — node identity:
- gpu_dispatch reads whatever the gpu-jobs fleet publishes (_node_label prefers
name/hostname/label; home-lab labels spark/dbc/evo land there) — no parallel
registry invented. dispatch_if_worker records the target node (the single online
GPU is definitive; ambiguous 'gpu' with several) on the render_jobs row; _dispatch
copies it onto the work-log row. Local-core jobs are 'studio pod'.
- GET /v1/nodes: the org's GPU nodes (online/offline) + the pod, for header badges.
Fleet is token-scoped → only the caller's nodes.
WHEN — position + ETA (server-side, never client math over private data):
- worklog records queued/started/finished timestamps (mark_started when a job first
heads its lane; mark_finished from the output file mtime). Pure functions:
median_durations (rolling median render seconds per kind) and estimate_lanes
(position N of M per lane; wait ≈ jobs-ahead × medians + running remaining;
eta = wait + own median). Unknown kinds fall back to a default; labeled honestly.
- GET /v1/queue/status: this org's active jobs with {position, of, node, status,
elapsed_seconds, wait_seconds, remaining_seconds, eta_seconds} + medians. Reads
the org work log only; no fleet call on the hot path.
UI: 'Your work' rows and the BYO-GPU row show ⚙ node · #N/M · running <elapsed> ·
~Xm left / #N/M in line · ~X min est. Queue header shows per-node online/offline
badges + the median-render stat. Auto-refresh while the queue view is open (6s
positions, 20s nodes — no hammering); the running item's elapsed ticks each second
client-side; a completion toast fires when an open page sees a job land.
Tests (11 new): _node_label + online-node filtering; record-node; started/finished
+ median; median_durations per kind + ts fallback; estimate_lanes positions/ETA +
default-median. Route smoke: positions #1/2/3 with ETAs from the org's own median,
landed→finished drop, globex sees none of acme's queue, /v1/nodes maps fleet→badges.
Browser smoke: node+position+ETA render, elapsed ticks 0s→2s, median stat + pod
badge, zero console errors.
Co-authored-by: Hanzo AI <ai@hanzo.ai>
Cancel and amend any of the caller's OWN queued/running renders, resolved
server-side; a prompt_id that isn't the caller's 404s (never reveals it exists).
Cancel — POST /v1/queue/cancel {prompt_id}:
- local pod queue: delete a pending item OR interrupt a running one, org-scoped
(reuses the core prompt_queue primitives — one tenant can't touch another's).
- BYO-GPU lane: drop the render_jobs.json tracking row (per-org file → inherently
owned); a job already rendering on the worker may still finish (the pod can't
stop a remote process) — the UI says so honestly.
- marks the work-log row cancelled. _owns_job gates it: work-log OR org-stamped
local-queue item OR the org's gpu tracking file; else 404.
Amend — POST /v1/queue/amend {prompt_id, instruction, paths?, uploads?}:
- honest atomic swap (queued graphs are immutable): cancel the original, then
re-dispatch through the ONE funnel with the amended prompt + the original refs
(kept, read from the authoritative work-log row) + new library picks/uploads.
Records a fresh work-log row; position resets, shown honestly in the UI.
UI: Edit + Cancel on every queued/running work row and in the context panel;
Cancel on the BYO-GPU live row; the local 'Remove' now routes through the unified
cancel (dead delQueued removed). Amend reuses the Fix dialog in amend mode
(openAmend): prompt primed, current refs shown locked, add more via the same
dropzone + history picker. Errors surface in-dialog (no silent swallow).
Hardening: _queue_prompt tolerates a non-JSON downstream response (reports it
instead of 500-ing).
Tests: cancel org-scoped + gpu-row removal + cross-tenant not-owned; amend atomic
swap keeps original refs + adds new, unknown→NotOwned(404), terminal rejected (5).
Route smoke: amend swaps + re-queues, gpu cancel removes tracking, globex gets 404
on acme's job (cancel AND amend). Browser: Edit/Cancel on queued rows, amend dialog
primed with prompt + current ref, correct POST, zero console errors.
Co-authored-by: Hanzo AI <ai@hanzo.ai>
Adds a per-org work log — the ONE record of every render dispatched — as the
backbone for a queue/history that never loses a job and shows what produced it.
Backend (middleware/worklog.py, one JSON per org at orgs/<org>/output/worklog.json;
same pattern as library.json/render_jobs.json; no new DB, no backfill):
- The single dispatch funnel (dispatch_fix/dispatch_compose/rerun) records one row
at dispatch via _dispatch(): {id, ts, kind, prompt, refs, uploads, parents,
output_prefix, params, lane, status, events}. A FAILED dispatch is recorded too
(status=failed + reason) so the deployed 'nothing showed as queued' can't recur —
every dispatch is visible with real status, gpu-lane or local.
- GET /v1/worklog — org's dispatched renders, reverse-chron, ?q= over prompt+kind,
status enriched (done when the output lands). Server-resolved org only.
- GET /v1/worklog/lineage?path= — a library asset's version chain (ancestors it was
derived from + later fixes), a pure function over this org's rows.
- _landed_output matches SaveImage's exact <prefix>_<counter>_.png (not a loose
<prefix>*, which collapsed distinct lineage steps).
Frontend (studio_home.html):
- Queue & History → 'Your work': searchable, reverse-chron list with status badge +
lane + prompt + thumbnail; click → context panel (prompt, reference/result thumbs,
workflow family, params, process trail, and the Versions chain inline).
- 'Versions' affordance on every library card → the same lineage chain; click any
version to open it.
- 'Reuse context' / 'Fix this version' open the Fix dialog primed with the item's
prompt + references (openFixPath/primeFix) — reuse = continue from any version.
Tenant isolation: one log file per org; routes resolve org server-side; lineage is
pure over the caller's rows — cross-tenant leakage is impossible by construction.
Tests: worklog record/failed/status/cap, lineage chain + cycle-termination,
cross-tenant isolation (7). Route smoke: dispatch→done+output, 2-step lineage,
failed dispatch recorded with reason, globex sees none of acme's work. Browser
smoke: work list + status badge + context panel + Versions chain, zero console
errors.
Co-authored-by: Hanzo AI <ai@hanzo.ai>
Two user-reported live regressions on studio.hanzo.ai, root-caused at the
deployed image (sha-0d307dc, pre-#6):
1) Fix 'won't work / nothing queued' — the GPU-less prod pod fails /prompt
validation (models absent) or the router has no GPU worker; _queue_prompt
raised web.HTTPBadGateway(text=json.dumps(body)[:500]) — a raw 502 whose
TRUNCATED body the dialog couldn't parse, so the error was swallowed into an
empty 'Failed:'. Now _queue_prompt raises DispatchError carrying the REAL
reason (_prompt_error_message), which the fix/compose routes already return as
{"error"}. Every dialog/sender now SHOWS backend errors persistently and keeps
the dialog open on failure — no silent swallow anywhere on fix/compose paths.
2) The composer '+' attach button did nothing (no handler). Wired it to the
shared upload lane (POST /upload/image → org input dir, multi-tenant) with a
visible thumbnail; added drag-and-drop + paste to the Fix dialog AND the
composer, all through ONE shared helper (uploadImage/uploadMany/wireDrop/
wirePaste). Upload failures are shown, never swallowed. Reference tray now
holds library AND uploaded refs; a single uploaded photo drives a fix via the
new dispatch_fix(upload=) base (edit a fresh photo without landing it first).
Tests: text-only fix regression lock, uploaded-base fix + traversal refusal,
_prompt_error_message extraction, and a real-aiohttp test that a non-200 /prompt
surfaces the real reason (not a 502). Browser smoke (headless Chromium on the
served page): '+' opens the chooser and uploads, drag-drop adds a ref, text-only
Fix posts {path,instruction}, zero console errors.
Co-authored-by: Hanzo AI <ai@hanzo.ai>
The Fix action on a generated image becomes a reference-capable surface:
- base image (the clicked output) + up to 4 references
- drag-and-drop / click-to-upload image files as references, via the
existing core /upload/image route (org-scoped input dir — the same place
fix/compose already stage refs and the BYO-GPU collector ships from)
- add-from-history picker: attach past library generations as references
- no references -> single-image /v1/library/fix (unchanged); any reference
-> multi-input /v1/library/compose, base image first
Backend: dispatch_compose now accepts uploaded input-dir names (uploads[])
alongside library asset paths, validated as safe existing images; the 2-5
bound is on the combined reference count. One compose path, additive.
Tests: _resolve_uploads filtering, compose upload path + single-ref
rejection; corrected stale _STATUSES vocab (includes 'deleted').
Co-authored-by: Hanzo AI <ai@hanzo.ai>
Chat hits the cloud clients/chat orchestrator directly (credentials:'include'
carries the .hanzo.ai cookie cross-origin; handler replays it into the billed
completion). Engine calls stay same-origin. Needs gateway CORS to allow the
studio.hanzo.ai origin with credentials — infra config, no backend code.
Vite+React studio-chat: say what to create, the cloud /v1/chat orchestrator drives
studio's render tools, outputs land in the library. Increment 1 (own minimal UI
wired to /v1/chat create + library/queue/gpu). Served at /studio; the ComfyUI node
editor stays separate in studio-ui (was studio-frontend). Builds clean (npm-published
@hanzo/*).
MCP-over-HTTP (JSON-RPC 2.0) exposing the studio render pipeline as tools the
cloud /v1/chat 'create' capability drives via the tool registry. Reuses the ONE
dispatch path (studio_home.dispatch_fix/dispatch_compose, extracted from the
/v1/library/{fix,compose} routes) so an LLM-issued fix behaves identically to a
UI one. Registered per-org via cloud POST /v1/tools/servers. v0.17.3.
/studio: org-scoped asset gallery over library.json (status chips draft/
approved/queued/published), per-asset workflow provenance from the embedded
PNG graph, Re-run (exact graph, fresh seed) and chat-fix (instruction ->
Qwen-edit pass on that output) both through the ONE /prompt path with the
caller's own credentials. Pipelines tab reuses the 0.15.1 deep-link opener
for Edit/Run; STUDIO_HOME_DEFAULT=1 makes / land on /studio (SPA at
/?advanced=1). 5 unit tests.
Claude-Session: https://claude.ai/code/session_01Gq8suw7uuodAMPDRpo6iAB
A gallery/console/chat link lands you on the graph ready to render: sets the
session org (so a re-run dispatches to that org's BYO-GPU and outputs to its
gallery), loads the saved workflow, and optionally auto-queues. Injected as a
frontend shim (patch_deeplink.py, step 13 of apply-branding), guarded end-to-end.
/app/models is ephemeral overlay — every model vanished on pod restart
(CheckpointLoaderSimple: 'not in []'). Bake extra_model_paths.yaml into
the image registering /app/orgs/.models (PVC) as the default model store
for all model classes; main.py auto-loads it. Downloads land durable.
Serialized, SQLite-persisted queue fronting Hanzo Studio (ComfyUI :8188) that
turns a prompt into a real MP4 on the GB10. One GPU => one job at a time; a
single async worker drains an ordered queue and submits to Studio.
Two composable surfaces over one backend:
- OpenAI-style /v1/videos/generations (gateway-injected X-Org-Id identity, per
generated-second billing into a usage table).
- do-ai-compatible async /v1/videos + /{id} + /{id}/content (Bearer provider
key) — the exact Sora-style create->poll->download contract the cloud LLM
gateway's video client drives, so the ai registry adopts spark by pointing
the zen3-video provider at spark, no ai-side client change.
Zen video family public names (zen3-video/-fast/-pro); private Wan2.2 filenames
confined to zen_wan_workflow.py, never returned to callers. Includes the
systemd unit and studio_clean_restart.sh (breaks the Studio JIT/EADDRINUSE
crash-loop).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A GPU-less cloud pod validates ckpt_name & friends against ITS OWN (empty)
model list, rejecting every checkpoint-referencing graph with 'not in []'
before dispatch could ship it to the org's GPU box. Validation belongs where
execution happens: the dispatch attempt now runs first, and the worker
validates against its real node classes + models. Orgs with no online GPU
fall through to local validation unchanged.
BertModelWarper now delegates to the real BertModel.forward instead of
reimplementing internals whose private signatures drift between releases;
checkpoint weights remap bert.* -> bert.bert_model.* at load. The installer
force-checkouts each pin (discarding prior patches) then applies any
scripts/patches/<pack>-*.patch, so re-runs stay idempotent and the image
build gets the same fix the GB10 venv had by hand.
karma_manifest.py regenerates the ONE canonical library.json index of the
karma org's rendered assets (storefront + socials queue both read it).
studio_watchdog.sh keeps the local render backend on :8188 alive unattended;
paths derive from the script location (STUDIO/STUDIO_PY overridable).
A finished render becomes an `Asset` document — the DocType hanzoai/cloud
clients/content already declares (framework module "marketing") and already writes
itself when IT drives a render. Studio-initiated renders were the only ones with
no path into that record; this closes exactly that gap and adds NO second schema,
NO second lifecycle, and NO second storage plane. cloud is untouched.
POST /v1/framework/Asset (the already-live generic framework surface)
Same field shape clients/content writes (studio_render.go draftAsset): title,
kind, design, prompt, file, generator, workflow, source_prompt_id, render_params,
project, tags. `file` is the org-scoped output key (orgs/<org>/output/...) —
byte-identical to the key persistOrLink produces and the key Studio's own output
mirror lands, so the bytes are ALREADY persisted: this module uploads nothing and
holds no storage credential.
Written AS THE REQUESTING USER: the verified IAM access token rides to the prompt
worker on the queue's existing `sensitive` channel (never persisted into history)
and authenticates every write, so the Asset lands in that user's org with that
user's rights. No service account, no impersonation, no cross-org path.
Assets land in draft (the DocType default); the content lane's ONE state machine
(draft->in_review->approved->queued->published) owns every transition, and a human
does the promoting. Publishing is what a storefront shows, so an unattended render
can never push an image to karma.style.
Fire-and-forget on the same prompt-worker seam as billing: a render is never failed
because the lane was unreachable. Env-gated (STUDIO_CONTENT_PUBLISH); local dev
unaffected.
Add HanzoMusic (POST /v1/audio/music -> ACE-Step stereo AUDIO) and HanzoDub
(POST /v1/animate -> MuseTalk lip-sync: portrait IMAGE + driving AUDIO -> frames
+ audio). Fix HanzoImageTo3D to send the now-live `texture` flag (textured GLB
via SLAT + FlexiCubes) and offer obj. Correct HanzoTTS response_format to the
engine's actual wav/pcm. Drop the torchaudio dependency: studio_audio_to_wav_bytes
now uses the stdlib wave module, and studio_audio_to_data_url feeds the dub's
driving audio.
Ship user/default/workflows/hanzo-full-generative-pipeline.json: a prompt is
expanded by Chat and fans out to Image Gen -> (textured Image-to-3D AND WAN
Text-to-Video), plus TTS narration and ACE-Step Music -- every modality in one
graph, all on the native engine.
contract_check.py drives all 8 nodes over real HTTP against a stub engine that
enforces each Rust handler's request fields and the async job submit->poll->content
protocol.
Run Hanzo Studio AI workloads on the native-Rust Hanzo Engine instead of
in-process Python. Transport, config, and Studio<->engine type conversions
live in one client (client.py); each node is one endpoint, so a new engine
endpoint is a new small node with no framework change.
Nodes (category Hanzo/Engine): HanzoChat (SSE streaming, model dropdown from
/v1/models), HanzoImageGen (engine width/height/n schema), HanzoTTS
(/v1/audio/speech -> AUDIO), HanzoASR (/v1/audio/transcriptions -> text),
HanzoEngineRequest (generic /v1 escape hatch), HanzoVisionCaption,
HanzoTextToVideo, HanzoImageTo3D, HanzoSaveText.
Config: one way, HANZO_ENGINE_URL (default http://127.0.0.1:1234), /v1/* only.
Ships example user/default/workflows/hanzo-native-pipeline.json (LLM prompt ->
image prompt -> image gen -> save), e2e-verified against a CPU Qwen3-0.6B engine.
Uploaded images land in orgs/{org}/input on this GPU-less pod; the BYO
worker renders on its own disk and cannot read them, so LoadImage of an
uploaded photo failed. Collect every LoadImage-referenced file that
exists under the org input dir, base64-inline it into the studio.render
job input (capped, traversal-safe). The worker materializes them before
rendering (cloud cli/gpu.go). Files already on the worker are harmless to
resend.
The studio.render job input now carries this deployment's public base (STUDIO_PUBLIC_URL,
default studio.hanzo.ai) as uploadUrl, so the worker POSTs finished renders back to
/upload/output here — org derived from the user's token, landing in orgs/{org}/output.
A remote hanzod gpu-fleet worker renders a studio.render job on its own GPU
and POSTs each finished image to /upload/output with the user's IAM bearer.
The IAM middleware validates the token and derives org from owner/groups, so
the file lands in this org's orgs/{org}/output (PVC + S3 mirror -> gallery).
No S3/rclone credentials on the worker box; the session token is the only
credential. Replaces the old rclone/S3-creds output sync.
- verify_jwt accepts a set of first-party audiences (STUDIO_IAM_ALLOWED_AUDIENCES)
so a console/desktop/CLI-minted user token authorizes an org-scoped upload
without a per-service re-mint (org still comes from owner/groups, not aud).
- image_upload forwards the request so /upload/image|mask|output are org-scoped.
The bundled studio/sd1_tokenizer/vocab.json had the token 'comfy</w>' scrubbed
(49407 tokens vs canonical 49408) while merges.txt still contained the 'com fy'
merge, so CLIPTokenizer.from_pretrained raised 'Error while initializing BPE:
Token comfy</w> out of vocabulary' for ANY FLUX.1 CLIP load (DualCLIPLoader type
'flux'). Only ever hit now because FLUX.1-Fill is the first CLIP-L consumer here
(Qwen/Klein use other tokenizers). Restored the canonical openai/clip-vit-large-
patch14 vocab.json (49408 tokens). Required by couture_best.json.
The bottom Docs/GitHub/Console/Discord row had no .product a rule, so it fell
back to the browser default link color + underline. Match the card's muted
console tokens (#cdd2da, no underline, hover -> #eef0f3).
Task 2 — login/header look & feel to match console.hanzo.ai. The console is a
deliberately monochrome true-black system (no brand-purple; white primary
action; 9px radii; hairline #23262b borders), so the prior black+#8b5cf6 purple
login + identity pill did not read as the same product family. Restyled to
console's tokens:
- web/login.html: rebuilt as a console-style centered card (surface #050505,
1px #23262b, 9px radius) with the 5-path Hanzo H mark, 'Sign in with Hanzo'
as a white (#fbfcfd) primary button with near-black text — not the default
ComfyUI splash.
- branding/hanzo-identity.css: pill + dropdown restyled monochrome to match
console's OrgSwitcher/user menu (surface/border/text tokens, neutral avatar,
red only for logout). Bumped identity bundle v2 -> v3 (cache-bust).
Task 3 — Nodes 2.0 (Comfy.VueNodes.Enabled) shipped as the only node renderer.
New idempotent branding/patch_vuenodes.py (wired as apply-branding step 12):
forces the registered default !1 -> !0, sets type boolean -> hidden so the
Settings dialog row is gone (ComfyUI-native, like Bookmarks.V2), flips the
inline ??!1 read fallbacks to ??!0, and deletes the logo-menu nodes-2.0-toggle
item. Same hidden treatment for the sibling AutoScaleLayout so the whole
'Nodes 2.0' settings category collapses. Verified: forced on, no user toggle.
Locally (IAM off) /v1/session returns 200 with authenticated:false; the
menu now renders the functional Local User pill instead of a Sign in
button whose /login 404s locally. The real signed-out state is the cloud
401 (auth on, no token) -> Sign in. Verified unmocked against the live
backend. Bundle v2.
Add a comfy->studio import shim so upstream ComfyUI custom nodes run
unmodified against this fork's renamed core, then vendor the popular packs
(Impact-Pack/Subpack, IPAdapter_plus, controlnet_aux, Ultimate SD Upscale,
Frame-Interpolation, VideoHelperSuite, rgthree, Custom-Scripts, KJNodes,
cg-use-everywhere, SAM2, segment_anything) pinned to immutable commits and
reproducibly rebuilt in the image.
- studio_compat: one meta-path finder aliasing comfy*->studio* and mirroring
Comfy*->Studio* API classes (the deferred phase-4 of scripts/rename_modules.py,
done with the 3.12 find_spec API); activated by one import at top of main.py.
- nodes.py: honor the upstream `comfy_entrypoint` V3 extension name too.
- custom_nodes/hanzo-packs.txt: pinned pack manifest (SHAs + SPDX licenses).
- custom_nodes-requirements.txt + scripts/install_custom_nodes.sh: pack deps
with the cu130 torch / NumPy 2.x / Transformers 5.x stack locked (derived
from the live env, never downgraded); ultralytics installed --no-deps.
- Dockerfile/.dockerignore/.gitignore: image vendors the packs at build.
Live GB10 worker verified: node types 644 -> 1290 (+646), 0 import failures.
was-node-suite quarantined (hard numba dep incompatible with NumPy 2.5).
Top-right user menu driven by GET /v1/session (user, org switcher,
project scope, Console link, Logout). Org switching is real via one
hook: the auth middleware applies session.resolve_org to a COPY of the
validated user so iam_user["org_id"] -- the id every tenancy/output
path already reads -- reflects the selection; membership from
_orgs_from_claims (owner + organizations/orgs/groups). POST
/v1/session/{org,project} set validated cookies; /logout clears the
session + selection cookies and bounces to the IAM end-session -> /login.
Frontend is a SPA-agnostic pill + dropdown (branding/hanzo-identity.*),
injected like the copilot (patch_identity.py, apply-branding step 10).
patch_destring.py (step 11) sweeps English display 'ComfyUI' -> 'Hanzo
Studio' (boundary-safe: skips code identifiers, i18n keys, hyphenated
product names, and non-English locale bundles) and rebrands the Settings
section header value 'Comfy' -> 'Studio' / 'Comfy-Desktop' -> 'Studio
Desktop' while preserving the setting KEYS. Served English display
'ComfyUI' 63->0. Tests: middleware_test/session_test.py (12 new).
Backend POST /v1/copilot/chat (middleware/copilot.py, registered in server.py)
runs one OpenAI tool-calling round to a chat endpoint (STUDIO_COPILOT_URL,
default local engine :1234 else api.hanzo.ai) and returns {reply, ops}. add_node
ops validated against nodes.NODE_CLASS_MAPPINGS so the model can't invent nodes.
Frontend bundle (branding/hanzo-copilot.js + on-brand css) registers a Copilot
sidebar tab via app.extensionManager.registerSidebarTab and applies ops against
app.graph (LiteGraph): add/set_widget/connect/set_prompt/move/delete/layout/queue.
Every op guarded. Injected by branding/patch_copilot.py (step 9 of apply-branding.sh),
survives frontend upgrades. window.HanzoCopilot.applyOps exposed for tests.
Tests: middleware_test/copilot_test.py (13, op-validation + mocked tool loop),
branding/copilot_smoke.py (headless: applier mutates a mock graph + panel renders).
Live route needs one server restart to register.
On a successful prompt execution, emit exactly one fire-and-forget usage
event to Commerce meter-events (POST /v1/billing/meter-events): userId =
IAM org_id, value = artifacts produced, idempotency = prompt_id. Timeout +
retry-once; a billing failure only logs and is dropped — a render is never
blocked or failed for billing.
Replaces the execution-time-priced record_usage (which billed against the
STUDIO_ORG_ID env, not the real IAM tenant) with metered record_render:
Studio reports raw quantity, pricing lives on the meter in Commerce.
Env-gated on STUDIO_BILLING_URL + STUDIO_COMMERCE_TOKEN + STUDIO_BILLING_METER
(all absent locally = no-op, zero behavior change). check_balance left as-is.
Tests: tests-unit/middleware_test/billing_test.py — success shape, idempotency
key, timeout-doesn't-block + retry-once, 4xx-no-retry, 5xx-retry, unconfigured
and partial-config no-ops. Full middleware_test suite green (52 passed).
The SPA "Sign in with Hanzo" button hand-rolled an authorize URL
(IAM /login?redirect_uri=<studio root>), which the IAM default login app
(hanzo-console) rejected because the studio root is not in its allowed
redirect-URI list -- blocking all customer login.
Give the flow exactly one entry point: the button now links to the studio
server's own /login route, which runs the middleware's OIDC Authorization
Code flow (client_id=hanzo-studio, redirect_uri=<origin>/callback, PKCE +
signed state). /callback is already an allowed redirect URI, so login
completes and lands on /. Removes the client-side URL builder and the dead
{{IAM_URL}} templating.
Tests: /login is public; middleware exposes handle_login; unauth /login
builds the hanzo-studio authorize URL to /callback; post-login return path is /.
Co-authored-by: hanzo-dev <dev@hanzo.ai>
Move the workspace-tab seeder out of a hand-edited index.html and into a
dedicated branding patcher wired into apply-branding.sh, so it survives a
frontend-package reinstall and re-applies on every branding pass.
Root cause the seeder addresses: the frontend restores open tabs from
localStorage via getStorageValue/setStorageValue, whose clientId-suffixed
sessionStorage layer is read BEFORE plain localStorage. A previously-saved
1-tab shadow (Comfy.OpenWorkflowsPaths:<clientId>) masked the seed, and the
old guard was burned at v2 so existing profiles never re-seeded.
Fix: bump guard v2->v3 (existing profiles re-seed once) and purge the stale
sessionStorage shadows on re-seed so the fresh localStorage seed wins even on
a same-tab reload. Clean-profile boot now opens all 25 fashion workflows
(alongside ComfyUI's own default tab).
SqlitePromptQueue (middleware/tasks_queue.py): drop-in for PromptQueue backed
by one SQLite file (stdlib sqlite3, WAL, zero external processes). Opt in with
STUDIO_QUEUE_DB; precedence STUDIO_QUEUE_DB > STUDIO_PERSIST_QUEUE > memory so
local default is unchanged. The put/get/task_done seam maps to a durable job
state machine: idempotent submit (INSERT OR IGNORE on prompt_id), exactly-once
claim (BEGIN IMMEDIATE), retry-on-error, and crash-recovery via a lease + reap
that re-runs an abandoned claim (idempotent — SaveImage suffixes increment).
Multiple Studio processes on the same DB compete safely.
Engine selector (middleware/engine_selector.py): GET /v1/engines lists execution
targets for the org (local + registered compute_config workers), PUT
/v1/engines/default sets a per-org default stored on ComputeProfile; route_prompt
honors it (local = unchanged). Leased cloud machines are a future engine class
(interface stubbed). Frontend picker is a TODO.
docs/federation.md rewritten to the durable-queue reality with one future
paragraph (remote Tasks backend + Go/unified-binary HIP-0106 migration + leased
machines). Tests: middleware_test/{tasks_queue,engine_selector}_test.py, incl.
crash-recovery across queue instances.
ci@v1 defaults runs-on to [self-hosted,linux,amd64]; ARC scale sets are
matched by installation name, so the job sat queued with no runner. Pin the
runner to the hanzo build pool (same as cloud/release.yml).
- Remove .github/workflows/deploy.yml — the canonical build+test+deploy path
is cicd.yml (imports hanzoai/ci build.yml@v1, reads hanzo.yml). One way to
build. The old Docker workflow called hanzoai/.github docker-build.yml and
failed at 0s.
- prompt_queue_persist_test.py: drop the 4 diagnostic prints (T201); pytest
reports pass/fail and the standalone runner raises on failure.
- zen_video_adapter.py: remove unused json/asyncio imports (F401).
Root cause of the "Event loop stopped before Future completed" crash that
wiped the in-memory queue/history (seen 3x today): the fork's _graceful_shutdown
handler (added in a45a441) called event_loop.stop(). The server coroutine passed
to run_until_complete() never completes on its own, so stopping the loop makes
run_until_complete() raise RuntimeError; the surrounding try only caught
KeyboardInterrupt, so the process died non-zero with a traceback and discarded
the queue. The dirty exit also failed to release :8188 promptly, and a duplicate
transient/user `hanzo-studio.service` racing the canonical studio.service turned
that into an EADDRINUSE restart storm.
main.py:
- handler is now idempotent, flags the queue closed, snapshots it, then stops
the loop; the outer try treats RuntimeError('Event loop stopped before Future
completed.') as the expected graceful-exit path (logs "Server stopped by
shutdown signal", exits 0). Verified live in journald.
execution.py:
- opt-in crash-durable PromptQueue behind STUDIO_PERSIST_QUEUE=1. Pending +
in-flight prompts are snapshotted atomically to user/queue_snapshot.json on
every change and re-queued on boot. Best-effort: all I/O is guarded so it
never affects the render path; inert when the flag is unset.
ops/systemd/studio.service:
- enable STUDIO_PERSIST_QUEUE=1 on the single canonical unit.
scripts/studio-service-swap.sh:
- batch-safe reconcile+apply: waits for an empty queue, removes any stray
duplicate unit, reinstalls the canonical unit, restarts, health-checks.
docs/ops-crash-and-service.md:
- topology (one canonical unit), how to read crash history, the fix, persistence.
tests-unit/execution_test/prompt_queue_persist_test.py:
- round-trip, inert-when-disabled, and corrupt-row-skipping tests.
Auth (middleware/iam_auth_middleware.py): local JWKS JWT validation against
Hanzo IAM (signature + exp + iss + aud=hanzo-studio; org from the `owner`
claim). Standard OIDC Authorization Code flow with PKCE and a /callback
session cookie for the standalone studio.hanzo.ai app; anonymous localhost
bypass unchanged. verify_jwt() extracted as a pure, unit-tested function.
Tenancy (folder_paths.py, app/user_manager.py): make get_public_user_directory
org-rooted so multi-tenant userdata resolves under user/orgs/{org}/... (it
previously always returned None). Scope execution outputs to output/orgs/{org}/
via set_execution_org(), bound by the prompt worker from the queue item org_id.
Also block a token subject from escaping into a System User namespace.
Federation (middleware/{worker_client,prompt_router}.py, server.py): move the
worker registry + execute surface to /v1/workers/register, /v1/workers,
/v1/worker/execute; add X-Worker-Token coordinator trust. Outbound pull channel
for NAT'd local boxes (GB10) specced in docs/federation.md.
Deploy: hanzo.yml + .github/workflows/cicd.yml (hanzoai/ci reusable) build
ghcr.io/hanzoai/studio and roll the studio operator Service CR at
studio.hanzo.ai. PyJWT[crypto] pinned.
Tests: tests-unit/middleware_test/iam_auth_test.py (RS256 sign/verify, state
CSRF, path exemptions) + folder_paths_test/org_scoping_test.py; 140 passing
across the auth/tenancy/user-manager/prompt-server suites.
Anoeses-style PDP pattern: product_<design> = garment-only ghost presentation on pure white (base image); hover_<design> = model in dynamic pose on subtle white studio (rollover image). Workspace: index.html seeds Comfy.OpenWorkflowsPaths with all 25 fashion tabs on first load (set-once, localStorage); Comfy.Workflow.Persist on. Seeder added to apply-branding.sh replay path.
Native TextEncodeQwenImageEditPlus path wired against the live /object_info
schema: LoadImage -> Plus encoder (image1+vae) + VAEEncode -> KSampler
(30 steps, cfg 2.5, euler/simple) -> VAEDecode -> SaveImage swimwear/antje.
Models installed under models/{diffusion_models,text_encoders,vae} (merged
from the Qwen-Image-Edit-2511 diffusers shards, gitignored). 8 of Antje's
CAD drawings preloaded in input/ and enumerated by LoadImage.
- Sidebar logo: replace the inline ComfyLogo Vue component (top-left mark,
compiled into the JS bundle, missed by the file-asset pass) with the Hanzo
mark via branding/patch_sidebar_logo.py (keyed off the stable __name:`ComfyLogo`).
- Theme: branding/hanzo-theme.css forces a pure-black shell + Hanzo-purple
accents (AA contrast), injected as the last stylesheet; patch_theme.py also
rewrites the dark palette's canvas clear color to #000000 (CSS can't reach the
litegraph <canvas>).
- Favicon: make_progress_favicons.py renders the 10 progress frames with the
Hanzo mark + purple ring, fixing the tab reverting to the Comfy mark during
generations. Committed PNGs like favicon.ico (Docker needs no rasterizer).
- Tab title: rebrand the runtime ' - ComfyUI' title suffix the string pass missed.
- apply-branding.sh replays all of the above on frontend-package upgrades.
- Rename fork-owned update_comfyui{,_stable}.bat -> update{,_stable}.bat.
Black background, white text, opacity-based visual hierarchy,
pill-shaped CTA button, Inter font, feature cards with subtle
white/5-10% backgrounds. Removes red accent entirely.
- Compute profiles: per-org CPU/GPU config stored in user dir with
thread-safe CRUD and atomic file writes
- Prompt routing: forwards GPU prompts to registered workers, falls
back to local execution when no worker available
- Worker mode: --worker-mode flag runs Studio as headless executor
that registers with coordinator via heartbeats
- Visor integration: provision/terminate GPU VMs on AWS (T4/V100/A100/H100)
with shell-safe startup scripts
- Login page: unauthenticated browsers see branded login gate with
"Sign in with Hanzo" button instead of broken ComfyUI interface
- 6 new API endpoints under /api/compute/*
- Fix branding patcher to target actual upstream logo filenames
(comfy-logo-*.svg, not studio-logo-*.svg)
- Add more display string patterns to patch_frontend.py
- Add IAM authentication middleware (hanzo.id/api/get-account)
with localhost bypass and browser login redirect
- Add multi-tenant org-scoped storage (folder_paths.get_org_*)
with per-org output/input/temp/user/models directories
- Add CLI flags: --enable-iam-auth, --iam-url, --multi-tenant,
--org-id, --no-localhost-bypass
- Support env var toggles for Docker: STUDIO_ENABLE_IAM_AUTH,
STUDIO_MULTI_TENANT, STUDIO_ORG_ID, STUDIO_IAM_URL
- Integrate IAM user context into UserManager and server routes
- Remove bare 'Comfy-Org' and 'ComfyOrg' from plain text replacements
(was corrupting i18n keys like OpenComfy-OrgDiscord)
- Only replace Comfy-Org/ComfyOrg in multi-word display phrases
- Add regex patterns for quoted string value contexts
- Add standalone quoted value patterns ("ComfyUI" → "Hanzo Studio")
- Add Comfy Cloud → Hanzo Cloud display text replacement
- Replace all user-facing "ComfyUI" strings with "Hanzo Studio"
- Update Help menu links: Discord → discord.gg/hanzoai, GitHub → hanzoai/studio
- Replace comfy.org URLs with hanzo.ai equivalents
- Update support email to support@hanzo.ai
- Rewrite branding script to patch ALL JS bundles comprehensively
- Update default filename prefixes from ComfyUI to HanzoStudio
- Fix CI to trigger on main branch (not hanzo/main)
- Update pyproject.toml metadata (name, URLs)
- Replace ComfyUI logo with Hanzo geometric H mark
- Replace favicon with Hanzo H favicon
- Patch index.html title, PWA manifest, and JS bundles
- Update AboutPanel and OrgHeader references
- Branding applied as Docker build step after pip install
* chore: tune CodeRabbit config to limit review scope and disable for drafts
- Add tone_instructions to focus only on newly introduced issues
- Add global path_instructions entry to ignore pre-existing issues in moved/reformatted code
- Disable draft PR reviews (drafts: false) and add WIP title keywords
- Disable ruff tool to prevent linter-based outside-diff-range comments
Addresses feedback from maintainers about CodeRabbit flagging pre-existing
issues in code that was merely moved or de-indented (e.g., PR #12557),
which can discourage community contributions and cause scope creep.
Amp-Thread-ID: https://ampcode.com/threads/T-019c82de-0481-7253-ad42-20cb595bb1ba
* chore: add 'DO NOT MERGE' to ignore_title_keywords
Amp-Thread-ID: https://ampcode.com/threads/T-019c82de-0481-7253-ad42-20cb595bb1ba
On Windows, Python defaults to cp1252 encoding when no encoding is
specified. JSON files containing UTF-8 characters (e.g., non-ASCII
characters) cause UnicodeDecodeError when read with cp1252.
This fixes the error that occurs when loading blueprint subgraphs
on Windows systems.
https://claude.ai/code/session_014WHi3SL9Gzsi3U6kbSjbSb
Co-authored-by: Claude <noreply@anthropic.com>
Integrate comfy-aimdo 0.2 which takes a different approach to
installing the memory allocator hook. Instead of using the complicated
and buggy pytorch MemPool+CudaPluggableAlloctor, cuda is directly hooked
making the process much more transparent to both comfy and pytorch. As
far as pytorch knows, aimdo doesnt exist anymore, and just operates
behind the scenes.
Remove all the mempool setup stuff for dynamic_vram and bump the
comfy-aimdo version. Remove the allocator object from memory_management
and demote its use as an enablment check to a boolean flag.
Comfy-aimdo 0.2 also support the pytorch cuda async allocator, so
remove the dynamic_vram based force disablement of cuda_malloc and
just go back to the old settings of allocators based on command line
input.
Add 24 non-cloud essential blueprints from comfyui-wiki/Subgraph-Blueprints.
These cover common workflows: text/image/video generation, editing,
inpainting, outpainting, upscaling, depth maps, pose, captioning, and more.
Cloud-only blueprints (5) are excluded and will be added once
client-side distribution filtering lands.
Amp-Thread-ID: https://ampcode.com/threads/T-019c6f43-6212-7308-bea6-bfc35a486cbf
gemini-3-pro-image-preview nondeterministically returns image/jpeg
instead of image/png. get_image_from_response() hardcoded
get_parts_by_type(response, "image/png"), silently dropping JPEG
responses and falling back to torch.zeros (all-black output).
Add _mime_matches() helper using fnmatch for glob-style MIME matching.
Change get_image_from_response() to request "image/*" so any image
format returned by the API is correctly captured.
This check was far too broad and the dtype is not a reliable indicator
of wanting the requant (as QT returns the compute dtype as the dtype).
So explictly plumb whether fp8mm wants the requant or not.
* lora: add weight shape calculations.
This lets the loader know if a lora will change the shape of a weight
so it can take appropriate action.
* MPDynamic: force load flux img_in weight
This weight is a bit special, in that the lora changes its geometry.
This is rather unique, not handled by existing estimate and doesn't
work for either offloading or dynamic_vram.
Fix for dynamic_vram as a special case. Ideally we can fully precalculate
these lora geometry changes at load time, but just get these models
working first.
* lora_extract: Add a trange
If you bite off more than your GPU can chew, this kinda just hangs.
Give a rough indication of progress counting the weights in a trange.
* lora_extract: Support on-the-fly patching
Use the on-the-fly approach from the regular model saving logic for
lora extraction too. Switch off force_cast_weights accordingly.
This gets extraction working in dynamic vram while also supporting
extraction on GPU offloaded.
If you have memory issues you can try disabling the smart memory management by running comfyui with:
If you have memory issues you can try disabling the smart memory management by running Hanzo Studio with:
run_amd_gpu_disable_smart_memory.bat
IF YOU GET A RED ERROR IN THE UI MAKE SURE YOU HAVE A MODEL/CHECKPOINT IN: ComfyUI\models\checkpoints
IF YOU GET A RED ERROR IN THE UI MAKE SURE YOU HAVE A MODEL/CHECKPOINT IN: HanzoStudio\models\checkpoints
You can download the stable diffusion XL one from: https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/sd_xl_base_1.0_0.9vae.safetensors
RECOMMENDED WAY TO UPDATE:
To update the ComfyUI code: update\update_comfyui.bat
To update the Hanzo Studio code: update\update_studio.bat
TO SHARE MODELS BETWEEN COMFYUI AND ANOTHER UI:
In the ComfyUI directory you will find a file: extra_model_paths.yaml.example
TO SHARE MODELS BETWEEN HANZO STUDIO AND ANOTHER UI:
In the HanzoStudio directory you will find a file: extra_model_paths.yaml.example
Rename this file to: extra_model_paths.yaml and edit it with your favorite text editor.
IF YOU GET A RED ERROR IN THE UI MAKE SURE YOU HAVE A MODEL/CHECKPOINT IN: ComfyUI\models\checkpoints
IF YOU GET A RED ERROR IN THE UI MAKE SURE YOU HAVE A MODEL/CHECKPOINT IN: HanzoStudio\models\checkpoints
You can download the stable diffusion 1.5 one from: https://huggingface.co/Comfy-Org/stable-diffusion-v1-5-archive/blob/main/v1-5-pruned-emaonly-fp16.safetensors
You can download the stable diffusion 1.5 one from: https://huggingface.co/hanzoai/stable-diffusion-v1-5-archive/blob/main/v1-5-pruned-emaonly-fp16.safetensors
RECOMMENDED WAY TO UPDATE:
To update the ComfyUI code: update\update_comfyui.bat
To update the Hanzo Studio code: update\update_studio.bat
To update ComfyUI with the python dependencies, note that you should ONLY run this if you have issues with python dependencies.
update\update_comfyui_and_python_dependencies.bat
To update Hanzo Studio with the python dependencies, note that you should ONLY run this if you have issues with python dependencies.
update\update_studio_and_python_dependencies.bat
TO SHARE MODELS BETWEEN COMFYUI AND ANOTHER UI:
In the ComfyUI directory you will find a file: extra_model_paths.yaml.example
TO SHARE MODELS BETWEEN HANZO STUDIO AND ANOTHER UI:
In the HanzoStudio directory you will find a file: extra_model_paths.yaml.example
Rename this file to: extra_model_paths.yaml and edit it with your favorite text editor.
echo If you see this and ComfyUI did not start try updating your Nvidia Drivers to the latest. If you get a c10.dll error you need to install vc redist that you can find: https://aka.ms/vc14/vc_redist.x64.exe
echo If you see this and Hanzo Studio did not start try updating your Nvidia Drivers to the latest. If you get a c10.dll error you need to install vc redist that you can find: https://aka.ms/vc14/vc_redist.x64.exe
echo If you see this and ComfyUI did not start try updating your Nvidia Drivers to the latest. If you get a c10.dll error you need to install vc redist that you can find: https://aka.ms/vc14/vc_redist.x64.exe
echo If you see this and Hanzo Studio did not start try updating your Nvidia Drivers to the latest. If you get a c10.dll error you need to install vc redist that you can find: https://aka.ms/vc14/vc_redist.x64.exe
echo If you see this and ComfyUI did not start try updating your Nvidia Drivers to the latest. If you get a c10.dll error you need to install vc redist that you can find: https://aka.ms/vc14/vc_redist.x64.exe
echo If you see this and Hanzo Studio did not start try updating your Nvidia Drivers to the latest. If you get a c10.dll error you need to install vc redist that you can find: https://aka.ms/vc14/vc_redist.x64.exe
tone_instructions:"Only comment on issues introduced by this PR's changes. Do not flag pre-existing problems in moved, re-indented, or reformatted code."
reviews:
profile:"chill"
request_changes_workflow:false
high_level_summary:false
poem:false
review_status:false
review_details:false
commit_status:true
collapse_walkthrough:true
changed_files_summary:false
sequence_diagrams:false
estimate_code_review_effort:false
assess_linked_issues:false
related_issues:false
related_prs:false
suggested_labels:false
auto_apply_labels:false
suggested_reviewers:false
auto_assign_reviewers:false
in_progress_fortune:false
enable_prompt_for_ai_agents:true
path_filters:
- "!studio_api_nodes/apis/**"
- "!**/generated/*.pyi"
- "!.ci/**"
- "!script_examples/**"
- "!**/__pycache__/**"
- "!**/*.ipynb"
- "!**/*.png"
- "!**/*.bat"
path_instructions:
- path:"**"
instructions:|
IMPORTANT: Only comment on issues directly introduced by this PR's code changes.
Do NOT flag pre-existing issues in code that was merely moved, re-indented,
de-indented, or reformatted without logic changes. If code appears in the diff
only due to whitespace or structural reformatting (e.g., removing a `with:` block),
treat it as unchanged. Contributors should not feel obligated to address
pre-existing issues outside the scope of their contribution.
- path:"studio/**"
instructions:|
Core ML/diffusion engine. Focus on:
- Backward compatibility (breaking changes affect all custom nodes)
- Memory management and GPU resource handling
- Performance implications in hot paths
- Thread safety for concurrent execution
- path:"studio_api_nodes/**"
instructions:|
Third-party API integration nodes. Focus on:
- No hardcoded API keys or secrets
- Proper error handling for API failures (timeouts, rate limits, auth errors)
- Correct Pydantic model usage
- Security of user data passed to external APIs
- path:"studio_extras/**"
instructions:|
Community-contributed extra nodes. Focus on:
- Consistency with node patterns (INPUT_TYPES, RETURN_TYPES, FUNCTION, CATEGORY)
- No breaking changes to existing node interfaces
- path:"studio_execution/**"
instructions:|
Execution engine (graph execution, caching, jobs). Focus on:
description:"Something is broken inside of ComfyUI. (Do not use this if you're just having issues and need help, or if the issue relates to a custom node)"
description:"Something is broken inside of Hanzo Studio. (Do not use this if you're just having issues and need help, or if the issue relates to a custom node)"
labels:["Potential Bug"]
body:
- type:markdown
@@ -7,23 +7,23 @@ body:
value:|
Before submitting a **Bug Report**, please ensure the following:
- **1:** You are running the latest version of ComfyUI.
- **2:** You have your ComfyUI logs and relevant workflow on hand and will post them in this bug report.
- **1:** You are running the latest version of Hanzo Studio.
- **2:** You have your Hanzo Studio logs and relevant workflow on hand and will post them in this bug report.
- **3:** You confirmed that the bug is not caused by a custom node. You can disable all custom nodes by passing
`--disable-all-custom-nodes` command line argument. If you have custom node try updating them to the latest version.
- **4:** This is an actual bug in ComfyUI, not just a support question. A bug is when you can specify exact
- **4:** This is an actual bug in Hanzo Studio, not just a support question. A bug is when you can specify exact
steps to replicate what went wrong and others will be able to repeat your steps and see the same issue happen.
## Very Important
Please make sure that you post ALL your ComfyUI logs in the bug report. A bug report without logs will likely be ignored.
Please make sure that you post ALL your Hanzo Studio logs in the bug report. A bug report without logs will likely be ignored.
- type:checkboxes
id:custom-nodes-test
attributes:
label:Custom Node Testing
description:Please confirm you have tried to reproduce the issue with all custom nodes disabled.
options:
- label:I have tried disabling custom nodes and the issue persists (see [how to disable custom nodes](https://docs.comfy.org/troubleshooting/custom-node-issues#step-1%3A-test-with-all-custom-nodes-disabled) if you need help)
- label:I have tried disabling custom nodes and the issue persists (see [how to disable custom nodes](https://docs.hanzo.ai/troubleshooting/custom-node-issues#step-1%3A-test-with-all-custom-nodes-disabled) if you need help)
required:false
- type:textarea
attributes:
@@ -40,7 +40,7 @@ body:
- type:textarea
attributes:
label:Steps to Reproduce
description:"Describe how to reproduce the issue. Please be sure to attach a workflow JSON or PNG, ideally one that doesn't require custom nodes to test. If the bug open happens when certain custom nodes are used, most likely that custom node is what has the bug rather than ComfyUI, in which case it should be reported to the node's author."
description:"Describe how to reproduce the issue. Please be sure to attach a workflow JSON or PNG, ideally one that doesn't require custom nodes to test. If the bug open happens when certain custom nodes are used, most likely that custom node is what has the bug rather than Hanzo Studio, in which case it should be reported to the node's author."
description:"You have an idea for something new you would like to see added to ComfyUI's core."
description:"You have an idea for something new you would like to see added to Hanzo Studio's core."
labels:["Feature"]
body:
- type:markdown
@@ -7,11 +7,11 @@ body:
value:|
Before submitting a **Feature Request**, please ensure the following:
**1:** You are running the latest version of ComfyUI.
**1:** You are running the latest version of Hanzo Studio.
**2:** You have looked to make sure there is not already a feature that does what you need, and there is not already a Feature Request listed for the same idea.
**3:** This is something that makes sense to add to ComfyUI Core, and wouldn't make more sense as a custom node.
**3:** This is something that makes sense to add to Hanzo Studio Core, and wouldn't make more sense as a custom node.
If unsure, ask on the [ComfyUI Matrix Space](https://app.element.io/#/room/%23comfyui_space%3Amatrix.org) or the [Comfy Org Discord](https://discord.gg/comfyorg) first.
If unsure, ask on the [Hanzo AI Discord](https://discord.gg/hanzoai) first.
Before submitting a **User Report** issue, please ensure the following:
**1:** You are running the latest version of ComfyUI.
**1:** You are running the latest version of Hanzo Studio.
**2:** You have made an effort to find public answers to your question before asking here. In other words, you googled it first, and scrolled through recent help topics.
If unsure, ask on the [ComfyUI Matrix Space](https://app.element.io/#/room/%23comfyui_space%3Amatrix.org) or the [Comfy Org Discord](https://discord.gg/comfyorg) first.
If unsure, ask on the [Hanzo AI Discord](https://discord.gg/hanzoai) first.
- type:checkboxes
id:custom-nodes-test
attributes:
label:Custom Node Testing
description:Please confirm you have tried to reproduce the issue with all custom nodes disabled.
options:
- label:I have tried disabling custom nodes and the issue persists (see [how to disable custom nodes](https://docs.comfy.org/troubleshooting/custom-node-issues#step-1%3A-test-with-all-custom-nodes-disabled) if you need help)
- label:I have tried disabling custom nodes and the issue persists (see [how to disable custom nodes](https://docs.hanzo.ai/troubleshooting/custom-node-issues#step-1%3A-test-with-all-custom-nodes-disabled) if you need help)
body: '(Automated Bot Message) CI Tests are running, you can view the results at https://ci.comfy.org/?branch=${{ github.event.pull_request.number }}%2Fmerge'
body: '(Automated Bot Message) CI Tests are running, you can view the results at https://ci.hanzo.ai/?branch=${{ github.event.pull_request.number }}%2Fmerge'
Welcome, and thank you for your interest in contributing to ComfyUI!
Welcome, and thank you for your interest in contributing to Hanzo Studio!
There are several ways in which you can contribute, beyond writing code. The goal of this document is to provide a high-level overview of how you can get involved.
## Asking Questions
Have a question? Instead of opening an issue, please ask on [Discord](https://comfy.org/discord) or [Matrix](https://app.element.io/#/room/%23comfyui_space%3Amatrix.org) channels. Our team and the community will help you.
Have a question? Instead of opening an issue, please ask on [Discord](https://discord.gg/hanzoai) first. Our team and the community will help you.
## Providing Feedback
@@ -16,24 +16,24 @@ See the `#bug-report`, `#feature-request` and `#feedback` channels on Discord.
## Reporting Issues
Have you identified a reproducible problem in ComfyUI? Do you have a feature request? We want to hear about it! Here's how you can report your issue as effectively as possible.
Have you identified a reproducible problem in Hanzo Studio? Do you have a feature request? We want to hear about it! Here's how you can report your issue as effectively as possible.
### Look For an Existing Issue
Before you create a new issue, please do a search in [open issues](https://github.com/comfyanonymous/ComfyUI/issues) to see if the issue or feature request has already been filed.
Before you create a new issue, please do a search in [open issues](https://github.com/hanzoai/studio/issues) to see if the issue or feature request has already been filed.
If you find your issue already exists, make relevant comments and add your [reaction](https://github.com/blog/2119-add-reactions-to-pull-requests-issues-and-comments). Use a reaction in place of a "+1" comment:
* 👍 - upvote
* 👎 - downvote
* upvote
* downvote
If you cannot find an existing issue that describes your bug or feature, create a new issue. We have an issue template in place to organize new issues.
### Creating Pull Requests
* Please refer to the article on [creating pull requests](https://github.com/comfyanonymous/ComfyUI/wiki/How-to-Contribute-Code) and contributing to this project.
* Please refer to the article on [creating pull requests](https://github.com/hanzoai/studio/wiki/How-to-Contribute-Code) and contributing to this project.
To represent these derived datatypes, ComfyUI uses a subclass of torch.Tensor to implements these using the `QuantizedTensor` class found in `comfy/quant_ops.py`
To represent these derived datatypes, Hanzo Studio uses a subclass of torch.Tensor to implements these using the `QuantizedTensor` class found in `studio/quant_ops.py`
A `Layout` class defines how a specific quantization format behaves:
- Required parameters
@@ -47,7 +47,7 @@ A `Layout` class defines how a specific quantization format behaves:
- De-Quantize method
```python
fromcomfy.quant_opsimportQuantizedLayout
fromstudio.quant_opsimportQuantizedLayout
classMyLayout(QuantizedLayout):
@classmethod
@@ -67,7 +67,7 @@ The first is a **generic registry** that handles operations common to all quanti
The second registry is layout-specific and allows to implement fast-paths like nn.Linear.
@@ -80,7 +80,7 @@ For any unsupported operation, QuantizedTensor will fallback to call `dequantize
### Mixed Precision
The `MixedPrecisionOps` class (lines 542-648 in `comfy/ops.py`) enables per-layer quantization decisions, allowing different layers in a model to use different precisions. This is activated when a model config contains a `layer_quant_config` dictionary that specifies which layers should be quantized and how.
The `MixedPrecisionOps` class (lines 542-648 in `studio/ops.py`) enables per-layer quantization decisions, allowing different layers in a model to use different precisions. This is activated when a model config contains a `layer_quant_config` dictionary that specifies which layers should be quantized and how.
**Architecture:**
@@ -125,7 +125,7 @@ We define 4 possible scaling parameters that should cover most recipes in the ne
</div>
ComfyUI lets you design and execute advanced stable diffusion pipelines using a graph/nodes/flowchart based interface. Available on Windows, Linux, and macOS.
Hanzo Studio lets you design and execute advanced stable diffusion pipelines using a graph/nodes/flowchart based interface. Available on Windows, Linux, and macOS.
## Upstream & License
Hanzo Studio is a fork of [**ComfyUI**](https://github.com/comfyanonymous/ComfyUI) by comfyanonymous and the ComfyUI contributors — full credit to the upstream project for the engine this builds on. ComfyUI is licensed under the **GNU General Public License v3.0 (GPL-3.0)**, and this fork stays under the same license. All modifications in this fork are likewise GPL-3.0.
See [`LICENSE`](LICENSE) for the full license text and [`NOTICE`](NOTICE) for the fork's attribution and a summary of what it changes (branding, Hanzo Engine integration nodes, and fashion workflow templates).
- Loading full workflows (with seeds) from generated PNG, WebP and FLAC files.
- Saving/Loading workflows as Json files.
- Nodes interface can be used to create complex workflows like one for [Hires fix](https://comfyanonymous.github.io/ComfyUI_examples/2_pass_txt2img/) or much more advanced ones.
- Works fully offline: core will never download anything unless you want to.
- Optional API nodes to use paid models from external providers through the online [Comfy API](https://docs.comfy.org/tutorials/api-nodes/overview) disable with: `--disable-api-nodes`
- Optional API nodes to use paid models from external providers through the online API. Disable with: `--disable-api-nodes`
- [Config file](extra_model_paths.yaml.example) to set the search paths for models.
Workflow examples can be found on the [Examples page](https://comfyanonymous.github.io/ComfyUI_examples/)
## Release Process
ComfyUI follows a weekly release cycle targeting Monday but this regularly changes because of model releases or large changes to the codebase. There are three interconnected repositories:
- Weekly frontend updates are merged into the core repository
- Features are frozen for the upcoming core release
- Development continues for the next release cycle
## Shortcuts
| Keybind | Explanation |
@@ -173,39 +133,6 @@ ComfyUI follows a weekly release cycle targeting Monday but this regularly chang
# Installing
## Windows Portable
There is a portable standalone build for Windows that should work for running on Nvidia GPUs or for running on your CPU only on the [releases page](https://github.com/comfyanonymous/ComfyUI/releases).
### [Direct link to download](https://github.com/comfyanonymous/ComfyUI/releases/latest/download/ComfyUI_windows_portable_nvidia.7z)
Simply download, extract with [7-Zip](https://7-zip.org) or with the windows explorer on recent windows versions and run. For smaller models you normally only need to put the checkpoints (the huge ckpt/safetensors files) in: ComfyUI\models\checkpoints but many of the larger models have multiple files. Make sure to follow the instructions to know which subfolder to put them in ComfyUI\models\
If you have trouble extracting it, right click the file -> properties -> unblock
The portable above currently comes with python 3.13 and pytorch cuda 13.0. Update your Nvidia drivers if it doesn't start.
#### Alternative Downloads:
[Experimental portable for AMD GPUs](https://github.com/comfyanonymous/ComfyUI/releases/latest/download/ComfyUI_windows_portable_amd.7z)
[Portable with pytorch cuda 12.8 and python 3.12](https://github.com/comfyanonymous/ComfyUI/releases/latest/download/ComfyUI_windows_portable_nvidia_cu128.7z).
[Portable with pytorch cuda 12.6 and python 3.12](https://github.com/comfyanonymous/ComfyUI/releases/latest/download/ComfyUI_windows_portable_nvidia_cu126.7z) (Supports Nvidia 10 series and older GPUs).
#### How do I share models between another UI and ComfyUI?
See the [Config file](extra_model_paths.yaml.example) to set the search paths for models. In the standalone windows build you can find this file in the ComfyUI directory. Rename this file to extra_model_paths.yaml and edit it with your favorite text editor.
You can install and start ComfyUI using comfy-cli:
```bash
pip install comfy-cli
comfy install
```
## Manual Install (Windows, Linux)
Python 3.14 works but some custom nodes may have issues. The free threaded variant works but some dependencies will enable the GIL so it's not fully supported.
@@ -229,9 +156,9 @@ AMD users can install rocm and pytorch with pip if you don't have it already ins
### AMD GPUs (Experimental: Windows and Linux), RDNA 3, 3.5 and 4 only.
@@ -282,24 +209,24 @@ And install it again with the command above.
### Dependencies
Install the dependencies by opening your terminal inside the ComfyUI folder and:
Install the dependencies by opening your terminal inside the Hanzo Studio folder and:
```pip install -r requirements.txt```
After this you should have everything installed and can proceed to running ComfyUI.
After this you should have everything installed and can proceed to running Hanzo Studio.
### Others:
#### Apple Mac silicon
You can install ComfyUI in Apple Mac silicon (M1 or M2) with any recent macOS version.
You can install Hanzo Studio in Apple Mac silicon (M1 or M2) with any recent macOS version.
1. Install pytorch nightly. For instructions, read the [Accelerated PyTorch training on Mac](https://developer.apple.com/metal/pytorch/) Apple Developer guide (make sure to install the latest pytorch nightly).
1. Follow the [ComfyUI manual installation](#manual-install-windows-linux) instructions for Windows and Linux.
1. Install the ComfyUI [dependencies](#dependencies). If you have another Stable Diffusion UI [you might be able to reuse the dependencies](#i-already-have-another-ui-for-stable-diffusion-installed-do-i-really-have-to-install-all-of-these-dependencies).
1. Launch ComfyUI by running `python main.py`
1. Follow the [Hanzo Studio manual installation](#manual-install-windows-linux) instructions for Windows and Linux.
1. Install the Hanzo Studio [dependencies](#dependencies). If you have another Stable Diffusion UI you might be able to reuse the dependencies.
1. Launch Hanzo Studio by running `python main.py`
> **Note**: Remember to add your models, VAE, LoRAs etc. to the corresponding Comfy folders, as discussed in [ComfyUI manual installation](#manual-install-windows-linux).
> **Note**: Remember to add your models, VAE, LoRAs etc. to the corresponding folders, as discussed in [Hanzo Studio manual installation](#manual-install-windows-linux).
#### Ascend NPUs
@@ -308,7 +235,7 @@ For models compatible with Ascend Extension for PyTorch (torch_npu). To get star
1. Begin by installing the recommended or newer kernel version for Linux as specified in the Installation page of torch-npu, if necessary.
2. Proceed with the installation of Ascend Basekit, which includes the driver, firmware, and CANN, following the instructions provided for your specific platform.
3. Next, install the necessary packages for torch-npu by adhering to the platform-specific instructions on the [Installation](https://ascend.github.io/docs/sources/pytorch/install.html#pytorch) page.
4. Finally, adhere to the [ComfyUI manual installation](#manual-install-windows-linux) guide for Linux. Once all components are installed, you can run ComfyUI as described earlier.
4. Finally, adhere to the [Hanzo Studio manual installation](#manual-install-windows-linux) guide for Linux. Once all components are installed, you can run Hanzo Studio as described earlier.
#### Cambricon MLUs
@@ -316,19 +243,19 @@ For models compatible with Cambricon Extension for PyTorch (torch_mlu). Here's a
1. Install the Cambricon CNToolkit by adhering to the platform-specific instructions on the [Installation](https://www.cambricon.com/docs/sdk_1.15.0/cntoolkit_3.7.2/cntoolkit_install_3.7.2/index.html)
2. Next, install the PyTorch(torch_mlu) following the instructions on the [Installation](https://www.cambricon.com/docs/sdk_1.15.0/cambricon_pytorch_1.17.0/user_guide_1.9/index.html)
3. Launch ComfyUI by running `python main.py`
3. Launch Hanzo Studio by running `python main.py`
#### Iluvatar Corex
For models compatible with Iluvatar Extension for PyTorch. Here's a step-by-step guide tailored to your platform and installation method:
1. Install the Iluvatar Corex Toolkit by adhering to the platform-specific instructions on the [Installation](https://support.iluvatar.com/#/DocumentCentre?id=1&nameCenter=2&productId=520117912052801536)
2. Launch ComfyUI by running `python main.py`
2. Launch Hanzo Studio by running `python main.py`
**ComfyUI-Manager** is an extension that allows you to easily install, update, and manage custom nodes for ComfyUI.
The **Manager** extension allows you to easily install, update, and manage custom nodes for Hanzo Studio.
### Setup
@@ -337,7 +264,7 @@ For models compatible with Iluvatar Extension for PyTorch. Here's a step-by-step
pip install -r manager_requirements.txt
```
2. Enable the manager with the `--enable-manager` flag when running ComfyUI:
2. Enable the manager with the `--enable-manager` flag when running Hanzo Studio:
```bash
python main.py --enable-manager
```
@@ -346,7 +273,7 @@ For models compatible with Iluvatar Extension for PyTorch. Here's a step-by-step
| Flag | Description |
|------|-------------|
| `--enable-manager` | Enable ComfyUI-Manager |
| `--enable-manager` | Enable the Manager |
| `--enable-manager-legacy-ui` | Use the legacy manager UI instead of the new UI (requires `--enable-manager`) |
| `--disable-manager-ui` | Disable the manager UI and endpoints while keeping background features like security checks and scheduled installation completion (requires `--enable-manager`) |
@@ -365,7 +292,7 @@ For AMD 7600 and maybe other RDNA3 cards: ```HSA_OVERRIDE_GFX_VERSION=11.0.0 pyt
### AMD ROCm Tips
You can enable experimental memory efficient attention on recent pytorch in ComfyUI on some AMD GPUs using this command, it should already be enabled by default on RDNA3. If this improves speed for you on latest pytorch on your GPU please report it so that I can enable it by default.
You can enable experimental memory efficient attention on recent pytorch on some AMD GPUs using this command, it should already be enabled by default on RDNA3. If this improves speed for you on latest pytorch on your GPU please report it so that we can enable it by default.
@@ -379,9 +306,9 @@ Only parts of the graph that change from each execution to the next will be exec
Dragging a generated png on the webpage or loading one will give you the full workflow including seeds that were used to create it.
You can use () to change emphasis of a word or phrase like: (good code:1.2) or (bad code:0.8). The default emphasis for () is 1.1. To use () characters in your actual prompt escape them like \\( or \\).
You can use () to change emphasis of a word or phrase like: (good code:1.2) or (bad code:0.8). The default emphasis for () is 1.1. To use () characters in your actual prompt escape them like \( or \).
You can use {day|night}, for wildcard/dynamic prompts. With this syntax "{wild|card|test}" will be randomly replaced by either "wild", "card" or "test" by the frontend every time you queue the prompt. To use {} characters in your actual prompt escape them like: \\{ or \\}.
You can use {day|night}, for wildcard/dynamic prompts. With this syntax "{wild|card|test}" will be randomly replaced by either "wild", "card" or "test" by the frontend every time you queue the prompt. To use {} characters in your actual prompt escape them like: \{ or \}.
Dynamic prompts also support C-style comments, like `// comment` or `/* comment */`.
@@ -394,7 +321,7 @@ To use a textual inversion concepts/embeddings in a text prompt put them in the
Use ```--preview-method auto``` to enable previews.
The default installation includes a fast latent preview method that's low-resolution. To enable higher-quality previews with [TAESD](https://github.com/madebyollin/taesd), download the [taesd_decoder.pth, taesdxl_decoder.pth, taesd3_decoder.pth and taef1_decoder.pth](https://github.com/madebyollin/taesd/) and place them in the `models/vae_approx` folder. Once they're installed, restart ComfyUI and launch it with `--preview-method taesd` to enable high-quality previews.
The default installation includes a fast latent preview method that's low-resolution. To enable higher-quality previews with [TAESD](https://github.com/madebyollin/taesd), download the [taesd_decoder.pth, taesdxl_decoder.pth, taesd3_decoder.pth and taef1_decoder.pth](https://github.com/madebyollin/taesd/) and place them in the `models/vae_approx` folder. Once they're installed, restart Hanzo Studio and launch it with `--preview-method taesd` to enable high-quality previews.
## How to use TLS/SSL?
Generate a self-signed certificate (not appropriate for shared/production use) and key by running the command: `openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -sha256 -days 3650 -nodes -subj "/C=XX/ST=StateName/L=CityName/O=CompanyName/OU=CompanySectionName/CN=CommonNameOrHostname"`
@@ -406,55 +333,6 @@ Use `--tls-keyfile key.pem --tls-certfile cert.pem` to enable TLS/SSL, the app w
## Support and dev channel
[Discord](https://comfy.org/discord): Try the #help or #feedback channels.
[Discord](https://discord.gg/hanzoai): Try the #help or #feedback channels.
[Matrix space: #comfyui_space:matrix.org](https://app.element.io/#/room/%23comfyui_space%3Amatrix.org) (it's like discord but open source).
See also: [https://www.comfy.org/](https://www.comfy.org/)
## Frontend Development
As of August 15, 2024, we have transitioned to a new frontend, which is now hosted in a separate repository: [ComfyUI Frontend](https://github.com/Comfy-Org/ComfyUI_frontend). This repository now hosts the compiled JS (from TS/Vue) under the `web/` directory.
### Reporting Issues and Requesting Features
For any bugs, issues, or feature requests related to the frontend, please use the [ComfyUI Frontend repository](https://github.com/Comfy-Org/ComfyUI_frontend). This will help us manage and address frontend-specific concerns more efficiently.
### Using the Latest Frontend
The new frontend is now the default for ComfyUI. However, please note:
1. The frontend in the main ComfyUI repository is updated fortnightly.
2. Daily releases are available in the separate frontend repository.
To use the most up-to-date frontend version:
1. For the latest daily release, launch ComfyUI with this command line argument:
This approach allows you to easily switch between the stable fortnightly release and the cutting-edge daily updates, or even specific versions for testing purposes.
### Accessing the Legacy Frontend
If you need to use the legacy frontend for any reason, you can access it using the following command line argument:
This will use a snapshot of the legacy frontend preserved in the [ComfyUI Legacy Frontend repository](https://github.com/Comfy-Org/ComfyUI_legacy_frontend).
# QA
### Which GPU should I buy for this?
[See this page for some recommendations](https://github.com/comfyanonymous/ComfyUI/wiki/Which-GPU-should-I-buy-for-ComfyUI)
See also: [https://www.hanzo.ai/](https://www.hanzo.ai/)
All routes under the `/internal` path are designated for **internal use by ComfyUI only**. These routes are not intended for use by external applications may change at any time without notice.
All routes under the `/internal` path are designated for **internal use by Hanzo Studio only**. These routes are not intended for use by external applications may change at any time without notice.
File diff suppressed because one or more lines are too long
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
hanzo-dev
z
