A finished render becomes an `Asset` document — the DocType hanzoai/cloud
clients/content already declares (framework module "marketing") and already writes
itself when IT drives a render. Studio-initiated renders were the only ones with
no path into that record; this closes exactly that gap and adds NO second schema,
NO second lifecycle, and NO second storage plane. cloud is untouched.
POST /v1/framework/Asset (the already-live generic framework surface)
Same field shape clients/content writes (studio_render.go draftAsset): title,
kind, design, prompt, file, generator, workflow, source_prompt_id, render_params,
project, tags. `file` is the org-scoped output key (orgs/<org>/output/...) —
byte-identical to the key persistOrLink produces and the key Studio's own output
mirror lands, so the bytes are ALREADY persisted: this module uploads nothing and
holds no storage credential.
Written AS THE REQUESTING USER: the verified IAM access token rides to the prompt
worker on the queue's existing `sensitive` channel (never persisted into history)
and authenticates every write, so the Asset lands in that user's org with that
user's rights. No service account, no impersonation, no cross-org path.
Assets land in draft (the DocType default); the content lane's ONE state machine
(draft->in_review->approved->queued->published) owns every transition, and a human
does the promoting. Publishing is what a storefront shows, so an unattended render
can never push an image to karma.style.
Fire-and-forget on the same prompt-worker seam as billing: a render is never failed
because the lane was unreachable. Env-gated (STUDIO_CONTENT_PUBLISH); local dev
unaffected.
Root cause of the "Event loop stopped before Future completed" crash that
wiped the in-memory queue/history (seen 3x today): the fork's _graceful_shutdown
handler (added in a45a441) called event_loop.stop(). The server coroutine passed
to run_until_complete() never completes on its own, so stopping the loop makes
run_until_complete() raise RuntimeError; the surrounding try only caught
KeyboardInterrupt, so the process died non-zero with a traceback and discarded
the queue. The dirty exit also failed to release :8188 promptly, and a duplicate
transient/user `hanzo-studio.service` racing the canonical studio.service turned
that into an EADDRINUSE restart storm.
main.py:
- handler is now idempotent, flags the queue closed, snapshots it, then stops
the loop; the outer try treats RuntimeError('Event loop stopped before Future
completed.') as the expected graceful-exit path (logs "Server stopped by
shutdown signal", exits 0). Verified live in journald.
execution.py:
- opt-in crash-durable PromptQueue behind STUDIO_PERSIST_QUEUE=1. Pending +
in-flight prompts are snapshotted atomically to user/queue_snapshot.json on
every change and re-queued on boot. Best-effort: all I/O is guarded so it
never affects the render path; inert when the flag is unset.
ops/systemd/studio.service:
- enable STUDIO_PERSIST_QUEUE=1 on the single canonical unit.
scripts/studio-service-swap.sh:
- batch-safe reconcile+apply: waits for an empty queue, removes any stray
duplicate unit, reinstalls the canonical unit, restarts, health-checks.
docs/ops-crash-and-service.md:
- topology (one canonical unit), how to read crash history, the fix, persistence.
tests-unit/execution_test/prompt_queue_persist_test.py:
- round-trip, inert-when-disabled, and corrupt-row-skipping tests.
Integrate comfy-aimdo 0.2 which takes a different approach to
installing the memory allocator hook. Instead of using the complicated
and buggy pytorch MemPool+CudaPluggableAlloctor, cuda is directly hooked
making the process much more transparent to both comfy and pytorch. As
far as pytorch knows, aimdo doesnt exist anymore, and just operates
behind the scenes.
Remove all the mempool setup stuff for dynamic_vram and bump the
comfy-aimdo version. Remove the allocator object from memory_management
and demote its use as an enablment check to a boolean flag.
Comfy-aimdo 0.2 also support the pytorch cuda async allocator, so
remove the dynamic_vram based force disablement of cuda_malloc and
just go back to the old settings of allocators based on command line
input.
* revert threaded model loader change
This change was only needed to get around the pytorch 2.7 mempool bugs,
and should have been reverted along with #12260. This fixes a different
memory leak where pytorch gets confused about cache emptying.
* load non comfy weights
* MPDynamic: Pre-generate the tensors for vbars
Apparently this is an expensive operation that slows down things.
* bump to aimdo 1.8
New features:
watermark limit feature
logging enhancements
-O2 build on linux
- Change error type from 'invalid_prompt' to 'missing_node_type' for frontend detection
- Add extra_info with node_id, class_type, and node_title (from _meta.title)
- Improve user-facing message: 'Node X not found. The custom node may not be installed.'
* Support Combo outputs in a more sane way
* Remove test validate_inputs function on test node
* Make curr_prefix be a list of strings instead of string for easier parsing as keys get added to dynamic types
* Start to account for id prefixes from frontend, need to fix bug with nested dynamics
* Ensure inputs/outputs/hidden are lists in schema finalize function, remove no longer needed 'is not None' checks
* Add raw_link and extra_dict to all relevant Inputs
* Make nested DynamicCombos work properly with prefixed keys on latest frontend; breaks old Autogrow, but is pretty much ready for upcoming Autogrow keys
* Replace ... usage with a MISSING sentinel for clarity in nodes_logic.py
* Added CustomCombo node in backend to reflect frontend node
* Prepare Autogrow's expand_schema_for_dynamic to work with upcoming frontend changes
* Prepare for look up table for dynamic input stuff
* More progress towards dynamic input lookup function stuff
* Finished converting _expand_schema_for_dynamic to be done via lookup instead of OOP to guarantee working with process isolation, did refactoring to remove old implementation + cleaning INPUT_TYPES definition including v3 hidden definition
* Change order of functions
* Removed some unneeded functions after dynamic refactor
* Make MatchType's output default displayname "MATCHTYPE"
* Fix DynamicSlot get_all
* Removed redundant code - dynamic stuff no longer happens in OOP way
* Natively support AnyType (*) without __ne__ hacks
* Remove stray code that made it in
* Remove expand_schema_for_dynamic left over on DynamicInput class
* get_dynamic() on DynamicInput/Output was not doing anything anymore, so removed it
* Make validate_inputs validate combo input correctly
* Temporarily comment out conversion to 'new' (9 month old) COMBO format in get_input_info
* Remove refrences to resources feature scrapped from V3
* Expose DynamicCombo in public API
* satisfy ruff after some code got commented out
* Make missing input error prettier for dynamic types
* Created a Switch2 node as a side-by-side test, will likely go with Switch2 as the initial switch node
* Figured out Switch situation
* Pass in v3_data in IsChangedCache.get function's fingerprint_inputs, add a from_v3_data helper method to HiddenHolder
* Switch order of Switch and Soft Switch nodes in file
* Temp test node for MatchType
* Fix missing v3_data for v1 nodes in validation
* For now, remove chacking duplicate id's for dynamic types
* Add Resize Image/Mask node that thanks to MatchType+DynamicCombo is 16-nodes-in-1
* Made DynamicCombo references in DCTestNode use public interface
* Add an AnyTypeTestNode
* Make lazy status for specific inputs on DynamicInputs work by having the values of the dictionary for check_lazy_status be a tuple, where the second element is the key of the input that can be returned
* Comment out test logic nodes
* Make primitive float's step make more sense
* Add (and leave commented out) some potential logic nodes
* Change default crop option to "center" on Resize Image/Mask node
* Changed copy.copy(d) to d.copy()
* Autogrow is available in stable frontend, so exposing it in public API
* Use outputs id as display_name if no display_name present, remove v3 outputs id restriction that made them have to have unique IDs from the inputs
* Enable Custom Combo node as stable frontend now supports it
* Make id properly act like display_name on outputs
* Add Batch Images/Masks/Latents node
* Comment out Batch Images/Masks/Latents node for now, as Autogrow has a bug with MatchType where top connection is disconnected upon refresh
* Removed code for a couple test nodes in nodes_logic.py
* Add Batch Images, Batch Masks, and Batch Latents nodes with Autogrow, deprecate old Batch Images + LatentBatch nodes
* Added output_matchtypes to generated json for v3, initial backend support for MatchType, created nodes_logic.py and added SwitchNode
* Fixed providing list of allowed_types
* Add workaround in validation.py for V3 Combo outputs not working as Combo inputs
* Make match type receive_type pass validation
* Also add MatchType check to input_type in validation - will likely trigger when connecting to non-lazy stuff
* Make sure this PR only has MatchType stuff
* Initial work on DynamicCombo
* Add get_dynamic function, not yet filled out correctly
* Mark Switch node as Beta
* Make sure other unfinished dynamic types are not accidentally used
* Send DynamicCombo.Option inputs in the same format as normal v1 inputs
* add dynamic combo test node
* Support validation of inputs and outputs
* Add missing input params to DynamicCombo.Input
* Add get_all function to inputs for id validation purposes
* Fix imports for v3 returning everything when doing io/ui/IO/UI instead of what is in __all__ of _io.py and _ui.py
* Modifying behavior of get_dynamic in V3 + serialization so can be used in execution code
* Fix v3 schema validation code after changes
* Refactor hidden_values for v3 in execution.py to be more general v3_data, add helper functions for dynamic behavior, preparing for restructuring dynamic type into object (not finished yet)
* Add nesting of inputs on DynamicCombo during execution
* Work with latest frontend commits
* Fix cringe arrows
* frontend will no longer namespace dynamic inputs widgets so reflect that in code, refactor build_nested_inputs
* Prepare Autogrow support for the love of the game
* satisfy ruff
* Create test nodes for Autogrow to collab with frontend development
* Add nested combo to DCTestNode
* Remove array support from build_nested_inputs, properly handle missing expected values
* Make execution.validate_inputs properly validate required dynamic inputs, renamed dynamic_data to dynamic_paths for clarity
* MatchType does not need any DynamicInput/Output features on backend; will increase compatibility with dynamic types
* Probably need this for ruff check
* Change MatchType to have template be the first and only required param; output id's do nothing right now, so no need
* Fix merge regression with LatentUpscaleModel type not being put in __all__ for _io.py, fix invalid type hint for validate_inputs
* Make Switch node inputs optional, disallow both inputs from being missing, and still work properly with lazy; when one input is missing, use the other no matter what the switch is set to
* Satisfy ruff
* Move MatchType code above the types that inherit from DynamicInput
* Add DynamicSlot type, awaiting frontend support
* Make curr_prefix creation happen in Autogrow, move curr_prefix in DynamicCombo to only be created if input exists in live_inputs
* I was confused, fixing accidentally redundant curr_prefix addition in Autogrow
* Make sure Autogrow inputs are force_input = True when WidgetInput, fix runtime validation by removing original input from expected inputs, fix min/max bounds, change test nodes slightly
* Remove unnecessary id usage in Autogrow test node outputs
* Commented out Switch node + test nodes
* Remove commented out code from Autogrow
* Make TemplatePrefix max more clear, allow max == 1
* Replace all dict[str] with dict[str, Any]
* Renamed add_to_dict_live_inputs to expand_schema_for_dynamic
* Fixed typo in DynamicSlot input code
* note about live_inputs not being present soon in get_v1_info (internal function anyway)
* For now, hide DynamicCombo and Autogrow from public interface
* Removed comment
* execution: Roll the UI cache into the outputs
Currently the UI cache is parallel to the output cache with
expectations of being a content superset of the output cache.
At the same time the UI and output cache are maintained completely
seperately, making it awkward to free the output cache content without
changing the behaviour of the UI cache.
There are two actual users (getters) of the UI cache. The first is
the case of a direct content hit on the output cache when executing a
node. This case is very naturally handled by merging the UI and outputs
cache.
The second case is the history JSON generation at the end of the prompt.
This currently works by asking the cache for all_node_ids and then
pulling the cache contents for those nodes. all_node_ids is the nodes
of the dynamic prompt.
So fold the UI cache into the output cache. The current UI cache setter
now writes to a prompt-scope dict. When the output cache is set, just
get this value from the dict and tuple up with the outputs.
When generating the history, simply iterate prompt-scope dict.
This prepares support for more complex caching strategies (like RAM
pressure caching) where less than 1 workflow will be cached and it
will be desirable to keep the UI cache and output cache in sync.
* sd: Implement RAM getter for VAE
* model_patcher: Implement RAM getter for ModelPatcher
* sd: Implement RAM getter for CLIP
* Implement RAM Pressure cache
Implement a cache sensitive to RAM pressure. When RAM headroom drops
down below a certain threshold, evict RAM-expensive nodes from the
cache.
Models and tensors are measured directly for RAM usage. An OOM score
is then computed based on the RAM usage of the node.
Note the due to indirection through shared objects (like a model
patcher), multiple nodes can account the same RAM as their individual
usage. The intent is this will free chains of nodes particularly
model loaders and associate loras as they all score similar and are
sorted in close to each other.
Has a bias towards unloading model nodes mid flow while being able
to keep results like text encodings and VAE.
* execution: Convert the cache entry to NamedTuple
As commented in review.
Convert this to a named tuple and abstract away the tuple type
completely from graph.py.
In the case of --cache-none lazy and subgraph execution can cause
anything to be run multiple times per workflow. If that rerun nodes is
in itself a subgraph generator, this will crash for two reasons.
pending_subgraph_results[] does not cleanup entries after their use.
So when a pending_subgraph_result is consumed, remove it from the list
so that if the corresponding node is fully re-executed this misses
lookup and it fall through to execute the node as it should.
Secondly, theres is an explicit enforcement against dups in the
addition of subgraphs nodes as ephemerals to the dymprompt. Remove this
enforcement as the use case is now valid.
* execution: fold in dependency aware caching
This makes --cache-none compatiable with lazy and expanded
subgraphs.
Currently the --cache-none option is powered by the
DependencyAwareCache. The cache attempts to maintain a parallel
copy of the execution list data structure, however it is only
setup once at the start of execution and does not get meaninigful
updates to the execution list.
This causes multiple problems when --cache-none is used with lazy
and expanded subgraphs as the DAC does not accurately update its
copy of the execution data structure.
DAC has an attempt to handle subgraphs ensure_subcache however
this does not accurately connect to nodes outside the subgraph.
The current semantics of DAC are to free a node ASAP after the
dependent nodes are executed.
This means that if a subgraph refs such a node it will be requed
and re-executed by the execution_list but DAC wont see it in
its to-free lists anymore and leak memory.
Rather than try and cover all the cases where the execution list
changes from inside the cache, move the while problem to the
executor which maintains an always up-to-date copy of the wanted
data-structure.
The executor now has a fast-moving run-local cache of its own.
Each _to node has its own mini cache, and the cache is unconditionally
primed at the time of add_strong_link.
add_strong_link is called for all of static workflows, lazy links
and expanded subgraphs so its the singular source of truth for
output dependendencies.
In the case of a cache-hit, the executor cache will hold the non-none
value (it will respect updates if they happen somehow as well).
In the case of a cache-miss, the executor caches a None and will
wait for a notification to update the value when the node completes.
When a node completes execution, it simply releases its mini-cache
and in turn its strong refs on its direct anscestor outputs, allowing
for ASAP freeing (same as the DependencyAwareCache but a little more
automatic).
This now allows for re-implementation of --cache-none with no cache
at all. The dependency aware cache was also observing the dependency
sematics for the objects and UI cache which is not accurate (this
entire logic was always outputs specific).
This also prepares for more complex caching strategies (such as RAM
pressure based caching), where a cache can implement any freeing
strategy completely independently of the DepedancyAwareness
requirement.
* main: re-implement --cache-none as no cache at all
The execution list now tracks the dependency aware caching more
correctly that the DependancyAwareCache.
Change it to a cache that does nothing.
* test_execution: add --cache-none to the test suite
--cache-none is now expected to work universally. Run it through the
full unit test suite. Propagate the server parameterization for whether
or not the server is capabale of caching, so that the minority of tests
that specifically check for cache hits can if else. Hard assert NOT
caching in the else to give some coverage of --cache-none expected
behaviour to not acutally cache.
* execution: fold in dependency aware caching
This makes --cache-none compatiable with lazy and expanded
subgraphs.
Currently the --cache-none option is powered by the
DependencyAwareCache. The cache attempts to maintain a parallel
copy of the execution list data structure, however it is only
setup once at the start of execution and does not get meaninigful
updates to the execution list.
This causes multiple problems when --cache-none is used with lazy
and expanded subgraphs as the DAC does not accurately update its
copy of the execution data structure.
DAC has an attempt to handle subgraphs ensure_subcache however
this does not accurately connect to nodes outside the subgraph.
The current semantics of DAC are to free a node ASAP after the
dependent nodes are executed.
This means that if a subgraph refs such a node it will be requed
and re-executed by the execution_list but DAC wont see it in
its to-free lists anymore and leak memory.
Rather than try and cover all the cases where the execution list
changes from inside the cache, move the while problem to the
executor which maintains an always up-to-date copy of the wanted
data-structure.
The executor now has a fast-moving run-local cache of its own.
Each _to node has its own mini cache, and the cache is unconditionally
primed at the time of add_strong_link.
add_strong_link is called for all of static workflows, lazy links
and expanded subgraphs so its the singular source of truth for
output dependendencies.
In the case of a cache-hit, the executor cache will hold the non-none
value (it will respect updates if they happen somehow as well).
In the case of a cache-miss, the executor caches a None and will
wait for a notification to update the value when the node completes.
When a node completes execution, it simply releases its mini-cache
and in turn its strong refs on its direct anscestor outputs, allowing
for ASAP freeing (same as the DependencyAwareCache but a little more
automatic).
This now allows for re-implementation of --cache-none with no cache
at all. The dependency aware cache was also observing the dependency
sematics for the objects and UI cache which is not accurate (this
entire logic was always outputs specific).
This also prepares for more complex caching strategies (such as RAM
pressure based caching), where a cache can implement any freeing
strategy completely independently of the DepedancyAwareness
requirement.
* main: re-implement --cache-none as no cache at all
The execution list now tracks the dependency aware caching more
correctly that the DependancyAwareCache.
Change it to a cache that does nothing.
* test_execution: add --cache-none to the test suite
--cache-none is now expected to work universally. Run it through the
full unit test suite. Propagate the server parameterization for whether
or not the server is capabale of caching, so that the minority of tests
that specifically check for cache hits can if else. Hard assert NOT
caching in the else to give some coverage of --cache-none expected
behaviour to not acutally cache.
When a prompt is submitted, it can optionally include
`partial_execution_targets` as a list of ids. If it does, rather than
adding all outputs to the execution list, we add only those in the list.
Remove auth_token_comfy_org and api_key_comfy_org from extra_data before
storing prompt history to prevent sensitive authentication tokens from
being persisted in the history endpoint response.
* Support for async execution functions
This commit adds support for node execution functions defined as async. When
a node's execution function is defined as async, we can continue
executing other nodes while it is processing.
Standard uses of `await` should "just work", but people will still have
to be careful if they spawn actual threads. Because torch doesn't really
have async/await versions of functions, this won't particularly help
with most locally-executing nodes, but it does work for e.g. web
requests to other machines.
In addition to the execute function, the `VALIDATE_INPUTS` and
`check_lazy_status` functions can also be defined as async, though we'll
only resolve one node at a time right now for those.
* Add the execution model tests to CI
* Add a missing file
It looks like this got caught by .gitignore? There's probably a better
place to put it, but I'm not sure what that is.
* Add the websocket library for automated tests
* Add additional tests for async error cases
Also fixes one bug that was found when an async function throws an error
after being scheduled on a task.
* Add a feature flags message to reduce bandwidth
We now only send 1 preview message of the latest type the client can
support.
We'll add a console warning when the client fails to send a feature
flags message at some point in the future.
* Add async tests to CI
* Don't actually add new tests in this PR
Will do it in a separate PR
* Resolve unit test in GPU-less runner
* Just remove the tests that GHA can't handle
* Change line endings to UNIX-style
* Avoid loading model_management.py so early
Because model_management.py has a top-level `logging.info`, we have to
be careful not to import that file before we call `setup_logging`. If we
do, we end up having the default logging handler registered in addition
to our custom one.
* Handle Comfy API key based authorizaton (#167)
Co-authored-by: Jedrzej Kosinski <kosinkadink1@gmail.com>
* Bump frontend version to include API key features (#170)
* bump templates version
---------
Co-authored-by: Jedrzej Kosinski <kosinkadink1@gmail.com>
* add dependency aware cache that removed a cached node as soon as all of its decendents have executed. This allows users with lower RAM to run workflows they would otherwise not be able to run. The downside is that every workflow will fully run each time even if no nodes have changed.
* remove test code
* tidy code
* Reapply "Add union link connection type support (#5806)" (#5889)
This reverts commit edf21cf606.
* Fix union type breaks existing type workarounds
* Add non-string test
* Add tests for hacks and non-string types
* Support python versions lower than 3.11
* Less fragile memory management.
* Fix issue.
* Remove useless function.
* Prevent and detect some types of memory leaks.
* Run garbage collector when switching workflow if needed.
* Fix issue.
This change fixes a bug where non-constant values could be passed to the
IS_CHANGED function. This would result in workflows taking an extra
execution before they acted as if they were cached.
The actual change is like 4 characters -- the rest is adding unit tests.
