Library ingest: POST /v1/library/upload — every render lands in the library (#13)

* Library ingest: POST /v1/library/upload — every render lands in the library

Tenant-scoped ingest so EVERY render is stored in studio.hanzo.ai, including
renders produced on a GPU node outside the job path (the node's mirror loop
POSTs here). Org via _org_of like every route.

Body: multipart (field image/file) OR raw bytes + ?name=, optional ?subpath=
(default renders). Writes orgs/<org>/output/<subpath>/<name> atomically
(tmp+replace, same pattern as the work log). Rejects traversal (basename-only
name + allowlisted subpath chars), caps size at 30MB, and skips a byte-identical
existing file (name+size match → 200 {ok, existed:true}); else 200 {ok, path}.
The manifest sidecar already indexes output/ — no other wiring.

Tests (studio_home_test): round-trip via GET /v1/library/file, dedup, tenant
isolation (org A upload never visible to org B, no cross-tenant overwrite),
subpath-traversal rejected, traversal-y name neutralized to basename, non-image
rejected, oversize rejected. 259 passed.

* Ship scripts/ whole (unbreak manifest sidecar) + index renders at ingest

(A) Regression: .dockerignore `scripts/*` + `!install_custom_nodes.sh` dropped
scripts/library_manifest.py from the image, so the build-manifest sidecar's
`[ -f ]` guard silently no-op'd and NOTHING got indexed into any org library.
Ship scripts/ whole again.

(B) Make indexing event-driven at the ingest point: POST /v1/library/upload, after
writing the asset, atomically appends the org library.json entry ({path, design?,
kind, role?, status:"draft", tags:[], updatedAt}) AND a worklog row ({kind:"render",
node, prompt:"", refs:[], parents:[], output_prefix:<path>, status:"done"}) — so
every mirrored render is instantly findable in Assets AND Queue & History with its
source node, filterable by run/flow, the moment it exists. Optional ?design= ?kind=
?role= ?node= with safe defaults (node→"upload"). Upserts by path (no dup on
re-render); the existed/dedup path skips indexing. The manifest sidecar stays the
sweeper for files that appear outside the route. _row_output resolves an ingested
render's output (its stored path) so it shows a thumbnail + opens its graph.

Tests: upload → library.json entry + worklog row appear (node, done, resolved
output), tenancy holds, dedup adds no second entry. 260 passed.

---------

Co-authored-by: Hanzo AI <ai@hanzo.ai>

.dockerignore

@@ -23,9 +23,10 @@ venv/
 .ruff_cache/
 tests/
 tests-unit/
-scripts/*
-# ...except the pack installer, which the image build invokes.
-!scripts/install_custom_nodes.sh
+# scripts/ ships WHOLE: the image build runs scripts/install_custom_nodes.sh and
+# the studio pod runs scripts/library_manifest.py as the build-manifest sidecar
+# (indexes every org's output/ into library.json). Excluding it silently no-ops
+# the sidecar's `[ -f ]` guard, so nothing gets indexed — keep the tree intact.
 notebooks/
 *.swp
 .env

middleware/studio_home.py

@@ -38,6 +38,7 @@ import os
 import random
 import re
 import time
+import uuid
 from pathlib import Path
 
 import aiohttp
@@ -678,6 +679,52 @@ async def _wallet(request: web.Request, org: str) -> dict:
     return out
 
 
+_MAX_UPLOAD = 30 * 1024 * 1024        # 30 MB — a sane ceiling for one render
+_UPLOAD_EXT = (".png", ".jpg", ".jpeg", ".webp")
+_SUBPATH_RE = re.compile(r"[A-Za-z0-9_-]+(?:/[A-Za-z0-9_-]+)*")
+
+
+def _safe_subpath(raw: str) -> str | None:
+    """The library subfolder an upload lands in — allowlist chars only (no traversal,
+    no absolute), default ``renders``. Returns None for anything outside the allowlist
+    so the route can reject it."""
+    sub = (raw or "").strip("/") or "renders"
+    return sub if _SUBPATH_RE.fullmatch(sub) else None
+
+
+def _row_output(root: Path | None, output_prefix: str | None) -> str | None:
+    """The library-relative output for a work-log row: the file a SaveImage prefix
+    produced, or — for an INGESTED render whose output_prefix is already the stored
+    file's path — that path itself. Lets a mirrored render show its thumbnail and open
+    its graph, without a SaveImage-style counter suffix."""
+    landed = _landed_output(root, output_prefix)
+    if landed is None and root is not None and output_prefix and _safe_asset(root, output_prefix) is not None:
+        landed = output_prefix
+    return landed
+
+
+def _index_upload(org: str, root: Path, rel: str, *, design, kind, role, node) -> None:
+    """Event-driven index at the ingest point so a just-stored render is INSTANTLY
+    findable — in Assets (a library.json entry) and in Queue & History (a done
+    work-log row carrying its source node) — without waiting for the manifest sweeper.
+    Atomic per-org writes like every other metadata file; upserts by path so a
+    re-render of the same name never duplicates the asset."""
+    now_iso = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
+    lib = _load_library(root)
+    assets = lib.setdefault("assets", [])
+    ex = next((a for a in assets if a.get("path") == rel), None)
+    if ex is None:
+        assets.append({"path": rel, "design": design, "kind": kind, "role": role,
+                       "status": "draft", "tags": [], "updatedAt": now_iso})
+    else:
+        ex.update(design=design, kind=kind, role=role, updatedAt=now_iso)
+    _save_library(root, lib)
+    worklog.record(org, kind="render", prompt="", refs=[], uploads=[], parents=[],
+                   output_prefix=rel, node=node, status="done",
+                   lane=("gpu" if node not in ("upload", "studio pod") else "local"),
+                   pid="up-" + uuid.uuid4().hex[:12])
+
+
 def add_studio_home_routes(routes: web.RouteTableDef, server) -> None:
 
     @routes.get("/studio")
@@ -710,7 +757,7 @@ def add_studio_home_routes(routes: web.RouteTableDef, server) -> None:
         if row is None:
             return web.json_response({"error": "not found"}, status=404)
         root = _library_root(org)
-        landed = _landed_output(root, row.get("output_prefix"))
+        landed = _row_output(root, row.get("output_prefix"))
         p = _safe_asset(root, landed) if (root and landed) else None
         if p is None or p.suffix.lower() != ".png":
             return web.json_response({"error": "workflow not available yet"}, status=404)
@@ -879,6 +926,58 @@ def add_studio_home_routes(routes: web.RouteTableDef, server) -> None:
                 pass  # fall through to full-res on any thumbnail failure
         return web.FileResponse(p)
 
+    @routes.post("/v1/library/upload")
+    async def library_upload(request: web.Request):
+        """Tenant-scoped render ingest so EVERY render lands in the library — including
+        ones produced on a GPU node outside the job path (the node's mirror loop POSTs
+        here). Body is multipart (field ``image``/``file``) or raw bytes + ``?name=``,
+        with an optional ``?subpath=`` (default ``renders``). Writes
+        orgs/<org>/output/<subpath>/<name> atomically (tmp+replace, same as the work
+        log); rejects traversal (basename-only name + allowlisted subpath); caps size;
+        and skips a byte-identical existing file (name+size match → existed). The
+        manifest sidecar indexes output/ — no other wiring."""
+        org = _org_of(request)
+        sub = _safe_subpath(request.query.get("subpath", ""))
+        if sub is None:
+            return web.json_response({"error": "invalid subpath"}, status=400)
+        clen = request.content_length
+        if clen is not None and clen > _MAX_UPLOAD:
+            return web.json_response({"error": "file too large"}, status=413)
+        if request.headers.get("Content-Type", "").startswith("multipart/"):
+            post = await request.post()
+            f = post.get("image") or post.get("file")
+            if f is None or not hasattr(f, "file"):
+                return web.json_response({"error": "no image field"}, status=400)
+            name = os.path.basename(f.filename or "")
+            data = f.file.read()
+        else:
+            data = await request.read()
+            name = os.path.basename(request.query.get("name", ""))
+        if not name or os.path.splitext(name)[1].lower() not in _UPLOAD_EXT:
+            return web.json_response({"error": "image name required (.png/.jpg/.jpeg/.webp)"}, status=400)
+        if not data:
+            return web.json_response({"error": "empty body"}, status=400)
+        if len(data) > _MAX_UPLOAD:
+            return web.json_response({"error": "file too large"}, status=413)
+        out_root = Path(folder_paths.get_org_output_directory(org))
+        dest = (out_root / sub / name).resolve()
+        if not str(dest).startswith(str(out_root.resolve()) + os.sep):
+            return web.json_response({"error": "invalid path"}, status=400)
+        rel = f"{sub}/{name}"
+        if dest.is_file() and dest.stat().st_size == len(data):
+            return web.json_response({"ok": True, "existed": True, "path": rel})
+        dest.parent.mkdir(parents=True, exist_ok=True)
+        tmp = dest.with_name(dest.name + ".tmp")
+        tmp.write_bytes(data)
+        tmp.replace(dest)
+        # index at ingest → instantly findable in Assets + Queue & History (with node)
+        _index_upload(org, out_root, rel,
+                      design=request.query.get("design") or None,
+                      kind=request.query.get("kind") or "render",
+                      role=request.query.get("role") or None,
+                      node=request.query.get("node") or "upload")
+        return web.json_response({"ok": True, "path": rel})
+
     @routes.get("/v1/gpu-status")
     async def gpu_status(request: web.Request):
         """Does the caller's org have an online GPU to render on? The UI showed
@@ -1093,7 +1192,7 @@ def add_studio_home_routes(routes: web.RouteTableDef, server) -> None:
         out = []
         for r in reversed(rows):
             row = dict(r)
-            landed = _landed_output(root, r.get("output_prefix"))
+            landed = _row_output(root, r.get("output_prefix"))
             row["output"] = landed
             if landed and r.get("status") not in ("failed", "cancelled"):
                 row["status"] = "done"

tests-unit/studio_home_test/test_studio_home.py

@@ -478,3 +478,91 @@ async def test_wallet_proxies_and_scopes_org(tmp_path, monkeypatch):
     r = await client.get("/v1/wallet", headers={"X-Org": "acme"})
     assert await r.json() == {"org": "acme", "balance": None, "currency": None}
     await client.close()
+
+
+@pytest.mark.asyncio
+async def test_library_upload_roundtrip_dedup_and_tenant_isolation(tmp_path, monkeypatch):
+    import aiohttp
+    _orgaware(tmp_path, monkeypatch)
+    # library.json so _library_root resolves for the GET round-trip
+    (Path(sh.folder_paths.get_org_output_directory("acme")) / "library.json").write_text('{"assets":[]}')
+    (Path(sh.folder_paths.get_org_output_directory("globex")) / "library.json").write_text('{"assets":[]}')
+    client = await _client(_FakeServer())
+    png = b"\x89PNG\r\n\x1a\n" + b"acme-bytes"
+    # multipart upload as acme -> renders/r1.png
+    form = aiohttp.FormData()
+    form.add_field("image", png, filename="r1.png", content_type="image/png")
+    r = await client.post("/v1/library/upload", data=form, headers={"X-Org": "acme"})
+    body = await r.json()
+    assert r.status == 200 and body["ok"] and body["path"] == "renders/r1.png" and not body.get("existed")
+    # round-trip through the org-scoped file server
+    g = await client.get("/v1/library/file?path=renders/r1.png", headers={"X-Org": "acme"})
+    assert g.status == 200 and (await g.read()) == png
+    # dedup: identical name+size -> existed, no rewrite
+    form2 = aiohttp.FormData()
+    form2.add_field("image", png, filename="r1.png", content_type="image/png")
+    r2 = await client.post("/v1/library/upload", data=form2, headers={"X-Org": "acme"})
+    assert (await r2.json()) == {"ok": True, "existed": True, "path": "renders/r1.png"}
+    # tenant isolation: org B never sees org A's file; its own upload is a separate tree
+    assert (await client.get("/v1/library/file?path=renders/r1.png", headers={"X-Org": "globex"})).status == 404
+    rg = await client.post("/v1/library/upload?name=r1.png", data=b"\x89PNGglobex",
+                           headers={"X-Org": "globex", "Content-Type": "application/octet-stream"})
+    assert (await rg.json())["path"] == "renders/r1.png"
+    # org A's bytes are untouched by org B's same-named upload
+    assert (await (await client.get("/v1/library/file?path=renders/r1.png", headers={"X-Org": "acme"})).read()) == png
+    await client.close()
+
+
+@pytest.mark.asyncio
+async def test_library_upload_rejects_traversal_bad_name_and_oversize(tmp_path, monkeypatch):
+    _orgaware(tmp_path, monkeypatch)
+    client = await _client(_FakeServer())
+    hdr = {"X-Org": "acme", "Content-Type": "application/octet-stream"}
+    # subpath traversal -> 400
+    assert (await client.post("/v1/library/upload?name=x.png&subpath=../secret", data=b"\x89PNGx", headers=hdr)).status == 400
+    # non-image name -> 400
+    assert (await client.post("/v1/library/upload?name=notes.txt", data=b"hi", headers=hdr)).status == 400
+    # a traversal-y NAME is neutralized to its basename, lands safely under renders/
+    r = await client.post("/v1/library/upload?name=../../evil.png", data=b"\x89PNGevil", headers=hdr)
+    assert (await r.json())["path"] == "renders/evil.png"
+    assert (Path(sh.folder_paths.get_org_output_directory("acme")) / "renders" / "evil.png").is_file()
+    # size cap -> 413
+    monkeypatch.setattr(sh, "_MAX_UPLOAD", 8)
+    assert (await client.post("/v1/library/upload?name=big.png", data=b"0123456789", headers=hdr)).status == 413
+    await client.close()
+
+
+@pytest.mark.asyncio
+async def test_library_upload_indexes_into_assets_and_worklog(tmp_path, monkeypatch):
+    import aiohttp
+    _orgaware(tmp_path, monkeypatch)
+    client = await _client(_FakeServer())
+    png = b"\x89PNG\r\n\x1a\n" + b"render-bytes"
+    form = aiohttp.FormData()
+    form.add_field("image", png, filename="shot.png", content_type="image/png")
+    r = await client.post("/v1/library/upload?node=spark&design=karma&kind=render",
+                          data=form, headers={"X-Org": "acme"})
+    assert (await r.json())["path"] == "renders/shot.png"
+    # instantly findable in Assets (library.json entry, draft)
+    lib = await (await client.get("/v1/library", headers={"X-Org": "acme"})).json()
+    a = next(x for x in lib["assets"] if x["path"] == "renders/shot.png")
+    assert a["status"] == "draft" and a["kind"] == "render" and a["design"] == "karma" and a["tags"] == []
+    # instantly findable in Queue & History with its source node + resolved output
+    wl = await (await client.get("/v1/worklog", headers={"X-Org": "acme"})).json()
+    it = next(x for x in wl["items"] if x["output_prefix"] == "renders/shot.png")
+    assert it["kind"] == "render" and it["node"] == "spark" and it["status"] == "done"
+    assert it["output"] == "renders/shot.png" and it["lane"] == "gpu"
+    # tenant isolation: org B sees neither the asset nor the row
+    lib2 = await (await client.get("/v1/library", headers={"X-Org": "globex"})).json()
+    assert all(x["path"] != "renders/shot.png" for x in lib2.get("assets", []))
+    assert (await (await client.get("/v1/worklog", headers={"X-Org": "globex"})).json())["items"] == []
+    # dedup: re-upload identical bytes adds no second asset or row
+    form2 = aiohttp.FormData()
+    form2.add_field("image", png, filename="shot.png", content_type="image/png")
+    r2 = await client.post("/v1/library/upload?node=spark", data=form2, headers={"X-Org": "acme"})
+    assert (await r2.json()).get("existed") is True
+    lib3 = await (await client.get("/v1/library", headers={"X-Org": "acme"})).json()
+    assert sum(1 for x in lib3["assets"] if x["path"] == "renders/shot.png") == 1
+    wl3 = await (await client.get("/v1/worklog", headers={"X-Org": "acme"})).json()
+    assert sum(1 for x in wl3["items"] if x["output_prefix"] == "renders/shot.png") == 1
+    await client.close()