Root cause of the "Event loop stopped before Future completed" crash that
wiped the in-memory queue/history (seen 3x today): the fork's _graceful_shutdown
handler (added in a45a441) called event_loop.stop(). The server coroutine passed
to run_until_complete() never completes on its own, so stopping the loop makes
run_until_complete() raise RuntimeError; the surrounding try only caught
KeyboardInterrupt, so the process died non-zero with a traceback and discarded
the queue. The dirty exit also failed to release :8188 promptly, and a duplicate
transient/user `hanzo-studio.service` racing the canonical studio.service turned
that into an EADDRINUSE restart storm.
main.py:
- handler is now idempotent, flags the queue closed, snapshots it, then stops
the loop; the outer try treats RuntimeError('Event loop stopped before Future
completed.') as the expected graceful-exit path (logs "Server stopped by
shutdown signal", exits 0). Verified live in journald.
execution.py:
- opt-in crash-durable PromptQueue behind STUDIO_PERSIST_QUEUE=1. Pending +
in-flight prompts are snapshotted atomically to user/queue_snapshot.json on
every change and re-queued on boot. Best-effort: all I/O is guarded so it
never affects the render path; inert when the flag is unset.
ops/systemd/studio.service:
- enable STUDIO_PERSIST_QUEUE=1 on the single canonical unit.
scripts/studio-service-swap.sh:
- batch-safe reconcile+apply: waits for an empty queue, removes any stray
duplicate unit, reinstalls the canonical unit, restarts, health-checks.
docs/ops-crash-and-service.md:
- topology (one canonical unit), how to read crash history, the fix, persistence.
tests-unit/execution_test/prompt_queue_persist_test.py:
- round-trip, inert-when-disabled, and corrupt-row-skipping tests.
hanzo-dev
b521933ca3