Merge branch 'calesthio:main' into main
This commit is contained in:
Hanzo Dev
@@ -0,0 +1,47 @@
|
||||
---
|
||||
name: canvas-procedural-animation
|
||||
description: Use p5.js/canvas for local procedural character effects: particles, weather, squash/stretch, walk cycles, and environmental motion.
|
||||
license: MIT
|
||||
---
|
||||
|
||||
# Canvas Procedural Animation
|
||||
|
||||
Use this skill when p5.js or Canvas is used for character-supporting motion:
|
||||
rain, snow, leaves, feathers, ambient particles, squash/stretch, or procedural
|
||||
walk cycles.
|
||||
|
||||
## Proven Pattern
|
||||
|
||||
p5.js runs setup once and redraws continuously through `draw()`. Keep animation
|
||||
state deterministic from time/frame values when rendering previews.
|
||||
|
||||
```js
|
||||
function setup() {
|
||||
createCanvas(1920, 1080);
|
||||
}
|
||||
|
||||
function draw() {
|
||||
const t = millis() / 1000;
|
||||
clear();
|
||||
drawCharacter(width / 2, height / 2 + sin(t * 8) * 8);
|
||||
}
|
||||
```
|
||||
|
||||
## Use For
|
||||
|
||||
- Particle/weather overlays.
|
||||
- Environmental motion.
|
||||
- Simple procedural bodies.
|
||||
- Effects that do not need individually authored SVG parts.
|
||||
|
||||
## Avoid For
|
||||
|
||||
- Complex facial acting where SVG/layered rig parts are easier to inspect.
|
||||
- Final renders that need exact frame determinism unless the runtime exposes
|
||||
frame-index control.
|
||||
|
||||
## Sources
|
||||
|
||||
- p5.js `setup()` reference: https://p5js.org/reference/p5/setup/
|
||||
- p5.js `draw()` reference: https://p5js.org/reference/p5/draw/
|
||||
- p5.js animation examples: https://p5js.org/examples/
|
||||
@@ -0,0 +1,43 @@
|
||||
---
|
||||
name: character-animation-qa
|
||||
description: Review local character animation with schema checks, Playwright browser previews, frame sampling, and FFmpeg/ffprobe final output checks.
|
||||
license: MIT
|
||||
---
|
||||
|
||||
# Character Animation QA
|
||||
|
||||
Use this skill before presenting a character-animation preview or final render.
|
||||
|
||||
## Review Layers
|
||||
|
||||
1. Schema validation: character design, rig plan, pose library, action timeline.
|
||||
2. Static asset checks: referenced parts and backgrounds exist.
|
||||
3. Browser preview: load the preview, capture screenshots, collect console errors.
|
||||
4. Motion check: compare sampled frames for non-trivial differences.
|
||||
5. Final MP4 check: ffprobe metadata, duration, resolution, audio, frame samples.
|
||||
6. Agent visual review: inspect sampled frames for detached limbs, bad layers,
|
||||
off-frame characters, unreadable expressions, broken text.
|
||||
|
||||
## Playwright Pattern
|
||||
|
||||
```ts
|
||||
const browser = await chromium.launch();
|
||||
const page = await browser.newPage({ viewport: { width: 1280, height: 720 } });
|
||||
await page.goto(previewUrl, { waitUntil: "networkidle" });
|
||||
await page.screenshot({ path: "preview.png" });
|
||||
```
|
||||
|
||||
## Pass/Revise/Fail
|
||||
|
||||
- `pass`: technical checks pass, acting is readable.
|
||||
- `revise`: fixable rig/timeline issue.
|
||||
- `fail`: missing assets, blank render, runtime failure, or wrong runtime.
|
||||
|
||||
## Sources
|
||||
|
||||
- Playwright screenshots:
|
||||
https://playwright.dev/docs/screenshots
|
||||
- Playwright page navigation:
|
||||
https://playwright.dev/docs/api/class-page#page-goto
|
||||
- FFmpeg/ffprobe should be used for final media probing:
|
||||
https://ffmpeg.org/ffprobe.html
|
||||
@@ -0,0 +1,53 @@
|
||||
---
|
||||
name: character-rigging
|
||||
description: Build data-driven 2D character rigs for local animation: parts, pivots, layers, constraints, views, and reusable rig packages.
|
||||
license: MIT
|
||||
---
|
||||
|
||||
# Character Rigging
|
||||
|
||||
Use this skill when building OpenMontage `rig_plan` artifacts or renderer input
|
||||
for local 2D character animation.
|
||||
|
||||
## Proven Patterns
|
||||
|
||||
- Keep runtime code generic; make each character a data package.
|
||||
- Split characters into independently transformable parts.
|
||||
- Define pivots in the same coordinate space as the artwork.
|
||||
- Store constraints on moving parts to prevent impossible rotations.
|
||||
- Keep layer order explicit; do not rely on SVG source order after generation.
|
||||
- Start with one view and add views only when the shot list requires them.
|
||||
|
||||
## Rig Package
|
||||
|
||||
```json
|
||||
{
|
||||
"character_id": "mouse",
|
||||
"rig_type": "svg_rig",
|
||||
"parts": [
|
||||
{ "id": "body", "kind": "torso", "layer": 10 },
|
||||
{ "id": "head", "kind": "head", "layer": 30, "parent": "body" },
|
||||
{ "id": "arm_right", "kind": "limb", "layer": 40, "parent": "body" }
|
||||
],
|
||||
"joints": {
|
||||
"head": { "pivot": [320, 180], "rotation": [-20, 20] },
|
||||
"arm_right": { "pivot": [390, 310], "rotation": [-70, 95] }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Quality Checklist
|
||||
|
||||
- Every moving part has a pivot.
|
||||
- Every child part has a parent where hierarchy matters.
|
||||
- Mouth shapes are separate assets or separate path groups.
|
||||
- Eyes and pupils are separate when gaze needs to change.
|
||||
- Props are separate if the character touches or carries them.
|
||||
|
||||
## Sources
|
||||
|
||||
- SVG transform-origin behavior is browser-defined and can be sensitive to
|
||||
coordinate space; prefer explicit SVG-coordinate pivots when using GSAP
|
||||
`svgOrigin`: https://gsap.com/docs/v3/GSAP/CorePlugins/CSS/
|
||||
- Remotion animations must be frame-driven and deterministic via current frame:
|
||||
https://www.remotion.dev/docs/use-current-frame
|
||||
@@ -0,0 +1,97 @@
|
||||
---
|
||||
name: doubao-tts
|
||||
description: Generate Mandarin and multilingual narration with Volcengine Doubao Speech 2.0. Use when creating Chinese voiceovers, when the user prefers Doubao/Volcengine/火山引擎/豆包 TTS, or when narration needs character-level timestamp metadata for subtitles.
|
||||
---
|
||||
|
||||
# Doubao TTS
|
||||
|
||||
Requires `DOUBAO_SPEECH_API_KEY` in `.env`.
|
||||
Set `DOUBAO_SPEECH_VOICE_TYPE` for the default voice, or pass `voice_id` to the tool.
|
||||
|
||||
## Current API
|
||||
|
||||
Use the new-console API key flow:
|
||||
|
||||
```text
|
||||
X-Api-Key: ${DOUBAO_SPEECH_API_KEY}
|
||||
X-Api-Resource-Id: seed-tts-2.0
|
||||
```
|
||||
|
||||
Do not use `X-Api-App-Id` and `X-Api-Access-Key` with a new-console API Key. If the API returns `load grant: requested grant not found`, the key type or auth header is probably wrong.
|
||||
|
||||
For long-form video narration, prefer the async endpoint:
|
||||
|
||||
```text
|
||||
POST https://openspeech.bytedance.com/api/v3/tts/submit
|
||||
POST https://openspeech.bytedance.com/api/v3/tts/query
|
||||
```
|
||||
|
||||
This returns `audio_url` plus `sentences[].words[]` timing metadata that can be used to build subtitles.
|
||||
|
||||
## OpenMontage Usage
|
||||
|
||||
Generate with the TTS selector:
|
||||
|
||||
```python
|
||||
from tools.audio.tts_selector import TTSSelector
|
||||
|
||||
result = TTSSelector().execute({
|
||||
"preferred_provider": "doubao",
|
||||
"text": "如果 AI 真的会改变未来，普通人到底该怎么参与？",
|
||||
"voice_id": "zh_female_vv_uranus_bigtts",
|
||||
"output_path": "projects/my-video/assets/audio/narration.mp3",
|
||||
"speech_rate": 0,
|
||||
"enable_timestamp": True,
|
||||
})
|
||||
```
|
||||
|
||||
Or call the provider directly:
|
||||
|
||||
```python
|
||||
from tools.audio.doubao_tts import DoubaoTTS
|
||||
|
||||
result = DoubaoTTS().execute({
|
||||
"text": "短样本试听文本。",
|
||||
"voice_id": "zh_female_vv_uranus_bigtts",
|
||||
"output_path": "projects/my-video/assets/audio/doubao_sample.mp3",
|
||||
})
|
||||
```
|
||||
|
||||
The provider writes:
|
||||
|
||||
- `output_path`: downloaded audio file
|
||||
- `metadata_path`: full query response JSON, defaulting to `<output_path>.json`
|
||||
|
||||
## Recommended Workflow
|
||||
|
||||
1. Generate a 10-15 second sample before a full paid narration.
|
||||
2. Ask the user to approve voice naturalness, accent, and speed.
|
||||
3. Generate the full narration only after approval.
|
||||
4. Keep the query JSON. It is the source of truth for subtitle timing.
|
||||
5. Build captions from `sentences[].words[]`, not from estimated text length.
|
||||
6. Group captions by Chinese semantic phrases before applying timestamps. Do not split only by fixed character count; it can break phrases like "在不押单个公司的情况下" or "可能会被慢慢稀释" and hurt comprehension.
|
||||
7. Let the video duration follow the approved voice rhythm unless the user explicitly asks to match a prior runtime.
|
||||
|
||||
## Parameters
|
||||
|
||||
- `voice_id`: Doubao `speaker` / voice type. Defaults to `DOUBAO_SPEECH_VOICE_TYPE`.
|
||||
- `resource_id`: use `seed-tts-2.0` for Doubao Speech 2.0 voices.
|
||||
- `speech_rate`: `0` is normal, `100` is 2x, `-50` is 0.5x.
|
||||
- `sample_rate`: default `24000`.
|
||||
- `enable_timestamp`: default `true`.
|
||||
- `return_usage`: default `true`, requests usage metadata when available.
|
||||
|
||||
Do not pass `additions.explicit_language` by default. Some endpoint/key combinations reject `zh-cn` with `unsupported additions explicit language zh-cn`.
|
||||
|
||||
For calm Mandarin explainers, start with `speech_rate: 0`. If the result is too long for the approved format, make a short comparison sample with `speech_rate: 25` or `50` before regenerating the full narration. Do not speed up only to match a previous provider's duration if the user prefers Doubao's natural pace.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- `load grant: requested grant not found`: wrong key type or wrong auth header. Use `X-Api-Key` for new-console API Keys.
|
||||
- `speaker permission denied`: voice id is wrong or not authorized for the selected resource.
|
||||
- `quota exceeded`: quota, lifetime characters, or concurrency exceeded.
|
||||
- Missing timestamps: verify `enable_timestamp: true`, keep the query JSON, and confirm the selected endpoint returned `sentences`.
|
||||
|
||||
## Safety
|
||||
|
||||
Never print or write the API key to logs, metadata, patches, or project artifacts. `.env.example` should contain only empty variable names.
|
||||
@@ -0,0 +1,58 @@
|
||||
---
|
||||
name: pose-library-design
|
||||
description: Design reusable 2D character pose libraries, action cycles, and expression states for data-driven animation.
|
||||
license: MIT
|
||||
---
|
||||
|
||||
# Pose Library Design
|
||||
|
||||
Use this skill when producing `pose_library` artifacts.
|
||||
|
||||
## Pose Categories
|
||||
|
||||
- Neutral: idle, breathe, listening.
|
||||
- Attention: look_up, look_down, look_left, look_right.
|
||||
- Emotion: happy, sad, surprised, worried, determined.
|
||||
- Action: reach, point, hold, jump, flap, walk_contact, walk_passing.
|
||||
- Mouth: closed, small_o, wide_open, smile, frown, phoneme-ish shapes.
|
||||
|
||||
## Acting Pattern
|
||||
|
||||
Use timed pose sequences:
|
||||
|
||||
```text
|
||||
anticipation -> action -> hold -> settle
|
||||
```
|
||||
|
||||
Do not continuously animate every part. Holds make the acting readable.
|
||||
|
||||
## Pose Data Pattern
|
||||
|
||||
```json
|
||||
{
|
||||
"pose": "surprised",
|
||||
"parts": {
|
||||
"head": { "rotation": -6, "y": -4 },
|
||||
"pupil_left": { "x": 4, "y": -6 },
|
||||
"mouth": "small_o"
|
||||
},
|
||||
"hold_frames": 18,
|
||||
"transition": "back.out"
|
||||
}
|
||||
```
|
||||
|
||||
## Quality Checklist
|
||||
|
||||
- Required emotions have poses.
|
||||
- Required actions have poses or cycles.
|
||||
- Reused cycles have contact and passing poses.
|
||||
- Poses name only changed parts; defaults come from the rig.
|
||||
|
||||
## Sources
|
||||
|
||||
- GSAP timeline sequencing for readable multi-step poses:
|
||||
https://gsap.com/docs/v3/GSAP/Timeline/
|
||||
- Remotion interpolation for frame-based transitions:
|
||||
https://www.remotion.dev/docs/interpolate
|
||||
- Remotion spring for natural motion:
|
||||
https://www.remotion.dev/docs/spring
|
||||
@@ -95,7 +95,72 @@ seedance.execute({
|
||||
})
|
||||
```
|
||||
|
||||
## Prompt structure
|
||||
## Prompt structure — The Higgsfield Methodology (canonical as of 2026)
|
||||
|
||||
**CRITICAL: Open every prompt with a shot-structure declaration.** Seedance rewards prompts that declare format upfront before any creative description. This is the single biggest quality lever.
|
||||
|
||||
### Opener templates (copy one verbatim, then extend)
|
||||
|
||||
**For action/combat/multi-shot (highest-performing format):**
|
||||
```
|
||||
Montage, multi-shot Hollywood action, don't use one camera angle or single cut, cinematic lighting, photorealistic, 35mm film quality, ARRI ALEXA aesthetic, heavy film grain, sharp but imperfect focus, motion blur on fast actions, halation on highlights, soft highlight rolloff, wide-angle lens with strong distortion, subtle chromatic aberration near frame edges, no 3D, no cartoon, no VFX aesthetic.
|
||||
```
|
||||
|
||||
**For single-POV continuous shots (orbs, walkthrough):**
|
||||
```
|
||||
Single continuous shot, first-person POV perspective, the camera IS [his/her] eyes, hyper-chaotic handheld motion, completely unstabilized, violent raw human movement, constant micro-jitters, aggressive head swings, abrupt jerks, frequent over-rotation, no smoothness at all, no cuts, no zoom, 35mm film, photorealistic.
|
||||
```
|
||||
|
||||
**For locked-POV reaction scenes:**
|
||||
```
|
||||
One continuous shot, POV [setting] perspective, no cuts, no zoom, natural head movement, photorealistic, 35mm film grain.
|
||||
```
|
||||
|
||||
### Body structure (after the opener)
|
||||
|
||||
1. **Environment/location** — sensory detail (wet asphalt, sodium lamps, neon bleed, rain particulates, volumetric haze)
|
||||
2. **Character block** — with reference tags and identity-lock language (see Reference-to-video below)
|
||||
3. **Enemy/secondary character block** — same detail level
|
||||
4. **Beat-by-beat choreography** with TEMPORAL MARKERS: `0–3s: … 3–6s: … 6–10s: …`
|
||||
5. **VFX inline in brackets:** `[VFX: branching white-blue electric arcs pulsing along forearms, sparks jumping between fingers]`
|
||||
6. **Slow-motion markers:** write `RAMPS TO SLOW MOTION` before the impact beat, `SNAPS BACK TO REAL TIME` on resume
|
||||
7. **Sound design block:** either `no music, only raw SFX` or explicit SFX sequence. Music language stays textural.
|
||||
|
||||
### Combat vocabulary (proven to hit)
|
||||
|
||||
- `snaps forward`, `lunges`, `sprints`, `weaves`, `chambers`, `drives`, `pivots`, `redirects`, `ducks`, `slips`
|
||||
- `explodes outward`, `devastating`, `raw force`, `kinetic`, `overload`, `compresses`, `erupts`, `fractures`, `ripples`
|
||||
- Avoid soft verbs: `attacks`, `hits`, `fights` — these read generic and Seedance underdelivers on them
|
||||
|
||||
### Camera behavior — state what it IS and ISN'T doing
|
||||
|
||||
Seedance misfires when camera intent is ambiguous. Always explicitly negate what you don't want:
|
||||
- `no cuts` (for continuous POV)
|
||||
- `no zoom` (prevents unnatural perspective punch-ins)
|
||||
- `no stabilization` (when you want chaotic handheld)
|
||||
- `no smoothness at all`
|
||||
- `no 3D, no cartoon, no VFX aesthetic` — counter-intuitive but forces photoreal skin/texture/lighting even when the scene has heavy VFX elements
|
||||
|
||||
### Realism enforcement phrase
|
||||
|
||||
When the brief has VFX but you want photoreal skin/textures (not plastic Marvel-cartoon look), include:
|
||||
```
|
||||
no 3D, no cartoon, no VFX aesthetic — photorealistic textures, real skin pores, authentic fabric detail, grounded in reality
|
||||
```
|
||||
|
||||
### Format priority (Higgsfield empirical ordering)
|
||||
|
||||
| Format | Best for | Pattern |
|
||||
|---|---|---|
|
||||
| **Transformation** | calm → threat → transformation → aftermath | 6 numbered shots × 2.5s each @ 15s total |
|
||||
| **Orbs** | single continuous POV | 1 shot × 15s, hyper-chaotic handheld |
|
||||
| **Fights** | combat choreography | Beat-by-beat, clear power mismatch, RAMPS/SNAPS |
|
||||
| **POV** | locked reaction | Continuous, "no cuts no zoom" mantra |
|
||||
| **Animation** | stylized 3D | `@image` keyframe + timed segments |
|
||||
|
||||
The **2.5-second-per-shot rhythm** appears optimal for multi-shot generations.
|
||||
|
||||
## Legacy 8-part template (use only for single simple shots, not action)
|
||||
|
||||
Seedance 2.0 is unusually literal about camera language, multi-shot cuts, and quoted dialogue. Use this 8-part template:
|
||||
|
||||
@@ -138,15 +203,33 @@ Do **not** request complex multi-instrument scores — keep music language textu
|
||||
|
||||
### Reference-to-video
|
||||
|
||||
When you have character / product / wardrobe references, use the reference-to-video endpoint and name each asset in the prompt:
|
||||
When you have character / product / wardrobe references, use the reference-to-video endpoint. Seedance 2.0 honors an explicit bracket tagging syntax:
|
||||
|
||||
```
|
||||
Reference 1: hero character (Aang) — bald, blue arrow tattoo, orange robes.
|
||||
Reference 2: environment plate — snowy Air Temple courtyard at dawn.
|
||||
Shot 1: Aang (from reference 1) walks across the courtyard (reference 2),
|
||||
wind lifting his robes. Low-angle tracking shot, slow push-in.
|
||||
[reference_image: hero_portrait.png]
|
||||
[identity_lock]
|
||||
The same character — bald, blue arrow tattoo, orange robes — consistent across all shots, no drift or deformation. Do not alter clothing category or primary color.
|
||||
|
||||
Shot 1 (wide, slow push-in): hero walks across the snowy Air Temple courtyard, wind lifting robes.
|
||||
Shot 2 (medium close-up): hero turns toward camera, staff in hand.
|
||||
Shot 3 (extreme close-up, rack focus): hero's eyes open, wind whipping.
|
||||
```
|
||||
|
||||
**Identity-anchor phrases that measurably reduce face drift** (stack them — redundancy helps):
|
||||
- `the same character`
|
||||
- `consistent across different scenes / all shots`
|
||||
- `maintain exact appearance from reference image`
|
||||
- `no deformation, no drift, no face morph`
|
||||
- `Do not alter clothing category or primary color`
|
||||
|
||||
**Single-reference workflow (common in practice):** When you only have one photo:
|
||||
- Use a clear, front-facing portrait with neutral lighting and minimal motion blur; avoid occluded faces (e.g., phones, sunglasses, heavy shadow).
|
||||
- Reuse the SAME reference image across all shots — do not generate new refs per shot.
|
||||
- Put all shots in ONE prompt under a single `[identity_lock]` block so the model treats them as a coherent sequence.
|
||||
- If wardrobe is changing by design (e.g., civilian → costume), describe the costume verbatim on every shot it appears and add `Do not alter clothing category or primary color` to lock it once generated.
|
||||
|
||||
**Anti-drift fallback:** If face morphs across frames on first render, drop to a shorter duration (5-6s instead of 10s), tighten the identity-lock language, and if you have multiple reference images, cull to the 3 most consistent ones rather than flooding with 9.
|
||||
|
||||
## Parameter guidance
|
||||
|
||||
| Parameter | Guidance |
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
---
|
||||
name: svg-character-animation
|
||||
description: Animate SVG character rigs with GSAP, CSS transforms, Remotion frame control, and HyperFrames-compatible browser previews.
|
||||
license: MIT
|
||||
---
|
||||
|
||||
# SVG Character Animation
|
||||
|
||||
Use this skill when animating character rigs made from SVG parts.
|
||||
|
||||
## Runtime Rules
|
||||
|
||||
- Animate transforms (`x`, `y`, `scale`, `rotation`) rather than layout.
|
||||
- Use timelines for multi-part acting beats.
|
||||
- For SVG elements, use stable pivots (`svgOrigin` or correctly scoped
|
||||
transform origins).
|
||||
- In Remotion, do not let GSAP advance with `requestAnimationFrame`; drive a
|
||||
paused timeline from the current frame.
|
||||
|
||||
## Browser Pattern
|
||||
|
||||
```js
|
||||
gsap.set("#arm_right", { svgOrigin: "390 310" });
|
||||
const tl = gsap.timeline({ defaults: { ease: "power2.inOut" } });
|
||||
tl.to("#head", { rotation: -8, duration: 0.2 })
|
||||
.to("#arm_right", { rotation: 35, duration: 0.4 }, "<");
|
||||
```
|
||||
|
||||
## Remotion Pattern
|
||||
|
||||
```tsx
|
||||
const frame = useCurrentFrame();
|
||||
const progress = frame / durationInFrames;
|
||||
timeline.progress(progress);
|
||||
```
|
||||
|
||||
## HyperFrames Pattern
|
||||
|
||||
Use HTML/SVG/GSAP components with deterministic timelines and validate via the
|
||||
HyperFrames CLI before final render.
|
||||
|
||||
## Quality Checklist
|
||||
|
||||
- Parts stay connected at pivots during motion.
|
||||
- Blinks, gaze, and mouth shapes are separate enough to read.
|
||||
- Pose holds are long enough to communicate emotion.
|
||||
- Frame sampling shows meaningful deltas, not frozen animation.
|
||||
|
||||
## Sources
|
||||
|
||||
- GSAP core transform properties and SVG handling:
|
||||
https://gsap.com/docs/v3/GSAP/CorePlugins/CSS/
|
||||
- GSAP timelines and sequencing:
|
||||
https://gsap.com/docs/v3/GSAP/Timeline/
|
||||
- Remotion `useCurrentFrame`:
|
||||
https://www.remotion.dev/docs/use-current-frame
|
||||
@@ -13,6 +13,8 @@ GOOGLE_API_KEY= # Google Imagen images, Google Cloud TTS (700+ voic
|
||||
ELEVENLABS_API_KEY= # TTS narration, music generation, sound effects
|
||||
OPENAI_API_KEY= # OpenAI TTS fallback and DALL-E image generation
|
||||
XAI_API_KEY= # Grok image generation/editing and Grok video generation
|
||||
DOUBAO_SPEECH_API_KEY= # Volcengine Doubao Speech TTS (new console API Key)
|
||||
DOUBAO_SPEECH_VOICE_TYPE= # Default Doubao speaker/voice type, e.g. zh_female_vv_uranus_bigtts
|
||||
# Piper local voices do not require env vars; install `piper-tts` via pip
|
||||
|
||||
# --- Music ---
|
||||
|
||||
+4
-1
@@ -230,6 +230,7 @@ If the folder has tracks, the proposal and asset stages should present them as o
|
||||
| `podcast-repurpose` | Podcast highlights and derivatives | beta |
|
||||
| `cinematic` | Trailer, teaser, and mood-led edits | production |
|
||||
| `animation` | Motion-graphics and animation-first videos | production |
|
||||
| `character-animation` | Local rigged cartoon characters and reusable character acting | beta |
|
||||
| `hybrid` | Source footage plus support visuals | production |
|
||||
| `avatar-spokesperson` | Presenter-led avatar or lip-sync videos | production |
|
||||
| `localization-dub` | Subtitle, dub, and translated variants | beta |
|
||||
@@ -361,7 +362,7 @@ print('HyperFrames note:', info.get('hyperframes_note'))
|
||||
|--------|----------|----------|
|
||||
| **FFmpeg** | Video-only cuts, concat, trim, subtitle burn | `ffmpeg` binary (always available) |
|
||||
| **Remotion** | React-based composition: still images → animated video, text cards, stat cards, charts, callouts, comparisons, transitions with spring physics, word-level caption burn, TalkingHead avatar | Node.js (`npx`) + `remotion-composer/` + `node_modules` |
|
||||
| **HyperFrames** | HTML/CSS/GSAP composition: kinetic typography, product promos, launch reels, website-to-video, registry-block-driven scenes | Node.js ≥ 22 + FFmpeg + `npx` (consumed via `npx @hyperframes/cli`) |
|
||||
| **HyperFrames** | HTML/CSS/GSAP composition: kinetic typography, product promos, launch reels, website-to-video, registry-block-driven scenes, SVG character rigs | Node.js ≥ 22 + FFmpeg + `npx` (consumed via `npx hyperframes`) |
|
||||
|
||||
`render_runtime` is **locked at proposal** (`proposal_packet.production_plan.render_runtime`) and **carried through edit_decisions unchanged**. `video_compose` routes based on this field; silent runtime swaps are forbidden. If the chosen runtime becomes unavailable at compose time, surface a structured blocker per "Escalate Blockers Explicitly" above. See `skills/core/hyperframes.md` for the Remotion-vs-HyperFrames decision matrix.
|
||||
|
||||
@@ -448,6 +449,7 @@ Key capability families to look for in the output:
|
||||
- **audio_processing** — Mixing, enhancement (FFmpeg-based, always local).
|
||||
- **analysis** — Transcription, scene detection, frame sampling.
|
||||
- **avatar** — Talking head and lip sync generation.
|
||||
- **character_animation** — Local character specs, SVG rigs, pose libraries, action timelines, previews, and QA.
|
||||
- **enhancement** — Upscale, background removal, face enhance, color grading.
|
||||
|
||||
Each tool in the registry declares `best_for`, `install_instructions`, `runtime` (LOCAL, API, LOCAL_GPU, HYBRID), and `status`. Read these fields — do not assume tool strengths from memory.
|
||||
@@ -637,6 +639,7 @@ The `.agents/skills/` directory is large. When you're not coming in through a to
|
||||
|---|---|
|
||||
| **Composition runtime** | `remotion`, `remotion-best-practices`, `synthetic-screen-recording` (fake terminal/UI demos via Remotion TerminalScene) |
|
||||
| **Animation knowledge (generic)** | `gsap-core`, `gsap-timeline`, `gsap-plugins` (SplitText / MorphSVG / DrawSVG / MotionPath / Flip / CustomEase), `gsap-utils`, `gsap-react`, `gsap-performance`, `gsap-scrolltrigger`, `gsap-frameworks`, `framer-motion` (Disney 12 principles), `lottie-bodymovin` (Lottie export) |
|
||||
| **Character animation** | `character-rigging`, `svg-character-animation`, `pose-library-design`, `canvas-procedural-animation`, `character-animation-qa` |
|
||||
| **Image generation** | `bfl-api`, `flux-best-practices` |
|
||||
| **Video generation** | `seedance-2-0` (preferred premium default — cinematic, trailer, multi-shot, synced audio, lip-sync), `ai-video-gen`, `ltx2` |
|
||||
| **Audio** | `elevenlabs`, `music`, `sound-effects`, `acestep`, `text-to-speech`, `setup-api-key` |
|
||||
|
||||
+2
-1
@@ -72,6 +72,7 @@ Each tool's `agent_skills[]` field bridges Layer 1 → Layer 3. See `skills/INDE
|
||||
| `tools/video/video_stitch.py` | Multi-clip assembly (stitch, spatial, validate, preview) |
|
||||
| `tools/video/video_compose.py` | Runtime-aware composition orchestrator — routes to Remotion / HyperFrames / FFmpeg based on `edit_decisions.render_runtime` |
|
||||
| `tools/video/hyperframes_compose.py` | HyperFrames runtime — workspace materialization, `hyperframes lint`/`validate`/`render`, FFmpeg floor check |
|
||||
| `tools/character/character_animation.py` | Local character-animation tools — character specs, SVG rig plans, pose libraries, action timelines, HyperFrames packages, and QA reports |
|
||||
| `lib/hyperframes_style_bridge.py` | Playbook → CSS custom properties + `DESIGN.md` bridge for HyperFrames workspaces |
|
||||
| `remotion-composer/src/components/` | 8 Remotion components (TextCard, StatCard, ProgressBar, CalloutBox, ComparisonCard + charts/) |
|
||||
| `.agents/skills/hyperframes*/` | Vendored HyperFrames Layer 3 skills (authoring contract, CLI, registry, website-to-video) |
|
||||
@@ -90,6 +91,7 @@ Each tool's `agent_skills[]` field bridges Layer 1 → Layer 3. See `skills/INDE
|
||||
| `podcast-repurpose` | `pipeline_defs/podcast-repurpose.yaml` | Podcast repurposing |
|
||||
| `cinematic` | `pipeline_defs/cinematic.yaml` | Cinematic edit |
|
||||
| `animation` | `pipeline_defs/animation.yaml` | Animation-first |
|
||||
| `character-animation` | `pipeline_defs/character-animation.yaml` | Local rigged character animation |
|
||||
| `hybrid` | `pipeline_defs/hybrid.yaml` | Source-plus-support hybrid |
|
||||
| `avatar-spokesperson` | `pipeline_defs/avatar-spokesperson.yaml` | Avatar presenter |
|
||||
| `localization-dub` | `pipeline_defs/localization-dub.yaml` | Localization and dubbing |
|
||||
@@ -115,4 +117,3 @@ Each tool's `agent_skills[]` field bridges Layer 1 → Layer 3. See `skills/INDE
|
||||
6. Let discovery happen through `tools/tool_registry.py`; do not depend on ad hoc imports
|
||||
7. Add a JSON schema in `schemas/tools/` if the tool has complex I/O
|
||||
8. Add tests only after the runtime path is correct
|
||||
|
||||
|
||||
@@ -205,15 +205,16 @@ You don't need paid API keys to make real videos. Out of the box, `make setup` g
|
||||
| **Open footage** | Archive.org + NASA + Wikimedia Commons | Free/open archival footage, educational media, and documentary texture |
|
||||
| **Extra stock** | Pexels + Unsplash + Pixabay | Free stock footage/images (developer keys are free to get) |
|
||||
| **Composition (React)** | Remotion | React-based rendering — spring-animated image scenes, text cards, stat cards, charts, TikTok-style word-level captions, TalkingHead |
|
||||
| **Composition (HTML/GSAP)** | HyperFrames | HTML/CSS/GSAP rendering — kinetic typography, product promos, launch reels, registry blocks, website-to-video |
|
||||
| **Composition (HTML/GSAP)** | HyperFrames | HTML/CSS/GSAP rendering — kinetic typography, product promos, launch reels, registry blocks, website-to-video, rigged SVG character animation |
|
||||
| **Post-production** | FFmpeg | Encoding, subtitle burn-in, audio mixing, color grading |
|
||||
| **Subtitles** | Built-in | Auto-generated captions with word-level timing |
|
||||
|
||||
OpenMontage picks between Remotion and HyperFrames at proposal time (locked as `render_runtime`). Remotion is the default for data-driven explainers and anything using the existing React scene stack; HyperFrames is the default for motion-graphics-heavy briefs that express naturally as HTML + GSAP. See `skills/core/hyperframes.md` for the full decision matrix.
|
||||
OpenMontage picks between Remotion and HyperFrames at proposal time (locked as `render_runtime`). Remotion is the default for data-driven explainers and anything using the existing React scene stack; HyperFrames is the default for motion-graphics-heavy briefs that express naturally as HTML + GSAP, including the `character-animation` pipeline's SVG/GSAP rig output. See `skills/core/hyperframes.md` for the full decision matrix.
|
||||
|
||||
**Two free-ish paths:**
|
||||
|
||||
- **Image-based video:** Piper narrates your script, images provide the visuals, and Remotion animates them into a polished edit.
|
||||
- **Local character animation:** SVG rigs, pose libraries, GSAP timelines, and HyperFrames render cartoon character acting to `projects/<project-name>/renders/final.mp4`.
|
||||
- **Real-footage video:** the documentary montage pipeline builds a CLIP-searchable corpus from Archive.org, NASA, Wikimedia Commons, and optional free-key sources like Pexels and Unsplash, then cuts together actual motion footage into a finished video.
|
||||
|
||||
If you want the second one, prompt for a **documentary montage**, **tone poem**, or **stock-footage collage**, and explicitly say **use real footage only**.
|
||||
@@ -374,8 +375,8 @@ OpenMontage/
|
||||
│ ├── avatar/ # Talking head, lip sync
|
||||
│ └── subtitle/ # SRT/VTT generation
|
||||
│
|
||||
├── pipeline_defs/ # 11 YAML pipeline manifests (the agent's playbook)
|
||||
├── skills/ # 124 Markdown skill files (the agent's knowledge)
|
||||
├── pipeline_defs/ # YAML pipeline manifests (the agent's playbook)
|
||||
├── skills/ # Markdown skill files (the agent's knowledge)
|
||||
│ ├── pipelines/ # Per-pipeline stage director skills
|
||||
│ ├── creative/ # Creative technique skills
|
||||
│ ├── core/ # Core tool skills
|
||||
@@ -393,7 +394,7 @@ OpenMontage/
|
||||
```
|
||||
Layer 1: tools/ + pipeline_defs/ "What exists" — executable capabilities + orchestration
|
||||
Layer 2: skills/ "How to use it" — OpenMontage conventions and quality bars
|
||||
Layer 3: .agents/skills/ "How it works" — 47 external technology knowledge packs
|
||||
Layer 3: .agents/skills/ "How it works" — external technology knowledge packs
|
||||
```
|
||||
|
||||
Each tool declares which Layer 3 skills it relies on. The agent reads Layer 1 to know what's available, Layer 2 to know how OpenMontage wants it used, and Layer 3 for deep technical knowledge when needed.
|
||||
@@ -509,7 +510,7 @@ Each tool declares which Layer 3 skills it relies on. The agent reads Layer 1 to
|
||||
| Engine | Type | What It Does |
|
||||
|--------|------|-------------|
|
||||
| **Remotion** | Local (Node.js) | React-based programmatic video — spring-animated image scenes, stat reveals, section titles, hero cards, TikTok-style word-by-word captions, scene transitions (fade/slide/wipe/flip), Google Fonts, audio with fade curves, and the TalkingHead avatar composition. **When no video generation providers are configured, the agent generates still images and Remotion turns them into fully animated video.** |
|
||||
| **HyperFrames** | Local (Node.js ≥ 22) | HTML/CSS/GSAP programmatic video — kinetic typography, product promos, launch reels, custom motion graphics, registry blocks (data charts, grain overlays, shader transitions), website-to-video workflows. Consumed via `npx @hyperframes/cli`; no monorepo checkout needed. |
|
||||
| **HyperFrames** | Local (Node.js ≥ 22) | HTML/CSS/GSAP programmatic video — kinetic typography, product promos, launch reels, custom motion graphics, registry blocks (data charts, grain overlays, shader transitions), website-to-video workflows, and rigged SVG character animation. Consumed via `npx hyperframes`; no monorepo checkout needed. |
|
||||
| **FFmpeg** | Local | Core video assembly, encoding, subtitle burn, audio muxing, color grading |
|
||||
|
||||
Runtime is chosen at proposal (`render_runtime`) and locked through `edit_decisions`. Silent swaps between runtimes are a governance violation — see `skills/core/hyperframes.md`.
|
||||
|
||||
+14
-7
@@ -53,7 +53,7 @@ OpenMontage/
|
||||
│ ├── subtitle/ # SRT/VTT generation from timestamps
|
||||
│ └── video/ # 13 video gen providers, composition, stitching, trimming
|
||||
│
|
||||
├── pipeline_defs/ # 11 YAML pipeline manifests
|
||||
├── pipeline_defs/ # YAML pipeline manifests
|
||||
├── schemas/ # JSON Schema definitions for validation
|
||||
│ ├── artifacts/ # 11 artifact schemas (brief → publish_log)
|
||||
│ ├── checkpoints/ # Checkpoint state schema
|
||||
@@ -65,9 +65,9 @@ OpenMontage/
|
||||
│ ├── core/ # FFmpeg, Remotion, WhisperX, color grading skills
|
||||
│ ├── creative/ # Video editing, enhancement, data viz, prompt engineering
|
||||
│ ├── meta/ # reviewer, checkpoint-protocol, skill-creator
|
||||
│ └── pipelines/ # Per-pipeline stage-director skills (10 pipelines)
|
||||
│ └── pipelines/ # Per-pipeline stage-director skills
|
||||
│
|
||||
├── .agents/skills/ # Layer 3: 47 external technology skills (FFmpeg, ElevenLabs, FLUX, etc.)
|
||||
├── .agents/skills/ # Layer 3: external technology skills (FFmpeg, HyperFrames, GSAP, etc.)
|
||||
├── styles/ # Visual style playbooks (YAML) + loader
|
||||
├── remotion-composer/ # Node.js/React — Remotion video composition renderer
|
||||
├── tests/ # Contract tests, QA integration tests, eval harness
|
||||
@@ -200,13 +200,14 @@ stages:
|
||||
# ... through publish
|
||||
```
|
||||
|
||||
### Available Pipelines (11)
|
||||
### Available Pipelines
|
||||
|
||||
| Pipeline | Category | Description |
|
||||
|----------|----------|-------------|
|
||||
| `animated-explainer` | generated | AI-produced explainer with research, narration, visuals, music |
|
||||
| `animation` | animation | Motion graphics, kinetic typography |
|
||||
| `avatar-spokesperson` | talking_head | Avatar-driven presenter videos |
|
||||
| `character-animation` | animation | Local rigged cartoon characters with SVG rigs, pose libraries, GSAP timelines, and HyperFrames rendering |
|
||||
| `cinematic` | cinematic | Trailer, teaser, mood-driven edits |
|
||||
| `clip-factory` | custom | Batch short-form clips from long source |
|
||||
| `hybrid` | hybrid | Source footage + AI-generated support visuals |
|
||||
@@ -218,7 +219,7 @@ stages:
|
||||
|
||||
### Standard Stage Progression
|
||||
|
||||
All production pipelines follow a canonical 8-stage flow:
|
||||
Most production pipelines follow a canonical 8-stage flow:
|
||||
|
||||
```
|
||||
research → proposal → script → scene_plan → assets → edit → compose → publish
|
||||
@@ -231,6 +232,11 @@ Each stage:
|
||||
4. Has **review_focus** criteria and **success_criteria**
|
||||
5. Can require **human approval** before proceeding
|
||||
|
||||
Specialized pipelines may insert domain-specific stages. For example,
|
||||
`character-animation` adds `character_design` and `rig_plan` before
|
||||
`scene_plan`, then emits a HyperFrames workspace and final deliverable at
|
||||
`projects/<project-name>/renders/final.mp4`.
|
||||
|
||||
---
|
||||
|
||||
## Checkpoint System
|
||||
@@ -437,11 +443,12 @@ A standalone Node.js/React subproject in `remotion-composer/` using [Remotion](h
|
||||
|
||||
### HyperFrames (HTML/CSS/GSAP)
|
||||
|
||||
Consumed via `npx @hyperframes/cli` (no monorepo checkout needed). Runtime floor: Node.js ≥ 22, FFmpeg, `npx`.
|
||||
Consumed via `npx hyperframes` (no monorepo checkout needed). Runtime floor: Node.js ≥ 22, FFmpeg, `npx`.
|
||||
|
||||
- Handles kinetic typography, product promos, launch reels, website-to-video, registry blocks
|
||||
- Handles kinetic typography, product promos, launch reels, website-to-video, registry blocks, and SVG/GSAP character rigs
|
||||
- Driver: `tools/video/hyperframes_compose.py` materializes a workspace under `projects/<name>/hyperframes/`, then runs `lint → validate → render`
|
||||
- Layer 3 skills vendored at `.agents/skills/hyperframes*/`; Layer 2 guide at `skills/core/hyperframes.md`
|
||||
- The `character-animation` pipeline uses HyperFrames as the production render package. Browser previews are QA/debug artifacts only, not the render path.
|
||||
|
||||
### FFmpeg (fallback / simple cuts)
|
||||
|
||||
|
||||
@@ -39,6 +39,8 @@ GOOGLE_API_KEY= # Google TTS + Google Imagen
|
||||
ELEVENLABS_API_KEY= # TTS, music, sound effects (10K chars/month free)
|
||||
OPENAI_API_KEY= # OpenAI TTS + DALL-E 3 images
|
||||
XAI_API_KEY= # xAI Grok image generation/editing + Grok video generation
|
||||
DOUBAO_SPEECH_API_KEY= # Volcengine Doubao Speech TTS (strong Mandarin narration)
|
||||
DOUBAO_SPEECH_VOICE_TYPE= # Default Doubao speaker/voice type
|
||||
|
||||
# MULTI-MODEL GATEWAY (one key, 6+ tools)
|
||||
FAL_KEY= # FLUX, Recraft, Kling, Veo, MiniMax video
|
||||
@@ -159,6 +161,52 @@ No subscription — pure pay-as-you-go, no minimum spend.
|
||||
|
||||
---
|
||||
|
||||
### Doubao Speech — Mandarin TTS
|
||||
|
||||
> **Strong Mandarin narration.** Volcengine Doubao Speech is a good choice for Chinese explainer voiceovers and long-form narration that needs subtitle timing metadata.
|
||||
|
||||
**Tools unlocked:** `doubao_tts`
|
||||
**Env vars:** `DOUBAO_SPEECH_API_KEY`, `DOUBAO_SPEECH_VOICE_TYPE`
|
||||
|
||||
#### Setup
|
||||
|
||||
1. Open the Volcengine Doubao Speech console and enable Speech Synthesis 2.0.
|
||||
2. Create a new-console API Key.
|
||||
3. Choose a Speech 2.0 voice type, for example `zh_female_vv_uranus_bigtts`.
|
||||
4. Add to `.env`:
|
||||
```bash
|
||||
DOUBAO_SPEECH_API_KEY=your-api-key
|
||||
DOUBAO_SPEECH_VOICE_TYPE=zh_female_vv_uranus_bigtts
|
||||
```
|
||||
|
||||
#### API Notes
|
||||
|
||||
OpenMontage uses the new-console API key flow:
|
||||
|
||||
```text
|
||||
X-Api-Key: ${DOUBAO_SPEECH_API_KEY}
|
||||
X-Api-Resource-Id: seed-tts-2.0
|
||||
```
|
||||
|
||||
Do not pass a new-console API Key as `X-Api-App-Id` or `X-Api-Access-Key`. That mismatch can produce `load grant: requested grant not found`.
|
||||
|
||||
#### What It Is Best For
|
||||
|
||||
- Natural Mandarin narration for Chinese-language explainers
|
||||
- Async long-form narration via `/api/v3/tts/submit` and `/api/v3/tts/query`
|
||||
- Character-level timing metadata for subtitle alignment
|
||||
- Calm educational pacing where the video duration can follow the approved voice rhythm
|
||||
|
||||
#### Pacing
|
||||
|
||||
Start with `speech_rate: 0` for natural Mandarin delivery. If the approved format needs a tighter runtime, compare short samples at `speech_rate: 25` or `50` before generating the full narration. Do not force Doubao to match another provider's duration unless the user explicitly wants that tradeoff.
|
||||
|
||||
#### Pricing
|
||||
|
||||
Doubao Speech 2.0 is billed by character package or usage in Volcengine. OpenMontage estimates cost from text length and prefers provider-returned usage metadata when available.
|
||||
|
||||
---
|
||||
|
||||
### Google — TTS + Imagen (Shared Key)
|
||||
|
||||
> **One key, two tools.** Google Cloud TTS has 700+ voices in 50+ languages — the strongest localization option. Imagen 4 generates high-quality images.
|
||||
@@ -503,6 +551,39 @@ If Remotion is not installed, compositions fall back to FFmpeg Ken Burns pan-and
|
||||
|
||||
---
|
||||
|
||||
### HyperFrames - HTML/CSS/GSAP Video Composition
|
||||
|
||||
> **GSAP-native local rendering.** HyperFrames is the preferred runtime for motion-graphics-heavy HTML compositions and the `character-animation` pipeline's rigged SVG character acting.
|
||||
|
||||
**Tool:** `hyperframes_compose` directly, or `video_compose` with `edit_decisions.render_runtime="hyperframes"`
|
||||
**Runtime:** CPU (Node.js >= 22, FFmpeg, and `npx` required)
|
||||
**Env var:** None
|
||||
|
||||
#### Setup
|
||||
|
||||
```bash
|
||||
node --version
|
||||
ffmpeg -version
|
||||
npx --yes hyperframes doctor
|
||||
```
|
||||
|
||||
The CLI is consumed as `npx hyperframes`. Do not use `npx @hyperframes/cli`; that package name is not the OpenMontage runtime path.
|
||||
|
||||
#### What HyperFrames Renders
|
||||
|
||||
| Use case | What it produces |
|
||||
|----------|------------------|
|
||||
| **Kinetic typography** | HTML/CSS text animation driven by GSAP timelines |
|
||||
| **Product / launch videos** | Structured HTML scenes, registry blocks, and transitions |
|
||||
| **Website-to-video** | Browser-captured site compositions with HyperFrames validation |
|
||||
| **Character animation** | SVG character rigs, pose/action timelines, and GSAP acting beats rendered to `renders/final.mp4` |
|
||||
|
||||
HyperFrames workspaces live under `projects/<project-name>/hyperframes/`. Final videos still follow the normal OpenMontage convention: `projects/<project-name>/renders/final.mp4`.
|
||||
|
||||
**Cost:** Free. Always local.
|
||||
|
||||
---
|
||||
|
||||
### Piper TTS — Offline Text-to-Speech
|
||||
|
||||
> **Completely free, fully offline TTS.** No network required. Good quality for drafts and budget-constrained projects.
|
||||
|
||||
@@ -44,7 +44,7 @@ def _probe_video(path: Path, tool_registry: Any) -> dict[str, Any]:
|
||||
|
||||
# Technical probe via audio_probe or ffprobe
|
||||
try:
|
||||
audio_probe = tool_registry.get_tool("audio_probe")
|
||||
audio_probe = tool_registry.get("audio_probe")
|
||||
if audio_probe:
|
||||
probe_result = audio_probe.execute({"input_path": str(path)})
|
||||
if probe_result.success:
|
||||
@@ -84,7 +84,7 @@ def _probe_video(path: Path, tool_registry: Any) -> dict[str, Any]:
|
||||
|
||||
# Sample frames
|
||||
try:
|
||||
frame_sampler = tool_registry.get_tool("frame_sampler")
|
||||
frame_sampler = tool_registry.get("frame_sampler")
|
||||
if frame_sampler:
|
||||
duration = result["technical_probe"].get("duration_seconds", 0)
|
||||
timestamps = _sample_timestamps(duration, count=4)
|
||||
@@ -122,7 +122,7 @@ def _probe_audio(path: Path, tool_registry: Any) -> dict[str, Any]:
|
||||
result: dict[str, Any] = {"technical_probe": {}, "quality_risks": []}
|
||||
|
||||
try:
|
||||
audio_probe = tool_registry.get_tool("audio_probe")
|
||||
audio_probe = tool_registry.get("audio_probe")
|
||||
if audio_probe:
|
||||
probe_result = audio_probe.execute({"input_path": str(path)})
|
||||
if probe_result.success:
|
||||
@@ -195,7 +195,7 @@ def _transcribe_if_available(
|
||||
return None
|
||||
|
||||
try:
|
||||
transcriber = tool_registry.get_tool("transcriber")
|
||||
transcriber = tool_registry.get("transcriber")
|
||||
if transcriber and transcriber.get_status().value == "available":
|
||||
result = transcriber.execute({"input_path": str(path)})
|
||||
if result.success:
|
||||
|
||||
@@ -0,0 +1,302 @@
|
||||
name: character-animation
|
||||
version: "0.1"
|
||||
description: >
|
||||
Character animation pipeline for local, reusable cartoon characters. It turns
|
||||
a script and scene plan into character specs, rig plans, pose libraries, action
|
||||
timelines, and browser-rendered SVG/Canvas/Remotion/HyperFrames animation.
|
||||
The pipeline is designed for deterministic local motion, not remote video-gen
|
||||
replacement.
|
||||
category: animation
|
||||
stability: beta
|
||||
default_checkpoint_policy: guided
|
||||
|
||||
reference_input:
|
||||
supported: true
|
||||
analysis_depth: standard
|
||||
analysis_tools:
|
||||
- video_analyzer
|
||||
- transcript_fetcher
|
||||
- video_downloader
|
||||
- scene_detect
|
||||
- frame_sampler
|
||||
|
||||
extensions:
|
||||
custom_scripts: true
|
||||
custom_playbooks: true
|
||||
custom_skills: true
|
||||
custom_tools: false
|
||||
|
||||
required_skills:
|
||||
- pipelines/character-animation/executive-producer
|
||||
- pipelines/character-animation/research-director
|
||||
- pipelines/character-animation/proposal-director
|
||||
- pipelines/character-animation/script-director
|
||||
- pipelines/character-animation/character-design-director
|
||||
- pipelines/character-animation/rig-plan-director
|
||||
- pipelines/character-animation/scene-director
|
||||
- pipelines/character-animation/asset-director
|
||||
- pipelines/character-animation/edit-director
|
||||
- pipelines/character-animation/compose-director
|
||||
- pipelines/character-animation/publish-director
|
||||
- meta/reviewer
|
||||
- meta/checkpoint-protocol
|
||||
- meta/animation-runtime-selector
|
||||
|
||||
orchestration:
|
||||
mode: executive-producer
|
||||
skill: pipelines/character-animation/executive-producer
|
||||
budget_default_usd: 2.00
|
||||
max_revisions_per_stage: 3
|
||||
max_send_backs: 3
|
||||
max_wall_time_minutes: 20
|
||||
|
||||
compatible_playbooks:
|
||||
recommended:
|
||||
- flat-motion-graphics
|
||||
also_works:
|
||||
- clean-professional
|
||||
- minimalist-diagram
|
||||
custom_allowed: true
|
||||
|
||||
stages:
|
||||
- name: research
|
||||
skill: pipelines/character-animation/research-director
|
||||
produces:
|
||||
- research_brief
|
||||
tools_available: []
|
||||
checkpoint_required: false
|
||||
human_approval_default: false
|
||||
review_focus:
|
||||
- Reference style and character-animation technique are researched when a reference exists
|
||||
- At least 3 comparable character-animation examples or techniques are summarized
|
||||
- Feasibility separates rigged local animation from frame-by-frame traditional animation
|
||||
success_criteria:
|
||||
- Schema-valid research_brief
|
||||
- Technique notes identify reusable rigs, pose libraries, and effects needs
|
||||
|
||||
- name: proposal
|
||||
skill: pipelines/character-animation/proposal-director
|
||||
required_artifacts_in:
|
||||
- research_brief
|
||||
produces:
|
||||
- proposal_packet
|
||||
- decision_log
|
||||
tools_available: []
|
||||
checkpoint_required: true
|
||||
human_approval_default: true
|
||||
review_focus:
|
||||
- Concepts are differentiated and not a carbon copy of any reference
|
||||
- Character count, action complexity, and reuse strategy are explicit
|
||||
- Render runtime selection presents Remotion and HyperFrames when both are available
|
||||
- Motion requirement is preserved; FFmpeg-only fallback is not proposed for character acting
|
||||
- Sample-first plan is included before full production
|
||||
success_criteria:
|
||||
- Schema-valid proposal_packet
|
||||
- Selected concept includes character_count, rig_complexity, reuse_strategy, and render_runtime
|
||||
- User approval is recorded before character assets are generated
|
||||
sub_stages:
|
||||
- name: sample
|
||||
description: "10-15 second character-animation proof before full production"
|
||||
condition: "approved_concept_exists"
|
||||
human_approval_default: true
|
||||
tools_available:
|
||||
- character_spec_generator
|
||||
- svg_rig_builder
|
||||
- pose_library_builder
|
||||
- action_timeline_compiler
|
||||
- character_rig_renderer
|
||||
- character_animation_reviewer
|
||||
- video_compose
|
||||
review_focus:
|
||||
- Sample proves character style, rig integrity, pose readability, and motion timing
|
||||
|
||||
- name: script
|
||||
skill: pipelines/character-animation/script-director
|
||||
required_artifacts_in:
|
||||
- proposal_packet
|
||||
optional_artifacts_in:
|
||||
- research_brief
|
||||
- video_analysis_brief
|
||||
produces:
|
||||
- script
|
||||
tools_available:
|
||||
- transcriber
|
||||
checkpoint_required: true
|
||||
human_approval_default: true
|
||||
review_focus:
|
||||
- Script is written as action beats, not only narration
|
||||
- Dialogue/narration architecture is locked
|
||||
- Every emotional turn can be expressed with poses and actions
|
||||
success_criteria:
|
||||
- Schema-valid script
|
||||
- Character beats and audio architecture are explicit
|
||||
|
||||
- name: character_design
|
||||
skill: pipelines/character-animation/character-design-director
|
||||
required_artifacts_in:
|
||||
- script
|
||||
- proposal_packet
|
||||
produces:
|
||||
- character_design
|
||||
tools_available:
|
||||
- character_spec_generator
|
||||
- image_selector
|
||||
checkpoint_required: true
|
||||
human_approval_default: true
|
||||
review_focus:
|
||||
- Each character has a distinct role, silhouette, emotional range, and action list
|
||||
- Character count is realistic for local rigging
|
||||
- Style anchors are reusable across scenes
|
||||
success_criteria:
|
||||
- Schema-valid character_design
|
||||
- Every main character has required emotions, actions, and views
|
||||
|
||||
- name: rig_plan
|
||||
skill: pipelines/character-animation/rig-plan-director
|
||||
required_artifacts_in:
|
||||
- character_design
|
||||
produces:
|
||||
- rig_plan
|
||||
- pose_library
|
||||
tools_available:
|
||||
- svg_rig_builder
|
||||
- pose_library_builder
|
||||
checkpoint_required: true
|
||||
human_approval_default: false
|
||||
review_focus:
|
||||
- Parts, pivots, layers, constraints, views, and required poses are complete
|
||||
- Rig plan avoids per-character code paths; character differences are data
|
||||
- Known risky motions are surfaced before asset generation
|
||||
success_criteria:
|
||||
- Schema-valid rig_plan
|
||||
- Schema-valid pose_library
|
||||
- Every required action has at least one pose or action strategy
|
||||
|
||||
- name: scene_plan
|
||||
skill: pipelines/character-animation/scene-director
|
||||
required_artifacts_in:
|
||||
- script
|
||||
- character_design
|
||||
- rig_plan
|
||||
- pose_library
|
||||
optional_artifacts_in:
|
||||
- proposal_packet
|
||||
produces:
|
||||
- scene_plan
|
||||
tools_available: []
|
||||
checkpoint_required: true
|
||||
human_approval_default: true
|
||||
review_focus:
|
||||
- Scenes use character_scene or animation types with timed actions
|
||||
- Each scene has characters, actions, camera/framing, background, and effects
|
||||
- Scene complexity fits the approved sample and budget
|
||||
success_criteria:
|
||||
- Schema-valid scene_plan
|
||||
- Every scene maps to rigged characters and required assets
|
||||
|
||||
- name: assets
|
||||
skill: pipelines/character-animation/asset-director
|
||||
required_artifacts_in:
|
||||
- character_design
|
||||
- rig_plan
|
||||
- pose_library
|
||||
- scene_plan
|
||||
optional_artifacts_in:
|
||||
- script
|
||||
- proposal_packet
|
||||
produces:
|
||||
- asset_manifest
|
||||
tools_available:
|
||||
- image_selector
|
||||
- tts_selector
|
||||
- music_gen
|
||||
- character_rig_renderer
|
||||
checkpoint_required: true
|
||||
human_approval_default: false
|
||||
review_focus:
|
||||
- Character parts, backgrounds, props, audio, and effects are linked to scenes
|
||||
- Layer 3 skills are read for every generation or animation-runtime tool
|
||||
- Asset provenance and prompts are recorded
|
||||
- Missing rig assets are blocked, not hidden
|
||||
success_criteria:
|
||||
- Schema-valid asset_manifest
|
||||
- All referenced asset files exist
|
||||
- Character assets are organized under projects/<name>/assets/characters
|
||||
|
||||
- name: edit
|
||||
skill: pipelines/character-animation/edit-director
|
||||
required_artifacts_in:
|
||||
- scene_plan
|
||||
- asset_manifest
|
||||
- pose_library
|
||||
optional_artifacts_in:
|
||||
- script
|
||||
- rig_plan
|
||||
produces:
|
||||
- edit_decisions
|
||||
- action_timeline
|
||||
tools_available:
|
||||
- action_timeline_compiler
|
||||
checkpoint_required: true
|
||||
human_approval_default: false
|
||||
review_focus:
|
||||
- Action timeline preserves acting beats and emotional readability
|
||||
- Pose transitions are timed with holds, anticipation, and follow-through
|
||||
- render_runtime is carried from proposal unchanged
|
||||
success_criteria:
|
||||
- Schema-valid edit_decisions
|
||||
- Schema-valid action_timeline
|
||||
- Every scene has timed actions
|
||||
|
||||
- name: compose
|
||||
skill: pipelines/character-animation/compose-director
|
||||
required_artifacts_in:
|
||||
- edit_decisions
|
||||
- action_timeline
|
||||
- asset_manifest
|
||||
optional_artifacts_in:
|
||||
- rig_plan
|
||||
- pose_library
|
||||
- proposal_packet
|
||||
produces:
|
||||
- render_report
|
||||
- final_review
|
||||
- character_qa_report
|
||||
tools_available:
|
||||
- character_rig_renderer
|
||||
- character_animation_reviewer
|
||||
- video_compose
|
||||
- audio_mixer
|
||||
checkpoint_required: true
|
||||
human_approval_default: false
|
||||
review_focus:
|
||||
- Runtime chosen in proposal is the runtime actually used
|
||||
- Browser preview passes Playwright/screenshot checks where available
|
||||
- Final MP4 passes ffprobe, frame sampling, character QA, and visual self-review
|
||||
- No silent downgrade to still-image motion
|
||||
success_criteria:
|
||||
- Schema-valid render_report
|
||||
- Schema-valid final_review
|
||||
- Schema-valid character_qa_report
|
||||
- Output exists and passes technical validation
|
||||
|
||||
- name: publish
|
||||
skill: pipelines/character-animation/publish-director
|
||||
required_artifacts_in:
|
||||
- render_report
|
||||
- final_review
|
||||
- character_qa_report
|
||||
optional_artifacts_in:
|
||||
- proposal_packet
|
||||
- script
|
||||
produces:
|
||||
- publish_log
|
||||
tools_available: []
|
||||
checkpoint_required: true
|
||||
human_approval_default: true
|
||||
review_focus:
|
||||
- Metadata describes the character-animation style honestly
|
||||
- Thumbnail/poster frame features the main character and emotional hook
|
||||
- Any limitations are surfaced in delivery notes
|
||||
success_criteria:
|
||||
- Schema-valid publish_log
|
||||
@@ -24,6 +24,7 @@ When you add a new component, append it here and in `src/components/index.ts`.
|
||||
| `progress_bar` | `ProgressBar` | `progress` | `progressLabel`, `progressColor`, `progressSegments` | Animated progress |
|
||||
| `anime_scene` | `AnimeScene` | `images` (list) | `particles`, `lightingFrom`, `lightingTo`, `vignette` | Still-image anime scene with particles + camera motion |
|
||||
| **`terminal_scene`** | **`TerminalScene`** | **`steps`** (list of cmd/out/pause/pill) | **`terminalTitle`, `prompt`, `accentColor`** | **Synthetic terminal animation — NO real capture needed. See [`.agents/skills/synthetic-screen-recording/SKILL.md`](../.agents/skills/synthetic-screen-recording/SKILL.md)** |
|
||||
| **`screenshot_scene`** | **`ScreenshotScene`** | **`backgroundImage`** (path in `public/`), **`screenshotSteps`** (list of overlays) | **`screenshotSize` (natural px w/h), `cursorStartAt`, `accentColor`** | **Approach-1 synthetic UI — drop any screenshot, animate scripted overlays on top (cursor, click_pulse, type_into, bubble_append, typing_dots, highlight_box, callout_balloon). Viewer-indistinguishable from a real recording for 15–30s focused demos. Coordinates are normalized (0–1) against the contain-fit rect. See [`.agents/skills/synthetic-ui-recording/SKILL.md`](../.agents/skills/synthetic-ui-recording/SKILL.md) (planned).** |
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -174,6 +174,10 @@ const TitleCard: React.FC<{
|
||||
titleFontSize: number;
|
||||
titleWidth: number;
|
||||
signalLineCount: number;
|
||||
backgroundSrc?: string;
|
||||
backgroundTrimBeforeSeconds?: number;
|
||||
backgroundTrimAfterSeconds?: number;
|
||||
variant?: "plate" | "overlay";
|
||||
}> = ({
|
||||
text,
|
||||
accent,
|
||||
@@ -181,85 +185,210 @@ const TitleCard: React.FC<{
|
||||
titleFontSize,
|
||||
titleWidth,
|
||||
signalLineCount,
|
||||
backgroundSrc,
|
||||
backgroundTrimBeforeSeconds,
|
||||
backgroundTrimAfterSeconds,
|
||||
variant = "plate",
|
||||
}) => {
|
||||
const frame = useCurrentFrame();
|
||||
const { fps, durationInFrames } = useVideoConfig();
|
||||
|
||||
const reveal = spring({
|
||||
const container = spring({
|
||||
fps,
|
||||
frame,
|
||||
config: { damping: 18, stiffness: 90 },
|
||||
config: { damping: 22, stiffness: 80 },
|
||||
});
|
||||
|
||||
const exit = interpolate(
|
||||
frame,
|
||||
[durationInFrames - 12, durationInFrames],
|
||||
[durationInFrames - 14, durationInFrames],
|
||||
[1, 0],
|
||||
{
|
||||
extrapolateLeft: "clamp",
|
||||
extrapolateRight: "clamp",
|
||||
},
|
||||
{ extrapolateLeft: "clamp", extrapolateRight: "clamp" },
|
||||
);
|
||||
|
||||
const y = interpolate(reveal, [0, 1], [18, 0]);
|
||||
const letterSpacing = interpolate(reveal, [0, 1], [0.3, 0.18]);
|
||||
// Split by newlines first (each line rendered in its own block),
|
||||
// then word-stagger inside each line. This preserves intentional
|
||||
// \n separators (e.g. "TITLE 1\nTITLE 2") that the old whitespace
|
||||
// regex was collapsing into a single space.
|
||||
const lines = text.split(/\r?\n/);
|
||||
const staggerFrames = 3;
|
||||
const wordFadeFrames = 14;
|
||||
|
||||
const lineGrow = interpolate(frame, [0, 22], [0, 1], {
|
||||
extrapolateLeft: "clamp",
|
||||
extrapolateRight: "clamp",
|
||||
});
|
||||
const lineExit = exit;
|
||||
const flareOpacity =
|
||||
0.18 + Math.max(0, Math.sin(frame * 0.08)) * 0.14 * intensity;
|
||||
0.22 + Math.max(0, Math.sin(frame * 0.09)) * 0.18 * intensity;
|
||||
|
||||
const bgScale = interpolate(frame, [0, durationInFrames], [1.04, 1.1], {
|
||||
extrapolateLeft: "clamp",
|
||||
extrapolateRight: "clamp",
|
||||
});
|
||||
|
||||
const bgTrimBefore =
|
||||
backgroundTrimBeforeSeconds !== undefined
|
||||
? Math.round(backgroundTrimBeforeSeconds * fps)
|
||||
: undefined;
|
||||
const bgTrimAfter =
|
||||
backgroundTrimAfterSeconds !== undefined
|
||||
? Math.round(backgroundTrimAfterSeconds * fps)
|
||||
: undefined;
|
||||
|
||||
const plateBg =
|
||||
variant === "overlay"
|
||||
? "transparent"
|
||||
: "radial-gradient(ellipse at 50% 50%, rgba(8,14,22,0.78) 0%, rgba(2,4,8,0.92) 58%, rgba(0,0,0,1) 100%)";
|
||||
|
||||
return (
|
||||
<AbsoluteFill
|
||||
style={{
|
||||
background:
|
||||
"radial-gradient(circle at 50% 42%, rgba(16,28,40,0.9) 0%, rgba(3,5,8,1) 58%, rgba(0,0,0,1) 100%)",
|
||||
background: "#000",
|
||||
justifyContent: "center",
|
||||
alignItems: "center",
|
||||
}}
|
||||
>
|
||||
{backgroundSrc ? (
|
||||
<>
|
||||
<AbsoluteFill style={{ transform: `scale(${bgScale})`, opacity: 0.62 }}>
|
||||
<OffthreadVideo
|
||||
muted
|
||||
src={resolveAsset(backgroundSrc)}
|
||||
trimBefore={bgTrimBefore}
|
||||
trimAfter={bgTrimAfter}
|
||||
style={{
|
||||
width: "100%",
|
||||
height: "100%",
|
||||
objectFit: "cover",
|
||||
filter: "contrast(1.08) saturate(0.55) brightness(0.55) blur(4px)",
|
||||
}}
|
||||
/>
|
||||
</AbsoluteFill>
|
||||
<AbsoluteFill
|
||||
style={{
|
||||
background:
|
||||
"linear-gradient(180deg, rgba(0,0,0,0.65) 0%, rgba(0,0,0,0.35) 40%, rgba(0,0,0,0.35) 60%, rgba(0,0,0,0.78) 100%)",
|
||||
}}
|
||||
/>
|
||||
</>
|
||||
) : null}
|
||||
|
||||
<AbsoluteFill
|
||||
style={{
|
||||
background: plateBg,
|
||||
}}
|
||||
/>
|
||||
|
||||
<SignalTexture
|
||||
accent={accent}
|
||||
intensity={intensity}
|
||||
intensity={intensity * 0.7}
|
||||
lineCount={signalLineCount}
|
||||
/>
|
||||
|
||||
{/* Top accent line — grows from center outwards */}
|
||||
<div
|
||||
style={{
|
||||
position: "absolute",
|
||||
width: 880,
|
||||
height: 2,
|
||||
width: 1100 * lineGrow,
|
||||
height: 1,
|
||||
background: accent,
|
||||
boxShadow: `0 0 28px ${accent}`,
|
||||
opacity: flareOpacity,
|
||||
transform: "translateY(-126px)",
|
||||
boxShadow: `0 0 24px ${accent}`,
|
||||
opacity: flareOpacity * lineExit,
|
||||
transform: "translateY(-118px)",
|
||||
}}
|
||||
/>
|
||||
<div
|
||||
style={{
|
||||
position: "absolute",
|
||||
width: 880,
|
||||
height: 2,
|
||||
width: 1100 * lineGrow,
|
||||
height: 1,
|
||||
background: accent,
|
||||
boxShadow: `0 0 28px ${accent}`,
|
||||
opacity: flareOpacity * 0.7,
|
||||
transform: "translateY(126px)",
|
||||
boxShadow: `0 0 24px ${accent}`,
|
||||
opacity: flareOpacity * 0.75 * lineExit,
|
||||
transform: "translateY(118px)",
|
||||
}}
|
||||
/>
|
||||
|
||||
{/* Word-stagger text reveal */}
|
||||
<div
|
||||
style={{
|
||||
opacity: reveal * exit,
|
||||
transform: `translateY(${y}px)`,
|
||||
fontFamily,
|
||||
fontWeight: 700,
|
||||
fontSize: titleFontSize,
|
||||
lineHeight: 1.06,
|
||||
letterSpacing: `${letterSpacing}em`,
|
||||
textAlign: "center",
|
||||
color: "#f3f6fa",
|
||||
textTransform: "uppercase",
|
||||
opacity: exit,
|
||||
width: titleWidth,
|
||||
textShadow: "0 0 22px rgba(255,255,255,0.08)",
|
||||
textAlign: "center",
|
||||
fontFamily,
|
||||
fontWeight: 500,
|
||||
fontSize: titleFontSize,
|
||||
lineHeight: 1.12,
|
||||
letterSpacing: "0.16em",
|
||||
color: "#f6f4ee",
|
||||
textTransform: "uppercase",
|
||||
textShadow: "0 0 34px rgba(255,255,255,0.10), 0 0 2px rgba(0,0,0,0.8)",
|
||||
}}
|
||||
>
|
||||
{text}
|
||||
{(() => {
|
||||
let wordCounter = 0;
|
||||
return lines.map((line, lineIdx) => {
|
||||
const tokens = line.split(/(\s+)/).filter((w) => w.length > 0);
|
||||
return (
|
||||
<div key={lineIdx} style={{ display: "block" }}>
|
||||
{tokens.map((w, ti) => {
|
||||
if (/^\s+$/.test(w)) {
|
||||
return <span key={ti}> </span>;
|
||||
}
|
||||
const startFrame = wordCounter * staggerFrames;
|
||||
wordCounter += 1;
|
||||
const wordOpacity = interpolate(
|
||||
frame,
|
||||
[startFrame, startFrame + wordFadeFrames],
|
||||
[0, 1],
|
||||
{ extrapolateLeft: "clamp", extrapolateRight: "clamp" },
|
||||
);
|
||||
const blur = interpolate(
|
||||
frame,
|
||||
[startFrame, startFrame + wordFadeFrames],
|
||||
[6, 0],
|
||||
{ extrapolateLeft: "clamp", extrapolateRight: "clamp" },
|
||||
);
|
||||
const ty = interpolate(
|
||||
frame,
|
||||
[startFrame, startFrame + wordFadeFrames],
|
||||
[14, 0],
|
||||
{ extrapolateLeft: "clamp", extrapolateRight: "clamp" },
|
||||
);
|
||||
return (
|
||||
<span
|
||||
key={ti}
|
||||
style={{
|
||||
display: "inline-block",
|
||||
opacity: wordOpacity,
|
||||
filter: `blur(${blur}px)`,
|
||||
transform: `translateY(${ty}px)`,
|
||||
}}
|
||||
>
|
||||
{w}
|
||||
</span>
|
||||
);
|
||||
})}
|
||||
</div>
|
||||
);
|
||||
});
|
||||
})()}
|
||||
</div>
|
||||
|
||||
{/* Subtle accent dot centered under text */}
|
||||
<div
|
||||
style={{
|
||||
position: "absolute",
|
||||
width: 6,
|
||||
height: 6,
|
||||
borderRadius: "50%",
|
||||
background: accent,
|
||||
boxShadow: `0 0 18px ${accent}`,
|
||||
opacity: 0.55 * container * lineExit,
|
||||
transform: "translateY(172px)",
|
||||
}}
|
||||
/>
|
||||
</AbsoluteFill>
|
||||
);
|
||||
};
|
||||
@@ -384,6 +513,10 @@ export const CinematicRenderer: React.FC<CinematicRendererProps> = ({
|
||||
titleFontSize={titleFontSize}
|
||||
titleWidth={titleWidth}
|
||||
signalLineCount={signalLineCount}
|
||||
backgroundSrc={scene.backgroundSrc}
|
||||
backgroundTrimBeforeSeconds={scene.backgroundTrimBeforeSeconds}
|
||||
backgroundTrimAfterSeconds={scene.backgroundTrimAfterSeconds}
|
||||
variant={scene.variant}
|
||||
/>
|
||||
)}
|
||||
</Sequence>
|
||||
|
||||
@@ -0,0 +1,608 @@
|
||||
import {
|
||||
AbsoluteFill,
|
||||
Img,
|
||||
OffthreadVideo,
|
||||
Sequence,
|
||||
interpolate,
|
||||
random,
|
||||
spring,
|
||||
staticFile,
|
||||
useCurrentFrame,
|
||||
useVideoConfig,
|
||||
} from "remotion";
|
||||
import React from "react";
|
||||
import { loadFont as loadPlayfair } from "@remotion/google-fonts/PlayfairDisplay";
|
||||
|
||||
const { fontFamily: playfairFamily } = loadPlayfair("normal", {
|
||||
weights: ["400", "700"],
|
||||
subsets: ["latin"],
|
||||
});
|
||||
const { fontFamily: playfairItalic } = loadPlayfair("italic", {
|
||||
weights: ["400", "700"],
|
||||
subsets: ["latin"],
|
||||
});
|
||||
|
||||
function resolveAsset(src: string): string {
|
||||
if (src.startsWith("http://") || src.startsWith("https://") || src.startsWith("data:")) return src;
|
||||
const clean = src.replace(/^file:\/\/\/?/, "");
|
||||
if (clean.startsWith("/") || /^[A-Za-z]:[\\/]/.test(clean)) {
|
||||
return `file:///${clean.replace(/\\/g, "/")}`;
|
||||
}
|
||||
return staticFile(clean);
|
||||
}
|
||||
|
||||
export type CollageTransition =
|
||||
| "pop"
|
||||
| "slide-zoom"
|
||||
| "spin"
|
||||
| "shutter"
|
||||
| "glitch"
|
||||
| "swoop";
|
||||
|
||||
export interface CollageClip {
|
||||
src: string;
|
||||
kind: "image" | "video";
|
||||
inSeconds: number;
|
||||
outSeconds: number;
|
||||
x: number; // 0..1 center
|
||||
y: number; // 0..1 center
|
||||
widthPct: number; // 0..1 of frame width
|
||||
aspect?: number; // width/height, default 3/4 (portrait card)
|
||||
rotation: number; // degrees
|
||||
sourceInSeconds?: number;
|
||||
transition?: CollageTransition;
|
||||
hero?: boolean; // adds extra polish + flash boost
|
||||
seed?: number;
|
||||
}
|
||||
|
||||
export interface CollageBurstProps {
|
||||
backgroundSrc: string;
|
||||
backgroundInSeconds?: number;
|
||||
curtainStartSeconds: number;
|
||||
curtainEndSeconds: number;
|
||||
clips: CollageClip[];
|
||||
}
|
||||
|
||||
// ----------------------------------------------------------------------------
|
||||
// Opening text — elegant serif card that lives in the pre-reveal black, then
|
||||
// fades out as the curtain opens.
|
||||
// ----------------------------------------------------------------------------
|
||||
const OpeningText: React.FC<{
|
||||
lineOne: string;
|
||||
lineTwo: string;
|
||||
fadeInStart: number;
|
||||
fadeInEnd: number;
|
||||
fadeOutStart: number;
|
||||
fadeOutEnd: number;
|
||||
}> = ({ lineOne, lineTwo, fadeInStart, fadeInEnd, fadeOutStart, fadeOutEnd }) => {
|
||||
const frame = useCurrentFrame();
|
||||
if (frame < fadeInStart || frame > fadeOutEnd) return null;
|
||||
|
||||
const fadeIn = interpolate(frame, [fadeInStart, fadeInEnd], [0, 1], {
|
||||
extrapolateLeft: "clamp",
|
||||
extrapolateRight: "clamp",
|
||||
});
|
||||
const fadeOut = interpolate(frame, [fadeOutStart, fadeOutEnd], [1, 0], {
|
||||
extrapolateLeft: "clamp",
|
||||
extrapolateRight: "clamp",
|
||||
});
|
||||
const opacity = fadeIn * fadeOut;
|
||||
|
||||
// Slight rise on entry, slight drift on exit
|
||||
const yIn = interpolate(fadeIn, [0, 1], [18, 0]);
|
||||
const yOut = interpolate(fadeOut, [0, 1], [-10, 0]);
|
||||
const y = yIn + yOut;
|
||||
|
||||
// Line draws in from center outward
|
||||
const lineProgress = interpolate(frame, [fadeInStart, fadeInEnd + 8], [0, 1], {
|
||||
extrapolateLeft: "clamp",
|
||||
extrapolateRight: "clamp",
|
||||
});
|
||||
|
||||
const cream = "#F5E7C5";
|
||||
const gold = "rgba(255, 214, 150, 0.9)";
|
||||
|
||||
return (
|
||||
<AbsoluteFill
|
||||
style={{
|
||||
pointerEvents: "none",
|
||||
display: "flex",
|
||||
flexDirection: "column",
|
||||
alignItems: "center",
|
||||
justifyContent: "center",
|
||||
paddingTop: "8%",
|
||||
opacity,
|
||||
}}
|
||||
>
|
||||
<div
|
||||
style={{
|
||||
transform: `translateY(${y}px)`,
|
||||
textAlign: "center",
|
||||
filter: `drop-shadow(0 0 22px rgba(255, 200, 120, 0.3))`,
|
||||
}}
|
||||
>
|
||||
{/* Decorative rule */}
|
||||
<div
|
||||
style={{
|
||||
width: 160 * lineProgress,
|
||||
height: 1.5,
|
||||
background: `linear-gradient(90deg, rgba(245,231,197,0) 0%, ${gold} 50%, rgba(245,231,197,0) 100%)`,
|
||||
margin: "0 auto 28px",
|
||||
}}
|
||||
/>
|
||||
<div
|
||||
style={{
|
||||
fontFamily: playfairFamily,
|
||||
fontWeight: 400,
|
||||
fontSize: 46,
|
||||
color: cream,
|
||||
letterSpacing: "0.01em",
|
||||
lineHeight: 1.35,
|
||||
textShadow: "0 2px 18px rgba(0,0,0,0.7)",
|
||||
}}
|
||||
>
|
||||
{lineOne}
|
||||
</div>
|
||||
<div
|
||||
style={{
|
||||
fontFamily: playfairItalic,
|
||||
fontStyle: "italic",
|
||||
fontWeight: 400,
|
||||
fontSize: 78,
|
||||
color: cream,
|
||||
letterSpacing: "0.005em",
|
||||
lineHeight: 1.15,
|
||||
marginTop: 14,
|
||||
textShadow: "0 2px 22px rgba(0,0,0,0.75)",
|
||||
}}
|
||||
>
|
||||
{lineTwo}
|
||||
</div>
|
||||
{/* Small jewel divider */}
|
||||
<div
|
||||
style={{
|
||||
marginTop: 30,
|
||||
display: "flex",
|
||||
justifyContent: "center",
|
||||
alignItems: "center",
|
||||
gap: 14,
|
||||
opacity: lineProgress,
|
||||
}}
|
||||
>
|
||||
<div
|
||||
style={{
|
||||
width: 60,
|
||||
height: 1,
|
||||
background: `linear-gradient(90deg, rgba(245,231,197,0) 0%, ${gold} 100%)`,
|
||||
}}
|
||||
/>
|
||||
<div
|
||||
style={{
|
||||
width: 6,
|
||||
height: 6,
|
||||
borderRadius: 999,
|
||||
background: gold,
|
||||
boxShadow: `0 0 14px ${gold}`,
|
||||
}}
|
||||
/>
|
||||
<div
|
||||
style={{
|
||||
width: 60,
|
||||
height: 1,
|
||||
background: `linear-gradient(90deg, ${gold} 0%, rgba(245,231,197,0) 100%)`,
|
||||
}}
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
</AbsoluteFill>
|
||||
);
|
||||
};
|
||||
|
||||
// ----------------------------------------------------------------------------
|
||||
// White flash — brief burst when a card lands. Rendered at composition level.
|
||||
// ----------------------------------------------------------------------------
|
||||
const CardFlash: React.FC<{ atFrame: number; strength?: number }> = ({
|
||||
atFrame,
|
||||
strength = 0.4,
|
||||
}) => {
|
||||
const frame = useCurrentFrame();
|
||||
if (frame < atFrame - 1 || frame > atFrame + 6) return null;
|
||||
const t = (frame - atFrame) / 6;
|
||||
const opacity = Math.max(0, (1 - t) * strength);
|
||||
return (
|
||||
<AbsoluteFill
|
||||
style={{
|
||||
backgroundColor: "white",
|
||||
opacity,
|
||||
pointerEvents: "none",
|
||||
mixBlendMode: "screen",
|
||||
}}
|
||||
/>
|
||||
);
|
||||
};
|
||||
|
||||
// ----------------------------------------------------------------------------
|
||||
// CollageCard — picks an entry transition by `clip.transition` and plays video
|
||||
// starting at its in-time via <Sequence>.
|
||||
// ----------------------------------------------------------------------------
|
||||
const CollageCard: React.FC<{ clip: CollageClip; frameW: number; frameH: number }> = ({
|
||||
clip,
|
||||
frameW,
|
||||
frameH,
|
||||
}) => {
|
||||
const frame = useCurrentFrame();
|
||||
const { fps } = useVideoConfig();
|
||||
const inFrame = clip.inSeconds * fps;
|
||||
const outFrame = clip.outSeconds * fps;
|
||||
if (frame < inFrame - 2 || frame > outFrame + 8) return null;
|
||||
|
||||
const rel = frame - inFrame;
|
||||
const relOut = frame - outFrame;
|
||||
const seed = clip.seed ?? 0;
|
||||
const transition = clip.transition ?? "pop";
|
||||
|
||||
// Common spring value 0→1 over ~18f for entrance
|
||||
const entry = spring({
|
||||
frame: rel,
|
||||
fps,
|
||||
config: { damping: 10, stiffness: 130, mass: 0.85 },
|
||||
durationInFrames: 20,
|
||||
});
|
||||
|
||||
const exit = spring({
|
||||
frame: relOut,
|
||||
fps,
|
||||
config: { damping: 14, stiffness: 200, mass: 0.8 },
|
||||
durationInFrames: 12,
|
||||
});
|
||||
|
||||
const isExiting = frame >= outFrame;
|
||||
|
||||
// Base transform values derived from transition
|
||||
let scale = 1;
|
||||
let rot = clip.rotation;
|
||||
let offX = 0;
|
||||
let offY = 0;
|
||||
let opacity = 1;
|
||||
let clipPathCss: string | undefined;
|
||||
let blur = 0;
|
||||
|
||||
if (!isExiting) {
|
||||
const e = entry;
|
||||
const inv = 1 - e;
|
||||
switch (transition) {
|
||||
case "pop":
|
||||
scale = e;
|
||||
opacity = e;
|
||||
break;
|
||||
case "slide-zoom": {
|
||||
const dir = (seed % 4); // 0=left, 1=right, 2=top, 3=bottom
|
||||
const slide = frameW * 0.6 * inv;
|
||||
if (dir === 0) offX = -slide;
|
||||
else if (dir === 1) offX = slide;
|
||||
else if (dir === 2) offY = -slide;
|
||||
else offY = slide;
|
||||
scale = 0.7 + 0.3 * e;
|
||||
opacity = e;
|
||||
blur = inv * 8;
|
||||
break;
|
||||
}
|
||||
case "spin": {
|
||||
scale = e;
|
||||
rot = clip.rotation + (1 - e) * (seed % 2 === 0 ? 540 : -540);
|
||||
opacity = e;
|
||||
break;
|
||||
}
|
||||
case "shutter": {
|
||||
scale = 0.92 + 0.08 * e;
|
||||
opacity = e > 0.1 ? 1 : 0;
|
||||
const pct = Math.round(inv * 50);
|
||||
clipPathCss = `inset(${pct}% 0 ${pct}% 0 round 14px)`;
|
||||
break;
|
||||
}
|
||||
case "glitch": {
|
||||
scale = 0.85 + 0.15 * e;
|
||||
opacity = e;
|
||||
// High-frequency jitter first ~8f then settles
|
||||
const jitterAmt = Math.max(0, 1 - rel / 8);
|
||||
offX = (random(`gx${seed}-${Math.floor(rel / 1)}`) - 0.5) * 40 * jitterAmt;
|
||||
offY = (random(`gy${seed}-${Math.floor(rel / 1)}`) - 0.5) * 40 * jitterAmt;
|
||||
rot = clip.rotation + (random(`gr${seed}-${Math.floor(rel / 2)}`) - 0.5) * 16 * jitterAmt;
|
||||
break;
|
||||
}
|
||||
case "swoop": {
|
||||
// arc in from a corner
|
||||
const fromX = (seed % 2 === 0 ? -1 : 1) * frameW * 0.55;
|
||||
const fromY = -frameH * 0.3;
|
||||
offX = fromX * inv;
|
||||
offY = fromY * inv;
|
||||
scale = 0.6 + 0.4 * e;
|
||||
rot = clip.rotation + inv * (seed % 2 === 0 ? -35 : 35);
|
||||
opacity = e;
|
||||
break;
|
||||
}
|
||||
}
|
||||
} else {
|
||||
// Exit: quick shrink + fade
|
||||
const o = exit;
|
||||
scale = 1 - 0.7 * o;
|
||||
opacity = 1 - o;
|
||||
rot = clip.rotation + o * (seed % 2 === 0 ? -8 : 8);
|
||||
}
|
||||
|
||||
// Gentle idle breathing while held
|
||||
const idle = Math.sin((rel - 20) / 14 + seed) * 0.012;
|
||||
const held = !isExiting && rel > 20;
|
||||
const idleScale = held ? 1 + idle : 1;
|
||||
|
||||
const aspect = clip.aspect ?? 3 / 4; // default portrait
|
||||
const w = frameW * clip.widthPct;
|
||||
const h = w / aspect;
|
||||
const cx = clip.x * frameW;
|
||||
const cy = clip.y * frameH;
|
||||
|
||||
// Source offset for videos — Sequence starts at inFrame so OffthreadVideo
|
||||
// naturally begins playback on card entry.
|
||||
const startFromFrames = Math.round((clip.sourceInSeconds ?? 0) * fps);
|
||||
|
||||
const borderWidth = clip.hero ? 10 : 8;
|
||||
const glowColor = clip.hero ? "rgba(255, 210, 140, 0.5)" : "rgba(255,255,255,0.05)";
|
||||
|
||||
const cardContent =
|
||||
clip.kind === "image" ? (
|
||||
<Img
|
||||
src={resolveAsset(clip.src)}
|
||||
style={{
|
||||
width: "100%",
|
||||
height: "100%",
|
||||
objectFit: "cover",
|
||||
display: "block",
|
||||
}}
|
||||
/>
|
||||
) : (
|
||||
<Sequence from={inFrame} layout="none">
|
||||
<OffthreadVideo
|
||||
src={resolveAsset(clip.src)}
|
||||
startFrom={startFromFrames}
|
||||
muted
|
||||
playbackRate={1}
|
||||
style={{
|
||||
width: "100%",
|
||||
height: "100%",
|
||||
objectFit: "cover",
|
||||
display: "block",
|
||||
}}
|
||||
/>
|
||||
</Sequence>
|
||||
);
|
||||
|
||||
return (
|
||||
<div
|
||||
style={{
|
||||
position: "absolute",
|
||||
left: cx - w / 2,
|
||||
top: cy - h / 2,
|
||||
width: w,
|
||||
height: h,
|
||||
transform: `translate(${offX}px, ${offY}px) rotate(${rot}deg) scale(${scale * idleScale})`,
|
||||
transformOrigin: "center center",
|
||||
opacity,
|
||||
filter: blur > 0 ? `blur(${blur}px)` : undefined,
|
||||
willChange: "transform",
|
||||
}}
|
||||
>
|
||||
<div
|
||||
style={{
|
||||
position: "absolute",
|
||||
inset: 0,
|
||||
borderRadius: 16,
|
||||
overflow: "hidden",
|
||||
boxShadow: clip.hero
|
||||
? "0 40px 100px rgba(0,0,0,0.7), 0 0 90px rgba(255,200,120,0.55), 0 8px 18px rgba(0,0,0,0.55)"
|
||||
: "0 24px 60px rgba(0,0,0,0.6), 0 8px 18px rgba(0,0,0,0.4)",
|
||||
border: `${borderWidth}px solid #FAFAF5`,
|
||||
backgroundColor: "#FAFAF5",
|
||||
clipPath: clipPathCss,
|
||||
}}
|
||||
>
|
||||
{cardContent}
|
||||
{/* Subtle inner vignette on hero cards for depth */}
|
||||
{clip.hero && (
|
||||
<AbsoluteFill
|
||||
style={{
|
||||
pointerEvents: "none",
|
||||
background:
|
||||
"radial-gradient(ellipse at center, rgba(0,0,0,0) 55%, rgba(0,0,0,0.4) 100%)",
|
||||
}}
|
||||
/>
|
||||
)}
|
||||
{/* Warm film edge glow for hero */}
|
||||
{clip.hero && (
|
||||
<AbsoluteFill
|
||||
style={{
|
||||
pointerEvents: "none",
|
||||
boxShadow: `inset 0 0 60px ${glowColor}`,
|
||||
}}
|
||||
/>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
};
|
||||
|
||||
export const CollageBurst: React.FC<CollageBurstProps> = ({
|
||||
backgroundSrc,
|
||||
backgroundInSeconds = 0,
|
||||
curtainStartSeconds,
|
||||
curtainEndSeconds,
|
||||
clips,
|
||||
}) => {
|
||||
const frame = useCurrentFrame();
|
||||
const { fps, width, height } = useVideoConfig();
|
||||
|
||||
const curtainStart = curtainStartSeconds * fps;
|
||||
const curtainEnd = curtainEndSeconds * fps;
|
||||
|
||||
const t = interpolate(frame, [curtainStart, curtainEnd], [0, 1], {
|
||||
extrapolateLeft: "clamp",
|
||||
extrapolateRight: "clamp",
|
||||
});
|
||||
const ease = t < 0.5 ? 4 * t * t * t : 1 - Math.pow(-2 * t + 2, 3) / 2;
|
||||
const panelShift = ease * (width / 2 + 40);
|
||||
|
||||
const seamOpacity = frame < curtainStart ? 0.65 + 0.35 * Math.sin(frame / 3) : 0;
|
||||
const seamGlow = frame < curtainStart ? 0.6 + 0.4 * Math.sin(frame / 4) : 0;
|
||||
|
||||
const bgStartFrame = Math.max(0, curtainStart - 6);
|
||||
const bgVisible = frame >= bgStartFrame;
|
||||
|
||||
// Slow cinematic zoom
|
||||
const bgZoom = interpolate(frame, [curtainStart, curtainStart + 900], [1.05, 1.18], {
|
||||
extrapolateLeft: "clamp",
|
||||
extrapolateRight: "clamp",
|
||||
});
|
||||
|
||||
// Screen shake intensity — spikes briefly around each clip entry
|
||||
let shakeX = 0;
|
||||
let shakeY = 0;
|
||||
for (const c of clips) {
|
||||
const entryFrame = c.inSeconds * fps;
|
||||
const dist = frame - entryFrame;
|
||||
if (dist >= 0 && dist < 6) {
|
||||
const falloff = 1 - dist / 6;
|
||||
const amp = (c.hero ? 12 : 5) * falloff;
|
||||
shakeX += (random(`sx-${entryFrame}-${dist}`) - 0.5) * amp;
|
||||
shakeY += (random(`sy-${entryFrame}-${dist}`) - 0.5) * amp;
|
||||
}
|
||||
}
|
||||
|
||||
return (
|
||||
<AbsoluteFill style={{ backgroundColor: "#0a0a0a" }}>
|
||||
{/* ---------- Hazy muted background ---------- */}
|
||||
{bgVisible && (
|
||||
<AbsoluteFill
|
||||
style={{
|
||||
transform: `scale(${bgZoom})`,
|
||||
filter: "saturate(0.35) brightness(0.55) contrast(0.95) blur(6px)",
|
||||
opacity: 0.55,
|
||||
}}
|
||||
>
|
||||
<OffthreadVideo
|
||||
src={resolveAsset(backgroundSrc)}
|
||||
startFrom={Math.round(backgroundInSeconds * fps)}
|
||||
muted
|
||||
style={{
|
||||
width: "100%",
|
||||
height: "100%",
|
||||
objectFit: "cover",
|
||||
display: "block",
|
||||
}}
|
||||
/>
|
||||
</AbsoluteFill>
|
||||
)}
|
||||
|
||||
{/* Dirty-gold / grey atmospheric wash layered over bg */}
|
||||
{bgVisible && (
|
||||
<AbsoluteFill
|
||||
style={{
|
||||
pointerEvents: "none",
|
||||
background:
|
||||
"radial-gradient(ellipse at 50% 40%, rgba(180,150,110,0.18) 0%, rgba(30,28,24,0.75) 70%, rgba(5,5,5,0.95) 100%)",
|
||||
mixBlendMode: "multiply",
|
||||
}}
|
||||
/>
|
||||
)}
|
||||
|
||||
{/* Grainy noise via CSS — subtle film texture */}
|
||||
{bgVisible && (
|
||||
<AbsoluteFill
|
||||
style={{
|
||||
pointerEvents: "none",
|
||||
opacity: 0.22,
|
||||
background:
|
||||
"repeating-conic-gradient(rgba(255,255,255,0.04) 0deg 1deg, rgba(0,0,0,0) 1deg 2deg)",
|
||||
mixBlendMode: "overlay",
|
||||
}}
|
||||
/>
|
||||
)}
|
||||
|
||||
{/* ---------- Curtain ---------- */}
|
||||
<AbsoluteFill>
|
||||
<div
|
||||
style={{
|
||||
position: "absolute",
|
||||
top: 0,
|
||||
left: 0,
|
||||
width: "50%",
|
||||
height: "100%",
|
||||
backgroundColor: "#050505",
|
||||
transform: `translateX(${-panelShift}px)`,
|
||||
boxShadow: "inset -30px 0 60px rgba(0,0,0,0.9)",
|
||||
}}
|
||||
/>
|
||||
<div
|
||||
style={{
|
||||
position: "absolute",
|
||||
top: 0,
|
||||
right: 0,
|
||||
width: "50%",
|
||||
height: "100%",
|
||||
backgroundColor: "#050505",
|
||||
transform: `translateX(${panelShift}px)`,
|
||||
boxShadow: "inset 30px 0 60px rgba(0,0,0,0.9)",
|
||||
}}
|
||||
/>
|
||||
<div
|
||||
style={{
|
||||
position: "absolute",
|
||||
top: 0,
|
||||
left: "50%",
|
||||
width: 8,
|
||||
height: "100%",
|
||||
transform: "translateX(-50%)",
|
||||
background:
|
||||
"linear-gradient(180deg, rgba(255,220,160,0) 0%, rgba(255,220,160,1) 50%, rgba(255,220,160,0) 100%)",
|
||||
opacity: Math.max(0, seamOpacity - ease * 1.2),
|
||||
filter: `blur(${2 + 6 * seamGlow}px)`,
|
||||
boxShadow: `0 0 ${50 + 80 * seamGlow}px rgba(255, 200, 120, 0.95)`,
|
||||
}}
|
||||
/>
|
||||
</AbsoluteFill>
|
||||
|
||||
{/* ---------- Opening text (pre-reveal, on black) ---------- */}
|
||||
<OpeningText
|
||||
lineOne="When you realize your daughters"
|
||||
lineTwo="deserve these lines"
|
||||
fadeInStart={Math.round(0.3 * fps)}
|
||||
fadeInEnd={Math.round(0.9 * fps)}
|
||||
fadeOutStart={Math.round(1.8 * fps)}
|
||||
fadeOutEnd={Math.round(2.6 * fps)}
|
||||
/>
|
||||
|
||||
{/* ---------- Foreground shake wrapper ---------- */}
|
||||
<AbsoluteFill style={{ transform: `translate(${shakeX}px, ${shakeY}px)` }}>
|
||||
{clips.map((clip, i) => (
|
||||
<CollageCard key={i} clip={clip} frameW={width} frameH={height} />
|
||||
))}
|
||||
</AbsoluteFill>
|
||||
|
||||
{/* ---------- Flash bursts layered on each clip entry ---------- */}
|
||||
{clips.map((c, i) => (
|
||||
<CardFlash
|
||||
key={`f${i}`}
|
||||
atFrame={c.inSeconds * fps}
|
||||
strength={c.hero ? 0.55 : 0.22}
|
||||
/>
|
||||
))}
|
||||
|
||||
{/* Frame top/bottom vignette for focus */}
|
||||
<AbsoluteFill
|
||||
style={{
|
||||
background:
|
||||
"linear-gradient(180deg, rgba(0,0,0,0.4) 0%, rgba(0,0,0,0) 14%, rgba(0,0,0,0) 86%, rgba(0,0,0,0.5) 100%)",
|
||||
pointerEvents: "none",
|
||||
}}
|
||||
/>
|
||||
</AbsoluteFill>
|
||||
);
|
||||
};
|
||||
@@ -43,6 +43,8 @@ import { AnimeScene } from "./components/AnimeScene";
|
||||
import type { CameraMotion } from "./components/AnimeScene";
|
||||
import { TerminalScene } from "./components/TerminalScene";
|
||||
import type { TerminalStep } from "./components/TerminalScene";
|
||||
import { ScreenshotScene } from "./components/ScreenshotScene";
|
||||
import type { ScreenshotStep } from "./components/ScreenshotScene";
|
||||
import { ProviderChip } from "./components/ProviderChip";
|
||||
import type { ParticleType } from "./components/ParticleOverlay";
|
||||
import { resolveTheme, type ThemeConfig, DEFAULT_THEME } from "./Root";
|
||||
@@ -262,6 +264,10 @@ interface Cut {
|
||||
steps?: TerminalStep[];
|
||||
terminalTitle?: string;
|
||||
prompt?: string;
|
||||
// Screenshot scene props (type: "screenshot_scene")
|
||||
screenshotSteps?: ScreenshotStep[];
|
||||
screenshotSize?: { width: number; height: number };
|
||||
cursorStartAt?: [number, number];
|
||||
}
|
||||
|
||||
interface Overlay {
|
||||
@@ -602,6 +608,17 @@ const SceneRenderer: React.FC<{ cut: Cut; theme: ThemeConfig }> = ({ cut, theme
|
||||
/>
|
||||
);
|
||||
}
|
||||
if (cut.type === "screenshot_scene" && cut.backgroundImage && cut.screenshotSteps) {
|
||||
return (
|
||||
<ScreenshotScene
|
||||
backgroundImage={cut.backgroundImage}
|
||||
backgroundSize={cut.screenshotSize}
|
||||
steps={cut.screenshotSteps as ScreenshotStep[]}
|
||||
accentColor={accent}
|
||||
cursorStartAt={cut.cursorStartAt}
|
||||
/>
|
||||
);
|
||||
}
|
||||
|
||||
// --- Chart types — use theme.chartColors as default palette ---
|
||||
if (cut.type === "bar_chart" && cut.chartData) {
|
||||
|
||||
@@ -0,0 +1,173 @@
|
||||
import {
|
||||
AbsoluteFill,
|
||||
Audio,
|
||||
OffthreadVideo,
|
||||
interpolate,
|
||||
staticFile,
|
||||
useCurrentFrame,
|
||||
useVideoConfig,
|
||||
} from "remotion";
|
||||
import React from "react";
|
||||
import { loadFont as loadPlayfair } from "@remotion/google-fonts/PlayfairDisplay";
|
||||
|
||||
const { fontFamily: playfairItalic } = loadPlayfair("italic", {
|
||||
weights: ["400", "700"],
|
||||
subsets: ["latin"],
|
||||
});
|
||||
|
||||
function resolveAsset(src: string): string {
|
||||
if (src.startsWith("http://") || src.startsWith("https://") || src.startsWith("data:")) return src;
|
||||
const clean = src.replace(/^file:\/\/\/?/, "");
|
||||
if (clean.startsWith("/") || /^[A-Za-z]:[\\/]/.test(clean)) {
|
||||
return `file:///${clean.replace(/\\/g, "/")}`;
|
||||
}
|
||||
return staticFile(clean);
|
||||
}
|
||||
|
||||
export interface Lyric {
|
||||
text: string;
|
||||
inSeconds: number;
|
||||
outSeconds: number;
|
||||
}
|
||||
|
||||
export interface LyricOverlayProps {
|
||||
videoSrc: string;
|
||||
lyrics: Lyric[];
|
||||
bottomY?: number; // 0..1, vertical center of subtitle band
|
||||
}
|
||||
|
||||
const LyricLine: React.FC<{ lyric: Lyric; bottomY: number }> = ({ lyric, bottomY }) => {
|
||||
const frame = useCurrentFrame();
|
||||
const { fps, width, height } = useVideoConfig();
|
||||
const inFrame = lyric.inSeconds * fps;
|
||||
const outFrame = lyric.outSeconds * fps;
|
||||
if (frame < inFrame - 1 || frame > outFrame + 8) return null;
|
||||
|
||||
const fadeInDur = 6;
|
||||
const fadeOutDur = 8;
|
||||
|
||||
const fadeIn = interpolate(frame, [inFrame, inFrame + fadeInDur], [0, 1], {
|
||||
extrapolateLeft: "clamp",
|
||||
extrapolateRight: "clamp",
|
||||
});
|
||||
const fadeOut = interpolate(frame, [outFrame, outFrame + fadeOutDur], [1, 0], {
|
||||
extrapolateLeft: "clamp",
|
||||
extrapolateRight: "clamp",
|
||||
});
|
||||
const opacity = fadeIn * fadeOut;
|
||||
|
||||
const yRise = interpolate(fadeIn, [0, 1], [10, 0]);
|
||||
|
||||
const cream = "#F5E7C5";
|
||||
const gold = "rgba(255, 214, 150, 0.85)";
|
||||
|
||||
// Line draws in from center outward beneath the text
|
||||
const lineProgress = interpolate(frame, [inFrame, inFrame + fadeInDur + 4], [0, 1], {
|
||||
extrapolateLeft: "clamp",
|
||||
extrapolateRight: "clamp",
|
||||
});
|
||||
|
||||
return (
|
||||
<AbsoluteFill
|
||||
style={{
|
||||
pointerEvents: "none",
|
||||
display: "flex",
|
||||
flexDirection: "column",
|
||||
alignItems: "center",
|
||||
justifyContent: "flex-end",
|
||||
paddingBottom: height * (1 - bottomY),
|
||||
opacity,
|
||||
}}
|
||||
>
|
||||
{/* Subtle dark backdrop behind text for readability */}
|
||||
<div
|
||||
style={{
|
||||
position: "absolute",
|
||||
bottom: height * (1 - bottomY) - 80,
|
||||
left: 0,
|
||||
right: 0,
|
||||
height: 240,
|
||||
background:
|
||||
"linear-gradient(180deg, rgba(0,0,0,0) 0%, rgba(0,0,0,0.55) 55%, rgba(0,0,0,0.75) 100%)",
|
||||
opacity,
|
||||
}}
|
||||
/>
|
||||
<div
|
||||
style={{
|
||||
transform: `translateY(${yRise}px)`,
|
||||
textAlign: "center",
|
||||
padding: "0 60px",
|
||||
filter: "drop-shadow(0 0 18px rgba(255, 200, 120, 0.22))",
|
||||
position: "relative",
|
||||
zIndex: 2,
|
||||
}}
|
||||
>
|
||||
<div
|
||||
style={{
|
||||
fontFamily: playfairItalic,
|
||||
fontStyle: "italic",
|
||||
fontWeight: 400,
|
||||
fontSize: 54,
|
||||
lineHeight: 1.15,
|
||||
color: cream,
|
||||
letterSpacing: "0.01em",
|
||||
textShadow: "0 2px 18px rgba(0,0,0,0.85), 0 0 22px rgba(0,0,0,0.5)",
|
||||
}}
|
||||
>
|
||||
{lyric.text}
|
||||
</div>
|
||||
{/* Gold underline */}
|
||||
<div
|
||||
style={{
|
||||
marginTop: 18,
|
||||
display: "flex",
|
||||
justifyContent: "center",
|
||||
alignItems: "center",
|
||||
gap: 10,
|
||||
}}
|
||||
>
|
||||
<div
|
||||
style={{
|
||||
width: 90 * lineProgress,
|
||||
height: 1.2,
|
||||
background: `linear-gradient(90deg, rgba(245,231,197,0) 0%, ${gold} 100%)`,
|
||||
}}
|
||||
/>
|
||||
<div
|
||||
style={{
|
||||
width: 5,
|
||||
height: 5,
|
||||
borderRadius: 999,
|
||||
background: gold,
|
||||
opacity: lineProgress,
|
||||
boxShadow: `0 0 10px ${gold}`,
|
||||
}}
|
||||
/>
|
||||
<div
|
||||
style={{
|
||||
width: 90 * lineProgress,
|
||||
height: 1.2,
|
||||
background: `linear-gradient(90deg, ${gold} 0%, rgba(245,231,197,0) 100%)`,
|
||||
}}
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
</AbsoluteFill>
|
||||
);
|
||||
};
|
||||
|
||||
export const LyricOverlay: React.FC<LyricOverlayProps> = ({
|
||||
videoSrc,
|
||||
lyrics,
|
||||
bottomY = 0.88,
|
||||
}) => {
|
||||
const { durationInFrames } = useVideoConfig();
|
||||
return (
|
||||
<AbsoluteFill style={{ backgroundColor: "#000" }}>
|
||||
<OffthreadVideo src={resolveAsset(videoSrc)} />
|
||||
{lyrics.map((l, i) => (
|
||||
<LyricLine key={i} lyric={l} bottomY={bottomY} />
|
||||
))}
|
||||
</AbsoluteFill>
|
||||
);
|
||||
};
|
||||
@@ -14,6 +14,8 @@ import { EndTag, EndTagProps } from "./components/EndTag";
|
||||
import { HeroTitle } from "./components/HeroTitle";
|
||||
import { ProductReveal, ProductRevealProps } from "./components/ProductReveal";
|
||||
import { CaptionOverlay, WordCaption } from "./components/CaptionOverlay";
|
||||
import { CollageBurst, CollageBurstProps } from "./CollageBurst";
|
||||
import { LyricOverlay, LyricOverlayProps } from "./LyricOverlay";
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Theme System — prevents every video from looking like dark fintech
|
||||
@@ -266,6 +268,34 @@ export const Root: React.FC = () => {
|
||||
backgroundColor: "rgba(15, 23, 42, 0.75)",
|
||||
}}
|
||||
/>
|
||||
<Composition
|
||||
id="CollageBurst"
|
||||
component={CollageBurst}
|
||||
durationInFrames={30 * 30}
|
||||
fps={30}
|
||||
width={1080}
|
||||
height={1920}
|
||||
defaultProps={{
|
||||
backgroundSrc: "",
|
||||
backgroundInSeconds: 0,
|
||||
curtainStartSeconds: 1.5,
|
||||
curtainEndSeconds: 3.0,
|
||||
clips: [],
|
||||
} as CollageBurstProps}
|
||||
/>
|
||||
<Composition
|
||||
id="LyricOverlay"
|
||||
component={LyricOverlay}
|
||||
durationInFrames={30 * 28}
|
||||
fps={30}
|
||||
width={1080}
|
||||
height={1920}
|
||||
defaultProps={{
|
||||
videoSrc: "",
|
||||
lyrics: [],
|
||||
bottomY: 0.88,
|
||||
} as LyricOverlayProps}
|
||||
/>
|
||||
<Composition
|
||||
id="EndTag"
|
||||
component={EndTag}
|
||||
|
||||
@@ -22,6 +22,10 @@ export interface CinematicTitleScene extends CinematicBaseScene {
|
||||
text: string;
|
||||
accent?: string;
|
||||
intensity?: number;
|
||||
backgroundSrc?: string;
|
||||
backgroundTrimBeforeSeconds?: number;
|
||||
backgroundTrimAfterSeconds?: number;
|
||||
variant?: "plate" | "overlay";
|
||||
}
|
||||
|
||||
export type CinematicScene = CinematicVideoScene | CinematicTitleScene;
|
||||
|
||||
@@ -0,0 +1,668 @@
|
||||
import {
|
||||
AbsoluteFill,
|
||||
Img,
|
||||
interpolate,
|
||||
spring,
|
||||
staticFile,
|
||||
useCurrentFrame,
|
||||
useVideoConfig,
|
||||
} from "remotion";
|
||||
|
||||
/**
|
||||
* ScreenshotScene — approach-1 synthetic UI demo.
|
||||
*
|
||||
* Takes any screenshot as a frozen backdrop and animates scripted overlays
|
||||
* (cursor, click pulses, typing, chat bubbles, highlight rings, callouts)
|
||||
* on top at normalized coordinates. Viewer-indistinguishable from a real
|
||||
* screen recording for ~15-30s focused demos.
|
||||
*
|
||||
* Coordinate system: everything is 0-1 normalized against the rendered
|
||||
* backdrop rectangle (not the raw canvas), so overlays track the image
|
||||
* correctly regardless of letterboxing.
|
||||
*
|
||||
* See .agents/skills/synthetic-ui-recording/SKILL.md for authoring guidance.
|
||||
*/
|
||||
|
||||
// ---------- Types ----------
|
||||
|
||||
export type Region = { x: number; y: number; w: number; h: number }; // all 0-1
|
||||
export type Point = [number, number]; // [x, y], 0-1 normalized
|
||||
|
||||
export type ScreenshotStep =
|
||||
| { kind: "cursor_move"; to: Point; durationSeconds?: number }
|
||||
| { kind: "click_pulse"; at?: Point; durationSeconds?: number; color?: string }
|
||||
| {
|
||||
kind: "type_into";
|
||||
region: Region;
|
||||
text: string;
|
||||
typeSpeed?: number; // seconds per char
|
||||
fontSize?: number; // in normalized-height units; default 0.022
|
||||
color?: string;
|
||||
}
|
||||
| {
|
||||
kind: "bubble_append";
|
||||
region: Region; // where the bubble lands (its bounding box)
|
||||
text: string;
|
||||
role?: "user" | "assistant";
|
||||
durationSeconds?: number;
|
||||
stream?: boolean; // if true, text reveals word-by-word over the duration
|
||||
fontSize?: number;
|
||||
}
|
||||
| {
|
||||
kind: "typing_dots";
|
||||
at: Point;
|
||||
durationSeconds?: number;
|
||||
color?: string;
|
||||
}
|
||||
| {
|
||||
kind: "highlight_box";
|
||||
region: Region;
|
||||
durationSeconds?: number;
|
||||
color?: string;
|
||||
pulses?: number;
|
||||
}
|
||||
| {
|
||||
kind: "callout_balloon";
|
||||
anchor: Point; // the element being pointed at
|
||||
text: string;
|
||||
position?: "top" | "bottom" | "left" | "right"; // where balloon sits relative to anchor
|
||||
durationSeconds?: number;
|
||||
color?: string;
|
||||
}
|
||||
| { kind: "pause"; seconds: number };
|
||||
|
||||
interface ScreenshotSceneProps {
|
||||
backgroundImage: string;
|
||||
/** Natural pixel size of the image — used to compute the contain-fit
|
||||
* rectangle so overlays land on correct pixels. Defaults to 16:9. */
|
||||
backgroundSize?: { width: number; height: number };
|
||||
steps: ScreenshotStep[];
|
||||
accentColor?: string;
|
||||
/** Starting cursor position. Default: top-right area. */
|
||||
cursorStartAt?: Point;
|
||||
}
|
||||
|
||||
// ---------- Helpers ----------
|
||||
|
||||
function resolveAsset(src: string): string {
|
||||
if (src.startsWith("http://") || src.startsWith("https://") || src.startsWith("data:")) {
|
||||
return src;
|
||||
}
|
||||
const clean = src.replace(/^file:\/\/\/?/, "");
|
||||
if (clean.startsWith("/") || /^[A-Za-z]:[\\/]/.test(clean)) {
|
||||
return `file:///${clean.replace(/\\/g, "/")}`;
|
||||
}
|
||||
return staticFile(clean);
|
||||
}
|
||||
|
||||
/** Compute the rendered bounding box of the backdrop inside a canvas,
|
||||
* using object-fit: contain semantics. Returns pixel offsets/sizes. */
|
||||
function containRect(
|
||||
imgW: number,
|
||||
imgH: number,
|
||||
cvW: number,
|
||||
cvH: number
|
||||
): { x: number; y: number; w: number; h: number } {
|
||||
const imgAspect = imgW / imgH;
|
||||
const cvAspect = cvW / cvH;
|
||||
if (imgAspect > cvAspect) {
|
||||
// image is wider → fit width, letterbox top/bottom
|
||||
const w = cvW;
|
||||
const h = cvW / imgAspect;
|
||||
return { x: 0, y: (cvH - h) / 2, w, h };
|
||||
} else {
|
||||
// image is taller → fit height, letterbox left/right
|
||||
const h = cvH;
|
||||
const w = cvH * imgAspect;
|
||||
return { x: (cvW - w) / 2, y: 0, w, h };
|
||||
}
|
||||
}
|
||||
|
||||
// ---------- Timing walk — assign frame windows to each step ----------
|
||||
|
||||
interface TimedStep {
|
||||
step: ScreenshotStep;
|
||||
startFrame: number;
|
||||
endFrame: number;
|
||||
/** cursor position at start of this step (inclusive) */
|
||||
cursorBefore: Point;
|
||||
/** cursor position at end of this step */
|
||||
cursorAfter: Point;
|
||||
}
|
||||
|
||||
function walkTimeline(
|
||||
steps: ScreenshotStep[],
|
||||
fps: number,
|
||||
cursorStart: Point
|
||||
): { timed: TimedStep[]; totalFrames: number } {
|
||||
const timed: TimedStep[] = [];
|
||||
let cursor = cursorStart;
|
||||
let frameCursor = 0;
|
||||
|
||||
for (const step of steps) {
|
||||
let duration = 0;
|
||||
const before = cursor;
|
||||
let after = cursor;
|
||||
let blocks = true; // whether step advances timeline cursor
|
||||
|
||||
switch (step.kind) {
|
||||
case "cursor_move":
|
||||
duration = (step.durationSeconds ?? 0.9) * fps;
|
||||
after = step.to;
|
||||
break;
|
||||
case "click_pulse":
|
||||
duration = (step.durationSeconds ?? 0.45) * fps;
|
||||
if (step.at) after = step.at;
|
||||
break;
|
||||
case "type_into": {
|
||||
const speed = step.typeSpeed ?? 0.04;
|
||||
duration = step.text.length * speed * fps + 0.25 * fps;
|
||||
break;
|
||||
}
|
||||
case "bubble_append":
|
||||
duration = (step.durationSeconds ?? 0.9) * fps;
|
||||
break;
|
||||
case "typing_dots":
|
||||
duration = (step.durationSeconds ?? 1.2) * fps;
|
||||
break;
|
||||
case "highlight_box":
|
||||
duration = (step.durationSeconds ?? 1.5) * fps;
|
||||
blocks = false; // non-blocking — subsequent steps can overlap
|
||||
break;
|
||||
case "callout_balloon":
|
||||
duration = (step.durationSeconds ?? 2.2) * fps;
|
||||
blocks = false;
|
||||
break;
|
||||
case "pause":
|
||||
duration = step.seconds * fps;
|
||||
break;
|
||||
}
|
||||
|
||||
timed.push({
|
||||
step,
|
||||
startFrame: Math.round(frameCursor),
|
||||
endFrame: Math.round(frameCursor + duration),
|
||||
cursorBefore: before,
|
||||
cursorAfter: after,
|
||||
});
|
||||
cursor = after;
|
||||
if (blocks) frameCursor += duration;
|
||||
}
|
||||
|
||||
const totalFrames = Math.max(
|
||||
...timed.map((t) => t.endFrame),
|
||||
Math.round(frameCursor)
|
||||
);
|
||||
return { timed, totalFrames };
|
||||
}
|
||||
|
||||
// ---------- SVG cursor ----------
|
||||
|
||||
const CursorArrow: React.FC<{ size?: number }> = ({ size = 28 }) => (
|
||||
<svg width={size} height={size * 1.2} viewBox="0 0 16 20" style={{ display: "block" }}>
|
||||
<path
|
||||
d="M2 2 L2 16 L6 12 L8.5 17 L10.5 16 L8 11 L13 11 Z"
|
||||
fill="#FFFFFF"
|
||||
stroke="#111"
|
||||
strokeWidth={1.2}
|
||||
strokeLinejoin="round"
|
||||
/>
|
||||
</svg>
|
||||
);
|
||||
|
||||
// ---------- Main component ----------
|
||||
|
||||
export const ScreenshotScene: React.FC<ScreenshotSceneProps> = ({
|
||||
backgroundImage,
|
||||
backgroundSize,
|
||||
steps,
|
||||
accentColor = "#F59E0B",
|
||||
cursorStartAt = [0.95, 0.05],
|
||||
}) => {
|
||||
const frame = useCurrentFrame();
|
||||
const { fps, width: cvW, height: cvH } = useVideoConfig();
|
||||
|
||||
const imgW = backgroundSize?.width ?? 1920;
|
||||
const imgH = backgroundSize?.height ?? 1080;
|
||||
const rect = containRect(imgW, imgH, cvW, cvH);
|
||||
|
||||
// Convert normalized (0-1) backdrop coord to absolute canvas pixels
|
||||
const abs = (p: Point): { x: number; y: number } => ({
|
||||
x: rect.x + p[0] * rect.w,
|
||||
y: rect.y + p[1] * rect.h,
|
||||
});
|
||||
const absRect = (r: Region) => ({
|
||||
left: rect.x + r.x * rect.w,
|
||||
top: rect.y + r.y * rect.h,
|
||||
width: r.w * rect.w,
|
||||
height: r.h * rect.h,
|
||||
});
|
||||
|
||||
// Walk timeline once
|
||||
const { timed } = walkTimeline(steps, fps, cursorStartAt);
|
||||
|
||||
// --- Cursor position at current frame ---
|
||||
// Find the active cursor_move or the completed-most-recent one.
|
||||
let cursorPos = cursorStartAt;
|
||||
for (const t of timed) {
|
||||
if (frame >= t.endFrame) {
|
||||
cursorPos = t.cursorAfter;
|
||||
} else if (frame >= t.startFrame && t.step.kind === "cursor_move") {
|
||||
const p = interpolate(frame, [t.startFrame, t.endFrame], [0, 1], {
|
||||
extrapolateRight: "clamp",
|
||||
});
|
||||
// Ease-out so cursor decelerates as it arrives
|
||||
const eased = 1 - Math.pow(1 - p, 3);
|
||||
cursorPos = [
|
||||
t.cursorBefore[0] + (t.cursorAfter[0] - t.cursorBefore[0]) * eased,
|
||||
t.cursorBefore[1] + (t.cursorAfter[1] - t.cursorBefore[1]) * eased,
|
||||
];
|
||||
break;
|
||||
} else if (frame < t.startFrame) {
|
||||
cursorPos = t.cursorBefore;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
const cursorAbs = abs(cursorPos);
|
||||
|
||||
return (
|
||||
<AbsoluteFill style={{ background: "#000" }}>
|
||||
{/* Backdrop */}
|
||||
<Img
|
||||
src={resolveAsset(backgroundImage)}
|
||||
style={{
|
||||
position: "absolute",
|
||||
left: rect.x,
|
||||
top: rect.y,
|
||||
width: rect.w,
|
||||
height: rect.h,
|
||||
objectFit: "fill",
|
||||
}}
|
||||
/>
|
||||
|
||||
{/* Overlays — render in order so later steps paint on top.
|
||||
Sticky kinds (type_into, bubble_append) persist once they appear;
|
||||
transient kinds fade out after their duration. */}
|
||||
{timed.map((t, i) => {
|
||||
const kind = t.step.kind;
|
||||
const sticky = kind === "type_into" || kind === "bubble_append";
|
||||
const active = sticky
|
||||
? frame >= t.startFrame
|
||||
: frame >= t.startFrame && frame <= t.endFrame + fps * 0.4;
|
||||
if (!active) return null;
|
||||
return (
|
||||
<OverlayForStep
|
||||
key={i}
|
||||
timed={t}
|
||||
frame={frame}
|
||||
fps={fps}
|
||||
rect={rect}
|
||||
abs={abs}
|
||||
absRect={absRect}
|
||||
accentColor={accentColor}
|
||||
/>
|
||||
);
|
||||
})}
|
||||
|
||||
{/* Cursor — always on top */}
|
||||
<div
|
||||
style={{
|
||||
position: "absolute",
|
||||
left: cursorAbs.x - 4,
|
||||
top: cursorAbs.y - 2,
|
||||
pointerEvents: "none",
|
||||
filter: "drop-shadow(0 2px 4px rgba(0,0,0,0.4))",
|
||||
}}
|
||||
>
|
||||
<CursorArrow size={Math.round(rect.w * 0.018)} />
|
||||
</div>
|
||||
</AbsoluteFill>
|
||||
);
|
||||
};
|
||||
|
||||
// ---------- Per-step overlay renderers ----------
|
||||
|
||||
interface OverlayProps {
|
||||
timed: TimedStep;
|
||||
frame: number;
|
||||
fps: number;
|
||||
rect: { x: number; y: number; w: number; h: number };
|
||||
abs: (p: Point) => { x: number; y: number };
|
||||
absRect: (r: Region) => { left: number; top: number; width: number; height: number };
|
||||
accentColor: string;
|
||||
}
|
||||
|
||||
const OverlayForStep: React.FC<OverlayProps> = ({
|
||||
timed,
|
||||
frame,
|
||||
fps,
|
||||
rect,
|
||||
abs,
|
||||
absRect,
|
||||
accentColor,
|
||||
}) => {
|
||||
const { step, startFrame, endFrame } = timed;
|
||||
const localFrame = frame - startFrame;
|
||||
|
||||
if (step.kind === "click_pulse") {
|
||||
const at = step.at ?? timed.cursorBefore;
|
||||
const p = abs(at);
|
||||
const progress = interpolate(localFrame, [0, endFrame - startFrame], [0, 1], {
|
||||
extrapolateRight: "clamp",
|
||||
});
|
||||
const size = interpolate(progress, [0, 1], [10, 80]);
|
||||
const alpha = interpolate(progress, [0, 1], [0.85, 0]);
|
||||
const color = step.color ?? accentColor;
|
||||
return (
|
||||
<div
|
||||
style={{
|
||||
position: "absolute",
|
||||
left: p.x - size / 2,
|
||||
top: p.y - size / 2,
|
||||
width: size,
|
||||
height: size,
|
||||
borderRadius: "50%",
|
||||
border: `3px solid ${color}`,
|
||||
opacity: alpha,
|
||||
pointerEvents: "none",
|
||||
}}
|
||||
/>
|
||||
);
|
||||
}
|
||||
|
||||
if (step.kind === "type_into") {
|
||||
const r = absRect(step.region);
|
||||
const speed = step.typeSpeed ?? 0.04;
|
||||
const totalChars = step.text.length;
|
||||
const typeFrames = totalChars * speed * fps;
|
||||
const revealed = Math.min(
|
||||
totalChars,
|
||||
Math.floor(interpolate(localFrame, [0, typeFrames], [0, totalChars], { extrapolateRight: "clamp" }))
|
||||
);
|
||||
const typed = step.text.slice(0, revealed);
|
||||
const fontPx = Math.round(rect.h * (step.fontSize ?? 0.024));
|
||||
const blink = Math.floor(frame / (fps * 0.5)) % 2 === 0;
|
||||
return (
|
||||
<div
|
||||
style={{
|
||||
position: "absolute",
|
||||
left: r.left,
|
||||
top: r.top,
|
||||
width: r.width,
|
||||
height: r.height,
|
||||
display: "flex",
|
||||
alignItems: "center",
|
||||
paddingLeft: Math.round(rect.w * 0.012),
|
||||
fontFamily: "Inter, -apple-system, sans-serif",
|
||||
fontSize: fontPx,
|
||||
color: step.color ?? "#E5E7EB",
|
||||
pointerEvents: "none",
|
||||
whiteSpace: "nowrap",
|
||||
overflow: "hidden",
|
||||
}}
|
||||
>
|
||||
<span>{typed}</span>
|
||||
{blink && (
|
||||
<span
|
||||
style={{
|
||||
display: "inline-block",
|
||||
width: 2,
|
||||
height: fontPx * 0.95,
|
||||
background: step.color ?? "#E5E7EB",
|
||||
marginLeft: 2,
|
||||
}}
|
||||
/>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
if (step.kind === "bubble_append") {
|
||||
const r = absRect(step.region);
|
||||
const springIn = spring({
|
||||
frame: localFrame,
|
||||
fps,
|
||||
config: { damping: 16, stiffness: 140 },
|
||||
durationInFrames: Math.ceil(fps * 0.5),
|
||||
});
|
||||
const isUser = step.role === "user";
|
||||
const fontPx = Math.round(rect.h * (step.fontSize ?? 0.021));
|
||||
|
||||
// Streaming text reveal (word-by-word)
|
||||
let displayText = step.text;
|
||||
if (step.stream) {
|
||||
const words = step.text.split(/(\s+)/); // keep whitespace
|
||||
const totalRevealFrames = Math.max(1, endFrame - startFrame - fps * 0.3);
|
||||
const wordCount = words.filter((w) => w.trim()).length;
|
||||
const revealedWords = Math.floor(
|
||||
interpolate(localFrame, [fps * 0.3, fps * 0.3 + totalRevealFrames], [0, wordCount], {
|
||||
extrapolateRight: "clamp",
|
||||
extrapolateLeft: "clamp",
|
||||
})
|
||||
);
|
||||
let count = 0;
|
||||
const pieces: string[] = [];
|
||||
for (const w of words) {
|
||||
if (w.trim()) {
|
||||
if (count < revealedWords) {
|
||||
pieces.push(w);
|
||||
count++;
|
||||
} else {
|
||||
break;
|
||||
}
|
||||
} else {
|
||||
pieces.push(w);
|
||||
}
|
||||
}
|
||||
displayText = pieces.join("");
|
||||
}
|
||||
|
||||
const bg = isUser ? "#2D3748" : "#1F2937";
|
||||
const border = isUser ? "#4A5568" : "#374151";
|
||||
|
||||
return (
|
||||
<div
|
||||
style={{
|
||||
position: "absolute",
|
||||
left: r.left,
|
||||
top: r.top,
|
||||
width: r.width,
|
||||
minHeight: r.height,
|
||||
background: bg,
|
||||
border: `1px solid ${border}`,
|
||||
borderRadius: Math.round(rect.w * 0.008),
|
||||
padding: `${Math.round(rect.h * 0.015)}px ${Math.round(rect.w * 0.012)}px`,
|
||||
fontFamily: "Inter, -apple-system, sans-serif",
|
||||
fontSize: fontPx,
|
||||
color: "#F1F5F9",
|
||||
lineHeight: 1.5,
|
||||
opacity: springIn,
|
||||
transform: `translateY(${(1 - springIn) * 20}px)`,
|
||||
boxShadow: "0 4px 20px rgba(0,0,0,0.3)",
|
||||
whiteSpace: "pre-wrap",
|
||||
pointerEvents: "none",
|
||||
overflow: "hidden",
|
||||
}}
|
||||
>
|
||||
{displayText}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
if (step.kind === "typing_dots") {
|
||||
const p = abs(step.at);
|
||||
const dotSize = Math.round(rect.h * 0.01);
|
||||
const dots = [0, 1, 2].map((i) => {
|
||||
const phase = (frame / (fps * 0.35) - i * 0.3) % 2;
|
||||
const alpha = phase < 1 ? 0.3 + phase * 0.7 : 1 - (phase - 1) * 0.7;
|
||||
return alpha;
|
||||
});
|
||||
const color = step.color ?? accentColor;
|
||||
return (
|
||||
<div
|
||||
style={{
|
||||
position: "absolute",
|
||||
left: p.x,
|
||||
top: p.y,
|
||||
display: "flex",
|
||||
gap: dotSize * 0.7,
|
||||
pointerEvents: "none",
|
||||
}}
|
||||
>
|
||||
{dots.map((a, i) => (
|
||||
<div
|
||||
key={i}
|
||||
style={{
|
||||
width: dotSize,
|
||||
height: dotSize,
|
||||
borderRadius: "50%",
|
||||
background: color,
|
||||
opacity: Math.max(0.3, a),
|
||||
}}
|
||||
/>
|
||||
))}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
if (step.kind === "highlight_box") {
|
||||
const r = absRect(step.region);
|
||||
const dur = endFrame - startFrame;
|
||||
const pulses = step.pulses ?? 2;
|
||||
const color = step.color ?? accentColor;
|
||||
// Pulsing ring: oscillate opacity + scale
|
||||
const wave = Math.sin((localFrame / dur) * pulses * Math.PI * 2) * 0.5 + 0.5;
|
||||
const alpha = 0.4 + wave * 0.5;
|
||||
const glow = 10 + wave * 18;
|
||||
return (
|
||||
<div
|
||||
style={{
|
||||
position: "absolute",
|
||||
left: r.left - 6,
|
||||
top: r.top - 6,
|
||||
width: r.width + 12,
|
||||
height: r.height + 12,
|
||||
border: `3px solid ${color}`,
|
||||
borderRadius: Math.round(rect.w * 0.006),
|
||||
boxShadow: `0 0 ${glow}px ${color}`,
|
||||
opacity: alpha,
|
||||
pointerEvents: "none",
|
||||
}}
|
||||
/>
|
||||
);
|
||||
}
|
||||
|
||||
if (step.kind === "callout_balloon") {
|
||||
const a = abs(step.anchor);
|
||||
const pos = step.position ?? "top";
|
||||
const springIn = spring({
|
||||
frame: localFrame,
|
||||
fps,
|
||||
config: { damping: 14, stiffness: 160 },
|
||||
durationInFrames: Math.ceil(fps * 0.4),
|
||||
});
|
||||
const dur = endFrame - startFrame;
|
||||
const fadeOut = interpolate(
|
||||
localFrame,
|
||||
[dur - fps * 0.4, dur],
|
||||
[1, 0],
|
||||
{ extrapolateLeft: "clamp", extrapolateRight: "clamp" }
|
||||
);
|
||||
const alpha = Math.min(springIn, fadeOut);
|
||||
const color = step.color ?? accentColor;
|
||||
const fontPx = Math.round(rect.h * 0.024);
|
||||
const maxW = rect.w * 0.28;
|
||||
|
||||
// Balloon offset from anchor
|
||||
const offset = rect.h * 0.06;
|
||||
let bx = a.x;
|
||||
let by = a.y;
|
||||
let tailStyle: React.CSSProperties = {};
|
||||
if (pos === "top") {
|
||||
by = a.y - offset - fontPx * 2.5;
|
||||
bx = a.x - maxW / 2;
|
||||
tailStyle = {
|
||||
position: "absolute",
|
||||
bottom: -10,
|
||||
left: "50%",
|
||||
transform: "translateX(-50%)",
|
||||
width: 0,
|
||||
height: 0,
|
||||
borderLeft: "10px solid transparent",
|
||||
borderRight: "10px solid transparent",
|
||||
borderTop: `12px solid ${color}`,
|
||||
};
|
||||
} else if (pos === "bottom") {
|
||||
by = a.y + offset;
|
||||
bx = a.x - maxW / 2;
|
||||
tailStyle = {
|
||||
position: "absolute",
|
||||
top: -10,
|
||||
left: "50%",
|
||||
transform: "translateX(-50%)",
|
||||
width: 0,
|
||||
height: 0,
|
||||
borderLeft: "10px solid transparent",
|
||||
borderRight: "10px solid transparent",
|
||||
borderBottom: `12px solid ${color}`,
|
||||
};
|
||||
} else if (pos === "left") {
|
||||
bx = a.x - offset - maxW;
|
||||
by = a.y - fontPx;
|
||||
tailStyle = {
|
||||
position: "absolute",
|
||||
right: -10,
|
||||
top: "50%",
|
||||
transform: "translateY(-50%)",
|
||||
width: 0,
|
||||
height: 0,
|
||||
borderTop: "10px solid transparent",
|
||||
borderBottom: "10px solid transparent",
|
||||
borderLeft: `12px solid ${color}`,
|
||||
};
|
||||
} else {
|
||||
bx = a.x + offset;
|
||||
by = a.y - fontPx;
|
||||
tailStyle = {
|
||||
position: "absolute",
|
||||
left: -10,
|
||||
top: "50%",
|
||||
transform: "translateY(-50%)",
|
||||
width: 0,
|
||||
height: 0,
|
||||
borderTop: "10px solid transparent",
|
||||
borderBottom: "10px solid transparent",
|
||||
borderRight: `12px solid ${color}`,
|
||||
};
|
||||
}
|
||||
|
||||
return (
|
||||
<div
|
||||
style={{
|
||||
position: "absolute",
|
||||
left: Math.max(rect.x + 8, Math.min(rect.x + rect.w - maxW - 8, bx)),
|
||||
top: by,
|
||||
width: maxW,
|
||||
background: color,
|
||||
color: "#0B0F1A",
|
||||
fontFamily: "Inter, -apple-system, sans-serif",
|
||||
fontWeight: 600,
|
||||
fontSize: fontPx,
|
||||
lineHeight: 1.35,
|
||||
padding: `${Math.round(fontPx * 0.6)}px ${Math.round(fontPx * 0.9)}px`,
|
||||
borderRadius: Math.round(rect.w * 0.008),
|
||||
opacity: alpha,
|
||||
transform: `scale(${interpolate(springIn, [0, 1], [0.9, 1])})`,
|
||||
boxShadow: "0 10px 30px rgba(0,0,0,0.35)",
|
||||
pointerEvents: "none",
|
||||
}}
|
||||
>
|
||||
{step.text}
|
||||
<div style={tailStyle} />
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
return null;
|
||||
};
|
||||
@@ -11,7 +11,9 @@ export { HeroTitle } from "./HeroTitle";
|
||||
export { ParticleOverlay } from "./ParticleOverlay";
|
||||
export { AnimeScene } from "./AnimeScene";
|
||||
export { TerminalScene } from "./TerminalScene";
|
||||
export { ScreenshotScene } from "./ScreenshotScene";
|
||||
export { ProviderChip } from "./ProviderChip";
|
||||
export type { ParticleType } from "./ParticleOverlay";
|
||||
export type { CameraMotion, AnimeSceneProps } from "./AnimeScene";
|
||||
export type { TerminalStep } from "./TerminalScene";
|
||||
export type { ScreenshotStep, Region, Point } from "./ScreenshotScene";
|
||||
|
||||
@@ -15,7 +15,11 @@ ARTIFACT_NAMES = [
|
||||
"proposal_packet",
|
||||
"brief",
|
||||
"script",
|
||||
"character_design",
|
||||
"rig_plan",
|
||||
"pose_library",
|
||||
"scene_plan",
|
||||
"action_timeline",
|
||||
"asset_manifest",
|
||||
"edit_decisions",
|
||||
"render_report",
|
||||
@@ -25,6 +29,7 @@ ARTIFACT_NAMES = [
|
||||
"decision_log",
|
||||
"source_media_review",
|
||||
"final_review",
|
||||
"character_qa_report",
|
||||
"video_analysis_brief",
|
||||
]
|
||||
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "openmontage/artifacts/action_timeline",
|
||||
"title": "Action Timeline",
|
||||
"description": "Timed character actions and poses compiled for rendering.",
|
||||
"type": "object",
|
||||
"required": ["version", "scenes"],
|
||||
"properties": {
|
||||
"version": { "type": "string", "const": "1.0" },
|
||||
"fps": { "type": "number", "minimum": 1, "default": 30 },
|
||||
"scenes": {
|
||||
"type": "array",
|
||||
"minItems": 1,
|
||||
"items": {
|
||||
"type": "object",
|
||||
"required": ["scene_id", "start_seconds", "end_seconds", "actions"],
|
||||
"properties": {
|
||||
"scene_id": { "type": "string" },
|
||||
"start_seconds": { "type": "number", "minimum": 0 },
|
||||
"end_seconds": { "type": "number", "minimum": 0 },
|
||||
"camera": { "type": "object" },
|
||||
"background": { "type": "string" },
|
||||
"effects": { "type": "array", "items": { "type": "string" } },
|
||||
"actions": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"required": ["at_seconds", "character_id", "action"],
|
||||
"properties": {
|
||||
"at_seconds": { "type": "number", "minimum": 0 },
|
||||
"duration_seconds": { "type": "number", "minimum": 0 },
|
||||
"character_id": { "type": "string" },
|
||||
"action": { "type": "string" },
|
||||
"pose": { "type": "string" },
|
||||
"emotion": { "type": "string" },
|
||||
"target": { "type": "string" },
|
||||
"easing": { "type": "string" },
|
||||
"notes": { "type": "string" }
|
||||
},
|
||||
"additionalProperties": false
|
||||
}
|
||||
}
|
||||
},
|
||||
"additionalProperties": false
|
||||
}
|
||||
},
|
||||
"metadata": { "type": "object" }
|
||||
},
|
||||
"additionalProperties": false
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "openmontage/artifacts/character_design",
|
||||
"title": "Character Design",
|
||||
"description": "Character definitions for local rigged animation.",
|
||||
"type": "object",
|
||||
"required": ["version", "characters"],
|
||||
"properties": {
|
||||
"version": { "type": "string", "const": "1.0" },
|
||||
"style": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"visual_style": { "type": "string" },
|
||||
"palette": { "type": "array", "items": { "type": "string" } },
|
||||
"line_style": { "type": "string" },
|
||||
"texture": { "type": "string" }
|
||||
},
|
||||
"additionalProperties": false
|
||||
},
|
||||
"characters": {
|
||||
"type": "array",
|
||||
"minItems": 1,
|
||||
"items": {
|
||||
"type": "object",
|
||||
"required": ["id", "role", "body_type", "style", "required_emotions", "required_actions"],
|
||||
"properties": {
|
||||
"id": { "type": "string" },
|
||||
"display_name": { "type": "string" },
|
||||
"role": { "type": "string" },
|
||||
"body_type": { "type": "string" },
|
||||
"style": { "type": "string" },
|
||||
"silhouette_notes": { "type": "string" },
|
||||
"required_emotions": { "type": "array", "items": { "type": "string" } },
|
||||
"required_actions": { "type": "array", "items": { "type": "string" } },
|
||||
"required_views": { "type": "array", "items": { "type": "string" } },
|
||||
"props": { "type": "array", "items": { "type": "string" } },
|
||||
"constraints": { "type": "array", "items": { "type": "string" } }
|
||||
},
|
||||
"additionalProperties": false
|
||||
}
|
||||
},
|
||||
"metadata": { "type": "object" }
|
||||
},
|
||||
"additionalProperties": false
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "openmontage/artifacts/character_qa_report",
|
||||
"title": "Character QA Report",
|
||||
"description": "Structured review of character rig integrity, motion, and render readiness.",
|
||||
"type": "object",
|
||||
"required": ["version", "status", "checks"],
|
||||
"properties": {
|
||||
"version": { "type": "string", "const": "1.0" },
|
||||
"status": { "type": "string", "enum": ["pass", "revise", "fail"] },
|
||||
"preview_path": { "type": "string" },
|
||||
"checks": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"schema_valid": { "type": "boolean" },
|
||||
"assets_exist": { "type": "boolean" },
|
||||
"pivots_defined": { "type": "boolean" },
|
||||
"poses_defined": { "type": "boolean" },
|
||||
"actions_timed": { "type": "boolean" },
|
||||
"motion_detected": { "type": "boolean" },
|
||||
"browser_preview_checked": { "type": "boolean" },
|
||||
"frame_samples_checked": { "type": "boolean" }
|
||||
},
|
||||
"additionalProperties": false
|
||||
},
|
||||
"issues": { "type": "array", "items": { "type": "string" } },
|
||||
"recommended_action": {
|
||||
"type": "string",
|
||||
"enum": ["present_to_user", "fix_rig", "fix_assets", "fix_timeline", "block"]
|
||||
},
|
||||
"metadata": { "type": "object" }
|
||||
},
|
||||
"additionalProperties": false
|
||||
}
|
||||
@@ -0,0 +1,41 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "openmontage/artifacts/pose_library",
|
||||
"title": "Pose Library",
|
||||
"description": "Named reusable poses, expressions, and transition hints for character rigs.",
|
||||
"type": "object",
|
||||
"required": ["version", "characters"],
|
||||
"properties": {
|
||||
"version": { "type": "string", "const": "1.0" },
|
||||
"characters": {
|
||||
"type": "array",
|
||||
"minItems": 1,
|
||||
"items": {
|
||||
"type": "object",
|
||||
"required": ["character_id", "poses"],
|
||||
"properties": {
|
||||
"character_id": { "type": "string" },
|
||||
"poses": {
|
||||
"type": "object",
|
||||
"additionalProperties": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"description": { "type": "string" },
|
||||
"parts": { "type": "object" },
|
||||
"expression": { "type": "string" },
|
||||
"hold_frames": { "type": "integer", "minimum": 0 },
|
||||
"transition": { "type": "string" }
|
||||
},
|
||||
"additionalProperties": true
|
||||
}
|
||||
},
|
||||
"mouth_shapes": { "type": "object" },
|
||||
"action_cycles": { "type": "object" }
|
||||
},
|
||||
"additionalProperties": false
|
||||
}
|
||||
},
|
||||
"metadata": { "type": "object" }
|
||||
},
|
||||
"additionalProperties": false
|
||||
}
|
||||
@@ -0,0 +1,59 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "openmontage/artifacts/rig_plan",
|
||||
"title": "Rig Plan",
|
||||
"description": "Rig parts, pivots, layers, constraints, and views for character animation.",
|
||||
"type": "object",
|
||||
"required": ["version", "characters"],
|
||||
"properties": {
|
||||
"version": { "type": "string", "const": "1.0" },
|
||||
"characters": {
|
||||
"type": "array",
|
||||
"minItems": 1,
|
||||
"items": {
|
||||
"type": "object",
|
||||
"required": ["character_id", "parts", "joints", "layers", "required_poses"],
|
||||
"properties": {
|
||||
"character_id": { "type": "string" },
|
||||
"rig_type": { "type": "string", "enum": ["svg_rig", "canvas_procedural", "lottie", "hybrid"], "default": "svg_rig" },
|
||||
"parts": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"required": ["id", "kind", "layer"],
|
||||
"properties": {
|
||||
"id": { "type": "string" },
|
||||
"kind": { "type": "string" },
|
||||
"layer": { "type": "integer" },
|
||||
"asset_path": { "type": "string" },
|
||||
"parent": { "type": "string" }
|
||||
},
|
||||
"additionalProperties": false
|
||||
}
|
||||
},
|
||||
"joints": {
|
||||
"type": "object",
|
||||
"additionalProperties": {
|
||||
"type": "object",
|
||||
"required": ["pivot"],
|
||||
"properties": {
|
||||
"pivot": { "type": "array", "items": { "type": "number" }, "minItems": 2, "maxItems": 2 },
|
||||
"rotation": { "type": "array", "items": { "type": "number" }, "minItems": 2, "maxItems": 2 },
|
||||
"scale": { "type": "array", "items": { "type": "number" }, "minItems": 2, "maxItems": 2 }
|
||||
},
|
||||
"additionalProperties": false
|
||||
}
|
||||
},
|
||||
"layers": { "type": "array", "items": { "type": "string" } },
|
||||
"views": { "type": "array", "items": { "type": "string" } },
|
||||
"required_poses": { "type": "array", "items": { "type": "string" } },
|
||||
"required_actions": { "type": "array", "items": { "type": "string" } },
|
||||
"risks": { "type": "array", "items": { "type": "string" } }
|
||||
},
|
||||
"additionalProperties": false
|
||||
}
|
||||
},
|
||||
"metadata": { "type": "object" }
|
||||
},
|
||||
"additionalProperties": false
|
||||
}
|
||||
@@ -17,7 +17,7 @@
|
||||
"id": { "type": "string" },
|
||||
"type": {
|
||||
"type": "string",
|
||||
"enum": ["talking_head", "broll", "animation", "diagram", "text_card", "transition", "generated", "screen_recording"]
|
||||
"enum": ["talking_head", "broll", "animation", "character_scene", "diagram", "text_card", "transition", "generated", "screen_recording"]
|
||||
},
|
||||
"description": { "type": "string" },
|
||||
"start_seconds": { "type": "number", "minimum": 0 },
|
||||
@@ -68,6 +68,27 @@
|
||||
"default": false,
|
||||
"description": "True if this is the visual peak of the video — deserves extra attention"
|
||||
},
|
||||
"character_actions": {
|
||||
"type": "array",
|
||||
"description": "Character-specific acting beats for rigged character animation scenes",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"required": ["character_id", "action_sequence"],
|
||||
"properties": {
|
||||
"character_id": { "type": "string" },
|
||||
"emotion": { "type": "string" },
|
||||
"action_sequence": {
|
||||
"type": "array",
|
||||
"items": { "type": "string" },
|
||||
"minItems": 1
|
||||
},
|
||||
"dialogue": { "type": "string" },
|
||||
"target": { "type": "string" },
|
||||
"notes": { "type": "string" }
|
||||
},
|
||||
"additionalProperties": false
|
||||
}
|
||||
},
|
||||
"texture_keywords": {
|
||||
"type": "array",
|
||||
"items": { "type": "string" },
|
||||
|
||||
+3
-1
@@ -58,6 +58,7 @@ Key capability families to look for in the output:
|
||||
| `audio_processing` | — | FFmpeg-based local tools |
|
||||
| `enhancement` | — | Mixed providers |
|
||||
| `analysis` | — | Mixed providers |
|
||||
| `character_animation` | — | Local character specs, SVG rigs, pose libraries, action timelines, previews, and QA |
|
||||
| `graphics` | — | Local rendering tools |
|
||||
| `music_generation` | — | Single-provider |
|
||||
| `subtitle` | — | Pure Python |
|
||||
@@ -92,7 +93,7 @@ Key capability families to look for in the output:
|
||||
| Enhancement Strategy | `creative/enhancement-strategy.md` | Overlay placement and density | `ffmpeg` |
|
||||
| Data Visualization | `creative/data-visualization.md` | Chart type selection, animation, label placement | `d3-viz`, `remotion-best-practices` |
|
||||
| Video Stitching | `creative/video-stitching.md` | Multi-clip assembly, AI clip chaining, spatial composition | `ffmpeg`, `video_toolkit` |
|
||||
| Video Gen Prompting | `creative/video-gen-prompting.md` | Universal video generation prompt vocabulary | `ai-video-gen`, `ltx2`, `create-video` |
|
||||
| Video Gen Prompting | `creative/video-gen-prompting.md` | Universal video generation prompt vocabulary; **canonical 5-aspect spec** (Subject / Motion / Scene / Spatial / Camera); ~200 cinematography primitives | `ai-video-gen`, `ltx2`, `create-video` |
|
||||
| ↳ Seedance Prompting | `creative/prompting/seedance-prompting.md` | **Preferred premium default.** Seedance 2.0 8-component structure, multi-shot, lip-sync, reference-to-video | `seedance-2-0`, `ai-video-gen` |
|
||||
| ↳ Grok Prompting | `creative/prompting/grok-prompting.md` | Grok image/video prompting, edit flows, reference-image video | `grok-media` |
|
||||
| ↳ Sora Prompting | `creative/prompting/sora-prompting.md` | Sora 2 structured template, advanced fields | `ai-video-gen` |
|
||||
@@ -127,6 +128,7 @@ Pipeline type skills provide production guidance for specific video formats, ind
|
||||
| Long-Form | `creative/long-form.md` | YouTube 10+ min â€" chapters, retention, end screens |
|
||||
| Screen Recording | `creative/screen-recording.md` | Code walkthroughs, tutorials, software demos |
|
||||
| Animation Pipeline | `creative/animation-pipeline.md` | Motion graphics, easing, transitions, composition |
|
||||
| Character Animation Pipeline | `pipelines/character-animation/` | Rigged local cartoon characters, pose libraries, action timelines, SVG/Canvas/Remotion/HyperFrames rendering |
|
||||
| Cinematic | `creative/cinematic.md` | Letterbox, film pacing, layered audio, color grading |
|
||||
|
||||
## Pipeline Stage Director Skills
|
||||
|
||||
+60
-38
@@ -288,54 +288,76 @@ Same pattern as Remotion.
|
||||
Things the upstream docs don't warn about but will cost you a 60-minute
|
||||
render to discover. Fix at author time, not at render time.
|
||||
|
||||
### Video elements need explicit size, in BOTH HTML attrs and `!important` CSS
|
||||
### Full-frame background video: source-resolution trap (NOT a framework bug)
|
||||
|
||||
HyperFrames runtime applies inline `style="width:...px; height:...px"` to
|
||||
every `<video>` element based on the video's **intrinsic** dimensions.
|
||||
That inline style beats any class-selector CSS you write, so even a
|
||||
`video.bg-video { width: 100%; height: 100% }` rule gets silently ignored
|
||||
and the video renders at whatever size the decoded stream reports —
|
||||
usually a small centered box inside an otherwise-black clip.
|
||||
Symptom: background video renders as a small centered box with black
|
||||
around it, even though you've told HyperFrames the clip is 1920×1080.
|
||||
Six renders of debugging revealed this is **almost always a source
|
||||
quality issue, not a HyperFrames framework bug.** Fix the input, not
|
||||
the CSS.
|
||||
|
||||
The bug is invisible in some scenarios:
|
||||
- Low `filter: brightness(0.25-0.35)` makes the small video box look
|
||||
mostly black.
|
||||
- Fullscreen typography layered on top hides the problem.
|
||||
- `object-fit: cover` on a class selector does NOT fix it because the
|
||||
inline width/height wins first.
|
||||
Root cause (observed on Pexels + Pixabay stock): many free stock
|
||||
clips ship at 640×360 or 960×540 even when you ask for the "large"
|
||||
size tier. If your pre-transform does
|
||||
`scale=1920:1080:force_original_aspect_ratio=decrease,pad=1920:1080:...`
|
||||
you'll produce a 1920×1080 file whose visible content is a centered
|
||||
640×360 rectangle with black padding. HyperFrames plays this clip
|
||||
faithfully — the small-video-in-black-frame you see is a correctly
|
||||
rendered letterboxed input.
|
||||
|
||||
It becomes obvious the moment you go footage-forward (brightness > 0.5,
|
||||
lower-third typography) — the small centered video box appears against
|
||||
a black frame that was supposed to be full-bleed b-roll.
|
||||
Diagnostic first: before blaming CSS, run
|
||||
`ffprobe -v error -select_streams v -show_entries stream=width,height`
|
||||
on the source clip and on your pre-transformed clip. If the pre-
|
||||
transform already shows a 640×360 active region inside 1920×1080
|
||||
padding, the problem is upstream.
|
||||
|
||||
**Correct pattern** for a full-frame background video:
|
||||
Fix the pre-transform to scale-to-COVER and crop, not scale-to-FIT
|
||||
and pad:
|
||||
|
||||
```
|
||||
-vf "scale=1920:1080:force_original_aspect_ratio=increase,\
|
||||
crop=1920:1080,unsharp=3:3:0.6"
|
||||
```
|
||||
|
||||
This upscales the small clip to at least cover 1920×1080, crops the
|
||||
overflow, and sharpens (unsharp) to offset perceived softness. The
|
||||
output is full-bleed and HyperFrames will render it edge-to-edge.
|
||||
|
||||
The canonical wrapper-div pattern (`patterns.md`) is still the right
|
||||
HTML shape for backgrounds:
|
||||
|
||||
```html
|
||||
<video
|
||||
class="clip bg-video"
|
||||
data-start="0" data-duration="8" data-track-index="0"
|
||||
src="assets/hero.mp4"
|
||||
muted playsinline preload="metadata"
|
||||
width="1920" height="1080"
|
||||
></video>
|
||||
<div style="position:absolute;top:0;left:0;width:1920px;height:1080px;overflow:hidden;">
|
||||
<video
|
||||
data-start="0" data-duration="60" data-track-index="0"
|
||||
src="..." muted playsinline
|
||||
style="width:100%;height:100%;object-fit:cover;"
|
||||
></video>
|
||||
</div>
|
||||
```
|
||||
|
||||
```css
|
||||
video.bg-video {
|
||||
position: absolute !important; inset: 0 !important;
|
||||
width: 100% !important; height: 100% !important;
|
||||
object-fit: cover !important;
|
||||
/* filter optional */
|
||||
}
|
||||
```
|
||||
Use it because `overflow: hidden` on the wrapper crops gracefully when
|
||||
the video's aspect differs slightly from 16:9, and because
|
||||
`object-fit: cover` on the inner `<video>` handles the (rare) case
|
||||
where source aspect isn't 16:9. It is **not** a workaround for a
|
||||
framework layout bug — the framework will size `<video class="clip">`
|
||||
correctly too if the source file has content filling the full frame.
|
||||
|
||||
Both the explicit `width="1920" height="1080"` HTML attrs AND the
|
||||
`!important` CSS are required. Either one alone isn't enough.
|
||||
Apply visual treatments (`filter`, `border-radius`, etc.) on the
|
||||
wrapper or on a scoped selector like `.bg-slot video { filter: ... }`.
|
||||
|
||||
Alternative: wrap the `<video>` in a `<div class="clip">` and put
|
||||
`data-start`/`data-duration` on the wrapper. Let the video fill its div
|
||||
via `position: absolute; inset: 0` without the framework's inline-style
|
||||
interference. Slightly more DOM but more robust.
|
||||
Invisible-vs-obvious failure mode: this trap hides when you stack
|
||||
dim filters (`brightness(0.25–0.35)`) or full-bleed typography on top —
|
||||
the letterbox reads as "moody darkness." It becomes obvious the moment
|
||||
you go footage-forward (brightness > 0.5, lower-third typography). If
|
||||
you're planning footage-forward from the start, probe your source
|
||||
resolutions in the asset stage.
|
||||
|
||||
When you truly can't source HD clips, switch pipelines: use FFmpeg
|
||||
hybrid-compositing (scale-to-cover + crop + unsharp in an external
|
||||
b-roll reel, then overlay HyperFrames typography via chromakey). That's
|
||||
the pattern `projects/quantum-willow-multiverse/` settled on after six
|
||||
HyperFrames renders with 640×360 Pexels sources.
|
||||
|
||||
### Always preview-scrub footage-forward scenes before render
|
||||
|
||||
|
||||
@@ -70,14 +70,18 @@ Fallback: AI-generated image of server racks
|
||||
|
||||
### Query Templates by Scene Type
|
||||
|
||||
| Scene Type | Query Template | Example |
|
||||
Add a **POV keyword** to every query. Stock libraries (Pexels, Pixabay, Storyblocks, Artgrid) explicitly index POV terms — drone, aerial, OTS (over-the-shoulder), macro, top-down, dashcam, FPV, handheld, locked-off — and adding the POV often unlocks better matches than refining the subject. The CMU/Harvard CHAI taxonomy treats POV as a first-class Scene aspect for the same reason: it changes which library shelf you're searching.
|
||||
|
||||
| Scene Type | Query Template | Example with POV |
|
||||
|-----------|---------------|---------|
|
||||
| Establishing | `[place] [time of day]` | "tokyo skyline night" |
|
||||
| Activity | `[person] [action]` | "scientist microscope" |
|
||||
| Object | `[object] [style]` | "circuit board closeup" |
|
||||
| Nature | `[element] [quality]` | "ocean waves aerial" |
|
||||
| Abstract motion | `[movement] [style]` | "light trails timelapse" |
|
||||
| Workplace | `[setting] [activity]` | "modern office meeting" |
|
||||
| Establishing | `[place] [time of day] [POV]` | "tokyo skyline night drone" |
|
||||
| Activity | `[person] [action] [POV]` | "scientist microscope OTS" |
|
||||
| Object | `[object] [style] [POV]` | "circuit board macro top-down" |
|
||||
| Nature | `[element] [quality] [POV]` | "ocean waves aerial drone" |
|
||||
| Abstract motion | `[movement] [style] [POV]` | "light trails timelapse locked-off" |
|
||||
| Workplace | `[setting] [activity] [POV]` | "modern office meeting handheld" |
|
||||
|
||||
If the scene description doesn't already imply a POV, ask the script/scene director — don't default to "no POV." A wrong-POV match (handheld when the scene needs drone) is harder to fix than a wrong color grade.
|
||||
|
||||
## Evaluating Stock Footage Quality
|
||||
|
||||
@@ -89,6 +93,7 @@ When the stock tool returns results, evaluate before using:
|
||||
- **Style compatibility:** Doesn't clash with the playbook's visual style
|
||||
- **No watermarks:** Pexels/Pixabay are license-free, but verify
|
||||
- **Composition:** Subject is well-framed, not cut off awkwardly
|
||||
- **POV match:** Does the footage's actual POV (drone, OTS, macro, handheld, locked-off, etc.) match what the scene needs? A wrong POV — e.g., handheld when the scene wants drone — is **more costly to fix than a wrong color grade**. Reject and re-query rather than try to crop your way out of it.
|
||||
|
||||
### Video Criteria (all image criteria plus)
|
||||
- **Duration:** At least as long as the scene needs (can trim, can't extend)
|
||||
|
||||
@@ -3,6 +3,8 @@
|
||||
> Sources: No Film School editorial guides, StudioBinder filmmaking resources, Film Riot
|
||||
> production tutorials, CinematographyDB shot databases, Walter Murch "In the Blink of an Eye"
|
||||
|
||||
> For the universal cinematography vocabulary (camera, lens, motion, focus primitives, plus the 5-aspect Subject / Subject Motion / Scene / Spatial Framing / Camera spec), see `skills/creative/video-gen-prompting.md`. This file layers cinematic-specific conventions on top of those primitives — it does not redefine them.
|
||||
|
||||
## Quick Reference Card
|
||||
|
||||
```
|
||||
@@ -16,6 +18,19 @@ MUSIC: 60-90 BPM, orchestral or ambient, dynamic (not loop-based)
|
||||
TARGET LUFS: -14 LUFS integrated, -24 LUFS for quiet moments
|
||||
```
|
||||
|
||||
## Replace Mood Adjectives with Visual Causes
|
||||
|
||||
> **"Cinematic" and "epic" don't constrain pixels.** The cinematic look comes from concrete choices: aspect ratio, lens, lighting key, color grade, shot duration, and audio layer count. State those — the rest is decoration.
|
||||
>
|
||||
> The CMU/Harvard CHAI study showed subjective phrasing varies wildly across annotators and model interpretations, which means a beat tagged "moody" routes to a different visual every render. Replace the adjective with the lighting + grade + shot-duration combination that produces moodiness. See `skills/creative/storytelling.md` "Anti-Subjective Rule" for the script-side equivalent.
|
||||
>
|
||||
> | Mood adjective | Cinematic translation |
|
||||
> |---|---|
|
||||
> | "epic" | 2.39:1 letterbox, 24fps, 8s+ shot duration, orchestral score with crescendo |
|
||||
> | "moody" | `moody_dark` grade at 0.6, key light at 1/8 fill, 6s+ contemplative shots, ambient bed at -28dB |
|
||||
> | "intimate" | 1.85:1, 40-50mm equivalent, shallow DoF, 2 audio layers (dialogue + room tone), no music under dialogue |
|
||||
> | "cinematic" | (this word is banned — pick one of the above or describe the actual choices) |
|
||||
|
||||
## Aspect Ratios
|
||||
|
||||
| Ratio | Resolution (in 1080p frame) | Feel | When to Use |
|
||||
@@ -113,6 +128,7 @@ Add a subtle ambient layer to fill silence and create depth:
|
||||
- Highlights should be slightly rolled off (never pure white)
|
||||
- Skin tones must stay on the vectorscope skin tone line
|
||||
- Consistency across all clips — one LUT/profile for the entire video
|
||||
- **If a beat is described as "moody," rewrite as the lighting + grade + shot-duration combination that produces moodiness.** Don't pass mood adjectives to the asset/edit stages.
|
||||
|
||||
## Applying to OpenMontage
|
||||
|
||||
|
||||
@@ -3,6 +3,8 @@
|
||||
> Source: [Tencent Prompt Handbook](https://github.com/Tencent-Hunyuan/HunyuanVideo-1.5/blob/main/assets/HunyuanVideo_1_5_Prompt_Handbook_EN.md)
|
||||
> For universal vocabulary, see: `skills/creative/video-gen-prompting.md`
|
||||
|
||||
**Word count:** Hunyuan 1.5 reads well at 80–200 words; doesn't reward 400-word essays.
|
||||
|
||||
## HunyuanVideo Prompt Formula
|
||||
|
||||
### Text-to-Video
|
||||
@@ -34,15 +36,28 @@ Describe lighting with multiple dimensions:
|
||||
|
||||
| Movement | Type | HunyuanVideo Prompt |
|
||||
|----------|------|-------------------|
|
||||
| Crane / Pedestal | Vertical | "camera rises vertically" |
|
||||
| Truck / Tracking | Horizontal | "camera tracks left alongside subject" |
|
||||
| Dolly In | Push | "camera pushes forward toward subject" |
|
||||
| Dolly Out | Pull | "camera pulls back from subject" |
|
||||
| Pan | Rotation | "camera pans right across the scene" |
|
||||
| Crane / Pedestal | Translation (vertical) | "camera rises vertically" |
|
||||
| Truck / Tracking | Translation (horizontal) | "camera tracks left alongside subject" |
|
||||
| Dolly In | Translation (push) | "camera pushes forward toward subject" |
|
||||
| Dolly Out | Translation (pull) | "camera pulls back from subject" |
|
||||
| Pan | Rotation (yaw) | "camera pans right across the scene" |
|
||||
| Tilt | Rotation (pitch) | "camera tilts upward to follow the rocket" |
|
||||
| Roll | Rotation (Z-axis / Dutch) | "camera rolls clockwise into a Dutch tilt" |
|
||||
| Orbit | Circular | "camera orbits around subject" |
|
||||
| Follow | Lock-on | "camera follows subject from behind" |
|
||||
| Zoom | Lens-only (focal length) | "camera slowly zooms in on the figure" |
|
||||
| Rack focus | Lens-only (focal plane, snap) | "rack focus from the foreground bottle to the figure in the background" |
|
||||
| Pull focus | Lens-only (focal plane, gradual) | "camera shifts focus from foreground X to background Y" |
|
||||
| Static | Fixed | "static camera, no movement" |
|
||||
|
||||
### Focal-plane labels at start AND end of dynamic-DoF shots
|
||||
|
||||
Hunyuan benefits when both endpoints of focus-changing shots are stated. State where focus starts AND where it lands — don't leave one implicit.
|
||||
|
||||
Example: "shallow DoF; focus on the foreground bottle at start; focus pulls to the figure in the background by end."
|
||||
|
||||
Without both endpoints, Hunyuan often defaults to deep focus or holds on the wrong plane.
|
||||
|
||||
### Style Keywords
|
||||
|
||||
**Photorealistic / Cinematic**:
|
||||
@@ -65,6 +80,10 @@ When using image-to-video, the input image defines appearance. Your prompt shoul
|
||||
|
||||
**Bad I2V prompt**: "A beautiful woman in a red dress standing in a forest" — this repeats what the image already shows.
|
||||
|
||||
### Order motions temporally
|
||||
|
||||
Describe motion in temporal order; if multiple movements occur, separate them ("first the camera pans right, then tilts upward"). Hunyuan executes motion in the order it appears in the prompt — bundling two movements into one clause causes one of them to be dropped or blended.
|
||||
|
||||
## Example (T2V)
|
||||
|
||||
```
|
||||
|
||||
@@ -11,9 +11,13 @@ LTX-2 uses a clean, focused prompt structure:
|
||||
2. **Set the scene** — lighting, color palette, textures, atmosphere
|
||||
3. **Describe the action** — natural sequence flowing from beginning to end
|
||||
4. **Define the character(s)** — physical cues (age, hair, clothes), not abstract labels
|
||||
5. **Camera movement(s)** — specify how and when; describe what appears AFTER the movement
|
||||
5. **Camera movement(s)** — specify how and when; describe what appears AFTER the movement. (LTX honors the translation/rotation/lens distinction: `dolly` ≠ `zoom`, `pan` ≠ `truck`. Pick the right family — translation moves the rig, rotation pivots it, lens-only changes focal length or focal plane without moving the camera.)
|
||||
6. **Describe the audio** — ambient sound, music, speech, or singing
|
||||
|
||||
### Strict-Static-Shot rule
|
||||
|
||||
If you write "static camera," the shot must have NO movement, NO focus change, NO zoom. LTX takes "static" literally — adding any motion verb later in the prompt will either be ignored or will produce a glitch where the camera contradicts itself. Pick one: static, OR a single named movement.
|
||||
|
||||
## LTX-Specific Tips
|
||||
|
||||
### Post-Movement Description
|
||||
@@ -51,6 +55,7 @@ LTX organizes styles into three families:
|
||||
| Overloaded scenes | Many characters/actions reduces coherence |
|
||||
| Conflicting lighting descriptions | Pick one setup, commit to it |
|
||||
| Starting complex | Build up: simple prompt first, add layers |
|
||||
| Prompts over ~80 words | LTX-2 degrades past that. Pick the most important 5–6 elements. |
|
||||
|
||||
## LTX Technical Notes
|
||||
|
||||
|
||||
@@ -44,6 +44,8 @@ Seedance is unusually literal about camera language, multi-shot cuts, and quoted
|
||||
|
||||
## Multi-shot pattern
|
||||
|
||||
> **Repeat identity verbatim across every shot.** "the same character" / pronouns / "Aang again" do not work. Repeat the 3–6 disambiguating visual attributes verbatim in every shot block. Seedance treats each shot as if you said it cold.
|
||||
|
||||
Seedance honors explicit shot lists:
|
||||
|
||||
```
|
||||
@@ -63,6 +65,22 @@ Style: anamorphic lens, teal-orange cinematic grade, 35mm film grain.
|
||||
Audio: rising orchestral swell with low taiko pulse, wind, distant wingbeats.
|
||||
```
|
||||
|
||||
### Subject transition primitives in multi-shot
|
||||
|
||||
Seedance handles four distinct ways a subject can enter or exit a shot. Naming the primitive explicitly helps the model build the right transition between shots.
|
||||
|
||||
- **Subject revealing** (by camera move OR subject move) — the subject becomes visible mid-shot.
|
||||
Example: `Shot 2 (slow truck right): empty corridor at first; the camera trucks right to reveal Aang — bald, blue arrow tattoo, orange robes — pressed flat against the wall.`
|
||||
|
||||
- **Subject disappearing** — the subject leaves frame, by motion or occlusion.
|
||||
Example: `Shot 4 (static wide): Aang — bald, blue arrow tattoo, orange robes — sprints into the temple doorway and is swallowed by shadow; camera holds on the empty threshold.`
|
||||
|
||||
- **Subject switching** (rack focus / camera move) — focus or framing transfers from one subject to another.
|
||||
Example: `Shot 5 (close-up, rack focus): rack focus from Aang's glowing arrow tattoo in foreground to Sokka — dark hair, blue tunic, boomerang on back — emerging from the mist behind.`
|
||||
|
||||
- **Complex alternating focus** — focus oscillates between two subjects within one shot.
|
||||
Example: `Shot 7 (medium two-shot, alternating rack focus): focus on Aang — bald, blue arrow tattoo, orange robes — as he speaks, then pulls to Katara — long brown hair, blue water-tribe parka — as she answers, then back to Aang on the final beat.`
|
||||
|
||||
## Lip-sync pattern
|
||||
|
||||
```
|
||||
@@ -84,6 +102,7 @@ Sokka, half a step behind, replies: "Then we fight."
|
||||
| `generate_audio` | Keep `true` — sync audio is the moat. Strip in compose if unused. |
|
||||
| `model_variant` | `standard` for hero + multi-shot + camera-heavy. `fast` for b-roll, previews, latency-capped jobs. |
|
||||
| `seed` | Lock once a shot composition reads; iterate variants with the same seed. |
|
||||
| `prompt length` | 200–400 words for hero shots; 80–150 for inserts. Seedance is one of the few models that rewards long, structured 5-aspect prompts. |
|
||||
|
||||
## Iteration strategy
|
||||
|
||||
|
||||
@@ -3,6 +3,8 @@
|
||||
> Source: [OpenAI Sora 2 Cookbook](https://developers.openai.com/cookbook/examples/sora/sora2_prompting_guide)
|
||||
> For universal vocabulary, see: `skills/creative/video-gen-prompting.md`
|
||||
|
||||
**Word count:** Sora 2 plateaus around 100–250 words. Past 250, additional detail rarely improves output.
|
||||
|
||||
## Sora-Specific Prompt Template
|
||||
|
||||
Sora responds best to a structured format with prose + cinematography block + action beats:
|
||||
@@ -40,6 +42,9 @@ Sora uniquely responds to these production-level details that most models ignore
|
||||
| **Wardrobe** | "navy coat, sleeves rolled, suspenders loose" |
|
||||
| **Finishing** | "fine-grain overlay, mild halation, gate weave, soft vignette" |
|
||||
| **Shutter** | "180° shutter angle" |
|
||||
| **Playback speed** | "speed ramp from 1x to 0.25x mid-shot", "stop-motion staccato", "time-reversed exhale" |
|
||||
| **Lens distortion** | "fisheye barrel distortion at the edges", "subtle barrel curvature on straight lines" |
|
||||
| **Focus mode** | "rack focus from foreground bottle to background figure", "deep focus, FG to BG sharp" |
|
||||
|
||||
## What Sora Does Differently
|
||||
|
||||
|
||||
@@ -3,6 +3,8 @@
|
||||
> Source: [Vertex AI Video Gen Prompt Guide](https://cloud.google.com/vertex-ai/generative-ai/docs/video/video-gen-prompt-guide)
|
||||
> For universal vocabulary, see: `skills/creative/video-gen-prompting.md`
|
||||
|
||||
**Word count:** VEO 3.1 sweet spot is 100–250 words; longer prompts stop helping.
|
||||
|
||||
## VEO-Specific 14-Component Structure
|
||||
|
||||
VEO responds to the most comprehensive prompt structure of any model:
|
||||
@@ -29,16 +31,30 @@ VEO responds to the most comprehensive prompt structure of any model:
|
||||
- **Negative prompts**: Explicitly supported — "no text overlays, no watermarks, no lens flare"
|
||||
- **Editing vocabulary**: Understands "match cut", "jump cut", "montage", "split diopter" as prompt terms.
|
||||
|
||||
### Camera vocabulary VEO honors literally
|
||||
|
||||
VEO 3.1 distinguishes the three camera-motion families and treats their tokens as separate primitives. Mixing them up (e.g. asking for a "zoom" when you mean a "dolly") will produce the wrong move.
|
||||
|
||||
- **Translation (rig physically moves):** `dolly` (in/out along the lens axis), `truck` (left/right laterally), `pedestal` (up/down vertically)
|
||||
- **Rotation (rig stays put, camera rotates):** `pan` (yaw, left/right), `tilt` (pitch, up/down), `roll` (Dutch / Z-axis)
|
||||
- **Lens-only (rig and body don't move):** `zoom` (focal length change), `rack focus` / `pull focus` / `focus tracking` (focal-plane change)
|
||||
|
||||
dolly ≠ zoom; pan ≠ truck. VEO follows whichever token leads.
|
||||
|
||||
## VEO Lens Effects (Unique)
|
||||
|
||||
VEO specifically responds to optical effects most models ignore:
|
||||
|
||||
| Effect | Prompt Language |
|
||||
|--------|----------------|
|
||||
| **Rack focus** | "rack focus from foreground flower to background figure" |
|
||||
| **Rack focus** | "rack focus from foreground flower to background figure" (snap shift) |
|
||||
| **Pull focus** | "slow pull focus from the candle in the foreground to the doorway behind" (gradual, slower than rack) |
|
||||
| **Focus tracking** | "focus tracks the runner as she crosses frame; background stays soft" (focus follows a moving subject) |
|
||||
| **Dolly zoom (vertigo)** | "vertigo effect as character realizes the truth" |
|
||||
| **Fisheye** | "fisheye lens distortion, skatepark POV" |
|
||||
| **Lens flare** | "anamorphic lens flare from setting sun" |
|
||||
| **Anamorphic lens flare** | "anamorphic lens flare streaking horizontally from setting sun" |
|
||||
|
||||
These three focus modes (rack, pull, tracking) are different — VEO 3.1 honors the distinction per the paper.
|
||||
|
||||
## VEO Art Movement References
|
||||
|
||||
|
||||
@@ -57,6 +57,32 @@ For a **3-minute explainer video** (scale proportionally for other lengths):
|
||||
| 3 min | 3-5 | 8s | 22s | 100s | 30s | 15s |
|
||||
| 5 min | 5-8 | 10s | 30s | 180s | 50s | 20s |
|
||||
|
||||
## Anti-Subjective Rule
|
||||
|
||||
> Hooks, beats, and section descriptions in OpenMontage scripts must describe the **visual cause** of the emotion, not the emotion itself. The CMU/Harvard CHAI study showed that subjective phrasing varies wildly across annotators and across model interpretations — so it does not constrain pixels and it doesn't reliably guide downstream generation tools.
|
||||
>
|
||||
> | Avoid | Use instead |
|
||||
> |---|---|
|
||||
> | "epic reveal" | "wide aerial pull-back; subject silhouetted against rising sun" |
|
||||
> | "inspiring moment" | "low angle on the subject's face; light catches the edge of a tear" |
|
||||
> | "moody atmosphere" | "low-key key light, lifted shadows by 2 stops, fog volumetrics" |
|
||||
> | "powerful music swell" | "music drops out at 0:42, holds 1.5s of silence, returns with low taiko at half tempo" |
|
||||
>
|
||||
> The rule applies to script narration AND to the metadata fields scene-director consumes. For the universal vocabulary that names these visual primitives, see `skills/creative/video-gen-prompting.md`.
|
||||
|
||||
## Subject Transitions in the Script
|
||||
|
||||
When a script beat introduces a new subject, kills one off, or hands focus from one subject to another, **name the transition explicitly** so the scene-director doesn't have to infer it. The CMU/Harvard taxonomy uses four labels:
|
||||
|
||||
| Label | What it means |
|
||||
|---|---|
|
||||
| **revealing** | A new subject enters frame or is uncovered (door opens, camera pans to find them, fog clears). |
|
||||
| **disappearing** | An existing subject leaves frame or is removed (walks out, fades, eclipsed). |
|
||||
| **switching** | Focus jumps from subject A to subject B (cut, rack focus, camera whip). |
|
||||
| **complex-alternating** | Multiple subjects trade focus repeatedly within a beat (debate cross-cutting, ensemble action). |
|
||||
|
||||
Add 1-2 sentences in the beat describing the mechanism (cut, pan, reveal-by-light, etc.). This propagates into the scene_plan as a transition primitive.
|
||||
|
||||
## Hook Types
|
||||
|
||||
| Type | Pattern | Best For |
|
||||
@@ -114,6 +140,20 @@ Don't explain the answer. **Reconstruct the reasoning path** so the viewer feels
|
||||
**Progressive Revelation:** Never show the full picture at once. Build visuals layer by layer.
|
||||
Each layer arrives exactly when the narration references it.
|
||||
|
||||
## Camera Intent Per Beat
|
||||
|
||||
When writing a beat, attach one line of camera intent so the scene-director doesn't have to invent it from a blank slate. Use the universal vocabulary in `skills/creative/video-gen-prompting.md` (Subject / Subject Motion / Scene / Spatial Framing / Camera). One line is enough — the scene-director will expand it.
|
||||
|
||||
Example beat:
|
||||
|
||||
```
|
||||
[0:30] Concept 1 — atoms aren't tiny planets
|
||||
Narration: "We grew up imagining electrons as tiny planets orbiting the nucleus..."
|
||||
Camera intent: medium shot of stylized atom; slow rotation; deep focus.
|
||||
```
|
||||
|
||||
The camera-intent line is consumed verbatim by the scene-director's 5-aspect spec — keep it concrete, no mood adjectives.
|
||||
|
||||
## Pacing Rules
|
||||
|
||||
| Rule | Value | Source |
|
||||
|
||||
@@ -24,18 +24,50 @@ For model-specific tips, see the linked guides below.
|
||||
| **Kling 2.6** | [Kling Prompt Guide](https://fal.ai/learn/devs/kling-2-6-pro-prompt-guide) | 4-part structure. Supports `++emphasis++` syntax for key elements. |
|
||||
| **Wan 2.1 / CogVideoX** | Use this generic guide | No official prompt guide. Standard cinematographic vocabulary works well. |
|
||||
|
||||
## Order Matters
|
||||
|
||||
When listing multiple subjects or events:
|
||||
|
||||
- **Temporal order** when events unfold over time ("First X enters, then Y reacts").
|
||||
- **Prominence order** when temporal isn't relevant — humans before objects, largest/most-centered first, then secondary subjects.
|
||||
|
||||
## Self-Contained Prompt
|
||||
|
||||
> Write the prompt so that someone who has never seen the intended video could picture the subjects, scene, motion, and camera work from your text alone. If a reader could not picture it, a generation model will not render it.
|
||||
|
||||
## Universal Prompt Formula
|
||||
|
||||
All video generation models respond to this structure. Include what's relevant, omit what's not.
|
||||
Prior work (CMU/Harvard, "Building a Precise Video Language with Human-AI Oversight") shows VLMs reliably describe subject + scene but fail on motion, spatial, and camera. **Forcing prompts to fill all five slots is the highest-leverage change.**
|
||||
|
||||
The OpenMontage canonical 5-aspect skeleton:
|
||||
|
||||
```
|
||||
[Shot type/framing] + [Camera movement] + [Subject description] +
|
||||
[Action/motion in beats] + [Setting/environment] + [Lighting] +
|
||||
[Style/aesthetic] + [Audio/atmosphere]
|
||||
[Subject] type + key visual attributes + how to disambiguate when multiple
|
||||
[Subject Motion] actions in temporal order; subject↔object and subject↔subject interactions; group action
|
||||
[Scene] overlays (separately!) + POV + setting + time of day + scene dynamics
|
||||
[Spatial] shot size + position-in-frame + depth (FG/MG/BG) + camera-height-relative
|
||||
— and how those CHANGE during the clip
|
||||
[Camera] playback speed → lens distortion → height → angle → focus/DoF → steadiness → movement
|
||||
```
|
||||
|
||||
**Shorter prompts = more creative freedom. Longer prompts = more control.**
|
||||
|
||||
### Prompt Length by Model
|
||||
|
||||
Empirical sweet spots from the paper's Section 6 findings — different models reward different prompt densities:
|
||||
|
||||
| Model | Sweet Spot | Notes |
|
||||
|---|---|---|
|
||||
| Seedance 2.0 | 200–400 words for hero shots, 80–150 for inserts | Reward long, structured 5-aspect prompts |
|
||||
| Wan 2.2 | 200–400 words | Fine-tuned on long captions |
|
||||
| Sora 2 / VEO 3.1 | 100–250 words | Plateau past ~250 |
|
||||
| LTX-2 | ≤ 80 words | Degrades past that, keep tight |
|
||||
| Runway Gen-4 | ≤ 60 words | "Focus on motion, not appearance" |
|
||||
|
||||
### Overlays Are Not Scene Depth
|
||||
|
||||
> Overlays (titles, HUD, subtitles, watermarks, framing graphics) are NOT part of the scene's foreground/midground/background depth axis. List them separately with content and placement. Never say "overlay in the foreground."
|
||||
|
||||
---
|
||||
|
||||
## Camera Shot Types
|
||||
@@ -58,23 +90,56 @@ All video generation models respond to this structure. Include what's relevant,
|
||||
|
||||
## Camera Movements
|
||||
|
||||
| Movement | What It Does | Best For |
|
||||
|----------|-------------|----------|
|
||||
| **Static / fixed** | No movement | Dialogue, contemplation, stability |
|
||||
| **Pan** (left/right) | Rotates horizontally | Revealing a scene, following action |
|
||||
| **Tilt** (up/down) | Rotates vertically | Revealing height, slow reveal |
|
||||
| **Dolly in / out** | Physically moves toward/away | Building tension, emphasis |
|
||||
| **Truck** (left/right) | Moves sideways | Parallels subject movement |
|
||||
| **Pedestal** (up/down) | Moves vertically | Smooth elevation changes |
|
||||
| **Crane shot** | Sweeping vertical arcs | Epic reveals, transitions |
|
||||
| **Tracking / follow** | Follows subject | Action sequences, walk-and-talk |
|
||||
| **Arc shot** | Circles around subject | Dramatic emphasis, 360° reveal |
|
||||
| **Zoom** (in/out) | Lens focal length change | Quick emphasis (cheaper than dolly) |
|
||||
| **Whip pan** | Extremely fast pan (blurs) | Transitions, energy, surprise |
|
||||
| **Handheld / shaky cam** | Unstable, human feel | Documentary, urgency, realism |
|
||||
| **Aerial / drone** | High altitude, smooth | Landscapes, establishing shots |
|
||||
| **Slow push-in** | Gradual forward movement | Building intimacy or tension |
|
||||
| **Dolly zoom (vertigo)** | Dolly one way, zoom opposite | Disorientation, revelation |
|
||||
The paper shows current models confuse translation, rotation, and lens-only changes — group your prompts so the model can't conflate them:
|
||||
|
||||
| Group | Primitives | Rule |
|
||||
|---|---|---|
|
||||
| **Translation** (camera physically moves) | dolly in/out, truck left/right, pedestal up/down | "dolly forward toward subject" |
|
||||
| **Rotation** (camera pivots in place) | pan left/right, tilt up/down, roll CW/CCW | "pan right across the room" |
|
||||
| **Lens-only** (no camera move) | zoom in/out, rack focus, pull focus, focus tracking | "zoom in" ≠ "dolly in" |
|
||||
| **Hybrid / signature** | dolly zoom (vertigo), arc/orbit, crane, whip pan, tracking/follow, handheld | "vertigo" only at moments of revelation |
|
||||
| **Stillness states** | static (NO movement at all — strict), micro-shake, locked-off | "static" requires zero movement, focus change, or zoom |
|
||||
|
||||
> **dolly ≠ zoom.** dolly is camera translation; zoom is focal-length change. Models follow whichever token dominates. **pan ≠ truck.** pan rotates, truck translates laterally.
|
||||
|
||||
> **Static shot is strict.** A static shot has zero movement, zero focus change, zero zoom. If any of those occur, do NOT write "static camera" — pick the right movement primitive.
|
||||
|
||||
## Camera Height (relative to ground)
|
||||
|
||||
| Primitive | Example |
|
||||
|---|---|
|
||||
| Aerial-level | "drone-altitude wide of the city" |
|
||||
| Overhead-level | "rooftop height looking across the street" |
|
||||
| Eye-level | "framed at eye level" |
|
||||
| Hip-level | "hip-height tracking shot" |
|
||||
| Ground-level | "low to the ground, ankle height" |
|
||||
| Water-level | "skimming the water surface" |
|
||||
| Underwater | "submerged below the surface" |
|
||||
|
||||
## Camera Angle (relative to subject)
|
||||
|
||||
| Primitive | Definition |
|
||||
|---|---|
|
||||
| **Bird's-eye** | strict top-down. Not the same as aerial. |
|
||||
| High angle | looking down on subject |
|
||||
| Level angle | camera and subject at same height |
|
||||
| Low angle | looking up at subject |
|
||||
| Worm's-eye | looking straight up |
|
||||
| **Dutch angle (fixed)** | tilted horizon held steady |
|
||||
| **Dutch angle (rolling)** | horizon tilt changes during shot |
|
||||
|
||||
> **bird's-eye = strict top-down. aerial = altitude.** A drone shot at 45° looking down is a high angle from aerial height, NOT bird's-eye.
|
||||
|
||||
## Point of View (POV)
|
||||
|
||||
| POV | Example |
|
||||
|---|---|
|
||||
| First-person | "the camera follows the character's viewpoint as they walk" |
|
||||
| Drone | "aerial drone footage of city skyline" |
|
||||
| Over-the-shoulder | "OTS framing of the laptop screen" |
|
||||
| Top-down oblique | "top-down view of the chess board, tilted slightly" |
|
||||
| Dashcam | "vehicle dashcam framing of the road" |
|
||||
| Objective / Neutral | (default — use when no specific POV) |
|
||||
|
||||
## Lighting Vocabulary
|
||||
|
||||
@@ -100,14 +165,51 @@ All video generation models respond to this structure. Include what's relevant,
|
||||
|
||||
| Effect | Result |
|
||||
|--------|--------|
|
||||
| **Shallow depth of field** | Subject sharp, background bokeh |
|
||||
| **Deep focus** | Everything sharp, foreground to background |
|
||||
| **Wide-angle lens** (24-35mm) | Broader view, exaggerated perspective |
|
||||
| **Telephoto** (85mm+) | Compressed perspective, subject isolation |
|
||||
| **Anamorphic** | Stretched aspect, signature lens flares |
|
||||
| **Lens flare** | Streaks from bright light hitting lens |
|
||||
| **Rack focus** | Shift focus between subjects in-shot |
|
||||
| **Fisheye** | Ultra-wide, barrel distortion |
|
||||
|
||||
### Lens Distortion
|
||||
|
||||
The paper distinguishes two primitives that models honor as separate effects — they are NOT interchangeable:
|
||||
|
||||
| Primitive | Effect |
|
||||
|---|---|
|
||||
| **Fisheye** | extreme curvature, edges bent strongly outward |
|
||||
| **Barrel** | mild distortion, straight lines bow slightly outward |
|
||||
|
||||
### Focus / Depth of Field
|
||||
|
||||
| Primitive | Definition |
|
||||
|---|---|
|
||||
| Deep focus | everything sharp, FG to BG |
|
||||
| Shallow DoF | subject sharp, background bokeh |
|
||||
| Extremely shallow DoF | razor-thin focal plane |
|
||||
| Rack focus | shifts focus between two subjects mid-shot |
|
||||
| Pull focus | gradual focus shift (slower than rack) |
|
||||
| Focus tracking | focus follows a moving subject |
|
||||
|
||||
When DoF changes during a shot, label start AND end focal plane (FG/MG/BG/out-of-focus).
|
||||
|
||||
## Subject Transitions
|
||||
|
||||
When subjects enter, leave, or hand off focus, name the transition explicitly:
|
||||
|
||||
| Primitive | When |
|
||||
|---|---|
|
||||
| **Subject revealing** | a new subject enters frame (by subject movement OR camera movement) |
|
||||
| **Subject disappearing** | a subject exits frame |
|
||||
| **Subject switching** | focus shifts from one subject to another (often via rack focus or camera move) |
|
||||
| **Complex alternating** | subjects alternate focus multiple times |
|
||||
|
||||
Always name the cause: "by subject movement" or "by camera movement". This unlocks reveal-style camerawork in multi-shot prompts.
|
||||
|
||||
## Identity Anchoring for Multi-Shot Prompts
|
||||
|
||||
> Models lose character identity across cuts unless you re-state it. In every shot of a multi-shot prompt, repeat the same 3–6 disambiguating visual attributes for each named subject verbatim. Pronouns and "the same character" do not work.
|
||||
>
|
||||
> Example: "Aang — bald, blue arrow tattoo on forehead, orange-and-yellow robes — plants his staff. … Aang — bald, blue arrow tattoo on forehead, orange-and-yellow robes — turns to camera."
|
||||
|
||||
## Style & Aesthetic References
|
||||
|
||||
@@ -137,10 +239,23 @@ All video generation models respond to this structure. Include what's relevant,
|
||||
|
||||
## Temporal Effects
|
||||
|
||||
### Playback Speed
|
||||
|
||||
The paper defines six explicit playback-speed primitives. Use the right one — they're not synonymous:
|
||||
|
||||
| Primitive | Definition |
|
||||
|---|---|
|
||||
| Time-lapse | events significantly faster than real time (clouds racing) |
|
||||
| Fast-motion | slightly faster than real (1x–3x) |
|
||||
| Slow-motion | slower than real |
|
||||
| Stop-motion | frame-by-frame discrete movements |
|
||||
| Speed-ramp | mix of fast and slow within the same shot |
|
||||
| Time-reversed | plays in reverse |
|
||||
|
||||
### Other Temporal Devices
|
||||
|
||||
| Effect | Use |
|
||||
|--------|-----|
|
||||
| **Slow motion** | Emphasis, beauty, impact |
|
||||
| **Time-lapse** | Passage of time, processes |
|
||||
| **Freeze-frame** | Dramatic pause |
|
||||
| **Rapid cuts** | Energy, urgency |
|
||||
| **Continuous / long take** | Immersion, tension |
|
||||
@@ -160,6 +275,15 @@ Put dialogue in quotation marks: `Character says: "Hello world."`
|
||||
|
||||
## What to Avoid
|
||||
|
||||
> **Replace emotional adjectives with the visual cause of the emotion.**
|
||||
> - "sad character" → "tears on cheek, shoulders slumped, staring at empty chair"
|
||||
> - "cinematic mood" → "low-key Rembrandt key + 35mm anamorphic + crushed shadows, lifted-by-2-stops shadow detail"
|
||||
> - "epic" → "low-angle, 24mm wide, sun directly behind subject, lens flare on the rim"
|
||||
>
|
||||
> "Inspiring," "powerful," "moody," "epic" do not constrain pixels.
|
||||
|
||||
> **Static shot is strict.** A static shot has zero movement, zero focus change, zero zoom. If any of those occur, do NOT write "static camera" — pick the right movement primitive.
|
||||
|
||||
| Don't | Why | Do Instead |
|
||||
|-------|-----|-----------|
|
||||
| "Beautiful scene" | Too vague, no visual info | "Wet cobblestone street, warm streetlamp glow reflecting in puddles" |
|
||||
|
||||
+16
-3
@@ -6,6 +6,18 @@ After completing any pipeline stage's work — before checkpointing. You are the
|
||||
|
||||
Every stage gets reviewed. No exceptions. The review quality determines whether the final video is worth watching.
|
||||
|
||||
## Critique Quality (CHAI Rules)
|
||||
|
||||
> Findings ≠ critiques. A finding identifies a problem; a critique tells the next stage how to fix it. The CMU/Harvard CHAI study ("Building a Precise Video Language with Human-AI Oversight", arXiv 2604.21718v2) showed that critique quality, measured on three axes, directly governs downstream output quality. Apply all three to every reviewer pass.
|
||||
>
|
||||
> **Accurate.** Every finding must reference a concrete artifact field, line number, or visible asset frame. Forbid hallucinated criticism — if you cannot point to where the problem is, you are guessing.
|
||||
>
|
||||
> **Complete.** A reviewer pass that catches one mistake while missing a second is worse than scoring "needs another pass" and continuing. If you find one critical issue, scan for the rest of the same class before returning. Pattern-match: where else in this artifact could the same mistake be hiding?
|
||||
>
|
||||
> **Constructive.** Every "critical" finding MUST propose a concrete fix, not just identify the problem. "Caption is wrong" → "Caption says 'man on the right'; the man is on the left of the frame. Replace with 'the man on the left of the frame.'" If you cannot propose a fix, label the finding as "investigation" not "critical."
|
||||
>
|
||||
> Removing any of these three properties measurably hurts pipeline output. The reviewer is the choke point — be rigorous.
|
||||
|
||||
## Protocol
|
||||
|
||||
### Step 1: Load Review Context
|
||||
@@ -27,9 +39,10 @@ First, the non-negotiable check:
|
||||
For each `review_focus` item from the manifest:
|
||||
1. Evaluate the artifact against this specific criterion
|
||||
2. Assign a severity:
|
||||
- **critical** — Must fix before proceeding. The artifact is broken, incomplete, or dangerously wrong.
|
||||
- **suggestion** — Should fix. Improves quality significantly but doesn't block progress.
|
||||
- **nitpick** — Could fix. Minor polish that's nice-to-have.
|
||||
- **critical** — Must fix before proceeding. The artifact is broken, incomplete, or dangerously wrong. **Per CHAI rules, every critical finding MUST carry a `proposed_fix` (concrete replacement text, exact field value, or specific corrective action). A critical finding without a proposed fix is downgraded to `investigation`.**
|
||||
- **suggestion** — Should fix. Improves quality significantly but doesn't block progress. **Suggestions MUST carry a `proposed_change` describing how to improve.**
|
||||
- **nitpick** — Could fix. Minor polish that's nice-to-have. May stand alone without a proposed change.
|
||||
- **investigation** — A real concern but you cannot pinpoint the fix. Surface it for the next round; do not block on it.
|
||||
3. Write a specific, actionable finding (not vague)
|
||||
|
||||
**Good finding:** "Section 3 narration is 180 words for a 10-second window — that's 1080 wpm, impossible to speak. Cut to 25 words."
|
||||
|
||||
@@ -38,7 +38,8 @@ video_analyzer.execute({
|
||||
```
|
||||
|
||||
Read the resulting VideoAnalysisBrief. Before proceeding, present a summary to the
|
||||
user. This is NOT a raw dump. It's a conversational interpretation:
|
||||
user. This is NOT a raw dump. It's a conversational interpretation, and it MUST be
|
||||
structured by the 5 aspects so downstream stages can lift fields directly:
|
||||
|
||||
```
|
||||
"I've watched the video. Here's what I see:
|
||||
@@ -48,12 +49,22 @@ user. This is NOT a raw dump. It's a conversational interpretation:
|
||||
**Structure:** [X scenes over Y seconds, pacing style]
|
||||
**Motion:** [N of M scenes are motion clips / animated stills / static images.
|
||||
This video uses [AI-generated video clips / still images with pan-zoom / a mix].]
|
||||
|
||||
**5-aspect breakdown (per shot or per shot-group):**
|
||||
- Subject: [type, count, attributes; subject transitions across shots: revealing / disappearing / switching / complex-alternating; or N/A]
|
||||
- Subject Motion: [actions in temporal order; interactions; or N/A]
|
||||
- Scene: [overlays (text/graphics) listed separately; POV (drone/OTS/macro/etc.); setting; time of day; dynamics]
|
||||
- Spatial Framing: [shot size; subject position; depth; height-relative; how it changes]
|
||||
- Camera: [playback speed; lens; height; angle; focus/DoF; steadiness; movement]
|
||||
|
||||
**What makes it work:** [2-3 specific things — the hook technique, the pacing,
|
||||
the visual transitions, the narration style]
|
||||
|
||||
Now let me check what I can do with your current setup..."
|
||||
```
|
||||
|
||||
The 5-aspect block above is the **canonical form** that `proposal-director`, `script-director`, and `scene-director` will read. Do not collapse it back into prose — keep the labels.
|
||||
|
||||
**Motion classification is critical.** The VideoAnalysisBrief now includes per-scene
|
||||
`motion_type` ("motion_clip", "animated_still", "static_image") and `flow_variance`.
|
||||
Use this to determine the production approach:
|
||||
@@ -81,6 +92,22 @@ Update the brief's `content_analysis`, `style_profile`, and `replication_guidanc
|
||||
fields with your visual observations. This is where the analysis becomes truly
|
||||
comprehensive — the tools provide structure; your vision provides understanding.
|
||||
|
||||
### 5-Aspect Structured Output (MANDATORY)
|
||||
|
||||
The analyst's report MUST break down the reference video into the **five aspects** from the CMU/Harvard CHAI study (also the canonical structure used in `skills/creative/video-gen-prompting.md`). A narrative-only summary is no longer sufficient — downstream stages (proposal, script, scene-director) ingest the 5-aspect form directly without re-parsing prose.
|
||||
|
||||
**Decision-tree captioning policy.** For each detected shot, walk all five aspects in order:
|
||||
|
||||
> - **Subject:** type, attributes (count, age, role, costume, distinguishing features), multiple-subject disambiguation, transitions across shots (revealing / disappearing / switching / complex-alternating).
|
||||
> - **Subject Motion:** actions in temporal order; group/interaction patterns (parallel, sequential, reactive); locomotion vs gesture vs facial.
|
||||
> - **Scene:** **overlays separately** (text, lower thirds, graphics, watermark — call these out as their own layer, do not merge into setting) + POV (drone, aerial, OTS, macro, top-down, dashcam, FPV, handheld, locked-off) + setting + time of day + dynamics (weather, particles, crowd movement).
|
||||
> - **Spatial Framing:** shot size (ECU/CU/MS/WS/EWS), subject position in frame, depth (foreground/midground/background usage), height-relative (above/at/below subject) — and how each of these **changes** across the shot if the camera or subject moves.
|
||||
> - **Camera:** playback speed (real-time / slow-mo / time-lapse), lens distortion (anamorphic, fish-eye, tilt-shift), height (ground / eye / overhead), angle (high / low / Dutch), focus / DoF (rack focus, deep focus, shallow), steadiness (locked / handheld / gimbal), movement (push / pull / pan / tilt / dolly / truck / crane / orbit).
|
||||
>
|
||||
> **Mark any aspect explicitly as N/A** if it doesn't apply (e.g., "Subject: N/A — pure scenery shot," or "Scene overlays: N/A — no graphics"). **Silent omission is the most common analyst failure** and produces ambiguous downstream prompts.
|
||||
|
||||
See `skills/creative/video-gen-prompting.md` for primitive definitions and the canonical vocabulary used at every aspect.
|
||||
|
||||
### Step 2: Capability Audit
|
||||
|
||||
Run standard preflight:
|
||||
|
||||
@@ -85,9 +85,25 @@ Recommended metadata keys:
|
||||
- `tool_path_map`
|
||||
- `reusable_motifs`
|
||||
|
||||
### 5. Quality Gate
|
||||
### 5. 5-Aspect Scene-Plan Checklist
|
||||
|
||||
> Every scene must specify all five aspects, BUT the load shifts with the scene's `animation_mode`. Manim and other diagrammatic/programmatic scenes care most about **Subject** and **Spatial Framing** — Camera and Subject Motion in the cinematographic sense often map to N/A or to abstract equivalents. AI-video / `image_animation` / `anime_scene` scenes care about all five and behave like cinematic shots. Marking an aspect as N/A is allowed but must be explicit per scene; silent omission is forbidden.
|
||||
>
|
||||
> 1. **Subject** — type + key visual attributes; for Manim, the equation/object/graph being foregrounded; for `anime_scene`, the character or environment in focus.
|
||||
> 2. **Subject Motion** — for Manim, the order of `Create`/`Transform`/`FadeIn` and what each animation conveys; for AI-video, the actions and interactions in temporal order.
|
||||
> 3. **Scene** — overlays (separately!) + POV + setting + time of day + scene dynamics. For Manim, "setting" is the canvas background + axis style; for `anime_scene`, the environment + lighting gradient.
|
||||
> 4. **Spatial Framing** — shot size + position-in-frame + depth (FG/MG/BG) + camera-height-relative; and how those CHANGE. Manim cares about layout grid + element positions; AI-video cares about full cinematographic framing.
|
||||
> 5. **Camera** — playback speed → lens distortion → height → angle → focus/DoF → steadiness → movement. For Manim and pure motion-graphics, default to N/A unless using a virtual camera move (`MoveCamera`, `self.frame`). For `anime_scene` and AI-video, specify fully.
|
||||
>
|
||||
> Tie this back to `animation_mode` in scene metadata: a Manim scene that lists Camera fully is over-specified; an AI-video scene that omits Camera is under-specified. See `skills/creative/video-gen-prompting.md` for the primitive vocabulary.
|
||||
|
||||
> **Overlays callout.** Overlays (titles, subtitles, HUD, watermarks, framing graphics, lower-thirds, `hero_title`, `section_title`, `provider_chip`) are NOT part of the scene's foreground/midground/background depth axis. List them separately in scene metadata (`overlays: [...]`) with content and placement. Never describe an overlay as "in the foreground" — that confuses both downstream tools and any video-understanding model that re-analyzes the output.
|
||||
|
||||
### 6. Quality Gate
|
||||
|
||||
- every scene has a clear timing intent,
|
||||
- the 5-aspect checklist is satisfied for the scene's `animation_mode` (with explicit N/A where appropriate),
|
||||
- overlays live under `overlays:`, never inside the framing description,
|
||||
- the transition system is limited and meaningful,
|
||||
- the tool path is explicit,
|
||||
- the sequence feels like one designed system.
|
||||
|
||||
@@ -0,0 +1,57 @@
|
||||
# Asset Director - Character Animation Pipeline
|
||||
|
||||
## Goal
|
||||
|
||||
Produce `asset_manifest` with character parts, backgrounds, props, audio, music,
|
||||
and preview artifacts.
|
||||
|
||||
## Layer 3 Gate
|
||||
|
||||
Before authoring or generating animation assets, read the relevant Layer 3 skills:
|
||||
|
||||
- `character-rigging`
|
||||
- `svg-character-animation`
|
||||
- `pose-library-design`
|
||||
- `canvas-procedural-animation` when p5/canvas effects are used
|
||||
- `character-animation-qa` before review
|
||||
- `gsap-core`, `gsap-timeline`, and `gsap-react` for GSAP/Remotion work
|
||||
- `remotion` and `remotion-best-practices` for Remotion render work
|
||||
- `hyperframes` and `hyperframes-cli` for HyperFrames work
|
||||
|
||||
Before image/TTS/music generation, read the tool's `agent_skills` from the
|
||||
registry.
|
||||
|
||||
## Asset Organization
|
||||
|
||||
Write character assets under:
|
||||
|
||||
```text
|
||||
projects/<project-name>/assets/characters/<character-id>/
|
||||
```
|
||||
|
||||
Use subfolders:
|
||||
|
||||
```text
|
||||
parts/
|
||||
poses/
|
||||
previews/
|
||||
```
|
||||
|
||||
Generated backgrounds go under:
|
||||
|
||||
```text
|
||||
projects/<project-name>/assets/backgrounds/
|
||||
```
|
||||
|
||||
## Process
|
||||
|
||||
1. Produce or source only the parts required by `rig_plan`.
|
||||
2. Keep each moving part separate.
|
||||
3. Preserve transparent backgrounds for parts.
|
||||
4. Record prompts, seeds, providers, and model names.
|
||||
5. Build a small preview before full asset expansion.
|
||||
|
||||
## Quality Bar
|
||||
|
||||
All parts referenced by `rig_plan` must exist before compose. Missing parts are a
|
||||
blocker unless the action timeline removes the action requiring them.
|
||||
@@ -0,0 +1,32 @@
|
||||
# Character Design Director - Character Animation Pipeline
|
||||
|
||||
## Goal
|
||||
|
||||
Produce `character_design`: a small cast with clear silhouettes, roles,
|
||||
emotions, actions, and style anchors.
|
||||
|
||||
## Process
|
||||
|
||||
1. List every character with `id`, role, body type, and style.
|
||||
2. Identify the minimum emotional range needed by the story.
|
||||
3. Identify the minimum action list needed by the story.
|
||||
4. Decide required views: front, 3/4, side, back. Keep MVPs to one or two views.
|
||||
5. Note props attached to characters, such as scarf, feather, bag, glasses.
|
||||
|
||||
## Constraints
|
||||
|
||||
- One or two characters is the MVP sweet spot.
|
||||
- Animal characters need species-specific parts and action cycles.
|
||||
- More views multiply asset and pose requirements.
|
||||
- Do not invent more poses than the approved duration can use.
|
||||
|
||||
## Tool Use
|
||||
|
||||
Use `character_spec_generator` for structured drafts. Use `image_selector` only
|
||||
after the visual style and character sheet requirements are explicit. Before
|
||||
using image generation, read the tool's Layer 3 skills from the registry.
|
||||
|
||||
## Quality Bar
|
||||
|
||||
A character design is ready only when an animator or tool can infer what parts,
|
||||
expressions, and actions must exist.
|
||||
@@ -0,0 +1,49 @@
|
||||
# Compose Director - Character Animation Pipeline
|
||||
|
||||
## Goal
|
||||
|
||||
Render the approved character animation and prove it was reviewed.
|
||||
|
||||
## Runtime Routing
|
||||
|
||||
First read `edit_decisions.render_runtime`. It must match the runtime locked in
|
||||
proposal unless a `render_runtime_selection` decision explicitly changed it.
|
||||
|
||||
- `remotion`: stage assets into `remotion-composer/public`, build composition
|
||||
JSON, render via `video_compose`.
|
||||
- `hyperframes`: materialize a HyperFrames workspace and let `video_compose`
|
||||
delegate to `hyperframes_compose`. `hyperframes lint` and `validate` must pass.
|
||||
- `ffmpeg`: only for post-processing or simple video assembly; not enough for
|
||||
character acting by itself.
|
||||
|
||||
## Review Workflow
|
||||
|
||||
1. Run `character_rig_renderer` to produce or refresh the HyperFrames package.
|
||||
The browser preview is a QA/debug artifact only, not the render path.
|
||||
2. Verify the renderer emitted a HyperFrames `workspace_path`, composition HTML,
|
||||
`asset_manifest`, and `edit_decisions.render_runtime: "hyperframes"` handoff.
|
||||
3. Run `character_animation_reviewer` against rig, poses, timeline, and preview.
|
||||
4. Render final video through `video_compose` using the renderer handoff or the
|
||||
approved Remotion/HyperFrames package. The deliverable path is
|
||||
`projects/<project-name>/renders/final.mp4`, matching the standard
|
||||
OpenMontage project convention.
|
||||
5. Run standard `final_review`: ffprobe, frame sampling, visual spotcheck, audio
|
||||
spotcheck, promise preservation.
|
||||
|
||||
## Browser QA
|
||||
|
||||
When Playwright is available:
|
||||
|
||||
- open the preview,
|
||||
- capture opening/middle/end frames,
|
||||
- check for console errors,
|
||||
- verify characters are visible,
|
||||
- compare frame deltas to ensure motion exists.
|
||||
|
||||
When Playwright is unavailable, use static artifact checks and FFmpeg frame
|
||||
sampling, and report the reduced confidence.
|
||||
|
||||
## Quality Bar
|
||||
|
||||
Do not present the output as complete when `character_qa_report.status` is
|
||||
`revise` or `fail`.
|
||||
@@ -0,0 +1,33 @@
|
||||
# Edit Director - Character Animation Pipeline
|
||||
|
||||
## Goal
|
||||
|
||||
Produce `edit_decisions` and `action_timeline`.
|
||||
|
||||
## Process
|
||||
|
||||
1. Carry `render_runtime` forward from the approved proposal.
|
||||
2. Convert scene beats into timed character actions.
|
||||
3. Add anticipation, hold, action, and follow-through where appropriate.
|
||||
4. Align mouth/gesture beats to dialogue or music.
|
||||
5. Keep action density readable.
|
||||
|
||||
## Timing Pattern
|
||||
|
||||
Most acting beats need:
|
||||
|
||||
```text
|
||||
anticipation -> action -> hold/reaction -> settle
|
||||
```
|
||||
|
||||
Do not animate everything continuously. Holds are part of acting.
|
||||
|
||||
## Tool Use
|
||||
|
||||
Use `action_timeline_compiler` for a first pass, then revise the timeline if the
|
||||
acting or rhythm is weak.
|
||||
|
||||
## Quality Bar
|
||||
|
||||
Every scene has timed actions. Every action maps to a pose, action cycle, or
|
||||
procedural effect that the renderer can understand.
|
||||
@@ -0,0 +1,48 @@
|
||||
# Executive Producer - Character Animation Pipeline
|
||||
|
||||
## When To Use
|
||||
|
||||
Use this pipeline when the requested deliverable depends on reusable animated
|
||||
characters: cartoon shorts, mascot explainers, music-led character scenes,
|
||||
dialogue between simple characters, or reference-inspired local animation.
|
||||
|
||||
Do not use this pipeline for one-off motion graphics with no acting. Route those
|
||||
to `animation`. Do not use it for avatar presenter lip-sync. Route that to
|
||||
`avatar-spokesperson`.
|
||||
|
||||
## Contract
|
||||
|
||||
The pipeline produces local, deterministic character animation. It does not
|
||||
silently substitute still-image motion for acting. If the character motion cannot
|
||||
be built with the available rigs, assets, or runtime, surface a blocker.
|
||||
|
||||
## Stage Order
|
||||
|
||||
1. `research` - understand reference, technique, and feasibility.
|
||||
2. `proposal` - present concepts, runtime options, cost, music plan, sample plan.
|
||||
3. `script` - write action-friendly beats and dialogue/narration.
|
||||
4. `character_design` - define characters, silhouettes, emotions, actions.
|
||||
5. `rig_plan` - define parts, pivots, layers, constraints, poses.
|
||||
6. `scene_plan` - map story beats to character scenes.
|
||||
7. `assets` - produce or source character parts, backgrounds, props, audio.
|
||||
8. `edit` - compile timed action timeline.
|
||||
9. `compose` - render through the approved runtime and run QA.
|
||||
10. `publish` - package the final output.
|
||||
|
||||
## Governance Rules
|
||||
|
||||
- Run registry preflight before proposal.
|
||||
- If both Remotion and HyperFrames are available, present both before locking
|
||||
`render_runtime`.
|
||||
- Produce a 10-15 second sample before full asset generation.
|
||||
- Character differences belong in rig data, not one-off code paths.
|
||||
- Every generated or runtime-authored asset must list Layer 3 skills read.
|
||||
- Use `character_animation_reviewer` plus final `final_review` before delivery.
|
||||
|
||||
## Send-Back Triggers
|
||||
|
||||
- `character_design` lacks required actions or emotional range.
|
||||
- `rig_plan` lacks pivots for moving parts.
|
||||
- `pose_library` has no readable acting poses.
|
||||
- `action_timeline` has actions that cannot be rendered by the rig.
|
||||
- Compose used a runtime not approved in proposal.
|
||||
@@ -0,0 +1,58 @@
|
||||
# Proposal Director - Character Animation Pipeline
|
||||
|
||||
## Goal
|
||||
|
||||
Present character-animation concepts that are honest about local rigged motion,
|
||||
reuse, cost, and runtime choice.
|
||||
|
||||
## Required Proposal Elements
|
||||
|
||||
Each option must include:
|
||||
|
||||
- characters and roles,
|
||||
- visual style,
|
||||
- action complexity,
|
||||
- rig reuse strategy,
|
||||
- sample plan,
|
||||
- audio architecture,
|
||||
- music plan,
|
||||
- render runtime options,
|
||||
- cost estimate,
|
||||
- honest limitation note.
|
||||
|
||||
## Runtime Selection
|
||||
|
||||
Read `skills/meta/animation-runtime-selector.md` before recommending a runtime.
|
||||
|
||||
When both Remotion and HyperFrames are available:
|
||||
|
||||
- Remotion: best when the final composition needs deterministic React-rendered
|
||||
video, captions, audio, scene JSON, and final MP4 governance.
|
||||
- HyperFrames: best when the character scene is HTML/SVG/GSAP-heavy and benefits
|
||||
from web-native authoring, lint, validate, and registry blocks.
|
||||
- FFmpeg: post-processing only. Do not pick FFmpeg as the primary runtime for
|
||||
character acting.
|
||||
|
||||
Wait for user approval before locking `render_runtime`.
|
||||
|
||||
## Sample-First Rule
|
||||
|
||||
Before full production, propose a 10-15 second sample containing:
|
||||
|
||||
- one main character,
|
||||
- one expression change,
|
||||
- one body action,
|
||||
- one camera/background treatment,
|
||||
- one audio/music cue if relevant.
|
||||
|
||||
Do not batch-generate all assets until this sample is approved.
|
||||
|
||||
## Cost Honesty
|
||||
|
||||
Local rigging is cheap at render time but expensive in authoring complexity.
|
||||
Report the difference:
|
||||
|
||||
- asset generation cost,
|
||||
- TTS/music cost,
|
||||
- local render cost,
|
||||
- manual complexity risk.
|
||||
@@ -0,0 +1,26 @@
|
||||
# Publish Director - Character Animation Pipeline
|
||||
|
||||
## Goal
|
||||
|
||||
Package the final character-animation deliverable with honest metadata and a
|
||||
strong character-forward thumbnail concept.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Mention the actual visual treatment: local rigged character animation,
|
||||
procedural effects, Remotion/HyperFrames render, or mixed.
|
||||
- Pick a poster frame where the main character's emotion is readable.
|
||||
- If the output is a sample, label it as a sample.
|
||||
- If the final is inspired by a reference, describe the inspiration without
|
||||
claiming duplication.
|
||||
|
||||
## Output
|
||||
|
||||
Produce `publish_log` with:
|
||||
|
||||
- final video path,
|
||||
- thumbnail/poster-frame notes,
|
||||
- title ideas,
|
||||
- description,
|
||||
- platform-specific export notes,
|
||||
- limitations or follow-up recommendations.
|
||||
@@ -0,0 +1,45 @@
|
||||
# Research Director - Character Animation Pipeline
|
||||
|
||||
## Goal
|
||||
|
||||
Ground the character-animation plan in real references and current technique.
|
||||
For reference videos, start from `video_analysis_brief`: content, pacing, motion
|
||||
classification, keyframes, color, and production complexity.
|
||||
|
||||
## Process
|
||||
|
||||
1. Identify what the reference actually uses:
|
||||
- rigged local animation,
|
||||
- frame-by-frame traditional animation,
|
||||
- video generation,
|
||||
- still-image motion,
|
||||
- mixed techniques.
|
||||
2. Research 3-5 relevant examples or techniques.
|
||||
3. Separate what the pipeline can reproduce locally from what requires manual
|
||||
illustration, video generation, or a larger asset library.
|
||||
4. Record reusable animation primitives:
|
||||
- walk cycle,
|
||||
- blink,
|
||||
- head turn,
|
||||
- reach,
|
||||
- wing flap,
|
||||
- squash/stretch,
|
||||
- camera pan/parallax,
|
||||
- particles/weather.
|
||||
|
||||
## Output Guidance
|
||||
|
||||
The `research_brief` should include:
|
||||
|
||||
- `character_animation_fit`: high/medium/low,
|
||||
- `reference_motion_type`,
|
||||
- `required_character_actions`,
|
||||
- `rig_complexity`,
|
||||
- `manual_asset_risks`,
|
||||
- `local_runtime_candidates`.
|
||||
|
||||
## Quality Bar
|
||||
|
||||
Be explicit when a reference is hand-drawn or frame-by-frame. The user can still
|
||||
choose an inspired local rigged style, but the proposal must not imply exact
|
||||
traditional-animation quality from an automatic rig.
|
||||
@@ -0,0 +1,40 @@
|
||||
# Rig Plan Director - Character Animation Pipeline
|
||||
|
||||
## Goal
|
||||
|
||||
Produce `rig_plan` and `pose_library` from `character_design`.
|
||||
|
||||
## Process
|
||||
|
||||
1. Convert each character into rig parts:
|
||||
- body,
|
||||
- head,
|
||||
- eyes/pupils,
|
||||
- brows,
|
||||
- mouth shapes,
|
||||
- limbs/wings,
|
||||
- tail/accessories,
|
||||
- props.
|
||||
2. Define pivots for every moving part.
|
||||
3. Define layer order.
|
||||
4. Define constraints so limbs do not rotate into impossible positions.
|
||||
5. Define named poses for the approved scenes.
|
||||
6. Define action cycles only when reused at least twice or central to the story.
|
||||
|
||||
## Runtime Pattern
|
||||
|
||||
Character differences are data. The renderer should not need one-off code for a
|
||||
mouse versus a bird. A bird may have `wing_left`; a mouse may have `tail`, but
|
||||
both feed the same pose interpolation and timeline compiler.
|
||||
|
||||
## Quality Checks
|
||||
|
||||
- Every moving part has a pivot.
|
||||
- Every required action has poses or a procedural strategy.
|
||||
- Every pose names the changed parts.
|
||||
- Risky actions are called out, not hidden.
|
||||
|
||||
## Tool Use
|
||||
|
||||
Use `svg_rig_builder` to draft rig data and `pose_library_builder` to draft the
|
||||
initial pose library. The agent may revise their output before checkpointing.
|
||||
@@ -0,0 +1,37 @@
|
||||
# Scene Director - Character Animation Pipeline
|
||||
|
||||
## Goal
|
||||
|
||||
Produce a `scene_plan` where each scene is feasible for rigged character
|
||||
animation.
|
||||
|
||||
## Scene Planning Fields
|
||||
|
||||
For each scene, include:
|
||||
|
||||
- character IDs,
|
||||
- emotional beat,
|
||||
- action sequence,
|
||||
- camera/framing,
|
||||
- background,
|
||||
- props,
|
||||
- effects,
|
||||
- required assets,
|
||||
- transition notes.
|
||||
|
||||
Use `type: "character_scene"` for rigged character acting scenes. Store
|
||||
character-specific detail in `character_actions`; do not put per-scene acting
|
||||
data in arbitrary metadata because the shared `scene_plan` schema rejects
|
||||
unknown per-scene fields.
|
||||
|
||||
## Complexity Budget
|
||||
|
||||
Prefer fewer, stronger shots:
|
||||
|
||||
- one establish,
|
||||
- one action beat,
|
||||
- one reaction beat,
|
||||
- one resolution beat.
|
||||
|
||||
Avoid scenes that require many unique views or complex physical contact unless
|
||||
the user approved that complexity.
|
||||
@@ -0,0 +1,37 @@
|
||||
# Script Director - Character Animation Pipeline
|
||||
|
||||
## Goal
|
||||
|
||||
Write scripts as performable animation beats, not just narration.
|
||||
|
||||
## Process
|
||||
|
||||
1. Lock audio architecture:
|
||||
- music-only,
|
||||
- narrator,
|
||||
- character dialogue,
|
||||
- narrator plus character sounds/dialogue.
|
||||
2. Break the story into beats that can be acted with poses.
|
||||
3. For each beat, state what changes visually:
|
||||
- emotion,
|
||||
- gaze,
|
||||
- body pose,
|
||||
- prop interaction,
|
||||
- camera,
|
||||
- environment.
|
||||
|
||||
## Writing Rules
|
||||
|
||||
- Prefer short visual beats with readable holds.
|
||||
- Avoid action that needs many unique hand-drawn poses unless approved.
|
||||
- Dialogue should be short enough for mouth-shape approximation.
|
||||
- Silent/music-led scenes need stronger physical acting notes.
|
||||
|
||||
## Output Notes
|
||||
|
||||
In the `script` artifact metadata, include:
|
||||
|
||||
- `audio_architecture`,
|
||||
- `character_beats`,
|
||||
- `required_emotions`,
|
||||
- `required_actions`.
|
||||
@@ -101,6 +101,22 @@ Recommended metadata keys:
|
||||
- `generated_support_assets`
|
||||
- `rights_notes`
|
||||
|
||||
### Pre/Post Self-Review for Generation Prompts
|
||||
|
||||
> Before sending a prompt to any image or video generation tool, run a three-step self-review modeled on the CHAI oversight loop ("Building a Precise Video Language with Human-AI Oversight", arXiv 2604.21718v2). Cost is small (no extra tool calls); benefit is large (avoids wasted generations). For cinematic, this matters most for **hero-frame prompts** — one bad hero frame ruins the piece, and hero frames are the most expensive shots to regenerate.
|
||||
>
|
||||
> **Step 1 — Pre-caption pass.** Write the prompt the way you'd write it today. Do not over-edit; aim for a complete first draft.
|
||||
>
|
||||
> **Step 2 — Critique pass.** Score the draft against the 5-aspect checklist (Subject / Subject Motion / Scene / Spatial Framing / Camera). For each aspect:
|
||||
> - Is it specified? If not, is the omission deliberate (e.g., "no subject — scenery shot") or accidental?
|
||||
> - Are confusable terms disambiguated? (dolly vs zoom, pan vs truck, bird's-eye vs aerial, fisheye vs barrel, full shot vs close-up)
|
||||
> - Are emotional adjectives ("epic", "moody", "cinematic") replaced with their visual causes (low-key lighting, slow push-in, anamorphic flare, deep shadows)?
|
||||
> - For multi-shot prompts and identity-anchored hero frames: is identity anchored verbatim across shots?
|
||||
>
|
||||
> **Step 3 — Post-caption pass.** Rewrite filling the missing aspects, fixing confusable terms, and replacing subjective language. The post-caption is what gets sent to the generation tool.
|
||||
>
|
||||
> Log the (pre, critique, post) triplet in the asset metadata for traceability. This mirrors the CHAI workflow and creates a record the reviewer can audit.
|
||||
|
||||
### 5. Quality Gate
|
||||
|
||||
- source and support assets are clearly distinguished,
|
||||
|
||||
@@ -49,11 +49,26 @@ Recommended metadata keys:
|
||||
- `title_card_rules`
|
||||
- `support_insert_rules`
|
||||
|
||||
### 5. Quality Gate
|
||||
### 5. 5-Aspect Scene-Plan Checklist
|
||||
|
||||
> Every scene beat — and especially every hero frame — must specify all five aspects. Cinematic relies on a small number of memorable frames; vague hero-frame specs are the single most common failure mode and produce unpredictable model output. Marking an aspect as N/A is allowed but must be explicit (e.g., "no subject — establishing scenery shot"). Silent omission is forbidden.
|
||||
>
|
||||
> 1. **Subject** — type + key visual attributes; if multiple, how to disambiguate. For hero frames, identity must be anchored verbatim across shots.
|
||||
> 2. **Subject Motion** — actions in temporal order; subject↔object / subject↔subject interactions.
|
||||
> 3. **Scene** — overlays (separately!) + POV + setting + time of day + scene dynamics.
|
||||
> 4. **Spatial Framing** — shot size + position-in-frame + depth (FG/MG/BG) + camera-height-relative; and how those CHANGE across the beat.
|
||||
> 5. **Camera** — playback speed → lens distortion → height → angle → focus/DoF → steadiness → movement.
|
||||
>
|
||||
> See `skills/creative/video-gen-prompting.md` for the primitive vocabulary.
|
||||
|
||||
> **Overlays callout.** Overlays (titles, subtitles, HUD, watermarks, framing graphics, lower-thirds, name plates, end-tag cards) are NOT part of the scene's foreground/midground/background depth axis. List them separately in scene metadata (`overlays: [...]`) with content and placement. Never describe an overlay as "in the foreground" — that confuses both downstream tools and any video-understanding model that re-analyzes the output.
|
||||
|
||||
### 6. Quality Gate
|
||||
|
||||
- every beat has a scene treatment,
|
||||
- hero frames are identifiable,
|
||||
- hero frames are identifiable AND fully specified across all 5 aspects,
|
||||
- support inserts are justified,
|
||||
- overlays are recorded under `overlays:`, never inside the depth/framing description,
|
||||
- the visual language stays consistent across the piece.
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
@@ -178,6 +178,22 @@ Assemble all generated assets into the manifest:
|
||||
}
|
||||
```
|
||||
|
||||
### Pre/Post Self-Review for Generation Prompts
|
||||
|
||||
> Before sending a prompt to any generation tool — `image_selector`, `diagram_gen`, `video_selector`, even `code_snippet` styling prompts — run a three-step self-review modeled on the CHAI oversight loop ("Building a Precise Video Language with Human-AI Oversight", arXiv 2604.21718v2). Cost is small (no extra tool calls); benefit is large (avoids wasted generations). This applies just as strongly to `diagram_gen` Mermaid prompts and `image_selector` illustration prompts as it does to video generation — bad explainer visuals fail in the same way: missing subject, missing framing, vague "make it look educational."
|
||||
>
|
||||
> **Step 1 — Pre-caption pass.** Write the prompt the way you'd write it today. Do not over-edit; aim for a complete first draft.
|
||||
>
|
||||
> **Step 2 — Critique pass.** Score the draft against the 5-aspect checklist (Subject / Subject Motion / Scene / Spatial Framing / Camera). For each aspect:
|
||||
> - Is it specified? If not, is the omission deliberate (e.g., "Camera N/A — Remotion native scene", "no subject motion — static diagram") or accidental?
|
||||
> - Are confusable terms disambiguated? (dolly vs zoom, pan vs truck, bird's-eye vs aerial, fisheye vs barrel, full shot vs close-up; for diagrams: flowchart vs sequence vs state diagram, top-down vs left-right)
|
||||
> - Are emotional adjectives ("clean", "professional", "modern") replaced with their visual causes (sans-serif typography, generous whitespace, monochromatic palette with one accent)?
|
||||
> - For multi-shot prompts: is identity anchored verbatim across shots? For `image_selector` prompts that recur (a character or world appearing in multiple scenes), are consistency anchors specified verbatim?
|
||||
>
|
||||
> **Step 3 — Post-caption pass.** Rewrite filling the missing aspects, fixing confusable terms, and replacing subjective language. The post-caption is what gets sent to the generation tool.
|
||||
>
|
||||
> Log the (pre, critique, post) triplet in the asset metadata for traceability. This mirrors the CHAI workflow and creates a record the reviewer can audit.
|
||||
|
||||
### Step 7: Verify All Assets
|
||||
|
||||
**Existence check:**
|
||||
|
||||
@@ -156,6 +156,20 @@ If the video includes narration, the script **must** be written to fit the video
|
||||
- If narration exceeds video by >1s, either trim the script and regenerate, or extend the video's closing scene.
|
||||
- Always run `composition_validator` before rendering to catch mismatches automatically.
|
||||
|
||||
### Step 4c: 5-Aspect Scene-Plan Checklist
|
||||
|
||||
> Every scene must specify all five aspects. For diagram, chart, and Remotion-native scenes, "Subject" can map to a foregrounded data element and "Camera" can be marked N/A — but only EXPLICITLY (e.g., `"camera": "N/A — Remotion native scene, no virtual camera"`). Silent omission is the most common failure mode and produces unpredictable model output, brittle prompts, and reviewer churn.
|
||||
>
|
||||
> 1. **Subject** — type + key visual attributes; if multiple, how to disambiguate. For diagram/chart scenes, this is the foregrounded data element (the node, the bar, the KPI being highlighted). For generated images, it's the person/object/concept being illustrated.
|
||||
> 2. **Subject Motion** — actions in temporal order; for animated diagrams, the order in which nodes/edges/values appear or change.
|
||||
> 3. **Scene** — overlays (separately!) + POV + setting + time of day + scene dynamics. For Remotion scenes, "setting" maps to background treatment + theme.
|
||||
> 4. **Spatial Framing** — shot size + position-in-frame + depth (FG/MG/BG) + camera-height-relative; and how those CHANGE. For static Remotion scenes, document the layout grid + which element occupies the visual center.
|
||||
> 5. **Camera** — playback speed → lens distortion → height → angle → focus/DoF → steadiness → movement. Mark N/A for native-Remotion scenes; specify fully for `generated`/`broll`/`image_animation` scenes.
|
||||
>
|
||||
> See `skills/creative/video-gen-prompting.md` for the primitive vocabulary.
|
||||
|
||||
> **Overlays callout.** Overlays (titles, subtitles, HUD, watermarks, framing graphics, lower-thirds, section_title bars, stat_reveal chips, hero_title overlays, provider chips) are NOT part of the scene's foreground/midground/background depth axis. List them separately in scene metadata (`overlays: [...]`) with content and placement. Never describe an overlay as "in the foreground" — that confuses both downstream tools and any video-understanding model that re-analyzes the output.
|
||||
|
||||
### Step 5: Validate Against Playbook
|
||||
|
||||
The style playbook constrains your visual choices:
|
||||
|
||||
@@ -0,0 +1,276 @@
|
||||
"""Contract tests for the local character-animation pipeline."""
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
|
||||
from lib.pipeline_loader import get_required_tools, get_stage_order, load_pipeline
|
||||
from schemas.artifacts import ARTIFACT_NAMES, validate_artifact
|
||||
from tools.character.character_animation import (
|
||||
ActionTimelineCompiler,
|
||||
CharacterAnimationReviewer,
|
||||
CharacterRigRenderer,
|
||||
CharacterSpecGenerator,
|
||||
PoseLibraryBuilder,
|
||||
SvgRigBuilder,
|
||||
)
|
||||
from tools.tool_registry import registry
|
||||
from tools.video.hyperframes_compose import HyperFramesCompose
|
||||
from tools.video.video_compose import VideoCompose
|
||||
|
||||
|
||||
def test_character_animation_manifest_contract():
|
||||
manifest = load_pipeline("character-animation")
|
||||
|
||||
assert manifest["name"] == "character-animation"
|
||||
assert get_stage_order(manifest) == [
|
||||
"research",
|
||||
"proposal",
|
||||
"script",
|
||||
"character_design",
|
||||
"rig_plan",
|
||||
"scene_plan",
|
||||
"assets",
|
||||
"edit",
|
||||
"compose",
|
||||
"publish",
|
||||
]
|
||||
assert {
|
||||
"character_spec_generator",
|
||||
"svg_rig_builder",
|
||||
"pose_library_builder",
|
||||
"action_timeline_compiler",
|
||||
"character_rig_renderer",
|
||||
"character_animation_reviewer",
|
||||
}.issubset(set(get_required_tools(manifest)))
|
||||
|
||||
|
||||
def test_character_artifacts_are_registered():
|
||||
assert {
|
||||
"character_design",
|
||||
"rig_plan",
|
||||
"pose_library",
|
||||
"action_timeline",
|
||||
"character_qa_report",
|
||||
}.issubset(set(ARTIFACT_NAMES))
|
||||
|
||||
|
||||
def test_character_tools_discover_in_registry():
|
||||
registry.discover()
|
||||
|
||||
names = {tool.name for tool in registry.get_by_capability("character_animation")}
|
||||
assert {
|
||||
"character_spec_generator",
|
||||
"svg_rig_builder",
|
||||
"pose_library_builder",
|
||||
"action_timeline_compiler",
|
||||
"character_rig_renderer",
|
||||
"character_animation_reviewer",
|
||||
}.issubset(names)
|
||||
|
||||
|
||||
def test_character_animation_smoke_flow(tmp_path):
|
||||
character_result = CharacterSpecGenerator().execute(
|
||||
{
|
||||
"characters": [
|
||||
{
|
||||
"id": "mouse_lead",
|
||||
"role": "curious lead",
|
||||
"body_type": "mouse with tail",
|
||||
"required_actions": ["idle", "gesture", "tail_swish"],
|
||||
},
|
||||
{
|
||||
"id": "bird_friend",
|
||||
"role": "expressive sidekick",
|
||||
"body_type": "round bird",
|
||||
"required_actions": ["idle", "wing_flap", "react"],
|
||||
},
|
||||
],
|
||||
"output_path": str(tmp_path / "character_design.json"),
|
||||
}
|
||||
)
|
||||
assert character_result.success
|
||||
character_design = character_result.data["character_design"]
|
||||
validate_artifact("character_design", character_design)
|
||||
|
||||
rig_result = SvgRigBuilder().execute(
|
||||
{
|
||||
"character_design": character_design,
|
||||
"output_path": str(tmp_path / "rig_plan.json"),
|
||||
}
|
||||
)
|
||||
assert rig_result.success
|
||||
rig_plan = rig_result.data["rig_plan"]
|
||||
validate_artifact("rig_plan", rig_plan)
|
||||
|
||||
pose_result = PoseLibraryBuilder().execute(
|
||||
{"rig_plan": rig_plan, "output_path": str(tmp_path / "pose_library.json")}
|
||||
)
|
||||
assert pose_result.success
|
||||
pose_library = pose_result.data["pose_library"]
|
||||
validate_artifact("pose_library", pose_library)
|
||||
|
||||
scene_plan = {
|
||||
"version": "1.0",
|
||||
"scenes": [
|
||||
{
|
||||
"id": "scene-1",
|
||||
"type": "character_scene",
|
||||
"start_seconds": 0,
|
||||
"end_seconds": 4,
|
||||
"description": "The mouse discovers a glowing seed while the bird reacts.",
|
||||
"hero_moment": True,
|
||||
"character_actions": [
|
||||
{
|
||||
"character_id": "mouse_lead",
|
||||
"emotion": "surprised",
|
||||
"action_sequence": ["anticipate", "perform", "settle"],
|
||||
},
|
||||
{
|
||||
"character_id": "bird_friend",
|
||||
"emotion": "surprised",
|
||||
"action_sequence": ["react", "follow", "settle"],
|
||||
},
|
||||
],
|
||||
}
|
||||
],
|
||||
}
|
||||
validate_artifact("scene_plan", scene_plan)
|
||||
timeline_result = ActionTimelineCompiler().execute(
|
||||
{
|
||||
"scene_plan": scene_plan,
|
||||
"character_ids": ["mouse_lead", "bird_friend"],
|
||||
"output_path": str(tmp_path / "action_timeline.json"),
|
||||
}
|
||||
)
|
||||
assert timeline_result.success
|
||||
action_timeline = timeline_result.data["action_timeline"]
|
||||
validate_artifact("action_timeline", action_timeline)
|
||||
assert {action["character_id"] for action in action_timeline["scenes"][0]["actions"]} == {
|
||||
"mouse_lead",
|
||||
"bird_friend",
|
||||
}
|
||||
|
||||
preview_path = tmp_path / "preview.html"
|
||||
render_result = CharacterRigRenderer().execute(
|
||||
{
|
||||
"rig_plan": rig_plan,
|
||||
"pose_library": pose_library,
|
||||
"action_timeline": action_timeline,
|
||||
"output_path": str(preview_path),
|
||||
}
|
||||
)
|
||||
assert render_result.success
|
||||
assert preview_path.exists()
|
||||
preview_html = preview_path.read_text(encoding="utf-8")
|
||||
assert "character_mouse-lead" in preview_html
|
||||
assert "character_bird-friend" in preview_html
|
||||
|
||||
qa_result = CharacterAnimationReviewer().execute(
|
||||
{
|
||||
"rig_plan": rig_plan,
|
||||
"pose_library": pose_library,
|
||||
"action_timeline": action_timeline,
|
||||
"preview_path": str(preview_path),
|
||||
"output_path": str(tmp_path / "character_qa_report.json"),
|
||||
}
|
||||
)
|
||||
assert qa_result.success
|
||||
qa_report = qa_result.data["character_qa_report"]
|
||||
validate_artifact("character_qa_report", qa_report)
|
||||
assert qa_report["status"] == "pass"
|
||||
assert qa_report["checks"]["schema_valid"] is True
|
||||
|
||||
|
||||
def test_character_style_is_normalized_for_schema(tmp_path):
|
||||
result = CharacterSpecGenerator().execute(
|
||||
{
|
||||
"characters": [{"id": "style_test", "role": "lead", "body_type": "round"}],
|
||||
"style": {
|
||||
"name": "flat-motion-graphics",
|
||||
"palette": ["#ff8f68", "#75b8ff"],
|
||||
"unexpected": "should not leak into artifact",
|
||||
},
|
||||
"output_path": str(tmp_path / "character_design.json"),
|
||||
}
|
||||
)
|
||||
|
||||
assert result.success
|
||||
character_design = result.data["character_design"]
|
||||
validate_artifact("character_design", character_design)
|
||||
assert character_design["style"] == {
|
||||
"visual_style": "flat-motion-graphics",
|
||||
"palette": ["#ff8f68", "#75b8ff"],
|
||||
}
|
||||
|
||||
|
||||
def test_character_renderer_can_handoff_to_video_compose(tmp_path):
|
||||
hyperframes = HyperFramesCompose()
|
||||
runtime = hyperframes._runtime_check()
|
||||
if not runtime["runtime_available"]:
|
||||
pytest.skip("HyperFrames runtime is required for character render handoff")
|
||||
|
||||
character_design = CharacterSpecGenerator().execute(
|
||||
{"characters": [{"id": "mouse_lead", "role": "lead", "body_type": "mouse with tail"}]}
|
||||
).data["character_design"]
|
||||
rig_plan = SvgRigBuilder().execute({"character_design": character_design}).data["rig_plan"]
|
||||
pose_library = PoseLibraryBuilder().execute({"rig_plan": rig_plan}).data["pose_library"]
|
||||
scene_plan = {
|
||||
"version": "1.0",
|
||||
"scenes": [
|
||||
{
|
||||
"id": "scene-1",
|
||||
"type": "character_scene",
|
||||
"description": "Mouse reacts to a tiny surprise.",
|
||||
"start_seconds": 0,
|
||||
"end_seconds": 1,
|
||||
"character_actions": [
|
||||
{
|
||||
"character_id": "mouse_lead",
|
||||
"emotion": "surprised",
|
||||
"action_sequence": ["anticipate", "perform", "settle"],
|
||||
}
|
||||
],
|
||||
}
|
||||
],
|
||||
}
|
||||
validate_artifact("scene_plan", scene_plan)
|
||||
action_timeline = ActionTimelineCompiler().execute(
|
||||
{"scene_plan": scene_plan, "character_ids": ["mouse_lead"]}
|
||||
).data["action_timeline"]
|
||||
|
||||
render_result = CharacterRigRenderer().execute(
|
||||
{
|
||||
"rig_plan": rig_plan,
|
||||
"pose_library": pose_library,
|
||||
"action_timeline": action_timeline,
|
||||
"output_path": str(tmp_path / "preview.html"),
|
||||
"workspace_path": str(tmp_path / "hyperframes"),
|
||||
}
|
||||
)
|
||||
assert render_result.success
|
||||
validate_artifact("asset_manifest", render_result.data["asset_manifest"])
|
||||
validate_artifact("edit_decisions", render_result.data["edit_decisions"])
|
||||
assert render_result.data["edit_decisions"]["render_runtime"] == "hyperframes"
|
||||
assert Path(render_result.data["composition_path"]).exists()
|
||||
|
||||
output_path = tmp_path / "renders" / "final.mp4"
|
||||
compose_result = VideoCompose().execute(
|
||||
{
|
||||
"operation": "render",
|
||||
"asset_manifest": render_result.data["asset_manifest"],
|
||||
"edit_decisions": render_result.data["edit_decisions"],
|
||||
"workspace_path": render_result.data["hyperframes_workspace"],
|
||||
"output_path": str(output_path),
|
||||
"skip_contrast": True,
|
||||
"quality": "draft",
|
||||
"fps": 24,
|
||||
}
|
||||
)
|
||||
|
||||
assert compose_result.success, compose_result.error
|
||||
assert output_path.exists()
|
||||
@@ -627,6 +627,171 @@ def test_decision_log_accepts_render_runtime_selection_with_both_options():
|
||||
jsonschema.validate(log, schema)
|
||||
|
||||
|
||||
def test_transcript_comparison_catches_literal_punctuation_leak(tmp_path):
|
||||
"""Regression: Chirp3-HD (and some other TTS engines) read literal `...`
|
||||
as the word 'dot' in audio output. This failure is invisible to
|
||||
volume-based audio spotchecks but ships audio that literally says
|
||||
'dot dot dot' twelve times. The transcript_comparison check catches
|
||||
this automatically before the video is marked pass."""
|
||||
import json
|
||||
|
||||
# Real-world example: user's script had `...` everywhere for dramatic
|
||||
# pause, Chirp read them all as "dot", transcript contains "dot dot dot"
|
||||
# phrases the script never had.
|
||||
script_text = (
|
||||
"A computer just did in five minutes what would take every machine on Earth, "
|
||||
"running since the Big Bang, ten septillion years to finish. "
|
||||
"We may have gotten help from parallel universes."
|
||||
)
|
||||
transcript_data = {
|
||||
"word_timestamps": [
|
||||
{"word": "A", "start": 0.0, "end": 0.1},
|
||||
{"word": "computer", "start": 0.1, "end": 0.5},
|
||||
{"word": "just", "start": 0.5, "end": 0.7},
|
||||
{"word": "did", "start": 0.7, "end": 0.9},
|
||||
{"word": "in", "start": 0.9, "end": 1.0},
|
||||
{"word": "five", "start": 1.0, "end": 1.3},
|
||||
{"word": "minutes", "start": 1.3, "end": 1.7},
|
||||
{"word": "dot", "start": 1.7, "end": 1.9}, # leak!
|
||||
{"word": "dot", "start": 1.9, "end": 2.1}, # leak!
|
||||
{"word": "dot", "start": 2.1, "end": 2.3}, # leak!
|
||||
{"word": "what", "start": 2.5, "end": 2.8},
|
||||
{"word": "would", "start": 2.8, "end": 3.0},
|
||||
{"word": "take", "start": 3.0, "end": 3.3},
|
||||
{"word": "every", "start": 3.3, "end": 3.6},
|
||||
{"word": "machine", "start": 3.6, "end": 4.0},
|
||||
{"word": "on", "start": 4.0, "end": 4.2},
|
||||
{"word": "Earth", "start": 4.2, "end": 4.6},
|
||||
{"word": "running", "start": 4.7, "end": 5.1},
|
||||
{"word": "since", "start": 5.1, "end": 5.4},
|
||||
{"word": "the", "start": 5.4, "end": 5.5},
|
||||
{"word": "Big", "start": 5.5, "end": 5.8},
|
||||
{"word": "Bang", "start": 5.8, "end": 6.2},
|
||||
{"word": "ten", "start": 6.2, "end": 6.5},
|
||||
{"word": "septillion", "start": 6.5, "end": 7.3},
|
||||
{"word": "years", "start": 7.3, "end": 7.7},
|
||||
{"word": "to", "start": 7.7, "end": 7.9},
|
||||
{"word": "finish", "start": 7.9, "end": 8.3},
|
||||
{"word": "dot", "start": 8.3, "end": 8.5}, # another leak
|
||||
{"word": "We", "start": 9.0, "end": 9.2},
|
||||
{"word": "may", "start": 9.2, "end": 9.4},
|
||||
{"word": "have", "start": 9.4, "end": 9.6},
|
||||
{"word": "gotten", "start": 9.6, "end": 9.9},
|
||||
{"word": "help", "start": 9.9, "end": 10.3},
|
||||
{"word": "from", "start": 10.3, "end": 10.5},
|
||||
{"word": "parallel", "start": 10.5, "end": 11.0},
|
||||
{"word": "universes", "start": 11.0, "end": 11.7},
|
||||
]
|
||||
}
|
||||
transcript_path = tmp_path / "transcript.json"
|
||||
transcript_path.write_text(json.dumps(transcript_data), encoding="utf-8")
|
||||
|
||||
result = VideoCompose._compare_transcript_to_script(transcript_path, script_text)
|
||||
|
||||
# Must catch the punctuation leak
|
||||
assert result["spurious_punctuation_words"], (
|
||||
"transcript_comparison failed to detect the 'dot' leak from literal ... punctuation."
|
||||
)
|
||||
leak_counts = {
|
||||
entry["word"]: entry["count"]
|
||||
for entry in result["spurious_punctuation_words"]
|
||||
}
|
||||
assert leak_counts.get("dot") == 4, f"Expected 4 'dot' leaks, got {leak_counts}"
|
||||
|
||||
# Must produce a CRITICAL-severity issue message
|
||||
issue_text = " ".join(result["issues"]).lower()
|
||||
assert "tts punctuation leak" in issue_text
|
||||
assert "not in the script" in issue_text
|
||||
|
||||
# Must NOT mark the transcript as matching
|
||||
assert result["transcript_matches_script"] is False
|
||||
|
||||
|
||||
def test_transcript_comparison_passes_clean_audio(tmp_path):
|
||||
"""Clean audio with no punctuation leaks must NOT trigger a false
|
||||
positive."""
|
||||
import json
|
||||
|
||||
script_text = "The quick brown fox jumps over the lazy dog."
|
||||
transcript_data = {
|
||||
"word_timestamps": [
|
||||
{"word": "The", "start": 0.0, "end": 0.1},
|
||||
{"word": "quick", "start": 0.1, "end": 0.4},
|
||||
{"word": "brown", "start": 0.4, "end": 0.7},
|
||||
{"word": "fox", "start": 0.7, "end": 1.0},
|
||||
{"word": "jumps", "start": 1.0, "end": 1.3},
|
||||
{"word": "over", "start": 1.3, "end": 1.6},
|
||||
{"word": "the", "start": 1.6, "end": 1.7},
|
||||
{"word": "lazy", "start": 1.7, "end": 2.0},
|
||||
{"word": "dog", "start": 2.0, "end": 2.3},
|
||||
]
|
||||
}
|
||||
transcript_path = tmp_path / "transcript.json"
|
||||
transcript_path.write_text(json.dumps(transcript_data), encoding="utf-8")
|
||||
|
||||
result = VideoCompose._compare_transcript_to_script(transcript_path, script_text)
|
||||
assert result["spurious_punctuation_words"] == []
|
||||
assert result["transcript_matches_script"] is True
|
||||
assert result["word_accuracy"] >= 0.9
|
||||
# issues may still have informational content but no CRITICAL TTS leak
|
||||
assert not any("tts punctuation leak" in i.lower() for i in result["issues"])
|
||||
|
||||
|
||||
def test_transcript_comparison_graceful_when_inputs_missing(tmp_path):
|
||||
"""When transcript or script is unavailable, the check should NOT
|
||||
crash — it should record the skip in issues so the silence is visible."""
|
||||
# No transcript
|
||||
result = VideoCompose._compare_transcript_to_script(None, "some script text")
|
||||
assert any("not provided" in i for i in result["issues"])
|
||||
|
||||
# No script
|
||||
dummy = tmp_path / "t.json"
|
||||
dummy.write_text('{"word_timestamps": []}', encoding="utf-8")
|
||||
result = VideoCompose._compare_transcript_to_script(dummy, "")
|
||||
assert any("not provided" in i for i in result["issues"])
|
||||
|
||||
# Transcript file missing
|
||||
result = VideoCompose._compare_transcript_to_script(tmp_path / "nonexistent.json", "script")
|
||||
assert any("not provided" in i for i in result["issues"])
|
||||
|
||||
|
||||
def test_run_final_review_includes_transcript_comparison_section(tmp_path):
|
||||
"""Regression: the `transcript_comparison` section must ALWAYS appear in
|
||||
the final_review output — even when the caller doesn't provide a
|
||||
transcript. A missing section = silent governance failure."""
|
||||
import subprocess
|
||||
|
||||
# Build a minimal MP4 so _run_final_review can probe it.
|
||||
mp4 = tmp_path / "out.mp4"
|
||||
subprocess.run(
|
||||
[
|
||||
"ffmpeg", "-y",
|
||||
"-f", "lavfi", "-i", "color=c=#000000:s=320x240:d=2",
|
||||
"-f", "lavfi", "-i", "sine=frequency=440:duration=2",
|
||||
"-c:v", "libx264", "-pix_fmt", "yuv420p",
|
||||
"-c:a", "aac", "-shortest", str(mp4),
|
||||
],
|
||||
capture_output=True, check=True, timeout=30,
|
||||
)
|
||||
|
||||
review = VideoCompose()._run_final_review(
|
||||
mp4,
|
||||
edit_decisions={
|
||||
"version": "1.0",
|
||||
"renderer_family": "animation-first",
|
||||
"render_runtime": "hyperframes",
|
||||
"cuts": [{"id": "c1", "source": "x", "in_seconds": 0, "out_seconds": 2}],
|
||||
},
|
||||
)
|
||||
assert "transcript_comparison" in review["checks"], (
|
||||
"final_review must always include a transcript_comparison section. "
|
||||
"When the caller doesn't provide a transcript, the section should "
|
||||
"still appear with a 'skipped' issue entry — not be omitted."
|
||||
)
|
||||
tc = review["checks"]["transcript_comparison"]
|
||||
assert any("not provided" in i for i in tc["issues"])
|
||||
|
||||
|
||||
def test_hyperframes_root_composition_has_data_start_and_duration(tmp_path):
|
||||
"""Regression: the generated root composition was missing data-start
|
||||
and data-duration, violating the HyperFrames contract (SKILL.md table)."""
|
||||
|
||||
@@ -0,0 +1,420 @@
|
||||
"""Doubao Speech text-to-speech provider tool."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from tools.base_tool import (
|
||||
BaseTool,
|
||||
Determinism,
|
||||
ExecutionMode,
|
||||
ResourceProfile,
|
||||
RetryPolicy,
|
||||
ToolResult,
|
||||
ToolRuntime,
|
||||
ToolStability,
|
||||
ToolStatus,
|
||||
ToolTier,
|
||||
)
|
||||
|
||||
|
||||
class DoubaoTTS(BaseTool):
|
||||
name = "doubao_tts"
|
||||
version = "0.1.0"
|
||||
tier = ToolTier.VOICE
|
||||
capability = "tts"
|
||||
provider = "doubao"
|
||||
stability = ToolStability.EXPERIMENTAL
|
||||
execution_mode = ExecutionMode.ASYNC
|
||||
determinism = Determinism.STOCHASTIC
|
||||
runtime = ToolRuntime.API
|
||||
|
||||
dependencies = []
|
||||
install_instructions = (
|
||||
"Set DOUBAO_SPEECH_API_KEY to a Volcengine Doubao Speech API Key.\n"
|
||||
"Optional: set DOUBAO_SPEECH_VOICE_TYPE to the default speaker voice.\n"
|
||||
"Use the new console API key flow; do not pass app id/access token as the API key."
|
||||
)
|
||||
fallback = "google_tts"
|
||||
fallback_tools = ["google_tts", "elevenlabs_tts", "openai_tts", "piper_tts"]
|
||||
agent_skills = ["doubao-tts", "text-to-speech"]
|
||||
|
||||
capabilities = [
|
||||
"text_to_speech",
|
||||
"voice_selection",
|
||||
"multilingual",
|
||||
"timestamp_alignment",
|
||||
]
|
||||
supports = {
|
||||
"voice_cloning": False,
|
||||
"multilingual": True,
|
||||
"offline": False,
|
||||
"native_audio": True,
|
||||
"timestamps": True,
|
||||
"long_text_async": True,
|
||||
}
|
||||
best_for = [
|
||||
"natural Mandarin narration",
|
||||
"Chinese explainer voiceovers with character-level timestamps",
|
||||
"long-form narration that needs subtitle alignment",
|
||||
]
|
||||
not_good_for = [
|
||||
"fully offline production",
|
||||
"voice clone matching",
|
||||
"real-time interactive speech playback",
|
||||
]
|
||||
|
||||
input_schema = {
|
||||
"type": "object",
|
||||
"required": ["text"],
|
||||
"properties": {
|
||||
"text": {"type": "string", "description": "Text to convert to speech"},
|
||||
"voice_id": {
|
||||
"type": "string",
|
||||
"description": "Doubao speaker/voice_type. Defaults to DOUBAO_SPEECH_VOICE_TYPE.",
|
||||
},
|
||||
"resource_id": {
|
||||
"type": "string",
|
||||
"default": "seed-tts-2.0",
|
||||
"description": "Volcengine resource id. Use seed-tts-2.0 for Doubao Speech 2.0 voices.",
|
||||
},
|
||||
"format": {
|
||||
"type": "string",
|
||||
"default": "mp3",
|
||||
"enum": ["mp3", "ogg_opus", "pcm"],
|
||||
},
|
||||
"sample_rate": {
|
||||
"type": "integer",
|
||||
"default": 24000,
|
||||
"enum": [8000, 16000, 22050, 24000, 32000, 44100, 48000],
|
||||
},
|
||||
"speech_rate": {
|
||||
"type": "integer",
|
||||
"default": 0,
|
||||
"minimum": -50,
|
||||
"maximum": 100,
|
||||
"description": "Doubao speech rate. 0=normal, 100=2x, -50=0.5x.",
|
||||
},
|
||||
"enable_timestamp": {
|
||||
"type": "boolean",
|
||||
"default": True,
|
||||
"description": "Return sentence/word timing metadata when supported by the selected endpoint.",
|
||||
},
|
||||
"disable_markdown_filter": {
|
||||
"type": "boolean",
|
||||
"default": False,
|
||||
"description": "Pass through Doubao markdown filtering behavior. Defaults to API-safe false.",
|
||||
},
|
||||
"return_usage": {
|
||||
"type": "boolean",
|
||||
"default": True,
|
||||
"description": "Request usage token data from Volcengine when available.",
|
||||
},
|
||||
"output_path": {"type": "string"},
|
||||
"metadata_path": {
|
||||
"type": "string",
|
||||
"description": "Where to save the full query JSON. Defaults next to output_path.",
|
||||
},
|
||||
"poll_interval_seconds": {
|
||||
"type": "number",
|
||||
"default": 2.0,
|
||||
"minimum": 0.5,
|
||||
},
|
||||
"timeout_seconds": {
|
||||
"type": "integer",
|
||||
"default": 300,
|
||||
"minimum": 30,
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
output_schema = {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"output": {"type": "string"},
|
||||
"metadata_path": {"type": "string"},
|
||||
"task_id": {"type": "string"},
|
||||
"audio_duration_seconds": {"type": ["number", "null"]},
|
||||
"sentences": {"type": "array"},
|
||||
"usage": {"type": ["object", "null"]},
|
||||
},
|
||||
}
|
||||
artifact_schema = {
|
||||
"type": "array",
|
||||
"items": {"type": "string"},
|
||||
}
|
||||
|
||||
resource_profile = ResourceProfile(
|
||||
cpu_cores=1, ram_mb=256, vram_mb=0, disk_mb=50, network_required=True
|
||||
)
|
||||
retry_policy = RetryPolicy(
|
||||
max_retries=2,
|
||||
backoff_seconds=2.0,
|
||||
retryable_errors=["timeout", "rate_limit", "quota exceeded for types: concurrency"],
|
||||
)
|
||||
idempotency_key_fields = ["text", "voice_id", "resource_id", "speech_rate", "sample_rate"]
|
||||
side_effects = [
|
||||
"writes audio file to output_path",
|
||||
"writes Doubao query metadata JSON next to output_path",
|
||||
"calls Volcengine Doubao Speech API",
|
||||
]
|
||||
user_visible_verification = [
|
||||
"Listen to generated audio for Mandarin naturalness and pacing",
|
||||
"Check timestamp JSON before building subtitles",
|
||||
]
|
||||
quality_score = 0.88
|
||||
latency_p50_seconds = 8.0
|
||||
|
||||
SUBMIT_URL = "https://openspeech.bytedance.com/api/v3/tts/submit"
|
||||
QUERY_URL = "https://openspeech.bytedance.com/api/v3/tts/query"
|
||||
DEFAULT_RESOURCE_ID = "seed-tts-2.0"
|
||||
DEFAULT_VOICE_ENV = "DOUBAO_SPEECH_VOICE_TYPE"
|
||||
|
||||
def get_status(self) -> ToolStatus:
|
||||
if os.environ.get("DOUBAO_SPEECH_API_KEY"):
|
||||
return ToolStatus.AVAILABLE
|
||||
return ToolStatus.UNAVAILABLE
|
||||
|
||||
def estimate_cost(self, inputs: dict[str, Any]) -> float:
|
||||
# Volcengine bills Doubao Speech 2.0 by characters. Keep this conservative
|
||||
# and prefer provider-returned usage when available.
|
||||
return round(len(inputs.get("text", "")) * 0.000015, 4)
|
||||
|
||||
def execute(self, inputs: dict[str, Any]) -> ToolResult:
|
||||
api_key = os.environ.get("DOUBAO_SPEECH_API_KEY")
|
||||
if not api_key:
|
||||
return ToolResult(success=False, error="No Doubao Speech API key. " + self.install_instructions)
|
||||
|
||||
voice_id = inputs.get("voice_id") or os.environ.get(self.DEFAULT_VOICE_ENV)
|
||||
if not voice_id:
|
||||
return ToolResult(
|
||||
success=False,
|
||||
error=(
|
||||
"No Doubao voice_id provided. Pass voice_id or set "
|
||||
f"{self.DEFAULT_VOICE_ENV} in the environment."
|
||||
),
|
||||
)
|
||||
|
||||
start = time.time()
|
||||
try:
|
||||
result = self._generate(inputs, api_key=api_key, voice_id=voice_id)
|
||||
except Exception as exc:
|
||||
return ToolResult(success=False, error=f"Doubao TTS failed: {self._safe_error(exc)}")
|
||||
|
||||
result.duration_seconds = round(time.time() - start, 2)
|
||||
if not result.cost_usd:
|
||||
result.cost_usd = self.estimate_cost(inputs)
|
||||
return result
|
||||
|
||||
def _generate(self, inputs: dict[str, Any], *, api_key: str, voice_id: str) -> ToolResult:
|
||||
import requests
|
||||
|
||||
text = inputs["text"]
|
||||
fmt = inputs.get("format", "mp3")
|
||||
resource_id = inputs.get("resource_id", self.DEFAULT_RESOURCE_ID)
|
||||
output_path = Path(inputs.get("output_path", f"doubao_tts.{self._extension_for_format(fmt)}"))
|
||||
metadata_path = Path(
|
||||
inputs.get("metadata_path") or output_path.with_suffix(output_path.suffix + ".json")
|
||||
)
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
metadata_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
req_id = str(uuid.uuid4())
|
||||
headers = self._headers(
|
||||
api_key=api_key,
|
||||
resource_id=resource_id,
|
||||
request_id=req_id,
|
||||
return_usage=bool(inputs.get("return_usage", True)),
|
||||
)
|
||||
body = self._submit_body(inputs, voice_id=voice_id, request_id=req_id)
|
||||
|
||||
submit_response = requests.post(self.SUBMIT_URL, headers=headers, json=body, timeout=(10, 60))
|
||||
submit_data = self._json_or_raise(submit_response)
|
||||
self._raise_for_doubao_error(submit_response.status_code, submit_data)
|
||||
|
||||
task_id = submit_data.get("data", {}).get("task_id")
|
||||
if not task_id:
|
||||
raise RuntimeError("Doubao submit succeeded but did not return data.task_id")
|
||||
|
||||
query_data = self._poll_query(
|
||||
requests_module=requests,
|
||||
api_key=api_key,
|
||||
resource_id=resource_id,
|
||||
task_id=task_id,
|
||||
return_usage=bool(inputs.get("return_usage", True)),
|
||||
poll_interval=float(inputs.get("poll_interval_seconds", 2.0)),
|
||||
timeout_seconds=int(inputs.get("timeout_seconds", 300)),
|
||||
)
|
||||
data = query_data.get("data", {})
|
||||
audio_url = data.get("audio_url")
|
||||
if not audio_url:
|
||||
raise RuntimeError("Doubao task completed but did not return data.audio_url")
|
||||
|
||||
audio_response = requests.get(audio_url, timeout=(10, 120))
|
||||
audio_response.raise_for_status()
|
||||
output_path.write_bytes(audio_response.content)
|
||||
metadata_path.write_text(json.dumps(query_data, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||
|
||||
audio_duration = self._audio_duration(output_path)
|
||||
usage = data.get("usage")
|
||||
cost = self._cost_from_usage(usage) or self.estimate_cost(inputs)
|
||||
|
||||
return ToolResult(
|
||||
success=True,
|
||||
data={
|
||||
"provider": self.provider,
|
||||
"model": resource_id,
|
||||
"resource_id": resource_id,
|
||||
"voice_id": voice_id,
|
||||
"format": fmt,
|
||||
"sample_rate": inputs.get("sample_rate", 24000),
|
||||
"speech_rate": inputs.get("speech_rate", 0),
|
||||
"text_length": len(text),
|
||||
"task_id": task_id,
|
||||
"task_status": data.get("task_status"),
|
||||
"req_text_length": data.get("req_text_length"),
|
||||
"synthesize_text_length": data.get("synthesize_text_length"),
|
||||
"audio_duration_seconds": round(audio_duration, 2) if audio_duration else None,
|
||||
"output": str(output_path),
|
||||
"metadata_path": str(metadata_path),
|
||||
"sentences": data.get("sentences", []),
|
||||
"usage": usage,
|
||||
"url_expire_time": data.get("url_expire_time"),
|
||||
},
|
||||
artifacts=[str(output_path), str(metadata_path)],
|
||||
cost_usd=cost,
|
||||
model=resource_id,
|
||||
)
|
||||
|
||||
def _headers(
|
||||
self,
|
||||
*,
|
||||
api_key: str,
|
||||
resource_id: str,
|
||||
request_id: str,
|
||||
return_usage: bool,
|
||||
) -> dict[str, str]:
|
||||
headers = {
|
||||
"X-Api-Key": api_key,
|
||||
"X-Api-Resource-Id": resource_id,
|
||||
"X-Api-Request-Id": request_id,
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
if return_usage:
|
||||
headers["X-Control-Require-Usage-Tokens-Return"] = "true"
|
||||
return headers
|
||||
|
||||
def _submit_body(self, inputs: dict[str, Any], *, voice_id: str, request_id: str) -> dict[str, Any]:
|
||||
audio_params = {
|
||||
"format": inputs.get("format", "mp3"),
|
||||
"sample_rate": inputs.get("sample_rate", 24000),
|
||||
"speech_rate": inputs.get("speech_rate", 0),
|
||||
"enable_timestamp": bool(inputs.get("enable_timestamp", True)),
|
||||
}
|
||||
additions = {
|
||||
"disable_markdown_filter": bool(inputs.get("disable_markdown_filter", False)),
|
||||
}
|
||||
return {
|
||||
"user": {"uid": inputs.get("user_id", "openmontage")},
|
||||
"unique_id": request_id,
|
||||
"req_params": {
|
||||
"text": inputs["text"],
|
||||
"speaker": voice_id,
|
||||
"audio_params": audio_params,
|
||||
"additions": json.dumps(additions, ensure_ascii=False),
|
||||
},
|
||||
}
|
||||
|
||||
def _poll_query(
|
||||
self,
|
||||
*,
|
||||
requests_module: Any,
|
||||
api_key: str,
|
||||
resource_id: str,
|
||||
task_id: str,
|
||||
return_usage: bool,
|
||||
poll_interval: float,
|
||||
timeout_seconds: int,
|
||||
) -> dict[str, Any]:
|
||||
deadline = time.time() + timeout_seconds
|
||||
while time.time() < deadline:
|
||||
time.sleep(poll_interval)
|
||||
headers = self._headers(
|
||||
api_key=api_key,
|
||||
resource_id=resource_id,
|
||||
request_id=str(uuid.uuid4()),
|
||||
return_usage=return_usage,
|
||||
)
|
||||
response = requests_module.post(self.QUERY_URL, headers=headers, json={"task_id": task_id}, timeout=(10, 60))
|
||||
query_data = self._json_or_raise(response)
|
||||
self._raise_for_doubao_error(response.status_code, query_data)
|
||||
status = query_data.get("data", {}).get("task_status")
|
||||
if status == 2:
|
||||
return query_data
|
||||
if status == 3:
|
||||
raise RuntimeError(f"Doubao task failed: {query_data.get('message', 'unknown error')}")
|
||||
raise TimeoutError(f"Doubao task did not finish within {timeout_seconds} seconds")
|
||||
|
||||
@staticmethod
|
||||
def _json_or_raise(response: Any) -> dict[str, Any]:
|
||||
try:
|
||||
return response.json()
|
||||
except ValueError as exc:
|
||||
raise RuntimeError(f"Non-JSON response from Doubao API: HTTP {response.status_code}") from exc
|
||||
|
||||
def _raise_for_doubao_error(self, http_status: int, payload: dict[str, Any]) -> None:
|
||||
code = payload.get("code")
|
||||
if http_status < 400 and code == 20000000:
|
||||
return
|
||||
message = payload.get("message", "unknown error")
|
||||
hint = self._diagnostic_hint(message)
|
||||
raise RuntimeError(f"HTTP {http_status}, code {code}: {message}{hint}")
|
||||
|
||||
@staticmethod
|
||||
def _diagnostic_hint(message: str) -> str:
|
||||
lowered = message.lower()
|
||||
if "load grant" in lowered or "requested grant not found" in lowered:
|
||||
return " (check DOUBAO_SPEECH_API_KEY and use the new-console X-Api-Key flow)"
|
||||
if "speaker permission denied" in lowered or "access denied" in lowered:
|
||||
return " (check voice_id/DOUBAO_SPEECH_VOICE_TYPE and voice authorization)"
|
||||
if "quota exceeded" in lowered:
|
||||
return " (check quota, concurrency, or remaining character package)"
|
||||
if "unsupported additions explicit language" in lowered:
|
||||
return " (do not pass additions.explicit_language for this endpoint)"
|
||||
return ""
|
||||
|
||||
@staticmethod
|
||||
def _safe_error(exc: Exception) -> str:
|
||||
# Avoid ever echoing request headers or secrets in user-visible errors.
|
||||
return str(exc).replace(os.environ.get("DOUBAO_SPEECH_API_KEY", ""), "[redacted]")
|
||||
|
||||
@staticmethod
|
||||
def _extension_for_format(fmt: str) -> str:
|
||||
if fmt == "ogg_opus":
|
||||
return "ogg"
|
||||
if fmt == "pcm":
|
||||
return "pcm"
|
||||
return "mp3"
|
||||
|
||||
@staticmethod
|
||||
def _audio_duration(path: Path) -> float | None:
|
||||
try:
|
||||
from tools.analysis.audio_probe import probe_duration
|
||||
|
||||
return probe_duration(path)
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def _cost_from_usage(usage: Any) -> float | None:
|
||||
if not isinstance(usage, dict):
|
||||
return None
|
||||
text_words = usage.get("text_words")
|
||||
if not isinstance(text_words, (int, float)):
|
||||
return None
|
||||
return round(float(text_words) * 0.000015, 4)
|
||||
@@ -0,0 +1,2 @@
|
||||
"""Character animation tools."""
|
||||
|
||||
@@ -0,0 +1,896 @@
|
||||
"""Local character-animation contract tools.
|
||||
|
||||
These tools provide deterministic artifact generation and validation for the
|
||||
character-animation pipeline. They intentionally keep creative orchestration in
|
||||
skills and manifests; Python only creates structured artifacts and lightweight
|
||||
preview/review outputs.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import shutil
|
||||
import subprocess
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from schemas.artifacts import validate_artifact
|
||||
from tools.base_tool import (
|
||||
BaseTool,
|
||||
Determinism,
|
||||
ExecutionMode,
|
||||
ResourceProfile,
|
||||
ToolResult,
|
||||
ToolStability,
|
||||
ToolTier,
|
||||
)
|
||||
|
||||
|
||||
def _write_json(path: str | None, data: dict[str, Any]) -> list[str]:
|
||||
if not path:
|
||||
return []
|
||||
out = Path(path)
|
||||
out.parent.mkdir(parents=True, exist_ok=True)
|
||||
out.write_text(json.dumps(data, indent=2), encoding="utf-8")
|
||||
return [str(out)]
|
||||
|
||||
|
||||
def _slug(value: str) -> str:
|
||||
chars = [c.lower() if c.isalnum() else "-" for c in value.strip()]
|
||||
return "-".join("".join(chars).split("-")).strip("-") or "character"
|
||||
|
||||
|
||||
def _character_color(index: int) -> tuple[str, str]:
|
||||
palettes = [
|
||||
("#ff8f68", "#ffd39f"),
|
||||
("#75b8ff", "#ffe7a3"),
|
||||
("#8fd17f", "#f7c8ff"),
|
||||
("#f2c94c", "#fce6c9"),
|
||||
]
|
||||
return palettes[index % len(palettes)]
|
||||
|
||||
|
||||
def _normalize_style(style: Any) -> dict[str, Any]:
|
||||
if not isinstance(style, dict):
|
||||
return {}
|
||||
normalized: dict[str, Any] = {}
|
||||
visual_style = style.get("visual_style") or style.get("name") or style.get("style")
|
||||
if visual_style:
|
||||
normalized["visual_style"] = str(visual_style)
|
||||
palette = style.get("palette")
|
||||
if isinstance(palette, list):
|
||||
normalized["palette"] = [str(color) for color in palette]
|
||||
for key in ["line_style", "texture"]:
|
||||
if style.get(key):
|
||||
normalized[key] = str(style[key])
|
||||
return normalized
|
||||
|
||||
|
||||
def _render_preview_mp4(preview_path: Path, video_path: Path, duration_seconds: float, fps: int) -> None:
|
||||
if shutil.which("ffmpeg") is None:
|
||||
raise RuntimeError("ffmpeg is required to render preview MP4")
|
||||
try:
|
||||
from playwright.sync_api import sync_playwright
|
||||
except Exception as exc: # pragma: no cover - dependency-specific branch
|
||||
raise RuntimeError("Playwright is required to render preview MP4") from exc
|
||||
|
||||
frame_dir = video_path.parent / f"{video_path.stem}_frames"
|
||||
frame_dir.mkdir(parents=True, exist_ok=True)
|
||||
frame_count = max(2, int(duration_seconds * fps))
|
||||
with sync_playwright() as p:
|
||||
browser = p.chromium.launch()
|
||||
page = browser.new_page(viewport={"width": 1280, "height": 720})
|
||||
page.goto(preview_path.resolve().as_uri(), wait_until="networkidle")
|
||||
for frame in range(frame_count):
|
||||
if frame:
|
||||
page.wait_for_timeout(int(1000 / fps))
|
||||
page.screenshot(path=str(frame_dir / f"frame_{frame:04d}.png"))
|
||||
browser.close()
|
||||
|
||||
cmd = [
|
||||
"ffmpeg",
|
||||
"-y",
|
||||
"-framerate",
|
||||
str(fps),
|
||||
"-i",
|
||||
str(frame_dir / "frame_%04d.png"),
|
||||
"-r",
|
||||
str(fps),
|
||||
"-pix_fmt",
|
||||
"yuv420p",
|
||||
str(video_path),
|
||||
]
|
||||
result = subprocess.run(cmd, capture_output=True, text=True)
|
||||
if result.returncode != 0:
|
||||
raise RuntimeError(result.stderr.strip() or "ffmpeg failed to render preview MP4")
|
||||
|
||||
|
||||
class CharacterSpecGenerator(BaseTool):
|
||||
name = "character_spec_generator"
|
||||
version = "0.1.0"
|
||||
tier = ToolTier.CORE
|
||||
capability = "character_animation"
|
||||
provider = "openmontage"
|
||||
stability = ToolStability.BETA
|
||||
execution_mode = ExecutionMode.SYNC
|
||||
determinism = Determinism.DETERMINISTIC
|
||||
resource_profile = ResourceProfile(cpu_cores=1, ram_mb=128, vram_mb=0, disk_mb=10)
|
||||
agent_skills = ["character-rigging", "pose-library-design"]
|
||||
capabilities = ["draft_character_design", "normalize_character_specs"]
|
||||
best_for = ["Converting approved concepts into structured character_design artifacts"]
|
||||
not_good_for = ["Generating artwork pixels or finished animation"]
|
||||
input_schema = {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"characters": {"type": "array"},
|
||||
"brief": {"type": "string"},
|
||||
"style": {"type": "object"},
|
||||
"output_path": {"type": "string"},
|
||||
},
|
||||
}
|
||||
output_schema = {"type": "object", "properties": {"character_design": {"type": "object"}}}
|
||||
artifact_schema = {"artifact": "character_design"}
|
||||
side_effects = ["optionally writes character_design JSON to output_path"]
|
||||
user_visible_verification = ["Review character count, action list, and emotional range"]
|
||||
|
||||
def execute(self, inputs: dict[str, Any]) -> ToolResult:
|
||||
start = time.time()
|
||||
raw_characters = inputs.get("characters") or [
|
||||
{
|
||||
"id": "main_character",
|
||||
"role": "lead character",
|
||||
"body_type": "simple rounded cartoon character",
|
||||
"style": "local rigged cartoon",
|
||||
"required_emotions": ["neutral", "curious", "happy", "surprised"],
|
||||
"required_actions": ["idle", "blink", "look", "gesture"],
|
||||
}
|
||||
]
|
||||
characters: list[dict[str, Any]] = []
|
||||
for raw in raw_characters:
|
||||
name = raw.get("id") or raw.get("name") or raw.get("display_name") or "character"
|
||||
characters.append(
|
||||
{
|
||||
"id": _slug(str(name)),
|
||||
"display_name": raw.get("display_name", str(name).replace("_", " ").title()),
|
||||
"role": raw.get("role", "supporting character"),
|
||||
"body_type": raw.get("body_type", "simple cartoon body"),
|
||||
"style": raw.get("style", inputs.get("style", {}).get("visual_style", "cartoon")),
|
||||
"silhouette_notes": raw.get("silhouette_notes", ""),
|
||||
"required_emotions": raw.get("required_emotions", ["neutral", "happy", "surprised"]),
|
||||
"required_actions": raw.get("required_actions", ["idle", "blink", "look"]),
|
||||
"required_views": raw.get("required_views", ["front", "side"]),
|
||||
"props": raw.get("props", []),
|
||||
"constraints": raw.get("constraints", []),
|
||||
}
|
||||
)
|
||||
artifact = {
|
||||
"version": "1.0",
|
||||
"style": _normalize_style(inputs.get("style", {})),
|
||||
"characters": characters,
|
||||
"metadata": {
|
||||
"source": "character_spec_generator",
|
||||
"brief": inputs.get("brief", ""),
|
||||
},
|
||||
}
|
||||
artifacts = _write_json(inputs.get("output_path"), artifact)
|
||||
return ToolResult(
|
||||
success=True,
|
||||
data={"character_design": artifact},
|
||||
artifacts=artifacts,
|
||||
duration_seconds=round(time.time() - start, 2),
|
||||
)
|
||||
|
||||
|
||||
class SvgRigBuilder(BaseTool):
|
||||
name = "svg_rig_builder"
|
||||
version = "0.1.0"
|
||||
tier = ToolTier.CORE
|
||||
capability = "character_animation"
|
||||
provider = "openmontage"
|
||||
stability = ToolStability.BETA
|
||||
execution_mode = ExecutionMode.SYNC
|
||||
determinism = Determinism.DETERMINISTIC
|
||||
resource_profile = ResourceProfile(cpu_cores=1, ram_mb=128, vram_mb=0, disk_mb=10)
|
||||
agent_skills = ["character-rigging", "svg-character-animation", "gsap-core", "gsap-timeline"]
|
||||
capabilities = ["draft_svg_rig_plan", "define_parts_pivots_layers"]
|
||||
input_schema = {
|
||||
"type": "object",
|
||||
"required": ["character_design"],
|
||||
"properties": {
|
||||
"character_design": {"type": "object"},
|
||||
"output_path": {"type": "string"},
|
||||
},
|
||||
}
|
||||
artifact_schema = {"artifact": "rig_plan"}
|
||||
side_effects = ["optionally writes rig_plan JSON to output_path"]
|
||||
user_visible_verification = ["Check pivots and layers before asset generation"]
|
||||
|
||||
def execute(self, inputs: dict[str, Any]) -> ToolResult:
|
||||
start = time.time()
|
||||
design = inputs["character_design"]
|
||||
rig_characters: list[dict[str, Any]] = []
|
||||
for character in design.get("characters", []):
|
||||
cid = character["id"]
|
||||
actions = character.get("required_actions", [])
|
||||
base_parts = [
|
||||
("body", "torso", 20, None, [320, 380]),
|
||||
("head", "head", 40, "body", [320, 220]),
|
||||
("eye_left", "eye", 50, "head", [288, 210]),
|
||||
("eye_right", "eye", 50, "head", [352, 210]),
|
||||
("pupil_left", "pupil", 51, "eye_left", [288, 210]),
|
||||
("pupil_right", "pupil", 51, "eye_right", [352, 210]),
|
||||
("mouth", "mouth", 52, "head", [320, 260]),
|
||||
("arm_left", "limb", 35, "body", [260, 330]),
|
||||
("arm_right", "limb", 35, "body", [380, 330]),
|
||||
("leg_left", "limb", 10, "body", [285, 470]),
|
||||
("leg_right", "limb", 10, "body", [355, 470]),
|
||||
]
|
||||
if "tail" in character.get("body_type", "").lower() or "mouse" in cid:
|
||||
base_parts.append(("tail", "tail", 5, "body", [245, 425]))
|
||||
if any("wing" in a for a in actions) or "bird" in cid:
|
||||
base_parts.extend(
|
||||
[
|
||||
("wing_left", "wing", 30, "body", [275, 330]),
|
||||
("wing_right", "wing", 30, "body", [365, 330]),
|
||||
]
|
||||
)
|
||||
parts = [
|
||||
{
|
||||
"id": part_id,
|
||||
"kind": kind,
|
||||
"layer": layer,
|
||||
**({"parent": parent} if parent else {}),
|
||||
}
|
||||
for part_id, kind, layer, parent, _ in base_parts
|
||||
]
|
||||
joints = {
|
||||
part_id: {
|
||||
"pivot": pivot,
|
||||
"rotation": [-35, 35] if kind in {"head", "tail"} else [-75, 95],
|
||||
"scale": [0.8, 1.2],
|
||||
}
|
||||
for part_id, kind, _, _, pivot in base_parts
|
||||
}
|
||||
required_poses = sorted(
|
||||
set(["idle", "blink", "look_left", "look_right", "surprised"] + actions)
|
||||
)
|
||||
rig_characters.append(
|
||||
{
|
||||
"character_id": cid,
|
||||
"rig_type": "svg_rig",
|
||||
"parts": parts,
|
||||
"joints": joints,
|
||||
"layers": [p["id"] for p in sorted(parts, key=lambda p: p["layer"])],
|
||||
"views": character.get("required_views", ["front", "side"]),
|
||||
"required_poses": required_poses,
|
||||
"required_actions": actions,
|
||||
"risks": [
|
||||
"Generated pivots are first-pass estimates; review with preview frames.",
|
||||
],
|
||||
}
|
||||
)
|
||||
artifact = {"version": "1.0", "characters": rig_characters, "metadata": {"source": self.name}}
|
||||
artifacts = _write_json(inputs.get("output_path"), artifact)
|
||||
return ToolResult(
|
||||
success=True,
|
||||
data={"rig_plan": artifact},
|
||||
artifacts=artifacts,
|
||||
duration_seconds=round(time.time() - start, 2),
|
||||
)
|
||||
|
||||
|
||||
class PoseLibraryBuilder(BaseTool):
|
||||
name = "pose_library_builder"
|
||||
version = "0.1.0"
|
||||
tier = ToolTier.CORE
|
||||
capability = "character_animation"
|
||||
provider = "openmontage"
|
||||
stability = ToolStability.BETA
|
||||
execution_mode = ExecutionMode.SYNC
|
||||
determinism = Determinism.DETERMINISTIC
|
||||
resource_profile = ResourceProfile(cpu_cores=1, ram_mb=128, vram_mb=0, disk_mb=10)
|
||||
agent_skills = ["pose-library-design", "character-rigging", "svg-character-animation"]
|
||||
capabilities = ["draft_pose_library", "draft_action_cycles"]
|
||||
input_schema = {
|
||||
"type": "object",
|
||||
"required": ["rig_plan"],
|
||||
"properties": {"rig_plan": {"type": "object"}, "output_path": {"type": "string"}},
|
||||
}
|
||||
artifact_schema = {"artifact": "pose_library"}
|
||||
side_effects = ["optionally writes pose_library JSON to output_path"]
|
||||
|
||||
def execute(self, inputs: dict[str, Any]) -> ToolResult:
|
||||
start = time.time()
|
||||
characters = []
|
||||
for rig in inputs["rig_plan"].get("characters", []):
|
||||
cid = rig["character_id"]
|
||||
poses = {
|
||||
"idle": {"description": "Neutral readable hold", "parts": {}, "hold_frames": 24},
|
||||
"blink": {
|
||||
"description": "Quick eye close/open",
|
||||
"parts": {"eye_left": {"scaleY": 0.08}, "eye_right": {"scaleY": 0.08}},
|
||||
"hold_frames": 3,
|
||||
"transition": "power1.inOut",
|
||||
},
|
||||
"look_left": {
|
||||
"description": "Gaze shifts left",
|
||||
"parts": {"pupil_left": {"x": -6}, "pupil_right": {"x": -6}},
|
||||
"hold_frames": 18,
|
||||
},
|
||||
"look_right": {
|
||||
"description": "Gaze shifts right",
|
||||
"parts": {"pupil_left": {"x": 6}, "pupil_right": {"x": 6}},
|
||||
"hold_frames": 18,
|
||||
},
|
||||
"surprised": {
|
||||
"description": "Head lifts, eyes widen, mouth opens",
|
||||
"parts": {"head": {"y": -4, "rotation": -4}, "mouth": {"shape": "small_o"}},
|
||||
"expression": "surprised",
|
||||
"hold_frames": 24,
|
||||
"transition": "back.out",
|
||||
},
|
||||
}
|
||||
for action in rig.get("required_actions", []):
|
||||
poses.setdefault(
|
||||
action,
|
||||
{
|
||||
"description": f"First-pass pose for {action}",
|
||||
"parts": {},
|
||||
"hold_frames": 18,
|
||||
"transition": "power2.inOut",
|
||||
},
|
||||
)
|
||||
characters.append(
|
||||
{
|
||||
"character_id": cid,
|
||||
"poses": poses,
|
||||
"mouth_shapes": {
|
||||
"closed": {"description": "Neutral closed mouth"},
|
||||
"small_o": {"description": "Small open mouth for surprise or vowel"},
|
||||
"wide": {"description": "Wide open mouth"},
|
||||
"smile": {"description": "Smile shape"},
|
||||
},
|
||||
"action_cycles": {
|
||||
"walk": ["walk_contact", "walk_passing"],
|
||||
"breathe": ["idle"],
|
||||
},
|
||||
}
|
||||
)
|
||||
artifact = {"version": "1.0", "characters": characters, "metadata": {"source": self.name}}
|
||||
artifacts = _write_json(inputs.get("output_path"), artifact)
|
||||
return ToolResult(
|
||||
success=True,
|
||||
data={"pose_library": artifact},
|
||||
artifacts=artifacts,
|
||||
duration_seconds=round(time.time() - start, 2),
|
||||
)
|
||||
|
||||
|
||||
class ActionTimelineCompiler(BaseTool):
|
||||
name = "action_timeline_compiler"
|
||||
version = "0.1.0"
|
||||
tier = ToolTier.CORE
|
||||
capability = "character_animation"
|
||||
provider = "openmontage"
|
||||
stability = ToolStability.BETA
|
||||
execution_mode = ExecutionMode.SYNC
|
||||
determinism = Determinism.DETERMINISTIC
|
||||
resource_profile = ResourceProfile(cpu_cores=1, ram_mb=128, vram_mb=0, disk_mb=10)
|
||||
agent_skills = ["pose-library-design", "svg-character-animation", "gsap-timeline"]
|
||||
capabilities = ["compile_scene_actions", "draft_action_timeline"]
|
||||
input_schema = {
|
||||
"type": "object",
|
||||
"required": ["scene_plan"],
|
||||
"properties": {
|
||||
"scene_plan": {"type": "object"},
|
||||
"character_ids": {"type": "array", "items": {"type": "string"}},
|
||||
"fps": {"type": "number"},
|
||||
"output_path": {"type": "string"},
|
||||
},
|
||||
}
|
||||
artifact_schema = {"artifact": "action_timeline"}
|
||||
side_effects = ["optionally writes action_timeline JSON to output_path"]
|
||||
|
||||
def execute(self, inputs: dict[str, Any]) -> ToolResult:
|
||||
start = time.time()
|
||||
character_ids = inputs.get("character_ids") or ["main_character"]
|
||||
scenes = []
|
||||
for scene in inputs["scene_plan"].get("scenes", []):
|
||||
start_s = scene.get("start_seconds", 0)
|
||||
end_s = scene.get("end_seconds", start_s + 3)
|
||||
duration = max(0.1, end_s - start_s)
|
||||
actions = []
|
||||
for index, character_id in enumerate(character_ids):
|
||||
offset = min(duration * 0.08 * index, duration * 0.2)
|
||||
is_primary = index == 0
|
||||
actions.extend(
|
||||
[
|
||||
{
|
||||
"at_seconds": start_s + offset,
|
||||
"duration_seconds": min(0.5, duration / 4),
|
||||
"character_id": character_id,
|
||||
"action": "anticipate" if is_primary else "react",
|
||||
"pose": "idle",
|
||||
"easing": "power2.out",
|
||||
},
|
||||
{
|
||||
"at_seconds": start_s + duration * 0.25 + offset,
|
||||
"duration_seconds": duration * 0.35,
|
||||
"character_id": character_id,
|
||||
"action": "perform" if is_primary else "follow",
|
||||
"pose": (
|
||||
"surprised"
|
||||
if scene.get("hero_moment") or not is_primary
|
||||
else "look_right"
|
||||
),
|
||||
"easing": "back.out",
|
||||
"notes": scene.get("description", ""),
|
||||
},
|
||||
{
|
||||
"at_seconds": start_s + duration * 0.7 + offset,
|
||||
"duration_seconds": duration * 0.25,
|
||||
"character_id": character_id,
|
||||
"action": "settle",
|
||||
"pose": "idle",
|
||||
"easing": "power2.inOut",
|
||||
},
|
||||
]
|
||||
)
|
||||
scenes.append(
|
||||
{
|
||||
"scene_id": scene["id"],
|
||||
"start_seconds": start_s,
|
||||
"end_seconds": end_s,
|
||||
"camera": {"framing": scene.get("framing", "medium")},
|
||||
"background": scene.get("description", ""),
|
||||
"effects": [],
|
||||
"actions": actions,
|
||||
}
|
||||
)
|
||||
artifact = {
|
||||
"version": "1.0",
|
||||
"fps": inputs.get("fps", 30),
|
||||
"scenes": scenes,
|
||||
"metadata": {"source": self.name},
|
||||
}
|
||||
artifacts = _write_json(inputs.get("output_path"), artifact)
|
||||
return ToolResult(
|
||||
success=True,
|
||||
data={"action_timeline": artifact},
|
||||
artifacts=artifacts,
|
||||
duration_seconds=round(time.time() - start, 2),
|
||||
)
|
||||
|
||||
|
||||
class CharacterRigRenderer(BaseTool):
|
||||
name = "character_rig_renderer"
|
||||
version = "0.1.0"
|
||||
tier = ToolTier.CORE
|
||||
capability = "character_animation"
|
||||
provider = "openmontage"
|
||||
stability = ToolStability.BETA
|
||||
execution_mode = ExecutionMode.SYNC
|
||||
determinism = Determinism.DETERMINISTIC
|
||||
resource_profile = ResourceProfile(cpu_cores=1, ram_mb=128, vram_mb=0, disk_mb=50)
|
||||
agent_skills = [
|
||||
"character-rigging",
|
||||
"svg-character-animation",
|
||||
"canvas-procedural-animation",
|
||||
"gsap-core",
|
||||
"gsap-timeline",
|
||||
"remotion-best-practices",
|
||||
"hyperframes",
|
||||
]
|
||||
capabilities = ["write_browser_preview", "prepare_character_render_package"]
|
||||
input_schema = {
|
||||
"type": "object",
|
||||
"required": ["action_timeline"],
|
||||
"properties": {
|
||||
"action_timeline": {"type": "object"},
|
||||
"rig_plan": {"type": "object"},
|
||||
"pose_library": {"type": "object"},
|
||||
"output_path": {"type": "string"},
|
||||
"workspace_path": {"type": "string"},
|
||||
"video_output_path": {"type": "string"},
|
||||
"render_video": {"type": "boolean", "default": False},
|
||||
"duration_seconds": {"type": "number", "minimum": 0.1, "default": 3},
|
||||
"fps": {"type": "integer", "minimum": 1, "default": 12},
|
||||
},
|
||||
}
|
||||
output_schema = {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"preview_path": {"type": "string"},
|
||||
"hyperframes_workspace": {"type": "string"},
|
||||
"composition_path": {"type": "string"},
|
||||
"video_path": {"type": "string"},
|
||||
"asset_manifest": {"type": "object"},
|
||||
"edit_decisions": {"type": "object"},
|
||||
},
|
||||
}
|
||||
side_effects = [
|
||||
"writes a lightweight HTML preview to output_path",
|
||||
"writes a HyperFrames workspace/package",
|
||||
"optionally writes preview MP4",
|
||||
]
|
||||
user_visible_verification = ["Open preview and check character visibility and motion"]
|
||||
|
||||
def execute(self, inputs: dict[str, Any]) -> ToolResult:
|
||||
start = time.time()
|
||||
output_path = Path(inputs.get("output_path", "projects/character-preview/preview.html"))
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
timeline_json = json.dumps(inputs["action_timeline"])
|
||||
rig_characters = (inputs.get("rig_plan") or {}).get("characters", [])
|
||||
if not rig_characters:
|
||||
seen_ids = {
|
||||
action.get("character_id")
|
||||
for scene in inputs["action_timeline"].get("scenes", [])
|
||||
for action in scene.get("actions", [])
|
||||
if action.get("character_id")
|
||||
}
|
||||
rig_characters = [{"character_id": cid} for cid in sorted(seen_ids)] or [
|
||||
{"character_id": "main_character"}
|
||||
]
|
||||
count = len(rig_characters)
|
||||
spacing = 620 / max(count, 1)
|
||||
character_svgs = []
|
||||
for index, character in enumerate(rig_characters):
|
||||
cid = _slug(character.get("character_id", f"character-{index + 1}"))
|
||||
x = 110 + spacing * index if count > 1 else 320
|
||||
scale = 0.82 if count > 1 else 1
|
||||
body_fill, head_fill = _character_color(index)
|
||||
character_svgs.append(
|
||||
f"""
|
||||
<g class=\"character\" id=\"character_{cid}\" data-character=\"{cid}\" transform=\"translate({x - 320:.1f} 0) scale({scale})\">
|
||||
<ellipse class=\"shadow\" cx=\"320\" cy=\"560\" rx=\"120\" ry=\"22\" fill=\"rgba(0,0,0,.18)\" />
|
||||
<ellipse class=\"body outline\" cx=\"320\" cy=\"400\" rx=\"80\" ry=\"120\" fill=\"{body_fill}\" />
|
||||
<circle class=\"head outline\" cx=\"320\" cy=\"230\" r=\"90\" fill=\"{head_fill}\" />
|
||||
<ellipse class=\"eye eye-left outline\" cx=\"285\" cy=\"215\" rx=\"18\" ry=\"26\" fill=\"white\" />
|
||||
<ellipse class=\"eye eye-right outline\" cx=\"355\" cy=\"215\" rx=\"18\" ry=\"26\" fill=\"white\" />
|
||||
<circle class=\"pupil pupil-left\" cx=\"289\" cy=\"218\" r=\"8\" fill=\"#202632\" />
|
||||
<circle class=\"pupil pupil-right\" cx=\"359\" cy=\"218\" r=\"8\" fill=\"#202632\" />
|
||||
<path class=\"mouth outline\" d=\"M285 275 Q320 305 355 275\" fill=\"none\" />
|
||||
<path class=\"arm arm-left outline\" d=\"M255 360 C210 380 190 420 180 455\" fill=\"none\" />
|
||||
<path class=\"arm arm-right outline\" d=\"M385 360 C440 330 465 290 475 240\" fill=\"none\" />
|
||||
</g>"""
|
||||
)
|
||||
html = f"""<!doctype html>
|
||||
<html lang=\"en\">
|
||||
<head>
|
||||
<meta charset=\"utf-8\" />
|
||||
<meta name=\"viewport\" content=\"width=device-width, initial-scale=1\" />
|
||||
<title>Character Animation Preview</title>
|
||||
<script src=\"https://cdn.jsdelivr.net/npm/gsap@3.12.5/dist/gsap.min.js\"></script>
|
||||
<style>
|
||||
body {{ margin: 0; overflow: hidden; background: #9bd7ff; font-family: system-ui, sans-serif; }}
|
||||
#stage {{ width: 100vw; height: 100vh; display: grid; place-items: center; background: linear-gradient(#9bd7ff 0 65%, #75c878 65%); }}
|
||||
svg {{ width: min(82vw, 720px); overflow: visible; }}
|
||||
.outline {{ stroke: #202632; stroke-width: 7; stroke-linecap: round; stroke-linejoin: round; }}
|
||||
#note {{ position: fixed; left: 16px; bottom: 16px; background: white; border: 2px solid #202632; padding: 10px 12px; border-radius: 8px; }}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div id=\"stage\">
|
||||
<svg viewBox=\"0 0 640 640\" role=\"img\" aria-label=\"Character preview\">
|
||||
{''.join(character_svgs)}
|
||||
</svg>
|
||||
</div>
|
||||
<div id=\"note\">Local character preview. Characters: <span id=\"characters\"></span> · Scenes: <span id=\"count\"></span></div>
|
||||
<script>
|
||||
window.__ACTION_TIMELINE__ = {timeline_json};
|
||||
document.querySelector('#count').textContent = window.__ACTION_TIMELINE__.scenes.length;
|
||||
const characters = gsap.utils.toArray('.character');
|
||||
document.querySelector('#characters').textContent = characters.map((node) => node.dataset.character).join(', ');
|
||||
characters.forEach((node, index) => {{
|
||||
const q = gsap.utils.selector(node);
|
||||
gsap.set(q('.head'), {{ svgOrigin: '320 320' }});
|
||||
gsap.set(q('.arm-right'), {{ svgOrigin: '385 360' }});
|
||||
gsap.set(q('.arm-left'), {{ svgOrigin: '255 360' }});
|
||||
gsap.timeline({{ repeat: -1, defaults: {{ ease: 'power2.inOut' }}, delay: index * 0.12 }})
|
||||
.to(node, {{ y: -16, duration: 0.45 }})
|
||||
.to(node, {{ y: 0, duration: 0.45 }});
|
||||
gsap.timeline({{ repeat: -1, repeatDelay: 0.5, delay: index * 0.18 }})
|
||||
.to(q('.head'), {{ rotation: index % 2 ? 8 : -8, duration: 0.35 }})
|
||||
.to(q('.pupil'), {{ x: index % 2 ? -8 : 8, y: -3, duration: 0.2 }}, '<')
|
||||
.to(q('.arm-right'), {{ rotation: index % 2 ? -22 : 28, duration: 0.35 }}, '<')
|
||||
.to(q('.eye'), {{ scaleY: 0.08, transformOrigin: 'center', duration: 0.08 }})
|
||||
.to(q('.eye'), {{ scaleY: 1, duration: 0.1 }})
|
||||
.to(q('.head'), {{ rotation: index % 2 ? -6 : 6, duration: 0.35 }})
|
||||
.to(q('.pupil'), {{ x: index % 2 ? 6 : -6, y: 3, duration: 0.2 }}, '<')
|
||||
.to(q('.arm-right'), {{ rotation: index % 2 ? 8 : -8, duration: 0.35 }}, '<');
|
||||
}});
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
"""
|
||||
output_path.write_text(html, encoding="utf-8")
|
||||
total_duration = max(
|
||||
[
|
||||
float(scene.get("end_seconds", 0) or 0)
|
||||
for scene in inputs["action_timeline"].get("scenes", [])
|
||||
]
|
||||
or [float(inputs.get("duration_seconds", 3))]
|
||||
)
|
||||
workspace_path = Path(
|
||||
inputs.get("workspace_path")
|
||||
or output_path.parent / "hyperframes"
|
||||
)
|
||||
composition_dir = workspace_path / "compositions"
|
||||
composition_dir.mkdir(parents=True, exist_ok=True)
|
||||
(workspace_path / "assets").mkdir(parents=True, exist_ok=True)
|
||||
(workspace_path / "hyperframes.json").write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"registry": "https://raw.githubusercontent.com/heygen-com/hyperframes/main/registry",
|
||||
"paths": {
|
||||
"blocks": "compositions",
|
||||
"components": "compositions/components",
|
||||
"assets": "assets",
|
||||
},
|
||||
},
|
||||
indent=2,
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
(workspace_path / "DESIGN.md").write_text(
|
||||
"# DESIGN\n\n"
|
||||
"Generated for OpenMontage character animation.\n\n"
|
||||
"- Background: `#9bd7ff` sky and `#75c878` ground\n"
|
||||
"- Foreground: `#202632` ink outlines\n"
|
||||
"- Accent: saturated cartoon body colors\n"
|
||||
"- Motion: GSAP pose holds, squash/bounce, gaze, blink, and arm arcs\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
finite_bounce_repeats = max(0, int(total_duration / 0.9) - 1)
|
||||
finite_acting_repeats = max(0, int(total_duration / 2.1) - 1)
|
||||
composition_html = f"""<template id=\"character-scene-template\">
|
||||
<div data-composition-id=\"character-scene\" data-start=\"0\" data-duration=\"{total_duration:.3f}\" data-width=\"1280\" data-height=\"720\">
|
||||
<style>
|
||||
[data-composition-id=\"character-scene\"] {{ position: relative; width: 1280px; height: 720px; overflow: hidden; background: linear-gradient(#9bd7ff 0 65%, #75c878 65%); }}
|
||||
[data-composition-id=\"character-scene\"] svg {{ width: 920px; position: absolute; left: 180px; top: 42px; overflow: visible; }}
|
||||
[data-composition-id=\"character-scene\"] .outline {{ stroke: #202632; stroke-width: 7; stroke-linecap: round; stroke-linejoin: round; }}
|
||||
</style>
|
||||
<svg viewBox=\"0 0 640 640\" role=\"img\" aria-label=\"Character animation scene\">
|
||||
{''.join(character_svgs)}
|
||||
</svg>
|
||||
<script src=\"https://cdn.jsdelivr.net/npm/gsap@3.14.2/dist/gsap.min.js\"></script>
|
||||
<script>
|
||||
window.__timelines = window.__timelines || {{}};
|
||||
const tl = gsap.timeline({{ paused: true }});
|
||||
const characters = gsap.utils.toArray('[data-composition-id=\"character-scene\"] .character');
|
||||
characters.forEach((node, index) => {{
|
||||
const q = gsap.utils.selector(node);
|
||||
tl.set(q('.head'), {{ svgOrigin: '320 320' }}, 0);
|
||||
tl.set(q('.arm-right'), {{ svgOrigin: '385 360' }}, 0);
|
||||
tl.set(q('.arm-left'), {{ svgOrigin: '255 360' }}, 0);
|
||||
tl.from(node, {{ y: 26, scale: 0.94, opacity: 0, duration: 0.45, ease: 'back.out(1.8)' }}, 0.15 + index * 0.12);
|
||||
tl.to(node, {{ y: -16, duration: 0.45, repeat: {finite_bounce_repeats}, yoyo: true, ease: 'power2.inOut' }}, 0.7 + index * 0.08);
|
||||
tl.to(q('.head'), {{ rotation: index % 2 ? 8 : -8, duration: 0.35, repeat: {finite_acting_repeats}, yoyo: true, ease: 'sine.inOut' }}, 0.55 + index * 0.16);
|
||||
tl.to(q('.pupil'), {{ x: index % 2 ? -8 : 8, y: -3, duration: 0.2, repeat: {finite_acting_repeats}, yoyo: true, ease: 'power2.inOut' }}, 0.6 + index * 0.16);
|
||||
tl.to(q('.arm-right'), {{ rotation: index % 2 ? -22 : 28, duration: 0.35, repeat: {finite_acting_repeats}, yoyo: true, ease: 'back.inOut(1.4)' }}, 0.65 + index * 0.16);
|
||||
tl.to(q('.eye'), {{ scaleY: 0.08, transformOrigin: 'center', duration: 0.08, repeat: {finite_acting_repeats}, repeatDelay: 1.4, yoyo: true, ease: 'power1.inOut' }}, 1.1 + index * 0.12);
|
||||
}});
|
||||
window.__timelines['character-scene'] = tl;
|
||||
</script>
|
||||
</div>
|
||||
</template>
|
||||
"""
|
||||
composition_path = composition_dir / "character-scene.html"
|
||||
composition_path.write_text(composition_html, encoding="utf-8")
|
||||
asset_id = "character_scene_hyperframes"
|
||||
asset_manifest = {
|
||||
"version": "1.0",
|
||||
"assets": [
|
||||
{
|
||||
"id": asset_id,
|
||||
"type": "animation",
|
||||
"path": str(composition_path),
|
||||
"source_tool": self.name,
|
||||
"scene_id": "character_preview",
|
||||
"duration_seconds": total_duration,
|
||||
"format": "html",
|
||||
"generation_summary": "HyperFrames SVG/GSAP character composition package.",
|
||||
}
|
||||
],
|
||||
"total_cost_usd": 0,
|
||||
"metadata": {"source": self.name, "workspace_path": str(workspace_path)},
|
||||
}
|
||||
edit_decisions = {
|
||||
"version": "1.0",
|
||||
"render_runtime": "hyperframes",
|
||||
"renderer_family": "animation-first",
|
||||
"cuts": [
|
||||
{
|
||||
"id": "character-scene",
|
||||
"source": asset_id,
|
||||
"in_seconds": 0,
|
||||
"out_seconds": total_duration,
|
||||
"reason": "HyperFrames SVG/GSAP character scene generated by character_rig_renderer.",
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"proposal_render_runtime": "hyperframes",
|
||||
"title": "Character Animation",
|
||||
},
|
||||
}
|
||||
data: dict[str, Any] = {
|
||||
"preview_path": str(output_path),
|
||||
"render_package": "hyperframes_workspace",
|
||||
"hyperframes_workspace": str(workspace_path),
|
||||
"composition_path": str(composition_path),
|
||||
"asset_manifest": asset_manifest,
|
||||
"edit_decisions": edit_decisions,
|
||||
}
|
||||
artifacts = [str(output_path), str(workspace_path / "hyperframes.json"), str(composition_path)]
|
||||
render_video = bool(inputs.get("render_video") or inputs.get("video_output_path"))
|
||||
if render_video:
|
||||
video_path = Path(
|
||||
inputs.get("video_output_path")
|
||||
or output_path.with_suffix(".mp4")
|
||||
)
|
||||
video_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
duration_seconds = float(inputs.get("duration_seconds", 3))
|
||||
fps = int(inputs.get("fps", 12))
|
||||
_render_preview_mp4(output_path, video_path, duration_seconds, fps)
|
||||
video_asset_id = f"{output_path.stem}_preview_video"
|
||||
video_asset_manifest = {
|
||||
"version": "1.0",
|
||||
"assets": [
|
||||
{
|
||||
"id": video_asset_id,
|
||||
"type": "video",
|
||||
"path": str(video_path),
|
||||
"source_tool": self.name,
|
||||
"scene_id": "character_preview",
|
||||
"duration_seconds": duration_seconds,
|
||||
"format": "mp4",
|
||||
"generation_summary": "Rendered from local SVG/GSAP character preview via Playwright frame capture and ffmpeg.",
|
||||
}
|
||||
],
|
||||
"total_cost_usd": 0,
|
||||
"metadata": {"source": self.name, "preview_path": str(output_path)},
|
||||
}
|
||||
video_edit_decisions = {
|
||||
"version": "1.0",
|
||||
"render_runtime": "ffmpeg",
|
||||
"renderer_family": "animation-first",
|
||||
"cuts": [
|
||||
{
|
||||
"id": "character-preview-cut",
|
||||
"source": video_asset_id,
|
||||
"in_seconds": 0,
|
||||
"out_seconds": duration_seconds,
|
||||
"reason": "Local rendered character preview for video_compose handoff.",
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"proposal_render_runtime": "ffmpeg",
|
||||
"character_preview_path": str(output_path),
|
||||
},
|
||||
}
|
||||
data.update(
|
||||
{
|
||||
"video_path": str(video_path),
|
||||
"video_asset_manifest": video_asset_manifest,
|
||||
"video_edit_decisions": video_edit_decisions,
|
||||
}
|
||||
)
|
||||
artifacts.append(str(video_path))
|
||||
return ToolResult(
|
||||
success=True,
|
||||
data=data,
|
||||
artifacts=artifacts,
|
||||
duration_seconds=round(time.time() - start, 2),
|
||||
)
|
||||
|
||||
|
||||
class CharacterAnimationReviewer(BaseTool):
|
||||
name = "character_animation_reviewer"
|
||||
version = "0.1.0"
|
||||
tier = ToolTier.ANALYZE
|
||||
capability = "character_animation"
|
||||
provider = "openmontage"
|
||||
stability = ToolStability.BETA
|
||||
execution_mode = ExecutionMode.SYNC
|
||||
determinism = Determinism.DETERMINISTIC
|
||||
resource_profile = ResourceProfile(cpu_cores=1, ram_mb=128, vram_mb=0, disk_mb=10)
|
||||
agent_skills = ["character-animation-qa"]
|
||||
capabilities = ["review_character_artifacts", "draft_character_qa_report"]
|
||||
input_schema = {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"rig_plan": {"type": "object"},
|
||||
"pose_library": {"type": "object"},
|
||||
"action_timeline": {"type": "object"},
|
||||
"preview_path": {"type": "string"},
|
||||
"review_level": {"type": "string", "enum": ["static", "browser", "final"], "default": "static"},
|
||||
"browser_preview_checked": {"type": "boolean", "default": False},
|
||||
"frame_samples_checked": {"type": "boolean", "default": False},
|
||||
"output_path": {"type": "string"},
|
||||
},
|
||||
}
|
||||
artifact_schema = {"artifact": "character_qa_report"}
|
||||
side_effects = ["optionally writes character_qa_report JSON to output_path"]
|
||||
|
||||
def execute(self, inputs: dict[str, Any]) -> ToolResult:
|
||||
start = time.time()
|
||||
issues: list[str] = []
|
||||
rig = inputs.get("rig_plan") or {}
|
||||
poses = inputs.get("pose_library") or {}
|
||||
timeline = inputs.get("action_timeline") or {}
|
||||
preview_path = inputs.get("preview_path")
|
||||
review_level = inputs.get("review_level", "static")
|
||||
browser_preview_checked = bool(inputs.get("browser_preview_checked", False))
|
||||
frame_samples_checked = bool(inputs.get("frame_samples_checked", False))
|
||||
assets_exist = True
|
||||
if not preview_path:
|
||||
assets_exist = False
|
||||
issues.append("Preview path is required for character animation QA.")
|
||||
elif not Path(preview_path).exists():
|
||||
assets_exist = False
|
||||
issues.append(f"Preview path does not exist: {preview_path}")
|
||||
pivots_defined = all(
|
||||
bool(character.get("joints"))
|
||||
for character in rig.get("characters", [])
|
||||
) if rig else False
|
||||
poses_defined = all(
|
||||
bool(character.get("poses"))
|
||||
for character in poses.get("characters", [])
|
||||
) if poses else False
|
||||
actions_timed = all(
|
||||
bool(scene.get("actions"))
|
||||
for scene in timeline.get("scenes", [])
|
||||
) if timeline else False
|
||||
if not pivots_defined:
|
||||
issues.append("Rig plan is missing joints/pivots for one or more characters.")
|
||||
if not poses_defined:
|
||||
issues.append("Pose library is missing poses for one or more characters.")
|
||||
if not actions_timed:
|
||||
issues.append("Action timeline has scenes without timed actions.")
|
||||
schema_valid = True
|
||||
for artifact_name, artifact in [
|
||||
("rig_plan", rig),
|
||||
("pose_library", poses),
|
||||
("action_timeline", timeline),
|
||||
]:
|
||||
if not artifact:
|
||||
continue
|
||||
try:
|
||||
validate_artifact(artifact_name, artifact)
|
||||
except Exception as exc:
|
||||
schema_valid = False
|
||||
issues.append(f"{artifact_name} schema validation failed: {exc}")
|
||||
if review_level in {"browser", "final"} and not browser_preview_checked:
|
||||
issues.append("Browser preview check is required for browser/final QA.")
|
||||
if review_level == "final" and not frame_samples_checked:
|
||||
issues.append("Frame sample check is required for final QA.")
|
||||
status = "pass" if not issues else "revise"
|
||||
report = {
|
||||
"version": "1.0",
|
||||
"status": status,
|
||||
"preview_path": preview_path or "",
|
||||
"checks": {
|
||||
"schema_valid": schema_valid,
|
||||
"assets_exist": assets_exist,
|
||||
"pivots_defined": pivots_defined,
|
||||
"poses_defined": poses_defined,
|
||||
"actions_timed": actions_timed,
|
||||
"motion_detected": actions_timed,
|
||||
"browser_preview_checked": browser_preview_checked,
|
||||
"frame_samples_checked": frame_samples_checked,
|
||||
},
|
||||
"issues": issues,
|
||||
"recommended_action": "present_to_user" if status == "pass" else "fix_rig",
|
||||
"metadata": {
|
||||
"source": self.name,
|
||||
"confidence": "static artifact review; run Playwright/FFmpeg checks in compose for final output",
|
||||
},
|
||||
}
|
||||
artifacts = _write_json(inputs.get("output_path"), report)
|
||||
return ToolResult(
|
||||
success=True,
|
||||
data={"character_qa_report": report},
|
||||
artifacts=artifacts,
|
||||
duration_seconds=round(time.time() - start, 2),
|
||||
)
|
||||
@@ -1089,6 +1089,19 @@ class HyperFramesCompose(BaseTool):
|
||||
|
||||
# Unknown cut shape — render a placeholder text card so the render
|
||||
# still succeeds; lint/validate will surface the issue.
|
||||
if ext in {".html", ".htm"} and src_path:
|
||||
rel = self._rel_from_workspace(str(src_path))
|
||||
composition_id = Path(rel).stem
|
||||
html = (
|
||||
f'<div id="{cut_id}" class="clip composition-clip" '
|
||||
f'data-composition-id="{self._escape_attr(composition_id)}" '
|
||||
f'data-composition-src="{self._escape_attr(rel)}" '
|
||||
f'data-start="{self._f(in_s)}" data-duration="{self._f(duration)}" '
|
||||
f'data-width="{width}" data-height="{height}" '
|
||||
f'data-track-index="1"></div>'
|
||||
)
|
||||
return html, None
|
||||
|
||||
placeholder = self._escape_text(text or cut.get("reason") or f"Scene {index + 1}")
|
||||
html = (
|
||||
f'<div id="{cut_id}" class="clip text-card" '
|
||||
@@ -1182,5 +1195,10 @@ class HyperFramesCompose(BaseTool):
|
||||
# If it's already a relative path starting with assets/, keep as-is.
|
||||
if not p.is_absolute():
|
||||
return str(p).replace("\\", "/")
|
||||
parts = p.parts
|
||||
for anchor in ("assets", "compositions"):
|
||||
if anchor in parts:
|
||||
index = len(parts) - 1 - list(reversed(parts)).index(anchor)
|
||||
return "/".join(parts[index:])
|
||||
# Otherwise emit just the basename under assets/.
|
||||
return f"assets/{p.name}"
|
||||
|
||||
@@ -216,7 +216,7 @@ class SeedanceVideo(BaseTool):
|
||||
payload["image_url"] = inputs["image_url"]
|
||||
elif inputs.get("image_path"):
|
||||
from tools.video._shared import upload_image_fal
|
||||
payload["image_url"] = upload_image_to_fal(inputs["image_path"])
|
||||
payload["image_url"] = upload_image_fal(inputs["image_path"])
|
||||
if inputs.get("end_image_url"):
|
||||
payload["end_image_url"] = inputs["end_image_url"]
|
||||
|
||||
@@ -224,7 +224,7 @@ class SeedanceVideo(BaseTool):
|
||||
ref_image_urls = list(inputs.get("reference_image_urls") or [])
|
||||
for local_path in inputs.get("reference_image_paths") or []:
|
||||
from tools.video._shared import upload_image_fal
|
||||
ref_image_urls.append(upload_image_to_fal(local_path))
|
||||
ref_image_urls.append(upload_image_fal(local_path))
|
||||
# Seedance 2.0 reference-to-video ceilings: 9 images + 3 video + 3 audio.
|
||||
if len(ref_image_urls) > 9:
|
||||
return ToolResult(
|
||||
@@ -257,7 +257,7 @@ class SeedanceVideo(BaseTool):
|
||||
|
||||
try:
|
||||
submit_resp = requests.post(
|
||||
f"https://queue.fal.run/fal-ai/{model_path}",
|
||||
f"https://queue.fal.run/{model_path}",
|
||||
headers=headers,
|
||||
json=payload,
|
||||
timeout=30,
|
||||
@@ -305,7 +305,7 @@ class SeedanceVideo(BaseTool):
|
||||
success=True,
|
||||
data={
|
||||
"provider": "seedance",
|
||||
"model": f"fal-ai/{model_path}",
|
||||
"model": model_path,
|
||||
"prompt": inputs["prompt"],
|
||||
"operation": operation,
|
||||
"variant": variant,
|
||||
@@ -321,5 +321,5 @@ class SeedanceVideo(BaseTool):
|
||||
artifacts=[str(output_path)],
|
||||
cost_usd=self.estimate_cost(inputs),
|
||||
duration_seconds=round(time.time() - start, 2),
|
||||
model=f"fal-ai/{model_path}",
|
||||
model=model_path,
|
||||
)
|
||||
|
||||
@@ -105,6 +105,32 @@ class VideoCompose(BaseTool):
|
||||
"edit_decisions.metadata.proposal_render_runtime."
|
||||
),
|
||||
},
|
||||
"narration_transcript_path": {
|
||||
"type": "string",
|
||||
"description": (
|
||||
"Path to a word-level transcript JSON (from `transcriber` "
|
||||
"tool output). Optional but STRONGLY recommended: when "
|
||||
"combined with script_path/script_text, final_review "
|
||||
"runs transcript_comparison and catches TTS failures "
|
||||
"like 'Chirp3-HD reads ... as the word dot'. Without "
|
||||
"it, content-level audio bugs ship silently."
|
||||
),
|
||||
},
|
||||
"script_path": {
|
||||
"type": "string",
|
||||
"description": (
|
||||
"Path to the source narration script (plain text). "
|
||||
"Used by transcript_comparison to diff against the "
|
||||
"transcribed audio. Provide this OR script_text."
|
||||
),
|
||||
},
|
||||
"script_text": {
|
||||
"type": "string",
|
||||
"description": (
|
||||
"Inline source narration script. Used by "
|
||||
"transcript_comparison when a file path is unavailable."
|
||||
),
|
||||
},
|
||||
"subtitle_path": {"type": "string"},
|
||||
"subtitle_style": {
|
||||
"type": "object",
|
||||
@@ -1043,7 +1069,13 @@ class VideoCompose(BaseTool):
|
||||
# --- Post-render: mandatory final self-review ---
|
||||
if render_result.success and output_path.exists():
|
||||
final_review = self._run_final_review(
|
||||
output_path, edit_decisions, inputs.get("proposal_packet")
|
||||
output_path,
|
||||
edit_decisions,
|
||||
inputs.get("proposal_packet"),
|
||||
narration_transcript_path=inputs.get("narration_transcript_path"),
|
||||
script_text=inputs.get("script_text") or self._read_text_file(
|
||||
inputs.get("script_path")
|
||||
),
|
||||
)
|
||||
|
||||
# Attach final_review to the ToolResult data so the compose-director
|
||||
@@ -1160,7 +1192,13 @@ class VideoCompose(BaseTool):
|
||||
# Post-render: mandatory final self-review (identical contract to the Remotion path).
|
||||
if output_path.exists():
|
||||
final_review = self._run_final_review(
|
||||
output_path, edit_decisions, inputs.get("proposal_packet")
|
||||
output_path,
|
||||
edit_decisions,
|
||||
inputs.get("proposal_packet"),
|
||||
narration_transcript_path=inputs.get("narration_transcript_path"),
|
||||
script_text=inputs.get("script_text") or self._read_text_file(
|
||||
inputs.get("script_path")
|
||||
),
|
||||
)
|
||||
if render_result.data is None:
|
||||
render_result.data = {}
|
||||
@@ -1214,7 +1252,13 @@ class VideoCompose(BaseTool):
|
||||
|
||||
if render_result.success and output_path.exists():
|
||||
final_review = self._run_final_review(
|
||||
output_path, edit_decisions, inputs.get("proposal_packet")
|
||||
output_path,
|
||||
edit_decisions,
|
||||
inputs.get("proposal_packet"),
|
||||
narration_transcript_path=inputs.get("narration_transcript_path"),
|
||||
script_text=inputs.get("script_text") or self._read_text_file(
|
||||
inputs.get("script_path")
|
||||
),
|
||||
)
|
||||
if render_result.data is None:
|
||||
render_result.data = {}
|
||||
@@ -1353,11 +1397,156 @@ class VideoCompose(BaseTool):
|
||||
# Final self-review — mandatory post-render inspection
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
# Punctuation/SSML-leak words that should NEVER appear in rendered audio.
|
||||
# When a TTS engine reads a literal "..." as the word "dot", or a "—" as
|
||||
# "hyphen", those leak into the transcript. Catching these in the final
|
||||
# review is the difference between catching a bad voice render in-tool
|
||||
# vs. shipping a video that says "dot dot dot" twelve times. CRITICAL.
|
||||
_TTS_PUNCTUATION_LEAK_WORDS = {
|
||||
"dot", "dots", "ellipsis", "period", "periods",
|
||||
"comma", "commas", "semicolon", "colon",
|
||||
"dash", "hyphen", "emdash", "endash",
|
||||
"parenthesis", "bracket", "brace",
|
||||
"asterisk", "slash", "backslash",
|
||||
"exclamation", "question mark",
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def _read_text_file(path: str | Path | None) -> str | None:
|
||||
"""Read a small text file if given a path; None-safe and exception-safe."""
|
||||
if not path:
|
||||
return None
|
||||
try:
|
||||
return Path(path).read_text(encoding="utf-8")
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
@classmethod
|
||||
def _tokenize(cls, text: str) -> list[str]:
|
||||
"""Split text into comparable word tokens (lowercased, punctuation
|
||||
stripped, numeric-word-aware). Empty tokens dropped."""
|
||||
import re
|
||||
|
||||
# Preserve hyphenated words as single tokens ("many-worlds" -> "many-worlds").
|
||||
# Drop everything except letters, digits, hyphens, apostrophes.
|
||||
cleaned = re.sub(r"[^A-Za-z0-9\-' ]+", " ", text.lower())
|
||||
return [t for t in cleaned.split() if t and t != "-"]
|
||||
|
||||
@classmethod
|
||||
def _compare_transcript_to_script(
|
||||
cls,
|
||||
transcript_path: Path,
|
||||
script_text: str,
|
||||
) -> dict[str, Any]:
|
||||
"""Compare a word-level transcript against the source script.
|
||||
|
||||
Purpose: catch TTS failures that look fine on audio-volume/duration
|
||||
checks but produce garbage content. The canonical example is
|
||||
Chirp3-HD reading ellipses ("...") literally as the word "dot" — our
|
||||
volume check says "narration present, not clipped" and the video
|
||||
ships. This check diffs the actual transcribed audio against what
|
||||
was supposed to be said, and flags:
|
||||
|
||||
- Spurious punctuation-leak words ("dot", "comma", "hyphen", etc.)
|
||||
that appear in audio but not script → CRITICAL
|
||||
- Overall word-accuracy ratio against script → SUGGESTION if < 0.9
|
||||
|
||||
Returns the transcript_comparison section of final_review, or a
|
||||
placeholder with an issue describing why the check couldn't run
|
||||
(missing transcript, missing script) so the review never goes
|
||||
silently quiet on this contract.
|
||||
"""
|
||||
result: dict[str, Any] = {
|
||||
"transcript_matches_script": False,
|
||||
"word_accuracy": None,
|
||||
"script_word_count": 0,
|
||||
"transcript_word_count": 0,
|
||||
"spurious_punctuation_words": [],
|
||||
"issues": [],
|
||||
}
|
||||
|
||||
if not transcript_path or not Path(transcript_path).is_file():
|
||||
result["issues"].append(
|
||||
"transcript_comparison skipped: narration_transcript not provided"
|
||||
)
|
||||
return result
|
||||
if not script_text:
|
||||
result["issues"].append(
|
||||
"transcript_comparison skipped: script_text not provided"
|
||||
)
|
||||
return result
|
||||
|
||||
try:
|
||||
transcript_data = json.loads(Path(transcript_path).read_text(encoding="utf-8"))
|
||||
except Exception as e:
|
||||
result["issues"].append(f"transcript_comparison could not parse transcript: {e}")
|
||||
return result
|
||||
|
||||
transcript_words = [
|
||||
w.get("word", "").strip() for w in transcript_data.get("word_timestamps", [])
|
||||
]
|
||||
transcript_tokens = cls._tokenize(" ".join(transcript_words))
|
||||
script_tokens = cls._tokenize(script_text)
|
||||
|
||||
result["script_word_count"] = len(script_tokens)
|
||||
result["transcript_word_count"] = len(transcript_tokens)
|
||||
|
||||
if not script_tokens or not transcript_tokens:
|
||||
result["issues"].append(
|
||||
f"transcript_comparison: empty token set "
|
||||
f"(script={len(script_tokens)}, transcript={len(transcript_tokens)})"
|
||||
)
|
||||
return result
|
||||
|
||||
# --- Punctuation-leak detection (TTS reading literal punctuation) ---
|
||||
script_set = set(script_tokens)
|
||||
leak_occurrences: dict[str, int] = {}
|
||||
for token in transcript_tokens:
|
||||
if token in cls._TTS_PUNCTUATION_LEAK_WORDS and token not in script_set:
|
||||
leak_occurrences[token] = leak_occurrences.get(token, 0) + 1
|
||||
|
||||
if leak_occurrences:
|
||||
formatted = ", ".join(
|
||||
f"{w!r}×{n}" for w, n in sorted(leak_occurrences.items(), key=lambda x: -x[1])
|
||||
)
|
||||
result["spurious_punctuation_words"] = [
|
||||
{"word": w, "count": n} for w, n in leak_occurrences.items()
|
||||
]
|
||||
result["issues"].append(
|
||||
f"TTS punctuation leak: transcript contains {formatted} — "
|
||||
f"these words are NOT in the script, which means the voice "
|
||||
f"engine is reading literal punctuation aloud. Rewrite the "
|
||||
f"script to eliminate the corresponding characters (ellipses, "
|
||||
f"em-dashes, etc.) and regenerate narration."
|
||||
)
|
||||
|
||||
# --- Word accuracy via set overlap (cheap & ordering-insensitive) ---
|
||||
# We don't penalize small word-order differences or minor TTS
|
||||
# hallucinations; we just want to know "did 90%+ of the script's
|
||||
# content make it into the audio." Using set overlap on the script
|
||||
# side is robust to transcription noise.
|
||||
matched = sum(1 for t in script_tokens if t in set(transcript_tokens))
|
||||
accuracy = matched / max(1, len(script_tokens))
|
||||
result["word_accuracy"] = round(accuracy, 3)
|
||||
result["transcript_matches_script"] = accuracy >= 0.9 and not leak_occurrences
|
||||
|
||||
if accuracy < 0.9:
|
||||
result["issues"].append(
|
||||
f"Low transcript-to-script match: only {accuracy:.0%} of script "
|
||||
f"words appear in the transcribed audio ({matched}/"
|
||||
f"{len(script_tokens)}). Narration may be truncated, mispronounced, "
|
||||
f"or the wrong script was used."
|
||||
)
|
||||
|
||||
return result
|
||||
|
||||
def _run_final_review(
|
||||
self,
|
||||
output_path: Path,
|
||||
edit_decisions: dict[str, Any] | None = None,
|
||||
proposal_packet: dict[str, Any] | None = None,
|
||||
narration_transcript_path: str | Path | None = None,
|
||||
script_text: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Run post-render self-review and produce a final_review artifact.
|
||||
|
||||
@@ -1706,12 +1895,24 @@ class VideoCompose(BaseTool):
|
||||
|
||||
issues.extend(subtitle_check.get("issues", []))
|
||||
|
||||
# --- 6. Determine overall status ---
|
||||
# --- 6. Transcript-vs-script comparison ---
|
||||
# Catches content-level TTS failures (the classic "Chirp reads `...`
|
||||
# as the word 'dot'" trap) that volume-based audio checks miss.
|
||||
# Only runs when caller provides both the transcript and script; when
|
||||
# skipped, issues list records that so the silence is visible.
|
||||
transcript_comparison = self._compare_transcript_to_script(
|
||||
Path(narration_transcript_path) if narration_transcript_path else None,
|
||||
script_text,
|
||||
)
|
||||
issues.extend(transcript_comparison.get("issues", []))
|
||||
|
||||
# --- 7. Determine overall status ---
|
||||
critical_issues = [
|
||||
i for i in issues
|
||||
if any(kw in i.lower() for kw in [
|
||||
"silent downgrade", "delivery promise violation",
|
||||
"effectively silent", "ffprobe failed", "suspiciously short",
|
||||
"tts punctuation leak", # reading literal punctuation aloud
|
||||
])
|
||||
]
|
||||
|
||||
@@ -1739,6 +1940,7 @@ class VideoCompose(BaseTool):
|
||||
"audio_spotcheck": audio_spotcheck,
|
||||
"promise_preservation": promise_preservation,
|
||||
"subtitle_check": subtitle_check,
|
||||
"transcript_comparison": transcript_comparison,
|
||||
},
|
||||
"issues_found": issues,
|
||||
"recommended_action": recommended_action,
|
||||
|
||||
Reference in New Issue
Block a user