screen-demo: add synthetic-terminal mode via Remotion TerminalScene

Make synthetic screen recording a first-class, discoverable capability
alongside real OS capture. For CLI / terminal / install-flow demos where
commands and output are predictable, author a `terminal_scene` cut instead
of driving a real screen recorder — deterministic, privacy-safe, pixel-
perfect, and frame-accurate to narration cues.

Components:
- TerminalScene.tsx: window chrome, char-by-char typing, blinking cursor,
  scrolling output, non-blocking floating pills, spring-based reveals
- ProviderChip.tsx: rotating badge overlay that cycles through provider
  names (used in AI-generated-motion scenes)
- BackgroundVideoLayer + source_in_seconds + backgroundVideo props on
  every scene type — supports video-behind-component composition

Discovery chain (six layers, so the next agent finds this without reading
source code):

1. pipeline_defs/screen-demo.yaml — bump to 2.1, declare production_modes
   (real_capture, synthetic_terminal) with required_tools, scene_type, and
   agent_skills pointers
2. skills/pipelines/screen-demo/idea-director.md — mode-selection table
   at brief time; brief.metadata.production_mode contract
3. skills/pipelines/screen-demo/asset-director.md — reads production_mode
   and branches asset production (capture+overlays vs steps+narration+
   pacing check)
4. .agents/skills/synthetic-screen-recording/SKILL.md — Layer 3 skill with
   step kinds (cmd/out/pause/pill), pacing rule, and the frozen-terminal
   failure mode captured from the showcase v3 retune
5. AGENT_GUIDE.md — TerminalScene added to Remotion routing; links
   SCENE_TYPES.md as the authoritative cut-type registry
6. remotion-composer/SCENE_TYPES.md — new cheat sheet of every cut.type
   and overlay.type with required fields, plus a "how to add a new scene
   type" section (candidates: ChatTranscript, EditorScene, PrReview,
   SlackThread, TicketBoard)

Guardrail:
- lib/verify_scene_pacing.py — reusable trace() and assert_alignment()
  helpers that mimic the TerminalScene frame math exactly. Fail loudly
  before render if steps burn through too fast or leave the scene frozen.
This commit is contained in:
calesthio
2026-04-16 19:22:46 -07:00
parent 33ba37704e
commit a36ce99793
11 changed files with 918 additions and 26 deletions
@@ -0,0 +1,176 @@
# Synthetic Screen Recording (Remotion TerminalScene)
**Decision this skill answers:** When the user wants a screen-recording-looking demo of a terminal, CLI tool, or coding workflow — do I **capture the real desktop** (OS screen recording via `screen_recorder`, Windows-MCP, Cap, or Playwright), or do I **synthesize it in Remotion** with the `TerminalScene` component?
> **Heuristic:** If the agent can author the exact command/output sequence in advance, synthesize. Only capture live when the real behavior is unpredictable, needs a real app UI, or the user explicitly asked for a real recording.
## Why this exists
v3 of the OpenMontage showcase tried to use Windows-MCP + `screen_recorder` to drive a Git-Bash window for the install walkthrough. It stalled on window positioning, focus races, and taskbar privacy concerns. We pivoted to **pure Remotion rendering** — a React component named `TerminalScene` that draws a fake terminal and types commands character-by-character. The output is visually indistinguishable from a real screen recording (same traffic-light window chrome, blinking cursor, scrolling output) but deterministic, privacy-safe, pixel-perfect at 1080p, and pace-controllable to the frame.
That component + pattern is the capability this skill makes discoverable.
## When to use synthetic (TerminalScene)
**YES, synthesize when:**
- The demo is a **terminal / CLI / coding session** where commands and outputs are predictable
- The user wants a polished tutorial feel (clean typography, floating pills, cursor blink)
- Install walkthroughs, setup demos, API key config, `make` targets, `git clone` flows
- You need tight sync with narration — every command must land on a specific beat
- You want the result reproducible (re-render gets identical pixels)
- The user's actual desktop has private apps/windows visible you'd otherwise have to crop
**NO, capture a real screen when:**
- The demo is **a real app UI** that can't be faked (Figma, Photoshop, a web app with live state, a browser flow)
- The user explicitly asked for a recording of *their* actual screen
- The behavior depends on timing you can't script (streaming LLM output, real network latency)
- There's a visual quirk (a cursor effect, a plugin pop-up) that only appears in the live environment
**For a browser demo**`playwright-recording` skill, not this one.
**For a real desktop**`screen_recorder` tool or Cap via `cap_recorder`.
## The component — `TerminalScene`
Located at: `remotion-composer/src/components/TerminalScene.tsx`
Exported from: `remotion-composer/src/components/index.ts`
Wired in dispatch: `remotion-composer/src/Explainer.tsx` (`if (cut.type === "terminal_scene")`)
**Props:**
```ts
interface TerminalSceneProps {
title?: string; // shown in the window title bar
steps: TerminalStep[]; // the timeline
prompt?: string; // "$", ">", etc.
accentColor?: string; // pill + prompt glow
backgroundColor?: string;
}
```
**Step kinds:**
```ts
{ kind: "cmd", text: string, typeSpeed?: number, holdSeconds?: number }
{ kind: "out", text: string, holdSeconds?: number }
{ kind: "pause", seconds: number }
{ kind: "pill", text: string, color?: string, durationSeconds?: number }
```
- `cmd` — prints the prompt, types the text character-by-character (`typeSpeed` is seconds per character, default 0.035), then holds for `holdSeconds` (default 0.3)
- `out` — a line of program output, reveals instantly with a short fade-in
- `pause` — dead time. Terminal holds on last visible state. USE THIS TO SYNC WITH NARRATION.
- `pill` — non-blocking floating badge (top-right). Spring-in, hold, spring-out. Does NOT advance the cursor — the next step runs in parallel.
## Authoring pattern
Author a new scene by adding a cut to `build_composition.py` (or your equivalent props builder):
```python
install_steps = [
{"kind": "pause", "seconds": 7.0}, # wait for intro narration
{"kind": "cmd", "text": "git clone https://github.com/calesthio/OpenMontage.git",
"typeSpeed": 0.045, "holdSeconds": 0.3},
{"kind": "out", "text": "Cloning into 'OpenMontage'..."},
{"kind": "out", "text": "remote: Enumerating objects: 2847, done."},
{"kind": "pill", "text": "repo cloned", "color": "#34D399", "durationSeconds": 2.6},
{"kind": "pause", "seconds": 3.8}, # bridge to next narration cue
# ...
]
cuts.append({
"id": "install-terminal",
"type": "terminal_scene",
"terminalTitle": "bash — OpenMontage setup",
"prompt": "$",
"accentColor": "#22D3EE",
"steps": install_steps,
"in_seconds": 50.0,
"out_seconds": 110.0,
})
```
## THE RULE: pace with narration, never ahead
**The #1 failure mode:** steps run continuously and burn through all content in the first 40% of the scene, leaving the terminal frozen for the remaining 60%. This is what killed the v3 first pass — the capability menu rendered at t=80s but narration didn't announce it until t=92s.
**Do this instead:**
1. **Know your narration cues** — for each scene, write down the exact video-time each narration segment starts.
2. **Start with a pause** that reaches the first narration cue before any command types.
3. **Time each command to land with its narration line**`cmd` should start typing the moment narration says its line, not before.
4. **Put pauses between command groups** that bridge to the next narration cue.
5. **End with a closer hold** — a pause long enough that the final state is readable after narration ends.
**Sanity-check your steps before rendering** — every minute of Remotion render is precious. Sum the step durations and verify they equal scene duration:
```python
import math
def trace(steps, scene_start, fps=30):
t = 0.0
for s in steps:
k = s["kind"]
if k == "cmd":
tf = math.ceil(len(s["text"]) * s.get("typeSpeed", 0.035) * fps)
t += tf / fps + s.get("holdSeconds", 0.3)
elif k == "out":
t += max(2, math.ceil(0.08 * fps)) / fps + s.get("holdSeconds", 0.15)
elif k == "pause":
t += s["seconds"]
# "pill" is non-blocking — does NOT advance cursor
print(f" {t + scene_start:6.2f}s {k}: {s.get('text', '')[:40]}")
trace(install_steps, 50)
```
Look at the output column. Each narration cue's video-time must appear adjacent to the command/output it announces. If a command lands 10s before or after its cue, adjust pauses.
See `lib/verify_scene_pacing.py` for a reusable version of this script.
## Design rules (inherited from the v3 retune)
- **Intro pause** — every terminal scene opens with at least 2s of empty-terminal-with-blinking-cursor before anything types. The viewer needs to register the window.
- **Pill timing** — a pill should fire at the exact moment its named event completes on screen (e.g., `repo cloned` immediately after the last `Receiving objects` line). Pills are your substitute for real-world UI notifications.
- **Command hold after typing** — keep `holdSeconds` ≥ 0.3 on every `cmd` so viewers register the completed command before the first output scrolls in.
- **Output cadence** — space `holdSeconds` on output lines between 0.4 and 1.0. Output that flies too fast feels like a bug; output that crawls feels boring.
- **Auto-scroll works** — the terminal holds the most recent 18 lines. Don't worry about off-screen content.
- **Cursor blinks only on the latest command line while typing + a ~0.2s tail** after typing completes.
## `ProviderChip` (companion component)
The `.agents/skills/synthetic-screen-recording` pattern also owns `ProviderChip` — a rotating badge overlay that cycles through a list of provider names at a fixed cadence. Used in the v3 showcase to cycle through all 11 AI video-gen providers during the "generated motion" section.
```python
overlays.append({
"type": "provider_chip",
"providers": ["Veo 3.1", "Seedance 2.0", "Kling 2.5", ...],
"cycleSeconds": 2.5,
"position": "bottom-right",
"accentColor": "#22D3EE",
"label": "generated with",
"in_seconds": 195.0,
"out_seconds": 222.5,
})
```
Wired in dispatch at: `remotion-composer/src/Explainer.tsx` overlay renderer (`overlay.type === "provider_chip"`).
## Adding new synthetic-UI components
The pattern generalizes. When you need to fake another UI surface (Claude Code chat bubbles, a Jira ticket view, a GitHub PR diff, a Slack message, a VS Code status bar):
1. Copy `TerminalScene.tsx` as a template.
2. Define a `steps` interface for the relevant timeline primitives.
3. Render each step by interpolating `frame` against cumulative start/end times.
4. Wire it into `Explainer.tsx`'s `SceneRenderer` dispatch with a new `cut.type`.
5. Add the type to the `Cut` interface in `Explainer.tsx` and to `components/index.ts`.
6. Add a section to this skill documenting it.
7. Update `remotion-composer/SCENE_TYPES.md` with the new cut type.
## Related skills
- `.agents/skills/remotion` — general Remotion authoring (hooks, springs, sequences)
- `.agents/skills/playwright-recording` — real browser-flow capture for web apps
- `tools/capture/screen_recorder` — ffmpeg-based desktop capture
- `tools/capture/cap_recorder` — Cap.so polished desktop capture
- `skills/pipelines/screen-demo/asset-director.md` — chooses between synthetic and real for a screen-demo project
## Provenance
Introduced: OpenMontage showcase v3 render (2026-04-16). Original motivation: the v3 setup walkthrough section needed a 60-second install demo where every command aligned to Chirp 3 HD narration cues, and Windows-MCP-driven real capture was too flaky in practice. See `projects/openmontage-showcase/build_composition.py` for the reference implementation.
+6
View File
@@ -356,6 +356,12 @@ For these requests:
- Explainer videos with `flat-motion-graphics` playbook -> Remotion animated scenes, not Ken Burns
- Data-driven videos -> Remotion stat cards and charts, not static image screenshots
- Any pipeline using still images -> Remotion spring animations, not FFmpeg pan-and-zoom
- **Screen demos of a CLI/terminal/install flow -> `TerminalScene` (synthetic screen recording), not OS-level capture.** See `.agents/skills/synthetic-screen-recording/SKILL.md`. Faster, deterministic, privacy-safe. Use real capture (`screen_recorder`, `cap_recorder`, `playwright-recording`) only when the demo is a real app UI or requires unpredictable live behavior.
### Remotion scene types available in `remotion-composer/`
See `remotion-composer/SCENE_TYPES.md` for the authoritative list and their cut schemas. Current scene types usable via `cut.type`:
`text_card`, `stat_card`, `callout`, `comparison`, `hero_title`, `terminal_scene`, `anime_scene`, `bar_chart`, `line_chart`, `pie_chart`, `kpi_grid`, `progress_bar`. Overlay types include `section_title`, `stat_reveal`, `hero_title`, `provider_chip`.
**When Remotion is NOT available**, `video_compose` falls back to FFmpeg Ken Burns motion on still images. This still works but produces less engaging visuals. Mention this tradeoff in the proposal.
+134
View File
@@ -0,0 +1,134 @@
"""Verify that a TerminalScene `steps` list paces with narration cues.
Use from any build_composition.py or synthetic-UI builder:
from lib.verify_scene_pacing import trace, assert_alignment
trace(install_steps, scene_start=50.0)
assert_alignment(
install_steps,
scene_start=50.0,
scene_end=110.0,
narration_cues=[
(57.0, "seg07 Clone the repo"),
(65.5, "seg08 Run make setup"),
(83.0, "seg09 Open the folder"),
(92.0, "seg10 agent reads guide"),
],
tolerance=1.0,
)
The tracer mimics the frame math inside TerminalScene.tsx so video-time
estimates are exact to 1/fps. Fails loudly if any narration cue has no
matching command/output within `tolerance` seconds.
"""
from __future__ import annotations
import math
from dataclasses import dataclass
from typing import Any
def step_duration(step: dict[str, Any], fps: int = 30) -> float:
"""Return the cursor-advancement for a single step (frame-accurate).
Pills DO NOT advance the cursor — they're non-blocking overlays.
"""
k = step["kind"]
if k == "cmd":
type_frames = math.ceil(len(step["text"]) * step.get("typeSpeed", 0.035) * fps)
return type_frames / fps + step.get("holdSeconds", 0.3)
if k == "out":
reveal_frames = max(2, math.ceil(0.08 * fps))
return reveal_frames / fps + step.get("holdSeconds", 0.15)
if k == "pause":
return float(step["seconds"])
if k == "pill":
return 0.0
raise ValueError(f"Unknown step kind: {k!r}")
@dataclass
class Landmark:
video_time: float
kind: str
text: str
def trace(steps: list[dict[str, Any]], scene_start: float = 0.0, fps: int = 30, *, quiet: bool = False) -> list[Landmark]:
"""Walk the step list and print a video-time landmark for each visible event.
Returns the list of landmarks (useful for alignment checks).
"""
cursor = 0.0
out: list[Landmark] = []
for s in steps:
k = s["kind"]
vt = round(cursor + scene_start, 2)
if k in ("cmd", "out", "pill"):
text = s.get("text", "")
out.append(Landmark(video_time=vt, kind=k.upper(), text=text))
if not quiet:
prefix = {"CMD": "CMD ", "OUT": "OUT ", "PILL": "PILL "}[k.upper()]
print(f" {vt:7.2f}s {prefix}{text[:60]}")
cursor += step_duration(s, fps)
end_vt = round(cursor + scene_start, 2)
if not quiet:
print(f" {end_vt:7.2f}s -- steps end --")
return out
def assert_alignment(
steps: list[dict[str, Any]],
scene_start: float,
scene_end: float,
narration_cues: list[tuple[float, str]],
*,
tolerance: float = 1.0,
fps: int = 30,
) -> None:
"""Validate that every narration cue has a visual landmark within tolerance.
Also checks that total step duration does not overflow scene_end.
Raises AssertionError on any mismatch.
"""
landmarks = trace(steps, scene_start, fps, quiet=True)
errors: list[str] = []
for cue_time, cue_desc in narration_cues:
# Find closest landmark by video-time
if not landmarks:
errors.append(f"cue {cue_time:.2f}s ({cue_desc}): no landmarks at all")
continue
closest = min(landmarks, key=lambda lm: abs(lm.video_time - cue_time))
delta = closest.video_time - cue_time
if abs(delta) > tolerance:
errors.append(
f"cue {cue_time:.2f}s ({cue_desc}) has no visual within ±{tolerance:.1f}s — "
f"closest is {closest.kind} at {closest.video_time:.2f}s ({delta:+.2f}s off): {closest.text[:40]}"
)
# Overflow check
cursor = sum(step_duration(s, fps) for s in steps)
end_vt = scene_start + cursor
scene_duration = scene_end - scene_start
if cursor > scene_duration + 0.5:
errors.append(
f"steps overflow scene: cursor ends at {end_vt:.2f}s but scene_end is {scene_end:.2f}s "
f"(overflow {cursor - scene_duration:.2f}s)"
)
if cursor < scene_duration - 5.0:
errors.append(
f"steps underfill scene by {scene_duration - cursor:.2f}s — last visible step holds "
f"frozen from {end_vt:.2f}s to {scene_end:.2f}s. Add a closer pause."
)
if errors:
raise AssertionError(
"Scene pacing check failed:\n - " + "\n - ".join(errors)
)
__all__ = ["step_duration", "trace", "assert_alignment", "Landmark"]
+36 -5
View File
@@ -1,14 +1,45 @@
name: screen-demo
version: "2.0"
version: "2.1"
description: >
Screen recording pipeline. Takes raw screen capture (app demo, browser walkthrough,
coding tutorial) and produces a polished demo video with callouts, zoom crops,
subtitles, and cleaned audio. EP orchestration adds quality gates for legibility,
audio clarity, and pacing.
Screen recording pipeline. Two production modes:
1. REAL CAPTURE — takes raw screen capture (app demo, browser walkthrough,
coding tutorial) via screen_recorder, cap_recorder, or playwright-recording
and produces a polished demo video with callouts, zoom crops, subtitles,
and cleaned audio.
2. SYNTHETIC (Remotion TerminalScene) — for CLI/terminal/install-flow demos,
renders a deterministic terminal animation with typed commands, scrolling
output, floating command pills, and blinking cursor. See
.agents/skills/synthetic-screen-recording/SKILL.md. Preferred when the
demo content is predictable (install walkthroughs, make targets, git clone,
API key config) because it's faster to iterate, privacy-safe, pixel-perfect,
and frame-accurate to narration cues.
The idea-director picks the mode at brief time based on whether the demo surface
is a real app UI (real capture) or a scriptable terminal session (synthetic).
category: screen_recording
stability: production
default_checkpoint_policy: guided
# Production modes this pipeline supports
production_modes:
- name: real_capture
description: "OS screen recording of a real desktop/browser session"
required_tools: [screen_recorder]
optional_tools: [cap_recorder]
best_for:
- Real app UIs (Figma, Photoshop, web apps with live state)
- Demos that depend on unpredictable live behavior
- User explicitly asked for a recording of their own screen
- name: synthetic_terminal
description: "Remotion TerminalScene — synthesized terminal animation"
required_tools: [video_compose] # Remotion path
scene_type: terminal_scene
agent_skills: [synthetic-screen-recording]
best_for:
- CLI / terminal / coding session demos
- Install walkthroughs, setup demos, config flows
- Tight narration-synced pacing where every command lands on a beat
- Reproducible renders (same input = identical pixels)
orchestration:
mode: executive-producer
skill: pipelines/screen-demo/executive-producer
+64
View File
@@ -0,0 +1,64 @@
# Remotion Composer — Scene & Overlay Cheat Sheet
Authoritative list of `cut.type` and `overlay.type` values the `Explainer` composition accepts. Each row maps to a dispatch case in `src/Explainer.tsx`.
When you add a new component, append it here and in `src/components/index.ts`.
---
## Cut types (`cut.type`)
| `type` | Component | Required fields | Common fields | Purpose |
|---|---|---|---|---|
| *(none — video)* | `OffthreadVideo` | `source` (path to mp4) | `source_in_seconds`, `animation` (zoom-in, ken-burns), `in_seconds`, `out_seconds` | Play an MP4 clip directly |
| *(none — image)* | `Img` | `source` (path to png/jpg) | `animation`, `in_seconds`, `out_seconds` | Play a still with Ken Burns |
| `text_card` | `TextCard` | `text` | `fontSize`, `backgroundVideo`, `backgroundOverlay`, `color` | Large-typography beat |
| `hero_title` | `HeroTitle` | `text` | `heroSubtitle`, `backgroundVideo`, `backgroundOverlay` | Title/end card |
| `stat_card` | `StatCard` | `stat` | `subtitle`, `accentColor`, `backgroundVideo` | A single big number |
| `callout` | `CalloutBox` | `text` | `callout_type` (info/warning/tip/quote), `title`, `backgroundVideo` | Boxed message with bullets |
| `comparison` | `ComparisonCard` | `leftLabel`, `leftValue`, `rightLabel`, `rightValue` | `title`, `backgroundColor` | Side-by-side compare |
| `bar_chart` | `BarChart` | `chartData` | `chartAnimation`, `showValues`, `showGrid`, `backgroundVideo` | Animated bars |
| `line_chart` | `LineChart` | `chartSeries` | `chartAnimation`, `xLabel`, `yLabel`, `showMarkers` | Animated line |
| `pie_chart` | `PieChart` | `chartData` | `donut`, `centerLabel`, `centerValue`, `showLegend` | Pie / donut |
| `kpi_grid` | `KPIGrid` | `chartData` | `title`, `columns`, `chartAnimation` | 24 column KPI grid |
| `progress_bar` | `ProgressBar` | `progress` | `progressLabel`, `progressColor`, `progressSegments` | Animated progress |
| `anime_scene` | `AnimeScene` | `images` (list) | `particles`, `lightingFrom`, `lightingTo`, `vignette` | Still-image anime scene with particles + camera motion |
| **`terminal_scene`** | **`TerminalScene`** | **`steps`** (list of cmd/out/pause/pill) | **`terminalTitle`, `prompt`, `accentColor`** | **Synthetic terminal animation — NO real capture needed. See [`.agents/skills/synthetic-screen-recording/SKILL.md`](../.agents/skills/synthetic-screen-recording/SKILL.md)** |
---
## Overlay types (`overlay.type`)
| `type` | Component | Required fields | Common fields | Purpose |
|---|---|---|---|---|
| `section_title` | `SectionTitle` | `text` | `accentColor`, `position` (top-left, etc.) | Tiny section label |
| `stat_reveal` | `StatReveal` | `text` | `subtitle`, `accentColor`, `position` | Corner stat badge |
| `hero_title` | `HeroTitle` (as overlay) | `text` | `subtitle` | Full-frame title overlay |
| **`provider_chip`** | **`ProviderChip`** | **`providers`** (list of strings) | **`cycleSeconds`, `position`, `accentColor`, `label`** | **Rotating badge that cycles through provider names — used in AI-generated-motion scenes to show which model produced the clip** |
---
## Adding a new scene type
1. Create the React component in `src/components/MyScene.tsx`. Use `interpolate(frame, [inFrame, outFrame], [from, to])` and `spring(...)` for motion. Read `useCurrentFrame()` and `useVideoConfig()`.
2. Export it in `src/components/index.ts`.
3. Add the `type` to the `Cut` interface in `src/Explainer.tsx` (and any new prop fields).
4. Add a dispatch case in `SceneRenderer`:
```tsx
if (cut.type === "my_scene" && cut.mySceneData) {
return maybeWrapWithBg(<MyScene ... />);
}
```
5. Document it in this file. That's what makes it discoverable to the next agent.
## Existing synthetic-UI components
Currently only `TerminalScene` exists. The pattern generalizes — likely candidates to add next, if a pipeline needs them:
- `ChatTranscript` — Claude/Cursor/GPT chat-bubble timeline with typing animation
- `EditorScene` — VS Code-style code editor with syntax highlight + cursor motion
- `PrReview` — GitHub PR diff view with inline-comment reveals
- `SlackThread` — Slack thread with avatars + reaction pops
- `TicketBoard` — Jira / Linear card moving across columns
Pattern: follow `TerminalScene.tsx` — a `steps` list of timeline primitives, cursor-advancing durations, spring-based reveals, optional non-blocking pills/badges.
+102 -19
View File
@@ -41,6 +41,9 @@ import { StatReveal } from "./components/StatReveal";
import { HeroTitle } from "./components/HeroTitle";
import { AnimeScene } from "./components/AnimeScene";
import type { CameraMotion } from "./components/AnimeScene";
import { TerminalScene } from "./components/TerminalScene";
import type { TerminalStep } from "./components/TerminalScene";
import { ProviderChip } from "./components/ProviderChip";
import type { ParticleType } from "./components/ParticleOverlay";
import { resolveTheme, type ThemeConfig, DEFAULT_THEME } from "./Root";
@@ -197,6 +200,9 @@ interface Cut {
subtitle?: string;
callout_type?: "info" | "warning" | "tip" | "quote";
title?: string;
// Video source trim — seek to this point in the source before playback.
// Defaults to 0 (play from beginning). Use this instead of in_seconds for source trimming.
source_in_seconds?: number;
// Comparison props
leftLabel?: string;
rightLabel?: string;
@@ -228,7 +234,9 @@ interface Cut {
// Styling overrides
backgroundColor?: string;
backgroundImage?: string; // AI-generated or stock image rendered behind the component
backgroundOverlay?: number; // Opacity of dark overlay on backgroundImage (0-1, default 0.55)
backgroundVideo?: string; // Video clip rendered behind the component (takes priority over backgroundImage)
backgroundVideoStart?: number; // Seek position in seconds for background video (default 0)
backgroundOverlay?: number; // Opacity of dark overlay on backgroundImage/backgroundVideo (0-1, default 0.55)
color?: string;
accentColor?: string;
fontSize?: number;
@@ -250,16 +258,24 @@ interface Cut {
vignette?: boolean;
lightingFrom?: string;
lightingTo?: string;
// Terminal scene props (type: "terminal_scene")
steps?: TerminalStep[];
terminalTitle?: string;
prompt?: string;
}
interface Overlay {
type: "section_title" | "stat_reveal" | "hero_title";
type: "section_title" | "stat_reveal" | "hero_title" | "provider_chip";
in_seconds: number;
out_seconds: number;
text: string;
text?: string;
subtitle?: string;
accentColor?: string;
position?: string;
// provider_chip
providers?: string[];
cycleSeconds?: number;
label?: string;
}
interface AudioLayer {
@@ -472,9 +488,54 @@ const BackgroundImageLayer: React.FC<{
);
};
// Background video layer — plays a looping video behind component content with dark overlay
const BackgroundVideoLayer: React.FC<{
src: string;
startFrom?: number;
overlayOpacity?: number;
children: React.ReactNode;
}> = ({ src, startFrom = 0, overlayOpacity = 0.55, children }) => {
const { fps } = useVideoConfig();
return (
<AbsoluteFill style={{ overflow: "hidden" }}>
{/* Background video */}
<OffthreadVideo
src={resolveAsset(src)}
startFrom={Math.round(startFrom * fps)}
style={{
width: "100%",
height: "100%",
objectFit: "cover",
}}
muted
/>
{/* Dark overlay for readability */}
<AbsoluteFill
style={{
background: `rgba(15, 23, 42, ${overlayOpacity})`,
}}
/>
{/* Component content on top */}
{children}
</AbsoluteFill>
);
};
const SceneRenderer: React.FC<{ cut: Cut; theme: ThemeConfig }> = ({ cut, theme }) => {
// Wrap component with background image if specified
const maybeWrapWithBgImage = (element: React.ReactElement) => {
// Wrap component with background video or image if specified
const maybeWrapWithBg = (element: React.ReactElement) => {
if (cut.backgroundVideo) {
return (
<BackgroundVideoLayer
src={cut.backgroundVideo}
startFrom={cut.backgroundVideoStart ?? 0}
overlayOpacity={cut.backgroundOverlay ?? 0.55}
>
{element}
</BackgroundVideoLayer>
);
}
if (cut.backgroundImage) {
return (
<BackgroundImageLayer
@@ -491,24 +552,24 @@ const SceneRenderer: React.FC<{ cut: Cut; theme: ThemeConfig }> = ({ cut, theme
// Resolve the scene element based on cut type, then wrap with backgroundImage if set
// Use transparent bg so the animated gradient background shows through
// When no explicit backgroundColor on the cut, inherit from theme
const rawBg = cut.backgroundImage ? "transparent" : (cut.backgroundColor || theme.surfaceColor);
const rawBg = (cut.backgroundImage || cut.backgroundVideo) ? "transparent" : (cut.backgroundColor || theme.surfaceColor);
const bgColor = (rawBg === theme.backgroundColor || rawBg === "#0F172A" || rawBg === "#0f172a") ? "transparent" : rawBg;
const textColor = cut.color || theme.textColor;
const accent = cut.accentColor || theme.accentColor;
// Explicit component types — use theme-derived defaults for colors
if (cut.type === "text_card" && cut.text) {
return maybeWrapWithBgImage(
return maybeWrapWithBg(
<TextCard text={cut.text} fontSize={cut.fontSize} color={textColor} backgroundColor={bgColor} />
);
}
if (cut.type === "stat_card" && cut.stat) {
return maybeWrapWithBgImage(
return maybeWrapWithBg(
<StatCard stat={cut.stat} subtitle={cut.subtitle} accentColor={accent} backgroundColor={bgColor} />
);
}
if (cut.type === "callout" && cut.text) {
return maybeWrapWithBgImage(
return maybeWrapWithBg(
<CalloutBox
text={cut.text} type={cut.callout_type} title={cut.title}
borderColor={accent} backgroundColor={cut.backgroundColor || theme.surfaceColor}
@@ -517,7 +578,7 @@ const SceneRenderer: React.FC<{ cut: Cut; theme: ThemeConfig }> = ({ cut, theme
);
}
if (cut.type === "comparison" && cut.leftLabel && cut.rightLabel && cut.leftValue && cut.rightValue) {
return maybeWrapWithBgImage(
return maybeWrapWithBg(
<ComparisonCard
leftLabel={cut.leftLabel} rightLabel={cut.rightLabel}
leftValue={cut.leftValue} rightValue={cut.rightValue}
@@ -526,14 +587,25 @@ const SceneRenderer: React.FC<{ cut: Cut; theme: ThemeConfig }> = ({ cut, theme
);
}
if (cut.type === "hero_title" && cut.text) {
return maybeWrapWithBgImage(
return maybeWrapWithBg(
<HeroTitle title={cut.text} subtitle={cut.heroSubtitle || cut.subtitle} />
);
}
if (cut.type === "terminal_scene" && cut.steps) {
return maybeWrapWithBg(
<TerminalScene
title={cut.terminalTitle || "Terminal"}
steps={cut.steps as TerminalStep[]}
prompt={cut.prompt}
accentColor={accent}
backgroundColor={bgColor || theme.backgroundColor}
/>
);
}
// --- Chart types — use theme.chartColors as default palette ---
if (cut.type === "bar_chart" && cut.chartData) {
return maybeWrapWithBgImage(
return maybeWrapWithBg(
<BarChart
data={cut.chartData} title={cut.title} colors={cut.chartColors || theme.chartColors}
animationStyle={(cut.chartAnimation as any) || "grow-up"}
@@ -542,7 +614,7 @@ const SceneRenderer: React.FC<{ cut: Cut; theme: ThemeConfig }> = ({ cut, theme
);
}
if (cut.type === "line_chart" && cut.chartSeries) {
return maybeWrapWithBgImage(
return maybeWrapWithBg(
<LineChart
series={cut.chartSeries} title={cut.title} colors={cut.chartColors || theme.chartColors}
animationStyle={(cut.chartAnimation as any) || "draw"}
@@ -552,7 +624,7 @@ const SceneRenderer: React.FC<{ cut: Cut; theme: ThemeConfig }> = ({ cut, theme
);
}
if (cut.type === "pie_chart" && cut.chartData) {
return maybeWrapWithBgImage(
return maybeWrapWithBg(
<PieChart
data={cut.chartData} title={cut.title} colors={cut.chartColors || theme.chartColors}
animationStyle={(cut.chartAnimation as any) || "expand"}
@@ -562,7 +634,7 @@ const SceneRenderer: React.FC<{ cut: Cut; theme: ThemeConfig }> = ({ cut, theme
);
}
if (cut.type === "kpi_grid" && cut.chartData) {
return maybeWrapWithBgImage(
return maybeWrapWithBg(
<KPIGrid
metrics={cut.chartData} title={cut.title} columns={cut.columns}
colors={cut.chartColors || theme.chartColors} animationStyle={(cut.chartAnimation as any) || "count-up"}
@@ -571,7 +643,7 @@ const SceneRenderer: React.FC<{ cut: Cut; theme: ThemeConfig }> = ({ cut, theme
);
}
if (cut.type === "progress_bar" && cut.progress !== undefined) {
return maybeWrapWithBgImage(
return maybeWrapWithBg(
<AbsoluteFill
style={{
background: bgColor || theme.surfaceColor,
@@ -620,16 +692,16 @@ const SceneRenderer: React.FC<{ cut: Cut; theme: ThemeConfig }> = ({ cut, theme
const animation = cut.animation || cut.transform?.animation;
if (cut.source && isImage(cut.source)) {
return <ImageScene src={cut.source} animation={animation} />;
return maybeWrapWithBg(<ImageScene src={cut.source} animation={animation} />);
}
if (cut.source && isVideo(cut.source)) {
return <VideoScene src={cut.source} startFrom={cut.in_seconds} />;
return maybeWrapWithBg(<VideoScene src={cut.source} startFrom={cut.source_in_seconds ?? 0} />);
}
// Final fallback — try as image if source exists, otherwise show text_card
if (cut.source) {
return <ImageScene src={cut.source} animation={animation} />;
return maybeWrapWithBg(<ImageScene src={cut.source} animation={animation} />);
}
// No source, no type — render as text card with cut id as fallback
@@ -664,6 +736,17 @@ const OverlayRenderer: React.FC<{ overlay: Overlay }> = ({ overlay }) => {
if (overlay.type === "hero_title") {
return <HeroTitle title={overlay.text} subtitle={overlay.subtitle} />;
}
if (overlay.type === "provider_chip" && overlay.providers) {
return (
<ProviderChip
providers={overlay.providers as string[]}
cycleSeconds={overlay.cycleSeconds}
position={(overlay.position as any) || "bottom-right"}
accentColor={overlay.accentColor}
label={overlay.label}
/>
);
}
return null;
};
@@ -0,0 +1,101 @@
import { AbsoluteFill, interpolate, spring, useCurrentFrame, useVideoConfig } from "remotion";
/**
* ProviderChip rotating pill of AI video provider names that cycle through.
* Positioned in a corner over background video.
*/
interface ProviderChipProps {
providers: string[];
cycleSeconds?: number;
position?: "top-left" | "top-right" | "bottom-left" | "bottom-right";
accentColor?: string;
label?: string;
}
const POS_STYLES: Record<string, React.CSSProperties> = {
"top-left": { top: 48, left: 48 },
"top-right": { top: 48, right: 48 },
"bottom-left": { bottom: 96, left: 48 }, // avoid caption zone
"bottom-right": { bottom: 96, right: 48 },
};
export const ProviderChip: React.FC<ProviderChipProps> = ({
providers,
cycleSeconds = 2.5,
position = "bottom-right",
accentColor = "#22D3EE",
label = "generated with",
}) => {
const frame = useCurrentFrame();
const { fps } = useVideoConfig();
const cycleFrames = Math.max(1, Math.round(cycleSeconds * fps));
const idx = Math.floor(frame / cycleFrames) % providers.length;
const current = providers[idx];
const framesIntoCycle = frame % cycleFrames;
// Spring in on cycle start
const springIn = spring({
frame: framesIntoCycle,
fps,
config: { damping: 14, stiffness: 200 },
durationInFrames: Math.ceil(fps * 0.35),
});
// Fade out before cycle end
const fadeOut =
framesIntoCycle > cycleFrames - fps * 0.25
? interpolate(framesIntoCycle, [cycleFrames - fps * 0.25, cycleFrames], [1, 0], { extrapolateRight: "clamp" })
: 1;
const alpha = Math.min(springIn, fadeOut);
const translateY = interpolate(springIn, [0, 1], [12, 0]);
return (
<AbsoluteFill pointerEvents="none">
<div
style={{
position: "absolute",
...POS_STYLES[position],
display: "flex",
flexDirection: "column",
alignItems: position.includes("right") ? "flex-end" : "flex-start",
gap: 8,
opacity: alpha,
transform: `translateY(${translateY}px)`,
}}
>
<div
style={{
fontSize: 16,
color: "rgba(255,255,255,0.6)",
fontFamily: "Inter, sans-serif",
fontWeight: 500,
letterSpacing: 1.5,
textTransform: "uppercase",
}}
>
{label}
</div>
<div
style={{
padding: "14px 26px",
background: "rgba(11, 15, 26, 0.82)",
border: `2px solid ${accentColor}`,
borderRadius: 999,
color: accentColor,
fontFamily: "'Space Grotesk', Inter, sans-serif",
fontWeight: 700,
fontSize: 28,
letterSpacing: 0.3,
backdropFilter: "blur(8px)",
boxShadow: `0 8px 32px ${accentColor}30`,
}}
>
{current}
</div>
</div>
</AbsoluteFill>
);
};
@@ -0,0 +1,266 @@
import { AbsoluteFill, interpolate, spring, useCurrentFrame, useVideoConfig } from "remotion";
/**
* TerminalScene animated terminal with typed commands and scrolling output.
*
* Each "step" is either:
* { kind: "cmd", text: "git clone ...", typeSpeed?: number } typed char-by-char with prompt
* { kind: "out", text: "cloning into 'OpenMontage'..." } reveals instantly
* { kind: "pause", seconds: number } silent dwell
* { kind: "pill", text: "Piper TTS installed", color?: string } floating badge
*
* Steps execute in order at the specified durations. Terminal auto-scrolls when
* it fills up.
*/
export type TerminalStep =
| { kind: "cmd"; text: string; typeSpeed?: number; holdSeconds?: number }
| { kind: "out"; text: string; holdSeconds?: number }
| { kind: "pause"; seconds: number }
| { kind: "pill"; text: string; color?: string; durationSeconds?: number };
interface TerminalSceneProps {
title?: string;
steps: TerminalStep[];
prompt?: string;
accentColor?: string;
backgroundColor?: string;
}
interface RenderedLine {
text: string;
isCmd: boolean;
startFrame: number;
endFrame: number; // frame at which typing completes
}
interface RenderedPill {
text: string;
color: string;
startFrame: number;
endFrame: number;
}
export const TerminalScene: React.FC<TerminalSceneProps> = ({
title = "Terminal",
steps,
prompt = "$",
accentColor = "#22D3EE",
backgroundColor = "#0B0F1A",
}) => {
const frame = useCurrentFrame();
const { fps } = useVideoConfig();
// Lay out timing in frames
const lines: RenderedLine[] = [];
const pills: RenderedPill[] = [];
let cursorFrame = 0;
for (const step of steps) {
if (step.kind === "cmd") {
const speed = step.typeSpeed ?? 0.035; // seconds per char
const typeFrames = Math.ceil(step.text.length * speed * fps);
const hold = Math.ceil((step.holdSeconds ?? 0.3) * fps);
lines.push({
text: step.text,
isCmd: true,
startFrame: cursorFrame,
endFrame: cursorFrame + typeFrames,
});
cursorFrame += typeFrames + hold;
} else if (step.kind === "out") {
const revealFrames = Math.max(2, Math.ceil(0.08 * fps));
const hold = Math.ceil((step.holdSeconds ?? 0.15) * fps);
lines.push({
text: step.text,
isCmd: false,
startFrame: cursorFrame,
endFrame: cursorFrame + revealFrames,
});
cursorFrame += revealFrames + hold;
} else if (step.kind === "pause") {
cursorFrame += Math.ceil(step.seconds * fps);
} else if (step.kind === "pill") {
const dur = Math.ceil((step.durationSeconds ?? 2.2) * fps);
pills.push({
text: step.text,
color: step.color ?? accentColor,
startFrame: cursorFrame,
endFrame: cursorFrame + dur,
});
// pill is non-blocking — don't advance cursor
}
}
// Only render lines that have started
const visibleLines = lines.filter(l => frame >= l.startFrame);
// Auto-scroll: keep last N lines in view
const MAX_VISIBLE = 18;
const scrollStart = Math.max(0, visibleLines.length - MAX_VISIBLE);
const renderedLines = visibleLines.slice(scrollStart);
// Cursor blinks on most recent command
const blinkPhase = Math.floor(frame / (fps * 0.55)) % 2 === 0;
// Terminal window frame fade-in
const windowOpacity = spring({ frame, fps, config: { damping: 25, stiffness: 100 } });
return (
<AbsoluteFill
style={{
background: backgroundColor,
justifyContent: "center",
alignItems: "center",
padding: "80px",
fontFamily: "'JetBrains Mono', 'Consolas', 'Monaco', monospace",
}}
>
<div
style={{
width: "85%",
maxWidth: 1600,
height: "80%",
opacity: windowOpacity,
transform: `scale(${interpolate(windowOpacity, [0, 1], [0.97, 1])})`,
borderRadius: 16,
overflow: "hidden",
boxShadow: "0 40px 120px rgba(0,0,0,0.6), 0 0 1px rgba(255,255,255,0.2) inset",
background: "#12151F",
position: "relative",
}}
>
{/* Title bar */}
<div
style={{
display: "flex",
alignItems: "center",
gap: 8,
padding: "14px 18px",
background: "#1A1F2E",
borderBottom: "1px solid rgba(255,255,255,0.05)",
}}
>
<div style={{ width: 12, height: 12, borderRadius: "50%", background: "#FF5F56" }} />
<div style={{ width: 12, height: 12, borderRadius: "50%", background: "#FFBD2E" }} />
<div style={{ width: 12, height: 12, borderRadius: "50%", background: "#27C93F" }} />
<div
style={{
flex: 1,
textAlign: "center",
color: "#8E8E93",
fontSize: 16,
fontFamily: "Inter, sans-serif",
}}
>
{title}
</div>
</div>
{/* Terminal body */}
<div
style={{
padding: "32px 40px",
fontSize: 26,
lineHeight: 1.55,
color: "#E5E7EB",
height: "calc(100% - 46px)",
overflow: "hidden",
}}
>
{renderedLines.map((line, idx) => {
if (line.isCmd) {
// Char-by-char typed command
const progress = interpolate(
frame,
[line.startFrame, line.endFrame],
[0, line.text.length],
{ extrapolateRight: "clamp" }
);
const typed = line.text.slice(0, Math.floor(progress));
const isLatest = idx === renderedLines.length - 1;
const isActive = frame <= line.endFrame + fps * 0.2;
return (
<div key={`${line.startFrame}-${idx}`} style={{ display: "flex", alignItems: "baseline" }}>
<span style={{ color: accentColor, marginRight: 12, fontWeight: 600 }}>{prompt}</span>
<span style={{ color: "#F1F5F9" }}>{typed}</span>
{isLatest && isActive && blinkPhase && (
<span
style={{
display: "inline-block",
width: 12,
height: 26,
background: "#F1F5F9",
marginLeft: 2,
transform: "translateY(4px)",
}}
/>
)}
</div>
);
} else {
// Instant-reveal output line with fade-in
const alpha = interpolate(
frame,
[line.startFrame, line.endFrame],
[0, 1],
{ extrapolateRight: "clamp" }
);
return (
<div
key={`${line.startFrame}-${idx}`}
style={{ color: "#9CA3AF", opacity: alpha, paddingLeft: 4 }}
>
{line.text}
</div>
);
}
})}
</div>
{/* Floating command pills */}
{pills
.filter(p => frame >= p.startFrame && frame <= p.endFrame)
.map((pill, idx) => {
const lifeProgress = (frame - pill.startFrame) / Math.max(1, pill.endFrame - pill.startFrame);
// spring in (0 → 1), hold, spring out (0.8 → 1.0)
const inAlpha = spring({
frame: frame - pill.startFrame,
fps,
config: { damping: 14, stiffness: 180 },
durationInFrames: Math.ceil(fps * 0.35),
});
const outAlpha =
lifeProgress > 0.82
? interpolate(lifeProgress, [0.82, 1], [1, 0], { extrapolateRight: "clamp" })
: 1;
const alpha = Math.min(inAlpha, outAlpha);
const translateY = interpolate(inAlpha, [0, 1], [14, 0]);
return (
<div
key={`${pill.startFrame}-${idx}`}
style={{
position: "absolute",
top: 28 + idx * 62,
right: 32,
padding: "12px 20px",
background: pill.color,
color: "#0B0F1A",
borderRadius: 999,
fontFamily: "Inter, sans-serif",
fontWeight: 700,
fontSize: 20,
letterSpacing: 0.2,
opacity: alpha,
transform: `translateY(${translateY}px)`,
boxShadow: `0 10px 30px ${pill.color}40`,
}}
>
{pill.text}
</div>
);
})}
</div>
</AbsoluteFill>
);
};
@@ -10,5 +10,8 @@ export { StatReveal } from "./StatReveal";
export { HeroTitle } from "./HeroTitle";
export { ParticleOverlay } from "./ParticleOverlay";
export { AnimeScene } from "./AnimeScene";
export { TerminalScene } from "./TerminalScene";
export { ProviderChip } from "./ProviderChip";
export type { ParticleType } from "./ParticleOverlay";
export type { CameraMotion, AnimeSceneProps } from "./AnimeScene";
export type { TerminalStep } from "./TerminalScene";
@@ -4,6 +4,23 @@
This stage produces the minimal but high-leverage assets that make a screen demo easier to follow: subtitles, audio cleanup, reusable overlays, masks, and optional light-weight support cards.
## Two production modes — pick before generating assets
**Read the brief's `production_mode` field.** If `idea` didn't set one, decide here:
| Mode | When | Asset production looks like |
|---|---|---|
| **`real_capture`** | Real app UI (browser, design tool, IDE with plugins); live behavior; user asked for their own screen recorded | Clean audio + subtitles + callout overlays (arrows, highlight masks) applied on top of the captured MP4 |
| **`synthetic_terminal`** | CLI, terminal, install flow, make targets, git/npm commands, `.env` config — anything scriptable | **No capture at all.** Author a `terminal_scene` cut for `video_compose` (Remotion). Commands type char-by-char, output scrolls, pills announce completions. See `.agents/skills/synthetic-screen-recording/SKILL.md`. |
**Mode selection heuristic:** *"Can I predict every command and its output before shooting?"* If yes → synthetic. If no → real capture.
For synthetic mode, the asset stage produces:
- **narration (tts_selector)** aligned to the exact video-time each command should type
- **a `steps` list** (cmd/out/pause/pill primitives) that paces with narration cues
- **a pacing verification** via `lib.verify_scene_pacing.assert_alignment(...)` — must pass before render
- **no** screen-recorder footage, no callout arrows, no zoom-crop regions (those are real-capture concepts)
## Prerequisites
| Layer | Resource | Purpose |
+13 -2
View File
@@ -2,9 +2,20 @@
## When To Use
Use this pipeline when the source is already a screen recording: product walkthrough, software tutorial, coding demo, browser flow, or troubleshooting capture.
Use this pipeline whenever the deliverable is a screen-recording-style demo. There are **two production modes** — pick one in the brief:
Your job is to turn raw capture into a clear procedural video. The main deliverable at this stage is a schema-valid `brief`, with pipeline-specific detail stored in `brief.metadata`.
| Mode | Source material | Pick when |
|---|---|---|
| **`real_capture`** | An actual screen recording (MP4) captured via `screen_recorder`, `cap_recorder`, or `playwright-recording` | Real app UI, live behavior, browser flows, IDE plugins, user asked for their own screen |
| **`synthetic_terminal`** | None — nothing is captured. You author a `terminal_scene` cut for Remotion | CLI / terminal / install flow / make targets / git clone / API key config — anything scriptable where every command and output is predictable |
**Decision question:** *"Can I predict every command and its output before shooting?"* If yes → synthetic. If no → real capture.
**Record the mode in `brief.metadata.production_mode`.** The asset-director reads this field to choose between capture+overlay assets vs a `steps` list paced with narration.
For `synthetic_terminal`, also read `.agents/skills/synthetic-screen-recording/SKILL.md` before proceeding — it encodes the pacing rule that killed an earlier showcase render (commands burned through in 40% of scene time, then terminal froze for the remaining 60%).
Your job at this stage is to turn the user's request into a clear procedural video plan. The main deliverable is a schema-valid `brief`, with pipeline-specific detail stored in `brief.metadata`.
## Operating Principles