mirror of
https://github.com/zenlm/papers.git
synced 2026-07-27 03:34:41 +00:00
Honest media-paper provenance: attribute real upstreams, withdraw fabricated
Rewrite 5 media whitepapers to truthfully attribute their permissively-licensed upstreams (dropping the fabricated "Zen MoDE" architecture + invented benchmarks): - zen-3d -> Microsoft TRELLIS (MIT) - zen-video-i2v -> Alibaba Wan2.2-I2V-A14B (Apache-2.0) - zen-world -> Alibaba Wan2.1-T2V-14B (Apache-2.0) - zen-director -> Alibaba Wan2.2-TI2V-5B (Apache-2.0) - zen-musician -> M-A-P YuE (Apache-2.0) Withdraw 4 fabricated/discontinued papers: - zen-mixture-of-experts (the fabricated "Zen MoDE" architecture) - zen-voyager (mislabeled Qwen3-32B LLM; repo deleted) - zen-foley (no permissively-licensed foley model exists) - zen-video (deleted Tencent-mix repo) Drop the withdrawn papers from INDEX.md and PAPER_TIMELINE.md. Co-authored-by: Hanzo Dev <dev@hanzo.ai>
This commit is contained in:
Hanzo Dev
@@ -29,7 +29,6 @@ Auto-generated catalogue of research papers.
|
||||
| `zen-enterprise-deployment` | ✓ | `zen-enterprise-deployment.tex` |
|
||||
| `zen-financial-ai` | ✓ | `zen-financial-ai.tex` |
|
||||
| `zen-finetuning` | ✓ | `zen-finetuning.tex` |
|
||||
| `zen-foley` | ✓ | `zen-foley.tex` |
|
||||
| `zen-guard-gen_whitepaper` | ✓ | `zen-guard-gen_whitepaper.tex` |
|
||||
| `zen-guard-stream_whitepaper` | ✓ | `zen-guard-stream_whitepaper.tex` |
|
||||
| `zen-hallucination-reduction` | ✓ | `zen-hallucination-reduction.tex` |
|
||||
@@ -41,7 +40,6 @@ Auto-generated catalogue of research papers.
|
||||
| `zen-mathematical-reasoning` | ✓ | `zen-mathematical-reasoning.tex` |
|
||||
| `zen-max_whitepaper` | ✓ | `zen-max_whitepaper.tex` |
|
||||
| `zen-medical` | ✓ | `zen-medical.tex` |
|
||||
| `zen-mixture-of-experts` | ✓ | `zen-mixture-of-experts.tex` |
|
||||
| `zen-multilingual` | ✓ | `zen-multilingual.tex` |
|
||||
| `zen-multimodal-architecture` | ✓ | `zen-multimodal-architecture.tex` |
|
||||
| `zen-musician` | ✓ | `zen-musician.tex` |
|
||||
@@ -57,11 +55,9 @@ Auto-generated catalogue of research papers.
|
||||
| `zen-training-methodology` | ✓ | `zen-training-methodology.tex` |
|
||||
| `zen-translator` | ✓ | `zen-translator.tex` |
|
||||
| `zen-video-i2v_whitepaper` | ✓ | `zen-video-i2v_whitepaper.tex` |
|
||||
| `zen-video` | ✓ | `zen-video.tex` |
|
||||
| `zen-vision-architecture` | ✓ | `zen-vision-architecture.tex` |
|
||||
| `zen-vl_whitepaper` | ✓ | `zen-vl_whitepaper.tex` |
|
||||
| `zen-voice-clone` | ✓ | `zen-voice-clone.tex` |
|
||||
| `zen-voyager` | ✓ | `zen-voyager.tex` |
|
||||
| `zen-world` | ✓ | `zen-world.tex` |
|
||||
| `zen3-embedding_whitepaper` | ✓ | `zen3-embedding_whitepaper.tex` |
|
||||
| `zen3-guard_whitepaper` | ✓ | `zen3-guard_whitepaper.tex` |
|
||||
@@ -79,4 +75,4 @@ Auto-generated catalogue of research papers.
|
||||
| `zen4-ultra_whitepaper` | ✓ | `zen4-ultra_whitepaper.tex` |
|
||||
| `zen5_whitepaper` | (pending) | `zen5_whitepaper.tex` |
|
||||
|
||||
**Total**: 74 papers, 151 PDFs compiled
|
||||
**Total**: 70 papers, 151 PDFs compiled
|
||||
|
||||
+3
-27
@@ -166,15 +166,7 @@ Zen models use **Zen MoDE (Mixture of Distilled Experts)** architecture, emphasi
|
||||
- **Status**: ✅ Ready for compilation
|
||||
- **File**: `zen-3d.tex`
|
||||
|
||||
#### 16. Zen-Foley (Audio Effects)
|
||||
- **Version**: v2025.01 (January 2025)
|
||||
- Sound effects generation
|
||||
- Audio matching for video
|
||||
- Spatial audio
|
||||
- **Status**: ✅ Ready for compilation
|
||||
- **File**: `zen-foley.tex`
|
||||
|
||||
#### 17. Zen-Musician
|
||||
#### 16. Zen-Musician
|
||||
- **Version**: v2025.01 (January 2025)
|
||||
- Music composition
|
||||
- Multiple genres
|
||||
@@ -206,22 +198,6 @@ Zen models use **Zen MoDE (Mixture of Distilled Experts)** architecture, emphasi
|
||||
- **Status**: ✅ Ready for compilation
|
||||
- **File**: `zen-world.tex`
|
||||
|
||||
#### 21. Zen-Video
|
||||
- **Version**: v2025.01 (January 2025)
|
||||
- Video generation
|
||||
- Frame interpolation
|
||||
- Video understanding
|
||||
- **Status**: ✅ Ready for compilation
|
||||
- **File**: `zen-video.tex`
|
||||
|
||||
#### 22. Zen-Voyager (Exploration)
|
||||
- **Version**: v2025.02 (February 2025)
|
||||
- Open-ended exploration
|
||||
- Curiosity-driven learning
|
||||
- Novel task discovery
|
||||
- **Status**: ✅ Ready for compilation
|
||||
- **File**: `zen-voyager.tex`
|
||||
|
||||
---
|
||||
|
||||
## Model Family Statistics
|
||||
@@ -304,8 +280,8 @@ Each paper includes:
|
||||
- **October 2024**: Zen-Nano (0.6B edge model)
|
||||
- **November 2024**: Zen-Coder, Zen-Eco, Zen-Scribe
|
||||
- **December 2024**: Zen-Artist family, Zen-Guard, Core papers (zen-base, zen-technical-paper)
|
||||
- **January 2025**: Zen-Omni, Zen-Next, Extended family (3D, Foley, Musician, Director, Agent, World)
|
||||
- **February 2025**: Zen-World, Zen-Voyager, Zen-VL, Zen-Coder-Flash, Zen-Inference-Optimization
|
||||
- **January 2025**: Zen-Omni, Zen-Next, Extended family (3D, Musician, Director, Agent, World)
|
||||
- **February 2025**: Zen-World, Zen-VL, Zen-Coder-Flash, Zen-Inference-Optimization
|
||||
- **March 2025**: Zen3 family (Nano, Omni, VL, Guard, Embedding), Zen-Code, Zen-Dub, Zen-Video-I2V
|
||||
- **April 2025**: Zen-Live, Zen-Guard-Gen/Stream, Context Extension, Fine-tuning, Architecture papers
|
||||
- **May 2025**: Domain papers (Medical, Financial, Legal, Privacy), Reward Modeling, Distillation
|
||||
|
||||
BIN
Binary file not shown.
+230
-333
@@ -33,7 +33,7 @@
|
||||
\vspace{0.5cm}
|
||||
\Huge \textbf{Zen-3D} \\
|
||||
\vspace{0.3cm}
|
||||
\large 3D Scene Understanding and Generation at Scale \\
|
||||
\large Image-to-3D Generation via a Redistribution of Microsoft TRELLIS \\
|
||||
\vspace{0.5cm}
|
||||
\normalsize Technical Report v2025.01
|
||||
}
|
||||
@@ -50,16 +50,20 @@
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
We present \textbf{Zen-3D}, an 8-billion parameter model specialized for 3D scene understanding,
|
||||
mesh generation, and point cloud processing. Built on the Zen MoDE (Mixture of Distilled Experts)
|
||||
architecture with domain-adapted 3D tokenization, Zen-3D achieves state-of-the-art results across
|
||||
all major 3D benchmarks: 95.2\% IoU on ShapeNet, 88.3\% mIoU on ScanNet semantic segmentation,
|
||||
and 94.1\% classification accuracy on ModelNet40. Unlike prior work that treats 3D understanding
|
||||
as a downstream task grafted onto language models, Zen-3D natively encodes spatial geometry through
|
||||
a novel triplane-based 3D tokenizer and dedicated attention mechanisms for volumetric context.
|
||||
The model supports interactive scene reconstruction, counterfactual 3D editing, and zero-shot
|
||||
generalization to unseen object categories---enabling downstream applications in robotics, augmented
|
||||
reality, and generative design at production scale.
|
||||
\textbf{Zen-3D} is a packaging and integration of \textbf{TRELLIS}---the open-source 3D
|
||||
generation system released by Microsoft Research \cite{xiang2024trellis}---into the Zen model
|
||||
stack. It is \emph{not} a from-scratch model: Zen-3D redistributes the publicly available
|
||||
TRELLIS code and pretrained weights (most notably the image-conditioned
|
||||
\texttt{JeffreyXiang/TRELLIS-image-large} checkpoint, $\approx$1.2B parameters) under their
|
||||
original \textbf{MIT license}, wrapped behind the Zen SDK and serving interfaces. TRELLIS
|
||||
generates high-quality 3D assets from a single image or a text prompt using a unified
|
||||
\emph{Structured LATent} (SLAT) representation and rectified-flow transformers, and can decode
|
||||
the same latent into 3D Gaussian splats, radiance fields, or textured meshes. This report
|
||||
describes what TRELLIS actually is and how it works, documents which artifacts Zen-3D
|
||||
redistributes and under what license, and explains the integration surface (API, output
|
||||
formats, deployment) that the Zen stack adds on top. All architectural details and any reported
|
||||
results are attributed to the original TRELLIS authors; we do not claim a novel architecture and
|
||||
report no benchmarks of our own.
|
||||
\end{abstract}
|
||||
|
||||
\tableofcontents
|
||||
@@ -67,31 +71,41 @@ reality, and generative design at production scale.
|
||||
|
||||
\section{Introduction}
|
||||
|
||||
Three-dimensional scene understanding has long been a frontier capability distinguishing physically
|
||||
situated AI from purely linguistic systems. Tasks such as reconstructing a scene from partial
|
||||
observations, segmenting semantically meaningful regions in a point cloud, or generating plausible
|
||||
3D meshes from text descriptions require models that reason natively about geometry, topology, and
|
||||
spatial relations---not merely pattern-match against 2D projections.
|
||||
\subsection{What Zen-3D is, and what it is not}
|
||||
|
||||
Prior multimodal approaches to 3D have taken two broad strategies: (1) projecting 3D data into
|
||||
sequences of 2D renders and applying vision-language models, or (2) building bespoke 3D encoders
|
||||
that feed compressed representations into a language backbone. Both strategies suffer from either
|
||||
information loss in the projection step or a semantic gap between the 3D encoder and the pretrained
|
||||
language prior.
|
||||
Zen-3D is the Zen stack's image-to-3D and text-to-3D capability. Concretely, it is a
|
||||
redistribution and integration of \textbf{TRELLIS} (``Structured 3D Latents for Scalable and
|
||||
Versatile 3D Generation''), an open-source system from Microsoft Research presented at
|
||||
CVPR~2025 \cite{xiang2024trellis}. The TRELLIS code and pretrained weights are released under
|
||||
the \textbf{MIT license} \cite{trellis_repo}, which permits redistribution and commercial use
|
||||
with attribution and license retention.
|
||||
|
||||
Zen-3D resolves this tension through three contributions:
|
||||
To be explicit about provenance:
|
||||
|
||||
\begin{enumerate}
|
||||
\item \textbf{Native 3D tokenization}: A triplane decomposition tokenizer that converts point
|
||||
clouds and voxel grids into discrete token sequences without lossy projection.
|
||||
\item \textbf{Geometry-aware attention}: Modified self-attention with relative 3D position
|
||||
encodings, enabling the model to reason about spatial adjacency, occlusion, and scale.
|
||||
\item \textbf{Unified generation and understanding}: A single model that handles segmentation,
|
||||
reconstruction, and conditional mesh generation through task-specific decoding heads built on
|
||||
the shared Zen MoDE backbone.
|
||||
\end{enumerate}
|
||||
\begin{itemize}
|
||||
\item Zen-3D does \textbf{not} introduce a new model architecture. There is no
|
||||
``from-scratch'' Zen 3D network and no proprietary 3D training run behind this report.
|
||||
\item The model, weights, and generation algorithm are those of TRELLIS, authored by
|
||||
Jianfeng Xiang, Zelong Lv, Sicheng Xu, Yu Deng, Ruicheng Wang, Bowen Zhang, Dong Chen,
|
||||
Xin Tong, and Jiaolong Yang \cite{xiang2024trellis}.
|
||||
\item Zen-3D's contribution is purely at the \emph{packaging and integration} layer:
|
||||
pinning a known-good TRELLIS checkpoint, exposing it through the Zen SDK, normalizing its
|
||||
output formats, and providing serving/deployment glue.
|
||||
\item All quantitative results belong to the original work; this report reproduces no
|
||||
benchmark numbers of its own and omits any figure we cannot attribute.
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Model Overview}
|
||||
\subsection{Why TRELLIS}
|
||||
|
||||
Single-image and text-conditioned 3D asset generation is a frontier capability for content
|
||||
creation, simulation, and AR/VR. Among openly licensed options, TRELLIS is notable for (1) a
|
||||
permissive MIT license on both code and weights, (2) a unified latent representation that can be
|
||||
decoded into multiple downstream 3D formats (Gaussian splats, radiance fields, meshes) from a
|
||||
single generation pass, and (3) strong reported quality at the 1--2B parameter scale
|
||||
\cite{xiang2024trellis}. These properties make it a practical foundation to integrate rather
|
||||
than a model to reinvent.
|
||||
|
||||
\subsection{Redistributed Artifacts}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
@@ -99,359 +113,242 @@ Zen-3D resolves this tension through three contributions:
|
||||
\toprule
|
||||
\textbf{Property} & \textbf{Value} \\
|
||||
\midrule
|
||||
Parameters & 8B \\
|
||||
Architecture & Zen MoDE (Mixture of Distilled Experts) \\
|
||||
3D Tokenizer & Triplane VQ-VAE (codebook size 16,384) \\
|
||||
Context Length & 32,768 tokens (spatial + language) \\
|
||||
Max Resolution & 256$^3$ voxels / 100K point clouds \\
|
||||
Output Modalities & Mesh, point cloud, segmentation mask, text \\
|
||||
Training Data & 47M 3D object-text pairs, 2.1M scene reconstructions \\
|
||||
Upstream project & TRELLIS (Microsoft Research) \cite{xiang2024trellis,trellis_repo} \\
|
||||
Task & Image-to-3D and text-to-3D generation \\
|
||||
Representation & Structured LATent (SLAT) \\
|
||||
Generative model & Rectified-flow transformers (two-stage) \\
|
||||
Primary weights & \texttt{JeffreyXiang/TRELLIS-image-large} ($\approx$1.2B) \\
|
||||
Other public weights & TRELLIS-text-base/large/xlarge (342M / 1.1B / 2.0B) \\
|
||||
Output formats & 3D Gaussian splats, radiance fields, textured meshes \\
|
||||
License (code + weights) & MIT \cite{trellis_repo} \\
|
||||
Training data (upstream) & TRELLIS-500K ($\approx$500K curated 3D assets) \\
|
||||
Zen-3D additions & SDK wrapper, serving, output normalization \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen-3D Model Specifications}
|
||||
\caption{Zen-3D: artifacts redistributed from TRELLIS and the integration layer added by Zen.
|
||||
Parameter counts and dataset sizes are as reported by the TRELLIS authors
|
||||
\cite{xiang2024trellis}.}
|
||||
\end{table}
|
||||
|
||||
\section{Architecture}
|
||||
\section{TRELLIS Architecture}
|
||||
|
||||
\subsection{Triplane 3D Tokenizer}
|
||||
This section summarizes the TRELLIS method as described by its authors \cite{xiang2024trellis}.
|
||||
It is included for completeness; none of it is original to Zen-3D.
|
||||
|
||||
Input 3D data---whether a point cloud, a voxel grid, or an implicit surface---is first projected
|
||||
onto three orthogonal feature planes ($XY$, $XZ$, $YZ$). Each plane is independently encoded by a
|
||||
convolutional backbone and then vector-quantized to produce a sequence of discrete tokens. The three
|
||||
plane token sequences are concatenated to form the full 3D token sequence fed to the transformer.
|
||||
|
||||
Formally, given input geometry $\mathcal{G} \in \mathbb{R}^{N \times 3}$:
|
||||
\begin{align}
|
||||
F_{XY}, F_{XZ}, F_{YZ} &= \text{TriplaneProject}(\mathcal{G}) \\
|
||||
T_i &= \text{VQ}(\text{ConvEnc}(F_i)), \quad i \in \{XY, XZ, YZ\} \\
|
||||
T_{3D} &= [T_{XY} \; || \; T_{XZ} \; || \; T_{YZ}]
|
||||
\end{align}
|
||||
|
||||
The VQ-VAE codebook is trained jointly with the transformer backbone using a commitment loss
|
||||
$\mathcal{L}_{commit} = \|z_e - \text{sg}(z_q)\|_2^2$ where $z_e$ is the encoder output and
|
||||
$z_q$ is the nearest codebook entry.
|
||||
|
||||
\subsection{Geometry-Aware Attention}
|
||||
|
||||
Standard self-attention treats tokens as an unordered set, discarding geometric structure. We
|
||||
introduce \textbf{Spatial Relative Position Encoding (SRPE)}: for any pair of 3D tokens $(i, j)$,
|
||||
we compute their approximate 3D centroid distance $d_{ij}$ and inject a learned distance bias into
|
||||
the attention logits:
|
||||
\subsection{Structured LATent (SLAT) Representation}
|
||||
|
||||
The cornerstone of TRELLIS is the \emph{Structured LATent} (SLAT) representation: a set of local
|
||||
latent vectors attached to the active (occupied) voxels of a sparse 3D grid. Formally, an asset
|
||||
is encoded as
|
||||
\begin{equation}
|
||||
a_{ij} = \frac{q_i \cdot k_j}{\sqrt{d_k}} + b(d_{ij})
|
||||
\mathbf{z} = \{(\boldsymbol{z}_i, \boldsymbol{p}_i)\}_{i=1}^{L},
|
||||
\qquad \boldsymbol{p}_i \in \{0,\dots,N-1\}^3,
|
||||
\end{equation}
|
||||
where each $\boldsymbol{p}_i$ is the integer coordinate of an active voxel and $\boldsymbol{z}_i$
|
||||
is its local latent vector. The active voxels $\boldsymbol{p}_i$ outline the coarse geometry of
|
||||
the asset, while the latents $\boldsymbol{z}_i$ capture finer structural and textural detail. By
|
||||
default the grid resolution is $N = 64$, which yields roughly $L \approx 20{,}000$ active voxels
|
||||
per asset on average \cite{xiang2024trellis}.
|
||||
|
||||
where $b: \mathbb{R}_{\geq 0} \to \mathbb{R}$ is a learned monotone function discretized into 64
|
||||
distance buckets covering 0--10 meters in log-scale.
|
||||
The per-voxel features used to fit the latents are obtained from 2D vision features rather than
|
||||
from raw geometry. TRELLIS renders an asset from many camera views sampled on a sphere, extracts
|
||||
feature maps with a pretrained \textbf{DINOv2} encoder, projects each active voxel into those
|
||||
multiview feature maps, and averages the retrieved features to form the voxel feature
|
||||
$\boldsymbol{f}_i$ \cite{xiang2024trellis}. This couples geometry (the sparse grid) with
|
||||
appearance (the aggregated visual features) in a single structured latent.
|
||||
|
||||
\subsection{Zen MoDE Backbone}
|
||||
\subsection{SLAT Variational Autoencoder and Decoders}
|
||||
|
||||
The transformer backbone follows the Zen MoDE architecture: 32 transformer layers, each with 16
|
||||
attention heads and a mixture-of-experts feed-forward network with 8 experts (top-2 routing per
|
||||
token). The 3D token stream and optional text tokens are interleaved via cross-attention at every
|
||||
fourth layer, enabling bidirectional grounding between language and geometry.
|
||||
A transformer-based VAE encodes the voxel features into SLAT and reconstructs 3D outputs from
|
||||
it. Voxel tokens are serialized with sinusoidal positional encodings derived from their
|
||||
coordinates. A distinctive property of TRELLIS is that the \emph{same} SLAT can be decoded into
|
||||
several 3D output formats by separate, format-specific decoders \cite{xiang2024trellis}:
|
||||
|
||||
\begin{itemize}
|
||||
\item \textbf{3D Gaussians.} Each latent decodes to a small set of Gaussians with position
|
||||
offsets, colors, scales, opacities, and rotations. Positions are anchored to the voxel and
|
||||
perturbed by a bounded offset, e.g.\ $\boldsymbol{x}_i^k = \boldsymbol{p}_i +
|
||||
\tanh(\boldsymbol{o}_i^k)$. The encoder and the Gaussian decoder are trained end-to-end.
|
||||
\item \textbf{Radiance fields.} Each latent decodes to a local radiance volume represented
|
||||
with a CP-decomposition (following Strivec), enabling neural volumetric rendering.
|
||||
\item \textbf{Meshes.} Each latent decodes to FlexiCubes parameters and signed-distance
|
||||
values, from which a textured surface mesh is extracted (upsampled to a higher resolution).
|
||||
\end{itemize}
|
||||
|
||||
The non-Gaussian decoders are trained separately on top of a frozen encoder
|
||||
\cite{xiang2024trellis}.
|
||||
|
||||
\subsection{Two-Stage Rectified-Flow Generation}
|
||||
|
||||
TRELLIS generates SLAT with two \textbf{rectified-flow transformers} trained with a conditional
|
||||
flow-matching objective, run in sequence \cite{xiang2024trellis}:
|
||||
|
||||
\begin{enumerate}
|
||||
\item \textbf{Sparse-structure stage.} A flow transformer $\mathcal{G}_S$ generates the
|
||||
sparse occupancy structure---i.e.\ which voxels are active. The binary grid is first
|
||||
compressed by a VAE into a low-resolution dense feature grid, and the flow model operates in
|
||||
that space before decoding back to the occupied voxel set.
|
||||
\item \textbf{Structured-latent stage.} A second flow transformer $\mathcal{G}_L$ generates
|
||||
the local latents $\boldsymbol{z}_i$ for the active voxels produced by stage~1, operating
|
||||
over the sparse set of occupied cells.
|
||||
\end{enumerate}
|
||||
|
||||
Both transformers use attention adapted to 3D sparsity (e.g.\ serialization of active voxels and
|
||||
shifted-window attention in 3D); $\mathcal{G}_S$ and $\mathcal{G}_L$ are trained separately
|
||||
\cite{xiang2024trellis}.
|
||||
|
||||
\subsection{Conditioning}
|
||||
|
||||
TRELLIS supports both image and text conditioning, injected into the flow transformers via
|
||||
cross-attention \cite{xiang2024trellis}:
|
||||
|
||||
\begin{itemize}
|
||||
\item \textbf{Image conditioning} uses DINOv2 visual features of the input image. This is
|
||||
the path exercised by the \texttt{TRELLIS-image-large} checkpoint that Zen-3D primarily
|
||||
redistributes.
|
||||
\item \textbf{Text conditioning} uses CLIP text features, corresponding to the
|
||||
text-conditioned TRELLIS checkpoints.
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Model Sizes and Training Data (as reported upstream)}
|
||||
|
||||
The TRELLIS authors report training models at three scales---\textbf{342M} (Base), \textbf{1.1B}
|
||||
(Large), and \textbf{2.0B} (X-Large)---on \textbf{TRELLIS-500K}, a curated set of roughly 500K
|
||||
3D assets sourced from Objaverse(XL), ABO, 3D-FUTURE, HSSD, and Toys4k and filtered by aesthetic
|
||||
score \cite{xiang2024trellis,trellis_repo}. The image-conditioned checkpoint redistributed by
|
||||
Zen-3D, \texttt{JeffreyXiang/TRELLIS-image-large}, is the $\approx$1.2B-parameter image variant.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{ll}
|
||||
\begin{tabular}{lll}
|
||||
\toprule
|
||||
\textbf{Component} & \textbf{Configuration} \\
|
||||
\textbf{Public checkpoint} & \textbf{Parameters} & \textbf{Conditioning} \\
|
||||
\midrule
|
||||
Transformer layers & 32 \\
|
||||
Attention heads & 16 \\
|
||||
Hidden dimension & 4096 \\
|
||||
FFN type & MoE (8 experts, top-2) \\
|
||||
Active parameters per token & $\approx$2.1B of 8B total \\
|
||||
Cross-attention layers & Every 4th (8 total) \\
|
||||
Positional encoding & Rotary (RoPE) + SRPE \\
|
||||
TRELLIS-image-large & $\approx$1.2B & Image \\
|
||||
TRELLIS-text-base & 342M & Text \\
|
||||
TRELLIS-text-large & 1.1B & Text \\
|
||||
TRELLIS-text-xlarge & 2.0B & Text \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen-3D Backbone Architecture}
|
||||
\caption{Publicly released TRELLIS checkpoints, as listed by the upstream project
|
||||
\cite{trellis_repo}. Zen-3D primarily integrates the image-conditioned model.}
|
||||
\end{table}
|
||||
|
||||
\subsection{Task-Specific Decoding Heads}
|
||||
\section{Reported Results (Upstream)}
|
||||
|
||||
Three decoding heads share the backbone:
|
||||
We do not run our own evaluation, and we do not restate numerical benchmarks here because we
|
||||
cannot independently verify them in this report. The TRELLIS paper reports that the method
|
||||
produces diverse, high-quality 3D assets and compares favorably against prior and contemporaneous
|
||||
3D generators at similar scale; for the exact quantitative comparisons, evaluation protocol, and
|
||||
qualitative galleries, we refer the reader to the original publication and project page
|
||||
\cite{xiang2024trellis,trellis_repo}. Any number not attributable to that source should be
|
||||
considered absent rather than implied.
|
||||
|
||||
\textbf{Segmentation head}: A lightweight MLP applied to per-token backbone features, producing
|
||||
per-point semantic logits over a 200-class ontology (ShapeNet taxonomy + ScanNet classes).
|
||||
\section{Zen Integration}
|
||||
|
||||
\textbf{Reconstruction head}: A triplane decoder that inverts the tokenizer, predicting an
|
||||
occupancy field from which meshes are extracted via marching cubes at configurable resolution.
|
||||
Everything in this section is the integration layer that Zen-3D adds; it does not modify the
|
||||
TRELLIS model itself.
|
||||
|
||||
\textbf{Generation head}: A diffusion decoder conditioned on text-derived embeddings, producing
|
||||
novel 3D shapes by iteratively denoising a triplane latent.
|
||||
\subsection{What Zen-3D Provides}
|
||||
|
||||
\section{Training}
|
||||
\begin{itemize}
|
||||
\item \textbf{Pinned redistribution.} A fixed, known-good TRELLIS checkpoint and code
|
||||
revision, repackaged under the Zen distribution while retaining the upstream MIT license and
|
||||
attribution (see Section~\ref{sec:licensing}).
|
||||
\item \textbf{SDK surface.} A thin Python wrapper that loads the upstream
|
||||
\texttt{TrellisImageTo3DPipeline} and exposes a uniform Zen API for image-to-3D (and
|
||||
text-to-3D where the corresponding checkpoint is used).
|
||||
\item \textbf{Output normalization.} Helpers to export the decoded SLAT into the format a
|
||||
caller wants---3D Gaussian splats, a radiance field, or a textured mesh (e.g.\ GLB/OBJ)---using
|
||||
TRELLIS's own decoders.
|
||||
\item \textbf{Serving glue.} Deployment configuration for running the pipeline behind the Zen
|
||||
serving stack.
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Dataset}
|
||||
\subsection{Output Formats}
|
||||
|
||||
Because a single SLAT can be decoded multiple ways, the same generation can yield different
|
||||
deliverables without re-running the flow models:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lrrl}
|
||||
\begin{tabular}{lll}
|
||||
\toprule
|
||||
\textbf{Source} & \textbf{Samples} & \textbf{Proportion} & \textbf{Modality} \\
|
||||
\textbf{Output} & \textbf{Decoder (TRELLIS)} & \textbf{Typical use} \\
|
||||
\midrule
|
||||
ShapeNet v2 & 51,300 & 0.1\% & Mesh + text \\
|
||||
Objaverse 1.0 & 800,000 & 1.7\% & Mesh + text \\
|
||||
Objaverse-XL & 10,200,000 & 21.6\% & Mesh + text \\
|
||||
ScanNet v2 & 1,513 scenes & 0.003\% & Point cloud + segmentation \\
|
||||
3D-FRONT & 18,000 scenes & 0.04\% & Scene mesh + layout \\
|
||||
Synthetic augmentation & 36,000,000 & 76.5\% & Rendered point clouds \\
|
||||
\midrule
|
||||
\textbf{Total} & \textbf{47,070,813} & 100\% & \\
|
||||
3D Gaussian splats & Gaussian decoder & Fast novel-view rendering \\
|
||||
Radiance field & CP-decomposition volume & Volumetric rendering \\
|
||||
Textured mesh & FlexiCubes + SDF & DCC tools, game/CAD pipelines \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen-3D Training Data Composition}
|
||||
\caption{TRELLIS output formats exposed through the Zen-3D API. Decoders are those of the
|
||||
upstream model \cite{xiang2024trellis}.}
|
||||
\end{table}
|
||||
|
||||
\subsection{Training Protocol}
|
||||
|
||||
Training proceeds in three stages:
|
||||
|
||||
\textbf{Stage 1 -- Tokenizer pretraining} (50K steps): The VQ-VAE is trained standalone on the
|
||||
full 3D corpus with reconstruction loss and commitment loss. Codebook utilization is monitored;
|
||||
dead entries are re-initialized via exponential moving average updates.
|
||||
|
||||
\textbf{Stage 2 -- Backbone pretraining} (200K steps): The Zen MoDE backbone is pretrained on
|
||||
tokenized 3D sequences with a masked-token prediction objective, analogous to masked language
|
||||
modeling but over the 3D token space.
|
||||
|
||||
\textbf{Stage 3 -- Task fine-tuning} (100K steps): All three decoding heads are trained jointly
|
||||
with task-specific losses: cross-entropy for segmentation, binary cross-entropy occupancy loss for
|
||||
reconstruction, and diffusion DDPM loss for generation. Tasks are sampled at ratio 4:3:3.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lllll}
|
||||
\toprule
|
||||
\textbf{Stage} & \textbf{Steps} & \textbf{Batch} & \textbf{LR} & \textbf{Hardware} \\
|
||||
\midrule
|
||||
Tokenizer & 50K & 512 & 1e-4 & 16$\times$A100 80GB \\
|
||||
Backbone & 200K & 256 & 3e-4 & 64$\times$A100 80GB \\
|
||||
Fine-tune & 100K & 128 & 5e-5 & 32$\times$A100 80GB \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Training Configuration by Stage}
|
||||
\end{table}
|
||||
|
||||
\subsection{Data Augmentation}
|
||||
|
||||
To improve robustness to real-world sensor noise, all point cloud inputs during training undergo:
|
||||
random dropout (10--30\% of points), Gaussian jitter ($\sigma = 0.01$ m), random rotation about
|
||||
the vertical axis, and random scale perturbation ($\pm 10\%$).
|
||||
|
||||
\section{Evaluation}
|
||||
|
||||
\subsection{ShapeNet Object Reconstruction}
|
||||
|
||||
We evaluate 3D reconstruction quality on the ShapeNet v2 test split using Intersection over Union
|
||||
(IoU) at 0.5 occupancy threshold.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lccccc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{Chair} & \textbf{Table} & \textbf{Lamp} & \textbf{Car} & \textbf{Mean IoU} \\
|
||||
\midrule
|
||||
ONet & 82.3 & 79.1 & 68.4 & 83.7 & 75.9 \\
|
||||
ConvONet & 88.6 & 85.2 & 74.1 & 88.9 & 82.4 \\
|
||||
IF-Net & 90.1 & 87.3 & 76.8 & 90.2 & 84.6 \\
|
||||
3D-LLM & 91.4 & 88.7 & 78.3 & 91.5 & 87.2 \\
|
||||
\textbf{Zen-3D} & \textbf{95.2} & \textbf{93.8} & \textbf{84.1} & \textbf{95.7} & \textbf{92.2} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{ShapeNet Reconstruction (IoU \%, higher is better)}
|
||||
\end{table}
|
||||
|
||||
\subsection{ScanNet Semantic Segmentation}
|
||||
|
||||
Point cloud semantic segmentation on ScanNet v2 validation set (20 classes).
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{mIoU} & \textbf{mAcc} & \textbf{oAcc} & \textbf{Parameters} \\
|
||||
\midrule
|
||||
PointNet++ & 53.5 & 64.2 & 85.3 & 1.5M \\
|
||||
KPConv & 68.4 & 79.1 & 88.1 & 14.3M \\
|
||||
PointTransformer & 70.6 & 81.9 & 90.2 & 7.8M \\
|
||||
Mask3D & 73.7 & 83.4 & 91.0 & 38.4M \\
|
||||
Point-MAE + ScanNet & 80.3 & 87.2 & 91.8 & 22.1M \\
|
||||
\textbf{Zen-3D} & \textbf{88.3} & \textbf{92.1} & \textbf{94.6} & 8B \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{ScanNet Semantic Segmentation (validation, higher is better)}
|
||||
\end{table}
|
||||
|
||||
\subsection{ModelNet40 Classification}
|
||||
|
||||
Object classification on ModelNet40 with 1,024 input points, 10-fold cross-validation.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{OA (\%)} & \textbf{mAcc (\%)} \\
|
||||
\midrule
|
||||
PointNet & 89.2 & 86.2 \\
|
||||
PointNet++ & 91.9 & 90.7 \\
|
||||
DGCNN & 92.9 & 90.2 \\
|
||||
Point-BERT & 93.2 & 90.6 \\
|
||||
Point-MAE & 93.8 & 90.6 \\
|
||||
PointGPT-L & 94.0 & 91.1 \\
|
||||
\textbf{Zen-3D} & \textbf{94.1} & \textbf{91.3} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{ModelNet40 Classification (OA = Overall Accuracy)}
|
||||
\end{table}
|
||||
|
||||
\subsection{Text-to-3D Generation Quality}
|
||||
|
||||
We evaluate generative quality using the Frechet Point Cloud Distance (FPCD) and CLIP-3D alignment
|
||||
score on a held-out set of 5,000 text prompts from the Cap3D dataset.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{FPCD} $\downarrow$ & \textbf{CLIP-3D} $\uparrow$ \\
|
||||
\midrule
|
||||
Shap-E & 143.2 & 0.312 \\
|
||||
Point-E & 128.7 & 0.331 \\
|
||||
DreamFusion & 115.4 & 0.347 \\
|
||||
One-2-3-45 & 98.3 & 0.369 \\
|
||||
\textbf{Zen-3D} & \textbf{71.4} & \textbf{0.403} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Text-to-3D Generation (Cap3D evaluation set)}
|
||||
\end{table}
|
||||
|
||||
\subsection{Zero-Shot Generalization}
|
||||
|
||||
To assess zero-shot generalization, we evaluate on the Objaverse-LVIS benchmark (1,156 novel
|
||||
categories not seen during training). Zen-3D achieves 61.3\% top-1 classification accuracy,
|
||||
compared to 43.7\% for the best prior approach, demonstrating strong zero-shot transfer from
|
||||
the large-scale pretraining regime.
|
||||
|
||||
\section{Applications}
|
||||
|
||||
\subsection{Robotics Scene Understanding}
|
||||
|
||||
Zen-3D has been integrated into the Hanzo robotics perception stack, where it operates on
|
||||
streaming LiDAR point clouds at 5 Hz (200ms per frame on a single A10G GPU). The model
|
||||
simultaneously provides semantic segmentation for grasp planning and 3D occupancy maps for
|
||||
collision-free path planning.
|
||||
|
||||
\subsection{Augmented Reality}
|
||||
|
||||
For AR applications, Zen-3D's reconstruction head is used to generate watertight meshes from
|
||||
partial RGBD observations in real time, enabling stable AR object placement without depth
|
||||
sensor drift artifacts.
|
||||
|
||||
\subsection{Generative 3D Design}
|
||||
|
||||
The text-to-3D generation pipeline enables designers to specify 3D shapes in natural language
|
||||
and receive printable mesh outputs in under 10 seconds. Integration with standard CAD tools
|
||||
(FreeCAD, Blender) is provided via the Zen SDK.
|
||||
|
||||
\subsection{Scene Counterfactual Editing}
|
||||
|
||||
Zen-3D supports counterfactual scene editing: given a complete scene reconstruction and a natural
|
||||
language instruction (e.g., ``remove the chair and place a desk''), the model regenerates the
|
||||
affected scene region while maintaining geometric consistency with the unchanged context.
|
||||
|
||||
\section{Inference and Deployment}
|
||||
|
||||
\subsection{Inference Performance}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lrrl}
|
||||
\toprule
|
||||
\textbf{Task} & \textbf{Latency (ms)} & \textbf{Throughput} & \textbf{Hardware} \\
|
||||
\midrule
|
||||
Segmentation (10K pts) & 45 & 22 scenes/s & A10G \\
|
||||
Reconstruction (ShapeNet) & 120 & 8 meshes/s & A10G \\
|
||||
Text-to-3D generation & 8,200 & 0.12 shapes/s & A100 \\
|
||||
Classification (1K pts) & 12 & 83 objects/s & T4 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen-3D Inference Performance}
|
||||
\end{table}
|
||||
|
||||
\subsection{Quantization}
|
||||
|
||||
Zen-3D supports INT8 quantization via the Zen SDK with less than 0.5\% degradation on all
|
||||
benchmarks. The quantized model fits in 10GB VRAM, enabling deployment on consumer-grade GPUs
|
||||
(RTX 3080 and above).
|
||||
|
||||
\subsection{Python API}
|
||||
|
||||
\begin{lstlisting}[language=Python, caption=Zen-3D Inference Example]
|
||||
The Zen-3D wrapper mirrors the upstream TRELLIS pipeline. Usage is illustrative:
|
||||
|
||||
\begin{lstlisting}[language=Python, caption=Zen-3D image-to-3D (wrapping TRELLIS)]
|
||||
from zen import Zen3D
|
||||
import numpy as np
|
||||
from PIL import Image
|
||||
|
||||
model = Zen3D.from_pretrained("zenlm/zen-3d-8b")
|
||||
# Loads the redistributed TRELLIS-image-large checkpoint (MIT licensed).
|
||||
model = Zen3D.from_pretrained("zenlm/zen-3d")
|
||||
|
||||
# Semantic segmentation from point cloud
|
||||
points = np.load("scene.npy") # (N, 3) float32
|
||||
labels = model.segment(points, taxonomy="scannet20")
|
||||
image = Image.open("input.png")
|
||||
outputs = model.generate(image)
|
||||
|
||||
# Text-to-3D generation
|
||||
mesh = model.generate(
|
||||
"a modern office chair with armrests",
|
||||
resolution=128,
|
||||
num_steps=50
|
||||
)
|
||||
mesh.export("chair.obj")
|
||||
|
||||
# Scene reconstruction from partial observations
|
||||
partial = points[:points.shape[0] // 2]
|
||||
complete = model.reconstruct(partial)
|
||||
# Decode the same SLAT into the format you need.
|
||||
outputs.gaussians.save("asset.ply") # 3D Gaussian splats
|
||||
outputs.mesh.export("asset.glb") # textured mesh (FlexiCubes)
|
||||
# outputs.radiance_field for volumetric rendering
|
||||
\end{lstlisting}
|
||||
|
||||
\section{Related Work}
|
||||
\subsection{Deployment Notes}
|
||||
|
||||
\subsection{3D Representation Learning}
|
||||
TRELLIS depends on differentiable rendering submodules (e.g.\ a differentiable octree renderer
|
||||
and a modified FlexiCubes) for training and for certain decoding paths; these submodules carry
|
||||
their own licenses (see Section~\ref{sec:licensing}) and have their own build requirements.
|
||||
Inference for the redistributed image model runs on a single modern GPU; for exact hardware
|
||||
requirements, dependency versions, and build steps, follow the upstream project documentation
|
||||
\cite{trellis_repo}.
|
||||
|
||||
PointNet \cite{qi2017pointnet} established the paradigm of operating directly on unordered point
|
||||
sets via symmetric functions. PointNet++ \cite{qi2017pointnet2} extended this with hierarchical
|
||||
local feature learning. Transformer-based approaches (Point-BERT, Point-MAE) adapted masked
|
||||
pretraining from NLP to 3D, achieving strong transfer performance.
|
||||
\section{Licensing and Attribution}
|
||||
\label{sec:licensing}
|
||||
|
||||
\subsection{3D Generation}
|
||||
|
||||
Occupancy Networks \cite{mescheder2019occupancy} and Implicit Functions \cite{park2019deepsdf}
|
||||
enabled continuous implicit representations. Diffusion-based generators (Shap-E, Point-E) brought
|
||||
generative quality closer to 2D image synthesis. Zen-3D advances this line by conditioning
|
||||
generation on the same backbone used for understanding, enabling tighter semantic alignment.
|
||||
|
||||
\subsection{Vision-Language 3D Models}
|
||||
|
||||
3D-LLM and related work demonstrated that language model priors can improve 3D understanding.
|
||||
Zen-3D builds on this direction but replaces the 2D-projection interface with native 3D
|
||||
tokenization, eliminating the projection bottleneck.
|
||||
\begin{itemize}
|
||||
\item \textbf{TRELLIS code and weights: MIT.} The TRELLIS repository and the released
|
||||
checkpoints (including \texttt{TRELLIS-image-large}) are MIT licensed
|
||||
\cite{trellis_repo}. Zen-3D's redistribution retains the upstream MIT license text and
|
||||
copyright notice, and credits Microsoft Research and the TRELLIS authors.
|
||||
\item \textbf{Submodule exceptions.} Some optional submodules are not MIT: the
|
||||
differentiable octree renderer (\texttt{diffoctreerast}) and the modified FlexiCubes are
|
||||
licensed under their own terms \cite{trellis_repo}. Deployments that build these components
|
||||
must comply with those licenses.
|
||||
\item \textbf{Upstream dependencies.} The vision/text encoders used for conditioning
|
||||
(DINOv2, CLIP) and the data sources behind TRELLIS-500K carry their own licenses and
|
||||
usage terms, which downstream users should review independently.
|
||||
\item \textbf{No relicensing of capabilities.} Zen-3D claims no ownership of the TRELLIS
|
||||
model or its outputs' underlying method. Where this report describes architecture or
|
||||
results, the source of record is the TRELLIS paper and repository
|
||||
\cite{xiang2024trellis,trellis_repo}.
|
||||
\end{itemize}
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
Zen-3D demonstrates that a single 8B parameter model, trained with the Zen MoDE architecture
|
||||
and a purpose-built triplane tokenizer, can simultaneously achieve state-of-the-art results
|
||||
across 3D segmentation (ScanNet mIoU 88.3\%), reconstruction (ShapeNet IoU 95.2\%), and
|
||||
classification (ModelNet40 94.1\%). The unified backbone enables cross-task transfer that
|
||||
benefits all three capabilities simultaneously.
|
||||
|
||||
Future work will extend Zen-3D to dynamic 4D scene understanding (point cloud video), tighter
|
||||
integration with the Zen-Omni multimodal backbone for joint 2D+3D understanding, and
|
||||
distillation of the 8B model to a sub-1B edge variant for real-time mobile AR applications.
|
||||
Zen-3D is an honest packaging of Microsoft's TRELLIS into the Zen stack: an MIT-licensed,
|
||||
image-conditioned (and optionally text-conditioned) 3D generator built on the Structured LATent
|
||||
representation and two-stage rectified-flow transformers, capable of decoding to 3D Gaussian
|
||||
splats, radiance fields, or meshes. The model, its architecture, and any reported quality are
|
||||
the work of the TRELLIS authors \cite{xiang2024trellis}; Zen-3D adds only redistribution, an SDK
|
||||
wrapper, output normalization, and serving integration. We make no from-scratch architecture or
|
||||
benchmark claims, and we encourage users to consult the upstream project for authoritative
|
||||
technical detail and evaluation.
|
||||
|
||||
\begin{thebibliography}{10}
|
||||
\bibitem{qi2017pointnet} R. Q. Charles et al., ``PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation,'' CVPR, 2017.
|
||||
\bibitem{qi2017pointnet2} C. R. Qi et al., ``PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space,'' NeurIPS, 2017.
|
||||
\bibitem{mescheder2019occupancy} L. Mescheder et al., ``Occupancy Networks: Learning 3D Reconstruction in Function Space,'' CVPR, 2019.
|
||||
\bibitem{park2019deepsdf} J. J. Park et al., ``DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation,'' CVPR, 2019.
|
||||
\bibitem{xiang2024trellis} J. Xiang, Z. Lv, S. Xu, Y. Deng, R. Wang, B. Zhang, D. Chen, X. Tong, and J. Yang, ``Structured 3D Latents for Scalable and Versatile 3D Generation,'' CVPR, 2025 (arXiv:2412.01506). \url{https://arxiv.org/abs/2412.01506}
|
||||
\bibitem{trellis_repo} Microsoft Research, ``TRELLIS,'' GitHub repository (MIT license; weights \texttt{JeffreyXiang/TRELLIS-image-large}). \url{https://github.com/microsoft/TRELLIS}
|
||||
\end{thebibliography}
|
||||
|
||||
\end{document}
|
||||
|
||||
Binary file not shown.
+191
-239
@@ -33,9 +33,11 @@
|
||||
\vspace{0.5cm}
|
||||
\Huge \textbf{Zen-Director} \\
|
||||
\vspace{0.3cm}
|
||||
\large Video Scene Generation, Storyboarding, and Cinematic Direction \\
|
||||
\large Single-GPU Unified Text- and Image-to-Video Generation \\
|
||||
\vspace{0.2cm}
|
||||
\large \normalsize A Packaging of Alibaba's Wan2.2-TI2V-5B (Apache-2.0) \\
|
||||
\vspace{0.5cm}
|
||||
\normalsize Technical Report v2025.01
|
||||
\normalsize Technical Report
|
||||
}
|
||||
|
||||
\author{
|
||||
@@ -43,23 +45,28 @@
|
||||
Zoo Labs Foundation\thanks{foundation@zoo.ngo}
|
||||
}
|
||||
|
||||
\date{January 2025}
|
||||
\date{2026}
|
||||
|
||||
\begin{document}
|
||||
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
We present \textbf{Zen-Director}, a 7-billion parameter vision-language model specialized for
|
||||
video scene generation, cinematographic planning, and multi-shot storyboard synthesis.
|
||||
Built on the Zen MoDE (Mixture of Distilled Experts) architecture with a temporal transformer
|
||||
extension, Zen-Director understands and generates structured cinematic descriptions---shot lists,
|
||||
scene compositions, camera movements, and narrative arcs---that downstream video generation
|
||||
models can consume directly. The model achieves BLEU-4 of 0.423 on the VideoCaption benchmark,
|
||||
81.2\% accuracy on CinematicQA (our novel evaluation of cinematographic knowledge), and
|
||||
produces storyboards rated 4.1/5.0 by professional cinematographers in blind evaluation.
|
||||
Zen-Director bridges the gap between natural language creative intent and the precise technical
|
||||
specifications required to direct AI video generation systems at professional quality.
|
||||
\textbf{Zen-Director} is a permissively licensed redistribution and integration of
|
||||
\textbf{Wan2.2-TI2V-5B}, the open-source unified text- and image-to-video generation model
|
||||
released by the Alibaba Wan team (\texttt{Wan-AI/Wan2.2-TI2V-5B}) under the
|
||||
\textbf{Apache-2.0 license}~\cite{wan22hf,wan22github}. Zen-Director is \emph{not} a
|
||||
from-scratch model: it packages Wan2.2-TI2V-5B---a 5-billion-parameter \emph{dense} diffusion
|
||||
transformer trained with flow matching, paired with the high-compression Wan2.2 3D causal
|
||||
VAE---into the Zen model family with consistent tooling, weights distribution, and
|
||||
serving integration. The defining property of Wan2.2-TI2V-5B is that it generates 720P video
|
||||
at 24~fps on a \emph{single consumer-grade GPU} (e.g. an NVIDIA RTX~4090 with under 24~GB of
|
||||
VRAM), generating a 5-second 720P clip in under nine minutes without specialized
|
||||
optimization~\cite{wan22hf,wan22github}. A single unified framework natively serves both
|
||||
text-to-video (T2V) and image-to-video (I2V). This report documents the model we redistribute,
|
||||
its true architecture, its capabilities, and how it is integrated into the Zen stack. All
|
||||
architectural facts and figures below are reported by the upstream Wan team; we add no
|
||||
independent benchmarks.
|
||||
\end{abstract}
|
||||
|
||||
\tableofcontents
|
||||
@@ -67,19 +74,23 @@ specifications required to direct AI video generation systems at professional qu
|
||||
|
||||
\section{Introduction}
|
||||
|
||||
The rapid maturation of text-to-video generation systems has exposed a critical bottleneck:
|
||||
the gap between a creator's high-level creative intent and the precise technical specifications
|
||||
these systems require to produce coherent, professional-quality video. Current generation
|
||||
workflows demand that users specify camera angles, focal lengths, lighting conditions, shot
|
||||
durations, and scene transitions---knowledge traditionally held by professional cinematographers
|
||||
and directors.
|
||||
Open-source video generation has reached the point where high-resolution, high-frame-rate
|
||||
synthesis is feasible on a single consumer GPU. The Alibaba Wan team's \textbf{Wan2.2}
|
||||
family~\cite{wan22github,wan21paper} is a leading example of this shift. Within that family,
|
||||
\textbf{Wan2.2-TI2V-5B} is a compact 5-billion-parameter \emph{dense} model that unifies
|
||||
text-to-video and image-to-video generation and runs within the memory budget of widely
|
||||
available hardware. Crucially, both the code and the weights are released under the
|
||||
\textbf{Apache-2.0 license}~\cite{wan22hf,wan22github}, which permits commercial redistribution
|
||||
and modification with attribution.
|
||||
|
||||
Zen-Director addresses this bottleneck as a \textbf{directorial intelligence}: a model that
|
||||
understands narrative intent and translates it into actionable cinematic specifications. Rather
|
||||
than generating video pixels directly, Zen-Director generates structured \textbf{shot plans}:
|
||||
hierarchical scene descriptions that video generation systems consume as structured prompts.
|
||||
\textbf{Zen-Director} is our packaging of Wan2.2-TI2V-5B. We make no claim to having designed,
|
||||
trained, or fine-tuned a new architecture. Our contribution is integration: distributing the
|
||||
upstream Apache-2.0 weights through the Zen model family, wiring the model into the Zen serving
|
||||
and tooling stack, and providing a consistent interface alongside other Zen models. This
|
||||
document therefore describes the \emph{real} Wan2.2-TI2V-5B architecture and capabilities so
|
||||
that users of Zen-Director understand exactly what they are running and under what license.
|
||||
|
||||
\subsection{Model Overview}
|
||||
\subsection{Provenance and License}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
@@ -87,261 +98,202 @@ hierarchical scene descriptions that video generation systems consume as structu
|
||||
\toprule
|
||||
\textbf{Property} & \textbf{Value} \\
|
||||
\midrule
|
||||
Parameters & 7B \\
|
||||
Architecture & Zen MoDE 7B + Temporal Transformer \\
|
||||
Context Length & 32K tokens + 512 video frames \\
|
||||
Visual Encoder & ViT-L/14 (336px) \\
|
||||
Temporal Depth & 12 temporal transformer layers \\
|
||||
Shot Plan Format & Structured JSON + prose description \\
|
||||
Training Data & 8.2M scene-annotation pairs, 1.4M film scripts \\
|
||||
Upstream model & Wan2.2-TI2V-5B \\
|
||||
Upstream author & Alibaba Wan team (Wan-Video / Wan-AI) \\
|
||||
Upstream repository & \texttt{Wan-AI/Wan2.2-TI2V-5B} (Hugging Face) \\
|
||||
License & Apache-2.0 \\
|
||||
Zen-Director role & Packaging / integration, not from-scratch training \\
|
||||
Modality & Unified text-to-video (T2V) + image-to-video (I2V) \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen-Director Model Specifications}
|
||||
\caption{Zen-Director provenance. Zen-Director redistributes the upstream model under its
|
||||
original Apache-2.0 license with attribution to the Alibaba Wan team.}
|
||||
\end{table}
|
||||
|
||||
\subsection{Model Overview}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{ll}
|
||||
\toprule
|
||||
\textbf{Property} & \textbf{Value (as reported upstream~\cite{wan22hf,wan22github})} \\
|
||||
\midrule
|
||||
Parameters & 5B (dense; not Mixture-of-Experts) \\
|
||||
Architecture & Diffusion Transformer (DiT) + Wan2.2 3D causal VAE \\
|
||||
Text encoder & umT5 (multilingual) \\
|
||||
Training objective & Flow matching \\
|
||||
VAE compression ($T\times H\times W$) & $4\times16\times16$ (overall rate $\approx 64$) \\
|
||||
With patchification & $4\times32\times32$ \\
|
||||
Output resolution & 720P ($1280\times704$ or $704\times1280$) \\
|
||||
Frame rate & 24 fps \\
|
||||
Clip length & up to 5 seconds \\
|
||||
Text input & up to 512 tokens \\
|
||||
Image input (I2V) & up to $1280\times704$ \\
|
||||
GPU requirement & single consumer GPU, $<24$ GB VRAM (e.g. RTX 4090) \\
|
||||
Reported speed & 5-second 720P clip in $<9$ minutes (single GPU, no special optimization) \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Wan2.2-TI2V-5B specifications as reported by the upstream Wan team. Zen-Director
|
||||
redistributes these weights unchanged.}
|
||||
\end{table}
|
||||
|
||||
\subsection{Key Capabilities}
|
||||
|
||||
\begin{itemize}
|
||||
\item \textbf{Scene storyboarding}: Convert natural language scene descriptions into
|
||||
complete shot-by-shot storyboards with camera specifications.
|
||||
\item \textbf{Shot composition}: Recommend and generate compositional guidelines (rule of
|
||||
thirds, leading lines, depth of field) for each shot.
|
||||
\item \textbf{Narrative arc planning}: Structure multi-scene video narratives with consistent
|
||||
pacing, tension arcs, and visual motifs.
|
||||
\item \textbf{Cinematic vocabulary}: Understand and generate industry-standard cinematographic
|
||||
terminology (establishing shot, dolly zoom, rack focus, etc.).
|
||||
\item \textbf{Video comprehension}: Analyze existing video clips and generate directorial
|
||||
notes describing their cinematographic techniques.
|
||||
\item \textbf{Unified text-to-video}: Generate a 720P, 24~fps clip (up to 5~seconds) from
|
||||
a text prompt of up to 512 tokens.
|
||||
\item \textbf{Unified image-to-video}: Animate a reference image (up to $1280\times704$)
|
||||
into a coherent video within the same single framework.
|
||||
\item \textbf{Single-GPU operation}: Run end-to-end on one consumer-grade GPU with under
|
||||
24~GB of VRAM, enabled by the high-compression 3D causal VAE.
|
||||
\item \textbf{Permissive licensing}: Apache-2.0 weights and code, suitable for commercial
|
||||
redistribution with attribution.
|
||||
\end{itemize}
|
||||
|
||||
\section{Architecture}
|
||||
|
||||
\subsection{Zen MoDE 7B Language Backbone}
|
||||
Zen-Director runs the Wan2.2-TI2V-5B model unchanged. Its architecture follows the Wan design
|
||||
lineage~\cite{wan21paper,wan22github}: a 3D causal variational autoencoder (VAE), a umT5 text
|
||||
encoder, and a diffusion transformer (DiT), trained with flow matching. We describe each
|
||||
component as documented upstream.
|
||||
|
||||
The language backbone is Zen MoDE at 7B scale: 28 transformer layers, 28 attention heads with
|
||||
grouped-query attention (4 KV heads), and MoE feed-forward networks (4 experts, top-2 routing).
|
||||
This provides strong natural language understanding for parsing creative briefs and generating
|
||||
rich cinematic descriptions.
|
||||
\subsection{5B Dense Diffusion Transformer}
|
||||
|
||||
\subsection{Visual Encoder}
|
||||
The generative backbone is a \emph{dense} diffusion transformer with approximately 5 billion
|
||||
parameters~\cite{wan22hf,wan22github}. Unlike the larger Wan2.2 A14B variants---which use a
|
||||
Mixture-of-Experts design with a high-noise expert for early denoising and a low-noise expert
|
||||
for later refinement---the TI2V-5B model is a single dense network with no expert routing. The
|
||||
DiT operates on the compressed video latents produced by the VAE: a patchifying module forms
|
||||
tokens from the latent, a stack of transformer blocks denoises them while attending to the text
|
||||
embedding via cross-attention, and an unpatchifying module reconstructs the latent.
|
||||
|
||||
A ViT-L/14 vision encoder operating at 336px resolution encodes reference images and video
|
||||
frames into 256 visual tokens per frame. A two-layer MLP projection maps visual tokens into
|
||||
the language model's embedding space.
|
||||
\subsection{High-Compression Wan2.2 3D Causal VAE}
|
||||
|
||||
\subsection{Temporal Transformer Extension}
|
||||
The capability that lets a 720P 24~fps video model fit on a single 24~GB GPU is the
|
||||
high-compression \textbf{Wan2.2-VAE}, a 3D causal autoencoder~\cite{wan22github,wan22hf}. It
|
||||
achieves a spatio-temporal ($T\times H\times W$) compression ratio of $4\times16\times16$,
|
||||
raising the overall information compression rate to roughly~64 while maintaining high-quality
|
||||
reconstruction. Combined with the patchification applied inside the DiT, the effective spatial
|
||||
reduction reaches $4\times32\times32$. The 3D \emph{causal} structure preserves temporal
|
||||
causality across frames and substantially reduces the memory footprint of the latent sequence,
|
||||
which is what makes single-GPU 720P generation practical.
|
||||
|
||||
The key architectural innovation is a 12-layer temporal transformer that operates \textit{across}
|
||||
frames rather than within them. Given $T$ frames, each encoded to 256 tokens, the temporal
|
||||
transformer attends over the time dimension to build a coherent representation of motion,
|
||||
continuity, and scene evolution:
|
||||
\subsection{umT5 Text Encoder}
|
||||
|
||||
Text conditioning uses the umT5 multilingual text encoder~\cite{wan21paper}, which encodes the
|
||||
input prompt (up to 512 tokens) into embeddings consumed by the DiT through cross-attention.
|
||||
The Wan team selected umT5 for its strong multilingual (notably Chinese and English) encoding
|
||||
and favorable training convergence.
|
||||
|
||||
\subsection{Flow-Matching Training Objective}
|
||||
|
||||
The diffusion transformer is trained with \textbf{flow matching}~\cite{wan21paper}. Given a
|
||||
target latent $x_1$, a noise sample $x_0$, and a timestep $t \in [0,1]$, the training input is
|
||||
the linear interpolation
|
||||
\begin{equation}
|
||||
H_{\text{temporal}} = \text{TransformerEncoder}([h_1^{CLS}, h_2^{CLS}, \ldots, h_T^{CLS}])
|
||||
x_t = (1-t)\,x_0 + t\,x_1 ,
|
||||
\end{equation}
|
||||
and the model is trained to predict the velocity field
|
||||
\begin{equation}
|
||||
v_t = x_1 - x_0 .
|
||||
\end{equation}
|
||||
At inference, samples are produced by integrating the learned velocity field, avoiding iterative
|
||||
per-step velocity re-prediction schemes used by some alternative diffusion formulations.
|
||||
|
||||
where $h_t^{CLS}$ is the CLS token representation of frame $t$ from the visual encoder.
|
||||
The temporal representation is concatenated with text tokens before the language model's
|
||||
cross-attention layers.
|
||||
\subsection{Unified T2V + I2V Framework}
|
||||
|
||||
\subsection{Shot Plan Generation}
|
||||
A single model serves both modalities~\cite{wan22hf,wan22github}. For text-to-video the model is
|
||||
conditioned on the umT5 text embedding alone; for image-to-video the reference image (up to
|
||||
$1280\times704$) provides additional conditioning. Both paths share the same VAE, DiT, and
|
||||
flow-matching sampler, so no separate model or checkpoint is required to switch between
|
||||
text-driven and image-driven generation.
|
||||
|
||||
Zen-Director generates shot plans as structured JSON objects:
|
||||
\section{Inference and Integration}
|
||||
|
||||
\begin{lstlisting}[language=Python, caption=Shot Plan Schema]
|
||||
{
|
||||
"scene": {
|
||||
"id": "s01",
|
||||
"location": "rain-soaked rooftop, night",
|
||||
"mood": "tense, noir",
|
||||
"duration_sec": 45
|
||||
},
|
||||
"shots": [
|
||||
{
|
||||
"id": "s01_001",
|
||||
"type": "establishing",
|
||||
"camera": {"position": "wide", "angle": "high angle", "move": "slow push in"},
|
||||
"focal_length": "24mm",
|
||||
"subject": "city skyline with protagonist silhouette",
|
||||
"duration_sec": 8,
|
||||
"lighting": "practical neon, rain reflections",
|
||||
"notes": "City should feel overwhelming relative to protagonist"
|
||||
}
|
||||
]
|
||||
}
|
||||
\subsection{Running Zen-Director}
|
||||
|
||||
Because Zen-Director is the upstream Wan2.2-TI2V-5B model, it is invoked through the standard
|
||||
Wan2.2 generation interface. The example below illustrates the conceptual call (refer to the
|
||||
upstream repository for the authoritative API and flags)~\cite{wan22github}.
|
||||
|
||||
\begin{lstlisting}[language=Python, caption=Invoking Zen-Director (Wan2.2-TI2V-5B)]
|
||||
# Text-to-video: 720P, 24 fps, up to 5 seconds, single 24GB GPU
|
||||
generate(
|
||||
task="ti2v-5B",
|
||||
size="1280*704",
|
||||
prompt="a lighthouse at dusk, waves breaking, slow push-in",
|
||||
)
|
||||
|
||||
# Image-to-video: animate a reference image in the same framework
|
||||
generate(
|
||||
task="ti2v-5B",
|
||||
size="1280*704",
|
||||
image="reference.png",
|
||||
prompt="gentle camera drift, wind moving through grass",
|
||||
)
|
||||
\end{lstlisting}
|
||||
|
||||
\section{Training}
|
||||
\subsection{Hardware Footprint}
|
||||
|
||||
\subsection{Dataset}
|
||||
The upstream Wan team reports that Wan2.2-TI2V-5B generates a 5-second 720P (24~fps) clip in
|
||||
under nine minutes on a single consumer-grade GPU, without specialized optimization, within a
|
||||
sub-24~GB VRAM budget such as an NVIDIA RTX~4090~\cite{wan22hf,wan22github}. This single-GPU
|
||||
profile is the primary reason Zen-Director is suitable for self-hosted and on-premise
|
||||
deployments where multi-GPU clusters are unavailable.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lrrl}
|
||||
\toprule
|
||||
\textbf{Source} & \textbf{Samples} & \textbf{Proportion} & \textbf{Content} \\
|
||||
\midrule
|
||||
Film scripts + frames & 2,400,000 & 29.3\% & Script-to-scene alignment \\
|
||||
Cinematography textbooks & 180,000 & 2.2\% & Technical knowledge \\
|
||||
Film criticism corpus & 820,000 & 10.0\% & Aesthetic analysis \\
|
||||
Video-caption pairs & 3,100,000 & 37.8\% & Visual understanding \\
|
||||
Storyboard collections & 1,700,000 & 20.7\% & Shot plan examples \\
|
||||
\midrule
|
||||
\textbf{Total} & \textbf{8,200,000} & 100\% & \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen-Director Training Data}
|
||||
\end{table}
|
||||
\subsection{Role in the Zen Stack}
|
||||
|
||||
\subsection{Training Protocol}
|
||||
Within the Zen model family, Zen-Director is the video-generation component. It consumes a text
|
||||
prompt (and optionally a reference image) and produces video, complementing other Zen models
|
||||
that handle language and image tasks. Integration consists of distributing the Apache-2.0
|
||||
weights, exposing the unified T2V/I2V interface through Zen tooling, and providing consistent
|
||||
serving alongside the rest of the family. No retraining or architectural modification is
|
||||
performed.
|
||||
|
||||
\textbf{Stage 1 -- Visual encoder alignment} (20K steps): The ViT encoder and MLP projection
|
||||
are trained to align visual representations with cinematic language descriptions.
|
||||
\section{Reported Results}
|
||||
|
||||
\textbf{Stage 2 -- Temporal pretraining} (40K steps): The temporal transformer is pretrained
|
||||
on video sequences with a masked-frame prediction objective.
|
||||
The Wan team positions Wan2.2-TI2V-5B as one of the fastest 720P@24fps open models currently
|
||||
available and reports favorable quality on its internal Wan-Bench~2.0 evaluation relative to
|
||||
contemporary models~\cite{wan22github}. The public model card does not publish a specific
|
||||
per-metric VBench table for the TI2V-5B variant, and \emph{we deliberately report no
|
||||
independent benchmark numbers of our own}: Zen-Director redistributes the upstream weights
|
||||
without modification, so any quantitative claims should be taken from, and attributed to, the
|
||||
upstream Wan releases~\cite{wan22hf,wan22github,wan21paper}.
|
||||
|
||||
\textbf{Stage 3 -- Directorial SFT} (60K steps): The full model is fine-tuned on storyboard
|
||||
generation, scene description, and cinematographic Q\&A tasks jointly.
|
||||
\section{Licensing and Attribution}
|
||||
|
||||
\textbf{Stage 4 -- RLHF from cinematographers} (15K steps): A reward model trained on 50,000
|
||||
pairwise comparisons from professional cinematographers (recruited via film school partnerships)
|
||||
is used to further refine shot plan quality via PPO.
|
||||
|
||||
\section{Evaluation}
|
||||
|
||||
\subsection{VideoCaption Benchmark}
|
||||
|
||||
We evaluate video description quality on a held-out set of 5,000 film clips spanning 20 genres.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{BLEU-4} & \textbf{METEOR} & \textbf{CIDEr} & \textbf{ROUGE-L} \\
|
||||
\midrule
|
||||
BLIP-2 & 0.281 & 0.314 & 0.872 & 0.512 \\
|
||||
InstructBLIP & 0.308 & 0.341 & 0.941 & 0.534 \\
|
||||
VideoChat2 & 0.352 & 0.374 & 1.021 & 0.561 \\
|
||||
Video-LLaMA2 & 0.387 & 0.402 & 1.103 & 0.584 \\
|
||||
\textbf{Zen-Director} & \textbf{0.423} & \textbf{0.441} & \textbf{1.187} & \textbf{0.612} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{VideoCaption Benchmark Results (higher is better)}
|
||||
\end{table}
|
||||
|
||||
\subsection{CinematicQA}
|
||||
|
||||
CinematicQA is a novel benchmark we introduce comprising 2,000 multiple-choice questions
|
||||
testing cinematographic knowledge: shot types, camera movements, lighting techniques, editing
|
||||
principles, and genre conventions. Questions were authored by five professional cinematographers.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{Accuracy (\%)} \\
|
||||
\midrule
|
||||
GPT-4o (zero-shot) & 68.3 \\
|
||||
Claude 3.5 Sonnet & 71.4 \\
|
||||
Gemini 1.5 Pro & 66.8 \\
|
||||
Specialist fine-tuned 7B & 74.2 \\
|
||||
\textbf{Zen-Director 7B} & \textbf{81.2} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{CinematicQA Accuracy (\%)}
|
||||
\end{table}
|
||||
|
||||
\subsection{Professional Storyboard Evaluation}
|
||||
|
||||
Twenty professional cinematographers evaluated storyboards generated from 100 scene descriptions,
|
||||
rating each on a 5-point scale across five dimensions.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lc}
|
||||
\toprule
|
||||
\textbf{Dimension} & \textbf{Mean Score (1--5)} \\
|
||||
\midrule
|
||||
Technical accuracy of shot specifications & 4.3 \\
|
||||
Narrative coherence across shots & 4.1 \\
|
||||
Creative quality / originality & 3.9 \\
|
||||
Pacing appropriateness & 4.0 \\
|
||||
Overall directorial vision & 4.1 \\
|
||||
\midrule
|
||||
\textbf{Overall Mean} & \textbf{4.1} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Professional Cinematographer Evaluation (N=20 evaluators, 100 storyboards)}
|
||||
\end{table}
|
||||
|
||||
\subsection{Downstream Video Generation Quality}
|
||||
|
||||
We evaluate whether Zen-Director shot plans improve final video quality when used as structured
|
||||
prompts for a text-to-video generation system. Using the same creative brief:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Prompt Method} & \textbf{Human Preference (\%)} & \textbf{FVD} $\downarrow$ \\
|
||||
\midrule
|
||||
Raw creative brief (baseline) & 18.4\% & 412 \\
|
||||
Manual cinematographer spec & 42.1\% & 287 \\
|
||||
Zen-Director shot plan & 39.5\% & 298 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Downstream Video Quality with Zen-Director Shot Plans}
|
||||
\end{table}
|
||||
|
||||
Zen-Director reaches 94\% of manual cinematographer performance at a fraction of the cost and
|
||||
time, validating its role as an effective creative intermediary.
|
||||
|
||||
\section{Applications}
|
||||
|
||||
\subsection{AI Film Production Pipeline}
|
||||
|
||||
Zen-Director is integrated into the Hanzo AI film production pipeline as the directorial layer
|
||||
between a human creative brief and the Zen-Video generation system. A typical workflow:
|
||||
|
||||
\begin{enumerate}
|
||||
\item Human writes a scene description in natural language.
|
||||
\item Zen-Director generates a shot plan JSON with full cinematographic specs.
|
||||
\item Human reviews and optionally edits the shot plan (typically 3--5 minutes).
|
||||
\item Zen-Video consumes the shot plan and generates video clips per shot.
|
||||
\item Zen-Director evaluates temporal consistency across clips and suggests retakes.
|
||||
\end{enumerate}
|
||||
|
||||
\subsection{Film Education}
|
||||
|
||||
Zen-Director serves as an interactive cinematography tutor: students can submit scene
|
||||
descriptions and receive expert-level directorial notes explaining the reasoning behind
|
||||
each shot choice.
|
||||
|
||||
\subsection{Video Game Cutscene Direction}
|
||||
|
||||
Game studios use Zen-Director to generate cinematic specifications for in-engine cutscene
|
||||
directors, reducing the time from narrative script to playable sequence by an estimated 60\%.
|
||||
Zen-Director is governed by the \textbf{Apache-2.0 license} of the upstream
|
||||
Wan2.2-TI2V-5B release~\cite{wan22hf,wan22github}. Redistribution under the Zen name retains
|
||||
this license and includes attribution to the Alibaba Wan team as the originating authors of the
|
||||
model, weights, and architecture. Apache-2.0 permits commercial use, modification, and
|
||||
redistribution provided the license and notices are preserved. Users remain responsible for
|
||||
ensuring that generated content complies with applicable laws and does not infringe third-party
|
||||
rights.
|
||||
|
||||
\section{Related Work}
|
||||
|
||||
Video understanding models (VideoChat, Video-LLaMA) focus primarily on description generation.
|
||||
Text-to-video generation systems (Sora, CogVideo, ModelScope) focus on pixel synthesis.
|
||||
Zen-Director uniquely occupies the directorial planning layer between these two stages,
|
||||
drawing on the rich tradition of computational narrative research and cinematography theory.
|
||||
Wan2.2-TI2V-5B belongs to the broader Wan family of open large-scale video generative
|
||||
models~\cite{wan21paper,wan22github}, which also includes the larger A14B Mixture-of-Experts
|
||||
text-to-video and image-to-video models. The Wan line builds on the diffusion-transformer
|
||||
approach to video synthesis, combining a 3D causal VAE for efficient latent representation, a
|
||||
multilingual text encoder, and flow-matching training. Zen-Director's specific value is in
|
||||
making the compact, single-GPU TI2V-5B configuration of this family conveniently available and
|
||||
integrated within the Zen stack.
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
Zen-Director establishes a new model category: the AI cinematographer. By training on film
|
||||
scripts, storyboards, and cinematographic knowledge at 7B scale, the model achieves 81.2\%
|
||||
on CinematicQA and generates shot plans rated 4.1/5 by professional cinematographers.
|
||||
Integration with the Zen-Video generation backbone creates a complete AI film production
|
||||
pipeline from creative brief to rendered footage.
|
||||
Zen-Director is an honest redistribution of Alibaba's Apache-2.0 Wan2.2-TI2V-5B model: a 5B
|
||||
dense diffusion transformer with a high-compression 3D causal VAE that generates 720P, 24~fps
|
||||
video from text or images on a single consumer GPU. It is not a from-scratch architecture and
|
||||
introduces no new benchmarks; its contribution is packaging and integration. By attributing the
|
||||
upstream Wan team and preserving the Apache-2.0 license, Zen-Director brings state-of-the-art,
|
||||
permissively licensed, single-GPU video generation into the Zen model family with full
|
||||
transparency about its provenance.
|
||||
|
||||
\begin{thebibliography}{10}
|
||||
\bibitem{li2023blip} J. Li et al., ``BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models,'' ICML, 2023.
|
||||
\bibitem{videochat} K. Li et al., ``VideoChat: Chat-Centric Video Understanding,'' arXiv:2305.06355, 2023.
|
||||
\bibitem{makarov2022} A. Makarov et al., ``Computational Cinematography: A Survey,'' IEEE TPAMI, 2022.
|
||||
\bibitem{wan22hf} Alibaba Wan team, ``Wan2.2-TI2V-5B,'' Hugging Face model card, \texttt{Wan-AI/Wan2.2-TI2V-5B}. \url{https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B}. License: Apache-2.0.
|
||||
\bibitem{wan22github} Wan-Video, ``Wan2.2: Open and Advanced Large-Scale Video Generative Models,'' GitHub repository. \url{https://github.com/Wan-Video/Wan2.2}. License: Apache-2.0.
|
||||
\bibitem{wan21paper} Wan team, ``Wan: Open and Advanced Large-Scale Video Generative Models,'' arXiv:2503.20314, 2025. \url{https://arxiv.org/abs/2503.20314}.
|
||||
\end{thebibliography}
|
||||
|
||||
\end{document}
|
||||
|
||||
Binary file not shown.
-373
@@ -1,373 +0,0 @@
|
||||
\documentclass[11pt,a4paper]{article}
|
||||
\usepackage[utf8]{inputenc}
|
||||
\usepackage[T1]{fontenc}
|
||||
\usepackage{amsmath,amsfonts,amssymb}
|
||||
\usepackage{graphicx}
|
||||
\usepackage{hyperref}
|
||||
\usepackage{listings}
|
||||
\usepackage[dvipsnames]{xcolor}
|
||||
\usepackage{booktabs}
|
||||
\usepackage{float}
|
||||
\usepackage{geometry}
|
||||
\geometry{margin=1in}
|
||||
|
||||
\definecolor{zenblue}{RGB}{41,121,255}
|
||||
\definecolor{zengreen}{RGB}{52,199,89}
|
||||
\definecolor{codegray}{RGB}{245,245,245}
|
||||
|
||||
\hypersetup{colorlinks=true,linkcolor=zenblue,urlcolor=zenblue,citecolor=zenblue}
|
||||
|
||||
\lstset{
|
||||
backgroundcolor=\color{codegray},
|
||||
basicstyle=\ttfamily\small,
|
||||
breaklines=true,
|
||||
captionpos=b,
|
||||
frame=single,
|
||||
numbers=left,
|
||||
numberstyle=\tiny\color{gray}
|
||||
}
|
||||
|
||||
\title{
|
||||
\vspace{-2cm}
|
||||
\Large \textbf{Zen AI Model Family} \\
|
||||
\vspace{0.5cm}
|
||||
\Huge \textbf{Zen-Foley} \\
|
||||
\vspace{0.3cm}
|
||||
\large Intelligent Audio Effects Generation and Video-Sound Synchronization \\
|
||||
\vspace{0.5cm}
|
||||
\normalsize Technical Report v2025.01
|
||||
}
|
||||
|
||||
\author{
|
||||
Hanzo AI Research Team\thanks{research@hanzo.ai} \and
|
||||
Zoo Labs Foundation\thanks{foundation@zoo.ngo}
|
||||
}
|
||||
|
||||
\date{January 2025}
|
||||
|
||||
\begin{document}
|
||||
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
We present \textbf{Zen-Foley}, a 1.5-billion parameter audio generation model specialized for
|
||||
sound effect synthesis, audio-video synchronization, and spatial audio rendering. Named after
|
||||
Jack Foley, the pioneer of synchronized sound production, Zen-Foley learns from a corpus of
|
||||
4.2 million video-audio pairs and 800,000 annotated sound effect libraries to generate
|
||||
contextually appropriate, physically plausible audio effects synchronized to video content.
|
||||
The model achieves a mean average precision (mAP) of 0.432 on AudioSet classification,
|
||||
produces sound effects rated 4.2/5.0 MOS (Mean Opinion Score) by audio engineers, and
|
||||
reduces audio-video synchronization error to under 40ms on the AVSync benchmark. Zen-Foley
|
||||
operates in three modes: video-conditioned generation (produces sound track matching visual
|
||||
events), text-conditioned generation (produces sound from natural language descriptions),
|
||||
and spatial audio rendering (positions sound sources in 3D space for immersive audio).
|
||||
\end{abstract}
|
||||
|
||||
\tableofcontents
|
||||
\newpage
|
||||
|
||||
\section{Introduction}
|
||||
|
||||
Professional audio post-production for video is a time-intensive craft. A single minute of
|
||||
film dialogue requires hours of Foley work: recording footsteps, clothing rustles, object
|
||||
impacts, and environmental ambience that were not captured on set. Automation of this process
|
||||
has long been sought but has remained elusive due to the complex physical and perceptual
|
||||
relationships between visual events and their associated sounds.
|
||||
|
||||
Zen-Foley addresses this challenge through a generative model that is jointly trained on
|
||||
visual and audio modalities, learning the mapping between physical events in video and their
|
||||
acoustic signatures. The model operates at three levels of granularity:
|
||||
|
||||
\begin{enumerate}
|
||||
\item \textbf{Event-level}: Generating a specific sound for a specific visual event
|
||||
(a door slamming, footsteps on gravel, glass breaking).
|
||||
\item \textbf{Scene-level}: Generating a complete ambient soundscape for a scene
|
||||
(a busy city street, a quiet forest, a factory floor).
|
||||
\item \textbf{Spatial}: Positioning sound sources in 3D space relative to camera
|
||||
position, with appropriate reverberation and distance cues.
|
||||
\end{enumerate}
|
||||
|
||||
\subsection{Model Overview}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{ll}
|
||||
\toprule
|
||||
\textbf{Property} & \textbf{Value} \\
|
||||
\midrule
|
||||
Parameters & 1.5B \\
|
||||
Architecture & Conditional diffusion (latent audio) + visual cross-attention \\
|
||||
Audio Representation & Mel spectrogram, 128 bins, 44.1kHz \\
|
||||
Max Audio Duration & 30 seconds per generation \\
|
||||
Spatial Audio & First-order Ambisonics (4-channel) \\
|
||||
Latency (10s clip) & 2.1s on A10G \\
|
||||
Training Data & 4.2M video-audio pairs, 800K SFX libraries \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen-Foley Model Specifications}
|
||||
\end{table}
|
||||
|
||||
\section{Architecture}
|
||||
|
||||
\subsection{Audio Representation}
|
||||
|
||||
Zen-Foley operates in the latent space of a pre-trained audio VAE. The VAE encodes 44.1kHz
|
||||
stereo audio into a compact latent representation at 50Hz temporal resolution with 64
|
||||
latent channels. This reduces the dimensionality of audio generation by 43$\times$ relative
|
||||
to raw waveform synthesis while preserving perceptually relevant features.
|
||||
|
||||
The VAE decoder converts latent representations back to mel spectrograms, which are then
|
||||
vocoded to waveforms using a HiFi-GAN vocoder fine-tuned jointly with the main model.
|
||||
|
||||
\subsection{Visual Conditioning}
|
||||
|
||||
Video frames are encoded using a lightweight ViT-S/16 encoder at 224px resolution. For a
|
||||
video of $T$ frames at 24fps, we sample 8 frames per second and encode each independently.
|
||||
Temporal context is provided by a 4-layer temporal transformer that produces a sequence of
|
||||
\textbf{event embeddings} aligned to the audio time axis.
|
||||
|
||||
The core conditioning mechanism uses cross-attention between audio latent tokens (at 50Hz)
|
||||
and visual event embeddings (at 8fps), with interpolated alignment:
|
||||
|
||||
\begin{equation}
|
||||
z_{\text{audio}}^t = \text{CrossAttend}(z_{\text{audio}}^t, \{e_v^{t'} : |t - t'| \leq W\})
|
||||
\end{equation}
|
||||
|
||||
where $W = 0.5$s is a causal window preventing the model from using future visual information
|
||||
to generate past audio.
|
||||
|
||||
\subsection{Diffusion Backbone}
|
||||
|
||||
The generative backbone is a U-Net diffusion model operating on the 64-channel audio latent
|
||||
space. The U-Net has four encoder stages (1024, 512, 256, 128 channels) and symmetric decoder
|
||||
stages. Each stage uses 2 residual blocks with self-attention at the two lowest-resolution
|
||||
stages. Conditioning is injected via:
|
||||
|
||||
\begin{itemize}
|
||||
\item \textbf{Visual cross-attention}: In every attention layer.
|
||||
\item \textbf{Text conditioning}: CLIP text embeddings added to the diffusion timestep
|
||||
embedding and concatenated to the U-Net bottleneck.
|
||||
\item \textbf{Spatial conditioning}: Camera position and listener orientation (azimuth,
|
||||
elevation) encoded as learned embeddings and injected at the bottleneck.
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Spatial Audio Rendering}
|
||||
|
||||
For spatial audio output, Zen-Foley generates four channels corresponding to First-order
|
||||
Ambisonics (W, X, Y, Z). A source position network predicts the 3D trajectory of each sound
|
||||
source from visual features, and the Ambisonic encoder applies appropriate gain patterns:
|
||||
|
||||
\begin{align}
|
||||
W &= \frac{1}{\sqrt{2}} S \\
|
||||
X &= S \cos\phi \cos\theta \\
|
||||
Y &= S \cos\phi \sin\theta \\
|
||||
Z &= S \sin\phi
|
||||
\end{align}
|
||||
|
||||
where $S$ is the source signal, $\phi$ is elevation, and $\theta$ is azimuth.
|
||||
|
||||
\section{Training}
|
||||
|
||||
\subsection{Dataset}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lrrl}
|
||||
\toprule
|
||||
\textbf{Source} & \textbf{Samples} & \textbf{Proportion} & \textbf{Description} \\
|
||||
\midrule
|
||||
AudioSet (video-audio) & 2,000,000 & 47.6\% & YouTube video clips, 526 classes \\
|
||||
VGGSound & 200,000 & 4.8\% & Visual-audio correspondence \\
|
||||
FreeSound + video sync & 400,000 & 9.5\% & SFX library aligned to video \\
|
||||
BBC Sound Effects & 33,000 & 0.8\% & Professional SFX library \\
|
||||
Film/TV Foley sessions & 800,000 & 19.0\% & Professional Foley recordings \\
|
||||
Synthetic augmentation & 767,000 & 18.3\% & Physics-based audio simulation \\
|
||||
\midrule
|
||||
\textbf{Total} & \textbf{4,200,000} & 100\% & \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen-Foley Training Data Composition}
|
||||
\end{table}
|
||||
|
||||
\subsection{Training Protocol}
|
||||
|
||||
\textbf{Stage 1 -- Audio VAE pretraining} (100K steps on AudioSet): The VAE learns a general
|
||||
audio representation with reconstruction loss ($L_2$ in mel space) + perceptual loss
|
||||
(STFT multi-scale).
|
||||
|
||||
\textbf{Stage 2 -- Diffusion pretraining} (200K steps, text-conditioned): The U-Net is
|
||||
trained on text-audio pairs to learn general audio generation.
|
||||
|
||||
\textbf{Stage 3 -- Video-conditioned fine-tuning} (100K steps): The visual conditioning
|
||||
components are added and trained while the audio backbone is frozen for the first 20K steps.
|
||||
|
||||
\textbf{Stage 4 -- Synchronization fine-tuning} (50K steps): Fine-tuned specifically on
|
||||
video-audio pairs with precise onset alignment, optimizing audio-visual synchrony loss.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lllll}
|
||||
\toprule
|
||||
\textbf{Stage} & \textbf{Steps} & \textbf{Batch} & \textbf{LR} & \textbf{Hardware} \\
|
||||
\midrule
|
||||
VAE pretrain & 100K & 256 & 1e-4 & 8$\times$A100 \\
|
||||
Diffusion pretrain & 200K & 128 & 2e-4 & 32$\times$A100 \\
|
||||
Video fine-tune & 100K & 64 & 5e-5 & 16$\times$A100 \\
|
||||
Sync fine-tune & 50K & 32 & 1e-5 & 8$\times$A100 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen-Foley Training Configuration}
|
||||
\end{table}
|
||||
|
||||
\section{Evaluation}
|
||||
|
||||
\subsection{AudioSet Classification (Sound Recognition)}
|
||||
|
||||
We evaluate the model's ability to recognize and generate sounds from the 527 AudioSet classes.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{mAP} $\uparrow$ & \textbf{Parameters} \\
|
||||
\midrule
|
||||
PANNs (CNN14) & 0.431 & 80M \\
|
||||
AST & 0.448 & 87M \\
|
||||
HTS-AT & 0.471 & 31M \\
|
||||
AudioMAE & 0.489 & 86M \\
|
||||
\textbf{Zen-Foley (generation + recognition)} & \textbf{0.432} & 1.5B \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{AudioSet mAP (recognition, not the primary task of Zen-Foley)}
|
||||
\end{table}
|
||||
|
||||
Note that Zen-Foley is primarily a generative model; AudioSet mAP is reported as a measure of
|
||||
audio understanding capability, not the primary optimization target.
|
||||
|
||||
\subsection{Mean Opinion Score (MOS)}
|
||||
|
||||
Professional audio engineers ($N=15$) rated generated sound effects on a 5-point scale for:
|
||||
naturalness, appropriateness to visual content, and overall quality.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Model / Condition} & \textbf{Naturalness} & \textbf{Appropriateness} & \textbf{Overall MOS} \\
|
||||
\midrule
|
||||
Ground truth Foley & 4.8 & 4.9 & 4.85 \\
|
||||
FoleyCrafter & 3.7 & 3.5 & 3.6 \\
|
||||
Diff-Foley & 3.9 & 3.7 & 3.8 \\
|
||||
V2A-Mapper & 4.0 & 3.9 & 4.0 \\
|
||||
\textbf{Zen-Foley} & \textbf{4.2} & \textbf{4.3} & \textbf{4.2} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Mean Opinion Score Evaluation (1--5 scale, N=15 audio engineers)}
|
||||
\end{table}
|
||||
|
||||
\subsection{AVSync Benchmark (Synchronization)}
|
||||
|
||||
Audio-video synchronization measured by onset alignment error (ms) on 500 held-out clips
|
||||
containing discrete sound-producing events (impacts, clicks, slams).
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{Mean Onset Error (ms)} $\downarrow$ & \textbf{Within 100ms (\%)} \\
|
||||
\midrule
|
||||
Diff-Foley & 182 & 64.3 \\
|
||||
FoleyCrafter & 127 & 78.1 \\
|
||||
TempoFoley & 68 & 89.4 \\
|
||||
\textbf{Zen-Foley} & \textbf{38} & \textbf{96.2} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{AVSync Synchronization Benchmark}
|
||||
\end{table}
|
||||
|
||||
\subsection{Spatial Audio Quality}
|
||||
|
||||
Spatial audio accuracy evaluated by blind listening tests ($N=20$ listeners) with binaural
|
||||
rendering via standard HRTF. Participants localized synthesized sound sources and compared
|
||||
to reference spatial positions.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Metric} & \textbf{Value} & \textbf{Human Reference} \\
|
||||
\midrule
|
||||
Azimuth error (mean $\pm$ std) & $6.2 \pm 3.1$\textdegree & $4.1 \pm 2.3$\textdegree \\
|
||||
Elevation error (mean $\pm$ std) & $9.4 \pm 4.2$\textdegree & $7.3 \pm 3.8$\textdegree \\
|
||||
Distance rank correlation & 0.84 & 0.91 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Spatial Audio Localization Accuracy}
|
||||
\end{table}
|
||||
|
||||
\section{Applications}
|
||||
|
||||
\subsection{Automated Film Post-Production}
|
||||
|
||||
Zen-Foley integrates into the Hanzo AI film pipeline as the Foley layer, automatically
|
||||
generating synchronized sound tracks from director-approved video clips. In production use
|
||||
at a partner studio, Zen-Foley reduced Foley session time by 73\% while requiring minimal
|
||||
human cleanup for non-dialogue sound elements.
|
||||
|
||||
\subsection{Game Audio Engine Integration}
|
||||
|
||||
Game engines can call Zen-Foley via the Hanzo SDK to generate procedural audio for dynamic
|
||||
game events, replacing hand-crafted sound libraries with contextually appropriate generated
|
||||
audio that varies with game state.
|
||||
|
||||
\subsection{Accessibility: Audio Description Enhancement}
|
||||
|
||||
Zen-Foley is used to enhance audio descriptions for visually impaired viewers by generating
|
||||
rich ambient soundscapes that communicate scene context without explicit narration.
|
||||
|
||||
\section{Integration}
|
||||
|
||||
\begin{lstlisting}[language=Python, caption=Zen-Foley Video-Conditioned Generation]
|
||||
from zen import ZenFoley
|
||||
import moviepy.editor as mp
|
||||
|
||||
model = ZenFoley.from_pretrained("zenlm/zen-foley-1.5b")
|
||||
|
||||
# Generate foley for a video clip
|
||||
video = mp.VideoFileClip("scene.mp4")
|
||||
audio = model.generate_foley(
|
||||
video=video,
|
||||
spatial=True, # Enable Ambisonics output
|
||||
style="cinematic",
|
||||
duration=video.duration
|
||||
)
|
||||
|
||||
# Composite with original audio
|
||||
final = video.set_audio(audio)
|
||||
final.write_videofile("scene_with_foley.mp4")
|
||||
\end{lstlisting}
|
||||
|
||||
\section{Related Work}
|
||||
|
||||
SpecVQGAN \cite{iashin2021taming} was among the first to apply VQ-VAE to video-to-audio
|
||||
generation. Diff-Foley \cite{luo2024difffoley} introduced latent diffusion for this task.
|
||||
FoleyCrafter \cite{zhang2024foleycrafter} improved temporal alignment. Zen-Foley advances
|
||||
synchronization accuracy to sub-40ms while adding full spatial audio generation capability
|
||||
and achieving higher MOS scores through large-scale professional Foley data training.
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
Zen-Foley demonstrates that a 1.5B parameter diffusion model, trained on a carefully curated
|
||||
mix of professional Foley sessions and large-scale video-audio data, can generate synchronized,
|
||||
spatially aware sound effects at quality approaching professional Foley artists (4.2/5 MOS
|
||||
vs. 4.85 ground truth). The 38ms synchronization accuracy and Ambisonics spatial audio
|
||||
output make Zen-Foley suitable for direct integration into professional film and game
|
||||
production pipelines.
|
||||
|
||||
\begin{thebibliography}{10}
|
||||
\bibitem{iashin2021taming} V. Iashin and E. Rahtu, ``Taming Visually Guided Sound Generation,'' BMVC, 2021.
|
||||
\bibitem{luo2024difffoley} S. Luo et al., ``Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models,'' NeurIPS, 2024.
|
||||
\bibitem{zhang2024foleycrafter} Y. Zhang et al., ``FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Sounds,'' arXiv:2407.01494, 2024.
|
||||
\end{thebibliography}
|
||||
|
||||
\end{document}
|
||||
Binary file not shown.
@@ -1,316 +0,0 @@
|
||||
\documentclass[11pt,a4paper]{article}
|
||||
\usepackage[utf8]{inputenc}
|
||||
\usepackage[T1]{fontenc}
|
||||
\usepackage{amsmath,amsfonts,amssymb}
|
||||
\usepackage{graphicx}
|
||||
\usepackage{hyperref}
|
||||
\usepackage{listings}
|
||||
\usepackage{color}
|
||||
\usepackage{booktabs}
|
||||
\usepackage{float}
|
||||
\usepackage{geometry}
|
||||
\geometry{margin=1in}
|
||||
\definecolor{zenblue}{RGB}{41,121,255}
|
||||
\hypersetup{colorlinks=true,linkcolor=zenblue,urlcolor=zenblue,citecolor=zenblue}
|
||||
|
||||
\title{\textbf{Zen MoDE: Mixture of Distilled Experts Architecture}\\
|
||||
\large Technical Report v2025.06}
|
||||
\author{Antje Worring, Zach Kelling \\ Zen LM Research Team\\
|
||||
\texttt{research@zenlm.org}}
|
||||
\date{June 2025}
|
||||
|
||||
\begin{document}
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
We present Zen MoDE (Mixture of Distilled Experts), the core architectural innovation underlying the Zen model family. Zen MoDE addresses two fundamental challenges in large language model scaling: computational efficiency via sparse expert routing, and knowledge transfer via distilled expert initialization. We introduce semantic routing, where experts specialize by input domain and linguistic style rather than by position-hashing or random assignment. Combined with distilled expert initialization from smaller dense teacher models, Zen MoDE achieves performance matching a 400B-parameter dense model using only 47B active parameters (480B total). We present the router architecture, expert specialization analysis, load balancing algorithm, and performance benchmarks comparing Zen MoDE against dense equivalents.
|
||||
\end{abstract}
|
||||
|
||||
\section{Introduction}
|
||||
|
||||
Scaling language models requires navigating a fundamental tension: more parameters improve capability, but increase inference cost proportionally. Mixture-of-Experts (MoE) architectures offer an elegant resolution: maintain a large parameter count while activating only a sparse subset per forward pass.
|
||||
|
||||
Conventional MoE implementations route tokens to experts via a learned linear router, trained to maximize routing utility. In practice, this leads to load imbalance (popular experts become overloaded while others are underutilized) and poor expert specialization (experts do not develop interpretable specializations).
|
||||
|
||||
Zen MoDE introduces three innovations that address these limitations:
|
||||
|
||||
\begin{enumerate}
|
||||
\item \textbf{Semantic routing}: Expert selection guided by semantic content embeddings, enabling interpretable domain and style specialization.
|
||||
\item \textbf{Distilled expert initialization}: Each expert is initialized from a smaller dense teacher model specialized on a semantic domain, providing warm-start knowledge.
|
||||
\item \textbf{Adaptive capacity factors}: Dynamic capacity allocation based on real-time expert load, preventing token dropping under skewed distributions.
|
||||
\end{enumerate}
|
||||
|
||||
\section{Architecture}
|
||||
|
||||
\subsection{Overall Structure}
|
||||
|
||||
Zen MoDE follows the standard Transformer architecture with MoE layers replacing every-other dense feed-forward network (FFN) layer:
|
||||
|
||||
\begin{itemize}
|
||||
\item \textbf{Layer structure}: Alternating dense attention + {dense FFN, MoE FFN} pattern.
|
||||
\item \textbf{Expert count}: 128 experts per MoE layer.
|
||||
\item \textbf{Active experts}: Top-$K=8$ experts per token ($K=2$ for the 14B model).
|
||||
\item \textbf{Expert FFN size}: $4d_{\text{model}}$ hidden dimension (same as dense FFN equivalent).
|
||||
\item \textbf{Shared experts}: 4 experts always activated (``shared'' experts, active for every token).
|
||||
\end{itemize}
|
||||
|
||||
The total parameter count $P_{\text{total}}$ and active parameters $P_{\text{active}}$ for an $L$-layer model with $N_E$ experts:
|
||||
|
||||
\begin{align}
|
||||
P_{\text{total}} &= P_{\text{attn}} + P_{\text{shared}} + N_E \cdot P_{\text{expert}} \\
|
||||
P_{\text{active}} &= P_{\text{attn}} + P_{\text{shared}} + K \cdot P_{\text{expert}}
|
||||
\end{align}
|
||||
|
||||
For the 480B Zen MoDE: $P_{\text{total}} = 480B$, $P_{\text{active}} = 47B$ (9.8\% activation ratio).
|
||||
|
||||
\subsection{Model Scale Configurations}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Zen MoDE model configurations}
|
||||
\label{tab:configs}
|
||||
\begin{tabular}{lcccccc}
|
||||
\toprule
|
||||
Model & Total Params & Active Params & Experts & $K$ & Layers & $d_{\text{model}}$ \\
|
||||
\midrule
|
||||
Zen MoDE 14B & 14B & 2.4B & 64 & 2 & 28 & 2048 \\
|
||||
Zen MoDE 72B & 72B & 8.1B & 128 & 4 & 64 & 4096 \\
|
||||
Zen MoDE 236B & 236B & 22B & 128 & 8 & 80 & 7168 \\
|
||||
Zen MoDE 480B & 480B & 47B & 128 & 8 & 94 & 7680 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\section{Semantic Routing}
|
||||
|
||||
\subsection{Router Architecture}
|
||||
|
||||
Conventional MoE routers apply a linear projection to the token representation and take the top-$K$ softmax scores:
|
||||
|
||||
\begin{equation}
|
||||
\text{router}_{\text{standard}}(x) = \text{TopK}(\text{softmax}(\mathbf{W}_r x), K)
|
||||
\end{equation}
|
||||
|
||||
The Zen MoDE semantic router adds a two-stage process:
|
||||
|
||||
\paragraph{Stage 1: Semantic Embedding.} A lightweight embedding model $f_s$ maps the input sequence prefix to a 256-dimensional semantic vector:
|
||||
\begin{equation}
|
||||
\mathbf{s} = f_s(\text{prefix}) \in \mathbb{R}^{256}
|
||||
\end{equation}
|
||||
|
||||
This semantic vector captures topic domain, linguistic register, and task type at the sequence level.
|
||||
|
||||
\paragraph{Stage 2: Expert Scoring.} Token-level routing is biased toward experts whose learned prototype $\mathbf{p}_e$ is nearest to the semantic vector:
|
||||
|
||||
\begin{equation}
|
||||
\text{score}(x, e) = \underbrace{\mathbf{w}_e^\top x}_{\text{token routing}} + \underbrace{\beta \cdot \text{sim}(\mathbf{s}, \mathbf{p}_e)}_{\text{semantic bias}}
|
||||
\end{equation}
|
||||
|
||||
where $\beta = 0.3$ balances local (token) and global (semantic) routing signals.
|
||||
|
||||
\subsection{Expert Specialization}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Expert specialization analysis: top domains per expert cluster (128 experts, 16 clusters)}
|
||||
\label{tab:specialization}
|
||||
\begin{tabular}{lll}
|
||||
\toprule
|
||||
Cluster & Primary Domain & Secondary Domains \\
|
||||
\midrule
|
||||
1--8 & Code (Python, JS, Rust) & Algorithms, data structures \\
|
||||
9--16 & Mathematics (formal) & Proofs, theorems \\
|
||||
17--24 & Scientific writing & Academic, citations \\
|
||||
25--32 & Medical/clinical & Pharmacology, anatomy \\
|
||||
33--40 & Legal text & Contracts, statutes \\
|
||||
41--48 & Financial text & Earnings, risk, regulation \\
|
||||
49--56 & Creative writing & Fiction, poetry \\
|
||||
57--64 & Conversational & Dialogue, casual register \\
|
||||
65--72 & News and journalism & Current events, AP style \\
|
||||
73--80 & Technical documentation & Manuals, API docs \\
|
||||
81--88 & Multilingual (CJK) & Japanese, Chinese, Korean \\
|
||||
89--96 & Multilingual (Arabic) & Arabic, Farsi, Hebrew \\
|
||||
97--104 & Multilingual (Slavic) & Russian, Polish, Czech \\
|
||||
105--112 & Low-resource languages & African, SE Asian \\
|
||||
113--120 & Instruction following & Commands, structured output \\
|
||||
121--128 & Reasoning (chain-of-thought) & Logic, planning \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
Expert specialization is measured by the Jensen-Shannon divergence between the token distribution of expert $e$'s input and the corpus-wide distribution. Higher divergence indicates stronger specialization. Zen MoDE experts achieve average JS-divergence of 0.42 versus 0.18 for standard top-K routing (2.3$\times$ improvement).
|
||||
|
||||
\section{Distilled Expert Initialization}
|
||||
|
||||
\subsection{Teacher Model Preparation}
|
||||
|
||||
For each semantic domain cluster $c$, we prepare a specialist dense teacher model:
|
||||
\begin{enumerate}
|
||||
\item Start from a 7B dense model pre-trained on the full corpus.
|
||||
\item Fine-tune on domain-specific data (e.g., code corpus for code experts, PubMed for medical experts).
|
||||
\item Extract the FFN weight matrices $\{W_1^c, W_2^c\}$ for each layer as expert initialization.
|
||||
\end{enumerate}
|
||||
|
||||
\subsection{Expert Initialization Protocol}
|
||||
|
||||
Expert $e$ in cluster $c$ is initialized by projecting the teacher FFN weights to expert dimensions:
|
||||
|
||||
\begin{equation}
|
||||
W_{e,\text{init}} = \mathbf{P}_e \cdot W^c_{\text{teacher}} \cdot \mathbf{Q}_e
|
||||
\end{equation}
|
||||
|
||||
where $\mathbf{P}_e, \mathbf{Q}_e$ are random orthogonal projections that map from teacher dimensions to expert dimensions. This preserves the spectral properties of the teacher weights in a lower-dimensional subspace.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Distilled initialization vs. random initialization (perplexity on held-out eval, lower is better)}
|
||||
\label{tab:distillation}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
Initialization & Step 1K & Step 10K & Step 100K \\
|
||||
\midrule
|
||||
Random & 48.2 & 18.4 & 8.4 \\
|
||||
Copy from dense base & 22.4 & 12.8 & 7.8 \\
|
||||
Distilled (ours) & \textbf{16.4} & \textbf{10.2} & \textbf{7.2} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
Distilled initialization provides significant early-training advantage, converging to final loss in 31\% fewer training steps.
|
||||
|
||||
\section{Load Balancing}
|
||||
|
||||
\subsection{Expert Load Imbalance Problem}
|
||||
|
||||
Without load balancing, popular experts become overloaded while others are underutilized. Overloaded experts either drop tokens (capacity-limited) or become bottlenecks (unlimited capacity). Both outcomes degrade performance.
|
||||
|
||||
\subsection{Auxiliary Load Balancing Loss}
|
||||
|
||||
Zen MoDE minimizes an auxiliary load balancing loss alongside the main language modeling objective:
|
||||
|
||||
\begin{equation}
|
||||
\mathcal{L}_{\text{balance}} = \alpha \cdot N \sum_{e=1}^{N_E} f_e \cdot P_e
|
||||
\end{equation}
|
||||
|
||||
where $f_e = \frac{1}{T} \sum_t \mathbf{1}[e \in \text{TopK}(t)]$ is the fraction of tokens routed to expert $e$, $P_e = \frac{1}{T} \sum_t p_t^e$ is the average routing probability for expert $e$, and $\alpha = 0.01$ is the load balancing coefficient.
|
||||
|
||||
\subsection{Adaptive Capacity Factors}
|
||||
|
||||
The capacity factor $C$ limits the maximum tokens an expert can process per batch:
|
||||
\begin{equation}
|
||||
\text{capacity}_e = C \cdot \frac{\text{batch size} \cdot \text{seq len}}{N_E}
|
||||
\end{equation}
|
||||
|
||||
Zen MoDE uses adaptive capacity factors that adjust per-expert based on rolling load statistics:
|
||||
\begin{equation}
|
||||
C_e(t) = C_{\text{base}} + \gamma \cdot \left(\bar{f}_e - \frac{1}{N_E}\right)
|
||||
\end{equation}
|
||||
|
||||
Under-utilized experts ($\bar{f}_e < 1/N_E$) get reduced capacity, over-utilized experts get increased capacity, preventing token dropping under natural load distribution skew.
|
||||
|
||||
\section{Benchmark Results}
|
||||
|
||||
\subsection{Performance vs. Dense Equivalent}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Zen MoDE vs. dense equivalent: MMLU 5-shot (\%)}
|
||||
\label{tab:perf_dense}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
Model & Params & Active & MMLU \\
|
||||
\midrule
|
||||
Dense 7B & 7B & 7B & 63.4 \\
|
||||
Zen MoDE 14B & 14B & 2.4B & 72.8 \\
|
||||
Dense 13B & 13B & 13B & 67.2 \\
|
||||
Dense 70B & 70B & 70B & 80.2 \\
|
||||
Zen MoDE 72B & 72B & 8.1B & 84.8 \\
|
||||
Dense 400B (estimated) & 400B & 400B & 90.4 \\
|
||||
Zen MoDE 480B & 480B & 47B & 91.2 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
Zen MoDE 480B matches estimated dense-400B performance with only 47B active parameters—an 8.5$\times$ reduction in inference FLOPs.
|
||||
|
||||
\subsection{Expert Utilization}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Expert utilization metrics (Zen MoDE 72B)}
|
||||
\label{tab:utilization}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
Metric & Standard Top-K & Zen MoDE \\
|
||||
\midrule
|
||||
Utilization entropy & 5.42 bits & 6.81 bits (max 7 bits) \\
|
||||
CV (coefficient of variation) & 0.84 & 0.31 \\
|
||||
Token drop rate & 3.2\% & 0.8\% \\
|
||||
Expert utilization below 50\% & 24 experts & 8 experts \\
|
||||
Expert utilization above 200\% & 18 experts & 3 experts \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Routing Quality}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Routing quality metrics}
|
||||
\label{tab:routing}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
Metric & Standard Top-K & Semantic Routing \\
|
||||
\midrule
|
||||
Domain consistency (same-domain token routing) & 61.4\% & 84.2\% \\
|
||||
Expert JS-divergence & 0.18 & 0.42 \\
|
||||
Router confidence (avg top score) & 0.48 & 0.62 \\
|
||||
Cross-domain routing (noise) & 38.6\% & 15.8\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\section{Training Efficiency}
|
||||
|
||||
\subsection{Compute Budget}
|
||||
|
||||
The MoDE architecture provides significant training compute savings over dense models at equivalent capacity:
|
||||
|
||||
\begin{equation}
|
||||
\frac{\text{FLOPs}_{\text{MoDE}}}{\text{FLOPs}_{\text{dense-equivalent}}} = \frac{K + N_{\text{shared}}}{N_E} \approx \frac{12}{128} \approx 9.4\%
|
||||
\end{equation}
|
||||
|
||||
In practice, attention layers (non-sparse) and overhead reduce the realized savings to approximately 8$\times$ reduction in training FLOPs for the same total parameters.
|
||||
|
||||
\subsection{Communication Overhead (Distributed Training)}
|
||||
|
||||
Expert parallelism requires all-to-all communication to route tokens to the correct expert device. Communication overhead as a fraction of total training time:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Expert parallelism communication overhead}
|
||||
\label{tab:comm}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
Model & Expert Parallelism & Comm / Total & Effective Throughput \\
|
||||
\midrule
|
||||
72B & 8 nodes & 8.4\% & 91.6\% of compute \\
|
||||
236B & 16 nodes & 11.2\% & 88.8\% of compute \\
|
||||
480B & 32 nodes & 14.8\% & 85.2\% of compute \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
Zen MoDE demonstrates that principled mixture-of-experts design—semantic routing, distilled expert initialization, and adaptive load balancing—substantially improves over naive sparse routing approaches. The 480B Zen MoDE model matches estimated dense-400B performance at 8.5$\times$ lower inference FLOPs, and semantic routing increases expert specialization by 2.3$\times$ as measured by domain-routing consistency. Distilled expert initialization accelerates convergence by 31\%, enabling efficient training of very large expert counts.
|
||||
|
||||
\begin{thebibliography}{99}
|
||||
\bibitem{moe_original} Shazeer, N. et al. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. \textit{ICLR}, 2017.
|
||||
\bibitem{switch} Fedus, W. et al. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. \textit{JMLR}, 2022.
|
||||
\bibitem{gshard} Lepikhin, D. et al. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. \textit{ICLR}, 2021.
|
||||
\bibitem{mixtral} Jiang, A. et al. Mixtral of Experts. \textit{arXiv:2401.04088}, 2024.
|
||||
\bibitem{expert_specialization} Zuo, S. et al. Taming Sparsely Activated Transformer with Stochastic Experts. \textit{ICLR}, 2022.
|
||||
\end{thebibliography}
|
||||
|
||||
\end{document}
|
||||
Binary file not shown.
+210
-268
@@ -33,7 +33,7 @@
|
||||
\vspace{0.5cm}
|
||||
\Huge \textbf{Zen-Musician} \\
|
||||
\vspace{0.3cm}
|
||||
\large AI Music Composition, MIDI Generation, and Audio Synthesis \\
|
||||
\large Packaging YuE for Long-Form Lyrics-to-Song Generation in the Zen Stack \\
|
||||
\vspace{0.5cm}
|
||||
\normalsize Technical Report v2025.01
|
||||
}
|
||||
@@ -50,18 +50,28 @@
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
We present \textbf{Zen-Musician}, a 3-billion parameter generative music model capable of
|
||||
multi-genre composition, MIDI generation, audio synthesis, and real-time accompaniment.
|
||||
Zen-Musician unifies symbolic music representation (MIDI) and acoustic audio generation
|
||||
within a single model, enabling coherent compositions that span from high-level structural
|
||||
planning (chord progressions, song sections, orchestration) to low-level acoustic rendering
|
||||
(instrument timbre, expression, dynamics). Trained on 180,000 hours of licensed music spanning
|
||||
40 genres, the model achieves a Frechet Audio Distance (FAD) score of 2.3 (compared to 1.8
|
||||
for human composers in identical conditions), earns a Music Opinion Score (MOS) of 4.1/5.0,
|
||||
and generates 30-second compositions in under 4 seconds on an A10G GPU. A novel
|
||||
\textbf{Section-Aware Transformer (SAT)} architecture enforces long-range musical structure
|
||||
(verse/chorus/bridge coherence) that pure autoregressive models lack. Zen-Musician supports
|
||||
real-time accompaniment generation, adaptive score following, and style transfer across genres.
|
||||
\textbf{Zen-Musician} is a packaging and integration of \textbf{YuE}, the open
|
||||
long-form music generation foundation model developed by \textbf{M-A-P
|
||||
(Multimodal Art Projection)} and HKUST and released under the permissive
|
||||
\textbf{Apache-2.0} license \cite{yuan2025yue,yuehf}. We make no claim of a
|
||||
from-scratch model: Zen-Musician redistributes YuE's published weights
|
||||
unmodified and exposes them through the Zen tooling, runtime, and API surface.
|
||||
This report documents the \emph{real} architecture so that integrators understand
|
||||
what they are deploying. YuE is a lyrics-to-song system: given lyrics together
|
||||
with a short genre/style description, it generates full songs containing both
|
||||
sung vocals and accompaniment. It is built on a decoder-only LLaMA-2-style
|
||||
autoregressive transformer that operates over discrete audio tokens. The flagship
|
||||
Stage-1 checkpoint (e.g. \texttt{m-a-p/YuE-s1-7B-anneal-en-cot}) has
|
||||
approximately 7B parameters and performs track-decoupled next-token prediction to
|
||||
produce vocal and accompaniment token streams; a separate Stage-2 model
|
||||
(approximately 1B parameters) refines and upsamples those tokens toward the final
|
||||
waveform \cite{yuegh}. According to its authors, YuE can generate up to roughly
|
||||
five minutes of music while maintaining lyrical alignment, coherent musical
|
||||
structure, and vocal melodies with appropriate accompaniment, and supports style
|
||||
transfer and reference conditioning through an in-context-learning (ICL) variant
|
||||
\cite{yuan2025yue}. This document supersedes an earlier draft that incorrectly
|
||||
described a from-scratch ``Zen MoDE'' architecture and reported fabricated
|
||||
benchmarks; those claims were false and have been removed.
|
||||
\end{abstract}
|
||||
|
||||
\tableofcontents
|
||||
@@ -69,23 +79,51 @@ real-time accompaniment generation, adaptive score following, and style transfer
|
||||
|
||||
\section{Introduction}
|
||||
|
||||
Music generation has seen rapid progress, yet existing models suffer from two persistent failure
|
||||
modes. \textbf{Structural incoherence}: autoregressive models generate locally plausible music
|
||||
that lacks the large-scale structure (repetition, contrast, development) that defines
|
||||
professional composition. \textbf{Modality gap}: separate symbolic and acoustic models require
|
||||
clumsy pipelines (MIDI $\to$ synthesis) that break acoustic coherence.
|
||||
\subsection{What Zen-Musician Is (and Is Not)}
|
||||
|
||||
Zen-Musician resolves both through:
|
||||
Zen-Musician is the Zen stack's integration of \textbf{YuE}
|
||||
(\href{https://github.com/multimodal-art-projection/YuE}{multimodal-art-projection/YuE}),
|
||||
an open foundation model for full-song generation released by M-A-P and HKUST
|
||||
under the \textbf{Apache-2.0} license \cite{yuehf,yuan2025yue}. The Apache-2.0
|
||||
license permits redistribution, modification, and commercial use subject to its
|
||||
attribution and notice requirements; YuE's authors explicitly encourage creators
|
||||
to incorporate model outputs, with attribution to ``YuE by HKUST/M-A-P''
|
||||
recommended \cite{yuehf}.
|
||||
|
||||
\begin{enumerate}
|
||||
\item \textbf{Section-Aware Transformer (SAT)}: A hierarchical attention architecture that
|
||||
maintains a global structural plan (verse, chorus, bridge, solo) and conditions local
|
||||
generation on the current section's role in the overall form.
|
||||
\item \textbf{Unified token space}: MIDI events and audio codec tokens share a single
|
||||
vocabulary, enabling the model to interleave symbolic and acoustic generation.
|
||||
\item \textbf{Multi-genre training}: 180,000 hours of licensed music spanning 40 genres
|
||||
with rich metadata (key, tempo, mood, instrumentation).
|
||||
\end{enumerate}
|
||||
To be explicit about provenance:
|
||||
|
||||
\begin{itemize}
|
||||
\item Zen-Musician \textbf{does not} introduce a new architecture. There is
|
||||
no ``Zen MoDE,'' no ``Section-Aware Transformer,'' and no unified
|
||||
MIDI/audio model. Those concepts appeared in a prior draft of this report and
|
||||
were not real.
|
||||
\item Zen-Musician \textbf{does not} report its own benchmark numbers. The FAD,
|
||||
MOS, genre-consistency, structure-F1, and accompaniment-latency tables in the
|
||||
earlier draft were fabricated and have been deleted. Where this report
|
||||
mentions evaluation, it attributes claims to YuE's authors and does not invent
|
||||
metrics.
|
||||
\item Zen-Musician \textbf{is} a redistribution of YuE's published weights,
|
||||
wrapped with Zen tooling (a consistent loader, runtime integration, and the
|
||||
Zen API surface) so that YuE can be used alongside other Zen models.
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Correcting the README}
|
||||
|
||||
A previous Zen-Musician description listed the model as ``1B'' parameters and as a
|
||||
MIDI/symbolic system. Both were wrong. The packaged flagship is the
|
||||
\textbf{7B Stage-1} checkpoint; the ``1B'' figure corresponds only to the
|
||||
\emph{Stage-2} refinement/upsampling model, not the main generator. YuE generates
|
||||
\emph{audio} (sung vocals plus accompaniment) from \emph{lyrics and genre text}.
|
||||
It is not a MIDI model and does not generate symbolic scores.
|
||||
|
||||
\subsection{The Problem YuE Addresses}
|
||||
|
||||
Open music models have historically struggled with \emph{long-form} generation:
|
||||
producing a coherent multi-minute song with intelligible, aligned lyrics, a
|
||||
sensible verse/chorus structure, and a singing voice that stays on top of a
|
||||
suitable accompaniment. YuE targets this lyrics-to-song setting directly, scaling
|
||||
an autoregressive transformer over audio tokens to song-length outputs
|
||||
\cite{yuan2025yue}.
|
||||
|
||||
\subsection{Model Overview}
|
||||
|
||||
@@ -95,286 +133,190 @@ Zen-Musician resolves both through:
|
||||
\toprule
|
||||
\textbf{Property} & \textbf{Value} \\
|
||||
\midrule
|
||||
Parameters & 3B \\
|
||||
Architecture & Section-Aware Transformer (SAT) \\
|
||||
Token Vocabulary & 32,768 (MIDI events + Encodec audio tokens) \\
|
||||
Context Length & 30K tokens ($\approx$2 min of music) \\
|
||||
Genres & 40 (classical, jazz, pop, EDM, folk, metal, + 34 more) \\
|
||||
Max Generation Length & 10 minutes continuous \\
|
||||
Training Data & 180,000 hours licensed music \\
|
||||
Upstream model & YuE (M-A-P / HKUST) \\
|
||||
License & Apache-2.0 \cite{yuehf} \\
|
||||
Task & Lyrics + genre $\rightarrow$ full song (vocals + accompaniment) \\
|
||||
Backbone & Decoder-only, LLaMA-2-style autoregressive transformer \cite{yuan2025yue,touvron2023llama2} \\
|
||||
Stage-1 generator & $\approx$7B parameters (e.g. \texttt{YuE-s1-7B-anneal-en-cot}) \\
|
||||
Stage-2 refiner & $\approx$1B parameters (token refinement / upsampling) \\
|
||||
Representation & Discrete audio tokens (xcodec-style tokenizer) \\
|
||||
Languages & English, Mandarin, Cantonese, Japanese, Korean (checkpoint-specific) \\
|
||||
Conditioning & Lyrics with section tags; genre/style tags; optional audio reference (ICL) \\
|
||||
Reported length & Up to $\approx$5 minutes of music \cite{yuan2025yue} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen-Musician Model Specifications}
|
||||
\caption{Zen-Musician = packaged YuE: upstream specifications}
|
||||
\end{table}
|
||||
|
||||
\section{Architecture}
|
||||
\section{Architecture (as published by YuE's authors)}
|
||||
|
||||
\subsection{Unified Token Space}
|
||||
The description below summarizes YuE's design from its paper and model
|
||||
documentation \cite{yuan2025yue,yuehf,yuegh}. Zen-Musician does not alter this
|
||||
architecture.
|
||||
|
||||
Zen-Musician's key architectural decision is a unified vocabulary covering both symbolic music
|
||||
and acoustic audio:
|
||||
\subsection{Audio Tokenization}
|
||||
|
||||
\textbf{MIDI tokens} (4,096 tokens): Representing note-on, note-off, velocity, program change,
|
||||
tempo, and time shift events at 5ms resolution.
|
||||
YuE operates over \emph{discrete audio tokens} rather than raw waveforms. Audio is
|
||||
encoded into token sequences by an xcodec-style neural audio codec; the language
|
||||
model predicts tokens, and the decoder side reconstructs audio from them
|
||||
\cite{yuehf}. This lets a standard autoregressive transformer model music the same
|
||||
way a text LLM models language.
|
||||
|
||||
\textbf{Audio tokens} (28,672 tokens): Encodec \cite{defossez2022encodec} codes at 24kHz with
|
||||
4 codebooks at 75Hz frame rate, providing compact acoustic representation.
|
||||
\subsection{Stage-1: LLaMA-2-style Autoregressive Generator}
|
||||
|
||||
\textbf{Structural tokens} (128 tokens): Section markers (VERSE, CHORUS, BRIDGE, INTRO,
|
||||
OUTRO, SOLO, BREAKDOWN) and metadata (KEY, TEMPO, GENRE, INSTRUMENT).
|
||||
The Stage-1 model is a \textbf{decoder-only, LLaMA-2-style transformer}
|
||||
\cite{yuan2025yue,touvron2023llama2} with approximately \textbf{7B parameters}.
|
||||
Its central design choice, as described by the authors, is \textbf{track-decoupled
|
||||
(``dual-track'') next-token prediction}: rather than modeling a single dense
|
||||
mixture signal, it predicts \emph{vocal} and \emph{accompaniment} token streams in
|
||||
a decoupled fashion, which the authors report helps the model keep the singing
|
||||
voice coherent over long contexts \cite{yuan2025yue}. The authors further describe
|
||||
\emph{structural progressive conditioning} to maintain lyric alignment across
|
||||
long-form generation \cite{yuan2025yue}.
|
||||
|
||||
The model learns to transition fluidly between MIDI and audio modalities, using MIDI for
|
||||
structural planning and audio tokens for acoustic detail within each section.
|
||||
\subsection{Stage-2: Refinement / Upsampling}
|
||||
|
||||
\subsection{Section-Aware Transformer}
|
||||
A separate \textbf{Stage-2} model (approximately \textbf{1B parameters}) takes the
|
||||
Stage-1 token output and performs residual refinement / upsampling toward the
|
||||
final audio, improving fidelity before waveform reconstruction \cite{yuegh,yuehf}.
|
||||
This is the source of the ``1B'' figure that an earlier Zen description mistook for
|
||||
the size of the whole system.
|
||||
|
||||
The SAT adds a \textbf{structural level} atop the token-level transformer:
|
||||
\subsection{Conditioning Interface}
|
||||
|
||||
YuE is prompted with three kinds of input \cite{yuehf}:
|
||||
|
||||
\begin{itemize}
|
||||
\item A \textbf{Structure Encoder} (4 transformer layers) processes a sequence of section
|
||||
descriptors $(s_1, s_2, \ldots, s_K)$ representing the planned song form.
|
||||
\item A \textbf{Section Embedding} is computed for each section and injected into token-level
|
||||
generation via cross-attention at every fourth transformer layer.
|
||||
\item During generation, the model can dynamically revise the structural plan based on
|
||||
what has been generated, enabling coherent improvisation.
|
||||
\item \textbf{Genre / style tags}: a short free-text description (the project
|
||||
suggests including genre, instrumentation, mood, vocal gender, and timbre).
|
||||
\item \textbf{Lyrics}: the song's lyrics, annotated with structural labels such
|
||||
as \texttt{[verse]} and \texttt{[chorus]}, organized into sessions/segments.
|
||||
\item \textbf{Optional audio reference} (ICL variants): a reference clip used
|
||||
for style transfer or voice-style conditioning via in-context learning
|
||||
\cite{yuan2025yue}.
|
||||
\end{itemize}
|
||||
|
||||
Formally, at each generation step $t$ within section $k$:
|
||||
\subsection{Checkpoint Family}
|
||||
|
||||
\begin{equation}
|
||||
h_t = \text{TransformerLayer}(h_{t-1}, \text{CrossAttn}(h_{t-1}, E_k))
|
||||
\end{equation}
|
||||
YuE ships multiple Stage-1 checkpoints, differentiated by language coverage and by
|
||||
mode --- a chain-of-thought (\texttt{cot}) variant and an in-context-learning
|
||||
(\texttt{icl}) variant for reference-conditioned generation \cite{yuehf}. Example
|
||||
flagship checkpoint: \texttt{m-a-p/YuE-s1-7B-anneal-en-cot}.
|
||||
|
||||
where $E_k = \text{StructureEncoder}(s_k)$ is the section embedding.
|
||||
\section{Capabilities Reported Upstream}
|
||||
|
||||
\subsection{Accompaniment Mode}
|
||||
The following are claims made by YuE's authors \cite{yuan2025yue}; Zen-Musician
|
||||
inherits them by redistributing the same weights and does not independently
|
||||
re-benchmark them in this report.
|
||||
|
||||
In accompaniment mode, the model conditions on an input melody (provided as MIDI or audio)
|
||||
and generates harmonically and rhythmically coherent accompaniment. The melody is encoded
|
||||
via a lightweight 2-layer transformer and injected as an additional conditioning signal
|
||||
alongside the section embedding. Accompaniment latency is 120ms (suitable for real-time
|
||||
performance with modest lookahead).
|
||||
\begin{itemize}
|
||||
\item \textbf{Long-form songs}: generation of up to roughly five minutes of
|
||||
music with maintained lyrical alignment and coherent musical structure.
|
||||
\item \textbf{Vocals with accompaniment}: engaging vocal melodies sung over an
|
||||
appropriate instrumental backing, produced by the dual-track scheme.
|
||||
\item \textbf{Style transfer / reference conditioning}: an in-context-learning
|
||||
mode enabling style transfer (e.g. genre conversion while preserving
|
||||
accompaniment) and reference-based generation.
|
||||
\item \textbf{Multilingual coverage}: checkpoints for English, Mandarin,
|
||||
Cantonese, Japanese, and Korean.
|
||||
\end{itemize}
|
||||
|
||||
\section{Training}
|
||||
The YuE paper reports that the system ``matches or even surpasses some of the
|
||||
proprietary systems in musicality and vocal agility,'' and evaluates music
|
||||
\emph{understanding} on the MARBLE benchmark \cite{yuan2025yue}. We reproduce this
|
||||
as an attributed upstream claim only; no numeric scores are asserted here, and
|
||||
none should be attributed to Zen.
|
||||
|
||||
\subsection{Dataset}
|
||||
\section{Integration in the Zen Stack}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lrrl}
|
||||
\toprule
|
||||
\textbf{Source} & \textbf{Hours} & \textbf{Proportion} & \textbf{Licensing} \\
|
||||
\midrule
|
||||
Licensed studio recordings & 80,000 & 44.4\% & Commercial license \\
|
||||
Classical MIDI corpus & 12,000 & 6.7\% & Public domain \\
|
||||
Jazz Real Book corpus & 8,000 & 4.4\% & Licensed \\
|
||||
Folk music archives & 15,000 & 8.3\% & Creative Commons \\
|
||||
Electronic music library & 25,000 & 13.9\% & Commercial license \\
|
||||
Synthetic augmentation & 40,000 & 22.2\% & Generated \\
|
||||
\midrule
|
||||
\textbf{Total} & \textbf{180,000} & 100\% & \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen-Musician Training Data}
|
||||
\end{table}
|
||||
Zen-Musician's contribution is purely \textbf{packaging and integration}, not
|
||||
modeling:
|
||||
|
||||
\subsection{Training Protocol}
|
||||
\begin{itemize}
|
||||
\item \textbf{Distribution}: the YuE Stage-1 and Stage-2 weights are
|
||||
redistributed under Apache-2.0 with upstream attribution and license/notice
|
||||
files preserved.
|
||||
\item \textbf{Loader and runtime}: a consistent entry point so YuE can be
|
||||
invoked the same way as other Zen models.
|
||||
\item \textbf{API surface}: a thin wrapper exposing YuE's lyrics+genre
|
||||
conditioning and two-stage pipeline.
|
||||
\end{itemize}
|
||||
|
||||
\textbf{Stage 1 -- Audio codec training} (50K steps): Encodec-based audio codec is trained
|
||||
on the full audio corpus. Codebook utilization target $>$95\%.
|
||||
\subsection{Resource Notes}
|
||||
|
||||
\textbf{Stage 2 -- Symbolic pretraining} (100K steps): Token-level transformer is pretrained
|
||||
on MIDI token sequences with masked token prediction.
|
||||
YuE is a 7B-class generator over long audio-token contexts; full-song generation
|
||||
is GPU-memory intensive, and the upstream project documents substantial VRAM
|
||||
requirements for long outputs as well as multi-stage (Stage-1 then Stage-2)
|
||||
inference \cite{yuehf}. Integrators should budget accordingly rather than expecting
|
||||
real-time, low-memory operation.
|
||||
|
||||
\textbf{Stage 3 -- Joint pretraining} (150K steps): Training on the full unified token space
|
||||
(MIDI + audio) with next-token prediction.
|
||||
\section{Usage}
|
||||
|
||||
\textbf{Stage 4 -- Structural fine-tuning} (50K steps): SAT structural components are trained
|
||||
with section-boundary supervision from annotated music datasets.
|
||||
The snippet below is illustrative of the lyrics-to-song interface. Refer to the
|
||||
upstream YuE repository \cite{yuegh} for the authoritative inference pipeline,
|
||||
prompt format, and Stage-1/Stage-2 invocation.
|
||||
|
||||
\textbf{Stage 5 -- RLHF from musicians} (20K steps): Reward model trained on 30,000 pairwise
|
||||
comparisons from professional musicians, optimized via PPO.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lllll}
|
||||
\toprule
|
||||
\textbf{Stage} & \textbf{Steps} & \textbf{Batch} & \textbf{LR} & \textbf{Hardware} \\
|
||||
\midrule
|
||||
Codec & 50K & 256 & 1e-4 & 8$\times$A100 \\
|
||||
Symbolic & 100K & 128 & 3e-4 & 16$\times$A100 \\
|
||||
Joint & 150K & 64 & 2e-4 & 32$\times$A100 \\
|
||||
Structural & 50K & 64 & 5e-5 & 16$\times$A100 \\
|
||||
RLHF & 20K & 32 & 2e-5 & 8$\times$A100 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen-Musician Training Configuration}
|
||||
\end{table}
|
||||
|
||||
\section{Evaluation}
|
||||
|
||||
\subsection{Frechet Audio Distance (FAD)}
|
||||
|
||||
FAD measures distribution distance between generated and real music in VGGish embedding space.
|
||||
Lower is better; human composers in identical prompt conditions achieve 1.8.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{FAD} $\downarrow$ & \textbf{Parameters} \\
|
||||
\midrule
|
||||
MusicLM & 4.0 & N/A (proprietary) \\
|
||||
AudioCraft / MusicGen-Large & 3.8 & 3.3B \\
|
||||
Stable Audio Open & 4.4 & 1.1B \\
|
||||
JEN-1 & 2.7 & 0.3B \\
|
||||
Human composers (reference) & 1.8 & -- \\
|
||||
\textbf{Zen-Musician} & \textbf{2.3} & 3B \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Frechet Audio Distance on MusicCaps evaluation set}
|
||||
\end{table}
|
||||
|
||||
\subsection{Music Opinion Score (MOS)}
|
||||
|
||||
Twenty professional musicians rated 200 generated compositions on five criteria.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lc}
|
||||
\toprule
|
||||
\textbf{Criterion} & \textbf{Mean Score (1--5)} \\
|
||||
\midrule
|
||||
Harmonic coherence & 4.2 \\
|
||||
Rhythmic consistency & 4.4 \\
|
||||
Structural form (verse/chorus logic) & 3.9 \\
|
||||
Genre authenticity & 4.1 \\
|
||||
Emotional expressiveness & 4.0 \\
|
||||
\midrule
|
||||
\textbf{Overall MOS} & \textbf{4.1} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Music Opinion Score (N=20 professional musicians)}
|
||||
\end{table}
|
||||
|
||||
\subsection{Genre Consistency}
|
||||
|
||||
We evaluate genre consistency using a fine-tuned genre classifier on 500 generated samples
|
||||
per genre, measuring what fraction are correctly classified:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Genre} & \textbf{Zen-Musician} & \textbf{MusicGen-Large} \\
|
||||
\midrule
|
||||
Classical & 91.4\% & 83.2\% \\
|
||||
Jazz & 87.2\% & 74.1\% \\
|
||||
Pop & 93.8\% & 88.6\% \\
|
||||
Electronic/EDM & 95.1\% & 91.3\% \\
|
||||
Folk & 82.3\% & 69.4\% \\
|
||||
Metal & 88.7\% & 72.8\% \\
|
||||
\textbf{Average} & \textbf{89.8\%} & \textbf{79.9\%} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Genre Classification Accuracy of Generated Music}
|
||||
\end{table}
|
||||
|
||||
\subsection{Structural Analysis}
|
||||
|
||||
To evaluate large-scale structure, we measure the presence and proper placement of structural
|
||||
sections (intro, verse, chorus, bridge, outro) in generated 3-minute compositions, as
|
||||
detected by a structure segmentation model:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{Section F1} $\uparrow$ & \textbf{Repetition Consistency} \\
|
||||
\midrule
|
||||
MusicGen-Large & 0.52 & 61.3\% \\
|
||||
AudioCraft XL & 0.58 & 67.4\% \\
|
||||
\textbf{Zen-Musician (SAT)} & \textbf{0.79} & \textbf{84.2\%} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Musical Structure Quality Metrics}
|
||||
\end{table}
|
||||
|
||||
\subsection{Accompaniment Quality}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Metric} & \textbf{Value} \\
|
||||
\midrule
|
||||
Harmonic correctness (% in-key) & 94.7\% \\
|
||||
Beat alignment error (ms) & 22.3 \\
|
||||
Musician preference vs. random & 91.4\% \\
|
||||
Real-time latency & 120ms \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Accompaniment Mode Evaluation}
|
||||
\end{table}
|
||||
|
||||
\section{Applications}
|
||||
|
||||
\subsection{Content Creation}
|
||||
|
||||
Zen-Musician is integrated into Hanzo's content creation platform, enabling video creators to
|
||||
generate custom background music from natural language descriptions (``upbeat jazz, 90 seconds,
|
||||
suitable for a cooking tutorial''). Time-to-music is under 8 seconds for 90-second clips.
|
||||
|
||||
\subsection{Real-Time Performance}
|
||||
|
||||
Musicians use Zen-Musician in live performance via the Zen-Musician VST plugin, which provides
|
||||
real-time accompaniment that adapts to the performer's tempo, key, and dynamics.
|
||||
|
||||
\subsection{Film Scoring}
|
||||
|
||||
The model generates adaptive film scores conditioned on video (via integration with Zen-Foley's
|
||||
video encoder) that automatically adjust mood and intensity to match scene content.
|
||||
|
||||
\section{Integration}
|
||||
|
||||
\begin{lstlisting}[language=Python, caption=Zen-Musician Composition Generation]
|
||||
\begin{lstlisting}[language=Python, caption=Zen-Musician (packaged YuE) lyrics-to-song generation]
|
||||
from zen import ZenMusician
|
||||
|
||||
model = ZenMusician.from_pretrained("zenlm/zen-musician-3b")
|
||||
# Loads the packaged YuE Stage-1 (7B) generator; Stage-2 refinement
|
||||
# runs as part of the pipeline.
|
||||
model = ZenMusician.from_pretrained("zenlm/zen-musician") # wraps m-a-p/YuE-s1-7B-*
|
||||
|
||||
# Text-to-music generation
|
||||
audio = model.compose(
|
||||
prompt="An introspective jazz ballad in F minor, "
|
||||
"featuring solo piano with brushed drums, "
|
||||
"verse-chorus-bridge structure, 3 minutes",
|
||||
bpm=72,
|
||||
duration_sec=180,
|
||||
format="wav"
|
||||
)
|
||||
audio.save("composition.wav")
|
||||
genre = "inspiring pop female vocal bright piano uplifting"
|
||||
|
||||
# Accompaniment generation
|
||||
from zen import MidiStream
|
||||
melody = MidiStream.from_file("melody.mid")
|
||||
accompaniment = model.accompany(
|
||||
melody=melody,
|
||||
style="bossa nova",
|
||||
instruments=["guitar", "bass", "drums"]
|
||||
lyrics = """
|
||||
[verse]
|
||||
Lines of the verse go here
|
||||
Another line that scans and rhymes
|
||||
[chorus]
|
||||
The hook that repeats and stays
|
||||
Carrying the song through its days
|
||||
"""
|
||||
|
||||
song = model.generate(
|
||||
genre_tags=genre,
|
||||
lyrics=lyrics,
|
||||
# optional reference audio enables ICL style/voice conditioning
|
||||
# reference_audio="reference.wav",
|
||||
)
|
||||
song.save("song.wav")
|
||||
\end{lstlisting}
|
||||
|
||||
\section{Provenance and Attribution}
|
||||
|
||||
Zen-Musician redistributes YuE. Required and recommended attribution:
|
||||
|
||||
\begin{itemize}
|
||||
\item \textbf{Model}: YuE --- Open Full-Song Music Generation Foundation Model.
|
||||
\item \textbf{Authors / affiliations}: M-A-P (Multimodal Art Projection) and
|
||||
HKUST (R.\ Yuan, H.\ Lin, S.\ Guo, G.\ Zhang, et al.; Y.\ Guo)
|
||||
\cite{yuan2025yue}.
|
||||
\item \textbf{License}: Apache-2.0 \cite{yuehf}.
|
||||
\item \textbf{Suggested credit}: ``YuE by HKUST/M-A-P.''
|
||||
\item \textbf{Source}: \href{https://github.com/multimodal-art-projection/YuE}{github.com/multimodal-art-projection/YuE};
|
||||
weights at \href{https://huggingface.co/m-a-p}{huggingface.co/m-a-p}.
|
||||
\end{itemize}
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
Zen-Musician's Section-Aware Transformer architecture and unified MIDI-audio token space produce
|
||||
music with measurably better large-scale structure (SAT section F1: 0.79 vs. 0.58 for best
|
||||
baseline) and competitive perceptual quality (FAD 2.3, MOS 4.1/5). The model's real-time
|
||||
accompaniment capability opens new applications in live performance while its large-scale
|
||||
generation pipeline serves content creation at production scale.
|
||||
Zen-Musician is the Zen stack's packaging of YuE, an Apache-2.0 long-form
|
||||
lyrics-to-song model from M-A-P and HKUST. The real system is a decoder-only,
|
||||
LLaMA-2-style autoregressive transformer (Stage-1 $\approx$7B) that performs
|
||||
track-decoupled next-token prediction over discrete audio tokens to generate
|
||||
vocals and accompaniment, followed by a $\approx$1B Stage-2 refinement/upsampling
|
||||
model. Conditioning is by lyrics and genre/style text, with an optional ICL mode
|
||||
for reference-based style transfer. All credit for the architecture, training, and
|
||||
reported capabilities belongs to YuE's authors; Zen's contribution is integration
|
||||
and distribution under the upstream license, with attribution. The from-scratch
|
||||
architecture and benchmark claims in the earlier draft were fabricated and have
|
||||
been removed.
|
||||
|
||||
\begin{thebibliography}{10}
|
||||
\bibitem{defossez2022encodec} A. Defossez et al., ``High Fidelity Neural Audio Compression,'' arXiv:2210.13438, 2022.
|
||||
\bibitem{copet2023musicgen} J. Copet et al., ``Simple and Controllable Music Generation,'' NeurIPS, 2023.
|
||||
\bibitem{agostinelli2023musiclm} A. Agostinelli et al., ``MusicLM: Generating Music From Text,'' arXiv:2301.11325, 2023.
|
||||
\bibitem{yuan2025yue} R.\ Yuan, H.\ Lin, S.\ Guo, G.\ Zhang, et al., ``YuE: Scaling Open Foundation Models for Long-Form Music Generation,'' arXiv:2503.08638, 2025. \url{https://arxiv.org/abs/2503.08638}
|
||||
\bibitem{yuehf} M-A-P, ``YuE model cards (Apache-2.0),'' Hugging Face. \url{https://huggingface.co/m-a-p/YuE-s1-7B-anneal-en-cot}
|
||||
\bibitem{yuegh} Multimodal Art Projection, ``YuE: Open Full-song Music Generation Foundation Model,'' GitHub. \url{https://github.com/multimodal-art-projection/YuE}
|
||||
\bibitem{touvron2023llama2} H.\ Touvron et al., ``Llama 2: Open Foundation and Fine-Tuned Chat Models,'' arXiv:2307.09288, 2023.
|
||||
\end{thebibliography}
|
||||
|
||||
\end{document}
|
||||
|
||||
Binary file not shown.
+97
-264
@@ -13,350 +13,183 @@
|
||||
\definecolor{zenblue}{RGB}{41,121,255}
|
||||
\hypersetup{colorlinks=true,linkcolor=zenblue,urlcolor=zenblue,citecolor=zenblue}
|
||||
|
||||
\title{\textbf{Zen-Video-I2V: Photorealistic Image-to-Video Generation\\
|
||||
with Controllable Motion Dynamics}\\[0.5em]
|
||||
\large Technical Whitepaper v2025.04}
|
||||
\title{\textbf{Zen-Video-I2V: Packaging Wan2.2-I2V-A14B\\
|
||||
for Image-to-Video Generation in the Zen Stack}\\[0.5em]
|
||||
\large Technical Whitepaper}
|
||||
\author{Zach Kelling \\ Zen LM Research Team\\
|
||||
\texttt{research@zenlm.org}\\
|
||||
\href{https://papers.zenlm.org}{papers.zenlm.org}}
|
||||
\date{April 2025}
|
||||
\date{2026}
|
||||
|
||||
\begin{document}
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
Zen-Video-I2V is a 14 billion parameter image-to-video generation model that animates static input images into fluid video sequences up to 10 seconds at 1080p resolution, with controllable motion direction, intensity, and style. The model achieves a Fr\'{e}chet Video Distance (FVD) of 289 on UCF-101, a Warping Error of 0.023 (measuring temporal warp consistency), and 91.3\% temporal consistency on the Video Quality Assessment benchmark. Zen-Video-I2V supports both single-image animation (producing plausible dynamic motion from scene content) and image-sequence interpolation (generating smooth transitions between provided keyframes). This paper describes the model architecture, motion conditioning mechanisms, training methodology, and comprehensive video quality evaluations.
|
||||
Zen-Video-I2V is a redistribution and integration of \textbf{Wan2.2-I2V-A14B}, the open-source image-to-video (I2V) generation model released by the Alibaba Wan team (\texttt{Wan-AI/Wan2.2-I2V-A14B}) under the \textbf{Apache-2.0} license. It is \emph{not} a from-scratch model: the weights, architecture, and training are the work of the Wan team, and Zen-Video-I2V repackages them for use within the Zen stack (consistent tooling, inference defaults, provenance, and safety integration). This whitepaper accurately describes the underlying Wan2.2 architecture --- a latent diffusion transformer (DiT) operating over a 3D causal VAE, trained with a flow-matching objective, and using an A14B mixture-of-experts (MoE) design that splits denoising between a high-noise and a low-noise expert (approximately 27B total parameters, roughly 14B active per step). We summarize the model's reported capabilities as published by the Wan team, document our packaging and integration choices, and discuss licensing, provenance, and safety. We deliberately do not report independent benchmark numbers; where quantitative claims appear, they are attributed to the upstream Wan team.
|
||||
\end{abstract}
|
||||
|
||||
\tableofcontents
|
||||
\newpage
|
||||
|
||||
\section{Introduction}
|
||||
\section{Introduction and Attribution}
|
||||
|
||||
The transition from image generation to video generation requires solving two qualitatively different problems. Image generation learns the distribution of plausible spatial configurations. Video generation must additionally learn the distribution of plausible temporal dynamics: how objects in a scene move, deform, and interact over time while maintaining physical and visual coherence across frames.
|
||||
\subsection{What Zen-Video-I2V is}
|
||||
|
||||
Image-to-video generation constrains this problem by conditioning on a real input image, fixing the spatial configuration and shifting the task to learning what temporal evolutions are consistent with that fixed starting state. This constraint is practically valuable: it allows users to start from real photographs, generated images, or concept art and animate them into video without requiring full video synthesis from text prompts.
|
||||
Zen-Video-I2V is a packaging of the upstream open-source model \textbf{Wan2.2-I2V-A14B} by Alibaba's Wan team. The original model, weights, training, and architecture are entirely the work of the Wan team and are distributed at \texttt{Wan-AI/Wan2.2-I2V-A14B} on Hugging Face under the \textbf{Apache-2.0} license. Zen-Video-I2V exists to integrate this model into the Zen stack: it provides consistent command-line and API surfaces, sensible inference defaults, content-provenance (C2PA) emission, and safety wiring, while leaving the model architecture and weights unmodified except where explicitly noted.
|
||||
|
||||
Zen-Video-I2V advances image-to-video generation through three contributions:
|
||||
This document supersedes earlier drafts that incorrectly described a from-scratch ``Zen MoDE'' architecture and reported fabricated benchmark results. No such architecture exists, and those numbers were not real. The remainder of this paper describes the actual Wan2.2-I2V-A14B model and our integration of it.
|
||||
|
||||
\begin{enumerate}
|
||||
\item \textbf{Dual-path motion conditioning}: Separate conditioning pathways for camera motion (dolly, pan, tilt, zoom, orbit) and scene motion (object-level dynamics), enabling independent control of each.
|
||||
\item \textbf{Physical plausibility constraints}: A physics-informed regularizer trained on simulation data penalizes animations that violate basic physical constraints (rigid body motion, fluid dynamics, cloth behavior).
|
||||
\item \textbf{Temporal attention at scale}: A full-resolution temporal attention mechanism (not limited to downsampled feature maps) operating across all frames simultaneously, producing high temporal coherence even in long-form (10 second) generations.
|
||||
\end{enumerate}
|
||||
\subsection{The image-to-video task}
|
||||
|
||||
\subsection{Model Overview}
|
||||
Image generation learns the distribution of plausible spatial configurations. Video generation must additionally learn plausible temporal dynamics: how objects move, deform, and interact over time while maintaining spatial and temporal coherence across frames. Image-to-video (I2V) constrains the problem by conditioning on a real input image (a photograph, a generated image, or concept art), which fixes the starting frame and shifts the task to learning temporal evolutions consistent with that start state. Wan2.2-I2V-A14B targets this setting, optionally also accepting a text prompt to steer the motion and content of the generated clip.
|
||||
|
||||
\subsection{Model overview}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Zen-Video-I2V Model Specification}
|
||||
\caption{Zen-Video-I2V / Wan2.2-I2V-A14B specification, as reported by the Wan team. Quantitative items are upstream-reported, not independently measured by Zen.}
|
||||
\begin{tabular}{ll}
|
||||
\toprule
|
||||
\textbf{Parameter} & \textbf{Value} \\
|
||||
\textbf{Item} & \textbf{Value} \\
|
||||
\midrule
|
||||
Architecture & Diffusion Transformer (DiT) with temporal attention \\
|
||||
Total Parameters & 14B \\
|
||||
Output Resolution & 720p (1280$\times$720) or 1080p (1920$\times$1080) \\
|
||||
Output Duration & 2--10 seconds (configurable) \\
|
||||
Frame Rate & 24 fps (configurable: 8, 16, 24 fps) \\
|
||||
FVD on UCF-101 & 289 \\
|
||||
Warping Error & 0.023 \\
|
||||
Temporal Consistency & 91.3\% \\
|
||||
Version & v2025.04 \\
|
||||
Release Date & April 2025 \\
|
||||
Upstream model & Wan2.2-I2V-A14B (Alibaba Wan) \\
|
||||
Upstream repository & \texttt{Wan-AI/Wan2.2-I2V-A14B} \\
|
||||
License & Apache-2.0 \\
|
||||
Task & Image-to-video (image, optionally text-conditioned) \\
|
||||
Architecture & Latent diffusion transformer (DiT) + 3D causal VAE \\
|
||||
Training objective & Flow matching \\
|
||||
Expert design & A14B mixture-of-experts (high-noise + low-noise) \\
|
||||
Total parameters & $\approx$27B (two $\approx$14B experts) \\
|
||||
Active parameters / step & $\approx$14B \\
|
||||
VAE compression (T$\times$H$\times$W) & 4$\times$8$\times$8 (Wan VAE) \\
|
||||
Output resolutions & 480P (832$\times$480) and 720P (1280$\times$720) \\
|
||||
Default clip & 81 frames at 16 fps ($\approx$5 s) \\
|
||||
Text encoder & umT5-XXL \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\section{Architecture}
|
||||
\section{Architecture (Wan2.2-I2V-A14B)}
|
||||
|
||||
\subsection{Diffusion Transformer with Temporal Attention}
|
||||
The architecture described in this section is that of the upstream Wan2.2 model; it is reproduced here for completeness and is attributed to the Wan team \cite{wan22repo,wan22hf}.
|
||||
|
||||
Zen-Video-I2V is built on a Diffusion Transformer (DiT) \cite{dit} backbone extended with full-resolution temporal attention. The model operates in the latent space of a pretrained video VAE \cite{vae} that encodes video frames at 8$\times$ spatial compression with 4-frame temporal compression, reducing a 1080p 240-frame video to a 135$\times$240$\times$60-frame latent volume.
|
||||
\subsection{Latent diffusion with a 3D causal VAE}
|
||||
|
||||
The transformer architecture alternates between spatial attention (across spatial positions within a frame) and temporal attention (across frames at fixed spatial positions):
|
||||
Wan2.2 follows the latent-diffusion paradigm: a 3D causal variational autoencoder (the Wan VAE) compresses video into a compact spatio-temporal latent representation, and a diffusion transformer models the distribution over those latents. For the A14B models, the VAE applies a $T\times H\times W$ downsampling factor of $4\times 8\times 8$ --- 4$\times$ along time and 8$\times$ along each spatial axis. (The separate, smaller TI2V-5B variant of Wan2.2 instead uses a higher-compression Wan2.2-VAE with a $4\times 16\times 16$ ratio, reaching $4\times 32\times 32$ after patchification; this whitepaper concerns the A14B I2V model, not TI2V-5B.)
|
||||
|
||||
\begin{align}
|
||||
h^{\text{spatial}} &= \text{SelfAttn}_{\text{spatial}}(h) \\
|
||||
h^{\text{temporal}} &= \text{SelfAttn}_{\text{temporal}}(h^{\text{spatial}})
|
||||
\end{align}
|
||||
Operating in latent space rather than pixel space is what makes high-resolution, multi-second video generation computationally tractable: the diffusion transformer denoises a much smaller latent volume, and the VAE decoder reconstructs pixels at the end.
|
||||
|
||||
Temporal attention is applied at full latent resolution, not downsampled, which enables the model to maintain fine-grained temporal coherence in high-frequency details (hair, fabric texture, water surface). The computational cost of full-resolution temporal attention is managed through FlashAttention-3 \cite{flashattn3} and temporal sparse attention patterns for long (10s) generations.
|
||||
\subsection{Diffusion transformer (DiT) backbone}
|
||||
|
||||
\subsection{Dual-Path Motion Conditioning}
|
||||
The denoiser is a diffusion transformer (DiT) \cite{dit} operating over the VAE latent tokens. As is standard for DiT-based video models, the backbone uses self-attention over the flattened spatio-temporal latent tokens together with cross-attention to the conditioning signal, with timestep conditioning applied throughout. Text conditioning, when a prompt is supplied, is provided through a umT5-XXL text encoder \cite{umt5}; the Wan codebase and model card credit umT5-XXL as the text encoder used in the Wan family. Image conditioning supplies the fixed first-frame content that the I2V model animates.
|
||||
|
||||
Motion control is provided through two independent conditioning pathways injected via cross-attention:
|
||||
\subsection{Flow-matching training objective}
|
||||
|
||||
\textbf{Camera Motion Pathway}: Encodes camera trajectory as a sequence of 6-DoF camera poses $\{(R_t, T_t)\}_{t=1}^F$ where $R_t \in SO(3)$ is rotation and $T_t \in \mathbb{R}^3$ is translation. The camera trajectory is encoded by a lightweight 6-layer MLP producing a 512-dimensional embedding per frame.
|
||||
Wan2.2 is trained with a flow-matching objective rather than the classical $\epsilon$-prediction DDPM loss. Flow matching learns a continuous-time velocity field that transports samples between the noise distribution and the data distribution; the model is trained to predict the velocity along the probability path, and generation integrates this learned velocity field at inference time. We do not restate upstream training hyperparameters here, as they are the Wan team's and are documented in their release.
|
||||
|
||||
\textbf{Scene Motion Pathway}: Encodes object-level dynamics as an optical flow field $\{u_t\}_{t=1}^F$ where $u_t \in \mathbb{R}^{H \times W \times 2}$ is the dense flow from frame $t-1$ to frame $t$. The flow field is encoded by a small 4-layer U-Net producing a 256-dimensional embedding at the latent spatial resolution.
|
||||
\subsection{A14B mixture-of-experts: high-noise and low-noise experts}
|
||||
|
||||
The two pathways are combined through learned gating:
|
||||
|
||||
\begin{equation}
|
||||
c_{\text{motion}} = \sigma(g) \cdot c_{\text{camera}} + (1 - \sigma(g)) \cdot c_{\text{scene}}
|
||||
\end{equation}
|
||||
|
||||
where $g$ is a learned gating scalar conditioned on the generation context. In practice, the gate learns to balance camera and scene motion conditioning based on the input image content: outdoor scenes with clear horizon lines favor camera conditioning, while close-up portraits or product shots favor scene motion conditioning.
|
||||
|
||||
\subsection{Image Conditioning}
|
||||
|
||||
The input image is encoded by the video VAE and injected into the generation process as:
|
||||
The defining feature of the Wan2.2 ``A14B'' models is a two-expert mixture-of-experts (MoE) design tailored to the denoising trajectory of a diffusion model. Rather than routing tokens (as in a typical LLM MoE), Wan2.2 routes \emph{by denoising stage}, splitting the reverse process into two regimes:
|
||||
|
||||
\begin{enumerate}
|
||||
\item \textbf{First-frame conditioning}: The encoded input image is fixed as the first latent frame, with the model trained to denoise subsequent frames conditioned on this fixed start state.
|
||||
\item \textbf{Global image embedding}: A CLIP image encoder extracts a semantic embedding of the input image that conditions all transformer layers via cross-attention, maintaining semantic consistency throughout the generated video.
|
||||
\item \textbf{Pixel-level conditioning}: For high-fidelity reconstruction of the input image's specific textures and details, ControlNet-style feature injection \cite{controlnet} copies early encoder features directly into the decoder path.
|
||||
\item \textbf{High-noise expert} --- active during the early, high-noise steps of denoising. With little signal present, this expert focuses on the overall layout and coarse structure of the video.
|
||||
\item \textbf{Low-noise expert} --- active during the later, low-noise steps. With the coarse structure established, this expert refines fine details.
|
||||
\end{enumerate}
|
||||
|
||||
\subsection{Physics-Informed Regularization}
|
||||
The handoff between experts is governed by the signal-to-noise ratio (SNR), which decreases monotonically as the denoising step index increases. The Wan team selects the switching point at a threshold tied to the SNR (described in their materials as a threshold around half of the minimum SNR), so that the high-noise expert handles the noisiest portion of the trajectory and the low-noise expert handles the rest.
|
||||
|
||||
Physical plausibility is enforced through an auxiliary physics regularizer trained on simulation data from three domains:
|
||||
Because each expert has roughly 14B parameters and only one expert is active at any given denoising step, the model totals approximately 27B parameters but keeps the per-step active parameter count --- and therefore the per-step compute and memory footprint --- near that of a single $\approx$14B model. This is the same motivation that drives MoE in large language models (increase total capacity without proportionally increasing per-step cost), applied here along the diffusion timestep axis.
|
||||
|
||||
\begin{enumerate}
|
||||
\item \textbf{Rigid body}: Motion from a physics engine simulator covering 50K object types.
|
||||
\item \textbf{Fluid}: Water surface simulation data from a fluid dynamics solver.
|
||||
\item \textbf{Cloth and soft body}: Garment simulation covering 20K clothing styles.
|
||||
\end{enumerate}
|
||||
\subsection{Image conditioning}
|
||||
|
||||
The regularizer $R_\psi$ takes a generated optical flow sequence and predicts a physics plausibility score. This score is used as an auxiliary loss during training:
|
||||
As an I2V model, Wan2.2-I2V-A14B conditions generation on an input image so that the generated clip begins from, and remains consistent with, that image. The output resolution and aspect ratio follow the input image dimensions within the supported 480P/720P range. An optional text prompt can additionally steer motion and content.
|
||||
|
||||
\begin{equation}
|
||||
\mathcal{L}_{\text{phys}} = -\log R_\psi(u_{1:F})
|
||||
\end{equation}
|
||||
\section{Capabilities and Reported Results}
|
||||
|
||||
The physics regularizer is frozen during video model training (trained separately on simulation data only), serving as a learned prior that penalizes motions that violate physical constraints without requiring differentiable physics simulation.
|
||||
\subsection{Outputs}
|
||||
|
||||
\section{Training Methodology}
|
||||
Wan2.2-I2V-A14B generates short video clips --- by default 81 frames at 16 fps (approximately five seconds) --- at 480P ($832\times 480$) or 720P ($1280\times 720$), with the aspect ratio adapted to the input image. The Wan team describes the A14B MoE family as offering improved complex-motion generation and cinematic-quality output relative to the prior Wan2.1 generation.
|
||||
|
||||
\subsection{Training Data}
|
||||
\subsection{Training-data scale (upstream-reported)}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Training Data Composition}
|
||||
\begin{tabular}{lrr}
|
||||
\toprule
|
||||
\textbf{Source} & \textbf{Video Hours} & \textbf{Fraction} \\
|
||||
\midrule
|
||||
Licensed stock footage & 2,400,000 & 48.0\% \\
|
||||
Documentary and nature video & 800,000 & 16.0\% \\
|
||||
Film and television (licensed) & 600,000 & 12.0\% \\
|
||||
User-generated video (consented) & 500,000 & 10.0\% \\
|
||||
Physics simulation renders & 400,000 & 8.0\% \\
|
||||
3D rendering and animation & 300,000 & 6.0\% \\
|
||||
\midrule
|
||||
\textbf{Total} & \textbf{5,000,000} & \textbf{100\%} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
The Wan team reports that Wan2.2 was trained on substantially more data than Wan2.1: approximately \textbf{+65.6\% more images} and \textbf{+83.2\% more videos}. These figures are reported by the Wan team; Zen has not independently verified them.
|
||||
|
||||
All training data was filtered for quality (resolution $\geq$ 720p, motion blur $<$ threshold, compression artifact score $<$ threshold) and processed to remove frames with faces not covered by consent agreements.
|
||||
\subsection{Benchmarks (upstream-reported, attributed)}
|
||||
|
||||
\subsection{Training Curriculum}
|
||||
The Wan team reports that Wan2.2 achieves top performance among open- and closed-source models on their own \textbf{Wan-Bench 2.0} evaluation suite \cite{wan22repo}. We reproduce this only as an attributed upstream claim. \textbf{Zen-Video-I2V does not contribute independent benchmark numbers}, and we deliberately omit any FVD, warping-error, temporal-consistency, or mean-opinion-score tables: prior drafts of this document contained fabricated values for such metrics, and we will not substitute new unverified numbers. Readers seeking quantitative comparisons should consult the Wan team's published evaluations and independent third-party leaderboards.
|
||||
|
||||
Training proceeds through three stages:
|
||||
\section{Zen Integration}
|
||||
|
||||
\textbf{Stage 1 -- Short clip pretraining} (2 seconds, 48 frames): The model learns basic temporal coherence and motion dynamics on short clips, where the temporal attention computation is manageable at full resolution.
|
||||
This section documents what Zen-Video-I2V adds on top of the upstream model. None of it constitutes a new model architecture.
|
||||
|
||||
\textbf{Stage 2 -- Mid-length extension} (5 seconds, 120 frames): The context length is extended through curriculum continuation training, where the model is initialized from Stage 1 weights and fine-tuned on longer clips with sparse temporal attention for positions beyond 48 frames.
|
||||
\subsection{Packaging}
|
||||
|
||||
\textbf{Stage 3 -- Long-form fine-tuning} (10 seconds, 240 frames): Final stage fine-tuning on high-quality long clips (filtered for production quality), with the sparse temporal attention pattern expanded to cover the full temporal extent.
|
||||
|
||||
\subsection{Noise Schedule}
|
||||
|
||||
Zen-Video-I2V uses a continuous-time diffusion process with a learned noise schedule that is adaptive per-frame: the noise level at frame $t$ is conditioned on the frame index, applying higher signal-to-noise ratios for early frames (closer to the conditioning image) and allowing more variance for later frames (more uncertain motion trajectories):
|
||||
|
||||
\begin{equation}
|
||||
\sigma_t^{(f)} = \sigma_{\min} + \frac{f}{F}(\sigma_{\max} - \sigma_{\min}), \quad f \in \{1, \ldots, F\}
|
||||
\end{equation}
|
||||
|
||||
This schedule encourages the model to generate temporally consistent short-range dynamics while maintaining temporal diversity over longer horizons.
|
||||
|
||||
\section{Evaluation}
|
||||
|
||||
\subsection{Video Quality Benchmarks}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Video Quality Benchmark Results}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{FVD (UCF-101)} $\downarrow$ & \textbf{Warping Error} $\downarrow$ & \textbf{Temporal Consistency} $\uparrow$ & \textbf{PSNR (1st frame)} $\uparrow$ \\
|
||||
\midrule
|
||||
Zen-Video-I2V (14B) & \textbf{289} & \textbf{0.023} & \textbf{91.3\%} & 42.1 dB \\
|
||||
Comparable I2V model A & 341 & 0.031 & 87.6\% & 40.8 dB \\
|
||||
Comparable I2V model B & 387 & 0.038 & 84.2\% & 39.4 dB \\
|
||||
Comparable I2V model C & 312 & 0.027 & 89.1\% & 41.3 dB \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Human Evaluation}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Human Evaluation MOS (1--5 scale, 500 clips rated by 3 raters each)}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{Realism} & \textbf{Temporal Coherence} & \textbf{Motion Quality} & \textbf{Overall} \\
|
||||
\midrule
|
||||
Zen-Video-I2V (14B) & \textbf{4.1} & \textbf{4.0} & \textbf{3.9} & \textbf{4.0} \\
|
||||
Comparable model A & 3.7 & 3.5 & 3.4 & 3.5 \\
|
||||
Comparable model B & 3.4 & 3.3 & 3.1 & 3.3 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Motion Control Evaluation}
|
||||
|
||||
We evaluate the precision of the dual-path motion control on a benchmark of 200 clips with specified camera trajectories and object motion targets:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Motion Control Accuracy}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Motion Type} & \textbf{Direction Accuracy} & \textbf{Intensity Correlation} \\
|
||||
\midrule
|
||||
Camera pan / tilt & 94.2\% & 0.91 \\
|
||||
Camera dolly (zoom equivalent) & 91.7\% & 0.88 \\
|
||||
Camera orbit & 88.3\% & 0.84 \\
|
||||
Object translation & 87.4\% & 0.82 \\
|
||||
Object deformation (cloth, fluid) & 79.1\% & 0.76 \\
|
||||
\textbf{Average} & \textbf{88.1\%} & \textbf{0.84} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Resolution and Duration Scaling}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Quality vs. Resolution and Duration}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
\textbf{Resolution} & \textbf{Duration} & \textbf{FVD} & \textbf{Temporal Consistency} & \textbf{Gen Time} \\
|
||||
\midrule
|
||||
720p & 2s & 241 & 93.7\% & 18s \\
|
||||
720p & 5s & 267 & 92.1\% & 44s \\
|
||||
720p & 10s & 289 & 91.3\% & 87s \\
|
||||
1080p & 2s & 253 & 93.1\% & 31s \\
|
||||
1080p & 5s & 278 & 91.4\% & 74s \\
|
||||
1080p & 10s & 302 & 90.7\% & 148s \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
Generation times measured on 4$\times$H100 80GB SXM5 (FP8, 50 DDIM steps).
|
||||
|
||||
\subsection{Image-Sequence Interpolation}
|
||||
|
||||
When provided with multiple keyframe images, Zen-Video-I2V generates smooth interpolated video transitions:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Keyframe Interpolation Quality}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Metric} & \textbf{Zen-Video-I2V} & \textbf{Linear Interpolation} & \textbf{Morph baseline} \\
|
||||
\midrule
|
||||
Perceptual quality MOS & 4.2/5 & 2.1/5 & 3.1/5 \\
|
||||
Boundary frame fidelity (PSNR) & 41.3 dB & 44.2 dB & 38.7 dB \\
|
||||
Intermediate plausibility MOS & 4.0/5 & 1.8/5 & 2.9/5 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Per-Domain Performance}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{FVD by Content Domain}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Domain} & \textbf{Zen-Video-I2V FVD} & \textbf{Best Comparable FVD} \\
|
||||
\midrule
|
||||
Nature / landscape & 243 & 298 \\
|
||||
Human portrait & 318 & 394 \\
|
||||
Animal motion & 271 & 337 \\
|
||||
Urban / architecture & 261 & 319 \\
|
||||
Product / object & 234 & 287 \\
|
||||
Abstract / artistic & 312 & 381 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
Human portrait is the most challenging domain due to the perceptual sensitivity to uncanny motion artifacts in faces; Zen-Video-I2V's physics-informed regularizer is less effective for biological motion than for physical objects.
|
||||
|
||||
\section{Deployment}
|
||||
|
||||
\subsection{Inference Requirements}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Inference Hardware Requirements}
|
||||
\begin{tabular}{llll}
|
||||
\toprule
|
||||
\textbf{Config} & \textbf{Hardware} & \textbf{Resolution} & \textbf{Gen Time (5s)} \\
|
||||
\midrule
|
||||
High quality & 4 $\times$ H100 80GB & 1080p & 74s \\
|
||||
Standard & 4 $\times$ H100 80GB & 720p & 44s \\
|
||||
Fast (reduced steps) & 2 $\times$ H100 80GB & 720p & 28s \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{API Interface}
|
||||
|
||||
Zen-Video-I2V is accessible through the Zen LM API with the following parameters:
|
||||
Zen-Video-I2V wraps Wan2.2-I2V-A14B with:
|
||||
|
||||
\begin{itemize}
|
||||
\item \texttt{image}: Input image (JPEG/PNG, min 512px on shorter edge).
|
||||
\item \texttt{duration}: Output duration in seconds (2.0--10.0, default 5.0).
|
||||
\item \texttt{resolution}: Output resolution (``720p'' or ``1080p'').
|
||||
\item \texttt{camera\_motion}: Camera trajectory type (``static'', ``dolly\_in'', ``dolly\_out'', ``pan\_left'', ``pan\_right'', ``orbit'', or custom 6-DoF trajectory).
|
||||
\item \texttt{motion\_intensity}: Overall motion scale (0.0--1.0, default 0.5).
|
||||
\item \texttt{fps}: Output frame rate (8, 16, or 24).
|
||||
\item \texttt{seed}: Reproducibility seed for deterministic generation.
|
||||
\item Consistent command-line and API entry points aligned with the rest of the Zen stack.
|
||||
\item Pinned, reproducible inference defaults (resolution, frame count, fps, sampling steps, guidance) chosen to match the upstream reference configuration.
|
||||
\item Optional reduced-precision inference (e.g.\ FP8 weights) for lower-memory deployment, following community-standard quantization of the Wan weights; this is an inference-time packaging choice and does not alter the trained model.
|
||||
\item Provenance and safety wiring (Section~\ref{sec:provenance}).
|
||||
\end{itemize}
|
||||
|
||||
\section{Safety Considerations}
|
||||
\subsection{Inference characteristics}
|
||||
|
||||
\subsection{Identity and Consent}
|
||||
Inference cost is governed by the upstream A14B architecture: per denoising step, a single $\approx$14B expert is active, so the compute and memory profile is close to a dense $\approx$14B video DiT rather than a dense 27B one. Actual latency and memory depend on resolution, frame count, sampling-step count, precision, and hardware. Because Zen-Video-I2V does not change the model, we direct users to upstream and community measurements for hardware sizing rather than publishing our own performance tables here.
|
||||
|
||||
Zen-Video-I2V includes a face detection classifier that identifies human faces in input images. Generation involving detected faces requires:
|
||||
\subsection{API interface}
|
||||
|
||||
Zen-Video-I2V exposes parameters consistent with the I2V task:
|
||||
|
||||
\begin{itemize}
|
||||
\item Explicit consent acknowledgment from the API caller that they have rights to animate the depicted individual.
|
||||
\item Invisible watermarking applied to all generated video (C2PA-compatible provenance metadata).
|
||||
\item Rate limiting and audit logging for accounts generating face-containing animations.
|
||||
\item \texttt{image}: input image to be animated (the fixed first frame).
|
||||
\item \texttt{prompt}: optional text prompt to steer motion and content.
|
||||
\item \texttt{resolution}: \texttt{480p} ($832\times 480$) or \texttt{720p} ($1280\times 720$); aspect ratio follows the input image.
|
||||
\item \texttt{num\_frames}: number of frames to generate (default 81).
|
||||
\item \texttt{fps}: output frame rate (default 16).
|
||||
\item \texttt{steps}: number of sampling steps.
|
||||
\item \texttt{guidance}: classifier-free guidance scale.
|
||||
\item \texttt{seed}: reproducibility seed for deterministic generation.
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Deepfake Mitigation}
|
||||
\section{Licensing}
|
||||
|
||||
Zen-Video-I2V applies Content Authenticity Initiative (CAI) metadata to all generated videos, marking them as AI-generated in a tamper-evident way. The model is also trained to degrade gracefully (reduce quality) on inputs that appear to be composite images designed for deceptive impersonation (face-swapped images, morphed composites).
|
||||
The upstream model is released under the \textbf{Apache-2.0} license by the Alibaba Wan team \cite{wan22hf,wan22repo}. Apache-2.0 is a permissive license that allows commercial use, modification, and redistribution, subject to its notice and attribution requirements (preserving copyright and license notices, and stating significant changes). Zen-Video-I2V is distributed consistently with these terms: we retain the upstream license and notices, attribute the work to the Alibaba Wan team, and document our integration changes.
|
||||
|
||||
Because the upstream license is genuinely permissive, Zen-Video-I2V can be redistributed and used commercially without the geographic or usage caps attached to some other restrictively-licensed video models. We nonetheless apply the provenance and safety measures described below as a matter of policy.
|
||||
|
||||
\section{Provenance and Safety}
|
||||
\label{sec:provenance}
|
||||
|
||||
\subsection{Content provenance}
|
||||
|
||||
Zen-Video-I2V emits C2PA-compatible content-provenance metadata marking output as AI-generated, in a tamper-evident manifest. This supports disclosure obligations (e.g.\ AI-generated-content labeling requirements) and downstream verification. Provenance emission is part of the Zen integration, not the upstream model.
|
||||
|
||||
\subsection{Identity and consent}
|
||||
|
||||
For inputs containing recognizable people, Zen-Video-I2V's deployment policy requires that the caller affirm they have the rights to animate the depicted individual, applies provenance metadata to all output, and supports audit logging for accounts generating person-containing animations. These are deployment-layer controls; the underlying generative model is the upstream Wan2.2-I2V-A14B.
|
||||
|
||||
\subsection{Misuse mitigation}
|
||||
|
||||
Generated clips are marked as synthetic via the provenance metadata above so that downstream consumers can detect AI-generated origin. As with any open generative model, these measures reduce but do not eliminate misuse risk; responsible-use policy and access controls remain necessary.
|
||||
|
||||
\section{Related Work}
|
||||
|
||||
Image-to-video generation has advanced through GAN-based \cite{vidgan}, flow-based \cite{flowgen}, and diffusion-based \cite{stablevideo,svd} approaches. Temporal attention mechanisms in video diffusion have been studied in \cite{tune-a-video,videocrafter}. Motion controllability has been explored through trajectory conditioning \cite{dragnuwa} and optical flow guidance \cite{motionctrl}. Zen-Video-I2V advances this line of work through dual-path motion conditioning, physics-informed regularization, and full-resolution temporal attention at 14B parameter scale.
|
||||
Image-to-video generation has progressed through GAN-based \cite{vidgan}, flow-based \cite{flowgen}, and diffusion-based \cite{stablevideo,svd} approaches. Latent video diffusion with transformer backbones \cite{dit,svd} and large-scale video DiTs underpin the current generation of high-quality models. Wan2.2 \cite{wan22repo,wan22hf} contributes a denoising-stage mixture-of-experts (high-noise / low-noise experts) on top of a flow-matching latent video DiT with a 3D causal VAE. Zen-Video-I2V does not introduce new modeling techniques; it packages Wan2.2-I2V-A14B for use within the Zen stack.
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
Zen-Video-I2V establishes new state-of-the-art results on image-to-video generation benchmarks (FVD 289, Warping Error 0.023, Temporal Consistency 91.3\%) while introducing dual-path motion control and physics-informed regularization that improve the plausibility and controllability of generated animations. At 14B parameters, the model generates photorealistic 10-second 1080p video from single images, suitable for creative production, product visualization, and interactive media applications. The physics-informed regularizer and full-resolution temporal attention architecture provide strong foundations for future extensions to longer-form and higher-resolution video generation.
|
||||
Zen-Video-I2V is an Apache-2.0 redistribution and integration of Alibaba Wan's Wan2.2-I2V-A14B image-to-video model. The model is a flow-matching latent diffusion transformer over a 3D causal VAE ($4\times 8\times 8$ compression), with a two-expert (high-noise / low-noise) mixture-of-experts design totaling roughly 27B parameters while keeping about 14B active per denoising step. All architectural and training credit belongs to the Wan team. Zen's contribution is packaging: consistent tooling and inference defaults, content provenance, and safety wiring. We attribute all quantitative claims to upstream and intentionally publish no independent or fabricated benchmark numbers.
|
||||
|
||||
\begin{thebibliography}{9}
|
||||
\bibitem{wan22repo} Wan Team, Alibaba. \emph{Wan2.2: Open and Advanced Large-Scale Video Generative Models.} GitHub repository. \url{https://github.com/Wan-Video/Wan2.2}
|
||||
\bibitem{wan22hf} Wan Team, Alibaba. \emph{Wan2.2-I2V-A14B} (model card, Apache-2.0). Hugging Face. \url{https://huggingface.co/Wan-AI/Wan2.2-I2V-A14B}
|
||||
\bibitem{dit} Peebles, W. \& Xie, S. (2023). Scalable Diffusion Models with Transformers. ICCV 2023.
|
||||
\bibitem{vae} Kingma, D.P. \& Welling, M. (2014). Auto-Encoding Variational Bayes. ICLR 2014.
|
||||
\bibitem{flashattn3} Shah, J. et al. (2024). FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision. arXiv:2407.08608.
|
||||
\bibitem{controlnet} Zhang, L. et al. (2023). Adding Conditional Control to Text-to-Image Diffusion Models. ICCV 2023.
|
||||
\bibitem{umt5} Chung, H.W. et al. (2023). UniMax / umT5: Multilingual Pretraining for the Masses. ICLR 2023.
|
||||
\bibitem{vidgan} Vondrick, C. et al. (2016). Generating Videos with Scene Dynamics. NeurIPS 2016.
|
||||
\bibitem{flowgen} Holynski, A. et al. (2021). Animating Pictures with Eulerian Motion Fields. CVPR 2021.
|
||||
\bibitem{stablevideo} Chai, L. et al. (2023). StableVideo: Text-driven Consistency-aware Diffusion Video Editing. ICCV 2023.
|
||||
\bibitem{svd} Blattmann, A. et al. (2023). Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets. arXiv:2311.15127.
|
||||
\bibitem{tune-a-video} Wu, J.Z. et al. (2023). Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation. ICCV 2023.
|
||||
\bibitem{videocrafter} He, Y. et al. (2023). VideoCrafter1: Open Diffusion Models for High-Quality Video Generation. arXiv:2310.19512.
|
||||
\bibitem{dragnuwa} Wang, S. et al. (2024). DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectory. ECCV 2024.
|
||||
\bibitem{motionctrl} Wang, Z. et al. (2024). MotionCtrl: A Unified and Flexible Motion Controller for Video Generation. SIGGRAPH 2024.
|
||||
\end{thebibliography}
|
||||
|
||||
\end{document}
|
||||
|
||||
Binary file not shown.
-389
@@ -1,389 +0,0 @@
|
||||
\documentclass[11pt,a4paper]{article}
|
||||
\usepackage[utf8]{inputenc}
|
||||
\usepackage[T1]{fontenc}
|
||||
\usepackage{amsmath,amsfonts,amssymb}
|
||||
\usepackage{graphicx}
|
||||
\usepackage{hyperref}
|
||||
\usepackage{listings}
|
||||
\usepackage[dvipsnames]{xcolor}
|
||||
\usepackage{booktabs}
|
||||
\usepackage{float}
|
||||
\usepackage{geometry}
|
||||
\geometry{margin=1in}
|
||||
|
||||
\definecolor{zenblue}{RGB}{41,121,255}
|
||||
\definecolor{zengreen}{RGB}{52,199,89}
|
||||
\definecolor{codegray}{RGB}{245,245,245}
|
||||
|
||||
\hypersetup{colorlinks=true,linkcolor=zenblue,urlcolor=zenblue,citecolor=zenblue}
|
||||
|
||||
\lstset{
|
||||
backgroundcolor=\color{codegray},
|
||||
basicstyle=\ttfamily\small,
|
||||
breaklines=true,
|
||||
captionpos=b,
|
||||
frame=single,
|
||||
numbers=left,
|
||||
numberstyle=\tiny\color{gray}
|
||||
}
|
||||
|
||||
\title{
|
||||
\vspace{-2cm}
|
||||
\Large \textbf{Zen AI Model Family} \\
|
||||
\vspace{0.5cm}
|
||||
\Huge \textbf{Zen-Video} \\
|
||||
\vspace{0.3cm}
|
||||
\large Text-to-Video Generation with Temporal Consistency and Cinematic Quality \\
|
||||
\vspace{0.5cm}
|
||||
\normalsize Technical Report v2025.01
|
||||
}
|
||||
|
||||
\author{
|
||||
Hanzo AI Research Team\thanks{research@hanzo.ai} \and
|
||||
Zoo Labs Foundation\thanks{foundation@zoo.ngo}
|
||||
}
|
||||
|
||||
\date{January 2025}
|
||||
|
||||
\begin{document}
|
||||
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
We present \textbf{Zen-Video}, a 14-billion parameter text-to-video generation model achieving
|
||||
state-of-the-art quality on standard video generation benchmarks: UCF-101 Inception Score (IS)
|
||||
of 96.8, FID of 8.7, and Frechet Video Distance (FVD) of 312. Zen-Video generates up to
|
||||
60-second, 1080p video clips from text prompts, image-conditioned extensions, and structured
|
||||
shot plans (compatible with Zen-Director output). The model is built on a spatiotemporal
|
||||
diffusion transformer (ST-DiT) architecture with a novel \textbf{Temporal Coherence Module
|
||||
(TCM)} that enforces physical plausibility and motion continuity across frames---eliminating
|
||||
the frame flickering and motion artifacts that afflict prior latent diffusion approaches.
|
||||
Zen-Video also supports video editing (temporally consistent inpainting and style transfer),
|
||||
frame interpolation (4$\times$ and 8$\times$ upsampling of existing video), and controlled
|
||||
camera motion (pan, tilt, dolly, orbit) as first-class generation modes. The model operates
|
||||
entirely in a compressed latent video space, reducing generation cost by 32$\times$ relative
|
||||
to pixel-space approaches.
|
||||
\end{abstract}
|
||||
|
||||
\tableofcontents
|
||||
\newpage
|
||||
|
||||
\section{Introduction}
|
||||
|
||||
Text-to-video generation has advanced rapidly, yet production-quality generation remains elusive
|
||||
for most open models. Key persistent challenges include:
|
||||
|
||||
\begin{enumerate}
|
||||
\item \textbf{Temporal inconsistency}: Objects change appearance, teleport, or deform
|
||||
unnaturally between frames.
|
||||
\item \textbf{Motion quality}: Generated motion is often jerky, unphysical, or lacks
|
||||
the smooth acceleration profiles of real-world dynamics.
|
||||
\item \textbf{Prompt adherence over time}: Models often correctly generate the first frame
|
||||
but drift away from the prompt over longer generations.
|
||||
\item \textbf{Compute cost}: Generating 60 seconds of 1080p video requires tractable
|
||||
inference time and cost for production use.
|
||||
\end{enumerate}
|
||||
|
||||
Zen-Video addresses all four through architectural and training innovations, achieving
|
||||
state-of-the-art benchmark results while supporting generation of up to 60-second 1080p clips.
|
||||
|
||||
\subsection{Model Overview}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{ll}
|
||||
\toprule
|
||||
\textbf{Property} & \textbf{Value} \\
|
||||
\midrule
|
||||
Parameters & 14B \\
|
||||
Architecture & Spatiotemporal Diffusion Transformer (ST-DiT) \\
|
||||
Latent Space & 8$\times$ spatial, 4$\times$ temporal compression \\
|
||||
Max Resolution & 1920$\times$1080 (1080p) \\
|
||||
Max Duration & 60 seconds \\
|
||||
Frame Rates & 12, 24, 30 fps \\
|
||||
Text Encoder & T5-XXL + CLIP ViT-L \\
|
||||
Training Data & 850M video-text pairs (filtered), 2.4B frames \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen-Video Model Specifications}
|
||||
\end{table}
|
||||
|
||||
\section{Architecture}
|
||||
|
||||
\subsection{Video VAE}
|
||||
|
||||
A spatiotemporal VAE encodes video clips into a compressed latent representation:
|
||||
\begin{itemize}
|
||||
\item \textbf{Spatial compression}: 8$\times$ downsampling per spatial dimension
|
||||
(a 1920$\times$1080 frame $\to$ 240$\times$135 latent).
|
||||
\item \textbf{Temporal compression}: 4$\times$ downsampling along the time axis
|
||||
(24fps input $\to$ 6fps latent).
|
||||
\item \textbf{Latent channels}: 16.
|
||||
\end{itemize}
|
||||
|
||||
The combined compression factor is 8$\times$8$\times$4$\times$16/3 $\approx$ 110$\times$
|
||||
reduction in data volume relative to raw video, enabling tractable diffusion in latent space.
|
||||
|
||||
\subsection{Spatiotemporal Diffusion Transformer (ST-DiT)}
|
||||
|
||||
The ST-DiT extends the DiT (Diffusion Transformer) architecture to video by interleaving
|
||||
spatial and temporal attention blocks:
|
||||
|
||||
\begin{align}
|
||||
h &= h + \text{SpatialAttn}(\text{LayerNorm}(h)) \\
|
||||
h &= h + \text{TemporalAttn}(\text{LayerNorm}(h)) \\
|
||||
h &= h + \text{CrossAttn}(\text{LayerNorm}(h), c_{\text{text}}) \\
|
||||
h &= h + \text{FFN}(\text{LayerNorm}(h))
|
||||
\end{align}
|
||||
|
||||
where $c_{\text{text}}$ is the text conditioning from the dual encoder (T5-XXL for semantic
|
||||
content, CLIP for visual style).
|
||||
|
||||
The spatial attention operates over the $H \times W$ spatial positions independently for each
|
||||
frame. The temporal attention operates over the $T$ time steps independently for each spatial
|
||||
position. This factored design reduces the quadratic attention cost from $O((HWT)^2)$ to
|
||||
$O((HW)^2 T + HW T^2)$, enabling attention over long video sequences.
|
||||
|
||||
\subsection{Temporal Coherence Module (TCM)}
|
||||
|
||||
The TCM is the key innovation enabling temporal consistency. It operates as a post-attention
|
||||
consistency regularizer applied after every 4 transformer blocks:
|
||||
|
||||
\begin{equation}
|
||||
h_t = h_t + \alpha \cdot \text{TCM}(h_{t-1}, h_t, h_{t+1})
|
||||
\end{equation}
|
||||
|
||||
The TCM computes an optical-flow-aligned weighted average of adjacent frame features, where
|
||||
the flow is estimated by a lightweight 3-layer ConvNet operating on low-resolution latent
|
||||
features. The parameter $\alpha = 0.3$ is learned during training. TCM reduces frame-to-frame
|
||||
variation by 41\% (measured by latent-space cosine distance) while allowing genuine motion.
|
||||
|
||||
\subsection{Camera Motion Control}
|
||||
|
||||
Camera motion is specified as a conditioning signal consisting of:
|
||||
\begin{itemize}
|
||||
\item \textbf{Type}: pan, tilt, dolly in/out, orbit, static, handheld.
|
||||
\item \textbf{Speed}: normalized 0--1.
|
||||
\item \textbf{Direction}: angle in degrees for directional motions.
|
||||
\end{itemize}
|
||||
|
||||
Camera motion embeddings are injected into the temporal attention layers via AdaLN conditioning,
|
||||
similar to diffusion timestep conditioning.
|
||||
|
||||
\section{Training}
|
||||
|
||||
\subsection{Dataset}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lrrl}
|
||||
\toprule
|
||||
\textbf{Source} & \textbf{Videos} & \textbf{Proportion} & \textbf{Filtering} \\
|
||||
\midrule
|
||||
WebVid-10M & 10,000,000 & 11.8\% & Quality + NSFW filter \\
|
||||
HD-VILA-100M (subset) & 20,000,000 & 23.5\% & Motion + aesthetic score \\
|
||||
Panda-70M & 70,000,000 & 82.4\% & Caption quality filter \\
|
||||
Licensed studio content & 50,000 & 0.1\% & High-quality hand-curated \\
|
||||
Synthetic renders & 1,200,000 & 1.4\% & 3D engine renders \\
|
||||
\midrule
|
||||
\textbf{Total (after filter)} & \textbf{84,950,000} & 100\% & \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen-Video Training Data (850M raw, 85M after quality filtering)}
|
||||
\end{table}
|
||||
|
||||
\subsection{Training Protocol}
|
||||
|
||||
Training proceeds through four stages on progressively higher resolutions:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lllll}
|
||||
\toprule
|
||||
\textbf{Stage} & \textbf{Resolution} & \textbf{Duration} & \textbf{Steps} & \textbf{Hardware} \\
|
||||
\midrule
|
||||
1 & 256$\times$144, 4fps & 4s & 100K & 256$\times$A100 \\
|
||||
2 & 512$\times$288, 8fps & 8s & 100K & 256$\times$A100 \\
|
||||
3 & 1024$\times$576, 24fps & 30s & 60K & 512$\times$A100 \\
|
||||
4 & 1920$\times$1080, 24fps & 60s & 30K & 512$\times$A100 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Progressive Resolution Training Stages}
|
||||
\end{table}
|
||||
|
||||
\subsection{Inference Optimization}
|
||||
|
||||
At inference time, Zen-Video uses:
|
||||
\begin{itemize}
|
||||
\item \textbf{DDIM sampling}: 50 steps (quality mode) or 20 steps (fast mode).
|
||||
\item \textbf{Classifier-free guidance}: $w = 7.5$ (text), $w = 2.0$ (image conditioning).
|
||||
\item \textbf{Temporal tiling}: 60-second clips generated in overlapping 10-second tiles
|
||||
with 2-second overlap, blended via cosine fade in latent space.
|
||||
\item \textbf{Tensor parallelism}: 4-GPU inference for 1080p generation.
|
||||
\end{itemize}
|
||||
|
||||
\section{Evaluation}
|
||||
|
||||
\subsection{UCF-101 Generation Quality}
|
||||
|
||||
Standard video generation benchmark: generate 256$\times$256 videos for UCF-101 classes and
|
||||
evaluate using Inception Score (IS), FID, and FVD.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{IS} $\uparrow$ & \textbf{FID} $\downarrow$ & \textbf{FVD} $\downarrow$ \\
|
||||
\midrule
|
||||
VideoGPT & 24.7 & 2880 & 2880 \\
|
||||
TGAN & 28.2 & -- & 1209 \\
|
||||
MoCoGAN & 46.3 & -- & 1729 \\
|
||||
NUWA & 49.3 & -- & 693 \\
|
||||
CogVideo & 50.5 & -- & 701 \\
|
||||
Make-A-Video & 82.8 & -- & 367 \\
|
||||
Emu Video & 89.3 & 9.4 & 323 \\
|
||||
\textbf{Zen-Video} & \textbf{96.8} & \textbf{8.7} & \textbf{312} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{UCF-101 Video Generation Benchmarks}
|
||||
\end{table}
|
||||
|
||||
\subsection{Video Quality and Consistency}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Metric} & \textbf{Zen-Video} & \textbf{Best Prior} \\
|
||||
\midrule
|
||||
CLIP-SIM (text-video alignment) & 0.312 & 0.281 \\
|
||||
Frame consistency (CLIP cosine) & 0.943 & 0.891 \\
|
||||
Motion smoothness (RAFT optical flow) & 0.921 & 0.847 \\
|
||||
Human preference rate & 71.3\% & 28.7\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Video Quality Metrics (EvalCrafter benchmark)}
|
||||
\end{table}
|
||||
|
||||
\subsection{Video Editing Quality}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Task} & \textbf{PSNR} $\uparrow$ & \textbf{SSIM} $\uparrow$ & \textbf{Temporal Cons.} \\
|
||||
\midrule
|
||||
Style transfer & 28.3 & 0.847 & 0.932 \\
|
||||
Object replacement & 31.2 & 0.891 & 0.958 \\
|
||||
Background removal & 33.7 & 0.923 & 0.971 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Video Editing Benchmark Results}
|
||||
\end{table}
|
||||
|
||||
\subsection{Frame Interpolation}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{PSNR} $\uparrow$ & \textbf{SSIM} $\uparrow$ & \textbf{LPIPS} $\downarrow$ \\
|
||||
\midrule
|
||||
RIFE & 35.6 & 0.956 & 0.041 \\
|
||||
IFRNet & 36.2 & 0.961 & 0.038 \\
|
||||
FILM & 36.9 & 0.965 & 0.034 \\
|
||||
\textbf{Zen-Video (interp. mode)} & \textbf{37.4} & \textbf{0.968} & \textbf{0.031} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Frame Interpolation on Vimeo-90K (4$\times$ upsampling)}
|
||||
\end{table}
|
||||
|
||||
\section{Generation Performance}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lrrl}
|
||||
\toprule
|
||||
\textbf{Mode} & \textbf{Resolution} & \textbf{Duration} & \textbf{Latency} \\
|
||||
\midrule
|
||||
Fast (20 steps) & 512$\times$288 & 10s & 12s on 4$\times$A100 \\
|
||||
Quality (50 steps) & 1024$\times$576 & 30s & 3.2 min on 4$\times$A100 \\
|
||||
HD (50 steps) & 1920$\times$1080 & 60s & 11 min on 4$\times$A100 \\
|
||||
Frame interpolation & any & any & 0.3s/frame on A10G \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen-Video Generation Performance}
|
||||
\end{table}
|
||||
|
||||
\section{Applications}
|
||||
|
||||
\subsection{AI Film Production}
|
||||
|
||||
Zen-Video is the rendering layer in the Hanzo AI film production pipeline, consuming shot plans
|
||||
from Zen-Director and producing video clips per shot. The complete pipeline (Zen-Director plan
|
||||
$\to$ Zen-Video render $\to$ Zen-Foley audio $\to$ Zen-Musician score) produces rough-cut
|
||||
footage from a written scene description in under 15 minutes.
|
||||
|
||||
\subsection{Marketing Content}
|
||||
|
||||
Brand teams use Zen-Video to generate product demonstration videos from text briefs, reducing
|
||||
video production timelines from weeks to hours.
|
||||
|
||||
\subsection{Game Cinematics}
|
||||
|
||||
Game studios use Zen-Video to generate in-engine cinematic sequences from narrative scripts,
|
||||
with camera motion controlled by the shot plan output of Zen-Director.
|
||||
|
||||
\section{Integration}
|
||||
|
||||
\begin{lstlisting}[language=Python, caption=Zen-Video Generation]
|
||||
from zen import ZenVideo
|
||||
|
||||
model = ZenVideo.from_pretrained("zenlm/zen-video-14b")
|
||||
|
||||
# Text-to-video
|
||||
video = model.generate(
|
||||
prompt="A lone wolf standing on a snowy mountain peak at dusk, "
|
||||
"cinematic lighting, epic wide shot",
|
||||
duration_sec=15,
|
||||
fps=24,
|
||||
resolution=(1920, 1080),
|
||||
camera_motion="slow_dolly_in",
|
||||
num_steps=50
|
||||
)
|
||||
video.save("wolf_mountain.mp4")
|
||||
|
||||
# Image-conditioned extension
|
||||
from PIL import Image
|
||||
first_frame = Image.open("reference.jpg")
|
||||
video = model.extend(
|
||||
image=first_frame,
|
||||
prompt="The scene slowly zooms out to reveal the full landscape",
|
||||
duration_sec=10
|
||||
)
|
||||
\end{lstlisting}
|
||||
|
||||
\section{Related Work}
|
||||
|
||||
VideoGPT \cite{yan2021videogpt} established autoregressive video generation in VQ-VAE space.
|
||||
NUWA \cite{wu2022nuwa} scaled text-to-video with multimodal conditioning. Make-A-Video
|
||||
\cite{singer2022make} and Imagen Video demonstrated the power of video diffusion. CogVideo
|
||||
\cite{hong2022cogvideo} applied large language models to video generation. Zen-Video advances
|
||||
the ST-DiT architecture with the TCM consistency module, achieving best-in-class FVD and
|
||||
temporal consistency metrics.
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
Zen-Video's ST-DiT architecture with the Temporal Coherence Module achieves UCF-101 IS 96.8,
|
||||
FID 8.7, and FVD 312---surpassing prior art across all metrics. The 14B parameter model
|
||||
generates 60-second 1080p clips and integrates natively with the Zen-Director and Zen-Foley
|
||||
systems for complete AI film production pipelines.
|
||||
|
||||
\begin{thebibliography}{10}
|
||||
\bibitem{yan2021videogpt} W. Yan et al., ``VideoGPT: Video Generation using VQ-VAE and Transformers,'' arXiv:2104.10157, 2021.
|
||||
\bibitem{wu2022nuwa} C. Wu et al., ``NUWA: Visual Synthesis Pre-training for Neural visual World crEation,'' ECCV, 2022.
|
||||
\bibitem{singer2022make} U. Singer et al., ``Make-A-Video: Text-to-Video Generation without Text-Video Data,'' ICLR, 2023.
|
||||
\bibitem{hong2022cogvideo} W. Hong et al., ``CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers,'' ICLR, 2023.
|
||||
\end{thebibliography}
|
||||
|
||||
\end{document}
|
||||
Binary file not shown.
-356
@@ -1,356 +0,0 @@
|
||||
\documentclass[11pt,a4paper]{article}
|
||||
\usepackage[utf8]{inputenc}
|
||||
\usepackage[T1]{fontenc}
|
||||
\usepackage{amsmath,amsfonts,amssymb}
|
||||
\usepackage{graphicx}
|
||||
\usepackage{hyperref}
|
||||
\usepackage{listings}
|
||||
\usepackage[dvipsnames]{xcolor}
|
||||
\usepackage{booktabs}
|
||||
\usepackage{float}
|
||||
\usepackage{geometry}
|
||||
\geometry{margin=1in}
|
||||
|
||||
\definecolor{zenblue}{RGB}{41,121,255}
|
||||
\definecolor{zengreen}{RGB}{52,199,89}
|
||||
\definecolor{codegray}{RGB}{245,245,245}
|
||||
|
||||
\hypersetup{colorlinks=true,linkcolor=zenblue,urlcolor=zenblue,citecolor=zenblue}
|
||||
|
||||
\lstset{
|
||||
backgroundcolor=\color{codegray},
|
||||
basicstyle=\ttfamily\small,
|
||||
breaklines=true,
|
||||
captionpos=b,
|
||||
frame=single,
|
||||
numbers=left,
|
||||
numberstyle=\tiny\color{gray}
|
||||
}
|
||||
|
||||
\title{
|
||||
\vspace{-2cm}
|
||||
\Large \textbf{Zen AI Model Family} \\
|
||||
\vspace{0.5cm}
|
||||
\Huge \textbf{Zen-Voyager} \\
|
||||
\vspace{0.3cm}
|
||||
\large Open-Ended Exploration, Novel Task Discovery, and Self-Directed Learning \\
|
||||
\vspace{0.5cm}
|
||||
\normalsize Technical Report v2025.02
|
||||
}
|
||||
|
||||
\author{
|
||||
Hanzo AI Research Team\thanks{research@hanzo.ai} \and
|
||||
Zoo Labs Foundation\thanks{foundation@zoo.ngo}
|
||||
}
|
||||
|
||||
\date{February 2025}
|
||||
|
||||
\begin{document}
|
||||
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
We present \textbf{Zen-Voyager}, a 14-billion parameter curiosity-driven exploration model
|
||||
designed for open-ended task discovery, self-directed learning, and autonomous skill acquisition
|
||||
in unstructured environments. Unlike task-conditioned agents, Zen-Voyager is trained to actively
|
||||
seek novel states, generate its own learning curricula, and accumulate reusable skills without
|
||||
requiring external task specification. The model achieves 78.3\% skill acquisition success on
|
||||
the MineDojo open-world benchmark, 85.2\% normalized score on the OpenAI Procgen suite in
|
||||
zero-shot transfer evaluation, and 92.1\% exploration coverage on our novel Open-World Coverage
|
||||
Benchmark (OWCB). Zen-Voyager introduces two key innovations: (1) the \textbf{Intrinsic
|
||||
Curiosity Transformer (ICT)}, an architecture that generates exploration policies from a
|
||||
learned model of its own epistemic uncertainty, and (2) the \textbf{Skill Library Protocol},
|
||||
a structured memory system for accumulating, indexing, and reusing discovered skills---enabling
|
||||
accelerating returns to exploration as the skill library grows.
|
||||
\end{abstract}
|
||||
|
||||
\tableofcontents
|
||||
\newpage
|
||||
|
||||
\section{Introduction}
|
||||
|
||||
Reinforcement learning agents typically require explicit task reward signals to learn useful
|
||||
behaviors. This limits their applicability to settings where reward functions are well-specified
|
||||
in advance---a strong constraint that rules out the vast majority of real-world learning
|
||||
problems, where what is worth learning is itself unknown.
|
||||
|
||||
Open-ended learning systems---agents that learn continuously in unstructured environments
|
||||
without prescribed tasks---represent a more general and robust approach to intelligence. The
|
||||
challenge is motivation: without external reward, what should an agent do?
|
||||
|
||||
Zen-Voyager answers this question through \textbf{intrinsic motivation}: the agent is rewarded
|
||||
for reducing uncertainty about its world model, discovering novel state-action-outcome patterns,
|
||||
and acquiring skills that increase its capability to explore further. This creates a
|
||||
bootstrapping dynamic: exploration yields new skills, which enable reaching previously
|
||||
inaccessible states, which yield more novel discoveries.
|
||||
|
||||
\subsection{Model Overview}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{ll}
|
||||
\toprule
|
||||
\textbf{Property} & \textbf{Value} \\
|
||||
\midrule
|
||||
Parameters & 14B \\
|
||||
Architecture & Zen MoDE 14B + Intrinsic Curiosity Transformer \\
|
||||
Context Length & 64K tokens (trajectory + skill library index) \\
|
||||
Skill Library & Up to 10,000 indexed skills \\
|
||||
World Model & 3B parameter predictive model \\
|
||||
Benchmarks & MineDojo, Procgen, OWCB \\
|
||||
Training & 500M environment steps (MineDojo + NetHack + Procgen) \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen-Voyager Model Specifications}
|
||||
\end{table}
|
||||
|
||||
\section{Architecture}
|
||||
|
||||
\subsection{Zen MoDE 14B Policy Backbone}
|
||||
|
||||
The policy backbone uses Zen MoDE at 14B scale: 36 transformer layers, 40 attention heads with
|
||||
grouped-query attention, and MoE FFN layers (8 experts, top-2 routing). The backbone processes
|
||||
a multimodal context window including:
|
||||
|
||||
\begin{itemize}
|
||||
\item \textbf{Observation history}: Last 32 visual observations encoded by ViT-L.
|
||||
\item \textbf{Action history}: Last 128 actions in the environment's action space.
|
||||
\item \textbf{Skill library summary}: A compressed index of acquired skills from the
|
||||
Skill Library Protocol (see Section~\ref{sec:skill-library}).
|
||||
\item \textbf{Intrinsic reward signal}: The ICT's uncertainty estimate for the current state.
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Intrinsic Curiosity Transformer (ICT)}
|
||||
|
||||
The ICT is a 2B parameter world model that predicts the next observation given the current
|
||||
observation and action:
|
||||
|
||||
\begin{equation}
|
||||
\hat{o}_{t+1} = f_\phi(o_t, a_t, h_{t-1})
|
||||
\end{equation}
|
||||
|
||||
where $h_{t-1}$ is the ICT's recurrent hidden state. The intrinsic reward is the prediction
|
||||
error:
|
||||
|
||||
\begin{equation}
|
||||
r^{\text{intr}}_t = \|o_{t+1} - \hat{o}_{t+1}\|_2^2
|
||||
\end{equation}
|
||||
|
||||
This classic formulation of curiosity-driven exploration \cite{pathak2017curiosity} is extended
|
||||
in Zen-Voyager with two key modifications:
|
||||
|
||||
\textbf{Epistemic uncertainty weighting}: The ICT also maintains a distribution over next
|
||||
observations (via Monte Carlo dropout), and intrinsic reward is weighted by epistemic uncertainty
|
||||
$\sigma^2_{\text{epist}}$, not just prediction error. This prevents exploitation of stochastic
|
||||
environment elements (which produce perpetual prediction error without genuine novelty):
|
||||
|
||||
\begin{equation}
|
||||
r^{\text{intr}}_t = \sigma^2_{\text{epist}}(o_t, a_t) \cdot \|o_{t+1} - \hat{o}_{t+1}\|_2^2
|
||||
\end{equation}
|
||||
|
||||
\textbf{Temporal novelty}: Intrinsic reward decays exponentially for states that have been
|
||||
visited previously (tracked via a count-based visitation model), preventing the agent from
|
||||
repeatedly exploiting the same novel-appearing state.
|
||||
|
||||
\subsection{Skill Library Protocol}
|
||||
\label{sec:skill-library}
|
||||
|
||||
When the agent successfully completes a novel goal (detected as a sustained high-value state
|
||||
transition), a new \textbf{skill entry} is created:
|
||||
|
||||
\begin{lstlisting}[language=Python, caption=Skill Library Entry Schema]
|
||||
{
|
||||
"skill_id": "s_0847",
|
||||
"name": "craft_iron_pickaxe",
|
||||
"description": "Gather iron ore, smelt into ingots, craft pickaxe",
|
||||
"preconditions": ["has_workbench", "has_furnace", "has_iron_ore >= 3"],
|
||||
"postconditions": ["has_iron_pickaxe"],
|
||||
"trajectory_summary": "...", # compressed action sequence
|
||||
"success_rate": 0.87,
|
||||
"discovered_at_step": 142831
|
||||
}
|
||||
\end{lstlisting}
|
||||
|
||||
The skill library is indexed with FAISS for efficient similarity search. During exploration,
|
||||
the agent can retrieve relevant skills by querying the library with the current state embedding,
|
||||
enabling compositional skill reuse.
|
||||
|
||||
Skills are periodically consolidated: similar skills are merged, and low-success-rate skills
|
||||
are pruned or replaced by improved variants discovered through continued exploration.
|
||||
|
||||
\section{Training}
|
||||
|
||||
\subsection{Training Environments}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lrrl}
|
||||
\toprule
|
||||
\textbf{Environment} & \textbf{Steps} & \textbf{Proportion} & \textbf{Purpose} \\
|
||||
\midrule
|
||||
MineDojo (Minecraft) & 250M & 50\% & Open-world exploration \\
|
||||
NetHack Learning Env. & 150M & 30\% & Procedural dungeon exploration \\
|
||||
OpenAI Procgen (16 games) & 100M & 20\% & Generalization test set \\
|
||||
\midrule
|
||||
\textbf{Total} & \textbf{500M} & 100\% & \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen-Voyager Training Environments}
|
||||
\end{table}
|
||||
|
||||
\subsection{Training Protocol}
|
||||
|
||||
\textbf{Phase 1 -- World model pretraining} (100M steps): The ICT world model is pretrained
|
||||
with supervised prediction loss on environment rollouts generated by a random policy.
|
||||
|
||||
\textbf{Phase 2 -- Curiosity-driven exploration} (300M steps): The policy backbone is trained
|
||||
with PPO using only the intrinsic reward signal. The world model is updated online throughout.
|
||||
|
||||
\textbf{Phase 3 -- Skill consolidation} (100M steps): The skill library is populated from
|
||||
Phase 2 trajectories. The policy is fine-tuned with an additional skill-reuse reward: positive
|
||||
reward for successfully invoking a previously discovered skill in a new context.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{llll}
|
||||
\toprule
|
||||
\textbf{Phase} & \textbf{Steps} & \textbf{Reward} & \textbf{LR} \\
|
||||
\midrule
|
||||
World model & 100M & Reconstruction loss & 3e-4 \\
|
||||
Exploration & 300M & Intrinsic only & 1e-4 \\
|
||||
Skill reuse & 100M & Intrinsic + skill reward & 5e-5 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen-Voyager Training Phases}
|
||||
\end{table}
|
||||
|
||||
\section{Evaluation}
|
||||
|
||||
\subsection{MineDojo Skill Acquisition}
|
||||
|
||||
MineDojo provides 1,628 diverse Minecraft tasks defined in natural language. We evaluate the
|
||||
fraction of tasks that Zen-Voyager can successfully execute after open-ended exploration
|
||||
(without task-specific training):
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{Task Success Rate} & \textbf{Skills Discovered} \\
|
||||
\midrule
|
||||
PPO (task-conditioned) & 42.1\% & N/A (not open-ended) \\
|
||||
DREAMER-V3 & 53.4\% & 312 \\
|
||||
VOYAGER (prior work) & 67.4\% & 1,247 \\
|
||||
\textbf{Zen-Voyager} & \textbf{78.3\%} & \textbf{3,841} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{MineDojo Open-Ended Exploration Results}
|
||||
\end{table}
|
||||
|
||||
\subsection{OpenAI Procgen Zero-Shot Transfer}
|
||||
|
||||
After open-ended exploration training (no Procgen task rewards), we evaluate zero-shot
|
||||
performance on Procgen's 16 games by providing natural language task descriptions only:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{Mean Score} & \textbf{Normalized} & \textbf{Training} \\
|
||||
\midrule
|
||||
PPO (200M steps, task-specific) & 7.3 & 73.0\% & Task-specific \\
|
||||
IMPALA (1B steps) & 8.1 & 81.0\% & Task-specific \\
|
||||
\textbf{Zen-Voyager (zero-shot)} & \textbf{8.52} & \textbf{85.2\%} & Open-ended only \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{OpenAI Procgen Zero-Shot Transfer}
|
||||
\end{table}
|
||||
|
||||
Zen-Voyager's zero-shot performance exceeds task-specific baselines, demonstrating that
|
||||
open-ended exploration yields broadly transferable skills.
|
||||
|
||||
\subsection{Open-World Coverage Benchmark (OWCB)}
|
||||
|
||||
We introduce OWCB: a procedurally generated 3D environment with 1M distinct reachable states.
|
||||
Coverage is measured as the fraction of states reached within a fixed step budget (10M steps):
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Exploration Strategy} & \textbf{Coverage (\%)} & \textbf{Steps to 50\% Coverage} \\
|
||||
\midrule
|
||||
Random policy & 31.4\% & Never \\
|
||||
Count-based bonus & 58.7\% & 8.2M \\
|
||||
ICM (Pathak et al.) & 71.3\% & 5.1M \\
|
||||
RIDE & 78.4\% & 4.3M \\
|
||||
\textbf{Zen-Voyager (ICT)} & \textbf{92.1\%} & \textbf{2.8M} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Open-World Coverage Benchmark (OWCB)}
|
||||
\end{table}
|
||||
|
||||
\subsection{Skill Library Growth}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Exploration Steps} & \textbf{Skills Discovered} & \textbf{Mean Success Rate} \\
|
||||
\midrule
|
||||
1M & 127 & 0.91 \\
|
||||
10M & 843 & 0.88 \\
|
||||
100M & 2,847 & 0.85 \\
|
||||
500M (final) & 3,841 & 0.84 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Skill Library Growth Over Training}
|
||||
\end{table}
|
||||
|
||||
The skill success rate remains high as the library grows, indicating the model learns to
|
||||
discover progressively more complex but reliable skills rather than accumulating low-quality
|
||||
entries.
|
||||
|
||||
\section{Applications}
|
||||
|
||||
\subsection{Autonomous Research Assistant}
|
||||
|
||||
Zen-Voyager's open-ended exploration capability is applied to scientific literature exploration:
|
||||
given a research area, the model autonomously explores related papers, discovers novel
|
||||
connections, and builds a structured knowledge graph without explicit task specification.
|
||||
|
||||
\subsection{Robotic Exploration}
|
||||
|
||||
Physical robots equipped with Zen-Voyager navigate novel indoor environments, building spatial
|
||||
maps and skill libraries that enable rapid adaptation when given specific tasks later.
|
||||
|
||||
\subsection{Game AI}
|
||||
|
||||
In game development, Zen-Voyager serves as an automated game tester that systematically explores
|
||||
game states, discovering edge cases and bugs through curiosity-driven exploration.
|
||||
|
||||
\section{Related Work}
|
||||
|
||||
VOYAGER \cite{wang2023voyager} pioneered LLM-powered open-ended Minecraft exploration with a
|
||||
skill library. DREAMER-V3 \cite{hafner2023mastering} demonstrated model-based RL in diverse
|
||||
environments. ICM \cite{pathak2017curiosity} established the curiosity-driven exploration
|
||||
paradigm. RIDE \cite{raileanu2020ride} introduced episodic novelty for improved exploration.
|
||||
Zen-Voyager advances this line by scaling to 14B parameters, introducing the epistemically
|
||||
calibrated ICT, and demonstrating zero-shot transfer that exceeds task-specific baselines.
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
Zen-Voyager demonstrates that open-ended exploration at 14B scale, with epistemically calibrated
|
||||
intrinsic motivation and a structured Skill Library Protocol, enables agents to discover skills
|
||||
and achieve generalization that surpasses task-specific trained baselines. The 78.3\% MineDojo
|
||||
success rate, 85.2\% Procgen zero-shot transfer, and 92.1\% OWCB coverage represent significant
|
||||
advances over prior open-ended learning systems.
|
||||
|
||||
\begin{thebibliography}{10}
|
||||
\bibitem{pathak2017curiosity} D. Pathak et al., ``Curiosity-driven Exploration by Self-supervised Prediction,'' ICML, 2017.
|
||||
\bibitem{wang2023voyager} G. Wang et al., ``Voyager: An Open-Ended Embodied Agent with Large Language Models,'' NeurIPS, 2023.
|
||||
\bibitem{hafner2023mastering} D. Hafner et al., ``Mastering Diverse Domains through World Models,'' arXiv:2301.04104, 2023.
|
||||
\bibitem{raileanu2020ride} R. Raileanu and T. Rocktaschel, ``RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments,'' ICLR, 2020.
|
||||
\end{thebibliography}
|
||||
|
||||
\end{document}
|
||||
Binary file not shown.
+173
-286
@@ -33,9 +33,9 @@
|
||||
\vspace{0.5cm}
|
||||
\Huge \textbf{Zen-World} \\
|
||||
\vspace{0.3cm}
|
||||
\large World Modeling, Physics Simulation, and Causal Reasoning at Scale \\
|
||||
\large Packaging Wan2.1 Text-to-Video Generation for the Zen Stack \\
|
||||
\vspace{0.5cm}
|
||||
\normalsize Technical Report v2025.02
|
||||
\normalsize Technical Report
|
||||
}
|
||||
|
||||
\author{
|
||||
@@ -43,51 +43,62 @@
|
||||
Zoo Labs Foundation\thanks{foundation@zoo.ngo}
|
||||
}
|
||||
|
||||
\date{February 2025}
|
||||
\date{June 2026}
|
||||
|
||||
\begin{document}
|
||||
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
We present \textbf{Zen-World}, a 32-billion parameter world model capable of physics simulation,
|
||||
causal reasoning, counterfactual prediction, and long-horizon environment modeling. World models
|
||||
have emerged as a critical capability for planning-based AI systems: a model that can accurately
|
||||
predict how the world will evolve in response to actions enables agents to plan in imagination
|
||||
rather than through costly real-world trial and error. Zen-World achieves 87.3\% on the PhysBench
|
||||
physical reasoning benchmark, 82.1\% on CausalQA (causal question answering over dynamic scenes),
|
||||
and 91.4\% physics consistency in a novel Physics Consistency Benchmark (PCB) we introduce.
|
||||
The model is built on the Zen MoDE architecture extended with a \textbf{Structured State Space
|
||||
World Representation (SSWR)}: a factored state representation that separates entities, their
|
||||
properties, and the physical and causal relations between them---enabling systematic
|
||||
generalization to novel physical configurations not seen during training.
|
||||
\textbf{Zen-World} is a packaging and integration of \textbf{Wan2.1-T2V-14B}, the open-source
|
||||
text-to-video diffusion model developed and released by Alibaba's Team Wan, into the Zen model
|
||||
stack. Zen-World is \emph{not} a from-scratch model: it redistributes the upstream
|
||||
\texttt{Wan-AI/Wan2.1-T2V-14B} weights and architecture under the permissive Apache~2.0 license,
|
||||
adding Zen-stack loaders, configuration, and a unified inference interface. The underlying model is
|
||||
a $\sim$14-billion-parameter Diffusion Transformer (DiT) trained with a flow-matching objective,
|
||||
paired with the Wan2.1 3D causal variational autoencoder (Wan-VAE) for spatio-temporal latent
|
||||
compression and the multilingual umT5 text encoder for prompt conditioning. It generates 480P and
|
||||
720P video from English or Chinese text prompts. This report describes the real architecture as
|
||||
released upstream, attributes it to its authors, and documents how it is integrated into Zen. All
|
||||
architectural and capability claims here are those of the upstream Wan2.1 release; we do not report
|
||||
any independent benchmark numbers of our own.
|
||||
\end{abstract}
|
||||
|
||||
\section*{Attribution and License}
|
||||
\addcontentsline{toc}{section}{Attribution and License}
|
||||
|
||||
\textbf{Upstream model.} Zen-World redistributes \texttt{Wan-AI/Wan2.1-T2V-14B}, part of the
|
||||
\emph{Wan} series of open large-scale video generative models from Alibaba's Team Wan
|
||||
\cite{wan2025report}.
|
||||
|
||||
\textbf{License.} The Wan2.1 models and source code are released under the
|
||||
\textbf{Apache License 2.0}. Zen-World preserves this license and the upstream copyright and
|
||||
attribution notices. No relicensing is claimed; redistribution is permitted because Apache~2.0 is a
|
||||
permissive license.
|
||||
|
||||
\textbf{What Zen adds.} Zen-World contributes \emph{packaging only}: Zen-stack model loaders, a
|
||||
consistent configuration and weights layout, and a unified text-to-video inference API consistent
|
||||
with the rest of the Zen family. It does not modify the model architecture or retrain the weights.
|
||||
|
||||
\tableofcontents
|
||||
\newpage
|
||||
|
||||
\section{Introduction}
|
||||
|
||||
A world model is a model that answers the question: \textit{given the current state of the world
|
||||
and an action, what happens next?} This deceptively simple formulation encompasses some of the
|
||||
deepest challenges in AI: physical intuition, causal reasoning, object permanence, and
|
||||
counterfactual prediction.
|
||||
Text-to-video generation synthesizes a short video clip from a natural-language description. Modern
|
||||
systems follow the diffusion-transformer paradigm: a transformer denoiser operates in a compressed
|
||||
video latent space produced by a video autoencoder, conditioned on text embeddings from a language
|
||||
encoder, and is trained to reverse a noising (or flow-matching) process \cite{wan2025report}.
|
||||
|
||||
World models serve as the foundation for model-based reinforcement learning: an agent with an
|
||||
accurate world model can plan multi-step action sequences entirely in mental simulation,
|
||||
dramatically improving sample efficiency and safety (by testing dangerous plans in imagination
|
||||
before executing them in reality).
|
||||
Zen-World packages one such system---\textbf{Wan2.1-T2V-14B}---into the Zen stack. Wan2.1 was
|
||||
released by Alibaba's Team Wan as an open, Apache-2.0-licensed family of video generative models
|
||||
\cite{wan2025report}. The 14B text-to-video variant is the model redistributed here. Zen-World
|
||||
provides Zen-native loading and a unified inference interface; the generative capability, weights,
|
||||
and architecture are entirely those of the upstream release.
|
||||
|
||||
Zen-World is designed as a general-purpose world model with three primary capabilities:
|
||||
|
||||
\begin{enumerate}
|
||||
\item \textbf{Physics simulation}: Predict the physical evolution of scenes---trajectories,
|
||||
collisions, fluid dynamics, rigid body motion---from visual observations.
|
||||
\item \textbf{Causal reasoning}: Answer questions about causal structure (``Why did X
|
||||
happen?'', ``Would Y have happened if Z had not occurred?'').
|
||||
\item \textbf{Counterfactual prediction}: Simulate counterfactual scenarios (``What would
|
||||
have happened if the red ball had been heavier?'').
|
||||
\end{enumerate}
|
||||
This report (1) attributes the model to its authors and states its license, (2) describes the real
|
||||
Wan2.1 architecture and its components, (3) clarifies the scope of Zen's integration, and
|
||||
(4) reports only capabilities and figures stated by the upstream release, with attribution.
|
||||
|
||||
\subsection{Model Overview}
|
||||
|
||||
@@ -97,302 +108,178 @@ Zen-World is designed as a general-purpose world model with three primary capabi
|
||||
\toprule
|
||||
\textbf{Property} & \textbf{Value} \\
|
||||
\midrule
|
||||
Parameters & 32B \\
|
||||
Architecture & Zen MoDE 32B + SSWR module \\
|
||||
Context Length & 128K tokens (supports 10+ minute scene sequences) \\
|
||||
Physics Timestep & Configurable 10ms--1s \\
|
||||
Simulation Domains & Rigid body, fluid, cloth, soft body, molecular \\
|
||||
Counterfactual Depth & Up to 50 intervention steps \\
|
||||
Training Data & 2.1B simulated frames, 180M real-world video frames \\
|
||||
Upstream model & \texttt{Wan-AI/Wan2.1-T2V-14B} (Alibaba Team Wan) \\
|
||||
Task & Text-to-video generation \\
|
||||
License & Apache 2.0 \\
|
||||
Architecture & Diffusion Transformer (DiT), flow matching \\
|
||||
Denoiser parameters & $\sim$14B \\
|
||||
Latent autoencoder & Wan2.1 3D causal VAE (Wan-VAE) \\
|
||||
Text encoder & umT5 (multilingual; English / Chinese prompts) \\
|
||||
Output resolutions & 480P ($832\times480$), 720P ($1280\times720$) \\
|
||||
Zen contribution & Packaging / integration into the Zen stack \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen-World Model Specifications}
|
||||
\caption{Zen-World summary. Architecture and capability values are those of upstream Wan2.1.}
|
||||
\end{table}
|
||||
|
||||
\section{Architecture}
|
||||
|
||||
\subsection{Structured State Space World Representation (SSWR)}
|
||||
Wan2.1 follows the mainstream diffusion-transformer design and is composed of three primary
|
||||
components \cite{wan2025report}: a 3D causal VAE (\emph{Wan-VAE}), a multilingual umT5 text encoder,
|
||||
and a flow-matching Diffusion Transformer (DiT) denoiser. Figure-level overview:
|
||||
|
||||
The key architectural innovation in Zen-World is the SSWR: a factored representation of world
|
||||
state that separates four orthogonal components:
|
||||
|
||||
\textbf{Entity set} $\mathcal{E} = \{e_1, \ldots, e_N\}$: The set of objects in the scene,
|
||||
each with a learned embedding encoding its identity, material, shape, and learned physical
|
||||
properties.
|
||||
|
||||
\textbf{State tensor} $S \in \mathbb{R}^{N \times D_s}$: The current state of each entity:
|
||||
position, velocity, angular velocity, deformation fields.
|
||||
|
||||
\textbf{Relation graph} $\mathcal{R} \subseteq \mathcal{E} \times \mathcal{E} \times R$: Typed
|
||||
edges encoding physical relations: contact, attachment, fluid interaction, gravitational coupling.
|
||||
|
||||
\textbf{Intervention set} $\mathcal{I}$: Pending actions or counterfactual interventions to be
|
||||
applied at the next simulation step.
|
||||
|
||||
The world model's forward pass operates over this structured representation:
|
||||
|
||||
\begin{align}
|
||||
S_{t+1} &= f_\phi(S_t, \mathcal{R}_t, \mathcal{I}_t) \\
|
||||
\mathcal{R}_{t+1} &= g_\phi(S_{t+1}, \mathcal{E})
|
||||
\end{align}
|
||||
|
||||
where $f_\phi$ is the dynamics model and $g_\phi$ updates the relation graph based on the new
|
||||
entity states (e.g., detecting new contacts).
|
||||
|
||||
The factored representation enables systematic generalization: new entities can be added to the
|
||||
simulation, and new relation types can be composed from known primitives.
|
||||
|
||||
\subsection{Dynamics Model Architecture}
|
||||
|
||||
The dynamics model $f_\phi$ is a message-passing graph neural network (GNN) operating over the
|
||||
SSWR graph, followed by 12 layers of the Zen MoDE transformer for global context integration:
|
||||
|
||||
\begin{equation}
|
||||
m_{ij} = \text{MLP}_e([h_i, h_j, r_{ij}, \Delta t])
|
||||
\end{equation}
|
||||
|
||||
\begin{equation}
|
||||
h_i' = h_i + \text{MLP}_v\left(h_i, \sum_{j \in \mathcal{N}(i)} m_{ij}\right)
|
||||
\end{equation}
|
||||
|
||||
The GNN processes local physical interactions (contacts, collisions) efficiently, while the
|
||||
transformer handles long-range dependencies (e.g., a distant force field affecting multiple
|
||||
objects).
|
||||
|
||||
\subsection{Causal Inference Module}
|
||||
|
||||
For causal and counterfactual reasoning, Zen-World maintains a \textbf{causal graph}
|
||||
$\mathcal{C}$ alongside the physical simulation. The causal graph tracks which entities and
|
||||
actions causally contributed to each observed outcome.
|
||||
|
||||
Counterfactual queries (``What if X had not happened?'') are answered by:
|
||||
\begin{enumerate}
|
||||
\item Identifying the intervention point in the causal graph.
|
||||
\item Rolling back the simulation to just before the intervention.
|
||||
\item Re-simulating with the modified initial conditions.
|
||||
\item Comparing the counterfactual trajectory to the observed trajectory.
|
||||
\item \textbf{Wan-VAE} encodes video into a compact spatio-temporal latent and decodes
|
||||
generated latents back to pixels.
|
||||
\item \textbf{umT5} encodes the text prompt into embeddings.
|
||||
\item \textbf{DiT} denoises the video latent, conditioned on the text embeddings via
|
||||
cross-attention, under a flow-matching objective.
|
||||
\end{enumerate}
|
||||
|
||||
This approach is grounded in Pearl's do-calculus \cite{pearl2009causality}:
|
||||
$P(\text{outcome} \mid \text{do}(\text{intervention}))$ is approximated by direct simulation.
|
||||
\subsection{Wan-VAE: 3D Causal Video Autoencoder}
|
||||
|
||||
\section{Training}
|
||||
Generating video directly in pixel space is prohibitively expensive, so Wan2.1 operates in a
|
||||
compressed latent space produced by \textbf{Wan-VAE}, a 3D \emph{causal} variational autoencoder
|
||||
designed for video \cite{wan2025report}. The VAE performs spatio-temporal compression with a
|
||||
temporal downsampling factor of $t=4$ and spatial downsampling factors of $h=w=8$, while preserving
|
||||
temporal causality so that motion remains smooth across frames. The latent has 16 channels (the
|
||||
DiT's input and output channel dimension; see Table~\ref{tab:dit}). The causal design lets the VAE
|
||||
encode and decode long videos without breaking temporal consistency.
|
||||
|
||||
\subsection{Dataset}
|
||||
\subsection{umT5 Text Encoder}
|
||||
|
||||
Prompts are encoded by \textbf{umT5}, a multilingual T5-family text encoder. The resulting text
|
||||
embeddings condition the DiT through cross-attention. The multilingual encoder is one reason Wan2.1
|
||||
accepts both English and Chinese prompts, and---per the upstream release---Wan2.1 is described as
|
||||
the first video model able to generate both Chinese and English text within the rendered video
|
||||
\cite{wan2025report}. The text sequence length used by the 14B configuration is 512 tokens
|
||||
(Table~\ref{tab:dit}).
|
||||
|
||||
\subsection{Diffusion Transformer (DiT)}
|
||||
|
||||
The denoiser is a Diffusion Transformer. It processes the noised video latent as a sequence of
|
||||
patch tokens through a stack of transformer blocks. Each block (the \emph{WanAttentionBlock})
|
||||
combines, per the upstream design, vision self-attention over the video tokens, text-to-vision
|
||||
cross-attention that injects the umT5 prompt embeddings, and a feed-forward layer, with timestep
|
||||
conditioning applied to modulate the block \cite{wan2025report}.
|
||||
|
||||
The exact configuration of the redistributed 14B text-to-video model, taken directly from the
|
||||
upstream model configuration, is given in Table~\ref{tab:dit}.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lrrl}
|
||||
\begin{tabular}{ll}
|
||||
\toprule
|
||||
\textbf{Source} & \textbf{Frames} & \textbf{Proportion} & \textbf{Domain} \\
|
||||
\textbf{Hyperparameter} & \textbf{Value (T2V-14B)} \\
|
||||
\midrule
|
||||
MuJoCo simulation & 800M & 38.1\% & Rigid body, robotics \\
|
||||
PhysX synthetic & 600M & 28.6\% & General physics \\
|
||||
Fluid simulation (SPH) & 300M & 14.3\% & Fluid dynamics \\
|
||||
Molecular dynamics & 200M & 9.5\% & Molecular simulation \\
|
||||
Real-world video (curated) & 180M & 8.6\% & Physical events \\
|
||||
Cloth/soft body sim. & 20M & 1.0\% & Deformable objects \\
|
||||
\midrule
|
||||
\textbf{Total} & \textbf{2,100,000,000} & 100\% & \\
|
||||
Model type & \texttt{t2v} (text-to-video) \\
|
||||
Model / embedding dimension (\texttt{dim}) & 5120 \\
|
||||
Transformer layers (\texttt{num\_layers}) & 40 \\
|
||||
Attention heads (\texttt{num\_heads}) & 40 \\
|
||||
Feed-forward dimension (\texttt{ffn\_dim}) & 13824 \\
|
||||
Latent input / output channels (\texttt{in\_dim} / \texttt{out\_dim}) & 16 \\
|
||||
Frequency embedding dimension (\texttt{freq\_dim}) & 256 \\
|
||||
Text sequence length (\texttt{text\_len}) & 512 \\
|
||||
LayerNorm epsilon (\texttt{eps}) & $1\times10^{-6}$ \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen-World Training Data Composition}
|
||||
\caption{Wan2.1-T2V-14B DiT configuration, from the upstream \texttt{config.json}.}
|
||||
\label{tab:dit}
|
||||
\end{table}
|
||||
|
||||
\subsection{Training Protocol}
|
||||
\subsection{Flow-Matching Objective}
|
||||
|
||||
\textbf{Stage 1 -- Physics encoder pretraining} (200K steps): The visual encoder + SSWR
|
||||
extraction network is trained to infer entity states and relation graphs from observations.
|
||||
Supervised by ground-truth simulation state (position, velocity, contact forces).
|
||||
Rather than a traditional discrete-noise diffusion schedule, Wan2.1 is trained with
|
||||
\textbf{flow matching} \cite{lipman2023flow, wan2025report}. Flow matching defines intermediate
|
||||
states by linear interpolation between a sample and noise and trains the network to predict the
|
||||
corresponding velocity (the time derivative of the interpolation path). Concretely, for a clean
|
||||
latent $x_1$ and noise $x_0 \sim \mathcal{N}(0, I)$, an intermediate latent is
|
||||
\begin{equation}
|
||||
x_t = (1 - t)\, x_0 + t\, x_1, \qquad t \in [0, 1],
|
||||
\end{equation}
|
||||
and the model $v_\theta$ is trained to regress the target velocity $x_1 - x_0$:
|
||||
\begin{equation}
|
||||
\mathcal{L}_{\text{FM}} = \mathbb{E}_{t,\, x_0,\, x_1,\, c}
|
||||
\left\lVert v_\theta(x_t, t, c) - (x_1 - x_0) \right\rVert_2^2,
|
||||
\end{equation}
|
||||
where $c$ denotes the umT5 text conditioning. At inference, video latents are generated by
|
||||
integrating the learned velocity field from noise, then decoded to pixels by Wan-VAE.
|
||||
|
||||
\textbf{Stage 2 -- Forward dynamics training} (300K steps): The dynamics model is trained to
|
||||
predict future states from current states and actions. Loss is a combination of state prediction
|
||||
MSE and a physics-consistency regularizer (energy conservation, momentum conservation).
|
||||
\section{Capabilities (as reported upstream)}
|
||||
|
||||
\textbf{Stage 3 -- Causal graph learning} (100K steps): The causal inference module is trained
|
||||
on simulation traces with annotated causal graphs, learning to identify and propagate causal
|
||||
attributions.
|
||||
The capabilities below are those stated by the upstream Wan2.1 release \cite{wan2025report}; they
|
||||
are reproduced here with attribution. Zen-World does not add or independently re-measure these
|
||||
figures.
|
||||
|
||||
\textbf{Stage 4 -- Counterfactual fine-tuning} (50K steps): Fine-tuned on paired
|
||||
(observed, counterfactual) simulation trajectories generated by systematic intervention sampling.
|
||||
\begin{itemize}
|
||||
\item \textbf{Output.} Generates short video clips at 480P ($832\times480$) and 720P
|
||||
($1280\times720$) from text prompts.
|
||||
\item \textbf{Multilingual prompts and in-video text.} Accepts English and Chinese prompts;
|
||||
described upstream as the first video model able to render both Chinese and English text in the
|
||||
generated video.
|
||||
\item \textbf{Model family.} The Wan2.1 release also includes a smaller T2V-1.3B model and
|
||||
image-to-video (I2V), first-last-frame (FLF2V), and video-editing (VACE) variants; Zen-World
|
||||
redistributes the T2V-14B variant.
|
||||
\item \textbf{Efficiency (sibling model).} The upstream release reports that the smaller
|
||||
T2V-1.3B model can generate a 5-second 480P clip using about 8.19~GB of VRAM, enabling
|
||||
consumer-GPU use; the 14B model redistributed here is correspondingly heavier.
|
||||
\item \textbf{Evaluation.} The upstream report states the 14B model was evaluated on an internal
|
||||
suite of 1{,}035 prompts across 14 dimensions and that it is competitive with leading
|
||||
open-source and commercial systems. We reproduce this claim as reported and do not restate
|
||||
specific scores here.
|
||||
\end{itemize}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lllll}
|
||||
\toprule
|
||||
\textbf{Stage} & \textbf{Steps} & \textbf{Batch} & \textbf{LR} & \textbf{Hardware} \\
|
||||
\midrule
|
||||
Physics encoder & 200K & 256 & 3e-4 & 64$\times$A100 \\
|
||||
Forward dynamics & 300K & 128 & 1e-4 & 128$\times$A100 \\
|
||||
Causal learning & 100K & 64 & 5e-5 & 32$\times$A100 \\
|
||||
Counterfactual FT & 50K & 32 & 2e-5 & 16$\times$A100 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen-World Training Configuration}
|
||||
\end{table}
|
||||
\section{Integration into the Zen Stack}
|
||||
|
||||
\section{Evaluation}
|
||||
Zen-World's contribution is integration, not modeling. Specifically, Zen-World provides:
|
||||
\begin{itemize}
|
||||
\item Zen-stack loading of the upstream Wan2.1-T2V-14B weights (DiT, Wan-VAE, umT5).
|
||||
\item A consistent on-disk configuration and weights layout aligned with the Zen family.
|
||||
\item A unified text-to-video inference interface consistent with other Zen models.
|
||||
\end{itemize}
|
||||
No architectural changes are made and the weights are not retrained. The example below illustrates
|
||||
the intended Zen-stack interface for text-to-video generation.
|
||||
|
||||
\subsection{PhysBench}
|
||||
|
||||
PhysBench evaluates physical reasoning across 5 categories: projectile motion, fluid dynamics,
|
||||
rigid body collisions, elastic deformation, and gravitational interactions.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lccccc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{Projectile} & \textbf{Fluid} & \textbf{Collision} & \textbf{Elastic} & \textbf{Mean} \\
|
||||
\midrule
|
||||
GPT-4o (text) & 73.2 & 61.4 & 78.3 & 64.1 & 69.3 \\
|
||||
Claude 3.5 Sonnet & 76.8 & 65.2 & 81.4 & 68.3 & 72.9 \\
|
||||
DreamerV3 & 68.3 & 71.4 & 74.2 & 72.1 & 71.5 \\
|
||||
PHYRE & 79.4 & -- & 82.3 & -- & -- \\
|
||||
\textbf{Zen-World} & \textbf{89.2} & \textbf{83.4} & \textbf{91.7} & \textbf{84.8} & \textbf{87.3} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{PhysBench Results (\%, higher is better)}
|
||||
\end{table}
|
||||
|
||||
\subsection{CausalQA}
|
||||
|
||||
CausalQA presents video clips of physical events with questions about causal structure:
|
||||
``What caused the glass to fall?'', ``Would the ball have reached the target if the ramp
|
||||
had been steeper?''
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{Causal ID} & \textbf{Counterfactual} & \textbf{Intervention} & \textbf{Mean} \\
|
||||
\midrule
|
||||
GPT-4o + video & 74.3 & 61.2 & 68.4 & 68.0 \\
|
||||
Gemini 1.5 Pro & 76.1 & 63.4 & 70.2 & 69.9 \\
|
||||
CLEVRER-Base & 78.4 & 67.2 & 72.1 & 72.6 \\
|
||||
\textbf{Zen-World} & \textbf{84.2} & \textbf{79.8} & \textbf{82.3} & \textbf{82.1} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{CausalQA Results (\%, higher is better)}
|
||||
\end{table}
|
||||
|
||||
The counterfactual category shows the largest gap (79.8\% vs. 67.2\%), confirming that
|
||||
Zen-World's explicit causal graph and simulation rollback provide a meaningful advantage over
|
||||
purely language-based approaches.
|
||||
|
||||
\subsection{Physics Consistency Benchmark (PCB)}
|
||||
|
||||
PCB is a novel benchmark we introduce: given a simulated trajectory, does the model's prediction
|
||||
satisfy physical conservation laws? We test energy conservation, momentum conservation, and
|
||||
angular momentum conservation in isolated systems.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{Energy} & \textbf{Momentum} & \textbf{Ang. Mom.} & \textbf{Mean} \\
|
||||
\midrule
|
||||
Video prediction LSTM & 67.3 & 71.2 & 63.4 & 67.3 \\
|
||||
DreamerV3 & 78.4 & 82.1 & 74.3 & 78.3 \\
|
||||
\textbf{Zen-World} & \textbf{92.1} & \textbf{93.4} & \textbf{88.7} & \textbf{91.4} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Physics Consistency Benchmark (\% of predictions satisfying conservation laws)}
|
||||
\end{table}
|
||||
|
||||
\subsection{Long-Horizon Prediction}
|
||||
|
||||
We evaluate simulation accuracy as a function of prediction horizon (seconds ahead):
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Horizon} & \textbf{Zen-World} & \textbf{DreamerV3} & \textbf{RSSM} \\
|
||||
\midrule
|
||||
0.5s & 94.2\% & 91.3\% & 87.4\% \\
|
||||
2.0s & 88.7\% & 79.2\% & 71.3\% \\
|
||||
10.0s & 76.4\% & 58.1\% & 44.2\% \\
|
||||
60.0s & 61.3\% & 31.4\% & 18.7\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Long-Horizon Prediction Accuracy (\% states within 5\% error)}
|
||||
\end{table}
|
||||
|
||||
\section{Applications}
|
||||
|
||||
\subsection{Model-Based Planning for Robotics}
|
||||
|
||||
Zen-World serves as the environment model in the Hanzo robotics planning stack. Given a robot's
|
||||
sensor observations, the model maintains an internal simulation of the scene and allows the
|
||||
motion planner to evaluate candidate action sequences entirely in simulation before committing
|
||||
to execution, reducing physical trial attempts by 84\%.
|
||||
|
||||
\subsection{Scientific Simulation}
|
||||
|
||||
Zen-World's molecular dynamics domain enables protein folding stability prediction and drug
|
||||
binding affinity estimation by simulating molecular-scale physical interactions.
|
||||
|
||||
\subsection{Autonomous Driving Safety}
|
||||
|
||||
In autonomous driving contexts, Zen-World is used to simulate counterfactual accident scenarios:
|
||||
``Would the collision have been avoided if the vehicle had braked 0.5s earlier?'' This enables
|
||||
post-incident analysis and safety system certification without physical testing.
|
||||
|
||||
\section{Integration}
|
||||
|
||||
\begin{lstlisting}[language=Python, caption=Zen-World Physics Simulation]
|
||||
\begin{lstlisting}[language=Python, caption=Zen-World text-to-video (packaging of Wan2.1-T2V-14B)]
|
||||
from zen import ZenWorld
|
||||
import numpy as np
|
||||
|
||||
model = ZenWorld.from_pretrained("zenlm/zen-world-32b")
|
||||
# Loads the redistributed Wan2.1-T2V-14B weights (DiT + Wan-VAE + umT5).
|
||||
model = ZenWorld.from_pretrained("zenlm/zen-world")
|
||||
|
||||
# Parse scene from image
|
||||
scene = model.parse_scene("scene.jpg")
|
||||
print(f"Detected {len(scene.entities)} objects")
|
||||
|
||||
# Simulate 5 seconds forward
|
||||
trajectory = model.simulate(
|
||||
scene=scene,
|
||||
action={"push_object": "ball_1", "force": [2.0, 0.0, 0.0]},
|
||||
duration_sec=5.0,
|
||||
timestep_ms=50
|
||||
# Text-to-video generation.
|
||||
video = model.generate(
|
||||
prompt="A red panda walking through a snowy bamboo forest, cinematic lighting",
|
||||
resolution="720p", # 1280x720; also "480p" (832x480)
|
||||
num_frames=81,
|
||||
fps=16,
|
||||
)
|
||||
|
||||
# Counterfactual query
|
||||
cf_trajectory = model.counterfactual(
|
||||
trajectory=trajectory,
|
||||
intervention={"at_time": 0.5, "change_mass": {"entity": "ball_1", "new_mass": 5.0}}
|
||||
)
|
||||
|
||||
print(f"Original final position: {trajectory.final_state.entities['ball_1'].position}")
|
||||
print(f"Counterfactual final: {cf_trajectory.final_state.entities['ball_1'].position}")
|
||||
video.save("output.mp4")
|
||||
\end{lstlisting}
|
||||
|
||||
\section{Related Work}
|
||||
|
||||
RSSM \cite{hafner2019learning} introduced latent dynamics models for model-based RL.
|
||||
DreamerV3 \cite{hafner2023mastering} demonstrated general applicability across diverse domains.
|
||||
PHYRE \cite{bakhtin2019phyre} established physical reasoning benchmarks. CLEVRER
|
||||
\cite{yi2019clevrer} introduced causal and counterfactual video question answering.
|
||||
Zen-World advances this line by combining structured SSWR representations with 32B Zen MoDE
|
||||
scale, achieving state-of-the-art results across physical reasoning, causal QA, and physics
|
||||
consistency metrics.
|
||||
Zen-World is a redistribution of Wan2.1 \cite{wan2025report}, which belongs to the broader line of
|
||||
diffusion-transformer video generators. The diffusion-transformer (DiT) backbone derives from the
|
||||
scalable transformer denoiser of Peebles and Xie \cite{peebles2023dit}. Flow matching, the training
|
||||
objective Wan2.1 uses in place of a discrete noise schedule, was introduced by Lipman et al.
|
||||
\cite{lipman2023flow}. The Wan technical report \cite{wan2025report} details the Wan-VAE 3D causal
|
||||
autoencoder, the umT5 conditioning, and the model family.
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
Zen-World demonstrates that world modeling at 32B scale with structured state representations
|
||||
(SSWR), explicit causal graph tracking, and simulation-based counterfactual reasoning produces
|
||||
substantially better physical understanding than purely learned end-to-end predictors.
|
||||
The 87.3\% PhysBench, 82.1\% CausalQA, and 91.4\% PCB results represent the current state
|
||||
of the art across all three evaluation dimensions, with particularly strong gains in
|
||||
counterfactual prediction---the capability most critical for planning and scientific analysis.
|
||||
Zen-World is the Zen stack's packaging of \textbf{Wan2.1-T2V-14B}, the Apache-2.0 text-to-video
|
||||
diffusion model released by Alibaba's Team Wan. The real system is a $\sim$14B Diffusion Transformer
|
||||
trained with flow matching, operating in the latent space of the Wan2.1 3D causal VAE and
|
||||
conditioned by the multilingual umT5 text encoder, producing 480P and 720P video from English or
|
||||
Chinese prompts. Zen's contribution is integration: loaders, configuration, and a unified inference
|
||||
interface. All architecture and capability statements in this report are attributed to the upstream
|
||||
Wan2.1 release, and no independent benchmark numbers are claimed.
|
||||
|
||||
\begin{thebibliography}{10}
|
||||
\bibitem{pearl2009causality} J. Pearl, ``Causality: Models, Reasoning, and Inference,'' 2nd ed., Cambridge University Press, 2009.
|
||||
\bibitem{hafner2019learning} D. Hafner et al., ``Learning Latent Dynamics for Planning from Pixels,'' ICML, 2019.
|
||||
\bibitem{hafner2023mastering} D. Hafner et al., ``Mastering Diverse Domains through World Models,'' arXiv:2301.04104, 2023.
|
||||
\bibitem{bakhtin2019phyre} A. Bakhtin et al., ``PHYRE: A New Evaluation Framework for Physical Reasoning,'' NeurIPS, 2019.
|
||||
\bibitem{yi2019clevrer} K. Yi et al., ``CLEVRER: Collision Events for Video Representation and Reasoning,'' ICLR, 2020.
|
||||
\bibitem{wan2025report} Team Wan (Alibaba), ``Wan: Open and Advanced Large-Scale Video Generative Models,'' arXiv:2503.20314, 2025. Models and code: \url{https://github.com/Wan-Video/Wan2.1}; weights: \url{https://huggingface.co/Wan-AI/Wan2.1-T2V-14B} (Apache~2.0).
|
||||
\bibitem{lipman2023flow} Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le, ``Flow Matching for Generative Modeling,'' ICLR, 2023. arXiv:2210.02747.
|
||||
\bibitem{peebles2023dit} W. Peebles and S. Xie, ``Scalable Diffusion Models with Transformers,'' ICCV, 2023. arXiv:2212.09748.
|
||||
\end{thebibliography}
|
||||
|
||||
\end{document}
|
||||
|
||||
Reference in New Issue
Block a user