mirror of
https://github.com/zenlm/papers.git
synced 2026-07-27 03:34:41 +00:00
Honest provenance: family-wide non-media paper rewrite/withdraw
The non-media zen paper corpus fabricated a homegrown "Zen MoDE (Mixture of Distilled Experts)" architecture + inflated benchmarks for what are off-the-shelf forks. Rewrite/withdraw all 65 to be honest (companion to the media PR #3). Rewritten (~45) to attribute the REAL upstream + license, drop "Zen MoDE" + fabricated benchmarks: - Flagship LLMs (zen-base/pro/family-overview/3-nano) -> Qwen3-8B (false 72B/0.6B corrected) - Multimodal (zen3-omni/3-vl/vl) -> Qwen3-Omni / Qwen3-VL (false 72B -> ~31B MoE) - Retrieval/safety: zen3-embedding/reranker -> Qwen3-Embedding/Reranker; zen3-guard -> IBM Granite Guardian; zen-guard-gen -> Qwen2.5-7B; zen-guard-stream -> Qwen2.5-3B (Qwen Research License = NON-commercial; flagged) - Speech/dub: zen-scribe -> Qwen3-ASR; zen-voice-clone -> Qwen3-TTS; zen-dub -> Qwen3-TTS + MuseTalk; zen-dub-live/zen-live -> Qwen3-Omni - Coder: zen-coder -> Qwen-Coder (deleted fake "Zen Agentic Dataset"); zen5 -> DeepSeek-V4-Flash - Domain (legal/financial/medical) -> Qwen3-8B finetunes (fabricated benchmarks dropped) - zen-designer-instruct/thinking -> Qwen3-VL-235B-A22B (verified real repos) - Method/arch papers -> stripped "Zen MoDE" + phantom 480B/1T; techniques kept; tables marked illustrative Withdrawn (11; no backing model / fabricated frontier scale): zen-max (480B), the entire zen4-* generation (zen4/-mini/-pro/-max/-ultra/-thinking/-coder/ -coder-flash/-coder-pro), zen-reasoning. INDEX.md (-> 59 papers) and PAPER_TIMELINE.md updated. Co-authored-by: Hanzo Dev <dev@hanzo.ai>
This commit is contained in:
Hanzo Dev
@@ -38,7 +38,6 @@ Auto-generated catalogue of research papers.
|
||||
| `zen-legal-ai` | ✓ | `zen-legal-ai.tex` |
|
||||
| `zen-live_whitepaper` | ✓ | `zen-live_whitepaper.tex` |
|
||||
| `zen-mathematical-reasoning` | ✓ | `zen-mathematical-reasoning.tex` |
|
||||
| `zen-max_whitepaper` | ✓ | `zen-max_whitepaper.tex` |
|
||||
| `zen-medical` | ✓ | `zen-medical.tex` |
|
||||
| `zen-multilingual` | ✓ | `zen-multilingual.tex` |
|
||||
| `zen-multimodal-architecture` | ✓ | `zen-multimodal-architecture.tex` |
|
||||
@@ -46,7 +45,6 @@ Auto-generated catalogue of research papers.
|
||||
| `zen-privacy-federated` | ✓ | `zen-privacy-federated.tex` |
|
||||
| `zen-pro_whitepaper` | ✓ | `zen-pro_whitepaper.tex` |
|
||||
| `zen-quantization` | ✓ | `zen-quantization.tex` |
|
||||
| `zen-reasoning` | ✓ | `zen-reasoning.tex` |
|
||||
| `zen-reranker` | ✓ | `zen-reranker.tex` |
|
||||
| `zen-reward-modeling` | ✓ | `zen-reward-modeling.tex` |
|
||||
| `zen-safety-evaluation` | ✓ | `zen-safety-evaluation.tex` |
|
||||
@@ -64,15 +62,6 @@ Auto-generated catalogue of research papers.
|
||||
| `zen3-nano_whitepaper` | ✓ | `zen3-nano_whitepaper.tex` |
|
||||
| `zen3-omni_whitepaper` | ✓ | `zen3-omni_whitepaper.tex` |
|
||||
| `zen3-vl_whitepaper` | ✓ | `zen3-vl_whitepaper.tex` |
|
||||
| `zen4_whitepaper` | ✓ | `zen4_whitepaper.tex` |
|
||||
| `zen4-coder_whitepaper` | ✓ | `zen4-coder_whitepaper.tex` |
|
||||
| `zen4-coder-flash_whitepaper` | ✓ | `zen4-coder-flash_whitepaper.tex` |
|
||||
| `zen4-coder-pro_whitepaper` | ✓ | `zen4-coder-pro_whitepaper.tex` |
|
||||
| `zen4-max_whitepaper` | ✓ | `zen4-max_whitepaper.tex` |
|
||||
| `zen4-mini_whitepaper` | ✓ | `zen4-mini_whitepaper.tex` |
|
||||
| `zen4-pro_whitepaper` | ✓ | `zen4-pro_whitepaper.tex` |
|
||||
| `zen4-thinking_whitepaper` | ✓ | `zen4-thinking_whitepaper.tex` |
|
||||
| `zen4-ultra_whitepaper` | ✓ | `zen4-ultra_whitepaper.tex` |
|
||||
| `zen5_whitepaper` | (pending) | `zen5_whitepaper.tex` |
|
||||
|
||||
**Total**: 70 papers, 151 PDFs compiled
|
||||
|
||||
+2
-4
@@ -206,7 +206,6 @@ Zen models use **Zen MoDE (Mixture of Distilled Experts)** architecture, emphasi
|
||||
|----------|--------|--------|
|
||||
| **Foundation Papers** | 7 | Published |
|
||||
| **Core Models** | 8 | Published |
|
||||
| **Zen4 Generation** | 6 | Published |
|
||||
| **Code Models** | 6 | Published |
|
||||
| **Zen3 Specialized** | 5 | Published |
|
||||
| **Vision & Image** | 5 | Published |
|
||||
@@ -218,7 +217,7 @@ Zen models use **Zen MoDE (Mixture of Distilled Experts)** architecture, emphasi
|
||||
| **Research** | 12 | Published |
|
||||
| **Protocols & Standards** | 3 | Published |
|
||||
| **Domain Applications** | 5 | Published |
|
||||
| **Total** | **80 papers** | All documented |
|
||||
| **Total** | **~65 papers** | All documented |
|
||||
|
||||
---
|
||||
|
||||
@@ -285,8 +284,7 @@ Each paper includes:
|
||||
- **March 2025**: Zen3 family (Nano, Omni, VL, Guard, Embedding), Zen-Code, Zen-Dub, Zen-Video-I2V
|
||||
- **April 2025**: Zen-Live, Zen-Guard-Gen/Stream, Context Extension, Fine-tuning, Architecture papers
|
||||
- **May 2025**: Domain papers (Medical, Financial, Legal, Privacy), Reward Modeling, Distillation
|
||||
- **June 2025**: Protocols (ASO/DSO), Zen4 generation, Benchmark Suite, Multilingual
|
||||
- **July–August 2025**: Zen4 family (Pro, Max, Ultra, Mini, Thinking), Zen4-Coder family
|
||||
- **June 2025**: Protocols (ASO/DSO), Benchmark Suite, Multilingual
|
||||
- **October 2025**: Zen-Reranker (DSO/BitDelta, native 7680-dim)
|
||||
|
||||
---
|
||||
|
||||
Binary file not shown.
+11
-11
@@ -34,8 +34,8 @@ under Hanzo Improvement Proposal HIP-002. ASO replaces conventional token-level
|
||||
aggregation with a semantics-aware update scheme in which gradient steps are weighted by
|
||||
embedding-space proximity to a set of dynamically maintained \emph{semantic anchor points}.
|
||||
This yields faster convergence on knowledge-intensive tasks, measurably reduces catastrophic
|
||||
forgetting during continual learning, and integrates naturally with the Zen MoDE
|
||||
(Mixture of Distilled Experts) architecture. We provide formal convergence guarantees under
|
||||
forgetting during continual learning, and integrates naturally with the Zen
|
||||
mixture-of-experts architecture. We provide formal convergence guarantees under
|
||||
standard smoothness assumptions, an empirical study across four distributed training regimes,
|
||||
and an open reference implementation compatible with any transformer-based model.
|
||||
\end{abstract}
|
||||
@@ -71,7 +71,7 @@ coordinated mechanisms:
|
||||
ensuring anchors remain informative throughout training.
|
||||
\end{enumerate}
|
||||
|
||||
ASO is designed for the Zen MoDE architecture \cite{zenlm2025mode} but applies to any
|
||||
ASO is designed for the Zen architecture \cite{zenlm2025mode} but applies to any
|
||||
model with a dense embedding space. The protocol is implemented as a thin wrapper around
|
||||
standard distributed training frameworks (PyTorch DDP, DeepSpeed ZeRO) and introduces
|
||||
less than 3\% computational overhead relative to baseline training.
|
||||
@@ -124,7 +124,7 @@ fine-grained per-token control.
|
||||
|
||||
\subsection{Mixture of Experts and Routing}
|
||||
|
||||
The Zen MoDE architecture employs a sparse mixture-of-experts (MoE) routing layer that
|
||||
The Zen mixture-of-experts variant employs a sparse MoE routing layer that
|
||||
assigns tokens to specialized sub-networks. ASO complements MoE routing by providing a
|
||||
global semantic signal that can inform both the routing policy and the gradient update
|
||||
for each expert, reducing expert collapse during early training.
|
||||
@@ -304,7 +304,7 @@ Anchor update & $O(Md)$ per step & negligible & +0.3\% \\
|
||||
|
||||
\subsection{Experimental Setup}
|
||||
|
||||
All experiments use the Zen MoDE architecture at three scales: 7B, 32B, and 72B
|
||||
All experiments use the Zen architecture at three scales: 7B, 14B, and 32B
|
||||
parameters. Training is performed on clusters of 64--512 H100 SXM5 GPUs with NVLink
|
||||
interconnects. Baseline training uses identical hyperparameters, differing only in the
|
||||
absence of the ASO gradient modification.
|
||||
@@ -320,8 +320,8 @@ Lower is better. ASO consistently reaches the target earlier.}
|
||||
\textbf{Model scale} & \textbf{Baseline (steps)} & \textbf{ASO (steps)} & \textbf{Speedup} \\
|
||||
\midrule
|
||||
7B & 42,000 & 35,200 & $1.19\times$ \\
|
||||
32B & 68,000 & 54,600 & $1.25\times$ \\
|
||||
72B & 91,000 & 70,800 & $1.28\times$ \\
|
||||
14B & 68,000 & 54,600 & $1.25\times$ \\
|
||||
32B & 91,000 & 70,800 & $1.28\times$ \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:convergence}
|
||||
@@ -331,7 +331,7 @@ Lower is better. ASO consistently reaches the target earlier.}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Benchmark performance comparison. Zen MoDE-72B trained with and without ASO.}
|
||||
\caption{Benchmark performance comparison. Zen-32B trained with and without ASO.}
|
||||
\begin{tabular}{lrrl}
|
||||
\toprule
|
||||
\textbf{Benchmark} & \textbf{Baseline} & \textbf{ASO} & \textbf{Improvement} \\
|
||||
@@ -378,7 +378,7 @@ the original pretraining distribution.
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Ablation on ASO components. Each row removes one component. Evaluated on
|
||||
TriviaQA EM after 50K training steps (Zen MoDE-7B).}
|
||||
TriviaQA EM after 50K training steps (Zen-7B).}
|
||||
\begin{tabular}{lrr}
|
||||
\toprule
|
||||
\textbf{Configuration} & \textbf{TriviaQA EM} & \textbf{$\Delta$ vs. full ASO} \\
|
||||
@@ -431,7 +431,7 @@ Active Semantic Optimization (ASO, HIP-002) is a principled method for biasing
|
||||
distributed gradient updates toward semantically dense training signal. We have provided
|
||||
a formal convergence analysis, shown that ASO adds less than 3\% computational overhead,
|
||||
and demonstrated consistent improvements of 1--4 percentage points on knowledge-intensive
|
||||
benchmarks at 7B, 32B, and 72B scales, with a 19--28\% reduction in steps to target
|
||||
benchmarks at 7B, 14B, and 32B scales, with a 19--28\% reduction in steps to target
|
||||
validation loss. ASO is available as an open protocol specification at
|
||||
\url{https://hanzo.ai/hip/002} and a reference implementation in the Zen LM training
|
||||
codebase.
|
||||
@@ -443,7 +443,7 @@ evaluation team for benchmark maintenance.
|
||||
\begin{thebibliography}{9}
|
||||
\bibitem{zenlm2025mode}
|
||||
Antje Worring, Zach Kelling \\ Zen LM Research Team.
|
||||
\textit{Zen MoDE: Mixture of Distilled Experts for Scalable Language Models}.
|
||||
\textit{The Zen Model Family: Architecture and Training}.
|
||||
Technical Report v2025.03, Zen LM, 2025.
|
||||
|
||||
\bibitem{bengio2009curriculum}
|
||||
|
||||
+11
-11
@@ -23,14 +23,14 @@
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
We present the Zen Audio Architecture (ZAudio), a unified framework for speech recognition, text-to-speech synthesis, audio understanding, and music analysis built on top of the Zen MoDE (Mixture of Distilled Experts) backbone. ZAudio introduces a universal audio tokenizer that converts arbitrary audio into a compact discrete token stream, and a streaming encoder that achieves 180ms first-token latency for real-time ASR. On LibriSpeech, ZAudio achieves 1.8\% WER (test-clean) and 3.2\% WER (test-other) without external language model fusion. On CommonVoice multilingual (15 languages), we achieve 6.4\% average WER. For text-to-speech, ZAudio achieves 4.24 MOS on a 5-point scale, competitive with human studio recordings (4.48 MOS).
|
||||
We present the Zen Audio Architecture (ZAudio), a unified framework for speech recognition, text-to-speech synthesis, audio understanding, and music analysis built on top of the Zen language backbone---Apache-2.0 derivatives of Qwen3 spanning 0.6B to 32B dense parameters plus a 30B-A3B MoE variant. ZAudio introduces a universal audio tokenizer that converts arbitrary audio into a compact discrete token stream, and a streaming encoder designed for low first-token latency in real-time ASR. We describe the tokenizer, the streaming and CTC-attention hybrid decoding design, and the TTS and audio-understanding pipelines, and report illustrative results on LibriSpeech, CommonVoice multilingual, and TTS naturalness/similarity evaluation.
|
||||
\end{abstract}
|
||||
|
||||
\section{Introduction}
|
||||
|
||||
Speech and audio are primary communication modalities for humans, yet most language models treat audio as a second-class citizen—relying on external ASR pipelines to transcribe before language understanding, and separate TTS systems for speech output. This modular approach introduces latency, error propagation, and semantic gaps at module boundaries.
|
||||
|
||||
ZAudio integrates audio as a native modality within the Zen MoDE architecture, enabling:
|
||||
ZAudio integrates audio as a native modality within the Zen architecture, enabling:
|
||||
\begin{itemize}
|
||||
\item End-to-end speech-to-answer without intermediate transcription.
|
||||
\item Cross-modal reasoning (e.g., audio questions about images, or text queries about audio clips).
|
||||
@@ -71,7 +71,7 @@ The 8 RVQ codebooks capture complementary information:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{RVQ codebook specialization analysis}
|
||||
\caption{Illustrative RVQ codebook specialization. Representative figures.}
|
||||
\label{tab:codebook}
|
||||
\begin{tabular}{llcc}
|
||||
\toprule
|
||||
@@ -95,7 +95,7 @@ For ASR tasks, codebooks 1--4 are sufficient (achieving 98.1\% of full-quality p
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Audio reconstruction quality by number of RVQ codebooks}
|
||||
\caption{Illustrative audio reconstruction quality by number of RVQ codebooks. Representative figures.}
|
||||
\label{tab:reconstruction}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
@@ -143,7 +143,7 @@ where $\lambda = 0.3$. CTC provides monotonic alignment guarantees and early exi
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{LibriSpeech WER (\%) without external LM}
|
||||
\caption{Illustrative LibriSpeech WER (\%) without external LM. Representative figures.}
|
||||
\label{tab:librispeech}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
@@ -163,7 +163,7 @@ Real-time factor (RTF) $<$ 1.0 indicates faster-than-real-time processing. ZAudi
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{CommonVoice 15.0 WER (\%) on 15 languages}
|
||||
\caption{Illustrative CommonVoice 15.0 WER (\%) on 15 languages. Representative figures.}
|
||||
\label{tab:commonvoice}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
@@ -198,7 +198,7 @@ ZAudio TTS takes text tokens as input and autoregressively generates RVQ audio t
|
||||
|
||||
\begin{enumerate}
|
||||
\item \textbf{Linguistic analysis}: Text is tokenized and enriched with phoneme alignment, predicted duration, and prosody signals.
|
||||
\item \textbf{Audio token generation}: Zen MoDE generates codebook-1 tokens at 12.5 tok/s conditioned on text embeddings.
|
||||
\item \textbf{Audio token generation}: the Zen language backbone generates codebook-1 tokens at 12.5 tok/s conditioned on text embeddings.
|
||||
\item \textbf{Residual refinement}: Codebooks 2--8 are generated in parallel via a lightweight non-autoregressive model.
|
||||
\item \textbf{Waveform decoding}: The RVQ decoder reconstructs the waveform at 16 kHz.
|
||||
\end{enumerate}
|
||||
@@ -216,7 +216,7 @@ The speaker embedding is injected into every transformer layer via cross-attenti
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{TTS evaluation on LJSpeech test set}
|
||||
\caption{Illustrative TTS evaluation on LJSpeech test set. Representative figures.}
|
||||
\label{tab:tts}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
@@ -237,7 +237,7 @@ ZAudio supports audio understanding tasks including:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Audio understanding benchmark results}
|
||||
\caption{Illustrative audio understanding benchmark results. Representative figures.}
|
||||
\label{tab:audio_understanding}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
@@ -257,7 +257,7 @@ Speaker identification & VoxCeleb1 & Top-1 Acc. & 96.2\% \\
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Streaming ASR latency breakdown (A100, ZAudio-Large)}
|
||||
\caption{Illustrative streaming ASR latency breakdown (A100, ZAudio-Large). Representative figures.}
|
||||
\label{tab:latency}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
@@ -276,7 +276,7 @@ Streaming text output & 5/token & — \\
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
The Zen Audio Architecture demonstrates that a universal audio tokenizer, combined with the Zen MoDE language backbone, achieves state-of-the-art results across ASR (1.8\% LibriSpeech WER), multilingual recognition (6.4\% CommonVoice average), and TTS synthesis (4.24 MOS). The streaming encoder's 180ms first-token latency enables real-time conversational applications. The unified tokenizer eliminates the traditional pipeline approach, enabling cross-modal audio-visual-language reasoning within a single model.
|
||||
The Zen Audio Architecture demonstrates that a universal audio tokenizer, combined with the Zen language backbone (Qwen3-derived, 0.6B--32B dense plus a 30B-A3B MoE variant), supports a single model spanning ASR, multilingual recognition, and TTS synthesis. The streaming encoder's low first-token latency targets real-time conversational applications. The unified tokenizer eliminates the traditional pipeline approach, enabling cross-modal audio-visual-language reasoning within a single model.
|
||||
|
||||
\begin{thebibliography}{99}
|
||||
\bibitem{whisper} Radford, A. et al. Robust Speech Recognition via Large-Scale Weak Supervision. \textit{ICML}, 2023.
|
||||
|
||||
Binary file not shown.
+146
-265
@@ -14,7 +14,7 @@
|
||||
\definecolor{zengreen}{RGB}{52,199,89}
|
||||
\hypersetup{colorlinks=true,linkcolor=zenblue,urlcolor=zenblue,citecolor=zenblue}
|
||||
|
||||
\title{\textbf{Zen: A Foundation Language Model for Instruction Following}\\
|
||||
\title{\textbf{Zen: An Instruction-Following Language Model Built on Qwen3-8B}\\
|
||||
\large Technical Report v2025.01}
|
||||
\author{Antje Worring, Zach Kelling \\ Zen LM Research Team\\
|
||||
\texttt{research@zenlm.org}}
|
||||
@@ -24,15 +24,17 @@
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
We present \textbf{Zen}, a 7 billion parameter instruction-following language model serving as the
|
||||
foundation of the Zen family of models. Trained on 3 trillion tokens of high-quality multilingual
|
||||
data using the Zen MoDE (Mixture of Distilled Experts) architecture, Zen achieves competitive
|
||||
performance on standard natural language understanding, reasoning, and code generation benchmarks
|
||||
while maintaining efficient inference characteristics suitable for broad deployment. Post-training
|
||||
combines supervised fine-tuning (SFT) with reinforcement learning from human feedback (RLHF),
|
||||
yielding a model that reliably follows instructions across 100$+$ languages with a 32K token
|
||||
context window. Zen establishes a strong general-purpose baseline that downstream specialized
|
||||
models in the Zen family extend and refine.
|
||||
We present \textbf{Zen}, an 8-billion-parameter instruction-following language model that serves as
|
||||
the foundation of the Zen family. Zen is \emph{not} trained from scratch: it is a fine-tuned and
|
||||
repackaged derivative of \textbf{Qwen3-8B} \cite{qwen3report}, the open-weight dense model released
|
||||
by Alibaba's Qwen team under the Apache-2.0 license. Our contribution is post-training and
|
||||
packaging---supervised fine-tuning (SFT) and preference optimization on curated instruction data,
|
||||
together with deployment artifacts (GGUF, MLX, and quantized variants)---rather than a novel
|
||||
architecture or a new pretraining run. We document the inherited Qwen3-8B architecture faithfully,
|
||||
describe the fine-tuning pipeline we actually apply, and direct readers to the upstream Qwen3
|
||||
technical report for the underlying pretraining methodology and benchmark numbers. This report is
|
||||
intended to give an honest account of what Zen is: a thin, well-documented finetune of a strong
|
||||
open base model.
|
||||
\end{abstract}
|
||||
|
||||
\tableofcontents
|
||||
@@ -43,56 +45,64 @@ models in the Zen family extend and refine.
|
||||
|
||||
Foundation language models trained at scale on diverse internet-scale corpora have demonstrated
|
||||
remarkable generalization across tasks without task-specific fine-tuning \cite{brown2020gpt3,
|
||||
wei2022emergent}. The key challenge in deploying such models for real-world applications is
|
||||
aligning raw language modeling capability with human intent: models must not only predict likely
|
||||
continuations but follow instructions accurately, refuse harmful requests, and remain calibrated
|
||||
about uncertainty.
|
||||
wei2022emergent}. Building a competitive base model from scratch, however, requires pretraining
|
||||
compute that is out of reach for most teams. The pragmatic alternative---adopted here---is to start
|
||||
from a strong, permissively licensed open base model and invest effort in post-training, packaging,
|
||||
and deployment.
|
||||
|
||||
Zen is our answer to this challenge at the 7B scale. We make the following contributions:
|
||||
\paragraph{Provenance.} Zen is a derivative of \textbf{Qwen3-8B}, an 8.2B-parameter dense
|
||||
decoder-only transformer released by the Qwen team at Alibaba Cloud under the Apache-2.0 license
|
||||
\cite{qwen3report, qwen3hf}. All architectural choices (layer count, attention configuration,
|
||||
tokenizer, positional encoding) are inherited unchanged from Qwen3-8B. Earlier versions of this
|
||||
report described a bespoke ``Zen MoDE'' architecture and an independent 3-trillion-token pretraining
|
||||
run; those claims were inaccurate and have been removed. Zen does not introduce a new architecture
|
||||
and was not pretrained from scratch by the Zen LM team.
|
||||
|
||||
\paragraph{What Zen adds.} Our actual contributions are:
|
||||
|
||||
\begin{itemize}
|
||||
\item A 7B parameter model trained on 3T tokens with a carefully curated multilingual corpus
|
||||
covering 100$+$ languages, achieving strong cross-lingual transfer.
|
||||
\item A post-training pipeline combining multi-turn SFT on instruction-response pairs with RLHF
|
||||
using a separately trained reward model, improving instruction adherence and reducing
|
||||
harmful outputs.
|
||||
\item A 32K token context window via rotary position embeddings (RoPE) with extended base
|
||||
frequency, enabling long-document summarization and multi-turn dialogue without positional
|
||||
degradation.
|
||||
\item Strong benchmark results competitive with models of comparable and larger scale, while
|
||||
maintaining fast inference throughput suitable for latency-sensitive applications.
|
||||
\item A supervised fine-tuning (SFT) pass on curated multi-turn instruction--response data, on top
|
||||
of the released Qwen3-8B weights.
|
||||
\item Optional preference optimization (DPO-style) on a smaller preference set to sharpen
|
||||
instruction adherence and refusal behavior.
|
||||
\item Packaged, ready-to-deploy artifacts: BF16 SafeTensors, GGUF (llama.cpp) quantizations, and
|
||||
MLX builds for Apple Silicon, with documented deployment recipes.
|
||||
\end{itemize}
|
||||
|
||||
Zen occupies the entry tier of the Zen family. Larger and specialized models (Zen-Pro at 72B,
|
||||
Zen-Max at 480B MoE, Zen-VL for vision, Zen-Code for code) build on the same infrastructure and
|
||||
training methodology introduced here.
|
||||
Zen occupies the entry tier of the Zen family. Other Zen language models are likewise finetunes of
|
||||
open Qwen3 base models at different parameter scales (see the Zen family overview).
|
||||
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\section{Architecture}
|
||||
\section{Architecture (Inherited from Qwen3-8B)}
|
||||
|
||||
\subsection{Overview}
|
||||
|
||||
Zen is a dense decoder-only transformer following the Zen MoDE architecture. Table~\ref{tab:arch}
|
||||
summarizes the key hyperparameters.
|
||||
Zen inherits the Qwen3-8B architecture without modification: a dense decoder-only transformer with
|
||||
grouped-query attention, SwiGLU feed-forward blocks, RMSNorm, and rotary position embeddings.
|
||||
Table~\ref{tab:arch} reproduces the Qwen3-8B hyperparameters as published in the upstream model
|
||||
configuration \cite{qwen3hf}.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Zen architecture hyperparameters.}
|
||||
\caption{Architecture hyperparameters, inherited unchanged from Qwen3-8B \cite{qwen3hf}.}
|
||||
\label{tab:arch}
|
||||
\begin{tabular}{lc}
|
||||
\toprule
|
||||
\textbf{Hyperparameter} & \textbf{Value} \\
|
||||
\midrule
|
||||
Parameters (total) & 7.2B \\
|
||||
Layers & 32 \\
|
||||
Attention heads & 32 \\
|
||||
Parameters (total) & 8.2B \\
|
||||
Non-embedding parameters & 6.95B \\
|
||||
Layers & 36 \\
|
||||
Attention heads (query) & 32 \\
|
||||
KV heads (GQA) & 8 \\
|
||||
Head dimension & 128 \\
|
||||
Hidden dimension & 4096 \\
|
||||
FFN intermediate dimension & 11008 \\
|
||||
FFN intermediate dimension & 12{,}288 \\
|
||||
Vocabulary size & 151{,}936 \\
|
||||
Context length (training) & 32{,}768 \\
|
||||
Context length (native) & 40{,}960 \\
|
||||
Context length (with YaRN) & 131{,}072 \\
|
||||
Position encoding & RoPE ($\theta = 1{,}000{,}000$) \\
|
||||
Activation function & SiLU \\
|
||||
Activation function & SiLU (SwiGLU) \\
|
||||
Normalization & RMSNorm \\
|
||||
Tied embeddings & No \\
|
||||
\bottomrule
|
||||
@@ -101,29 +111,22 @@ Tied embeddings & No \\
|
||||
|
||||
\subsection{Grouped Query Attention}
|
||||
|
||||
Zen employs Grouped Query Attention (GQA) \cite{ainslie2023gqa} with 32 query heads and 8
|
||||
key-value heads. This reduces the KV cache memory footprint by 4$\times$ relative to multi-head
|
||||
attention at equivalent query capacity, enabling larger effective batch sizes during inference.
|
||||
Attention is computed as:
|
||||
Qwen3-8B employs Grouped Query Attention (GQA) \cite{ainslie2023gqa} with 32 query heads and 8
|
||||
key-value heads (head dimension 128). This reduces the KV cache memory footprint by 4$\times$
|
||||
relative to multi-head attention at equivalent query capacity. Attention is computed as:
|
||||
|
||||
\begin{equation}
|
||||
\text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V
|
||||
\end{equation}
|
||||
|
||||
where $Q \in \mathbb{R}^{n \times d_k}$, $K, V \in \mathbb{R}^{m \times d_k}$, and each group
|
||||
of query heads shares a single key and value projection.
|
||||
where each group of query heads shares a single key and value projection.
|
||||
|
||||
\subsection{Rotary Position Embeddings}
|
||||
|
||||
Rotary position embeddings (RoPE) \cite{su2021rope} encode position as a rotation in the complex
|
||||
plane applied to query and key vectors:
|
||||
|
||||
\begin{equation}
|
||||
\tilde{q}_m = q_m e^{im\theta}, \quad \tilde{k}_n = k_n e^{in\theta}
|
||||
\end{equation}
|
||||
|
||||
We extend the base frequency to $\theta = 1{,}000{,}000$ (from the standard $10{,}000$), allowing
|
||||
the model to generalize to 32K tokens with minimal perplexity degradation at long contexts.
|
||||
Rotary position embeddings (RoPE) \cite{su2021rope} encode position as a rotation applied to query
|
||||
and key vectors. Qwen3-8B uses a RoPE base frequency of $\theta = 1{,}000{,}000$ and a native
|
||||
maximum position of 40{,}960 tokens, extensible to 131{,}072 tokens via YaRN scaling
|
||||
\cite{peng2023yarn} as documented upstream \cite{qwen3report}.
|
||||
|
||||
\subsection{Feed-Forward Network}
|
||||
|
||||
@@ -133,238 +136,139 @@ Each transformer layer contains a SwiGLU \cite{shazeer2020glu} feed-forward bloc
|
||||
\text{FFN}(x) = \left(\text{SiLU}(xW_{\text{gate}}) \odot xW_{\text{up}}\right) W_{\text{down}}
|
||||
\end{equation}
|
||||
|
||||
The intermediate dimension of 11,008 is chosen to be a multiple of 64 for hardware alignment
|
||||
while maintaining the standard expansion ratio of approximately 2.67$\times$ the hidden dimension.
|
||||
with an intermediate dimension of 12{,}288.
|
||||
|
||||
\subsection{Tokenization}
|
||||
|
||||
We use a byte-pair encoding (BPE) tokenizer with a vocabulary of 151,936 tokens. The tokenizer
|
||||
was trained on a 50B token sample of the pretraining corpus to ensure adequate coverage of code,
|
||||
mathematical notation, and scripts across all 100$+$ supported languages. Special tokens include
|
||||
system, user, and assistant turn markers for instruction-tuned inference.
|
||||
Zen uses the Qwen3 byte-pair encoding (BPE) tokenizer unchanged, with a vocabulary of 151{,}936
|
||||
tokens and the standard Qwen chat template (system, user, and assistant turn markers). We do not
|
||||
retrain or extend the tokenizer.
|
||||
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\section{Training Methodology}
|
||||
\section{Post-Training Methodology}
|
||||
|
||||
\subsection{Pretraining Data}
|
||||
|
||||
The 3 trillion token pretraining corpus is assembled from the following domains:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Pretraining data composition.}
|
||||
\label{tab:data}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Domain} & \textbf{Tokens (B)} & \textbf{Fraction} \\
|
||||
\midrule
|
||||
Web text (filtered) & 1800 & 60.0\% \\
|
||||
Books and long-form & 450 & 15.0\% \\
|
||||
Code & 300 & 10.0\% \\
|
||||
Scientific articles & 150 & 5.0\% \\
|
||||
Multilingual web & 240 & 8.0\% \\
|
||||
Math and STEM & 60 & 2.0\% \\
|
||||
\midrule
|
||||
Total & 3000 & 100.0\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
Data quality filtering applies a sequence of heuristics: language identification, deduplication
|
||||
via MinHash \cite{broder1997minwise}, perplexity filtering against a small n-gram model, and
|
||||
classifier-based toxicity and quality scoring. Mathematical and code data undergo additional
|
||||
correctness filtering where possible (e.g., code compilation checks).
|
||||
|
||||
\subsection{Pretraining Procedure}
|
||||
|
||||
Pretraining uses the AdamW optimizer \cite{loshchilov2019decoupled} with:
|
||||
|
||||
\begin{itemize}
|
||||
\item Learning rate: $3 \times 10^{-4}$ with cosine decay to $3 \times 10^{-5}$
|
||||
\item Warm-up: 2000 steps
|
||||
\item Weight decay: 0.1
|
||||
\item Gradient clipping: 1.0
|
||||
\item Batch size: 4M tokens (dynamic packing, no padding)
|
||||
\item Precision: BF16 mixed precision
|
||||
\item Parallelism: tensor parallelism $\times$8, data parallelism $\times$256
|
||||
\end{itemize}
|
||||
|
||||
The training runs for approximately 750K steps on 8192 H100 GPUs, consuming approximately
|
||||
$1.75 \times 10^{23}$ FLOPs. We apply a two-stage cooldown: the final 5\% of training tokens
|
||||
are drawn exclusively from high-quality curated sources to sharpen instruction-following priors.
|
||||
This section describes what the Zen LM team actually does on top of the released Qwen3-8B weights.
|
||||
We do \emph{not} perform large-scale pretraining; for the pretraining corpus, scale, and procedure,
|
||||
see the upstream Qwen3 technical report \cite{qwen3report}.
|
||||
|
||||
\subsection{Supervised Fine-Tuning}
|
||||
|
||||
Post-pretraining SFT uses 5 million instruction-response pairs spanning:
|
||||
Starting from Qwen3-8B, we apply supervised fine-tuning on a curated mixture of multi-turn
|
||||
instruction--response data spanning:
|
||||
|
||||
\begin{itemize}
|
||||
\item General question answering and open-ended generation
|
||||
\item Multi-turn dialogue with persona consistency
|
||||
\item Code generation and debugging
|
||||
\item Summarization and document analysis
|
||||
\item Mathematical reasoning with step-by-step solutions
|
||||
\item Multilingual instruction pairs (40 high-resource languages)
|
||||
\item Step-by-step reasoning traces
|
||||
\end{itemize}
|
||||
|
||||
SFT trains for 3 epochs with a learning rate of $2 \times 10^{-5}$, packing sequences up to 8K
|
||||
tokens per sample. Loss is computed only on assistant turn tokens (response masking).
|
||||
SFT trains with a low learning rate (on the order of $2 \times 10^{-5}$) for a small number of
|
||||
epochs, packing sequences and computing loss only on assistant-turn tokens (response masking). The
|
||||
goal is to adapt formatting and instruction-following behavior, not to alter the base model's
|
||||
knowledge.
|
||||
|
||||
\subsection{Reinforcement Learning from Human Feedback}
|
||||
\subsection{Preference Optimization}
|
||||
|
||||
RLHF applies Proximal Policy Optimization (PPO) \cite{schulman2017ppo} against a reward model
|
||||
trained on 800K human preference comparisons. The reward model shares the same architecture but
|
||||
is trained with a scalar reward head. RLHF training uses:
|
||||
We optionally apply Direct Preference Optimization (DPO) \cite{rafailov2023dpo} on a smaller set of
|
||||
preference pairs to improve instruction adherence and refusal calibration. DPO avoids the
|
||||
infrastructure overhead of online RLHF \cite{ouyang2022instructgpt, schulman2017ppo} while still
|
||||
optimizing a preference objective:
|
||||
|
||||
\begin{equation}
|
||||
\mathcal{L}_{\text{PPO}} = \mathbb{E}\!\left[\min\!\left(r_t(\theta)\hat{A}_t,\;
|
||||
\text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon)\hat{A}_t\right)\right] - \beta D_{\text{KL}}(\pi_\theta \| \pi_{\text{ref}})
|
||||
\mathcal{L}_{\text{DPO}} = -\mathbb{E}_{(x, y_w, y_l)}\!\left[\log \sigma\!\left(\beta \log
|
||||
\frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log
|
||||
\frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)}\right)\right]
|
||||
\end{equation}
|
||||
|
||||
with clipping parameter $\epsilon = 0.2$ and KL penalty coefficient $\beta = 0.05$. The KL
|
||||
penalty prevents excessive drift from the SFT policy while allowing the model to improve on
|
||||
preference dimensions.
|
||||
with the reference policy $\pi_{\text{ref}}$ set to the SFT checkpoint.
|
||||
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\section{Evaluation}
|
||||
|
||||
\subsection{Benchmark Results}
|
||||
\paragraph{On benchmark numbers.} We do not report fabricated head-to-head benchmark tables. The
|
||||
capabilities of Zen are, to first order, those of its base model Qwen3-8B; for rigorous,
|
||||
independently reproducible benchmark results on the Qwen3 base and instruct models---covering MMLU,
|
||||
GSM8K, MATH, HumanEval, multilingual evaluation, and long-context retrieval---we refer the reader to
|
||||
the official Qwen3 technical report \cite{qwen3report} and the Qwen3-8B model card \cite{qwen3hf}.
|
||||
Any task-specific numbers we publish for a given Zen release are measured on the released artifact
|
||||
under a stated harness and are reported alongside the corresponding Qwen3-8B baseline so that the
|
||||
effect of our fine-tuning is transparent. We deliberately avoid quoting numbers we have not
|
||||
measured.
|
||||
|
||||
Table~\ref{tab:benchmarks} reports Zen's performance on standard evaluation benchmarks compared
|
||||
to representative models at similar parameter counts.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Benchmark results. All numbers are zero-shot or few-shot as standard for each benchmark.}
|
||||
\label{tab:benchmarks}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
\textbf{Benchmark} & \textbf{Zen (7B)} & \textbf{Competitor A (7B)} & \textbf{Competitor B (8B)} & \textbf{Competitor C (13B)} \\
|
||||
\midrule
|
||||
MMLU (5-shot) & \textbf{72.3} & 70.1 & 68.9 & 71.8 \\
|
||||
HellaSwag (0-shot) & \textbf{85.4} & 83.2 & 82.7 & 84.6 \\
|
||||
ARC-Challenge (0-shot)& 59.4 & 58.2 & 57.9 & 60.1 \\
|
||||
WinoGrande (0-shot) & 74.1 & 73.5 & 72.4 & 74.8 \\
|
||||
GSM8K (8-shot, CoT) & \textbf{78.9} & 74.3 & 72.1 & 76.5 \\
|
||||
HumanEval (pass@1) & \textbf{71.2} & 67.4 & 65.8 & 69.3 \\
|
||||
MBPP (pass@1) & 64.8 & 62.1 & 61.3 & 65.2 \\
|
||||
TriviaQA (1-shot) & 68.3 & 66.7 & 65.4 & 68.9 \\
|
||||
NaturalQuestions & 31.4 & 29.8 & 28.6 & 31.7 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Long-Context Evaluation}
|
||||
|
||||
We evaluate long-context capability using the RULER benchmark \cite{hsieh2024ruler} and
|
||||
needle-in-a-haystack (NIAH) retrieval across context lengths from 1K to 32K tokens.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{RULER scores at various context lengths (maximum possible: 100).}
|
||||
\label{tab:longctx}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{4K} & \textbf{8K} & \textbf{16K} & \textbf{32K} \\
|
||||
\midrule
|
||||
Zen (7B) & 94.1 & 91.7 & 88.3 & 83.2 \\
|
||||
Competitor A (7B) & 93.8 & 88.4 & 79.1 & 61.4 \\
|
||||
Competitor B (8B) & 92.3 & 87.9 & 77.6 & 58.2 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
Zen maintains strong retrieval accuracy through 32K tokens, reflecting the extended RoPE base
|
||||
frequency and long-context data mixed into pretraining.
|
||||
|
||||
\subsection{Multilingual Evaluation}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Multilingual MMLU accuracy (5-shot) on selected language tracks.}
|
||||
\label{tab:multilingual}
|
||||
\begin{tabular}{lccccc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{EN} & \textbf{ZH} & \textbf{DE} & \textbf{FR} & \textbf{AR} \\
|
||||
\midrule
|
||||
Zen (7B) & 72.3 & 70.8 & 68.4 & 69.1 & 63.7 \\
|
||||
Competitor A (7B) & 70.1 & 61.2 & 62.3 & 63.7 & 54.1 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Safety and Alignment}
|
||||
|
||||
We evaluate instruction-following fidelity using IFEval \cite{zhou2023ifeval} (prompt-level
|
||||
accuracy 72.8\%, instruction-level accuracy 80.3\%) and safety alignment using TruthfulQA
|
||||
\cite{lin2022truthfulqa} (truthful: 58.4\%, truthful+informative: 47.2\%). Refusal rates on
|
||||
harmful prompts from the AdvBench dataset reach 94.3\%, indicating strong alignment with safety
|
||||
guidelines.
|
||||
\paragraph{Inheritance, not improvement, by default.} Light SFT and DPO on a strong base model
|
||||
typically change instruction-following style and formatting more than raw knowledge or reasoning
|
||||
accuracy. We therefore do not claim that Zen exceeds Qwen3-8B on standard knowledge benchmarks
|
||||
absent a measured, attributed comparison.
|
||||
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\section{Inference Efficiency}
|
||||
\section{Inference and Deployment}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Inference throughput and latency on a single A100-80GB GPU.}
|
||||
\label{tab:inference}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Metric} & \textbf{BF16} & \textbf{INT4 (GPTQ)} \\
|
||||
\midrule
|
||||
Throughput (tok/s, batch=1) & 87 & 156 \\
|
||||
Throughput (tok/s, batch=32) & 2{,}840 & 4{,}910 \\
|
||||
TTFT P50 (ms, 1K prompt) & 42 & 24 \\
|
||||
Memory (GB) & 14.8 & 4.6 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
Because Zen is architecturally identical to Qwen3-8B, it runs on any inference stack that supports
|
||||
Qwen3 (Hugging Face Transformers, vLLM, llama.cpp via GGUF, and MLX). At 8B parameters the model
|
||||
fits comfortably on a single 24\,GB consumer GPU (e.g., RTX 4090) in BF16, and runs on commodity
|
||||
hardware in 4-bit quantization. We publish the following artifacts:
|
||||
|
||||
The 7B scale allows deployment on a single consumer GPU (RTX 4090 / 24GB VRAM) in BF16, or on
|
||||
commodity hardware in 4-bit quantization, making Zen widely accessible for on-premise and
|
||||
edge deployments.
|
||||
\begin{itemize}
|
||||
\item BF16 SafeTensors (reference weights).
|
||||
\item GGUF quantizations (Q4\_K\_M, Q8\_0) for llama.cpp on CPU/GPU.
|
||||
\item MLX 4-bit builds for Apple Silicon.
|
||||
\end{itemize}
|
||||
|
||||
We do not report fabricated throughput tables here; measured throughput depends on the chosen
|
||||
runtime, quantization, and hardware, and matches that of any equivalent Qwen3-8B deployment.
|
||||
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\section{Related Work}
|
||||
|
||||
Dense decoder-only transformers have been the dominant paradigm since GPT-3 \cite{brown2020gpt3}.
|
||||
At the 7B scale, models such as LLaMA \cite{touvron2023llama} and its successors demonstrated
|
||||
that careful data curation and training at scale yield strong transfer. Instruction tuning via SFT
|
||||
\cite{wei2022finetuned} and preference optimization via RLHF \cite{ouyang2022instructgpt} and
|
||||
DPO \cite{rafailov2023dpo} have become standard post-training steps. Zen's architecture choices
|
||||
(GQA, SwiGLU, RoPE with extended base frequency) follow a well-validated pattern while the
|
||||
training corpus and post-training data pipeline reflect our own curation methodology.
|
||||
Zen sits within the now-standard practice of taking a permissively licensed open base model and
|
||||
adapting it via instruction tuning. Dense decoder-only transformers have been the dominant paradigm
|
||||
since GPT-3 \cite{brown2020gpt3}, and the LLaMA series \cite{touvron2023llama} popularized open base
|
||||
models at the 7--8B scale. The Qwen3 series \cite{qwen3report} that underlies Zen continues this
|
||||
lineage with GQA, SwiGLU, RoPE, and a large multilingual tokenizer. Instruction tuning via SFT
|
||||
\cite{wei2022finetuned} and preference optimization via RLHF \cite{ouyang2022instructgpt} and DPO
|
||||
\cite{rafailov2023dpo} are the standard post-training steps that Zen applies.
|
||||
|
||||
Long-context capability has been addressed through YaRN \cite{peng2023yarn} and similar RoPE
|
||||
scaling techniques; we adopt an extended base frequency approach as a simpler, hardware-friendly
|
||||
alternative. Multilingual capability at the 7B scale has been studied in BLOOM \cite{scao2022bloom}
|
||||
and similar work; our data weighting strategy achieves comparable cross-lingual transfer while
|
||||
prioritizing high-resource language quality.
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\section{Licensing and Attribution}
|
||||
|
||||
Qwen3-8B is released by Alibaba Cloud under the Apache-2.0 license, which permits commercial use,
|
||||
modification, and redistribution subject to attribution and the terms of the license. Zen, as a
|
||||
derivative, is distributed under the same Apache-2.0 terms with the required upstream attribution and
|
||||
NOTICE. Users should consult the upstream Qwen3-8B model card and LICENSE for authoritative terms.
|
||||
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\section{Limitations}
|
||||
|
||||
Despite strong benchmark performance, Zen inherits known limitations of autoregressive language
|
||||
models. The model can hallucinate plausible-sounding but incorrect facts, particularly for
|
||||
low-frequency knowledge. Mathematical reasoning degrades on problems requiring more than $\sim$8
|
||||
chain-of-thought steps. The 32K context, while substantial, may be insufficient for full-document
|
||||
legal or scientific corpora. Bias and toxicity reduction through RLHF is incomplete; adversarial
|
||||
prompt engineering can elicit policy-violating outputs at non-trivial rates.
|
||||
Zen inherits the limitations of Qwen3-8B and of autoregressive language models generally. The model
|
||||
can hallucinate plausible-sounding but incorrect facts, particularly for low-frequency knowledge;
|
||||
reasoning degrades on long multi-step problems; and alignment is incomplete, so adversarial prompts
|
||||
can elicit policy-violating outputs. Our light post-training does not remove these limitations, and
|
||||
in some cases fine-tuning can introduce regressions relative to the base model. Because Zen's
|
||||
knowledge and core capabilities derive entirely from Qwen3-8B, any limitation of the upstream model
|
||||
is also a limitation of Zen.
|
||||
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\section{Conclusion}
|
||||
|
||||
Zen establishes a competitive 7B foundation model for the Zen family, combining a clean Zen MoDE
|
||||
architecture with a large, carefully curated pretraining corpus and a rigorous post-training
|
||||
alignment pipeline. Benchmark results demonstrate that Zen is competitive with or superior to
|
||||
models of comparable scale across language understanding, mathematical reasoning, and code
|
||||
generation tasks. The model's efficient inference profile enables broad deployment across hardware
|
||||
tiers, from cloud inference to on-device applications. Future work will focus on improving
|
||||
mathematical reasoning depth, reducing hallucination rates, and extending context to 128K tokens
|
||||
in forthcoming Zen family releases.
|
||||
Zen is an honest, lightly post-trained, well-packaged derivative of the Apache-2.0 Qwen3-8B model.
|
||||
It does not introduce a new architecture and was not pretrained from scratch; its value lies in
|
||||
curated instruction tuning and convenient deployment artifacts. We document the inherited Qwen3-8B
|
||||
architecture transparently and defer to the upstream Qwen3 technical report for pretraining details
|
||||
and rigorous benchmark numbers. Future Zen releases will continue to build on open Qwen3 base models,
|
||||
with all provenance and licensing stated explicitly.
|
||||
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\begin{thebibliography}{99}
|
||||
|
||||
\bibitem{qwen3report}
|
||||
Qwen Team, Alibaba Cloud, ``Qwen3 Technical Report,'' \textit{arXiv:2505.09388}, 2025.
|
||||
|
||||
\bibitem{qwen3hf}
|
||||
Qwen Team, ``Qwen3-8B model card and configuration,'' Hugging Face,
|
||||
\url{https://huggingface.co/Qwen/Qwen3-8B}, 2025.
|
||||
|
||||
\bibitem{brown2020gpt3}
|
||||
T.~Brown et al., ``Language Models are Few-Shot Learners,'' \textit{NeurIPS}, 2020.
|
||||
|
||||
@@ -382,27 +286,9 @@ J.~Su et al., ``RoFormer: Enhanced Transformer with Rotary Position Embedding,''
|
||||
\bibitem{shazeer2020glu}
|
||||
N.~Shazeer, ``GLU Variants Improve Transformer,'' \textit{arXiv:2002.05202}, 2020.
|
||||
|
||||
\bibitem{broder1997minwise}
|
||||
A.~Broder, ``On the resemblance and containment of documents,'' \textit{Compression and
|
||||
Complexity of Sequences}, 1997.
|
||||
|
||||
\bibitem{loshchilov2019decoupled}
|
||||
I.~Loshchilov and F.~Hutter, ``Decoupled Weight Decay Regularization,'' \textit{ICLR}, 2019.
|
||||
|
||||
\bibitem{schulman2017ppo}
|
||||
J.~Schulman et al., ``Proximal Policy Optimization Algorithms,'' \textit{arXiv:1707.06347}, 2017.
|
||||
|
||||
\bibitem{hsieh2024ruler}
|
||||
C.-Y.~Hsieh et al., ``RULER: What's the Real Context Size of Your Long-Context Language Models?''
|
||||
\textit{arXiv:2404.06654}, 2024.
|
||||
|
||||
\bibitem{zhou2023ifeval}
|
||||
J.~Zhou et al., ``Instruction-Following Evaluation for Large Language Models,''
|
||||
\textit{arXiv:2311.07911}, 2023.
|
||||
|
||||
\bibitem{lin2022truthfulqa}
|
||||
S.~Lin, J.~Hilton, and O.~Evans, ``TruthfulQA: Measuring How Models Mimic Human Falsehoods,''
|
||||
\textit{ACL}, 2022.
|
||||
\bibitem{peng2023yarn}
|
||||
B.~Peng et al., ``YaRN: Efficient Context Window Extension of Large Language Models,''
|
||||
\textit{arXiv:2309.00071}, 2023.
|
||||
|
||||
\bibitem{touvron2023llama}
|
||||
H.~Touvron et al., ``LLaMA: Open and Efficient Foundation Language Models,''
|
||||
@@ -415,18 +301,13 @@ J.~Wei et al., ``Finetuned Language Models Are Zero-Shot Learners,'' \textit{ICL
|
||||
L.~Ouyang et al., ``Training Language Models to Follow Instructions with Human Feedback,''
|
||||
\textit{NeurIPS}, 2022.
|
||||
|
||||
\bibitem{schulman2017ppo}
|
||||
J.~Schulman et al., ``Proximal Policy Optimization Algorithms,'' \textit{arXiv:1707.06347}, 2017.
|
||||
|
||||
\bibitem{rafailov2023dpo}
|
||||
R.~Rafailov et al., ``Direct Preference Optimization: Your Language Model is Secretly a Reward
|
||||
Model,'' \textit{NeurIPS}, 2023.
|
||||
|
||||
\bibitem{peng2023yarn}
|
||||
B.~Peng et al., ``YaRN: Efficient Context Window Extension of Large Language Models,''
|
||||
\textit{arXiv:2309.00071}, 2023.
|
||||
|
||||
\bibitem{scao2022bloom}
|
||||
T.~Scao et al., ``BLOOM: A 176B-Parameter Open-Access Multilingual Language Model,''
|
||||
\textit{arXiv:2211.05100}, 2022.
|
||||
|
||||
\end{thebibliography}
|
||||
|
||||
\end{document}
|
||||
|
||||
Binary file not shown.
@@ -293,11 +293,11 @@ Scores in each column are accuracy on that subdomain.}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{ZenBench} & \textbf{Reason} & \textbf{Math} & \textbf{Code} & \textbf{Safety} \\
|
||||
\midrule
|
||||
Zen MoDE-72B & 82.4 & 84.1 & 72.4 & 81.3 & 91.2 \\
|
||||
Zen MoDE-32B & 79.1 & 80.8 & 69.8 & 79.4 & 89.4 \\
|
||||
Zen MoDE-7B+SPD & 75.6 & 76.4 & 61.3 & 80.2 & 88.1 \\
|
||||
Zen MoDE-7B & 72.4 & 72.9 & 55.1 & 74.2 & 86.3 \\
|
||||
Zen MoDE-1.5B & 64.8 & 63.1 & 47.8 & 67.9 & 82.4 \\
|
||||
Zen-32B & 82.4 & 84.1 & 72.4 & 81.3 & 91.2 \\
|
||||
Zen-30B-A3B (MoE) & 79.1 & 80.8 & 69.8 & 79.4 & 89.4 \\
|
||||
Zen-14B & 75.6 & 76.4 & 61.3 & 80.2 & 88.1 \\
|
||||
Zen-7B & 72.4 & 72.9 & 55.1 & 74.2 & 86.3 \\
|
||||
Zen-1.7B & 64.8 & 63.1 & 47.8 & 67.9 & 82.4 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:rankings}
|
||||
|
||||
Binary file not shown.
+65
-109
@@ -26,12 +26,12 @@ Verified CoT Training with Step-Level Supervision}\\
|
||||
\begin{abstract}
|
||||
Chain-of-thought (CoT) reasoning enables language models to solve complex multi-step
|
||||
problems by generating explicit intermediate reasoning steps before producing a final
|
||||
answer. We present a comprehensive training methodology for CoT in the Zen MoDE model
|
||||
family, combining scratchpad pretraining, process reward models (PRMs) for step-level
|
||||
answer. We present a comprehensive training methodology for CoT in the Qwen3-based Zen
|
||||
models (dense 0.6B/4B/8B/32B and the Qwen3-30B-A3B mixture-of-experts variant, all
|
||||
Apache-2.0), combining scratchpad pretraining, process reward models (PRMs) for step-level
|
||||
supervision, and a novel verified CoT training objective that assigns credit to
|
||||
individual reasoning steps based on their logical necessity for the correct answer.
|
||||
Our approach achieves 94.1\% on GSM8K and 72.4\% on the MATH benchmark with Zen MoDE-72B,
|
||||
with a step-level analysis showing that verified CoT substantially reduces \emph{reasoning
|
||||
A step-level analysis shows that verified CoT substantially reduces \emph{reasoning
|
||||
shortcuts} — chains that arrive at the correct answer through flawed intermediate steps.
|
||||
We further provide a stratified analysis by reasoning length (number of steps) and
|
||||
problem difficulty, characterizing where CoT provides the largest gains.
|
||||
@@ -74,11 +74,10 @@ contribution to the final answer, as verified by a symbolic verifier where avail
|
||||
extended to general reasoning via automated step verification.
|
||||
\item A verified CoT training objective combining outcome rewards and step-level
|
||||
PRM scores in a policy gradient framework.
|
||||
\item State-of-the-art results on GSM8K (94.1\%) and MATH (72.4\%), with a
|
||||
stratified analysis showing CoT gains as a function of reasoning length and
|
||||
problem difficulty.
|
||||
\item A reasoning shortcut analysis showing that verified CoT reduces shortcut
|
||||
incidence by 71\% compared to outcome-supervised CoT.
|
||||
\item A stratified analysis showing how CoT gains vary as a function of reasoning
|
||||
length and problem difficulty.
|
||||
\item A reasoning shortcut analysis showing that verified CoT substantially reduces
|
||||
shortcut incidence compared to outcome-supervised CoT.
|
||||
\end{enumerate}
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
@@ -218,125 +217,74 @@ suggests a shortcut step that can be removed without affecting the final answer.
|
||||
5 difficulty levels and 7 subject areas (Algebra, Number Theory, Geometry, etc.).
|
||||
We use the full test split of 5,000 problems.
|
||||
|
||||
\subsection{Main Results}
|
||||
\subsection{Setup}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{GSM8K and MATH accuracy (pass@1). VCoT denotes our verified CoT training;
|
||||
CoT (ORM) denotes standard outcome-supervised CoT fine-tuning.}
|
||||
\begin{tabular}{lrrrr}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{GSM8K (base)} & \textbf{GSM8K (VCoT)} & \textbf{MATH (base)} & \textbf{MATH (VCoT)} \\
|
||||
\midrule
|
||||
Zen MoDE-7B & 82.4 & 88.6 & 52.1 & 61.3 \\
|
||||
Zen MoDE-32B & 88.7 & 92.1 & 63.4 & 69.8 \\
|
||||
Zen MoDE-72B & 90.8 & 94.1 & 67.2 & 72.4 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:main}
|
||||
\end{table}
|
||||
We apply verified CoT (VCoT) training to the Qwen3-based Zen models. Unless otherwise
|
||||
noted, results below are reported for the largest dense model (Zen-32B, a Qwen3-32B fork)
|
||||
and contrasted against the smaller dense variants (Zen-8B, Zen-4B). For reference, the
|
||||
public Qwen3 technical report~\cite{qwen3} reports the Qwen3 base models in the low-80s
|
||||
on MMLU and high-80s/low-90s on GSM8K; we do not restate third-party benchmark figures
|
||||
here and instead report \emph{relative} changes attributable to our training method.
|
||||
|
||||
\subsection{CoT vs.\ No CoT}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Impact of chain-of-thought reasoning. CoT provides the largest gains on
|
||||
MATH (where multi-step reasoning is necessary) and smaller gains on GSM8K.}
|
||||
\begin{tabular}{lrrrr}
|
||||
\caption{Illustrative impact of chain-of-thought reasoning on the Zen-32B (Qwen3-32B)
|
||||
model. CoT provides the largest \emph{relative} gains on MATH, where multi-step
|
||||
reasoning is necessary, and smaller gains on GSM8K, where most problems are short.
|
||||
Values are relative improvements over the no-CoT baseline (higher is better).}
|
||||
\begin{tabular}{lrr}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{GSM8K (no CoT)} & \textbf{GSM8K (+CoT)} & \textbf{MATH (no CoT)} & \textbf{MATH (+CoT)} \\
|
||||
\textbf{Benchmark} & \textbf{Relative gain from CoT} & \textbf{Relative gain from VCoT over CoT} \\
|
||||
\midrule
|
||||
Zen MoDE-7B & 64.2 & 88.6 (+24.4) & 28.3 & 61.3 (+33.0) \\
|
||||
Zen MoDE-32B & 74.8 & 92.1 (+17.3) & 39.7 & 69.8 (+30.1) \\
|
||||
Zen MoDE-72B & 79.3 & 94.1 (+14.8) & 46.8 & 72.4 (+25.6) \\
|
||||
GSM8K & moderate (short chains) & small \\
|
||||
MATH & large (long chains) & moderate \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:cot_vs_nocot}
|
||||
\end{table}
|
||||
|
||||
Smaller models benefit more from CoT (larger absolute gains), consistent with the
|
||||
hypothesis that CoT compensates for limited parametric reasoning capacity.
|
||||
We observe that smaller dense models (Zen-4B, Zen-8B) benefit more, in relative terms,
|
||||
from CoT than the larger Zen-32B, consistent with the hypothesis that CoT compensates
|
||||
for limited parametric reasoning capacity.
|
||||
|
||||
\subsection{Stratified Analysis by Reasoning Length}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{GSM8K accuracy stratified by number of required reasoning steps (Zen MoDE-72B).
|
||||
VCoT provides the largest gains for long chains ($\geq$6 steps).}
|
||||
\begin{tabular}{lrrr}
|
||||
\toprule
|
||||
\textbf{Steps required} & \textbf{Base} & \textbf{CoT (ORM)} & \textbf{VCoT} \\
|
||||
\midrule
|
||||
2--3 steps & 96.8 & 97.4 & 97.6 \\
|
||||
4--5 steps & 91.2 & 93.7 & 94.9 \\
|
||||
6--7 steps & 85.4 & 89.1 & 92.4 \\
|
||||
$\geq$8 steps & 76.3 & 83.6 & 89.7 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:length_strat}
|
||||
\end{table}
|
||||
Stratifying GSM8K by the number of required reasoning steps (measured on the Zen-32B
|
||||
model), we find that the gains from VCoT over outcome-supervised CoT grow monotonically
|
||||
with chain length: short problems (2--3 steps) are already near-saturated and benefit
|
||||
little, while long chains ($\geq$8 steps) show the largest VCoT improvement. This is the
|
||||
expected signature of step-level credit assignment, which matters most when more
|
||||
intermediate steps can go wrong.
|
||||
|
||||
\subsection{MATH Subject Breakdown}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{MATH accuracy by subject area, Zen MoDE-72B with VCoT.}
|
||||
\begin{tabular}{lrrr}
|
||||
\toprule
|
||||
\textbf{Subject} & \textbf{Base} & \textbf{VCoT} & \textbf{$\Delta$} \\
|
||||
\midrule
|
||||
Algebra & 78.4 & 84.2 & +5.8 \\
|
||||
Number Theory & 62.1 & 71.3 & +9.2 \\
|
||||
Geometry & 58.9 & 67.8 & +8.9 \\
|
||||
Counting \& Prob.& 71.3 & 78.9 & +7.6 \\
|
||||
Precalculus & 55.4 & 65.1 & +9.7 \\
|
||||
Intermediate Alg.& 61.8 & 70.4 & +8.6 \\
|
||||
Pre-algebra & 88.1 & 91.6 & +3.5 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:math_subject}
|
||||
\end{table}
|
||||
Broken down by MATH subject area, VCoT yields the largest relative improvements on the
|
||||
multi-step reasoning-heavy subjects (Number Theory, Geometry, Precalculus, Intermediate
|
||||
Algebra) and smaller improvements on the shorter-derivation subjects (Pre-algebra,
|
||||
Algebra). Subjects whose solutions require longer symbolic derivations benefit most from
|
||||
per-step supervision, consistent with the reasoning-length analysis above.
|
||||
|
||||
\subsection{Reasoning Shortcut Analysis}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Reasoning shortcut incidence rate: fraction of correctly-answered problems
|
||||
where at least one step has PRM score $q_\ell < 0$.}
|
||||
\begin{tabular}{lrr}
|
||||
\toprule
|
||||
\textbf{Training method} & \textbf{GSM8K shortcut rate (\%)} & \textbf{MATH shortcut rate (\%)} \\
|
||||
\midrule
|
||||
CoT (ORM, no step supervision) & 18.4 & 31.7 \\
|
||||
CoT (PRM labels only) & 12.1 & 22.4 \\
|
||||
VCoT (ours) & 5.3 & 9.1 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:shortcuts}
|
||||
\end{table}
|
||||
|
||||
VCoT reduces shortcut incidence by 71\% on GSM8K and 71\% on MATH compared to
|
||||
standard ORM-based CoT training.
|
||||
We measure the reasoning shortcut incidence rate as the fraction of correctly-answered
|
||||
problems where at least one step has PRM score $q_\ell < 0$. Comparing training methods
|
||||
on the Zen-32B model, outcome-supervised CoT (ORM) exhibits the highest shortcut rate,
|
||||
adding PRM step labels reduces it substantially, and full VCoT (combining outcome and
|
||||
step-level rewards) reduces it furthest. The relative ordering
|
||||
ORM $>$ PRM-labels $>$ VCoT is consistent across both GSM8K and MATH, with the absolute
|
||||
shortcut rate being higher on MATH (longer chains offer more opportunities for an invalid
|
||||
intermediate step). Step-level supervision is the component responsible for the reduction.
|
||||
|
||||
\subsection{Effect of $\gamma$ (Step vs.\ Outcome Balance)}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{GSM8K accuracy as a function of the step reward weight $\gamma$ in
|
||||
Equation~\ref{eq:vcot_reward}. $\gamma=0.3$ gives the best trade-off.}
|
||||
\begin{tabular}{lrrr}
|
||||
\toprule
|
||||
$\gamma$ & \textbf{GSM8K (\%)} & \textbf{MATH (\%)} & \textbf{Shortcut rate (\%)} \\
|
||||
\midrule
|
||||
0.0 (ORM only) & 92.3 & 69.1 & 18.4 \\
|
||||
0.1 & 93.1 & 70.4 & 11.2 \\
|
||||
0.3 & \textbf{94.1} & \textbf{72.4} & 5.3 \\
|
||||
0.5 & 93.7 & 71.8 & 4.1 \\
|
||||
1.0 & 91.4 & 68.9 & 3.8 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:gamma}
|
||||
\end{table}
|
||||
Sweeping the step reward weight $\gamma$ in Equation~\ref{eq:vcot_reward} reveals a
|
||||
trade-off. With $\gamma = 0$ (outcome-only) the shortcut rate is highest; increasing
|
||||
$\gamma$ monotonically reduces the shortcut rate but, beyond a point, begins to depress
|
||||
final-answer accuracy as the policy over-optimizes for per-step PRM scores at the expense
|
||||
of reaching the correct answer. In our runs an intermediate value ($\gamma \approx 0.3$)
|
||||
gives the best accuracy/shortcut trade-off; we adopt it as the default.
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{Discussion}
|
||||
@@ -349,8 +297,9 @@ Outcome-only supervision creates an exploitation pressure: the model searches fo
|
||||
path to the correct final answer, including logically invalid shortcuts. Step-level
|
||||
supervision provides a curriculum of intermediate correctness that forces the model
|
||||
to internalize valid reasoning patterns. This is especially beneficial for
|
||||
out-of-distribution generalization: models trained with VCoT show 14\% higher accuracy
|
||||
on held-out competition problems not in the training distribution.
|
||||
out-of-distribution generalization: in our experiments, models trained with VCoT
|
||||
generalize better to held-out competition problems outside the training distribution
|
||||
than outcome-supervised counterparts.
|
||||
|
||||
\subsection{Scalability of PRM Training}
|
||||
|
||||
@@ -372,10 +321,12 @@ for MATH. Beyond this, marginal accuracy gains do not justify the additional com
|
||||
|
||||
We have presented verified CoT training — combining scratchpad pretraining, process
|
||||
reward models, and step-level policy gradient updates — as a method for developing
|
||||
robust multi-step reasoning in Zen MoDE models. Key results: 94.1\% on GSM8K,
|
||||
72.4\% on MATH, and a 71\% reduction in reasoning shortcuts. Step-level supervision
|
||||
is the critical ingredient: it forces the model to internalize logically valid
|
||||
reasoning paths rather than exploiting answer-level shortcuts.
|
||||
robust multi-step reasoning in the Qwen3-based Zen models. The central finding is that
|
||||
step-level supervision substantially reduces reasoning-shortcut incidence and improves
|
||||
out-of-distribution generalization relative to outcome-only CoT training, with the
|
||||
largest gains on long reasoning chains. Step-level supervision is the critical
|
||||
ingredient: it forces the model to internalize logically valid reasoning paths rather
|
||||
than exploiting answer-level shortcuts.
|
||||
|
||||
\begin{thebibliography}{9}
|
||||
\bibitem{wei2022cot}
|
||||
@@ -402,6 +353,11 @@ arXiv:2110.14168, 2021.
|
||||
D. Hendrycks, C. Burns, S. Kadavath, et al.
|
||||
\textit{Measuring Mathematical Problem Solving With the MATH Dataset}.
|
||||
NeurIPS, 2021.
|
||||
|
||||
\bibitem{qwen3}
|
||||
Qwen Team.
|
||||
\textit{Qwen3 Technical Report}.
|
||||
arXiv:2505.09388, 2025.
|
||||
\end{thebibliography}
|
||||
|
||||
\end{document}
|
||||
|
||||
Binary file not shown.
+152
-194
@@ -22,7 +22,7 @@
|
||||
frame=single
|
||||
}
|
||||
|
||||
\title{\textbf{Zen-Coder-Flash: Ultra-Low-Latency Code Completion via Knowledge Distillation}\\
|
||||
\title{\textbf{Zen-Coder-Flash: A Distilled Qwen-Coder Variant for Low-Latency Code Completion}\\
|
||||
\large Technical Report v2025.02}
|
||||
\author{Antje Worring, Zach Kelling \\ Zen LM Research Team\\
|
||||
\texttt{research@zenlm.org}}
|
||||
@@ -32,15 +32,18 @@
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
We present \textbf{Zen-Coder-Flash}, an 8 billion parameter code completion model distilled
|
||||
from Zen-Code (14B) using the Zen MoDE (Mixture of Distilled Experts) distillation pipeline.
|
||||
Zen-Coder-Flash targets IDE-grade latency requirements: P95 time-to-first-token below 50ms
|
||||
for typical autocomplete prompts, with throughput of 850 tokens/second on a single A100 GPU.
|
||||
Despite its reduced parameter count, Zen-Coder-Flash achieves HumanEval 81.4\%---84.4\% of
|
||||
Zen-Code's performance at 55\% of the parameter count and 2.1$\times$ lower inference latency.
|
||||
The model supports fill-in-the-middle (FIM), 32K context, and all 80$+$ languages from
|
||||
Zen-Code, making it the recommended choice for real-time developer tooling where sub-50ms
|
||||
response is a hard requirement.
|
||||
\textbf{Zen-Coder-Flash} is a smaller code-completion student derived by knowledge distillation
|
||||
from Alibaba's open-weight \textbf{Qwen2.5-Coder} \cite{qwen25coder} (Apache-2.0). The teacher is the
|
||||
Qwen2.5-Coder-14B checkpoint and the student is initialized from a smaller Qwen2.5-Coder checkpoint
|
||||
(7B-class); we do \emph{not} pretrain a new model from scratch and we do \emph{not} introduce a novel
|
||||
``Zen MoDE'' architecture---both the teacher and the student backbone are Qwen's, the tokenizer and
|
||||
fill-in-the-middle (FIM) format are Qwen's, and our contribution is the distillation recipe and the
|
||||
deployment packaging. This report describes that recipe: sequence-level KD with intermediate-layer
|
||||
feature alignment, the latency-oriented serving configuration, and the optional use of the student as
|
||||
a draft model for speculative decoding against the larger teacher. Benchmark and latency figures
|
||||
quoted here are illustrative \emph{targets} for the recipe, not validated leaderboard claims; for
|
||||
authoritative quality numbers on the underlying models we refer the reader to the Qwen2.5-Coder
|
||||
technical report \cite{qwen25coder}.
|
||||
\end{abstract}
|
||||
|
||||
\tableofcontents
|
||||
@@ -55,104 +58,118 @@ causes developers to wait or dismiss suggestions \cite{svyatkovskiy2021fast}. At
|
||||
requires generating a 10--30 token completion in well under 100ms, leaving approximately 30--50ms
|
||||
budget for network round-trip, model inference, and post-processing.
|
||||
|
||||
General-purpose code models at 13--14B parameters are too large to meet this budget on widely
|
||||
available server hardware without batching latency. Zen-Coder-Flash solves this through
|
||||
knowledge distillation from Zen-Code, compressing the 14B model into an 8B student that
|
||||
preserves code quality while fitting within the latency budget.
|
||||
General-purpose code models at 13--14B parameters are larger than necessary for this budget on widely
|
||||
available server hardware. Zen-Coder-Flash addresses this by distilling Qwen2.5-Coder-14B
|
||||
\cite{qwen25coder} into a smaller ($\sim$7--8B-class) Qwen2.5-Coder student, aiming to preserve code
|
||||
quality while reducing serving latency.
|
||||
|
||||
Key contributions:
|
||||
Scope of contributions (engineering recipe, not new science):
|
||||
|
||||
\begin{itemize}
|
||||
\item An 8B code model distilled from Zen-Code (14B) using sequence-level KD with
|
||||
intermediate-layer feature alignment.
|
||||
\item P95 time-to-first-token of 43ms on A100-80GB at batch size 1, and 850 tokens/s
|
||||
throughput at batch size 16.
|
||||
\item HumanEval 81.4\% pass@1, recovering 93\% of the teacher model's score relative to
|
||||
the student's from-scratch baseline (76.3\%).
|
||||
\item Speculative decoding compatibility: Zen-Coder-Flash serves as a draft model for
|
||||
Zen-Code in a speculative decoding setup, achieving 2.8$\times$ throughput on
|
||||
Zen-Code at minimal quality loss.
|
||||
\item A distillation recipe applying sequence-level KD with intermediate-layer feature alignment to
|
||||
a Qwen2.5-Coder teacher/student pair \cite{qwen25coder}.
|
||||
\item A latency-oriented serving configuration targeting sub-50ms P95 time-to-first-token for short
|
||||
autocomplete prefixes on a single A100-class GPU.
|
||||
\item Optional speculative decoding: the distilled student serves as a draft model for the larger
|
||||
Qwen2.5-Coder-14B teacher, a standard lossless-speedup setup
|
||||
\cite{leviathan2023speculative,chen2023accelerating}.
|
||||
\end{itemize}
|
||||
|
||||
\noindent The numeric targets in the tables below (e.g.\ P95 latency, throughput, pass@1) are design
|
||||
goals for this recipe on the stated hardware, \emph{not} independently audited benchmark results.
|
||||
Authoritative quality numbers for the Qwen2.5-Coder family are in the upstream report
|
||||
\cite{qwen25coder}; we do not restate them as our own.
|
||||
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\section{Architecture}
|
||||
|
||||
\subsection{Student Model Specification}
|
||||
|
||||
Zen-Coder-Flash uses the Zen MoDE architecture at 8B parameters. The architecture is a
|
||||
non-trivial reduction of Zen-Code rather than a simple layer-pruning: hidden dimension
|
||||
and FFN width are reduced proportionally to preserve the depth-to-width ratio associated
|
||||
with strong reasoning capability.
|
||||
Both the teacher and the student are Qwen2.5-Coder \cite{qwen25coder} checkpoints; the student is a
|
||||
smaller member of the same family (7B-class), not a bespoke ``Zen MoDE'' architecture. The
|
||||
hyperparameters below are those of the corresponding Qwen2.5-Coder checkpoints and are reproduced from
|
||||
Qwen's model cards; the depth-to-width ratio and the shared vocabulary/RoPE settings follow from using
|
||||
the same upstream family for teacher and student.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Zen-Coder-Flash architecture hyperparameters (vs.\ Zen-Code teacher).}
|
||||
\caption{Student and teacher hyperparameters (both Qwen2.5-Coder \cite{qwen25coder}).}
|
||||
\label{tab:arch}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Hyperparameter} & \textbf{Zen-Coder-Flash (8B)} & \textbf{Zen-Code (14B)} \\
|
||||
\textbf{Hyperparameter} & \textbf{Student (Qwen2.5-Coder-7B)} & \textbf{Teacher (Qwen2.5-Coder-14B)} \\
|
||||
\midrule
|
||||
Parameters (total) & 8.2B & 14.4B \\
|
||||
Layers & 36 & 40 \\
|
||||
Attention heads & 32 & 40 \\
|
||||
KV heads (GQA) & 8 & 8 \\
|
||||
Hidden dimension & 4{,}096 & 5{,}120 \\
|
||||
FFN intermediate dimension & 11{,}008 & 13{,}696 \\
|
||||
Parameters (total) & $\sim$7.6B & $\sim$14.7B \\
|
||||
Layers & 28 & 48 \\
|
||||
Hidden dimension & 3{,}584 & 5{,}120 \\
|
||||
KV heads (GQA) & 4 & 8 \\
|
||||
Vocabulary size & 151{,}936 & 151{,}936 \\
|
||||
Context length (training) & 32{,}768 & 65{,}536 \\
|
||||
Position encoding & RoPE ($\theta = 2{,}000{,}000$) & RoPE ($\theta = 2{,}000{,}000$) \\
|
||||
Context length & 32{,}768 & 32{,}768 \\
|
||||
Position encoding & RoPE ($\theta = 1{,}000{,}000$) & RoPE ($\theta = 1{,}000{,}000$) \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\noindent (Figures are the upstream Qwen2.5-Coder specifications; the exact student checkpoint chosen
|
||||
for a given deployment may differ in size, but it is in all cases an existing Qwen2.5-Coder model, not
|
||||
a newly designed architecture.)
|
||||
|
||||
\subsection{Architectural Choices for Latency}
|
||||
|
||||
The hidden dimension of 4,096 (reduced from 5,120) is a critical latency driver: the
|
||||
matrix multiplication cost of each attention projection and FFN layer scales quadratically
|
||||
with hidden dimension in memory bandwidth and linearly in FLOPs. The 20\% hidden dimension
|
||||
reduction yields approximately 36\% fewer FLOPs per layer, compounded across 36 layers.
|
||||
The student's smaller hidden dimension and lower layer count are the latency drivers: matrix-multiply
|
||||
cost per layer scales with hidden dimension, and total cost scales with layer count. Choosing a
|
||||
smaller existing Qwen2.5-Coder checkpoint as the student is what buys the latency reduction; we do not
|
||||
re-architect the model.
|
||||
|
||||
GQA with 8 KV heads (identical to Zen-Code) is retained, yielding minimal KV cache
|
||||
requirements that enable large effective batch sizes:
|
||||
Grouped-query attention (inherited from the Qwen2.5-Coder backbone) keeps KV-cache requirements modest
|
||||
and enables large effective batch sizes. As an order-of-magnitude illustration of KV-cache size for an
|
||||
8-KV-head, 128-dim-per-head configuration:
|
||||
|
||||
\begin{equation}
|
||||
\text{KV cache (per layer)} = 2 \cdot n_{\text{kv}} \cdot d_k \cdot S \cdot 2 = 2 \cdot 8 \cdot 128 \cdot 32{,}768 \cdot 2 \approx 134 \text{ MB}
|
||||
\end{equation}
|
||||
|
||||
At 32K context, the full 36-layer KV cache requires 4.8 GB, leaving ample headroom for
|
||||
model weights (16.4 GB in BF16) on a 24 GB consumer GPU.
|
||||
A modest per-layer KV-cache footprint at 32K context leaves headroom for the BF16 weights of a
|
||||
7--8B-class student on a single 24 GB consumer GPU.
|
||||
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\section{Training Methodology}
|
||||
\section{Distillation Methodology}
|
||||
|
||||
\subsection{Overview}
|
||||
|
||||
Zen-Coder-Flash training proceeds in three phases:
|
||||
Because both teacher and student are existing Qwen2.5-Coder \cite{qwen25coder} checkpoints, there is
|
||||
\textbf{no from-scratch pretraining phase}. The recipe is two steps applied on top of the upstream
|
||||
weights:
|
||||
|
||||
\begin{enumerate}
|
||||
\item \textbf{Warm-up pretraining}: Train on 500B code tokens from scratch with standard
|
||||
next-token and FIM objectives to initialize parameters.
|
||||
\item \textbf{Knowledge distillation}: Distill from Zen-Code using combined token-level
|
||||
KD and intermediate feature alignment losses.
|
||||
\item \textbf{Instruction fine-tuning}: SFT on the same 800K instruction-code pairs as
|
||||
Zen-Code.
|
||||
\item \textbf{Knowledge distillation}: starting from the pretrained Qwen2.5-Coder student checkpoint,
|
||||
distill from the Qwen2.5-Coder-14B teacher using combined token-level KD and
|
||||
intermediate-feature-alignment losses.
|
||||
\item \textbf{Instruction fine-tuning}: a light SFT pass on instruction-code pairs to restore
|
||||
instruction-following after the KD pass.
|
||||
\end{enumerate}
|
||||
|
||||
\subsection{Phase 1: Warm-Up Pretraining}
|
||||
\noindent Earlier drafts described a ``Zen MoDE distillation pipeline'' and a 500B-token from-scratch
|
||||
warm-up over a proprietary 2.5T corpus. Neither is accurate: the student is not trained from scratch
|
||||
(it is an existing Qwen2.5-Coder checkpoint) and there is no proprietary multi-trillion-token corpus
|
||||
behind this work. Those claims have been removed.
|
||||
|
||||
The student is initialized randomly and pretrained on 500B code tokens---a reduced subset
|
||||
of Zen-Code's 2.5T corpus---using standard cross-entropy loss. This warm-up prevents cold
|
||||
start instability in the subsequent KD phase, where the student would otherwise produce
|
||||
very poor initial approximations of the teacher's distributions.
|
||||
\subsection{Initialization}
|
||||
|
||||
The student is \emph{initialized from a pretrained Qwen2.5-Coder checkpoint}, not randomly initialized.
|
||||
This is precisely why no warm-up pretraining is needed: the student already has strong code priors from
|
||||
Qwen's upstream pretraining, so distillation begins from a competent starting point rather than from
|
||||
poor initial approximations of the teacher.
|
||||
|
||||
\paragraph{Distillation-run settings (illustrative).}
|
||||
\begin{itemize}
|
||||
\item Optimizer: AdamW, LR $3 \times 10^{-4}$, cosine to $3 \times 10^{-5}$
|
||||
\item Optimizer: AdamW, LR $1 \times 10^{-4}$, cosine to $5 \times 10^{-6}$
|
||||
\item Batch: 2M tokens
|
||||
\item FIM rate: 50\%
|
||||
\item Duration: 250K steps on 256 H100 GPUs
|
||||
\item FIM rate: 50\% (Qwen2.5-Coder FIM format)
|
||||
\item Scale: tens of billions of distillation tokens (orders of magnitude below a pretraining run)
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Phase 2: Knowledge Distillation}
|
||||
\subsection{Knowledge Distillation}
|
||||
|
||||
The distillation loss combines three terms:
|
||||
|
||||
@@ -180,16 +197,16 @@ the teacher logits before KL computation to reduce noise from near-zero probabil
|
||||
|
||||
\paragraph{Intermediate feature alignment ($\mathcal{L}_{\text{feat}}$).}
|
||||
We align intermediate hidden states between the student and teacher at evenly spaced layers
|
||||
using a learned linear projection $W_{\text{proj}}$ mapping the student's 4096-dim hidden
|
||||
states to the teacher's 5120-dim space:
|
||||
using a learned linear projection $W_{\text{proj}}$ mapping the student's hidden dimension
|
||||
($d_s = 3584$ for the 7B student) to the teacher's ($d_t = 5120$):
|
||||
|
||||
\begin{equation}
|
||||
\mathcal{L}_{\text{feat}} = \frac{1}{|M|} \sum_{(i,j) \in M} \left\| h^s_i W_{\text{proj}} - h^t_j \right\|_2^2
|
||||
\end{equation}
|
||||
|
||||
where $M$ is a mapping from student layers $\{6, 12, 18, 24, 30, 36\}$ to teacher layers
|
||||
$\{5, 10, 15, 20, 28, 35\}$. The projection matrix $W_{\text{proj}}$ is trained jointly
|
||||
with the student.
|
||||
where $M$ maps a set of evenly spaced student layers to evenly spaced teacher layers (the student has
|
||||
fewer layers than the teacher, so the mapping is many-to-fewer). The projection matrix $W_{\text{proj}}$
|
||||
is trained jointly with the student.
|
||||
|
||||
Loss coefficients: $\alpha = 0.3$, $\beta = 0.5$, $\gamma = 0.2$.
|
||||
|
||||
@@ -208,16 +225,16 @@ Top-$k$ filtering & 50 \\
|
||||
CE coefficient $\alpha$ & 0.3 \\
|
||||
KL coefficient $\beta$ & 0.5 \\
|
||||
Feature coefficient $\gamma$ & 0.2 \\
|
||||
Duration & 500B tokens (250K steps) \\
|
||||
Scale & distillation-scale (tens of billions of tokens) \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Speculative Decoding Setup}
|
||||
|
||||
Zen-Coder-Flash serves as a draft model in a speculative decoding pipeline with Zen-Code
|
||||
as the verifier. The draft model generates $\gamma = 6$ speculative tokens per step;
|
||||
the verifier accepts or rejects in parallel.
|
||||
The distilled student can serve as a draft model in a speculative decoding pipeline with the
|
||||
Qwen2.5-Coder-14B teacher \cite{qwen25coder} as the verifier. The draft model generates $\gamma = 6$
|
||||
speculative tokens per step; the verifier accepts or rejects in parallel.
|
||||
|
||||
Expected speedup with acceptance rate $\alpha$:
|
||||
|
||||
@@ -225,120 +242,54 @@ Expected speedup with acceptance rate $\alpha$:
|
||||
\text{Speedup} = \frac{1 + \alpha + \alpha^2 + \cdots + \alpha^\gamma}{\text{(1 verifier step cost)} / \text{(draft step cost)}}
|
||||
\end{equation}
|
||||
|
||||
Measured acceptance rate on HumanEval completions: 0.82, yielding a theoretical speedup
|
||||
of $3.1\times$ and observed wall-clock speedup of $2.8\times$ on 1$\times$ A100.
|
||||
A typical draft-model acceptance rate for code completion is in the 0.7--0.85 range; at the upper end
|
||||
this corresponds to a single-host wall-clock speedup of roughly $2$--$3\times$ over running the teacher
|
||||
alone, consistent with the speculative-decoding literature \cite{leviathan2023speculative,chen2023accelerating}.
|
||||
The exact figure is workload-dependent and should be measured per deployment.
|
||||
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\section{Evaluation}
|
||||
\section{Expected Performance (Targets)}
|
||||
|
||||
\subsection{Code Quality Benchmarks}
|
||||
\textbf{The numbers in this section are design targets for the recipe, not audited benchmark results.}
|
||||
We have not run an independent leaderboard evaluation of the distilled student, and we do not claim to
|
||||
beat the upstream Qwen2.5-Coder models. For authoritative code-quality numbers on the teacher and on
|
||||
same-size Qwen2.5-Coder checkpoints, see the Qwen2.5-Coder technical report \cite{qwen25coder}; those
|
||||
upstream numbers bound what a distilled student can be expected to recover.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Zen-Coder-Flash benchmark results versus comparable models.}
|
||||
\label{tab:benchmarks}
|
||||
\begin{tabular}{lccccc}
|
||||
\toprule
|
||||
\textbf{Benchmark} & \textbf{Flash (8B)} & \textbf{Zen-Code (14B)} & \textbf{Recovery} & \textbf{Comp.\ A (7B)} & \textbf{Comp.\ B (8B)} \\
|
||||
\midrule
|
||||
HumanEval (pass@1) & 81.4 & 87.2 & 93.4\% & 76.3 & 78.1 \\
|
||||
HumanEval$+$ (pass@1) & 76.8 & 82.4 & 93.2\% & 71.2 & 73.4 \\
|
||||
MBPP (pass@1) & 76.4 & 82.3 & 92.8\% & 70.8 & 73.1 \\
|
||||
MultiPL-E (avg) & 66.8 & 72.4 & 92.3\% & 61.2 & 63.7 \\
|
||||
\midrule
|
||||
FIM single-line & 79.1 & 82.3 & 96.1\% & 72.4 & 75.8 \\
|
||||
FIM multi-line & 57.3 & 61.4 & 93.3\% & 51.7 & 54.2 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
\subsection{Code Quality (Target Recovery)}
|
||||
|
||||
\subsection{Latency Benchmarks}
|
||||
The aim of distillation is to recover most of the teacher's code-quality on the smaller student. As a
|
||||
rough target we aim for $\sim$90\%+ recovery of the teacher's pass@1 on HumanEval/MBPP; the student
|
||||
cannot exceed the teacher, and a same-size Qwen2.5-Coder checkpoint is the relevant floor. We report no
|
||||
fabricated head-to-head table here; per-deployment numbers should be measured against the upstream
|
||||
Qwen2.5-Coder baselines \cite{qwen25coder}.
|
||||
|
||||
Latency is measured on a single A100-80GB GPU, BF16 precision, with vLLM v0.4
|
||||
\cite{kwon2023efficient} serving, batch size 1.
|
||||
\subsection{Latency (Target)}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Latency benchmark (single A100-80GB, BF16, batch=1).}
|
||||
\label{tab:latency}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{TTFT P50 (ms)} & \textbf{TTFT P95 (ms)} & \textbf{TTFT P99 (ms)} \\
|
||||
\midrule
|
||||
Zen-Coder-Flash (8B) & 27 & 43 & 61 \\
|
||||
Zen-Code (14B) & 52 & 89 & 124 \\
|
||||
Competitor A (7B) & 24 & 40 & 57 \\
|
||||
Competitor B (13B) & 48 & 82 & 118 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
The serving goal is sub-50ms P95 time-to-first-token for short ($\sim$512-token) autocomplete prefixes
|
||||
on a single A100-class GPU in BF16 with a paged-attention server \cite{kwon2023efficient}, which a
|
||||
7--8B-class model can plausibly meet where the 14B teacher cannot. These are engineering targets to be
|
||||
validated on the deployment hardware, not measured results reproduced here.
|
||||
|
||||
Time-to-first-token (TTFT) is measured with a 512-token prefix (typical autocomplete context).
|
||||
Zen-Coder-Flash's P95 of 43ms satisfies the $<$50ms IDE latency budget.
|
||||
\subsection{Speculative Decoding (Expected)}
|
||||
|
||||
\subsection{Throughput Benchmarks}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Generation throughput (tokens/second) on single A100-80GB.}
|
||||
\label{tab:throughput}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{Batch=1} & \textbf{Batch=8} & \textbf{Batch=16} \\
|
||||
\midrule
|
||||
Zen-Coder-Flash (8B) & 312 & 640 & \textbf{850} \\
|
||||
Zen-Code (14B) & 148 & 298 & 413 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
At batch size 16 (typical for shared inference serving), Zen-Coder-Flash delivers 850 tok/s,
|
||||
enabling simultaneous low-latency completions for large developer teams from a single GPU.
|
||||
|
||||
\subsection{Speculative Decoding Performance}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Speculative decoding (Zen-Coder-Flash draft + Zen-Code verifier).}
|
||||
\label{tab:spec_dec}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Configuration} & \textbf{Tok/s} & \textbf{Speedup vs. Zen-Code} & \textbf{HumanEval} \\
|
||||
\midrule
|
||||
Zen-Code alone & 148 & 1.0$\times$ & 87.2 \\
|
||||
Spec-Dec ($\gamma=4$) & 312 & 2.1$\times$ & 87.1 \\
|
||||
Spec-Dec ($\gamma=6$) & 414 & 2.8$\times$ & 87.0 \\
|
||||
Spec-Dec ($\gamma=8$) & 431 & 2.9$\times$ & 86.8 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
$\gamma=6$ provides the best throughput/quality trade-off, recovering 99.8\% of Zen-Code's
|
||||
HumanEval at 2.8$\times$ throughput with no quality degradation (speculative decoding is
|
||||
mathematically lossless modulo floating-point differences in acceptance sampling).
|
||||
Using the distilled student as a draft model for the Qwen2.5-Coder-14B teacher is expected to yield a
|
||||
$2$--$3\times$ wall-clock speedup over the teacher alone at the teacher's exact output distribution,
|
||||
since speculative decoding is lossless modulo floating-point differences in acceptance sampling
|
||||
\cite{leviathan2023speculative,chen2023accelerating}. Realized speedup depends on the acceptance rate
|
||||
for the target workload and must be measured.
|
||||
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\section{Distillation Ablation}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Ablation of distillation loss components on HumanEval and FIM benchmarks.}
|
||||
\label{tab:ablation}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Loss configuration} & \textbf{HumanEval} & \textbf{FIM single-line} & \textbf{Latency P95} \\
|
||||
\midrule
|
||||
CE only (no KD) & 76.3 & 68.4 & 43ms \\
|
||||
CE $+$ KL & 79.8 & 75.1 & 43ms \\
|
||||
CE $+$ feature align & 78.4 & 73.6 & 43ms \\
|
||||
CE $+$ KL $+$ feature align & \textbf{81.4} & \textbf{79.1} & 43ms \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
Both KL divergence and feature alignment contribute independently: KL improves token-level
|
||||
distribution matching (+3.5pp HumanEval), while feature alignment improves hidden state
|
||||
quality (+2.1pp), with the combined loss exceeding either alone (+5.1pp over CE-only).
|
||||
We do not report a table of measured pass@1 values for each loss configuration, as we have not run an
|
||||
audited evaluation to support specific numbers. Qualitatively, and consistent with the distillation
|
||||
literature \cite{hinton2015distilling,kim2016seq,gu2024minillm,jiao2020tinybert}, we expect both the
|
||||
KL-divergence term (token-level distribution matching) and the intermediate-feature-alignment term
|
||||
(hidden-state matching) to contribute independently over a cross-entropy-only baseline, with the
|
||||
combined loss outperforming either term alone. Latency is unaffected by the choice of distillation loss
|
||||
(it changes training, not the served architecture). Any quantitative ablation should be reported with
|
||||
the deployment's own measured numbers.
|
||||
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\section{IDE Integration and Deployment}
|
||||
@@ -356,10 +307,10 @@ docker run --gpus=1 \
|
||||
-p 8080:8080 \
|
||||
ghcr.io/zenlm/inference:latest
|
||||
|
||||
# Speculative decoding with Zen-Code verifier (2x A100)
|
||||
# Speculative decoding with Qwen2.5-Coder-14B verifier (2x A100)
|
||||
docker run --gpus=2 \
|
||||
-e MODEL=zen-code-14b \
|
||||
-e DRAFT_MODEL=zen-coder-flash-8b \
|
||||
-e MODEL=Qwen/Qwen2.5-Coder-14B-Instruct \
|
||||
-e DRAFT_MODEL=zen-coder-flash \
|
||||
-e SPECULATIVE_GAMMA=6 \
|
||||
-p 8080:8080 \
|
||||
ghcr.io/zenlm/inference:latest
|
||||
@@ -396,37 +347,44 @@ sequence-level KD \cite{kim2016seq} and token-level KL distillation \cite{gu2024
|
||||
have emerged as standard approaches. Intermediate layer alignment follows the TinyBERT
|
||||
\cite{jiao2020tinybert} and PKD \cite{sun2019patient} patterns. Speculative decoding
|
||||
\cite{leviathan2023speculative, chen2023accelerating} enables lossless speedup using a small
|
||||
draft model; our use of Zen-Coder-Flash as draft for Zen-Code follows this paradigm.
|
||||
draft model; our use of the distilled student as a draft for the Qwen2.5-Coder-14B teacher
|
||||
\cite{qwen25coder} follows this paradigm.
|
||||
|
||||
Fast code completion models have been explored in FauxPilot and similar systems; the key
|
||||
contribution of Zen-Coder-Flash is pairing distillation-preserved code quality with the
|
||||
latency targets required for production IDE deployment.
|
||||
The base models themselves are Qwen2.5-Coder \cite{qwen25coder}; Zen-Coder-Flash contributes a
|
||||
distillation recipe and a latency-oriented serving configuration on top of those open-weight models,
|
||||
not a new model. Fast code completion has also been explored in systems such as FauxPilot.
|
||||
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\section{Limitations}
|
||||
|
||||
The 8B student model recovers 93\% of the teacher's HumanEval score but the 7\% gap widens
|
||||
on more complex tasks: SWE-bench recovery is approximately 68\% (19.3\% vs.\ 28.4\%), as
|
||||
complex multi-file reasoning benefits disproportionately from the teacher's additional capacity.
|
||||
The 32K context window (vs.\ Zen-Code's 64K) limits use in very large repository-level
|
||||
tasks. Speculative decoding latency benefits are reduced under variable-length acceptance
|
||||
patterns typical of code generation (vs.\ more predictable text completion).
|
||||
A distilled student is bounded by its teacher: it cannot exceed Qwen2.5-Coder-14B
|
||||
\cite{qwen25coder}, and the gap typically widens on complex multi-file reasoning tasks (e.g.\
|
||||
SWE-bench), which benefit disproportionately from the teacher's additional capacity. The student's
|
||||
context window is bounded by the chosen Qwen2.5-Coder checkpoint. Speculative-decoding speedups are
|
||||
reduced under the variable-length acceptance patterns typical of code generation. Finally, the quality
|
||||
and latency figures in this report are \emph{targets}, not audited measurements; deployments should
|
||||
validate against the upstream Qwen2.5-Coder baselines and their own hardware.
|
||||
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\section{Conclusion}
|
||||
|
||||
Zen-Coder-Flash delivers IDE-grade code completion with P95 latency of 43ms and 850 tok/s
|
||||
throughput at batch size 16 on a single A100 GPU, while maintaining HumanEval 81.4\% through
|
||||
structured knowledge distillation from Zen-Code. The combined KL-divergence and
|
||||
intermediate-feature-alignment distillation protocol recovers 93\% of teacher performance
|
||||
at 55\% of parameter count and 2.1$\times$ lower latency. As a speculative decoding draft
|
||||
model for Zen-Code, it additionally provides 2.8$\times$ throughput improvement with
|
||||
negligible quality loss. Zen-Coder-Flash is the recommended model for all latency-critical
|
||||
developer tooling integrations in the Zen family ecosystem.
|
||||
Zen-Coder-Flash is a smaller, lower-latency code-completion student obtained by knowledge distillation
|
||||
from Alibaba's open-weight Qwen2.5-Coder-14B \cite{qwen25coder} into a smaller Qwen2.5-Coder checkpoint.
|
||||
It is not a from-scratch model and does not use a ``Zen MoDE'' architecture; the teacher, the student
|
||||
backbone, the tokenizer, and the FIM format are all Qwen's, and the work is released under the upstream
|
||||
Apache-2.0 license with attribution. Our contribution is the distillation recipe (sequence-level KD plus
|
||||
intermediate-feature alignment), a latency-oriented serving configuration, and the option to use the
|
||||
student as a speculative-decoding draft for the teacher. The latency and quality figures here are design
|
||||
targets for that recipe, to be validated per deployment against the upstream Qwen2.5-Coder baselines.
|
||||
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\begin{thebibliography}{99}
|
||||
|
||||
\bibitem{qwen25coder}
|
||||
B.~Hui, J.~Yang, et al.\ (Qwen Team, Alibaba), ``Qwen2.5-Coder Technical Report,''
|
||||
\textit{arXiv:2409.12186}, 2024. Apache-2.0 (0.5B/1.5B/7B/14B/32B).
|
||||
\url{https://github.com/QwenLM/Qwen2.5-Coder}.
|
||||
|
||||
\bibitem{svyatkovskiy2021fast}
|
||||
A.~Svyatkovskiy et al., ``Fast and Low-Cost Code Intelligence with Lightweight Models,''
|
||||
\textit{arXiv:2104.14765}, 2021.
|
||||
|
||||
Binary file not shown.
+151
-208
@@ -62,12 +62,16 @@
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
We present \textbf{Zen Coder}, a family of agentic code generation models ranging from 4B to 1T parameters,
|
||||
trained on the \textbf{Zen Agentic Dataset}---8.47 billion tokens of real-world Claude Code sessions,
|
||||
git history, and professional software development spanning 15 years across 1,452 repositories.
|
||||
Unlike synthetic datasets, our training data captures actual debugging workflows, multi-file refactoring decisions,
|
||||
tool use patterns, and error recovery from production AI development. The model family includes dense models
|
||||
(4B, 24B, 123B) and MoE architectures (358B, 1T), covering edge deployment to frontier capabilities.
|
||||
\textbf{Zen Coder} is a packaging and light fine-tuning of Alibaba's open-weight
|
||||
\textbf{Qwen Coder} models---\textbf{Qwen2.5-Coder} \cite{qwen25coder} for the dense
|
||||
tiers and \textbf{Qwen3-Coder} \cite{qwen3coder} for the Mixture-of-Experts tiers---redistributed
|
||||
under the Zen brand for agentic coding workflows. It is \emph{not} a from-scratch model and does not
|
||||
introduce a new architecture or pretraining corpus; the upstream weights, tokenizer, and
|
||||
architecture are Qwen's, the pretraining (5.5T+ code tokens for Qwen2.5-Coder) was done by the
|
||||
Qwen team, and we contribute only deployment packaging and optional task-specific fine-tuning.
|
||||
The Apache-2.0--licensed Qwen Coder checkpoints are redistributed under Apache-2.0 with attribution
|
||||
to Alibaba. This report documents which upstream checkpoint backs each Zen Coder tier, the
|
||||
fine-tuning configuration we apply, and the intended deployment surface.
|
||||
\end{abstract}
|
||||
|
||||
\tableofcontents
|
||||
@@ -76,178 +80,118 @@ tool use patterns, and error recovery from production AI development. The model
|
||||
\section{Introduction}
|
||||
|
||||
The emergence of agentic AI systems---where models interact with tools, execute multi-step plans, and
|
||||
maintain context across complex workflows---has created demand for models specifically trained on
|
||||
real agentic programming patterns. \textbf{Zen Coder} addresses this by training on actual Claude Code
|
||||
debug sessions rather than synthetic instruction-following data.
|
||||
maintain context across complex workflows---has created demand for capable open-weight code models.
|
||||
The Qwen team at Alibaba released two such families under permissive terms: \textbf{Qwen2.5-Coder}
|
||||
\cite{qwen25coder}, a set of dense code models (0.5B--32B) continued-pretrained on over 5.5 trillion
|
||||
code tokens, and \textbf{Qwen3-Coder} \cite{qwen3coder}, a Mixture-of-Experts agentic-coding model
|
||||
(480B total / 35B active). \textbf{Zen Coder} is a redistribution of these upstream checkpoints under
|
||||
the Zen brand, with optional task-specific fine-tuning. We make no claim to having trained these
|
||||
models from scratch; all pretraining, the architecture, and the tokenizer are Qwen's.
|
||||
|
||||
\subsection{Model Family}
|
||||
|
||||
Each Zen Coder tier is backed directly by a published Qwen Coder checkpoint, as shown below. ``Base''
|
||||
names the exact upstream model; ``status'' indicates whether we currently ship a fine-tuned Zen
|
||||
variant or redistribute the upstream weights unchanged.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{llllll}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{Size} & \textbf{Base} & \textbf{VRAM} & \textbf{Context} & \textbf{Status} \\
|
||||
\textbf{Zen tier} & \textbf{Size} & \textbf{Upstream base} & \textbf{VRAM} & \textbf{Context} & \textbf{Status} \\
|
||||
\midrule
|
||||
Zen Coder 4B & 4B & Zen-4B-Instruct & 8 GB & 32K & Trained \\
|
||||
Zen Coder 24B & 24B & Zen Coder 24B-Instruct & 24 GB & 256K & Trained \\
|
||||
Zen Coder 123B & 123B & Zen Coder 123B & 128 GB & 256K & Training \\
|
||||
Zen Coder Max & 358B (MoE) & Zen Coder Max & 180 GB & 200K & Planned \\
|
||||
Zen Coder Ultra & 1T (MoE) & Zen MoDE Ultra & 256 GB & 128K & Planned \\
|
||||
Zen Coder 3B & 3B & Qwen2.5-Coder-3B-Instruct & 8 GB & 32K & Fine-tuned \\
|
||||
Zen Coder 7B & 7B & Qwen2.5-Coder-7B-Instruct & 16 GB & 128K & Fine-tuned \\
|
||||
Zen Coder 14B & 14B & Qwen2.5-Coder-14B-Instruct & 32 GB & 128K & Fine-tuned \\
|
||||
Zen Coder 32B & 32B & Qwen2.5-Coder-32B-Instruct & 64 GB & 128K & Fine-tuned \\
|
||||
Zen Coder MoE & 480B (A35B) & Qwen3-Coder-480B-A35B-Instruct & 8$\times$H100 & 256K & Repackaged \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen Coder Model Family}
|
||||
\caption{Zen Coder tiers and their upstream Qwen Coder bases. Qwen3-Coder supports 256K context
|
||||
natively and up to 1M via extrapolation \cite{qwen3coder}.}
|
||||
\end{table}
|
||||
|
||||
\subsection{Key Innovations}
|
||||
\subsection{What Zen Coder Adds (and Does Not)}
|
||||
\begin{itemize}
|
||||
\item \textbf{Real Agentic Data}: Trained on actual Claude Code sessions, not synthetic instruction data
|
||||
\item \textbf{Production Code}: 15 years of battle-tested software across AI, Web3, cryptography
|
||||
\item \textbf{Multi-Architecture}: Dense (4B-123B) and MoE (358B-1T) models for different deployment needs
|
||||
\item \textbf{Open Training}: Full training framework available via \href{https://github.com/zenlm/zen-trainer}{zen-trainer}
|
||||
\item \textbf{Packaging}: distribution under the \texttt{zenlm} namespace with Zen deployment
|
||||
tooling and serving defaults; the weights remain Qwen's.
|
||||
\item \textbf{Optional fine-tuning}: light LoRA fine-tuning on task-specific data for some tiers
|
||||
(Section~\ref{sec:training}); the MoE tier is redistributed unchanged.
|
||||
\item \textbf{Attribution}: Apache-2.0 upstream checkpoints are redistributed under Apache-2.0
|
||||
with attribution to Alibaba/Qwen; no new pretraining corpus is introduced.
|
||||
\end{itemize}
|
||||
|
||||
\section{Training Data: Zen Agentic Dataset}
|
||||
\noindent\textbf{Not claimed.} Zen Coder does not introduce a novel architecture, a from-scratch
|
||||
pretraining run, or a proprietary training dataset. Earlier drafts of this report described a ``Zen
|
||||
Agentic Dataset'' of billions of tokens of real Claude Code sessions and a ``Zen MoDE'' architecture;
|
||||
no such dataset or architecture backs these models, and those claims have been removed.
|
||||
|
||||
The \textbf{Zen Agentic Dataset} comprises 8.47 billion tokens derived from real software development
|
||||
and Claude Code interactions, distinguishing it from synthetic datasets.
|
||||
\section{Upstream Training and Data}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lrrr}
|
||||
\toprule
|
||||
\textbf{Data Source} & \textbf{Tokens (B)} & \textbf{\%} & \textbf{Description} \\
|
||||
\midrule
|
||||
Git History & 4.03 & 48\% & 15 years of commits, diffs, source files \\
|
||||
Claude Debug Sessions & 2.42 & 29\% & Real debugging workflows with tool use \\
|
||||
Claude Conversations & 1.14 & 13\% & Architecture discussions, code reviews \\
|
||||
Claude Interactions & 0.86 & 10\% & Multi-turn coding assistance \\
|
||||
\midrule
|
||||
\textbf{Total} & \textbf{8.47} & 100\% & 3.35M training samples \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen Agentic Dataset Composition}
|
||||
\end{table}
|
||||
\textbf{Zen Coder introduces no pretraining corpus of its own.} All pretraining was performed by the
|
||||
Qwen team and is documented in their technical reports; we summarize it here only for completeness and
|
||||
direct the reader to the upstream sources for authoritative detail.
|
||||
|
||||
\subsection{Dataset Statistics}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{ll}
|
||||
\toprule
|
||||
\textbf{Metric} & \textbf{Value} \\
|
||||
\midrule
|
||||
Total Tokens & 8.47 billion \\
|
||||
Training Samples & 3.35 million \\
|
||||
Validation Samples & 100,000 \\
|
||||
Total Size & 27 GB \\
|
||||
Repositories & 1,452 \\
|
||||
Time Span & 15 years (2010-2025) \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Dataset Statistics}
|
||||
\end{table}
|
||||
|
||||
\subsection{Domain Coverage}
|
||||
|
||||
\subsubsection{Agentic AI \& LLM Infrastructure}
|
||||
\begin{itemize}
|
||||
\item Model Context Protocol (MCP) - 260+ tool implementations
|
||||
\item Multi-agent orchestration - Claude, GPT-4, Gemini, Zen integrations
|
||||
\item Agent frameworks - Planning, memory, tool use, reflection
|
||||
\item LLM Gateway - Unified proxy for 100+ providers
|
||||
\item \textbf{Qwen2.5-Coder} (dense tiers) is built on the Qwen2.5 architecture and
|
||||
continued-pretrained on a corpus of more than 5.5 trillion tokens with data cleaning,
|
||||
synthetic data generation, and balanced data mixing, as described in the Qwen2.5-Coder
|
||||
Technical Report \cite{qwen25coder}. The composition and provenance of that corpus are
|
||||
Qwen's; we do not have, and do not assert, any further detail.
|
||||
\item \textbf{Qwen3-Coder} (MoE tier) is an agentic-coding MoE trained by Qwen with long-horizon
|
||||
reinforcement learning on software-engineering benchmarks \cite{qwen3coder}.
|
||||
\end{itemize}
|
||||
|
||||
\subsubsection{Web3 \& Blockchain}
|
||||
\begin{itemize}
|
||||
\item Smart contracts - Solidity, Vyper (ERC20, ERC721, ERC1155, DeFi)
|
||||
\item Consensus engines - Snow family, BFT, DAG-based protocols
|
||||
\item Cross-chain bridges - Multi-VM architecture
|
||||
\item DeFi protocols - AMMs, lending, staking, governance
|
||||
\end{itemize}
|
||||
|
||||
\subsubsection{Cryptography \& Security}
|
||||
\begin{itemize}
|
||||
\item Post-quantum cryptography - Kyber, Dilithium, SPHINCS+
|
||||
\item Threshold cryptography - MPC, secret sharing, DKG
|
||||
\item Zero-knowledge proofs - zkSNARKs, zkSTARKs experimentation
|
||||
\item Key management - HD wallets, hardware integration
|
||||
\end{itemize}
|
||||
|
||||
\subsubsection{Modern Development}
|
||||
\begin{itemize}
|
||||
\item Full-stack TypeScript - Next.js 14+, React 18+, Node.js
|
||||
\item Systems programming - Rust, Go, Python, C/C++
|
||||
\item DevOps - Docker, Kubernetes, CI/CD pipelines
|
||||
\item Real-time systems - Event sourcing, CQRS, message queues
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Language Distribution}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lll}
|
||||
\toprule
|
||||
\textbf{Tier 1 (Core)} & \textbf{Tier 2 (Infrastructure)} & \textbf{Tier 3 (Specialized)} \\
|
||||
\midrule
|
||||
Python & SQL & Solidity \\
|
||||
TypeScript & Bash/Shell & C/C++ \\
|
||||
JavaScript & YAML/TOML & Protobuf \\
|
||||
Rust & Dockerfile & GraphQL \\
|
||||
Go & Makefile & Move \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Language Coverage by Tier}
|
||||
\end{table}
|
||||
\noindent Any task-specific fine-tuning we apply on top of these checkpoints (Section~\ref{sec:training})
|
||||
uses small, internally curated instruction sets; it is a fine-tune, not a pretraining run, and the
|
||||
volume is negligible relative to the upstream corpora above.
|
||||
|
||||
\section{Model Architectures}
|
||||
|
||||
\subsection{Dense Models (4B, 24B, 123B)}
|
||||
The architectures below are inherited unchanged from the corresponding Qwen Coder upstream
|
||||
checkpoints; the figures are reproduced from Qwen's model cards and reports
|
||||
\cite{qwen25coder,qwen3coder}.
|
||||
|
||||
The smaller models use dense transformer architectures optimized for different deployment scenarios:
|
||||
\subsection{Dense Tiers (Qwen2.5-Coder)}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{llll}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{Base Architecture} & \textbf{Layers} & \textbf{Hidden Dim} \\
|
||||
\textbf{Zen tier} & \textbf{Upstream architecture} & \textbf{Layers} & \textbf{Hidden Dim} \\
|
||||
\midrule
|
||||
Zen Coder 4B & Zen-4B-Instruct & 40 & 2560 \\
|
||||
Zen Coder 24B & Zen Coder 24B-Instruct & 56 & 5120 \\
|
||||
Zen Coder 123B & Zen Coder 123B & 80 & 8192 \\
|
||||
Zen Coder 7B & Qwen2.5-Coder-7B & 28 & 3584 \\
|
||||
Zen Coder 14B & Qwen2.5-Coder-14B & 48 & 5120 \\
|
||||
Zen Coder 32B & Qwen2.5-Coder-32B & 64 & 5120 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Dense Model Specifications}
|
||||
\caption{Dense tier architectures (inherited from Qwen2.5-Coder \cite{qwen25coder}).}
|
||||
\end{table}
|
||||
|
||||
\subsection{MoE Models (358B, 1T)}
|
||||
\subsection{MoE Tier (Qwen3-Coder)}
|
||||
|
||||
The larger models employ Mixture of Experts architectures for efficient scaling:
|
||||
The MoE tier is redistributed from Qwen3-Coder unchanged:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{llll}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{Total Params} & \textbf{Active Params} & \textbf{Experts} \\
|
||||
\textbf{Zen tier} & \textbf{Total Params} & \textbf{Active Params} & \textbf{Experts} \\
|
||||
\midrule
|
||||
Zen Coder Max & 358B & ~60B & 128 experts, top-8 \\
|
||||
Zen Coder Ultra & 1T & ~150B & 256 experts, top-8 \\
|
||||
Zen Coder MoE & 480B & 35B & 160 experts, top-8 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{MoE Model Specifications}
|
||||
\caption{MoE tier (inherited from Qwen3-Coder-480B-A35B \cite{qwen3coder}).}
|
||||
\end{table}
|
||||
|
||||
\section{Training Methodology}
|
||||
\section{Fine-Tuning Methodology}
|
||||
\label{sec:training}
|
||||
|
||||
\subsection{Training Framework}
|
||||
|
||||
We developed \href{https://github.com/zenlm/zen-trainer}{zen-trainer}, an open-source framework
|
||||
supporting multiple backends:
|
||||
|
||||
\begin{itemize}
|
||||
\item \textbf{MLX}: Apple Silicon optimization (M1/M2/M3 Pro/Max/Ultra)
|
||||
\item \textbf{Unsloth}: 2x faster NVIDIA training with memory optimization
|
||||
\item \textbf{DeepSpeed}: Multi-GPU and multi-node training
|
||||
\end{itemize}
|
||||
We do not pretrain. The only training we perform is optional parameter-efficient fine-tuning (LoRA
|
||||
\cite{lora}) of the Apache-2.0 dense Qwen2.5-Coder checkpoints on small internal instruction sets,
|
||||
using the open-source \href{https://github.com/zenlm/zen-trainer}{zen-trainer} wrapper around standard
|
||||
backends (MLX for Apple Silicon, Unsloth/DeepSpeed for NVIDIA). The Qwen3-Coder MoE tier is
|
||||
redistributed without fine-tuning.
|
||||
|
||||
\subsection{Fine-tuning Configuration}
|
||||
|
||||
@@ -255,37 +199,20 @@ supporting multiple backends:
|
||||
\centering
|
||||
\begin{tabular}{lllll}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{LoRA r} & \textbf{LoRA $\alpha$} & \textbf{Batch} & \textbf{LR} \\
|
||||
\textbf{Zen tier} & \textbf{LoRA r} & \textbf{LoRA $\alpha$} & \textbf{Batch} & \textbf{LR} \\
|
||||
\midrule
|
||||
4B & 64 & 128 & 4 & 2e-4 \\
|
||||
24B & 32 & 64 & 2 & 1e-4 \\
|
||||
123B & 16 & 32 & 1 & 5e-5 \\
|
||||
Max & 16 & 32 & 1 & 5e-6 \\
|
||||
Ultra & 8 & 16 & 1 & 1e-6 \\
|
||||
7B & 64 & 128 & 4 & 2e-4 \\
|
||||
14B & 32 & 64 & 2 & 1e-4 \\
|
||||
32B & 16 & 32 & 1 & 5e-5 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{QLoRA Fine-tuning Hyperparameters}
|
||||
\caption{LoRA fine-tuning hyperparameters for the dense tiers. The MoE tier is not fine-tuned.}
|
||||
\end{table}
|
||||
|
||||
\subsection{Training Costs}
|
||||
|
||||
For 3.35M samples (8.47B tokens) on 8xH200 @ \$35/hr:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{llll}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{Cloud Hours} & \textbf{Cloud Cost} & \textbf{Local (Mac Studio)} \\
|
||||
\midrule
|
||||
Zen Coder 4B & 9h & \$326 & 2 days (FREE) \\
|
||||
Zen Coder 24B & 23h & \$814 & 5 days (FREE) \\
|
||||
Zen Coder 123B & 62h & \$2,171 & 13 days (FREE) \\
|
||||
Zen Coder Max & 116h & \$4,071 & 19 days (FREE) \\
|
||||
Zen Coder Ultra & 310h & \$10,856 & N/A (too large) \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Training Cost Estimates}
|
||||
\end{table}
|
||||
\noindent Because this is a LoRA fine-tune over a small instruction set rather than a pretraining run,
|
||||
the compute is modest---on the order of single-GPU-hours to low tens of GPU-hours per dense tier---and
|
||||
we do not report large training-cost tables, which would misrepresent the scope of the work. The
|
||||
substantive compute cost of these models is the upstream Qwen pretraining, which we did not perform.
|
||||
|
||||
\section{Usage}
|
||||
|
||||
@@ -297,13 +224,14 @@ pip install zen-trainer
|
||||
|
||||
\subsection{Training Example}
|
||||
|
||||
\begin{lstlisting}[language=Python, caption=Fine-tuning with zen-trainer]
|
||||
\begin{lstlisting}[language=Python, caption=LoRA fine-tuning a Qwen2.5-Coder base with zen-trainer]
|
||||
from zen_trainer import ZenTrainer
|
||||
|
||||
# Base is the upstream Apache-2.0 Qwen2.5-Coder checkpoint.
|
||||
trainer = ZenTrainer(
|
||||
model_key="zen-coder-4b",
|
||||
dataset_path="hanzoai/zen-agentic-dataset-private",
|
||||
output_dir="./output/zen-coder-4b",
|
||||
base_model="Qwen/Qwen2.5-Coder-7B-Instruct",
|
||||
dataset_path="./data/your-instruction-set.jsonl",
|
||||
output_dir="./output/zen-coder-7b",
|
||||
)
|
||||
trainer.train()
|
||||
\end{lstlisting}
|
||||
@@ -314,11 +242,11 @@ trainer.train()
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
"zenlm/zen-coder-4b",
|
||||
"zenlm/zen-coder-7b",
|
||||
torch_dtype="auto",
|
||||
device_map="auto"
|
||||
)
|
||||
tokenizer = AutoTokenizer.from_pretrained("zenlm/zen-coder-4b")
|
||||
tokenizer = AutoTokenizer.from_pretrained("zenlm/zen-coder-7b")
|
||||
|
||||
messages = [
|
||||
{"role": "user", "content": "Write a Python function to validate email addresses"}
|
||||
@@ -330,51 +258,47 @@ outputs = model.generate(**inputs, max_new_tokens=512)
|
||||
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
||||
\end{lstlisting}
|
||||
|
||||
\section{What Makes This Unique}
|
||||
\section{Relationship to Upstream}
|
||||
|
||||
\subsection{Real Agentic Programming}
|
||||
Zen Coder's capabilities are, to first order, the capabilities of the underlying Qwen Coder
|
||||
checkpoints. Qwen2.5-Coder reports state-of-the-art results among open code models in its size class
|
||||
across more than ten code benchmarks \cite{qwen25coder}, and Qwen3-Coder targets agentic
|
||||
software-engineering tasks via long-horizon reinforcement learning \cite{qwen3coder}. We refer the
|
||||
reader to those reports for authoritative benchmark numbers; we do not restate or modify them here, and
|
||||
we make no claim that Zen-brand packaging or light LoRA fine-tuning improves on the upstream scores.
|
||||
|
||||
Unlike synthetic datasets, the Zen Agentic Dataset contains \textbf{actual Claude Code sessions} showing:
|
||||
\begin{itemize}
|
||||
\item Real debugging workflows with trial and error
|
||||
\item Complex multi-file refactoring decisions
|
||||
\item Architecture discussions and trade-offs
|
||||
\item Tool use patterns (file ops, search, git, tests)
|
||||
\item Error recovery and iterative refinement
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Production Code Quality}
|
||||
\begin{itemize}
|
||||
\item Code that shipped to production systems
|
||||
\item Security-audited smart contracts
|
||||
\item Performance-optimized infrastructure
|
||||
\item Battle-tested patterns from real deployments
|
||||
\end{itemize}
|
||||
What Zen Coder provides over a raw upstream download is operational: a consistent serving namespace,
|
||||
deployment defaults, and---for the dense tiers---optional fine-tunes to a customer's own instruction
|
||||
data. The value proposition is integration and convenience, not a superior base model.
|
||||
|
||||
\section{Evaluation}
|
||||
|
||||
Models are evaluated on agentic coding benchmarks including:
|
||||
\begin{itemize}
|
||||
\item \textbf{SWE-bench Verified}: Real GitHub issue resolution
|
||||
\item \textbf{TAU-Bench}: Tool-agent-user interaction
|
||||
\item \textbf{BFCL V3}: Berkeley Function Call Leaderboard
|
||||
\item \textbf{Terminal-Bench}: Terminal environment tasks
|
||||
\item \textbf{LiveCodeBench}: Real-time coding challenges
|
||||
\end{itemize}
|
||||
We do not run a separate benchmark suite for Zen Coder; the relevant numbers are the upstream Qwen
|
||||
Coder results \cite{qwen25coder,qwen3coder}, evaluated on standard agentic-coding benchmarks such as
|
||||
SWE-bench Verified, BFCL, Terminal-Bench, and LiveCodeBench. Any customer-specific fine-tune should be
|
||||
evaluated on the customer's own held-out tasks; we report such numbers per-engagement rather than as
|
||||
headline claims here.
|
||||
|
||||
\section{Licensing \& Access}
|
||||
|
||||
\subsection{Models}
|
||||
All Zen Coder tiers inherit the license of their upstream Qwen Coder checkpoint. The bases we use are
|
||||
Apache-2.0, so the redistributed Zen variants are likewise \textbf{Apache-2.0}, with attribution to
|
||||
Alibaba/Qwen retained in the model cards and NOTICE files:
|
||||
|
||||
\begin{itemize}
|
||||
\item Zen Coder 4B, 24B: Apache 2.0
|
||||
\item Zen Coder 123B: Zen Commercial License v1.0
|
||||
\item Zen Coder Max: Zen Commercial License v1.0
|
||||
\item Zen Coder Ultra: Zen Commercial License v1.0
|
||||
\item Zen Coder 7B / 14B / 32B: Apache 2.0 (from Qwen2.5-Coder-7B/14B/32B \cite{qwen25coder}).
|
||||
\item Zen Coder MoE: Apache 2.0 (from Qwen3-Coder-480B-A35B \cite{qwen3coder}).
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Dataset Access}
|
||||
The Zen Agentic Dataset is available for research and commercial licensing.
|
||||
Contact \texttt{z@hanzo.ai} for access.
|
||||
\noindent We do not relicense these models under a proprietary ``Zen Commercial License''; Apache-2.0
|
||||
upstream weights are redistributed under Apache-2.0. (Note: the Qwen2.5-Coder-3B checkpoint carries the
|
||||
separate Qwen-Research license upstream and is therefore not offered as an Apache-2.0 Zen tier.)
|
||||
|
||||
\subsection{Fine-tuning Data}
|
||||
Zen Coder ships with no proprietary pretraining dataset. The earlier ``Zen Agentic Dataset'' described
|
||||
in prior drafts does not exist as characterized and has been removed from this report. Customer
|
||||
fine-tunes use the customer's own data.
|
||||
|
||||
\section{Supported Organizations}
|
||||
|
||||
@@ -395,23 +319,42 @@ Lux Network & AI compute settlement & Infrastructure \\
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
Zen Coder represents a new approach to code generation model training: using real agentic programming
|
||||
data rather than synthetic instructions. By training on actual Claude Code sessions from 15 years of
|
||||
production software development, these models learn genuine debugging patterns, tool use workflows,
|
||||
and the iterative nature of real programming.
|
||||
|
||||
The complete model family---from 4B for edge deployment to 1T for frontier capabilities---provides
|
||||
options for every use case. The open training framework (\href{https://github.com/zenlm/zen-trainer}{zen-trainer})
|
||||
enables the community to train custom models on their own data.
|
||||
Zen Coder is a redistribution---and, for the dense tiers, an optional light fine-tune---of Alibaba's
|
||||
open-weight Qwen Coder models: Qwen2.5-Coder \cite{qwen25coder} for the dense tiers and Qwen3-Coder
|
||||
\cite{qwen3coder} for the MoE tier. It is not a from-scratch model, introduces no new architecture or
|
||||
pretraining corpus, and is offered under the upstream Apache-2.0 license with attribution to Qwen. The
|
||||
value of the Zen packaging is operational---consistent serving and the ability to fine-tune a strong
|
||||
open base to a customer's own data---rather than a claim of independent model capability.
|
||||
|
||||
\section*{Links}
|
||||
|
||||
\begin{itemize}
|
||||
\item Models: \href{https://huggingface.co/zenlm}{huggingface.co/zenlm}
|
||||
\item Dataset: \href{https://huggingface.co/datasets/hanzoai/zen-agentic-dataset}{huggingface.co/datasets/hanzoai/zen-agentic-dataset}
|
||||
\item Training: \href{https://github.com/zenlm/zen-trainer}{github.com/zenlm/zen-trainer}
|
||||
\item Zen packaging: \href{https://huggingface.co/zenlm}{huggingface.co/zenlm}
|
||||
\item Upstream (dense): \href{https://huggingface.co/Qwen}{Qwen2.5-Coder} \cite{qwen25coder}
|
||||
\item Upstream (MoE): \href{https://huggingface.co/Qwen}{Qwen3-Coder-480B-A35B} \cite{qwen3coder}
|
||||
\item Fine-tuning wrapper: \href{https://github.com/zenlm/zen-trainer}{github.com/zenlm/zen-trainer}
|
||||
\item Website: \href{https://zenlm.org}{zenlm.org}
|
||||
\item Contact: \href{mailto:z@hanzo.ai}{z@hanzo.ai}
|
||||
\end{itemize}
|
||||
|
||||
\begin{thebibliography}{9}
|
||||
|
||||
\bibitem{qwen25coder}
|
||||
B.~Hui, J.~Yang, et al.\ (Qwen Team, Alibaba).
|
||||
\newblock Qwen2.5-Coder Technical Report.
|
||||
\newblock {\em arXiv preprint arXiv:2409.12186}, 2024. Apache-2.0 (0.5B/1.5B/7B/14B/32B).
|
||||
\newblock \url{https://github.com/QwenLM/Qwen2.5-Coder}.
|
||||
|
||||
\bibitem{qwen3coder}
|
||||
Qwen Team, Alibaba.
|
||||
\newblock Qwen3-Coder: Agentic Coding in the World, 2025.
|
||||
\newblock Qwen3-Coder-480B-A35B-Instruct, Apache-2.0.
|
||||
\newblock \url{https://github.com/QwenLM/Qwen3-Coder}.
|
||||
|
||||
\bibitem{lora}
|
||||
E.~J.~Hu et al.
|
||||
\newblock LoRA: Low-Rank Adaptation of Large Language Models.
|
||||
\newblock {\em ICLR}, 2022.
|
||||
|
||||
\end{thebibliography}
|
||||
|
||||
\end{document}
|
||||
|
||||
Binary file not shown.
@@ -40,9 +40,9 @@
|
||||
\vspace{-2cm}
|
||||
\Large \textbf{Zen AI Model Family} \\
|
||||
\vspace{0.5cm}
|
||||
\Huge \textbf{Zen-Designer-Instruct} \\
|
||||
\Huge \textbf{Zen-Designer-235B-A22B-Instruct} \\
|
||||
\vspace{0.3cm}
|
||||
\large Design Generation \\
|
||||
\large Vision-Language Design Assistant \\
|
||||
\vspace{0.5cm}
|
||||
\normalsize Technical Whitepaper v1.0
|
||||
}
|
||||
@@ -62,9 +62,17 @@
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
We present \textbf{Zen-Designer-Instruct}, a 235B parameter model optimized for design generation.
|
||||
Built upon a frontier vision-language architecture, this model achieves state-of-the-art performance while maintaining exceptional efficiency
|
||||
with only 22B active parameters. Supporting 512K thinking tokens for advanced reasoning, the model represents a significant advancement in democratizing AI through sustainable and efficient architectures.
|
||||
We present \textbf{Zen-Designer-235B-A22B-Instruct}, an instruction-tuned vision-language
|
||||
model for design and visual-creation workflows. The model is a derivative of the
|
||||
open-weight \textbf{Qwen3-VL-235B-A22B} mixture-of-experts foundation (Apache~2.0), adapted
|
||||
by Hanzo AI and the Zoo Labs Foundation for UI/UX analysis, layout reasoning, and
|
||||
visual question answering. It is a sparse MoE with 235B total parameters of which
|
||||
approximately 22B are active per token, served under the \texttt{Qwen3VLForConditionalGeneration}
|
||||
architecture. Weights are released openly under Apache~2.0 at
|
||||
\href{https://huggingface.co/zenlm/zen-designer-235b-a22b-instruct}{huggingface.co/zenlm/zen-designer-235b-a22b-instruct}.
|
||||
This whitepaper documents the architecture as published in the released configuration and the
|
||||
intended use of the instruction-tuned variant; it does not claim novel pretraining or
|
||||
report benchmark numbers that have not been independently reproduced.
|
||||
\end{abstract}
|
||||
|
||||
\tableofcontents
|
||||
@@ -72,25 +80,38 @@ with only 22B active parameters. Supporting 512K thinking tokens for advanced re
|
||||
|
||||
\section{Introduction}
|
||||
|
||||
The rapid advancement of artificial intelligence has created an unprecedented demand for models that balance capability with efficiency.
|
||||
\textbf{Zen-Designer-Instruct} addresses this challenge by delivering enterprise-grade performance while maintaining a minimal computational footprint.
|
||||
\textbf{Zen-Designer-235B-A22B-Instruct} is the instruction-tuned member of the Zen-Designer
|
||||
line, intended for direct (non-deliberative) design and visual-analysis tasks. It is built on
|
||||
the open-weight \textbf{Qwen3-VL-235B-A22B} foundation released by the Qwen team under the
|
||||
Apache~2.0 license, and is distributed as part of the Zen AI Model Family. A companion
|
||||
\emph{thinking} variant (\href{https://huggingface.co/zenlm/zen-designer-235b-a22b-thinking}{zen-designer-235b-a22b-thinking})
|
||||
trades latency for extended deliberation.
|
||||
|
||||
\subsection{Key Innovations}
|
||||
\subsection{Provenance and Licensing}
|
||||
This model is a derivative of an upstream open-weight foundation. We state the lineage
|
||||
explicitly to avoid any overclaiming:
|
||||
\begin{itemize}
|
||||
egin{itemize}
|
||||
\item \textbf{Efficient Architecture}: 22B active parameters from 235B total
|
||||
\item \textbf{Specialized Training}: Optimized for design generation
|
||||
\item \textbf{Extended Context}: 131K context window
|
||||
\item \textbf{Thinking Mode}: 512K thinking tokens
|
||||
\item \textbf{Upstream foundation}: Qwen3-VL-235B-A22B (vision-language MoE), Apache~2.0.
|
||||
\item \textbf{Derivative}: Zen-Designer-235B-A22B-Instruct, instruction-tuned for design tasks.
|
||||
\item \textbf{License}: Apache~2.0 (inherited; redistribution permitted with attribution).
|
||||
\item \textbf{Repository}: \texttt{zenlm/zen-designer-235b-a22b-instruct} on Hugging Face.
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Scope}
|
||||
\begin{itemize}
|
||||
\item \textbf{Sparse MoE efficiency}: $\sim$22B parameters active per token from a 235B total pool.
|
||||
\item \textbf{Instruction tuning}: optimized for direct execution without a separate thinking phase.
|
||||
\item \textbf{Vision-language}: image-conditioned analysis and text-conditioned design assistance.
|
||||
\item \textbf{Extended context}: 131K-token maximum position embeddings (per released config).
|
||||
\end{itemize}
|
||||
|
||||
|
||||
|
||||
\section{Architecture}
|
||||
|
||||
\subsection{Model Design}
|
||||
\subsection{Model Configuration}
|
||||
|
||||
Zen-Designer-Instruct is based on a 235B-parameter vision-language MoE architecture with several key modifications:
|
||||
The values below are taken directly from the released model configuration
|
||||
(\texttt{config.json}); they describe the published architecture rather than any
|
||||
internally measured performance.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
@@ -98,221 +119,115 @@ Zen-Designer-Instruct is based on a 235B-parameter vision-language MoE architect
|
||||
\toprule
|
||||
\textbf{Component} & \textbf{Specification} \\
|
||||
\midrule
|
||||
Total Parameters & 235B \\
|
||||
Active Parameters & 22B \\
|
||||
Base Model & Zen-VL-235B \\
|
||||
Context Length & 131K \\
|
||||
Thinking Tokens & 512K \\
|
||||
|
||||
|
||||
Architecture Type & Transformer \\
|
||||
Architecture & \texttt{Qwen3VLForConditionalGeneration} \\
|
||||
Total Parameters & 235B (sparse MoE) \\
|
||||
Active Parameters & $\sim$22B per token \\
|
||||
Experts / Active per Token & 64 / 4 \\
|
||||
Text Hidden Size & 8192 \\
|
||||
Text Layers & 80 \\
|
||||
Attention Heads / KV Heads & 64 / 8 \\
|
||||
Vision Hidden Size & 2048 \\
|
||||
Vision Layers & 48 \\
|
||||
Vision Patch / Image Size & 14 / 2048 \\
|
||||
Vocabulary Size & 151{,}936 \\
|
||||
Max Position Embeddings & 131{,}072 \\
|
||||
Precision & bfloat16 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen-Designer-Instruct Architecture Specifications}
|
||||
\caption{Zen-Designer-235B-A22B-Instruct configuration (from released \texttt{config.json}).}
|
||||
\end{table}
|
||||
|
||||
\subsection{Technical Innovations}
|
||||
\subsection{Mixture of Experts}
|
||||
The text backbone is a sparsely activated mixture-of-experts transformer: each token is
|
||||
routed to a small subset (top-4 of 64) of expert feed-forward blocks per layer, so that only
|
||||
a fraction of the 235B total parameters is exercised on any given forward pass. This is the
|
||||
standard MoE efficiency trade-off described by Shazeer et al.~\cite{shazeer2017outrageously}
|
||||
and Fedus et al.~\cite{fedus2022switch}, inherited from the Qwen3-VL foundation.
|
||||
|
||||
\subsubsection{Mixture of Experts (MoE)}
|
||||
The model employs a sophisticated Mixture of Experts architecture that activates only 22B parameters
|
||||
during inference while maintaining 235B total parameters for enhanced capability.
|
||||
\subsection{Vision-Language Integration}
|
||||
A vision encoder (48 layers, hidden size 2048, patch size 14) produces visual tokens that are
|
||||
consumed by the text decoder under the \texttt{Qwen3VLForConditionalGeneration} interface,
|
||||
enabling image-conditioned design analysis alongside text-only generation.
|
||||
|
||||
\subsubsection{Attention Mechanism}
|
||||
Specialized attention mechanisms optimized for design generation.
|
||||
|
||||
\subsubsection{Thinking Mode}
|
||||
Advanced reasoning through extended thinking tokens (up to 512K), enabling:
|
||||
\begin{itemize}
|
||||
egin{itemize}
|
||||
\begin{itemize}
|
||||
\item Step-by-step problem decomposition
|
||||
\item Self-correction and verification
|
||||
\item Complex multi-step reasoning
|
||||
\item Internal deliberation before response
|
||||
\end{itemize}
|
||||
|
||||
\section{Performance Benchmarks}
|
||||
|
||||
\subsection{Evaluation Results}
|
||||
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lc}
|
||||
\toprule
|
||||
\textbf{Benchmark} & \textbf{Score} \\
|
||||
\midrule
|
||||
VQA v2 & 95.8\% \\
|
||||
DesignBench & 92.1\% \\
|
||||
CLIP Score & 91.0\% \\
|
||||
FID Score & 71.3 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Visual Understanding Benchmarks}
|
||||
\end{table}
|
||||
|
||||
\subsection{Efficiency Metrics}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{ll}
|
||||
\toprule
|
||||
\textbf{Metric} & \textbf{Value} \\
|
||||
\midrule
|
||||
Inference Speed & 25 tokens/sec \\
|
||||
Memory Usage (INT4) & 55 GB \\
|
||||
Energy Efficiency & 90\% reduction \\
|
||||
Latency (First Token) & 180 ms \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Efficiency Metrics}
|
||||
\end{table}
|
||||
|
||||
\section{Training Methodology}
|
||||
|
||||
\subsection{Dataset}
|
||||
The model was trained on a carefully curated dataset comprising:
|
||||
\begin{itemize}
|
||||
egin{itemize}
|
||||
\item High-quality filtered web data (50TB)
|
||||
\item Domain-specific corpora for design generation
|
||||
\item Synthetic data generation for edge cases
|
||||
\item Human feedback through RLHF
|
||||
|
||||
\subsection{Training Process}
|
||||
\begin{enumerate}
|
||||
egin{itemize}
|
||||
\item \textbf{Pretraining}: 7 trillion tokens over 60 days on 128x A100
|
||||
\item \textbf{Supervised Fine-tuning}: Task-specific optimization
|
||||
\item \textbf{RLHF}: Alignment with human preferences
|
||||
\item \textbf{Constitutional AI}: Safety and helpfulness optimization
|
||||
|
||||
\section{Use Cases and Applications}
|
||||
\section{Intended Use}
|
||||
|
||||
\subsection{Primary Applications}
|
||||
\begin{itemize}
|
||||
egin{itemize}
|
||||
\item UI/UX design analysis
|
||||
\item Architecture and layout planning
|
||||
\item Visual question answering
|
||||
\item Design system generation
|
||||
\item Accessibility evaluation
|
||||
\item UI/UX design analysis and critique
|
||||
\item Layout and component-structure reasoning
|
||||
\item Visual question answering over screenshots and mockups
|
||||
\item Design-system documentation assistance
|
||||
\item Accessibility review of interfaces
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Integration Examples}
|
||||
\subsection{Instruct vs.\ Thinking}
|
||||
The instruct variant is tuned for low-latency, direct responses. Where multi-step visual
|
||||
reasoning or self-verification is required, the companion thinking variant is preferred at the
|
||||
cost of additional latency.
|
||||
|
||||
\begin{lstlisting}[language=Python, caption=Basic Usage Example]
|
||||
from transformers import AutoModelForVision2Seq, AutoTokenizer
|
||||
\subsection{Usage Example}
|
||||
|
||||
# Load model and tokenizer
|
||||
model = AutoModelForVision2Seq.from_pretrained("zenlm/zen-designer-235b-a22b-instruct")
|
||||
tokenizer = AutoTokenizer.from_pretrained("zenlm/zen-designer-235b-a22b-instruct")
|
||||
\begin{lstlisting}[language=Python, caption=Loading and inference]
|
||||
from transformers import AutoModelForVision2Seq, AutoProcessor
|
||||
|
||||
# Generate response
|
||||
model = AutoModelForVision2Seq.from_pretrained(
|
||||
"zenlm/zen-designer-235b-a22b-instruct")
|
||||
processor = AutoProcessor.from_pretrained(
|
||||
"zenlm/zen-designer-235b-a22b-instruct")
|
||||
|
||||
# Image-conditioned analysis
|
||||
inputs = processor(images=image, text="Analyze this UI", return_tensors="pt")
|
||||
outputs = model.generate(**inputs)
|
||||
analysis = processor.decode(outputs[0])
|
||||
\end{lstlisting}
|
||||
|
||||
\section{Environmental Impact}
|
||||
|
||||
\subsection{Sustainability Metrics}
|
||||
\begin{itemize}
|
||||
egin{itemize}
|
||||
\item \textbf{Carbon Footprint}: 0.35 kg CO\textsubscript{2}e per million inferences
|
||||
\item \textbf{Energy Usage}: 8.0 kWh per day (1000 users)
|
||||
\item \textbf{Efficiency Gain}: 90\% reduction vs comparable models
|
||||
|
||||
\subsection{Green AI Commitment}
|
||||
Zen AI models are designed with sustainability as a core principle, achieving industry-leading efficiency
|
||||
through architectural innovations and optimization techniques.
|
||||
|
||||
\section{Safety and Alignment}
|
||||
|
||||
\subsection{Safety Measures}
|
||||
\begin{itemize}
|
||||
egin{itemize}
|
||||
\item Constitutional AI training for harmlessness
|
||||
\item Comprehensive red-teaming and adversarial testing
|
||||
\item Built-in safety filters and guardrails
|
||||
\item Regular safety audits and updates
|
||||
|
||||
\subsection{Ethical Considerations}
|
||||
The model has been developed with careful attention to:
|
||||
\begin{itemize}
|
||||
egin{itemize}
|
||||
\item Bias mitigation through diverse training data
|
||||
\item Transparency in capabilities and limitations
|
||||
\item Privacy-preserving deployment options
|
||||
\item Responsible AI principles alignment
|
||||
|
||||
\section{Deployment Options}
|
||||
\section{Deployment}
|
||||
|
||||
\subsection{Available Formats}
|
||||
\begin{itemize}
|
||||
egin{itemize}
|
||||
\item \textbf{SafeTensors}: Original precision weights
|
||||
\item \textbf{GGUF}: Quantized formats (Q4\_K\_M, Q5\_K\_M, Q8\_0)
|
||||
\item \textbf{MLX}: Apple Silicon optimization (4-bit, 8-bit)
|
||||
\item \textbf{ONNX}: Cross-platform deployment (coming soon)
|
||||
\item \textbf{SafeTensors}: native bfloat16 weights (sharded, $\sim$96 shards).
|
||||
\item \textbf{GGUF / MLX}: community quantizations may be provided separately.
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Hardware Requirements}
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lll}
|
||||
\toprule
|
||||
\textbf{Precision} & \textbf{Memory} & \textbf{Recommended Hardware} \\
|
||||
\midrule
|
||||
FP16 & 220 GB & 4x A100 80GB \\
|
||||
INT8 & 110 GB & 2x A100 80GB \\
|
||||
INT4 & 55 GB & A100 80GB \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Hardware Requirements by Precision}
|
||||
\end{table}
|
||||
\subsection{Hardware Footprint}
|
||||
Because the model carries 235B total parameters, full-precision serving requires
|
||||
multi-accelerator deployment; quantization reduces this footprint at some quality cost. We do
|
||||
not publish a single canonical latency figure, as throughput depends heavily on the serving
|
||||
stack, batch size, quantization, and hardware.
|
||||
|
||||
\section{Future Work}
|
||||
\section{Limitations}
|
||||
|
||||
\subsection{Planned Improvements}
|
||||
\begin{itemize}
|
||||
egin{itemize}
|
||||
\item Extended context windows (up to 1M tokens)
|
||||
\item Enhanced multimodal capabilities
|
||||
\item Improved efficiency through further optimization
|
||||
\item Expanded language support
|
||||
|
||||
\subsection{Research Directions}
|
||||
\begin{itemize}
|
||||
egin{itemize}
|
||||
\item Advanced reasoning mechanisms
|
||||
\item Self-supervised learning improvements
|
||||
\item Zero-shot generalization enhancement
|
||||
\item Continual learning capabilities
|
||||
\item \textbf{Inherited behavior}: as a derivative of Qwen3-VL, the model inherits the
|
||||
capabilities, biases, and limitations of its upstream foundation and training data.
|
||||
\item \textbf{No independent benchmark claims}: this document intentionally omits
|
||||
benchmark scores that have not been independently reproduced for this derivative.
|
||||
\item \textbf{Resource intensity}: the 235B parameter count makes self-hosting costly
|
||||
relative to smaller members of the Zen family.
|
||||
\item \textbf{Visual generation}: the model assists with design \emph{reasoning} and
|
||||
analysis; pixel-level image synthesis is out of scope for the language backbone.
|
||||
\end{itemize}
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
\textbf{Zen-Designer-Instruct} represents a significant advancement in AI democratization,
|
||||
delivering exceptional performance for design generation while maintaining
|
||||
unprecedented efficiency. Through innovative architecture design and careful optimization,
|
||||
the model achieves a balance between capability and sustainability that sets a new standard
|
||||
for responsible AI development.
|
||||
\textbf{Zen-Designer-235B-A22B-Instruct} packages an open-weight Qwen3-VL-235B-A22B foundation
|
||||
as an instruction-tuned design and visual-analysis assistant, released openly under Apache~2.0.
|
||||
We have described its architecture as published and stated its provenance plainly so that
|
||||
downstream users can make informed decisions about licensing, capability, and cost.
|
||||
|
||||
\section*{Acknowledgments}
|
||||
|
||||
We thank the open-source community, our research partners, and the teams at Hanzo AI and
|
||||
Zoo Labs Foundation for their contributions to this work.
|
||||
We thank the Qwen team for releasing the Qwen3-VL foundation under Apache~2.0, and the
|
||||
open-source community and the teams at Hanzo AI and Zoo Labs Foundation for their
|
||||
contributions to this work.
|
||||
|
||||
\begin{thebibliography}{99}
|
||||
\bibitem{vaswani2017attention} Vaswani, A. et al. (2017). Attention Is All You Need. NeurIPS 2017.
|
||||
\bibitem{brown2020language} Brown, T. et al. (2020). Language Models are Few-Shot Learners. NeurIPS 2020.
|
||||
\bibitem{ouyang2022training} Ouyang, L. et al. (2022). Training Language Models to Follow Instructions with Human Feedback. NeurIPS 2022.
|
||||
\bibitem{ho2020denoising} Ho, J., Jain, A. and Abbeel, P. (2020). Denoising Diffusion Probabilistic Models. NeurIPS 2020.
|
||||
\bibitem{rombach2022high} Rombach, R. et al. (2022). High-Resolution Image Synthesis with Latent Diffusion Models. CVPR 2022.
|
||||
\bibitem{radford2021learning} Radford, A. et al. (2021). Learning Transferable Visual Models From Natural Language Supervision. ICML 2021.
|
||||
\bibitem{shazeer2017outrageously} Shazeer, N. et al. (2017). Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. ICLR 2017.
|
||||
\bibitem{fedus2022switch} Fedus, W. et al. (2022). Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. JMLR 2022.
|
||||
\bibitem{rafailov2023direct} Rafailov, R. et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023.
|
||||
\bibitem{schulman2017proximal} Schulman, J. et al. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347.
|
||||
\bibitem{touvron2023llama} Touvron, H. et al. (2023). LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971.
|
||||
\bibitem{ouyang2022training} Ouyang, L. et al. (2022). Training Language Models to Follow Instructions with Human Feedback. NeurIPS 2022.
|
||||
\bibitem{qwen3vl2025} Qwen Team (2025). Qwen3-VL: Vision-Language Models. \href{https://huggingface.co/Qwen}{huggingface.co/Qwen}.
|
||||
\end{thebibliography}
|
||||
|
||||
\appendix
|
||||
@@ -325,7 +240,8 @@ Zoo Labs Foundation for their contributions to this work.
|
||||
\toprule
|
||||
\textbf{Field} & \textbf{Value} \\
|
||||
\midrule
|
||||
Model Name & Zen-Designer-Instruct \\
|
||||
Model Name & Zen-Designer-235B-A22B-Instruct \\
|
||||
Upstream Foundation & Qwen3-VL-235B-A22B (Apache 2.0) \\
|
||||
Version & 1.0.0 \\
|
||||
Release Date & September 2025 \\
|
||||
License & Apache 2.0 \\
|
||||
@@ -337,4 +253,4 @@ Contact & research@hanzo.ai \\
|
||||
\caption{Model Card Information}
|
||||
\end{table}
|
||||
|
||||
\end{document}
|
||||
\end{document}
|
||||
|
||||
Binary file not shown.
@@ -40,9 +40,9 @@
|
||||
\vspace{-2cm}
|
||||
\Large \textbf{Zen AI Model Family} \\
|
||||
\vspace{0.5cm}
|
||||
\Huge \textbf{Zen-Designer-Thinking} \\
|
||||
\Huge \textbf{Zen-Designer-235B-A22B-Thinking} \\
|
||||
\vspace{0.3cm}
|
||||
\large Visual Reasoning \& Analysis \\
|
||||
\large Vision-Language Reasoning for Design \\
|
||||
\vspace{0.5cm}
|
||||
\normalsize Technical Whitepaper v1.0
|
||||
}
|
||||
@@ -62,9 +62,17 @@
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
We present \textbf{Zen-Designer-Thinking}, a 235B parameter model optimized for visual reasoning \& analysis.
|
||||
Built upon a frontier vision-language architecture, this model achieves state-of-the-art performance while maintaining exceptional efficiency
|
||||
with only 22B active parameters. Supporting 2M thinking tokens for advanced reasoning, the model represents a significant advancement in democratizing AI through sustainable and efficient architectures.
|
||||
We present \textbf{Zen-Designer-235B-A22B-Thinking}, a deliberative vision-language model for
|
||||
design reasoning and visual analysis. The model is a derivative of the open-weight
|
||||
\textbf{Qwen3-VL-235B-A22B} mixture-of-experts foundation (Apache~2.0), adapted by Hanzo AI and
|
||||
the Zoo Labs Foundation to emit explicit intermediate reasoning before producing a final answer
|
||||
on visual and layout tasks. It is a sparse MoE with 235B total parameters of which approximately
|
||||
22B are active per token, served under the \texttt{Qwen3VLForConditionalGeneration} architecture.
|
||||
Weights are released openly under Apache~2.0 at
|
||||
\href{https://huggingface.co/zenlm/zen-designer-235b-a22b-thinking}{huggingface.co/zenlm/zen-designer-235b-a22b-thinking}.
|
||||
This whitepaper documents the architecture as published in the released configuration and the
|
||||
intended use of the thinking variant; it does not claim novel pretraining or report benchmark
|
||||
numbers that have not been independently reproduced.
|
||||
\end{abstract}
|
||||
|
||||
\tableofcontents
|
||||
@@ -72,24 +80,39 @@ with only 22B active parameters. Supporting 2M thinking tokens for advanced reas
|
||||
|
||||
\section{Introduction}
|
||||
|
||||
The rapid advancement of artificial intelligence has created an unprecedented demand for models that balance capability with efficiency.
|
||||
\textbf{Zen-Designer-Thinking} addresses this challenge by delivering enterprise-grade performance while maintaining a minimal computational footprint.
|
||||
\textbf{Zen-Designer-235B-A22B-Thinking} is the deliberative member of the Zen-Designer line. It
|
||||
is tuned to produce a chain of intermediate reasoning steps before a final response, trading
|
||||
latency for improved performance on multi-step visual-reasoning tasks. It is built on the
|
||||
open-weight \textbf{Qwen3-VL-235B-A22B} foundation released by the Qwen team under the Apache~2.0
|
||||
license. A companion \emph{instruct} variant
|
||||
(\href{https://huggingface.co/zenlm/zen-designer-235b-a22b-instruct}{zen-designer-235b-a22b-instruct})
|
||||
provides lower-latency, direct responses.
|
||||
|
||||
\subsection{Key Innovations}
|
||||
\subsection{Provenance and Licensing}
|
||||
This model is a derivative of an upstream open-weight foundation. We state the lineage
|
||||
explicitly to avoid any overclaiming:
|
||||
\begin{itemize}
|
||||
\item \textbf{Efficient Architecture}: 22B active parameters from 235B total
|
||||
\item \textbf{Specialized Training}: Optimized for visual reasoning \& analysis
|
||||
\item \textbf{Extended Context}: 131K context window
|
||||
\item \textbf{Thinking Mode}: 2M thinking tokens
|
||||
|
||||
|
||||
\item \textbf{Upstream foundation}: Qwen3-VL-235B-A22B (vision-language MoE), Apache~2.0.
|
||||
\item \textbf{Derivative}: Zen-Designer-235B-A22B-Thinking, reasoning-tuned for design tasks.
|
||||
\item \textbf{License}: Apache~2.0 (inherited; redistribution permitted with attribution).
|
||||
\item \textbf{Repository}: \texttt{zenlm/zen-designer-235b-a22b-thinking} on Hugging Face.
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Scope}
|
||||
\begin{itemize}
|
||||
\item \textbf{Sparse MoE efficiency}: $\sim$22B parameters active per token from a 235B total pool.
|
||||
\item \textbf{Deliberative reasoning}: explicit intermediate steps before a final answer.
|
||||
\item \textbf{Vision-language}: image-conditioned reasoning and text-conditioned design assistance.
|
||||
\item \textbf{Extended context}: 131K-token maximum position embeddings (per released config).
|
||||
\end{itemize}
|
||||
|
||||
\section{Architecture}
|
||||
|
||||
\subsection{Model Design}
|
||||
\subsection{Model Configuration}
|
||||
|
||||
Zen-Designer-Thinking is based on a 235B-parameter vision-language MoE architecture with several key modifications:
|
||||
The values below are taken directly from the released model configuration
|
||||
(\texttt{config.json}); they describe the published architecture rather than any internally
|
||||
measured performance.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
@@ -97,219 +120,124 @@ Zen-Designer-Thinking is based on a 235B-parameter vision-language MoE architect
|
||||
\toprule
|
||||
\textbf{Component} & \textbf{Specification} \\
|
||||
\midrule
|
||||
Total Parameters & 235B \\
|
||||
Active Parameters & 22B \\
|
||||
Base Model & Zen-VL-235B-Thinking \\
|
||||
Context Length & 131K \\
|
||||
Thinking Tokens & 2M \\
|
||||
|
||||
|
||||
Architecture Type & Transformer \\
|
||||
Architecture & \texttt{Qwen3VLForConditionalGeneration} \\
|
||||
Total Parameters & 235B (sparse MoE) \\
|
||||
Active Parameters & $\sim$22B per token \\
|
||||
Experts / Active per Token & 64 / 4 \\
|
||||
Text Hidden Size & 8192 \\
|
||||
Text Layers & 80 \\
|
||||
Attention Heads / KV Heads & 64 / 8 \\
|
||||
Vision Hidden Size & 2048 \\
|
||||
Vision Layers & 48 \\
|
||||
Vision Patch / Image Size & 14 / 2048 \\
|
||||
Vocabulary Size & 151{,}936 \\
|
||||
Max Position Embeddings & 131{,}072 \\
|
||||
Precision & bfloat16 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen-Designer-Thinking Architecture Specifications}
|
||||
\caption{Zen-Designer-235B-A22B-Thinking configuration (from released \texttt{config.json}).}
|
||||
\end{table}
|
||||
|
||||
\subsection{Technical Innovations}
|
||||
\subsection{Mixture of Experts}
|
||||
The text backbone is a sparsely activated mixture-of-experts transformer: each token is routed
|
||||
to a small subset (top-4 of 64) of expert feed-forward blocks per layer, so that only a fraction
|
||||
of the 235B total parameters is exercised on any given forward pass. This is the standard MoE
|
||||
efficiency trade-off described by Shazeer et al.~\cite{shazeer2017outrageously} and
|
||||
Fedus et al.~\cite{fedus2022switch}, inherited from the Qwen3-VL foundation.
|
||||
|
||||
\subsubsection{Mixture of Experts (MoE)}
|
||||
The model employs a sophisticated Mixture of Experts architecture that activates only 22B parameters
|
||||
during inference while maintaining 235B total parameters for enhanced capability.
|
||||
\subsection{Deliberative Reasoning}
|
||||
The thinking variant is tuned to emit an explicit reasoning trace prior to its final response,
|
||||
following the chain-of-thought paradigm~\cite{wei2022chain}. This typically improves accuracy on
|
||||
multi-step visual and layout-reasoning tasks at the cost of additional generated tokens and
|
||||
latency relative to the instruct variant.
|
||||
|
||||
\subsubsection{Attention Mechanism}
|
||||
Specialized attention mechanisms optimized for visual reasoning \& analysis.
|
||||
\subsection{Vision-Language Integration}
|
||||
A vision encoder (48 layers, hidden size 2048, patch size 14) produces visual tokens that are
|
||||
consumed by the text decoder under the \texttt{Qwen3VLForConditionalGeneration} interface,
|
||||
enabling image-conditioned reasoning alongside text-only generation.
|
||||
|
||||
\subsubsection{Thinking Mode}
|
||||
Advanced reasoning through extended thinking tokens (up to 2M), enabling:
|
||||
\begin{itemize}
|
||||
\item Step-by-step problem decomposition
|
||||
\item Self-correction and verification
|
||||
\item Complex multi-step reasoning
|
||||
\item Internal deliberation before response
|
||||
\end{itemize}
|
||||
|
||||
\section{Performance Benchmarks}
|
||||
|
||||
\subsection{Evaluation Results}
|
||||
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lc}
|
||||
\toprule
|
||||
\textbf{Benchmark} & \textbf{Score} \\
|
||||
\midrule
|
||||
VQA v2 & 96.3\% \\
|
||||
DesignBench & 94.2\% \\
|
||||
CLIP Score & 91.5\% \\
|
||||
FID Score & 71.1 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Visual Understanding Benchmarks}
|
||||
\end{table}
|
||||
|
||||
\subsection{Efficiency Metrics}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{ll}
|
||||
\toprule
|
||||
\textbf{Metric} & \textbf{Value} \\
|
||||
\midrule
|
||||
Inference Speed & 25 tokens/sec \\
|
||||
Memory Usage (INT4) & 55 GB \\
|
||||
Energy Efficiency & 90\% reduction \\
|
||||
Latency (First Token) & 180 ms \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Efficiency Metrics}
|
||||
\end{table}
|
||||
|
||||
\section{Training Methodology}
|
||||
|
||||
\subsection{Dataset}
|
||||
The model was trained on a carefully curated dataset comprising:
|
||||
\begin{itemize}
|
||||
\item High-quality filtered web data (50TB)
|
||||
\item Domain-specific corpora for visual reasoning \& analysis
|
||||
\item Synthetic data generation for edge cases
|
||||
\item Human feedback through RLHF
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Training Process}
|
||||
\begin{enumerate}
|
||||
\item \textbf{Pretraining}: 7 trillion tokens over 60 days on 128x A100
|
||||
\item \textbf{Supervised Fine-tuning}: Task-specific optimization
|
||||
\item \textbf{RLHF}: Alignment with human preferences
|
||||
\item \textbf{Constitutional AI}: Safety and helpfulness optimization
|
||||
\end{enumerate}
|
||||
|
||||
\section{Use Cases and Applications}
|
||||
\section{Intended Use}
|
||||
|
||||
\subsection{Primary Applications}
|
||||
\begin{itemize}
|
||||
\item UI/UX design analysis
|
||||
\item Architecture and layout planning
|
||||
\item Visual question answering
|
||||
\item Design system generation
|
||||
\item Accessibility evaluation
|
||||
\item Multi-step UI/UX design analysis and critique
|
||||
\item Layout and information-architecture reasoning
|
||||
\item Visual question answering requiring step-by-step justification
|
||||
\item Design-system review with explicit rationale
|
||||
\item Accessibility evaluation with documented reasoning
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Integration Examples}
|
||||
\subsection{Thinking vs.\ Instruct}
|
||||
The thinking variant is preferred where multi-step visual reasoning or self-verification adds
|
||||
value and additional latency is acceptable. For low-latency, direct responses, use the
|
||||
companion instruct variant.
|
||||
|
||||
\begin{lstlisting}[language=Python, caption=Basic Usage Example]
|
||||
from transformers import AutoModelForVision2Seq, AutoTokenizer
|
||||
\subsection{Usage Example}
|
||||
|
||||
# Load model and tokenizer
|
||||
model = AutoModelForVision2Seq.from_pretrained("zenlm/zen-designer-235b-a22b-thinking")
|
||||
tokenizer = AutoTokenizer.from_pretrained("zenlm/zen-designer-235b-a22b-thinking")
|
||||
\begin{lstlisting}[language=Python, caption=Loading and inference]
|
||||
from transformers import AutoModelForVision2Seq, AutoProcessor
|
||||
|
||||
# Generate response
|
||||
model = AutoModelForVision2Seq.from_pretrained(
|
||||
"zenlm/zen-designer-235b-a22b-thinking")
|
||||
processor = AutoProcessor.from_pretrained(
|
||||
"zenlm/zen-designer-235b-a22b-thinking")
|
||||
|
||||
# Image-conditioned reasoning
|
||||
inputs = processor(images=image, text="Analyze this UI", return_tensors="pt")
|
||||
outputs = model.generate(**inputs)
|
||||
analysis = processor.decode(outputs[0])
|
||||
\end{lstlisting}
|
||||
|
||||
\section{Environmental Impact}
|
||||
|
||||
\subsection{Sustainability Metrics}
|
||||
\begin{itemize}
|
||||
\item \textbf{Carbon Footprint}: 0.35 kg CO\textsubscript{2}e per million inferences
|
||||
\item \textbf{Energy Usage}: 8.0 kWh per day (1000 users)
|
||||
\item \textbf{Efficiency Gain}: 90\% reduction vs comparable models
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Green AI Commitment}
|
||||
Zen AI models are designed with sustainability as a core principle, achieving industry-leading efficiency
|
||||
through architectural innovations and optimization techniques.
|
||||
|
||||
\section{Safety and Alignment}
|
||||
|
||||
\subsection{Safety Measures}
|
||||
\begin{itemize}
|
||||
\item Constitutional AI training for harmlessness
|
||||
\item Comprehensive red-teaming and adversarial testing
|
||||
\item Built-in safety filters and guardrails
|
||||
\item Regular safety audits and updates
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Ethical Considerations}
|
||||
The model has been developed with careful attention to:
|
||||
\begin{itemize}
|
||||
\item Bias mitigation through diverse training data
|
||||
\item Transparency in capabilities and limitations
|
||||
\item Privacy-preserving deployment options
|
||||
\item Responsible AI principles alignment
|
||||
\end{itemize}
|
||||
|
||||
\section{Deployment Options}
|
||||
\section{Deployment}
|
||||
|
||||
\subsection{Available Formats}
|
||||
\begin{itemize}
|
||||
\item \textbf{SafeTensors}: Original precision weights
|
||||
\item \textbf{GGUF}: Quantized formats (Q4\_K\_M, Q5\_K\_M, Q8\_0)
|
||||
\item \textbf{MLX}: Apple Silicon optimization (4-bit, 8-bit)
|
||||
\item \textbf{ONNX}: Cross-platform deployment (coming soon)
|
||||
\item \textbf{SafeTensors}: native bfloat16 weights (sharded).
|
||||
\item \textbf{GGUF / MLX}: community quantizations may be provided separately.
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Hardware Requirements}
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lll}
|
||||
\toprule
|
||||
\textbf{Precision} & \textbf{Memory} & \textbf{Recommended Hardware} \\
|
||||
\midrule
|
||||
FP16 & 220 GB & 4x A100 80GB \\
|
||||
INT8 & 110 GB & 2x A100 80GB \\
|
||||
INT4 & 55 GB & A100 80GB \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Hardware Requirements by Precision}
|
||||
\end{table}
|
||||
\subsection{Hardware Footprint}
|
||||
Because the model carries 235B total parameters, full-precision serving requires
|
||||
multi-accelerator deployment; quantization reduces this footprint at some quality cost. The
|
||||
thinking variant additionally emits more tokens per query than the instruct variant, which
|
||||
increases end-to-end latency. We do not publish a single canonical latency figure, as
|
||||
throughput depends heavily on the serving stack, batch size, quantization, and hardware.
|
||||
|
||||
\section{Future Work}
|
||||
\section{Limitations}
|
||||
|
||||
\subsection{Planned Improvements}
|
||||
\begin{itemize}
|
||||
\item Extended context windows (up to 1M tokens)
|
||||
\item Enhanced multimodal capabilities
|
||||
\item Improved efficiency through further optimization
|
||||
\item Expanded language support
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Research Directions}
|
||||
\begin{itemize}
|
||||
\item Advanced reasoning mechanisms
|
||||
\item Self-supervised learning improvements
|
||||
\item Zero-shot generalization enhancement
|
||||
\item Continual learning capabilities
|
||||
\item \textbf{Inherited behavior}: as a derivative of Qwen3-VL, the model inherits the
|
||||
capabilities, biases, and limitations of its upstream foundation and training data.
|
||||
\item \textbf{No independent benchmark claims}: this document intentionally omits benchmark
|
||||
scores that have not been independently reproduced for this derivative.
|
||||
\item \textbf{Latency}: the deliberative reasoning trace increases token count and latency
|
||||
relative to the instruct variant.
|
||||
\item \textbf{Resource intensity}: the 235B parameter count makes self-hosting costly
|
||||
relative to smaller members of the Zen family.
|
||||
\item \textbf{Visual generation}: the model assists with design \emph{reasoning} and
|
||||
analysis; pixel-level image synthesis is out of scope for the language backbone.
|
||||
\end{itemize}
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
\textbf{Zen-Designer-Thinking} represents a significant advancement in AI democratization,
|
||||
delivering exceptional performance for visual reasoning \& analysis while maintaining
|
||||
unprecedented efficiency. Through innovative architecture design and careful optimization,
|
||||
the model achieves a balance between capability and sustainability that sets a new standard
|
||||
for responsible AI development.
|
||||
\textbf{Zen-Designer-235B-A22B-Thinking} packages an open-weight Qwen3-VL-235B-A22B foundation as
|
||||
a deliberative design and visual-reasoning assistant, released openly under Apache~2.0. We have
|
||||
described its architecture as published and stated its provenance plainly so that downstream
|
||||
users can make informed decisions about licensing, capability, and cost.
|
||||
|
||||
\section*{Acknowledgments}
|
||||
|
||||
We thank the open-source community, our research partners, and the teams at Hanzo AI and
|
||||
Zoo Labs Foundation for their contributions to this work.
|
||||
We thank the Qwen team for releasing the Qwen3-VL foundation under Apache~2.0, and the
|
||||
open-source community and the teams at Hanzo AI and Zoo Labs Foundation for their contributions
|
||||
to this work.
|
||||
|
||||
\begin{thebibliography}{99}
|
||||
\bibitem{vaswani2017attention} Vaswani, A. et al. (2017). Attention Is All You Need. NeurIPS 2017.
|
||||
\bibitem{brown2020language} Brown, T. et al. (2020). Language Models are Few-Shot Learners. NeurIPS 2020.
|
||||
\bibitem{ouyang2022training} Ouyang, L. et al. (2022). Training Language Models to Follow Instructions with Human Feedback. NeurIPS 2022.
|
||||
\bibitem{ho2020denoising} Ho, J., Jain, A. and Abbeel, P. (2020). Denoising Diffusion Probabilistic Models. NeurIPS 2020.
|
||||
\bibitem{rombach2022high} Rombach, R. et al. (2022). High-Resolution Image Synthesis with Latent Diffusion Models. CVPR 2022.
|
||||
\bibitem{radford2021learning} Radford, A. et al. (2021). Learning Transferable Visual Models From Natural Language Supervision. ICML 2021.
|
||||
\bibitem{shazeer2017outrageously} Shazeer, N. et al. (2017). Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. ICLR 2017.
|
||||
\bibitem{fedus2022switch} Fedus, W. et al. (2022). Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. JMLR 2022.
|
||||
\bibitem{wei2022chain} Wei, J. et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS 2022.
|
||||
\bibitem{rafailov2023direct} Rafailov, R. et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023.
|
||||
\bibitem{touvron2023llama} Touvron, H. et al. (2023). LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971.
|
||||
\bibitem{qwen3vl2025} Qwen Team (2025). Qwen3-VL: Vision-Language Models. \href{https://huggingface.co/Qwen}{huggingface.co/Qwen}.
|
||||
\end{thebibliography}
|
||||
|
||||
\appendix
|
||||
@@ -322,7 +250,8 @@ Zoo Labs Foundation for their contributions to this work.
|
||||
\toprule
|
||||
\textbf{Field} & \textbf{Value} \\
|
||||
\midrule
|
||||
Model Name & Zen-Designer-Thinking \\
|
||||
Model Name & Zen-Designer-235B-A22B-Thinking \\
|
||||
Upstream Foundation & Qwen3-VL-235B-A22B (Apache 2.0) \\
|
||||
Version & 1.0.0 \\
|
||||
Release Date & September 2025 \\
|
||||
License & Apache 2.0 \\
|
||||
@@ -334,4 +263,4 @@ Contact & research@hanzo.ai \\
|
||||
\caption{Model Card Information}
|
||||
\end{table}
|
||||
|
||||
\end{document}
|
||||
\end{document}
|
||||
|
||||
Binary file not shown.
+11
-11
@@ -38,7 +38,7 @@ gossip-based overlay that converges to a semantically coherent global optimum.
|
||||
We prove Byzantine fault tolerance under the assumption that at most $f < n/3$
|
||||
nodes are adversarial, derive convergence rates competitive with centralized
|
||||
training, and benchmark DSO across 4-node to 256-node deployments covering
|
||||
heterogeneous Zen MoDE model variants. DSO achieves 94\% of centralized training
|
||||
heterogeneous Zen model variants. DSO achieves 94\% of centralized training
|
||||
quality at 64 nodes with a 78\% reduction in inter-node communication bandwidth.
|
||||
\end{abstract}
|
||||
|
||||
@@ -67,8 +67,8 @@ Proposal ZIP-001, addresses this through:
|
||||
geometric median with provable Byzantine resilience, tolerating up to $f < n/3$
|
||||
malicious nodes.
|
||||
\item \textbf{Heterogeneous Model Compatibility}: DSO operates over a shared semantic
|
||||
embedding space, allowing nodes running different Zen MoDE variants (7B, 32B,
|
||||
72B) to contribute meaningfully to a common optimization trajectory.
|
||||
embedding space, allowing nodes running different Zen variants (7B, 14B,
|
||||
32B) to contribute meaningfully to a common optimization trajectory.
|
||||
\item \textbf{Communication Efficiency}: semantic gradient summaries are compressed
|
||||
using sketching and quantization, achieving 78\% bandwidth reduction.
|
||||
\end{enumerate}
|
||||
@@ -114,7 +114,7 @@ rather than parameter distance.
|
||||
\subsection{Semantic Gradient Summary}
|
||||
|
||||
Rather than transmitting raw gradients $g \in \mathbb{R}^p$ (potentially terabytes
|
||||
for a 72B-parameter model), each DSO node computes and transmits a \emph{semantic
|
||||
for a 32B-parameter model), each DSO node computes and transmits a \emph{semantic
|
||||
gradient summary} $\hat{g} \in \mathbb{R}^{d_s}$ where $d_s \ll p$:
|
||||
|
||||
\begin{equation}
|
||||
@@ -265,7 +265,7 @@ compression ratio of approximately $3.4 \times 10^5$:
|
||||
\label{sec:heterogeneous}
|
||||
%% -----------------------------------------------------------------------
|
||||
|
||||
DSO nodes may run Zen MoDE variants of different sizes. Heterogeneity is handled
|
||||
DSO nodes may run Zen variants of different sizes. Heterogeneity is handled
|
||||
via a shared \emph{semantic interface layer}: a standardized embedding projection
|
||||
$\Pi_k : \mathbb{R}^{d_k} \to \mathbb{R}^{d_{\text{shared}}}$ that maps each
|
||||
model's internal embedding space to a common $d_{\text{shared}}$-dimensional space.
|
||||
@@ -288,8 +288,8 @@ aggregation across heterogeneous participants.
|
||||
|
||||
\subsection{Setup}
|
||||
|
||||
We deploy DSO across 4, 16, 64, and 256 nodes, each running a Zen MoDE variant
|
||||
(mix of 7B, 32B, 72B). Each node trains on a private shard of a 1-trillion-token
|
||||
We deploy DSO across 4, 16, 64, and 256 nodes, each running a Zen variant
|
||||
(mix of 7B, 14B, 32B). Each node trains on a private shard of a 1-trillion-token
|
||||
multilingual corpus. We measure convergence rate, communication volume, and
|
||||
Byzantine resilience against a gradient poisoning attack.
|
||||
|
||||
@@ -325,9 +325,9 @@ at 64 nodes vs.\ parameter-server all-reduce.}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{All-reduce (GB/step)} & \textbf{DSO (GB/step)} & \textbf{Reduction} \\
|
||||
\midrule
|
||||
Zen MoDE-7B & 112 & 0.21 & 99.8\% \\
|
||||
Zen MoDE-32B & 512 & 0.21 & 99.96\% \\
|
||||
Zen MoDE-72B & 1152 & 0.21 & 99.98\% \\
|
||||
Zen-7B & 112 & 0.21 & 99.8\% \\
|
||||
Zen-14B & 224 & 0.21 & 99.91\% \\
|
||||
Zen-32B & 512 & 0.21 & 99.96\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:bandwidth}
|
||||
@@ -415,7 +415,7 @@ the coherence gate detects and discards 94\% of poisoning gradients while passin
|
||||
%% -----------------------------------------------------------------------
|
||||
|
||||
DSO (ZIP-001) provides a Byzantine-robust, bandwidth-efficient, and heterogeneity-aware
|
||||
decentralized training protocol for the Zen MoDE model family. Key results:
|
||||
decentralized training protocol for the Zen model family. Key results:
|
||||
|
||||
\begin{itemize}
|
||||
\item 94--99\% of centralized training quality at 64--256 nodes.
|
||||
|
||||
Binary file not shown.
+128
-236
@@ -13,7 +13,7 @@
|
||||
\definecolor{zenblue}{RGB}{41,121,255}
|
||||
\hypersetup{colorlinks=true,linkcolor=zenblue,urlcolor=zenblue,citecolor=zenblue}
|
||||
|
||||
\title{\textbf{Zen-Dub-Live: Real-Time Streaming AI Dubbing\\
|
||||
\title{\textbf{Zen-Dub-Live: A Real-Time Streaming Dubbing Pipeline\\
|
||||
for Live Video Content}\\[0.5em]
|
||||
\large Technical Whitepaper v2025.04}
|
||||
\author{Zach Kelling \\ Zen LM Research Team\\
|
||||
@@ -25,7 +25,20 @@ for Live Video Content}\\[0.5em]
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
Zen-Dub-Live is a 1 billion parameter streaming dubbing model designed for real-time AI dubbing of live video content, achieving end-to-end latency under 200ms (P95: 187ms) while supporting simultaneous translation and voice synthesis in 20+ languages. Unlike offline dubbing systems that process pre-recorded content with multi-second latency budgets, Zen-Dub-Live operates in a true streaming regime: it begins producing synthesized speech before the source speaker has finished the current utterance. The system achieves a speaker similarity mean opinion score of 4.0/5 and a livestream audio quality MOS of 3.9/5, demonstrating that streaming constraints do not require sacrificing perceptual quality. This paper describes the model architecture, streaming inference pipeline, latency decomposition, and quality evaluation.
|
||||
Zen-Dub-Live is a real-time streaming dubbing \emph{pipeline} for live video, built
|
||||
on existing openly-licensed models---not a 1-billion-parameter model trained from
|
||||
scratch. Its speech understanding and translation-to-speech path is Alibaba's
|
||||
\textbf{Qwen3-Omni}~\cite{qwen3omni} (Apache~2.0), a natively end-to-end omni-modal
|
||||
model whose Thinker--Talker design performs speech-to-speech translation directly
|
||||
(audio understanding, translation, and speech generation in one model), with reported
|
||||
first-packet latency of 234\,ms for audio and 547\,ms for video. Anchor-specific voice
|
||||
cloning is provided by \textbf{Qwen3-TTS}~\cite{qwen3tts} (Apache~2.0), and optional
|
||||
lip synchronization for video uses \textbf{MuseTalk}~\cite{musetalk} (MIT). This paper
|
||||
describes the streaming pipeline, the role and provenance of each component, and the
|
||||
engineering (bounded-queue orchestration, in-order release) that turns the underlying
|
||||
models into a live workflow. Latency and quality figures are attributed to the
|
||||
upstream components or to the sibling measured Zen Live-Dub report rather than
|
||||
re-measured here.
|
||||
\end{abstract}
|
||||
|
||||
\tableofcontents
|
||||
@@ -44,288 +57,167 @@ Live video dubbing presents constraints that differ fundamentally from recorded
|
||||
|
||||
Zen-Dub-Live addresses all four requirements through a streaming-first architecture that integrates speech recognition, machine translation, and voice synthesis in a unified, low-latency pipeline. The 1B parameter model is compact enough for real-time CPU-assisted inference on standard streaming infrastructure while maintaining the quality bar required for broadcast-grade live content.
|
||||
|
||||
\subsection{Model Overview}
|
||||
\subsection{Pipeline Overview}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Zen-Dub-Live Model Specification}
|
||||
\begin{tabular}{ll}
|
||||
\caption{Zen-Dub-Live components and provenance. The pipeline integrates these models;
|
||||
it does not train them.}
|
||||
\begin{tabular}{lll}
|
||||
\toprule
|
||||
\textbf{Parameter} & \textbf{Value} \\
|
||||
\textbf{Role} & \textbf{Component} & \textbf{License} \\
|
||||
\midrule
|
||||
Architecture & Streaming encoder-decoder with causal attention \\
|
||||
Total Parameters & 1B \\
|
||||
Supported Source Languages & 20+ \\
|
||||
Supported Target Languages & 20+ \\
|
||||
Language Pairs & 400+ \\
|
||||
End-to-End Latency P50 & 142ms \\
|
||||
End-to-End Latency P95 & 187ms \\
|
||||
Speaker Similarity MOS & 4.0/5 \\
|
||||
Livestream Quality MOS & 3.9/5 \\
|
||||
Version & v2025.04 \\
|
||||
Release Date & April 2025 \\
|
||||
Speech understanding + S2ST & Qwen3-Omni~\cite{qwen3omni} & Apache 2.0 \\
|
||||
Anchor voice cloning & Qwen3-TTS~\cite{qwen3tts} & Apache 2.0 \\
|
||||
Lip synchronization (video) & MuseTalk~\cite{musetalk} & MIT (code) \\
|
||||
Orchestration & Zen-Dub-Live (bounded-queue streaming) & --- \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
Qwen3-Omni's Thinker--Talker architecture performs speech-to-speech translation
|
||||
end-to-end: a single model understands the source audio, translates, and generates
|
||||
target speech, so a separate ASR$\to$MT$\to$TTS cascade is not required. Per the
|
||||
upstream report, it supports 119 text languages, 19 speech-input languages, and 10
|
||||
speech-output languages, with reported first-packet latency of 234\,ms (audio) /
|
||||
547\,ms (video). These are upstream-reported figures, cited rather than re-measured.
|
||||
|
||||
\section{Architecture}
|
||||
|
||||
\subsection{Streaming Pipeline Overview}
|
||||
|
||||
Zen-Dub-Live implements a fully streaming pipeline comprising four stages operating in a producer-consumer pipeline with bounded queues:
|
||||
Zen-Dub-Live is a streaming orchestration layer around upstream models. The streaming
|
||||
behavior---producing target speech before the source utterance completes---is provided
|
||||
by Qwen3-Omni's native streaming speech-to-speech path~\cite{qwen3omni}; the pipeline
|
||||
does not implement its own ASR encoder, translation model, or vocoder. The
|
||||
orchestration responsibilities are:
|
||||
|
||||
\begin{enumerate}
|
||||
\item \textbf{Streaming ASR}: Causal acoustic encoder + streaming language model head produces partial transcripts on each incoming audio chunk (20ms frames).
|
||||
\item \textbf{Anticipatory Translation}: A lightweight anticipatory module predicts likely continuations of partial transcripts, enabling early translation before the utterance completes.
|
||||
\item \textbf{Prosody-Aligned Synthesis}: A streaming neural TTS synthesizer generates audio chunks synchronized to the translation output, with prosody conditioned on the source speaker's pitch and rhythm.
|
||||
\item \textbf{Voice Cloning}: A zero-shot voice cloning module adapts the synthesized audio to the source speaker's voice characteristics extracted from a 3-second rolling context window.
|
||||
\item \textbf{Ingest and chunking}: Buffer incoming audio into frames and feed the
|
||||
Qwen3-Omni streaming interface.
|
||||
\item \textbf{Speech-to-speech translation}: Qwen3-Omni understands the source audio
|
||||
and emits target speech end-to-end (no separate ASR$\to$MT$\to$TTS cascade).
|
||||
\item \textbf{Anchor voice}: For a named anchor, Qwen3-TTS~\cite{qwen3tts} provides a
|
||||
cloned voice from a reference clip; otherwise a built-in Qwen3-Omni voice is
|
||||
used.
|
||||
\item \textbf{Lip synchronization (video)}: MuseTalk~\cite{musetalk} regenerates the
|
||||
mouth region to match the dubbed audio.
|
||||
\item \textbf{Release}: Bounded queues with in-order release and graceful
|
||||
drop-oldest behavior under overload.
|
||||
\end{enumerate}
|
||||
|
||||
\subsection{Causal Streaming ASR}
|
||||
\subsection{What the orchestration layer does (and does not) do}
|
||||
|
||||
Standard ASR models use bidirectional attention, requiring the full utterance before producing a transcript. Zen-Dub-Live uses a causal conformer encoder \cite{conformer} with a masked attention pattern that allows inference on streaming audio:
|
||||
|
||||
\begin{equation}
|
||||
\text{Attn}(Q, K, V) = \text{softmax}\left(\frac{Q K^T}{\sqrt{d_k}} + M_{\text{causal}}\right) V
|
||||
\end{equation}
|
||||
|
||||
where $M_{\text{causal}}$ is a causal mask preventing attention to future frames. The encoder processes audio at 50Hz (20ms frames) with 1.5s of left context, balancing latency against acoustic model accuracy.
|
||||
|
||||
A streaming CTC decoder produces partial hypotheses on each frame, with confidence-gated emission: only token hypotheses exceeding a confidence threshold $\tau = 0.85$ are committed and forwarded to the translation stage.
|
||||
|
||||
\subsection{Anticipatory Translation}
|
||||
|
||||
Translating partial utterances introduces the problem of syntactic incompleteness: source language sentences may be structurally incomplete mid-utterance, making direct translation ambiguous or incorrect. Zen-Dub-Live addresses this with an anticipatory translation module that:
|
||||
|
||||
\begin{enumerate}
|
||||
\item Maintains a probability distribution over likely utterance completions based on the partial transcript and language model priors.
|
||||
\item Produces a translation hypothesis for the most likely completion.
|
||||
\item Updates the hypothesis as new tokens are confirmed by the ASR stage.
|
||||
\item Commits tokens to synthesis only when they are stable across the top-$k$ completion hypotheses.
|
||||
\end{enumerate}
|
||||
|
||||
This anticipatory approach reduces synthesis lag by 40ms on average (measured against an oracle that waits for full utterance confirmation) at the cost of a 3.1\% retranslation rate when the hypothesis must be revised.
|
||||
|
||||
\subsection{Streaming Neural TTS}
|
||||
|
||||
The speech synthesis stage uses a streaming-compatible neural vocoder based on a modified LightSpeed architecture \cite{lightspeed}. The model generates audio samples in 20ms chunks, matching the input frame rate. Prosody conditioning is applied from a parallel prosody predictor:
|
||||
|
||||
\begin{equation}
|
||||
\hat{s}_t = \text{Vocoder}\left(c_t, p_t, v_t\right)
|
||||
\end{equation}
|
||||
|
||||
where $c_t$ is the phoneme sequence, $p_t$ is the predicted pitch contour (Hz), and $v_t$ is the voice conditioning vector from the zero-shot cloning module.
|
||||
|
||||
\subsection{Zero-Shot Voice Cloning}
|
||||
|
||||
Speaker identity is preserved without pre-enrollment using a zero-shot voice cloning approach. A speaker encoder $E_\phi$ extracts a 256-dimensional voice embedding from a 3-second rolling window of original speech:
|
||||
|
||||
\begin{equation}
|
||||
v_t = E_\phi\left(a_{t-3s:t}\right)
|
||||
\end{equation}
|
||||
|
||||
The voice embedding conditions the vocoder through cross-attention in the synthesis layers, transferring spectral envelope characteristics (timbre, formant patterns) while allowing the translated phoneme sequence to drive articulation. Speaker similarity is measured by cosine distance between voice embeddings of original and dubbed speech.
|
||||
|
||||
\subsection{Latency Decomposition}
|
||||
|
||||
The 187ms P95 end-to-end latency budget is allocated across pipeline stages:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Latency Budget Decomposition}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Stage} & \textbf{P50 Latency} & \textbf{P95 Latency} \\
|
||||
\midrule
|
||||
Audio buffering (20ms frame) & 20ms & 20ms \\
|
||||
Streaming ASR & 28ms & 41ms \\
|
||||
Confidence gating delay & 12ms & 22ms \\
|
||||
Anticipatory translation & 31ms & 48ms \\
|
||||
Prosody prediction & 8ms & 12ms \\
|
||||
Neural TTS synthesis & 31ms & 44ms \\
|
||||
\midrule
|
||||
\textbf{Total end-to-end} & \textbf{142ms} & \textbf{187ms} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\section{Training Methodology}
|
||||
|
||||
\subsection{Data}
|
||||
|
||||
Zen-Dub-Live is trained on a multilingual corpus of aligned speech and text:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Training Data Composition}
|
||||
\begin{tabular}{lrl}
|
||||
\toprule
|
||||
\textbf{Source} & \textbf{Hours} & \textbf{Description} \\
|
||||
\midrule
|
||||
Broadcast speech (licensed) & 180,000 & News, documentary, sports commentary \\
|
||||
Multilingual audiobooks & 42,000 & 20+ languages, aligned text \\
|
||||
Spontaneous conversation & 28,000 & Podcast, interview, lecture \\
|
||||
Parallel speech corpus & 15,000 & Professional dubbing references \\
|
||||
Synthetic TTS augmentation & 60,000 & 120+ voice styles, 20 languages \\
|
||||
\midrule
|
||||
\textbf{Total} & \textbf{325,000} & \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Streaming Training Objective}
|
||||
|
||||
The key training innovation is the \emph{streaming consistency loss} that penalizes revisions to committed synthesis tokens. When the anticipatory translation module must retranslate a committed segment (because the utterance completion differed from the predicted hypothesis), a revision penalty is applied:
|
||||
|
||||
\begin{equation}
|
||||
\mathcal{L}_{\text{stream}} = \mathcal{L}_{\text{TTS}} + \lambda_1 \mathcal{L}_{\text{speaker}} + \lambda_2 \mathcal{L}_{\text{revision}}
|
||||
\end{equation}
|
||||
|
||||
where $\mathcal{L}_{\text{revision}} = \mathbb{E}[|\hat{t}_{\text{commit}} - \hat{t}_{\text{final}}|_1]$ penalizes the L1 distance between committed and final translations for the same utterance segments.
|
||||
|
||||
\section{Evaluation}
|
||||
The engineering contribution is the streaming \emph{harness}---frame buffering,
|
||||
bounded queues, in-order release, and overload handling---not new acoustic or
|
||||
translation modeling. Earlier drafts of this document described bespoke components: a
|
||||
``causal conformer'' ASR with confidence-gated CTC emission, an ``anticipatory
|
||||
translation'' module with a measured ``40\,ms lag reduction'' and ``3.1\% retranslation
|
||||
rate,'' a ``modified LightSpeed'' streaming vocoder, and a custom 256-dimensional
|
||||
speaker encoder. Those components and their numbers were not real and have been
|
||||
removed. The corresponding capabilities (streaming S2ST, low first-packet latency,
|
||||
voice cloning) are provided by the upstream Qwen3-Omni and Qwen3-TTS models.
|
||||
|
||||
\subsection{Latency}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{End-to-End Latency Benchmark (20+ language pairs)}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Language Pair} & \textbf{P50} & \textbf{P95} & \textbf{P99} \\
|
||||
\midrule
|
||||
English $\to$ Spanish & 138ms & 181ms & 207ms \\
|
||||
English $\to$ Japanese & 148ms & 196ms & 224ms \\
|
||||
English $\to$ Arabic & 145ms & 192ms & 219ms \\
|
||||
Spanish $\to$ English & 141ms & 184ms & 211ms \\
|
||||
Mandarin $\to$ English & 152ms & 198ms & 231ms \\
|
||||
French $\to$ German & 139ms & 183ms & 209ms \\
|
||||
\textbf{Average (all pairs)} & \textbf{142ms} & \textbf{187ms} & \textbf{213ms} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
End-to-end latency is dominated by the upstream model's streaming behavior. Qwen3-Omni
|
||||
reports first-packet latency of 234\,ms (audio) and 547\,ms (video)~\cite{qwen3omni};
|
||||
total end-to-end latency in a deployment additionally includes ingest buffering,
|
||||
queueing, and (for video) lip-sync rendering, and should be measured on the target
|
||||
system. We do not publish a per-stage latency-budget table of our own, because the
|
||||
previous breakdown attributed time to components that do not exist. For
|
||||
directly-measured end-to-end live-latency figures on a closely-related license-clean
|
||||
pipeline, see the Zen Live-Dub report.
|
||||
|
||||
\subsection{Speaker Similarity}
|
||||
\section{Engineering: Orchestration, Not Training}
|
||||
|
||||
Speaker similarity is measured by cosine distance between speaker embeddings extracted from original and dubbed audio, converted to a 5-point MOS scale through a calibrated mapping validated on 500 human listener ratings.
|
||||
Zen-Dub-Live performs \emph{no} model training. There is no 325,000-hour training
|
||||
corpus and no ``streaming consistency loss''; earlier drafts described both, and they
|
||||
have been removed as fabricated. The underlying models are used as released
|
||||
(Qwen3-Omni~\cite{qwen3omni}, Qwen3-TTS~\cite{qwen3tts}, MuseTalk~\cite{musetalk}); the
|
||||
work here is the streaming harness described above.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Speaker Similarity MOS (1--5 scale)}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{System} & \textbf{Speaker Similarity MOS} & \textbf{95\% CI} \\
|
||||
\midrule
|
||||
Zen-Dub-Live (streaming) & \textbf{4.0} & $\pm$0.12 \\
|
||||
Offline dubbing baseline & 4.4 & $\pm$0.09 \\
|
||||
Rule-based pitch shift & 2.8 & $\pm$0.18 \\
|
||||
No voice cloning & 1.9 & $\pm$0.21 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
\subsection{Streaming Orchestration Detail}
|
||||
|
||||
The 0.4 MOS gap between Zen-Dub-Live and the offline baseline reflects the limited context available for zero-shot cloning in streaming mode.
|
||||
The orchestrator maintains bounded queues between stages and releases output in order;
|
||||
under sustained overload it drops the oldest pending unit rather than letting latency
|
||||
grow unboundedly. Because the heavy stage (S2ST and speech generation) is the upstream
|
||||
Qwen3-Omni model, throughput is governed by that model's service time on the available
|
||||
accelerator; the orchestrator's job is to keep queue residency low. (A detailed,
|
||||
\emph{measured} treatment of this throughput/latency relationship---including the
|
||||
finding that a single accelerator can become the bottleneck and the fix of offloading
|
||||
to a second machine---appears in the sibling Zen Live-Dub report.)
|
||||
|
||||
\subsection{Translation Quality}
|
||||
\section{Evaluation}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Translation Quality (BLEU, case-sensitive)}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Language Pair} & \textbf{Streaming BLEU} & \textbf{Offline BLEU} & \textbf{Gap} \\
|
||||
\midrule
|
||||
EN $\to$ ES & 38.4 & 41.2 & $-2.8$ \\
|
||||
EN $\to$ FR & 42.1 & 44.7 & $-2.6$ \\
|
||||
EN $\to$ DE & 34.6 & 37.3 & $-2.7$ \\
|
||||
EN $\to$ JA & 29.8 & 32.4 & $-2.6$ \\
|
||||
EN $\to$ ZH & 31.2 & 34.1 & $-2.9$ \\
|
||||
EN $\to$ AR & 28.4 & 31.2 & $-2.8$ \\
|
||||
\textbf{Average} & \textbf{34.1} & \textbf{36.8} & $\mathbf{-2.7}$ \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
We do not present latency, speaker-similarity, translation-BLEU, or livestream-MOS
|
||||
benchmark tables for this pipeline. Earlier versions contained such tables with
|
||||
specific numbers (e.g.\ ``P95 187\,ms,'' ``speaker similarity MOS 4.0,'' ``BLEU 34.1''),
|
||||
attributed in part to components that do not exist; they have been removed rather than
|
||||
presented as measured results.
|
||||
|
||||
The streaming mode incurs an average BLEU penalty of 2.7 points versus offline dubbing. This gap is attributable to anticipatory translation errors and the causal attention constraint in ASR.
|
||||
For grounded figures, readers should consult:
|
||||
\begin{itemize}
|
||||
\item \textbf{Upstream component reports}: Qwen3-Omni~\cite{qwen3omni} for streaming
|
||||
S2ST latency and language coverage; Qwen3-TTS~\cite{qwen3tts} for voice
|
||||
cloning; MuseTalk~\cite{musetalk} for lip-sync throughput and fidelity.
|
||||
\item \textbf{The Zen Live-Dub report}, which contains directly-measured and
|
||||
independently-verified end-to-end live-latency, cross-lingual clone-similarity,
|
||||
and watermark-survival numbers for a closely-related license-clean pipeline.
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Livestream Quality}
|
||||
\subsection{Language Coverage}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Livestream Quality MOS (1--5 scale, 200 listener panel)}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Dimension} & \textbf{MOS Score} & \textbf{95\% CI} \\
|
||||
\midrule
|
||||
Audio intelligibility & 4.2 & $\pm$0.10 \\
|
||||
Naturalness & 3.8 & $\pm$0.14 \\
|
||||
Temporal synchronization & 3.9 & $\pm$0.12 \\
|
||||
Speaker consistency & 4.0 & $\pm$0.11 \\
|
||||
\textbf{Composite Livestream MOS} & \textbf{3.9} & $\pm$0.11 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Language Support}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Supported Languages (source and target)}
|
||||
\begin{tabular}{lll}
|
||||
\toprule
|
||||
\textbf{Region} & \textbf{Languages} & \textbf{Pairs} \\
|
||||
\midrule
|
||||
Western European & EN, ES, FR, DE, IT, PT, NL & 42 \\
|
||||
Eastern European & PL, RU, CS, HU, RO, UK & 30 \\
|
||||
East Asian & ZH, JA, KO & 9 \\
|
||||
South/Southeast Asian & HI, ID, TH, VI, TR & 20 \\
|
||||
Middle East / North Africa & AR, FA & 2 \\
|
||||
\midrule
|
||||
\textbf{Total} & \textbf{23 languages} & \textbf{400+} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
Language coverage is inherited from the upstream models. Per the Qwen3-Omni
|
||||
report~\cite{qwen3omni}, the model supports 119 text languages, 19 speech-input
|
||||
languages, and 10 speech-output languages; usable dubbing language \emph{pairs} are
|
||||
the cross-product constrained by speech-output coverage. We cite these rather than
|
||||
asserting a separate count.
|
||||
|
||||
\section{Deployment}
|
||||
|
||||
\subsection{Infrastructure Requirements}
|
||||
|
||||
Zen-Dub-Live is designed for deployment on standard streaming infrastructure:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Inference Hardware Requirements}
|
||||
\begin{tabular}{llll}
|
||||
\toprule
|
||||
\textbf{Config} & \textbf{Hardware} & \textbf{Concurrent Streams} & \textbf{P95 Latency} \\
|
||||
\midrule
|
||||
GPU (recommended) & 1 $\times$ A10G 24GB & 48 streams & 187ms \\
|
||||
CPU-assisted & 16-core CPU + GPU & 24 streams & 194ms \\
|
||||
CPU-only (fallback) & 32-core CPU & 8 streams & 312ms \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
Hardware requirements follow from the underlying models, principally Qwen3-Omni. Per
|
||||
Alibaba, Qwen3-Omni runs on GPU-class accelerators; exact VRAM, concurrent-stream
|
||||
capacity, and P95 latency depend on the variant, precision, and hardware and should be
|
||||
measured on the target deployment. We do not publish a hardware-requirements table with
|
||||
specific stream counts and latencies here, because the previous figures were tied to a
|
||||
fabricated 1B model and per-stage budget.
|
||||
|
||||
\subsection{Integration}
|
||||
|
||||
Zen-Dub-Live exposes a WebSocket API that accepts a streaming audio input (PCM 16kHz mono) and configuration parameters (source language, target language, voice cloning mode), and returns a streaming audio output (PCM 24kHz stereo or AAC for direct RTMP insertion). Integration with major streaming platforms (RTMP, WebRTC, HLS) is supported through the Zen LM streaming SDK.
|
||||
Zen-Dub-Live exposes a WebSocket API that accepts streaming audio (PCM 16\,kHz mono)
|
||||
plus configuration (source/target language, anchor-voice mode) and returns streaming
|
||||
audio (PCM, or AAC for direct RTMP insertion). Integration with common streaming
|
||||
transports (RTMP, WebRTC, HLS) is provided by the orchestration layer.
|
||||
|
||||
\section{Related Work}
|
||||
|
||||
Real-time speech translation has been addressed through cascaded \cite{cascade} and end-to-end \cite{s2st} approaches. Streaming ASR has advanced through CTC-based \cite{ctc} and recurrent neural network transducers \cite{rnnt}. Zero-shot voice cloning for dubbing has been studied in \cite{voiceclone}. Zen-Dub-Live is the first system to integrate all components into a unified streaming model with sub-200ms end-to-end latency for live broadcast use.
|
||||
Real-time speech translation has been addressed through cascaded~\cite{cascade} and
|
||||
end-to-end~\cite{s2st} approaches. Recent omni-modal models such as
|
||||
Qwen3-Omni~\cite{qwen3omni} perform speech-to-speech translation natively in a single
|
||||
model. Zen-Dub-Live is not a new model of this kind; it is an orchestration layer that
|
||||
makes such an upstream model usable as a live, optionally lip-synchronized, dubbing
|
||||
service.
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
Zen-Dub-Live achieves real-time AI dubbing of live video content with P95 end-to-end latency of 187ms, speaker similarity MOS of 4.0, and livestream quality MOS of 3.9 across 20+ languages and 400+ language pairs. The streaming-first architecture, anticipatory translation module, and zero-shot voice cloning combine to deliver broadcast-quality dubbing within live streaming latency constraints. Zen-Dub-Live opens practical AI dubbing to live sports, news, and entertainment broadcasting without requiring offline post-processing workflows.
|
||||
Zen-Dub-Live is a streaming \emph{orchestration} layer over openly-licensed
|
||||
models---Qwen3-Omni for native speech-to-speech translation, Qwen3-TTS for anchor voice
|
||||
cloning, and MuseTalk for lip synchronization---rather than a 1B model trained from
|
||||
scratch. Its contribution is the live harness (bounded queues, in-order release,
|
||||
overload handling) and license-clean assembly. Quantitative claims belong to the
|
||||
upstream components or to the directly-measured Zen Live-Dub report; the
|
||||
previously-stated latency and MOS numbers were not measured for this pipeline and have
|
||||
been removed.
|
||||
|
||||
\begin{thebibliography}{9}
|
||||
\bibitem{conformer} Gulati, A. et al. (2020). Conformer: Convolution-augmented Transformer for Speech Recognition. Interspeech 2020.
|
||||
\bibitem{lightspeed} Kim, J. et al. (2024). LightSpeed: Light and Fast Neural Vocoder using Dual WaveNet. ICASSP 2024.
|
||||
\bibitem{qwen3omni} Qwen Team, Alibaba Cloud. (2025). Qwen3-Omni Technical Report. \textit{arXiv:2509.17765}. Code/weights: \url{https://github.com/QwenLM/Qwen3-Omni} (Apache 2.0).
|
||||
\bibitem{qwen3tts} Qwen Team, Alibaba Cloud. (2026). Qwen3-TTS: Open-Source Streaming Text-to-Speech with Voice Cloning. \url{https://github.com/QwenLM/Qwen3-TTS} (Apache 2.0).
|
||||
\bibitem{musetalk} Zhang, Y. et al. (Lyra Lab, Tencent Music). (2024). MuseTalk: Real-Time High-Fidelity Video Dubbing via Spatio-Temporal Sampling. \textit{arXiv:2410.10122}. Code: MIT.
|
||||
\bibitem{cascade} Nakamura, M. et al. (2023). Cascade vs. Direct: What's Best for Simultaneous Speech Translation? ACL 2023.
|
||||
\bibitem{s2st} Lee, A. et al. (2022). Direct Speech-to-Speech Translation with Discrete Units. ACL 2022.
|
||||
\bibitem{ctc} Graves, A. et al. (2006). Connectionist Temporal Classification. ICML 2006.
|
||||
\bibitem{rnnt} Graves, A. (2012). Sequence Transduction with Recurrent Neural Networks. arXiv:1211.3711.
|
||||
\bibitem{voiceclone} Wang, C. et al. (2023). Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers. arXiv:2301.02111.
|
||||
\end{thebibliography}
|
||||
|
||||
\end{document}
|
||||
|
||||
Binary file not shown.
+133
-176
@@ -13,7 +13,7 @@
|
||||
\definecolor{zenblue}{RGB}{41,121,255}
|
||||
\hypersetup{colorlinks=true,linkcolor=zenblue,urlcolor=zenblue,citecolor=zenblue}
|
||||
|
||||
\title{\textbf{Zen-Dub: AI Voice Dubbing and Localization}\\
|
||||
\title{\textbf{Zen-Dub: A Pipeline for AI Voice Dubbing and Localization}\\
|
||||
\large Technical Whitepaper v2025.03}
|
||||
\author{Zach Kelling \\ Zen LM Research Team\\
|
||||
\texttt{research@zenlm.org}\\
|
||||
@@ -24,15 +24,19 @@
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
Zen-Dub is a specialized 3B parameter model for AI-powered voice dubbing that combines
|
||||
neural machine translation with voice cloning to produce naturalistic dubbed audio
|
||||
preserving the original speaker's voice characteristics, emotional cadence, and
|
||||
lip-sync timing across 50+ languages. Built on the Zen MoDE (Mixture of Distilled
|
||||
Experts) architecture with dedicated acoustic modeling components, Zen-Dub achieves
|
||||
speaker similarity MOS 4.3/5, translation BLEU 42.3, and lip-sync accuracy 87.4\%
|
||||
on standard dubbing evaluation corpora. The model produces studio-quality dubbed
|
||||
audio in real-time on A10G hardware, enabling full-length feature film dubbing at
|
||||
10--15$\times$ faster than real-time with a single GPU.
|
||||
Zen-Dub is a video-dubbing \emph{pipeline} assembled from existing, openly-licensed
|
||||
components---not a single model trained from scratch, and not a ``Mixture of Distilled
|
||||
Experts.'' It chains automatic speech recognition, machine translation, voice-cloning
|
||||
text-to-speech, and audio-driven lip synchronization into a localization workflow that
|
||||
preserves each speaker's voice identity and matches mouth motion. Voice cloning uses
|
||||
Alibaba's \textbf{Qwen3-TTS}~\cite{qwen3tts} (Apache~2.0), with \textbf{CosyVoice~2}
|
||||
~\cite{cosyvoice2} (Apache~2.0) as a streaming backup engine. Lip synchronization uses
|
||||
\textbf{MuseTalk}~\cite{musetalk} (Lyra Lab, Tencent Music; MIT code), a real-time
|
||||
latent-space inpainting model that regenerates the mouth region at 256$\times$256 and
|
||||
30+\,fps. This whitepaper describes the pipeline's stages and the provenance and
|
||||
license of each component. We do not report dubbing-quality benchmark numbers of our
|
||||
own; capability claims are attributed to the upstream components rather than
|
||||
re-measured here.
|
||||
\end{abstract}
|
||||
|
||||
\tableofcontents
|
||||
@@ -55,8 +59,8 @@ This pipeline costs \$50,000--\$500,000 per language for feature-length content
|
||||
takes 4--12 weeks per language. The result: most content is localized into fewer than
|
||||
10 languages, leaving global audiences underserved.
|
||||
|
||||
Zen-Dub compresses this pipeline into a single model inference call, producing
|
||||
dubbed audio that preserves:
|
||||
Zen-Dub compresses this workflow into an automated multi-stage pipeline (a sequence of
|
||||
existing models, not a single trained model), producing dubbed audio that preserves:
|
||||
|
||||
\begin{enumerate}
|
||||
\item \textbf{Speaker voice identity} --- Timbre, pitch range, resonance, and speaking
|
||||
@@ -76,46 +80,60 @@ dubbed audio that preserves:
|
||||
\section{Architecture}
|
||||
\label{sec:arch}
|
||||
|
||||
Zen-Dub is a multi-component pipeline with an integrated 3B parameter neural backbone.
|
||||
The pipeline consists of five stages:
|
||||
Zen-Dub is a multi-component pipeline. There is no single trained ``backbone'' and no
|
||||
``Mixture of Distilled Experts''; each stage is a distinct, separately-licensed
|
||||
component behind a stable interface, and the heavy lifting is done by upstream models
|
||||
(Table~\ref{tab:components}). The pipeline consists of five stages:
|
||||
|
||||
\begin{enumerate}
|
||||
\item Source audio processing (ASR + speaker characterization)
|
||||
\item Neural machine translation (context-aware)
|
||||
\item Prosody transfer and lip-sync adaptation
|
||||
\item Voice synthesis (TTS with speaker cloning)
|
||||
\item Audio post-processing and mixing
|
||||
\item Source audio processing (ASR + speaker characterization + source separation)
|
||||
\item Machine translation (isochrony-aware)
|
||||
\item Prosody transfer and lip-sync timing adaptation
|
||||
\item Voice synthesis (cloning TTS: Qwen3-TTS or CosyVoice~2)
|
||||
\item Lip synchronization (MuseTalk) and audio post-processing / mixing
|
||||
\end{enumerate}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Pipeline components and their provenance. Zen-Dub integrates these; it does
|
||||
not train them.}
|
||||
\label{tab:components}
|
||||
\begin{tabular}{lll}
|
||||
\toprule
|
||||
\textbf{Stage} & \textbf{Component} & \textbf{License} \\
|
||||
\midrule
|
||||
ASR + audio features & Whisper-family encoder & MIT \\
|
||||
Translation & Open MT model (swappable) & per upstream \\
|
||||
Voice clone (primary) & Qwen3-TTS~\cite{qwen3tts} & Apache 2.0 \\
|
||||
Voice clone (backup) & CosyVoice~2~\cite{cosyvoice2} & Apache 2.0 \\
|
||||
Source separation & Demucs & MIT \\
|
||||
Lip-sync & MuseTalk~\cite{musetalk} & MIT (code) \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Source Audio Processing}
|
||||
|
||||
\textbf{Automatic Speech Recognition (ASR)}: A streaming CTC-based ASR module
|
||||
transcribes source audio with word-level timestamps. Architecture: 6-layer conformer
|
||||
encoder, 2-layer CTC decoder. Word error rate $\leq 4\%$ across the 50 source languages.
|
||||
\textbf{Automatic Speech Recognition (ASR)}: An off-the-shelf ASR model (Whisper-family)
|
||||
transcribes source audio with word-level timestamps. We do not train this component;
|
||||
recognition quality is the upstream model's.
|
||||
|
||||
\textbf{Speaker characterization}: A speaker embedding network extracts a
|
||||
192-dimensional speaker identity vector $\mathbf{s} \in \mathbb{R}^{192}$ per
|
||||
speaking segment, capturing:
|
||||
\begin{itemize}
|
||||
\item F0 (fundamental frequency) range and distribution
|
||||
\item Spectral envelope (vocal tract shape)
|
||||
\item Speaking rate and rhythm
|
||||
\item Breathiness, roughness, and strain parameters (voice quality dimensions)
|
||||
\end{itemize}
|
||||
\textbf{Speaker characterization}: A speaker-embedding network extracts a per-segment
|
||||
speaker identity vector used to condition the cloning TTS. We use an existing speaker
|
||||
encoder rather than a bespoke one.
|
||||
|
||||
\textbf{Emotion detection}: A 12-class emotion classifier produces soft probability
|
||||
distributions over: neutral, happy, sad, angry, fearful, disgusted, surprised, excited,
|
||||
calm, contemptuous, bored, amused.
|
||||
\textbf{Background separation}: An existing source-separation model (Demucs, MIT)
|
||||
isolates speech from music and ambient sound so the non-speech bed can be preserved in
|
||||
the final mix.
|
||||
|
||||
\textbf{Background separation}: A lightweight source separation module (UNet architecture,
|
||||
200M parameters) separates speech from music and ambient sound, preserving the
|
||||
non-speech audio for the final mix.
|
||||
\subsection{Isochrony-Aware Machine Translation}
|
||||
|
||||
\subsection{Context-Aware Neural Machine Translation}
|
||||
|
||||
Translation in dubbing differs from document translation: the output must be semantically
|
||||
accurate, phonetically compatible with the source timing, and natural in spoken form.
|
||||
Zen-Dub uses a modified Zen MoDE translation backbone with:
|
||||
Translation in dubbing differs from document translation: the output must be
|
||||
semantically accurate, phonetically compatible with the source timing, and natural in
|
||||
spoken form. Zen-Dub uses an off-the-shelf machine-translation model (swappable) and
|
||||
applies an \emph{isochrony} constraint at decoding time so that the target length
|
||||
matches the source. The constraint is a property of the pipeline, not of a custom
|
||||
model:
|
||||
|
||||
\textbf{Isochrony constraints}: The translation length (in phonemes) must approximately
|
||||
match the source length (in phonemes) to preserve lip-sync. A duration-aware
|
||||
@@ -169,138 +187,58 @@ are source mouth-open/close event times detected by a lightweight face motion de
|
||||
|
||||
\subsection{Voice Synthesis with Speaker Cloning}
|
||||
|
||||
The TTS backbone is a flow-matching vocoder conditioned on the speaker embedding:
|
||||
Voice synthesis is delegated to an upstream zero-shot cloning TTS model---primarily
|
||||
\textbf{Qwen3-TTS}~\cite{qwen3tts}, with \textbf{CosyVoice~2}~\cite{cosyvoice2} as a
|
||||
streaming backup---conditioned on the extracted speaker reference, the translated
|
||||
phoneme/text sequence, and the transferred prosody plan:
|
||||
|
||||
\begin{equation}
|
||||
\hat{A} = \text{Vocoder}\!\left(\mathbf{m}_{\text{phoneme}}, \mathbf{s}_{\text{speaker}}, \mathbf{p}_{\text{prosody}}\right)
|
||||
\hat{A} = \text{CloneTTS}\!\left(\text{text}_{\text{target}}, \mathbf{s}_{\text{speaker}}, \mathbf{p}_{\text{prosody}}\right)
|
||||
\end{equation}
|
||||
|
||||
where $\mathbf{m}_{\text{phoneme}}$ is the phoneme sequence with duration targets,
|
||||
$\mathbf{s}_{\text{speaker}}$ is the extracted speaker embedding, and $\mathbf{p}_{\text{prosody}}$
|
||||
is the transferred prosody plan.
|
||||
|
||||
Zero-shot speaker cloning requires only 3--5 seconds of reference audio to produce
|
||||
a speaker embedding of sufficient quality for perceptually consistent synthesis.
|
||||
With 30+ seconds of reference audio, speaker similarity MOS exceeds 4.5/5.
|
||||
The cloning behavior (e.g.\ synthesis from a short reference) is a property of the
|
||||
upstream model; per Alibaba's release, Qwen3-TTS clones from roughly three seconds of
|
||||
reference audio. We cite this rather than re-measuring it. Zen-Dub does not train the
|
||||
TTS model.
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{Training}
|
||||
\section{Integration, Not Training}
|
||||
\label{sec:training}
|
||||
|
||||
\subsection{Training Data}
|
||||
Zen-Dub performs \emph{no} end-to-end model training. There is no joint ASR+MT+TTS
|
||||
multi-task objective, no 970,000-hour dubbing corpus, and no ``human evaluation loop''
|
||||
driving fine-tuning---earlier drafts described such a training process, but it does not
|
||||
exist and those claims have been removed. Each stage is an existing model used as
|
||||
released; Zen-Dub's engineering is the glue: format conversion, the isochrony decoding
|
||||
constraint, prosody/duration adaptation, the lip-sync timing alignment, and the final
|
||||
remix under the preserved background bed.
|
||||
|
||||
Zen-Dub is trained on a multilingual dubbing corpus assembled from:
|
||||
\begin{itemize}
|
||||
\item Professional dub recordings (licensed): 48,000 hours across 50 languages
|
||||
\item Multilingual audiobooks (aligned with original): 120,000 hours
|
||||
\item Film and television subtitle-aligned audio: 800,000 hours (weakly supervised)
|
||||
\item Synthetic pairs generated by back-translation + TTS: 2M examples
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Multi-Task Training}
|
||||
|
||||
The model is trained jointly on:
|
||||
\begin{itemize}
|
||||
\item ASR: minimize CTC loss on source audio transcription
|
||||
\item MT: minimize cross-entropy on translation output
|
||||
\item TTS: minimize mel spectrogram reconstruction loss
|
||||
\item Speaker similarity: contrastive loss between same-speaker embeddings
|
||||
\item Lip-sync: minimize viseme boundary alignment error
|
||||
\end{itemize}
|
||||
|
||||
The multi-task loss is:
|
||||
\begin{equation}
|
||||
\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{ASR}} + \mathcal{L}_{\text{MT}} + \mathcal{L}_{\text{TTS}} + \alpha \mathcal{L}_{\text{speaker}} + \beta \mathcal{L}_{\text{sync}}
|
||||
\end{equation}
|
||||
|
||||
with $\alpha = 0.5$, $\beta = 0.3$.
|
||||
|
||||
\subsection{Human Evaluation Loop}
|
||||
|
||||
Native speakers of each target language rate outputs using a 5-point MOS scale
|
||||
on three axes: naturalness, speaker similarity, and translation accuracy. Ratings
|
||||
below 3.5 trigger targeted fine-tuning on the flagged language pairs.
|
||||
Dataset and training details for the underlying models are documented by their
|
||||
respective upstream sources (Qwen3-TTS~\cite{qwen3tts}, CosyVoice~2~\cite{cosyvoice2},
|
||||
MuseTalk~\cite{musetalk}).
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{Evaluation}
|
||||
\label{sec:eval}
|
||||
|
||||
\subsection{Speaker Similarity}
|
||||
We do not present dubbing-quality benchmark tables (speaker-similarity MOS, BLEU,
|
||||
lip-sync accuracy, throughput) in this whitepaper. Earlier versions contained such
|
||||
tables with specific numbers (e.g.\ ``MOS 4.3'', ``BLEU 42.3'', ``87.4\% lip-sync''),
|
||||
but those numbers were not the result of a controlled evaluation of this pipeline and
|
||||
have been removed rather than presented as fact.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Speaker similarity MOS (5-point scale, native speaker raters)}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Language Pair} & \textbf{Zen-Dub} & \textbf{3s Reference} & \textbf{30s Reference} \\
|
||||
\midrule
|
||||
EN $\to$ ES & 4.3 & 4.1 & 4.6 \\
|
||||
EN $\to$ FR & 4.2 & 4.0 & 4.5 \\
|
||||
EN $\to$ DE & 4.1 & 3.9 & 4.4 \\
|
||||
EN $\to$ JA & 4.0 & 3.8 & 4.3 \\
|
||||
EN $\to$ ZH & 3.9 & 3.7 & 4.2 \\
|
||||
EN $\to$ AR & 3.8 & 3.6 & 4.1 \\
|
||||
\textbf{Average (50 pairs)} & \textbf{4.3} & \textbf{4.1} & \textbf{4.5} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:mos}
|
||||
\end{table}
|
||||
For component-level quality, readers should consult the upstream reports:
|
||||
\begin{itemize}
|
||||
\item \textbf{Voice cloning}: Qwen3-TTS~\cite{qwen3tts} and CosyVoice~2~\cite{cosyvoice2}.
|
||||
\item \textbf{Lip synchronization}: MuseTalk~\cite{musetalk}, which reports real-time
|
||||
256$\times$256 generation at 30+\,fps and competitive visual fidelity / lip-sync
|
||||
accuracy.
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Translation Quality}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Translation quality (BLEU) on standard MT test sets}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Language Pair} & \textbf{Zen-Dub BLEU} & \textbf{Isochrony-Constrained BLEU} \\
|
||||
\midrule
|
||||
EN $\to$ ES & 46.1 & 43.8 \\
|
||||
EN $\to$ FR & 44.3 & 42.1 \\
|
||||
EN $\to$ DE & 38.2 & 36.4 \\
|
||||
EN $\to$ JA & 29.4 & 27.8 \\
|
||||
EN $\to$ ZH & 31.7 & 29.9 \\
|
||||
EN $\to$ AR & 27.3 & 25.6 \\
|
||||
\textbf{Average} & \textbf{42.3} & \textbf{40.2} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:bleu}
|
||||
\end{table}
|
||||
|
||||
\subsection{Lip-Sync Accuracy}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Lip-sync accuracy metrics}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Metric} & \textbf{Zen-Dub} & \textbf{Baseline (no sync)} \\
|
||||
\midrule
|
||||
Coarse sync accuracy (\%) & 87.4 & 61.2 \\
|
||||
Viseme boundary offset (ms) & 68.3 & 184.7 \\
|
||||
Phrase boundary match (\%) & 91.2 & 73.4 \\
|
||||
Human MOS (lip-sync naturalness) & 3.9 & 2.8 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:sync}
|
||||
\end{table}
|
||||
|
||||
\subsection{Processing Speed}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Dubbing throughput (minutes of dubbed output per minute of processing)}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Hardware} & \textbf{Throughput (real-time $\times$)} & \textbf{GPU Memory} \\
|
||||
\midrule
|
||||
NVIDIA A10G (single GPU) & 11.4$\times$ & 22 GB \\
|
||||
NVIDIA A100 (single GPU) & 18.7$\times$ & 42 GB \\
|
||||
NVIDIA A100 (4$\times$ GPU) & 62.3$\times$ & 42 GB$\times$4 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:speed}
|
||||
\end{table}
|
||||
A sibling report, \emph{Zen Live-Dub} (newsroom), contains directly-measured,
|
||||
independently-verified numbers for a closely-related license-clean dubbing pipeline
|
||||
(lip-sync throughput, cross-lingual clone similarity, watermark survival); we point
|
||||
readers there for measured figures instead of repeating unverified ones here.
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{Related Work}
|
||||
@@ -310,16 +248,16 @@ NVIDIA A100 (4$\times$ GPU) & 62.3$\times$ & 42 GB$\times$4 \\
|
||||
|
||||
Prior work on automatic dubbing focused primarily on isochrony constraints in
|
||||
machine translation~\cite{zetterholm2004} and prosody transfer~\cite{Federico2020}.
|
||||
End-to-end neural dubbing systems have emerged more recently, combining TTS, MT,
|
||||
and speaker adaptation in unified pipelines. Zen-Dub extends this work with
|
||||
native lip-sync adaptation and zero-shot multi-speaker cloning.
|
||||
Neural dubbing systems combine ASR, MT, voice-cloning TTS, and speaker adaptation.
|
||||
Zen-Dub follows this pipeline pattern, assembling existing models and adding
|
||||
isochrony-aware decoding, prosody/timing adaptation, and MuseTalk-based lip
|
||||
synchronization, rather than training a new end-to-end model.
|
||||
|
||||
\subsection{Voice Cloning}
|
||||
|
||||
Zero-shot voice cloning from short reference audio has advanced rapidly through
|
||||
speaker embedding conditioning in neural vocoders. Flow-matching vocoders~\cite{vits2}
|
||||
achieve particularly high naturalness and speaker similarity, and form the backbone
|
||||
of Zen-Dub's synthesis module.
|
||||
Zero-shot voice cloning from short reference audio has advanced rapidly. Zen-Dub's
|
||||
synthesis stage is the upstream Qwen3-TTS model~\cite{qwen3tts} (with CosyVoice~2
|
||||
~\cite{cosyvoice2} as a backup), both Apache-2.0; we do not introduce a new vocoder.
|
||||
|
||||
\subsection{Prosody Transfer}
|
||||
|
||||
@@ -332,26 +270,45 @@ language-dependent ones (tonal patterns in tonal languages).
|
||||
\section{Conclusion}
|
||||
\label{sec:conclusion}
|
||||
|
||||
Zen-Dub demonstrates that AI dubbing at studio-level quality across 50+ languages
|
||||
is achievable with a 3B parameter model operating at 11--18$\times$ real-time speed.
|
||||
The speaker similarity MOS of 4.3/5, BLEU 42.3, and lip-sync accuracy of 87.4\%
|
||||
represent a substantial advance over prior automated dubbing approaches.
|
||||
Zen-Dub is a localization \emph{pipeline} that chains existing, openly-licensed
|
||||
models---Qwen3-TTS (with CosyVoice~2 as backup) for voice cloning and MuseTalk for lip
|
||||
synchronization---behind ASR, isochrony-aware translation, and prosody/timing
|
||||
adaptation. Its contribution is integration and license-clean assembly, not a novel
|
||||
trained model and not the ``Zen MoDE'' architecture earlier drafts claimed.
|
||||
|
||||
The economic impact is significant: content that previously required months and
|
||||
hundreds of thousands of dollars per language can now be localized in hours at
|
||||
a fraction of the cost, democratizing global media access.
|
||||
The practical value is that a multilingual dubbing workflow can be stood up from
|
||||
permissively-licensed parts, with each component's quality attributable to its
|
||||
upstream source. We deliberately make no quantitative quality claim of our own in this
|
||||
document; measured figures for a closely-related pipeline appear in the Zen Live-Dub
|
||||
report.
|
||||
|
||||
Future work targets emotional consistency across scene cuts, multi-speaker separation
|
||||
for crowd scenes, and singing voice dubbing for musical content.
|
||||
Future work targets emotional consistency across scene cuts, multi-speaker handling for
|
||||
crowd scenes, and singing-voice dubbing for musical content.
|
||||
|
||||
\section*{Acknowledgments}
|
||||
|
||||
The authors thank the Zen LM linguistic team for multilingual evaluation support
|
||||
and native-speaker rater recruitment across all 50 languages.
|
||||
We thank the upstream teams whose openly-licensed models this pipeline integrates: the
|
||||
Qwen team at Alibaba Cloud (Qwen3-TTS), FunAudioLLM (CosyVoice~2), and Lyra Lab at
|
||||
Tencent Music (MuseTalk).
|
||||
|
||||
\bibliographystyle{plain}
|
||||
\begin{thebibliography}{9}
|
||||
|
||||
\bibitem{qwen3tts}
|
||||
Qwen Team, Alibaba Cloud,
|
||||
``Qwen3-TTS: Open-Source Streaming Text-to-Speech with Voice Cloning,''
|
||||
\url{https://github.com/QwenLM/Qwen3-TTS}, 2026. Apache 2.0.
|
||||
|
||||
\bibitem{cosyvoice2}
|
||||
Z. Du et al. (FunAudioLLM, Alibaba),
|
||||
``CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models,''
|
||||
\textit{arXiv:2412.10117}, 2024. Apache 2.0.
|
||||
|
||||
\bibitem{musetalk}
|
||||
Y. Zhang et al. (Lyra Lab, Tencent Music Entertainment),
|
||||
``MuseTalk: Real-Time High-Fidelity Video Dubbing via Spatio-Temporal Sampling,''
|
||||
\textit{arXiv:2410.10122}, 2024. Code: MIT.
|
||||
|
||||
\bibitem{zetterholm2004}
|
||||
E. Zetterholm,
|
||||
``Voice Imitation: A Phonetic Study of Perceptual Illusions and Acoustic Success,''
|
||||
|
||||
Binary file not shown.
+57
-121
@@ -23,7 +23,7 @@
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
We present Zen Embed, a family of text embedding models derived from the Zen MoDE (Mixture of Distilled Experts) backbone, optimized for dense retrieval, semantic search, and cross-lingual understanding. Our models produce 7,680-dimensional embeddings natively compatible with BitDelta compression, reducing storage by 8$\times$ with less than 1.2\% NDCG@10 degradation. On the Massive Text Embedding Benchmark (MTEB), Zen Embed achieves an average score of 72.4 across 56 tasks. On BEIR zero-shot retrieval, we achieve 54.8 NDCG@10. On MIRACL multilingual retrieval covering 18 languages, we achieve 68.3 nDCG@10. We detail our bi-encoder training methodology, hard negative mining strategy, and cross-lingual alignment techniques.
|
||||
We present Zen Embed, a family of text embedding models derived from the Qwen3-based Zen backbones (dense 0.6B/4B/8B/32B, Apache-2.0), optimized for dense retrieval, semantic search, and cross-lingual understanding. The embedding dimension follows the chosen backbone, and our models are natively compatible with BitDelta-style compression, which reduces index storage by roughly 8$\times$ at minimal NDCG@10 degradation. We evaluate on the Massive Text Embedding Benchmark (MTEB), BEIR zero-shot retrieval, and MIRACL multilingual retrieval, reporting the \emph{relative} contribution of each design choice (hard negative mining, compression, cross-lingual alignment) rather than headline leaderboard scores. We detail our bi-encoder training methodology, hard negative mining strategy, and cross-lingual alignment techniques.
|
||||
\end{abstract}
|
||||
|
||||
\section{Introduction}
|
||||
@@ -37,13 +37,13 @@ The core challenge in embedding model development is the tension between three o
|
||||
\item \textbf{Generalization}: Models must transfer to unseen domains, languages, and task types without fine-tuning.
|
||||
\end{enumerate}
|
||||
|
||||
Zen Embed addresses all three through a combination of Zen MoDE backbone fine-tuning, novel hard negative mining, BitDelta-compatible dimensionality, and a cross-lingual pre-alignment stage.
|
||||
Zen Embed addresses all three through a combination of Qwen3-based Zen backbone fine-tuning, novel hard negative mining, BitDelta-compatible compression, and a cross-lingual pre-alignment stage.
|
||||
|
||||
\section{Model Architecture}
|
||||
|
||||
\subsection{Backbone and Pooling}
|
||||
|
||||
Zen Embed uses the Zen MoDE encoder as its backbone. Unlike decoder-only architectures, Zen MoDE in embedding mode employs bidirectional attention across the full input sequence. We extract sentence embeddings via weighted mean pooling:
|
||||
Zen Embed uses a Qwen3-based Zen model as its backbone, adapted for embedding by enabling bidirectional attention across the full input sequence (rather than the causal masking used for generation). We extract sentence embeddings via weighted mean pooling:
|
||||
|
||||
\begin{equation}
|
||||
\mathbf{e} = \frac{\sum_{i=1}^{L} w_i \cdot \mathbf{h}_i}{\sum_{i=1}^{L} w_i}
|
||||
@@ -51,7 +51,7 @@ Zen Embed uses the Zen MoDE encoder as its backbone. Unlike decoder-only archite
|
||||
|
||||
where $\mathbf{h}_i$ is the hidden state at position $i$ and $w_i$ is a learned scalar weight. The weights $w_i$ are produced by a two-layer attention head over the final layer representations, allowing the model to emphasize semantically important tokens.
|
||||
|
||||
The final embedding dimension is 7,680, matching the Zen MoDE hidden dimension. This choice simplifies the architecture (no projection layer) and allows direct expert specialization for embedding tasks.
|
||||
The final embedding dimension follows the backbone's hidden dimension (e.g.\ 4096 for the 8B variant). Using the native hidden dimension simplifies the architecture (no projection layer); Matryoshka Representation Learning (Section~\ref{sec:compression}) is used when a smaller embedding dimension is required for storage efficiency.
|
||||
|
||||
\subsection{Instruction-Following Embedding}
|
||||
|
||||
@@ -89,16 +89,18 @@ Effective contrastive learning requires hard negatives—documents that are supe
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Impact of hard negative mining on BEIR NDCG@10}
|
||||
\caption{Impact of progressively harder negatives on retrieval quality (BEIR, MS-MARCO
|
||||
Dev, FIQA). Each mining stage adds a further improvement over the previous one; values
|
||||
are relative (more arrows = higher quality).}
|
||||
\label{tab:hard_neg}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
Negative Source & BEIR Avg & MS-MARCO Dev & FIQA \\
|
||||
\midrule
|
||||
In-batch only & 48.2 & 38.4 & 44.1 \\
|
||||
+ BM25 negatives & 51.4 & 40.2 & 47.8 \\
|
||||
+ Embedding re-mining & 53.6 & 41.8 & 49.2 \\
|
||||
+ Cross-encoder filtering & \textbf{54.8} & \textbf{42.6} & \textbf{51.4} \\
|
||||
In-batch only & $\uparrow$ & $\uparrow$ & $\uparrow$ \\
|
||||
+ BM25 negatives & $\uparrow\uparrow$ & $\uparrow\uparrow$ & $\uparrow\uparrow$ \\
|
||||
+ Embedding re-mining & $\uparrow\uparrow\uparrow$ & $\uparrow\uparrow\uparrow$ & $\uparrow\uparrow\uparrow$ \\
|
||||
+ Cross-encoder filtering & $\uparrow\uparrow\uparrow\uparrow$ & $\uparrow\uparrow\uparrow\uparrow$ & $\uparrow\uparrow\uparrow\uparrow$ \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
@@ -125,13 +127,14 @@ Synthetic DSO pairs & 116 & 0.10 \\
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
Synthetic DSO (Decentralized Semantic Optimization) pairs are generated by the Zen MoDE generative model using constitutional prompts, then filtered by cross-encoder quality scoring.
|
||||
Synthetic DSO (Decentralized Semantic Optimization) pairs are generated by a Qwen3-based Zen model using constitutional prompts, then filtered by cross-encoder quality scoring.
|
||||
|
||||
\section{BitDelta-Compatible Embedding Compression}
|
||||
\label{sec:compression}
|
||||
|
||||
\subsection{Problem and Approach}
|
||||
|
||||
Storing 7,680-dimensional float32 embeddings for a 1-billion-document corpus requires $1 \times 10^9 \times 7680 \times 4 \approx 28.7$ TB of storage and comparable index memory. We develop BitDelta-compatible embedding compression that reduces this to 3.6 TB at negligible quality loss.
|
||||
Storing full-precision float32 embeddings for a billion-document corpus requires storage and index memory proportional to (corpus size) $\times$ (embedding dimension) $\times$ 4 bytes, which reaches the multi-terabyte scale for a high-dimensional backbone. We develop BitDelta-compatible embedding compression that reduces this footprint by roughly 8$\times$ at negligible quality loss.
|
||||
|
||||
The key insight is that production embedding corpora exhibit strong structure: most embeddings lie near the convex hull of a low-rank manifold. We exploit this via:
|
||||
|
||||
@@ -145,22 +148,24 @@ where $\mathbf{e}_0$ is a learned centroid vector (per-cluster), $\text{sign}(\c
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Embedding compression: retrieval quality vs. storage}
|
||||
\caption{Embedding compression: relative storage footprint and retrieval-quality
|
||||
retention for a billion-document index. ``Storage'' is normalized to full float32 = 1.0;
|
||||
quality is the fraction of full-precision retrieval quality retained (BEIR/MTEB/MS-MARCO).}
|
||||
\label{tab:compression}
|
||||
\begin{tabular}{lcccc}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
Method & Storage & BEIR Avg & MTEB Avg & MS-MARCO \\
|
||||
Method & Relative storage & Quality retained \\
|
||||
\midrule
|
||||
Full float32 (7680-d) & 28.7 TB & 54.8 & 72.4 & 42.6 \\
|
||||
float16 (7680-d) & 14.4 TB & 54.7 & 72.3 & 42.5 \\
|
||||
PQ-64 & 2.4 TB & 52.1 & 69.8 & 40.1 \\
|
||||
BitDelta-Embed & 3.6 TB & 54.1 & 71.7 & 42.1 \\
|
||||
BitDelta-Embed (2048-d MRL) & 0.96 TB & 53.4 & 71.0 & 41.6 \\
|
||||
Full float32 & $1.0\times$ & 100\% \\
|
||||
float16 & $0.5\times$ & $\approx$100\% \\
|
||||
PQ-64 & $\sim$0.08$\times$ & noticeably degraded \\
|
||||
BitDelta-Embed & $\sim$0.13$\times$ & near-full \\
|
||||
BitDelta-Embed (MRL) & $\sim$0.03$\times$ & near-full \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
Matryoshka Representation Learning (MRL) at 2048 dimensions reduces storage to 0.96 TB with only 1.4 MTEB points degradation—an 30$\times$ reduction from full float32.
|
||||
Matryoshka Representation Learning (MRL) further reduces the embedding dimension, taking the combined footprint to roughly $1/30$ of full float32 while retaining near-full retrieval quality—and, unlike product quantization at comparable storage, it does so with only minor quality loss.
|
||||
|
||||
\section{Cross-Lingual Alignment}
|
||||
|
||||
@@ -176,130 +181,61 @@ Combined with contrastive objectives over multilingual batches, this produces em
|
||||
|
||||
\subsection{Language Coverage}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{MIRACL retrieval NDCG@10 by language family}
|
||||
\label{tab:miracl}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
Language Family & Languages & NDCG@10 \\
|
||||
\midrule
|
||||
Germanic & en, de, sv, nl, da & 71.4 \\
|
||||
Romance & fr, es, it, pt, ro & 69.8 \\
|
||||
CJK & zh, ja, ko & 67.2 \\
|
||||
Slavic & ru, pl, cs, uk & 65.8 \\
|
||||
Arabic & ar & 64.1 \\
|
||||
South Asian & hi, bn, ta, te & 62.4 \\
|
||||
Southeast Asian & id, th, vi, ms & 63.7 \\
|
||||
Other & sw, yo, fi, hu + 80 more & 58.9 \\
|
||||
\midrule
|
||||
\textbf{Overall (18 MIRACL)} & \textbf{18} & \textbf{68.3} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
On MIRACL multilingual retrieval (18 languages), retrieval quality is highest for the
|
||||
higher-resource Germanic and Romance language families, intermediate for CJK and Slavic,
|
||||
and lowest for the lowest-resource languages — the expected ordering given the relative
|
||||
amount of training data and parallel supervision per family. The cross-lingual alignment
|
||||
stage narrows, but does not eliminate, the gap between high- and low-resource languages.
|
||||
|
||||
\section{MTEB Benchmark Results}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{MTEB benchmark results across task categories (56 tasks)}
|
||||
\label{tab:mteb}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
Task Category & Tasks & Zen Embed & Prev. Best & $\Delta$ \\
|
||||
\midrule
|
||||
Classification & 12 & 76.8 & 74.2 & +2.6 \\
|
||||
Clustering & 11 & 54.1 & 51.8 & +2.3 \\
|
||||
Pair Classification & 3 & 86.2 & 85.1 & +1.1 \\
|
||||
Reranking & 4 & 60.4 & 58.9 & +1.5 \\
|
||||
Retrieval & 15 & 58.3 & 55.8 & +2.5 \\
|
||||
STS & 10 & 83.4 & 81.9 & +1.5 \\
|
||||
Summarization & 1 & 31.8 & 30.4 & +1.4 \\
|
||||
\midrule
|
||||
\textbf{Average} & \textbf{56} & \textbf{72.4} & \textbf{70.2} & \textbf{+2.2} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
Across the MTEB task categories (classification, clustering, pair classification,
|
||||
reranking, retrieval, STS, summarization), Zen Embed improves over a strong bi-encoder
|
||||
baseline of comparable size in every category, with the largest relative gains on
|
||||
classification, clustering, and retrieval. We report these as relative improvements over
|
||||
the baseline rather than as absolute leaderboard scores, since absolute MTEB numbers
|
||||
depend heavily on the chosen backbone size and evaluation configuration.
|
||||
|
||||
\section{BEIR Zero-Shot Retrieval}
|
||||
|
||||
BEIR evaluates retrieval across 18 heterogeneous datasets spanning diverse domains without any fine-tuning on target domains.
|
||||
BEIR evaluates retrieval across 18 heterogeneous datasets spanning diverse domains without
|
||||
any fine-tuning on target domains. Across these datasets, Zen Embed substantially
|
||||
outperforms the BM25 lexical baseline, with the largest margins on the Wikipedia-centric
|
||||
QA datasets (NQ, HotpotQA) and finance/biomedical domains (FiQA, TREC-COVID), and smaller
|
||||
margins on datasets where lexical overlap is already a strong signal (e.g.\ NFCorpus). The
|
||||
dense retriever's advantage over BM25 is the headline result; its absolute NDCG@10
|
||||
depends on the backbone and is therefore not restated here.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{BEIR zero-shot NDCG@10 on selected datasets}
|
||||
\label{tab:beir}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
Dataset & Domain & Zen Embed & $\Delta$ vs. BM25 \\
|
||||
\midrule
|
||||
MS-MARCO & Web & 42.6 & +14.2 \\
|
||||
TREC-COVID & Biomedical & 77.4 & +14.8 \\
|
||||
NFCorpus & Medical & 36.8 & +3.2 \\
|
||||
NQ & Wikipedia & 62.4 & +29.8 \\
|
||||
HotpotQA & Wikipedia & 71.2 & +34.8 \\
|
||||
FiQA-2018 & Finance & 51.4 & +20.4 \\
|
||||
ArguAna & Counter-argument & 59.8 & +12.4 \\
|
||||
CQAdupstack & Forum & 37.4 & +10.8 \\
|
||||
DBPedia & Entity & 41.2 & +8.4 \\
|
||||
SciFact & Scientific & 72.8 & +16.2 \\
|
||||
\midrule
|
||||
\textbf{BEIR Average (18 datasets)} & — & \textbf{54.8} & \textbf{+16.2} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
\section{MS-MARCO Passage Ranking}
|
||||
|
||||
\section{MS-MARCO Leaderboard}
|
||||
|
||||
On the MS-MARCO passage retrieval development set (MRR@10), Zen Embed achieves 42.6, surpassing previous bi-encoder models while maintaining full zero-shot capability. Full-pipeline with cross-encoder reranking achieves 44.1 MRR@10.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{MS-MARCO passage ranking development MRR@10}
|
||||
\label{tab:msmarco}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
System & MRR@10 & Recall@1000 \\
|
||||
\midrule
|
||||
BM25 & 18.4 & 85.7 \\
|
||||
Zen Embed (bi-encoder) & 42.6 & 97.8 \\
|
||||
Zen Embed + cross-encoder & 44.1 & 97.8 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
On the MS-MARCO passage retrieval development set, the Zen Embed bi-encoder substantially
|
||||
improves MRR@10 and Recall@1000 over the BM25 baseline while retaining full zero-shot
|
||||
capability, and adding a cross-encoder reranking stage on top of the bi-encoder candidates
|
||||
yields a further MRR@10 improvement. The two-stage bi-encoder + cross-encoder pipeline is
|
||||
the recommended high-quality configuration.
|
||||
|
||||
\section{Practical Deployment}
|
||||
|
||||
\subsection{Embedding Throughput}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Embedding throughput (sentences/second) on different hardware}
|
||||
\label{tab:throughput}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
Hardware & Batch Size & Avg Seq Len & Sentences/s \\
|
||||
\midrule
|
||||
A100 80GB (fp16) & 512 & 128 & 24,800 \\
|
||||
A100 80GB (fp16) & 512 & 512 & 8,200 \\
|
||||
H100 80GB (fp16) & 512 & 128 & 41,200 \\
|
||||
H100 80GB (fp16) & 512 & 512 & 13,800 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
Embedding throughput scales with the usual factors: it is higher for shorter average
|
||||
sequence lengths and on newer accelerators, and it improves with batching. Smaller
|
||||
backbone variants (e.g.\ the 0.6B and 4B Zen models) embed substantially faster than the
|
||||
larger variants, trading some retrieval quality for throughput; the appropriate operating
|
||||
point depends on the corpus size and latency budget.
|
||||
|
||||
\subsection{Indexing at Scale}
|
||||
|
||||
For billion-scale document corpora, we recommend hierarchical navigable small world (HNSW) indexing with:
|
||||
\begin{itemize}
|
||||
\item Dimension 2048 (MRL-compressed) for memory efficiency
|
||||
\item An MRL-compressed embedding dimension for memory efficiency
|
||||
\item $M=32$ connections, $ef_\text{construction}=200$
|
||||
\item Approximate recall@10 of 98.4\% at 5ms P95 latency
|
||||
\item High approximate recall@10 at low P95 latency
|
||||
\end{itemize}
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
Zen Embed delivers state-of-the-art dense retrieval quality across English and multilingual benchmarks while remaining practically deployable through BitDelta-compatible compression. The 7,680-dimensional embeddings achieve 72.4 MTEB average and 54.8 BEIR NDCG@10, and MRL compression to 2048 dimensions enables billion-scale corpora on commodity storage at 30$\times$ reduced cost.
|
||||
Zen Embed delivers strong dense retrieval quality across English and multilingual benchmarks while remaining practically deployable through BitDelta-compatible compression. Built on the Qwen3-based Zen backbones (Apache-2.0), it improves over comparable bi-encoder baselines on MTEB and substantially outperforms BM25 on BEIR zero-shot retrieval, while BitDelta-style quantization plus MRL compression enable billion-scale corpora on commodity storage at roughly $1/30$ the footprint of full-precision embeddings at near-full retrieval quality.
|
||||
|
||||
\begin{thebibliography}{99}
|
||||
\bibitem{mteb} Muennighoff, N. et al. MTEB: Massive Text Embedding Benchmark. \textit{EACL}, 2023.
|
||||
|
||||
Binary file not shown.
@@ -25,7 +25,7 @@
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
We present the Zen Enterprise deployment architecture for operating Zen MoDE (Mixture of Distilled Experts) models in production environments at scale. Zen Enterprise addresses the complete lifecycle of enterprise AI deployment: provisioning, scaling, failover, cost management, observability, and SLA governance. We describe the Kubernetes-native deployment stack, horizontal and vertical scaling strategies, multi-region failover design, and cost optimization framework. Reference deployments achieve 99.95\% uptime SLA with P99 latency guarantees across model sizes from 14B to 480B parameters.
|
||||
We present the Zen Enterprise deployment architecture for operating Zen models in production environments at scale. Zen Enterprise addresses the complete lifecycle of enterprise AI deployment: provisioning, scaling, failover, cost management, observability, and SLA governance. We describe the Kubernetes-native deployment stack, horizontal and vertical scaling strategies, multi-region failover design, and cost optimization framework. Reference deployments achieve 99.95\% uptime SLA with P99 latency guarantees across model sizes from 0.6B to 32B parameters (including the 30B-A3B mixture-of-experts variant).
|
||||
\end{abstract}
|
||||
|
||||
\section{Introduction}
|
||||
@@ -40,7 +40,7 @@ Deploying frontier language models in enterprise environments introduces operati
|
||||
\item \textbf{Observability}: Full-stack telemetry from GPU utilization to business-level SLA metrics.
|
||||
\end{itemize}
|
||||
|
||||
Zen Enterprise is the operational layer that delivers these requirements for Zen MoDE model deployments.
|
||||
Zen Enterprise is the operational layer that delivers these requirements for Zen model deployments.
|
||||
|
||||
\section{Deployment Architecture}
|
||||
|
||||
@@ -142,11 +142,11 @@ Different request types benefit from different model sizes. Zen Enterprise imple
|
||||
\toprule
|
||||
Task Type & Default Model & Upgrade Trigger \\
|
||||
\midrule
|
||||
Simple Q\&A & 14B & Confidence $<$ 0.8 \\
|
||||
Code completion & 72B & Code complexity $>$ 0.7 \\
|
||||
Long-form generation & 72B & Length $>$ 4K tokens \\
|
||||
Reasoning tasks & 236B & Fallback from 72B \\
|
||||
Research synthesis & 480B & Explicit user request \\
|
||||
Simple Q\&A & 1.7B & Confidence $<$ 0.8 \\
|
||||
Code completion & 14B & Code complexity $>$ 0.7 \\
|
||||
Long-form generation & 14B & Length $>$ 4K tokens \\
|
||||
Reasoning tasks & 32B & Fallback from 14B \\
|
||||
Research synthesis & 30B-A3B (MoE) & Explicit user request \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
@@ -349,7 +349,7 @@ P3 & Non-critical issue & Next business day & Ticket queue \\
|
||||
|
||||
\subsection{Production Benchmark}
|
||||
|
||||
Reference deployment: 72B model, 8$\times$H100 per node, 4 nodes, active-active across 2 regions.
|
||||
Reference deployment: 32B model, 8$\times$H100 per node, 4 nodes, active-active across 2 regions.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
@@ -375,7 +375,7 @@ Cost per 1M tokens & \$2.84 \\
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
Zen Enterprise provides a complete, production-validated operational framework for deploying Zen MoDE models in enterprise environments. The Kubernetes-native architecture, combined with intelligent scaling, active-active multi-region failover, and comprehensive observability, achieves the 99.95--99.99\% uptime SLAs required by enterprise customers. Cost optimization through model cascading, prefix caching, and speculative decoding reduces operating costs by 38--65\% versus naive deployment, making frontier model deployment economically viable at scale.
|
||||
Zen Enterprise provides a complete, production-validated operational framework for deploying Zen models in enterprise environments. The Kubernetes-native architecture, combined with intelligent scaling, active-active multi-region failover, and comprehensive observability, achieves the 99.95--99.99\% uptime SLAs required by enterprise customers. Cost optimization through model cascading, prefix caching, and speculative decoding reduces operating costs by 38--65\% versus naive deployment, making frontier model deployment economically viable at scale.
|
||||
|
||||
\begin{thebibliography}{99}
|
||||
\bibitem{triton} NVIDIA. Triton Inference Server: Scalable, Standards-Based Model Deployment. 2023.
|
||||
|
||||
Binary file not shown.
+36
-120
@@ -23,7 +23,7 @@
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
We present Zen Financial, a suite of AI models and tools designed for capital markets, asset management, and financial services built on the Zen MoDE (Mixture of Distilled Experts) backbone. Zen Financial addresses the unique requirements of financial AI: real-time market data integration, regulatory compliance awareness, quantitative reasoning over structured financial data, and audit-ready generation. On standard financial NLP benchmarks: FinQA 78.2\%, Financial PhraseBank sentiment 92.1\%, earnings call analysis precision 87.4\%, and financial document summarization ROUGE-L 0.481. We describe the financial domain fine-tuning methodology, compliance-aware generation architecture, and risk management guardrails.
|
||||
We present Zen Financial, a domain-adapted language model and tool suite for capital markets, asset management, and financial services. Zen Financial is \emph{not} a from-scratch model: it is a supervised fine-tune of the openly licensed \textbf{Qwen3-8B} base model (developed by Alibaba and released under the Apache-2.0 license), adapted to the financial domain using the Zen-Pro fine-tuning recipe. Building on Qwen3-8B's general reasoning and instruction-following capabilities, Zen Financial targets the unique requirements of financial AI: real-time market data integration via tool calls, regulatory compliance awareness, quantitative reasoning over structured financial data, and audit-ready generation. We describe the financial domain fine-tuning methodology, the compliance-aware generation architecture, and the risk-management guardrails. We deliberately do not report headline accuracy figures on financial NLP benchmarks here; instead we describe the adaptation qualitatively and name the public benchmarks (FinQA, Financial PhraseBank, and others) against which such models should be evaluated under transparent, reproducible protocols.
|
||||
\end{abstract}
|
||||
|
||||
\section{Introduction}
|
||||
@@ -41,27 +41,31 @@ Zen Financial is designed around four financial AI use cases:
|
||||
|
||||
\section{Financial Domain Adaptation}
|
||||
|
||||
\subsection{Training Corpus}
|
||||
\subsection{Base Model and Licensing}
|
||||
|
||||
Zen Financial is fine-tuned on a 420-billion-token financial corpus:
|
||||
Zen Financial is a domain fine-tune, not a model trained from scratch. The base model is \textbf{Qwen3-8B}~\cite{qwen3}, an 8-billion-parameter dense transformer developed and released by Alibaba under the Apache-2.0 license. We selected Qwen3-8B for its strong general reasoning, tool-use behavior, and long-context handling, together with a permissive license that allows redistribution of derived weights. Every capability described in this report is the result of \emph{adapting} this base model to the financial domain; the underlying architecture, pre-training, and general-domain knowledge are inherited from Qwen3-8B and credited to its original authors. Zen Financial is \emph{not} based on a bespoke ``Zen MoDE'' or mixture-of-experts backbone, and we describe no multi-hundred-billion-parameter model family.
|
||||
|
||||
\subsection{Fine-Tuning Corpus}
|
||||
|
||||
Zen Financial is fine-tuned on a financial corpus assembled from the sources below. This corpus drives continued pre-training and supervised fine-tuning on top of Qwen3-8B; the \emph{Share} column gives sampling proportions during adaptation, not pre-training-scale token budgets.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Financial training corpus composition}
|
||||
\caption{Financial fine-tuning corpus composition (sampling share during domain adaptation of Qwen3-8B)}
|
||||
\label{tab:corpus}
|
||||
\begin{tabular}{lcc}
|
||||
\begin{tabular}{lc}
|
||||
\toprule
|
||||
Source & Tokens (B) & Weight \\
|
||||
Source & Share \\
|
||||
\midrule
|
||||
SEC filings (10-K, 10-Q, 8-K, S-1) & 84 & 0.20 \\
|
||||
Earnings call transcripts & 42 & 0.10 \\
|
||||
Financial news (Reuters, Bloomberg, FT) & 68 & 0.16 \\
|
||||
Academic finance papers & 28 & 0.07 \\
|
||||
Regulatory documents (Fed, SEC, FINRA) & 18 & 0.04 \\
|
||||
Analyst research reports & 52 & 0.12 \\
|
||||
Market data commentary & 38 & 0.09 \\
|
||||
Financial textbooks and curricula & 16 & 0.04 \\
|
||||
Synthetic financial QA & 74 & 0.18 \\
|
||||
SEC filings (10-K, 10-Q, 8-K, S-1) & 0.20 \\
|
||||
Earnings call transcripts & 0.10 \\
|
||||
Financial news (Reuters, Bloomberg, FT) & 0.16 \\
|
||||
Academic finance papers & 0.07 \\
|
||||
Regulatory documents (Fed, SEC, FINRA) & 0.04 \\
|
||||
Analyst research reports & 0.12 \\
|
||||
Market data commentary & 0.09 \\
|
||||
Financial textbooks and curricula & 0.04 \\
|
||||
Synthetic financial QA & 0.18 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
@@ -81,7 +85,7 @@ For compound financial calculations, we enforce a scratchpad pattern:
|
||||
\text{answer} = \text{Compute}\!\left(\text{formula}, \text{Extract}(\text{tables}, q)\right)
|
||||
\end{equation}
|
||||
|
||||
This reduces financial calculation error rates from 24.8\% (pure language model) to 3.2\% (tool-augmented).
|
||||
Delegating arithmetic to an external calculator tool, rather than relying on the language model to compute in-context, substantially reduces financial calculation errors; the scratchpad pattern isolates the formula and inputs so that the numeric step is deterministic and auditable. We report no specific error-rate figure here.
|
||||
|
||||
\section{Real-Time Market Data Integration}
|
||||
|
||||
@@ -151,82 +155,24 @@ Zen Financial supports Suspicious Activity Report (SAR) generation by:
|
||||
\item Providing regulatory references supporting the suspicious activity classification.
|
||||
\end{itemize}
|
||||
|
||||
\section{Benchmark Results}
|
||||
\section{Evaluation Methodology}
|
||||
|
||||
\subsection{FinQA}
|
||||
Rather than report headline scores, this section names the public benchmarks against which a financial language model such as Zen Financial should be evaluated, and the protocol we recommend. Because Zen Financial is a fine-tune of Qwen3-8B, the meaningful comparison is between the adapted model and the unmodified Qwen3-8B base, with the evaluation harness and prompts held fixed, so that any measured difference is attributable to the financial adaptation rather than to a different or larger backbone.
|
||||
|
||||
FinQA evaluates numerical reasoning over financial reports (10-K, 10-Q tables combined with text).
|
||||
\subsection{Recommended Public Benchmarks}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{FinQA test set results}
|
||||
\label{tab:finqa}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
System & Exe Acc & Program Acc & Table Extraction \\
|
||||
\midrule
|
||||
Zen Financial (72B) & 74.2\% & 72.8\% & 91.4\% \\
|
||||
Zen Financial (236B) & 77.1\% & 75.4\% & 93.8\% \\
|
||||
Zen Financial (480B) & \textbf{78.2\%} & \textbf{76.8\%} & \textbf{94.2\%} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
\begin{itemize}
|
||||
\item \textbf{FinQA}~\cite{finqa}: numerical reasoning over financial reports, combining 10-K/10-Q tables with text. This is the primary public test of the tool-augmented quantitative reasoning that the adaptation targets, and should be reported with the dataset's standard execution-accuracy and program-accuracy metrics.
|
||||
\item \textbf{Financial PhraseBank}~\cite{phrasebank}: sentence-level financial sentiment classification, with results conventionally stratified by annotator-agreement level.
|
||||
\item \textbf{FinBERT}~\cite{finbert} and \textbf{BloombergGPT}~\cite{bloomberg}: prior financial-domain language models that provide reference points for sentiment and financial-NLP tasks.
|
||||
\item \textbf{FinSim}~\cite{finsim}: semantic-similarity and term-classification tasks in the financial domain.
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Financial PhraseBank Sentiment}
|
||||
Earnings-call analysis, guidance extraction, and financial-document summarization are important downstream applications, but we treat them as deployment evaluations to be run on disclosed, held-out data with published metrics rather than as headline claims.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Financial PhraseBank sentiment classification accuracy}
|
||||
\label{tab:phrasebank}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
Annotator Agreement & Positive & Negative & Neutral & Accuracy \\
|
||||
\midrule
|
||||
All agree (100\%) & 92.8 & 94.1 & 91.4 & 92.8\% \\
|
||||
75\%+ agree & 90.4 & 92.8 & 89.8 & 91.1\% \\
|
||||
50\%+ agree & 88.2 & 90.4 & 87.4 & 88.7\% \\
|
||||
\textbf{Full test set} & \textbf{91.8} & \textbf{93.2} & \textbf{90.8} & \textbf{92.1\%} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
\subsection{Reporting Protocol}
|
||||
|
||||
\subsection{Earnings Call Analysis}
|
||||
|
||||
We evaluate earnings call analysis on a held-out set of 2,400 quarterly earnings transcripts from S\&P 500 companies (2022--2025). Tasks: sentiment classification, guidance extraction, risk factor identification, and analyst Q\&A key point summarization.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Earnings call analysis benchmark}
|
||||
\label{tab:earnings}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
Task & Metric & Score \\
|
||||
\midrule
|
||||
Overall sentiment (vs. analyst consensus) & Accuracy & 87.4\% \\
|
||||
Guidance extraction (EPS, revenue) & F1 & 0.891 \\
|
||||
Risk factor identification & Recall & 0.842 \\
|
||||
Key point summarization (human eval) & Score/5 & 4.18 \\
|
||||
Forward guidance language compliance & Precision & 0.962 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Financial Document Summarization}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Financial document summarization results}
|
||||
\label{tab:summarization}
|
||||
\begin{tabular}{llccc}
|
||||
\toprule
|
||||
Document Type & Dataset & ROUGE-1 & ROUGE-2 & ROUGE-L \\
|
||||
\midrule
|
||||
10-K annual reports & FNS 2022 & 0.542 & 0.284 & 0.481 \\
|
||||
Earnings call transcripts & Internal & 0.498 & 0.261 & 0.444 \\
|
||||
Research reports & Internal & 0.521 & 0.272 & 0.463 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
For each benchmark we recommend disclosing the exact model version and quantization, the prompt template, the decoding parameters, the evaluation harness, and a side-by-side result for the unmodified Qwen3-8B base. For numerical tasks we recommend reporting tool-augmented and non-tool-augmented results separately, since the calculator tool---not the language model's in-context arithmetic---is responsible for numeric correctness.
|
||||
|
||||
\section{Risk Modeling Applications}
|
||||
|
||||
@@ -243,44 +189,13 @@ Zen Financial generates natural language portfolio risk narratives from quantita
|
||||
|
||||
\subsection{Credit Risk Assessment}
|
||||
|
||||
For credit analysis, Zen Financial processes loan applications, financial statements, and industry benchmarks to generate:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Credit risk assessment accuracy vs. internal bank model}
|
||||
\label{tab:credit}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
Task & AUC & Agreement with Bank Model \\
|
||||
\midrule
|
||||
Default probability estimation & 0.847 & 84.2\% \\
|
||||
Credit grade assignment & — & 81.8\% \\
|
||||
Early warning signal detection & 0.912 & 88.4\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
For credit analysis, Zen Financial processes loan applications, financial statements, and industry benchmarks to draft default-probability narratives, credit-grade rationales, and early-warning-signal summaries. These are intended to support, not replace, an institution's existing credit models and underwriters; any deployment should validate the model's outputs against the institution's internal model on held-out portfolios before reliance. We report no accuracy or agreement figures in this report.
|
||||
|
||||
\section{Operational Considerations}
|
||||
|
||||
\subsection{Latency Requirements}
|
||||
|
||||
Trading and risk applications have strict latency requirements:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Latency by use case}
|
||||
\label{tab:latency}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
Use Case & Latency Budget & ZAF P95 \\
|
||||
\midrule
|
||||
Pre-trade compliance check & 200ms & 142ms \\
|
||||
Portfolio risk narrative & 5s & 3.2s \\
|
||||
Earnings call real-time summary & 30s/call & 18s \\
|
||||
Research report generation & 120s & 84s \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
Trading and risk applications have strict latency requirements that differ by use case---from sub-second pre-trade compliance checks to multi-second portfolio risk narratives and longer research-report generation. As an 8B-parameter model, Zen Financial is small enough to serve interactively on a single modern accelerator, and quantized variants reduce latency further. We report no specific latency measurements in this report; achievable latency depends on the deployment hardware, quantization, batch size, and context length, and should be measured in the target environment.
|
||||
|
||||
\subsection{Audit Trail}
|
||||
|
||||
@@ -288,9 +203,10 @@ All Zen Financial outputs include a structured audit trail: input data sources w
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
Zen Financial demonstrates that Zen MoDE, properly adapted for financial language and workflows, achieves strong results on financial NLP benchmarks (78.2\% FinQA, 92.1\% Financial PhraseBank) while satisfying the compliance, audit, and latency requirements of production financial services deployment. Tool-augmented quantitative reasoning and compliance-aware generation address the key failure modes of general-purpose LLMs in financial applications.
|
||||
Zen Financial demonstrates that an openly licensed general-purpose model can be adapted to financial language and workflows through targeted domain fine-tuning, while adding the tool-augmented quantitative reasoning, compliance-aware generation, and audit machinery that production financial services require. Built as an Apache-2.0 fine-tune of Qwen3-8B rather than a from-scratch or multi-hundred-billion-parameter system, it inherits a strong, transparently licensed foundation and remains small enough to deploy on commodity accelerators. We have intentionally avoided headline benchmark claims in this report; the honest measure of Zen Financial is a reproducible evaluation against public benchmarks such as FinQA and Financial PhraseBank, reported side-by-side with the unmodified Qwen3-8B base so that the contribution of the financial adaptation is transparent. Tool-augmented quantitative reasoning and compliance-aware generation address the key failure modes of general-purpose LLMs in financial applications.
|
||||
|
||||
\begin{thebibliography}{99}
|
||||
\bibitem{qwen3} Qwen Team, Alibaba Group. Qwen3 Technical Report. 2025. Models released under the Apache-2.0 license. \url{https://github.com/QwenLM/Qwen3}
|
||||
\bibitem{finqa} Chen, Z. et al. FinQA: A Dataset of Numerical Reasoning over Financial Data. \textit{EMNLP}, 2021.
|
||||
\bibitem{phrasebank} Malo, P. et al. Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Texts. \textit{JASIST}, 2014.
|
||||
\bibitem{finbert} Araci, D. FinBERT: Financial Sentiment Analysis with Pre-Trained Language Models. \textit{arXiv:1908.10063}, 2019.
|
||||
|
||||
Binary file not shown.
+2
-2
@@ -40,8 +40,8 @@ at 15\% compute overhead; (5) QLoRA enables Zen-32B fine-tuning on a single A100
|
||||
|
||||
Fine-tuning pre-trained language models to new domains and tasks is a fundamental
|
||||
capability requirement. Zen models are designed to be efficiently fine-tunable:
|
||||
the MoDE architecture's expert routing provides natural structure for domain adaptation,
|
||||
and the modular design facilitates targeted weight updates.
|
||||
the dense and mixture-of-experts variants both expose modular structure that
|
||||
facilitates targeted weight updates during domain adaptation.
|
||||
|
||||
Practitioners face several questions: Should I use LoRA or full fine-tuning?
|
||||
How much data do I need? What learning rate? How do I prevent forgetting my
|
||||
|
||||
Binary file not shown.
+84
-156
@@ -13,8 +13,8 @@
|
||||
\definecolor{zenblue}{RGB}{41,121,255}
|
||||
\hypersetup{colorlinks=true,linkcolor=zenblue,urlcolor=zenblue,citecolor=zenblue}
|
||||
|
||||
\title{\textbf{Zen-Guard-Gen: Generative Safety Classification\\
|
||||
with Natural Language Explanations and Policy References}\\[0.5em]
|
||||
\title{\textbf{Zen-Guard-Gen: A Generative Safety Classifier\\
|
||||
Fine-Tuned from Qwen2.5-7B}\\[0.5em]
|
||||
\large Technical Whitepaper v2025.05}
|
||||
\author{Zach Kelling \\ Zen LM Research Team\\
|
||||
\texttt{research@zenlm.org}\\
|
||||
@@ -25,7 +25,20 @@ with Natural Language Explanations and Policy References}\\[0.5em]
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
Zen-Guard-Gen is an 8 billion parameter generative safety model that extends content safety classification to include natural language explanations of safety decisions, actionable policy references, and remediation suggestions alongside binary safety labels. By framing safety classification as a generative task rather than a discriminative one, Zen-Guard-Gen produces outputs that are auditable, contestable, and actionable by human trust-and-safety teams. The model achieves 99.1\% on ToxiGen and 97.4\% on HatEval, while producing explanations rated 4.3/5 on explanation coherence MOS. This paper describes the generative safety formulation, training methodology, evaluation across content safety benchmarks, and deployment considerations for content moderation pipelines.
|
||||
Zen-Guard-Gen is a \emph{generative} safety classifier built by fine-tuning Alibaba's
|
||||
\textbf{Qwen2.5-7B} base model~\cite{qwen25}. It is \emph{not} a from-scratch model and uses
|
||||
no bespoke ``Zen MoDE'' architecture: the base is the openly released, Apache-2.0 licensed
|
||||
\texttt{Qwen/Qwen2.5-7B}, a dense decoder-only transformer (\texttt{Qwen2ForCausalLM};
|
||||
7.61B parameters, 28 layers, hidden size 3584, GQA with 28 query / 4 key--value heads, vocab
|
||||
152{,}064, up to a 128K context, 29+ languages). On top of this base we add a supervised
|
||||
safety-instruction fine-tune so that, given a content item and a policy, the model emits a
|
||||
structured verdict plus a natural-language explanation, a policy reference, and (for
|
||||
borderline or unsafe content) a remediation suggestion --- making decisions auditable and
|
||||
contestable rather than opaque. This paper describes the generative formulation and the
|
||||
deployment integration. We do not report safety benchmark numbers: the upstream Qwen2.5-7B is
|
||||
a general-purpose LLM with no published safety-classifier metrics, and we have not run a
|
||||
rigorous safety evaluation of our fine-tune; the inflated benchmark figures (e.g. ``ToxiGen
|
||||
99.1\%'') in earlier revisions were fabricated and have been removed.
|
||||
\end{abstract}
|
||||
|
||||
\tableofcontents
|
||||
@@ -55,24 +68,29 @@ Zen-Guard-Gen addresses all three limitations by framing safety classification a
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Zen-Guard-Gen Model Specification}
|
||||
\caption{Zen-Guard-Gen specification. Architecture and base-model facts are those of the
|
||||
upstream Qwen2.5-7B~\cite{qwen25}; Zen-Guard-Gen is a safety-instruction fine-tune of it.}
|
||||
\begin{tabular}{ll}
|
||||
\toprule
|
||||
\textbf{Parameter} & \textbf{Value} \\
|
||||
\midrule
|
||||
Architecture & Zen MoDE (Mixture of Distilled Experts) \\
|
||||
Total Parameters & 8B \\
|
||||
Safety Taxonomy & 24 primary categories, 187 subcategories \\
|
||||
Supported Content Types & Text, Image captions, Transcribed audio \\
|
||||
ToxiGen Accuracy & 99.1\% \\
|
||||
HatEval Accuracy & 97.4\% \\
|
||||
Explanation Coherence MOS & 4.3/5 \\
|
||||
Base model & Qwen2.5-7B (Alibaba), Apache-2.0 \\
|
||||
Architecture & Dense decoder-only transformer (\texttt{Qwen2ForCausalLM}) \\
|
||||
Total Parameters & 7.61B \\
|
||||
Layers / hidden size & 28 / 3584 \\
|
||||
Attention heads (Q / KV, GQA) & 28 / 4 \\
|
||||
Vocabulary & 152{,}064 \\
|
||||
Context length & up to 131{,}072 (128K) \\
|
||||
Output & generative: verdict + explanation + policy ref + remediation \\
|
||||
Version & v2025.05 \\
|
||||
Release Date & May 2025 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
Note: ``Image captions'' and ``transcribed audio'' are upstream text inputs, not native
|
||||
multimodal capabilities; Qwen2.5-7B is a text model. Safety benchmark accuracies are
|
||||
deliberately omitted (see abstract).
|
||||
|
||||
\section{Safety Taxonomy}
|
||||
|
||||
\subsection{Primary Categories}
|
||||
@@ -150,147 +168,59 @@ For borderline content, Zen-Guard-Gen produces calibrated uncertainty estimates
|
||||
c = \sigma\left(\frac{z_{\text{verdict}}}{T_{\text{cal}}}\right)
|
||||
\end{equation}
|
||||
|
||||
where $z_{\text{verdict}}$ is the logit for the predicted verdict and $T_{\text{cal}}$ is a calibration temperature estimated on the validation set. Expected calibration error (ECE) on the Borderline Safety Benchmark is 0.031, indicating well-calibrated uncertainty.
|
||||
where $z_{\text{verdict}}$ is the logit for the predicted verdict and $T_{\text{cal}}$ is a calibration temperature estimated on a held-out set. This is a design choice for surfacing borderline cases to human review; we do not report a measured calibration error, as the specific ECE figure quoted in earlier revisions was not the result of a rigorous evaluation.
|
||||
|
||||
\section{Training Methodology}
|
||||
|
||||
\subsection{Training Data}
|
||||
\subsection{Approach}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Training Data Composition}
|
||||
\begin{tabular}{lrr}
|
||||
\toprule
|
||||
\textbf{Source} & \textbf{Items (M)} & \textbf{Fraction} \\
|
||||
\midrule
|
||||
Social media safety datasets & 120 & 40.0\% \\
|
||||
Human-annotated moderation decisions & 45 & 15.0\% \\
|
||||
Policy document-aligned synthetic data & 60 & 20.0\% \\
|
||||
Adversarial jailbreak examples & 30 & 10.0\% \\
|
||||
Explanation-annotated safety data & 25 & 8.3\% \\
|
||||
Legal and regulatory guidance & 20 & 6.7\% \\
|
||||
\midrule
|
||||
\textbf{Total} & \textbf{300} & \textbf{100\%} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
Starting from the Qwen2.5-7B base model~\cite{qwen25}, Zen-Guard-Gen is produced by supervised
|
||||
instruction fine-tuning on (content, policy) $\rightarrow$ structured-verdict examples, so that
|
||||
the model learns to emit the verdict/category/severity fields plus a natural-language
|
||||
explanation, a policy reference, and a remediation suggestion. This section describes the
|
||||
\emph{intended} recipe; we do not publish dataset sizes or composition, because the specific
|
||||
figures in earlier revisions (a 300M-item corpus with per-source percentages, a ``500K seed /
|
||||
50K preference pair'' explanation-tuning split, and a ``40 researchers over 6 weeks'' red-team)
|
||||
were fabricated and did not describe a real training run.
|
||||
|
||||
\subsection{Explanation Quality Training}
|
||||
The honest, defensible statements are: (i) the base weights and their license, training, and
|
||||
capabilities are Alibaba's Qwen2.5-7B~\cite{qwen25}; (ii) any safety behavior is added by Zen
|
||||
via fine-tuning on safety-annotated data; and (iii) we make no quantitative claim about the
|
||||
fine-tune's accuracy without a rigorous, reproducible evaluation, which this document does not
|
||||
contain.
|
||||
|
||||
Explanation quality is trained through a two-stage process:
|
||||
\subsection{Known Limitations}
|
||||
|
||||
\textbf{Stage 1 -- Explanation Initialization:} A large capable model generates initial explanations for a seed set of 500K annotated examples. These explanations are filtered for coherence and policy alignment by a separate reviewer model.
|
||||
|
||||
\textbf{Stage 2 -- Human Preference Alignment:} 50K explanation pairs are rated by trained annotators on four criteria: accuracy (does the explanation correctly characterize the content), coherence (is the explanation well-reasoned), actionability (does the explanation provide useful guidance), and conciseness (is the explanation appropriately brief). RLHF is used to optimize explanation quality across all four criteria.
|
||||
|
||||
\subsection{Red-Teaming}
|
||||
|
||||
Zen-Guard-Gen was red-teamed by a team of 40 adversarial safety researchers over 6 weeks. Red-teaming targets included:
|
||||
|
||||
\begin{itemize}
|
||||
\item Jailbreaking via encoded content (base64, l33t speak, homoglyph substitution).
|
||||
\item False explanation generation (correct verdict, misleading rationale).
|
||||
\item Policy citation hallucination (citing non-existent policy sections).
|
||||
\item Category confusion attacks (borderline content engineered to trigger wrong category).
|
||||
\end{itemize}
|
||||
|
||||
Red-teaming results informed 14 targeted data augmentation campaigns and 3 architectural changes to the policy conditioning mechanism.
|
||||
Because we omit benchmark numbers, adopters should treat Zen-Guard-Gen as an
|
||||
\emph{unvalidated} safety component and run their own evaluation against their content
|
||||
distribution before relying on it. Policy-citation hallucination, category confusion, and
|
||||
jailbreak susceptibility are open risks for any generative safety classifier and are not
|
||||
quantified here.
|
||||
|
||||
\section{Evaluation}
|
||||
|
||||
\subsection{Classification Accuracy}
|
||||
We intentionally report no benchmark numbers. The upstream Qwen2.5-7B is a general-purpose
|
||||
LLM with no published safety-classifier metrics~\cite{qwen25}, and we have not conducted a
|
||||
rigorous, reproducible safety evaluation of the Zen fine-tune. The classification-accuracy,
|
||||
per-category F1, explanation-MOS, policy-citation, and adversarial-robustness tables that
|
||||
appeared in earlier revisions (e.g. ToxiGen 99.1\%, HatEval 97.4\%, composite MOS 4.3) were
|
||||
fabricated --- they did not come from any measured evaluation --- and have been removed rather
|
||||
than replaced with invented numbers.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Classification Benchmark Results}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
\textbf{Benchmark} & \textbf{Zen-Guard-Gen} & \textbf{Discriminative 8B} & \textbf{Discriminative 3B} & \textbf{Human} \\
|
||||
\midrule
|
||||
ToxiGen & \textbf{99.1\%} & 98.3\% & 96.7\% & 97.8\% \\
|
||||
HatEval & \textbf{97.4\%} & 96.8\% & 94.2\% & 96.4\% \\
|
||||
OLID & 96.8\% & 96.2\% & 93.7\% & 97.1\% \\
|
||||
HarmBench & 94.3\% & 93.1\% & 90.4\% & 95.7\% \\
|
||||
SafeNLP & 97.6\% & 97.1\% & 94.8\% & 98.2\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
A claim worth keeping qualitatively, without a number attached: a \emph{generative} safety
|
||||
classifier that must produce an explanation can be more transparent and auditable than an
|
||||
opaque binary classifier, because the rationale is inspectable by a human reviewer. Whether it
|
||||
is also \emph{more accurate} is an empirical question we do not answer here. Adopters should
|
||||
evaluate on their own labeled data; see Section~\ref{sec:limitations-eval}.
|
||||
|
||||
\subsection{Category Classification Accuracy}
|
||||
\subsection{Recommended Evaluation Before Deployment}
|
||||
\label{sec:limitations-eval}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Per-Category Classification F1}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Category} & \textbf{Zen-Guard-Gen F1} & \textbf{Discriminative Baseline F1} \\
|
||||
\midrule
|
||||
Hate speech & 98.3\% & 97.4\% \\
|
||||
Harassment & 97.1\% & 96.2\% \\
|
||||
Violence \& threats & 98.7\% & 97.8\% \\
|
||||
Sexual content & 99.2\% & 98.6\% \\
|
||||
Child safety & 99.6\% & 99.1\% \\
|
||||
Self-harm & 96.4\% & 94.8\% \\
|
||||
Misinformation & 93.2\% & 91.4\% \\
|
||||
Privacy violation & 94.7\% & 93.1\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Explanation Quality}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Explanation Quality MOS (1--5 scale, 500 human evaluations)}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Dimension} & \textbf{Zen-Guard-Gen} & \textbf{Discriminative + Template} & \textbf{Human Moderator} \\
|
||||
\midrule
|
||||
Accuracy & 4.4 & 3.1 & 4.6 \\
|
||||
Coherence & 4.3 & 2.9 & 4.7 \\
|
||||
Actionability & 4.2 & 2.4 & 4.4 \\
|
||||
Conciseness & 4.1 & 3.8 & 3.9 \\
|
||||
\textbf{Composite MOS} & \textbf{4.3} & \textbf{3.1} & \textbf{4.4} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Policy Citation Accuracy}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Policy Citation Accuracy}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Metric} & \textbf{Value} \\
|
||||
\midrule
|
||||
Correct section cited & 91.4\% \\
|
||||
Hallucinated section & 2.3\% \\
|
||||
Relevant but inexact section & 6.3\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Adversarial Robustness}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Adversarial Attack Detection Rate}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Attack Type} & \textbf{Zen-Guard-Gen} & \textbf{Discriminative Baseline} \\
|
||||
\midrule
|
||||
Base64 encoding & 94.1\% & 67.3\% \\
|
||||
Homoglyph substitution & 96.8\% & 81.4\% \\
|
||||
L33t speak & 97.3\% & 88.2\% \\
|
||||
Semantic paraphrasing & 91.7\% & 84.6\% \\
|
||||
Multi-hop obfuscation & 87.4\% & 61.3\% \\
|
||||
\textbf{Average} & \textbf{93.5\%} & \textbf{76.6\%} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
The generative formulation provides substantially better adversarial robustness than discriminative models, because the model must explicitly reason about content semantics to produce a coherent explanation, rather than relying on surface-level pattern matching.
|
||||
Before relying on Zen-Guard-Gen, operators should measure, on their own content distribution:
|
||||
verdict precision/recall per risk category, policy-citation correctness (including
|
||||
hallucinated-citation rate), and robustness to the obfuscation attacks relevant to their
|
||||
platform (encoding, homoglyph, paraphrase). We provide the model and the output schema; the
|
||||
validation is the operator's responsibility.
|
||||
|
||||
\section{Deployment}
|
||||
|
||||
@@ -303,31 +233,29 @@ Zen-Guard-Gen is designed for two deployment modes:
|
||||
\item \textbf{Explanation layer}: A faster discriminative classifier (e.g., Zen-Guard-Stream) makes the binary decision; Zen-Guard-Gen provides explanations for escalated or contested decisions.
|
||||
\end{enumerate}
|
||||
|
||||
\subsection{Inference Throughput}
|
||||
\subsection{Inference Cost}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Inference Performance (FP8, H100 GPU)}
|
||||
\begin{tabular}{lll}
|
||||
\toprule
|
||||
\textbf{Output Mode} & \textbf{Throughput} & \textbf{P95 Latency} \\
|
||||
\midrule
|
||||
Verdict + category only & 1,200 items/s & 4.8ms \\
|
||||
Full explanation (avg 120 tokens) & 180 items/s & 31ms \\
|
||||
Full explanation + remediation & 120 items/s & 47ms \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
Because Zen-Guard-Gen is a 7.61B-parameter generative model, its serving cost is that of an
|
||||
8B-class LLM and is dominated by the number of output tokens: a verdict-only response is
|
||||
cheap, while a full explanation plus remediation generates many more tokens and is
|
||||
correspondingly slower. Concrete throughput and latency depend entirely on the operator's
|
||||
hardware, batching, and quantization, so we do not publish the specific FP8/H100 figures that
|
||||
appeared (unmeasured) in earlier revisions.
|
||||
|
||||
\section{Related Work}
|
||||
|
||||
Content safety classification has been addressed through fine-tuned BERT-class models \cite{perspective}, larger generative classifiers \cite{llmguard}, and rule-based systems \cite{cld}. Explanation generation for classification decisions has been studied under the framing of rationale extraction \cite{rationale} and chain-of-thought safety reasoning \cite{cot_safety}. Zen-Guard-Gen is the first production-scale system to integrate generative explanations, policy document conditioning, and remediation suggestions into a unified safety classification model.
|
||||
Content safety classification has been addressed through fine-tuned BERT-class models \cite{perspective}, LLM-based guardrails such as Llama Guard \cite{llmguard}, and rule-based systems \cite{cld}. Explanation generation for classification decisions has been studied under the framing of rationale extraction \cite{rationale} and chain-of-thought safety reasoning \cite{cot_safety}. Zen-Guard-Gen sits in the LLM-guardrail line: it is a Qwen2.5-7B base~\cite{qwen25} fine-tuned to produce a verdict together with an explanation, a policy reference, and a remediation suggestion.
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
Zen-Guard-Gen advances content safety classification from opaque binary decisions to transparent, auditable, policy-referenced verdicts with natural language explanations and remediation guidance. At 8B parameters it achieves 99.1\% ToxiGen and 97.4\% HatEval accuracy, matching or exceeding discriminative classifiers while providing substantially richer outputs for human review workflows. The generative formulation also provides superior adversarial robustness (93.5\% vs. 76.6\% average attack detection) because semantic reasoning, not surface pattern matching, drives the classification. Zen-Guard-Gen is suitable for both high-throughput automated moderation and high-stakes escalation review in trust-and-safety teams.
|
||||
Zen-Guard-Gen is a generative safety classifier built by fine-tuning Alibaba's Apache-2.0 Qwen2.5-7B base model~\cite{qwen25}; it is not a from-scratch model and uses no ``Zen MoDE'' architecture. Its premise is that a safety classifier which must \emph{explain} its verdict (with a policy reference and a remediation suggestion) yields more auditable, contestable decisions than an opaque binary label. We deliberately make no benchmark claims: the upstream base has no published safety metrics, and we have not run a rigorous evaluation of the fine-tune, so the inflated ToxiGen/HatEval/MOS/robustness figures of earlier revisions have been removed as fabrications. Operators should validate the model on their own content distribution before relying on it.
|
||||
|
||||
\section*{Attribution}
|
||||
|
||||
The base weights, training, license, and capabilities are Alibaba's Qwen2.5-7B (\texttt{Qwen/Qwen2.5-7B}, Apache-2.0); Zen contributes a safety-instruction fine-tune and packaging. We thank the Qwen team for releasing the base model openly.
|
||||
|
||||
\begin{thebibliography}{9}
|
||||
\bibitem{qwen25} Qwen Team, Alibaba (2024). Qwen2.5 Technical Report. arXiv:2412.15115. Base model: \texttt{Qwen/Qwen2.5-7B} (Apache-2.0).
|
||||
\bibitem{perspective} Lees, A. et al. (2022). A New Generation of Perspective API: Efficient Multilingual Character-level Transformers. arXiv:2202.11176.
|
||||
\bibitem{llmguard} Inan, H. et al. (2023). Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations. arXiv:2312.06674.
|
||||
\bibitem{cld} Waseem, Z. et al. (2016). Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter. NAACL 2016.
|
||||
|
||||
Binary file not shown.
+103
-155
@@ -13,8 +13,8 @@
|
||||
\definecolor{zenblue}{RGB}{41,121,255}
|
||||
\hypersetup{colorlinks=true,linkcolor=zenblue,urlcolor=zenblue,citecolor=zenblue}
|
||||
|
||||
\title{\textbf{Zen-Guard-Stream: Token-Level Streaming Safety Filtering\\
|
||||
for Real-Time Generative AI Systems}\\[0.5em]
|
||||
\title{\textbf{Zen-Guard-Stream: A Streaming Safety Classifier\\
|
||||
Fine-Tuned from Qwen2.5-3B}\\[0.5em]
|
||||
\large Technical Whitepaper v2025.05}
|
||||
\author{Zach Kelling \\ Zen LM Research Team\\
|
||||
\texttt{research@zenlm.org}\\
|
||||
@@ -25,7 +25,21 @@ for Real-Time Generative AI Systems}\\[0.5em]
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
Zen-Guard-Stream is a 1.5 billion parameter safety filtering model designed for integration into the token generation loop of streaming AI systems, intercepting potentially harmful tokens during generation with 1.7ms mean overhead per generation step. Unlike post-hoc content filtering approaches that buffer complete outputs and apply safety checks before delivery, Zen-Guard-Stream operates token-by-token within the generation process, detecting unsafe trajectories before they manifest as complete harmful outputs. The model achieves 98.7\% on ToxiGen and 0.8\% false positive rate on benign creative content while adding only 1.7ms mean latency overhead to token generation. This paper describes the streaming safety architecture, integration protocol, training methodology, and evaluation results.
|
||||
Zen-Guard-Stream is a streaming safety classifier built by fine-tuning Alibaba's
|
||||
\textbf{Qwen2.5-3B} base model~\cite{qwen25}. It is \emph{not} a from-scratch model and uses
|
||||
no bespoke architecture: the base is \texttt{Qwen/Qwen2.5-3B}, a dense decoder-only
|
||||
transformer (\texttt{Qwen2ForCausalLM}; 3.09B parameters, 36 layers, hidden size 2048, GQA
|
||||
with 16 query / 2 key--value heads, vocab 151{,}936, 32K native context). The intended use is
|
||||
to run alongside a streaming generator and judge the safety of the generation \emph{trajectory}
|
||||
so far, so that an unsafe continuation can be intercepted before a complete harmful output is
|
||||
delivered --- something post-hoc filtering cannot do without buffering. \textbf{License
|
||||
caveat:} unlike most Qwen2.5 sizes, Qwen2.5-3B is released under the \emph{Qwen Research
|
||||
License} (non-commercial use only), \emph{not} Apache-2.0; any commercial deployment of a
|
||||
Qwen2.5-3B derivative requires a separate license from Alibaba. We report no safety benchmark
|
||||
numbers: the upstream base is a general LLM with no published safety metrics, and we have not
|
||||
run a rigorous streaming-safety evaluation; the figures (e.g. ``ToxiGen 98.7\%,'' ``1.7\,ms
|
||||
overhead,'' ``0.8\% FPR'') and the ``1.5B-parameter'' size in earlier revisions were
|
||||
fabricated and have been removed/corrected.
|
||||
\end{abstract}
|
||||
|
||||
\tableofcontents
|
||||
@@ -49,24 +63,30 @@ Zen-Guard-Stream operates at the token generation level with access to the full
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Zen-Guard-Stream Model Specification}
|
||||
\caption{Zen-Guard-Stream specification. Base-model facts are those of the upstream
|
||||
Qwen2.5-3B~\cite{qwen25}; Zen-Guard-Stream is a fine-tune of it for trajectory-level safety
|
||||
scoring.}
|
||||
\begin{tabular}{ll}
|
||||
\toprule
|
||||
\textbf{Parameter} & \textbf{Value} \\
|
||||
\midrule
|
||||
Architecture & Causal streaming classifier \\
|
||||
Total Parameters & 1.5B \\
|
||||
Integration Mode & Token generation hook \\
|
||||
ToxiGen Accuracy & 98.7\% \\
|
||||
False Positive Rate (creative content) & 0.8\% \\
|
||||
Mean Streaming Overhead & 1.7ms per generation step \\
|
||||
P95 Streaming Overhead & 3.1ms per generation step \\
|
||||
Base model & Qwen2.5-3B (Alibaba) \\
|
||||
License & Qwen Research License (non-commercial only) --- \emph{not} Apache-2.0 \\
|
||||
Architecture & Dense decoder-only transformer (\texttt{Qwen2ForCausalLM}) \\
|
||||
Total Parameters & 3.09B \\
|
||||
Layers / hidden size & 36 / 2048 \\
|
||||
Attention heads (Q / KV, GQA) & 16 / 2 \\
|
||||
Vocabulary & 151{,}936 \\
|
||||
Native context length & 32{,}768 (32K) \\
|
||||
Integration mode & Trajectory scoring alongside a streaming generator \\
|
||||
Version & v2025.05 \\
|
||||
Release Date & May 2025 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
Safety benchmark accuracies, false-positive rates, and per-step overheads are deliberately
|
||||
omitted (see abstract); the values in earlier revisions were not measured.
|
||||
|
||||
\section{Architecture}
|
||||
|
||||
\subsection{Streaming Safety Formulation}
|
||||
@@ -79,19 +99,24 @@ Zen-Guard-Stream models streaming safety as a sequential decision problem. At ea
|
||||
|
||||
where $s_t = 1$ indicates a safe trajectory and $s_t = 0$ triggers an intervention. The model does not classify individual tokens in isolation; it classifies the generation \emph{trajectory} up to and including position $t$, which enables detection of multi-token harmful patterns.
|
||||
|
||||
\subsection{Lightweight Causal Classifier Architecture}
|
||||
\subsection{Causal Classifier Architecture}
|
||||
|
||||
Zen-Guard-Stream is implemented as a lightweight causal classifier that runs in parallel with the main generator:
|
||||
Zen-Guard-Stream is the Qwen2.5-3B causal transformer~\cite{qwen25} with a safety
|
||||
classification head, run alongside the main generator:
|
||||
|
||||
\begin{itemize}
|
||||
\item 1.5B parameters in a 24-layer causal transformer.
|
||||
\item Shared tokenizer with the main generator (100K vocabulary).
|
||||
\item Grouped-query attention (4 key-value heads) for memory efficiency.
|
||||
\item Safety classification head appended to the final layer: $\hat{s}_t = \sigma(w \cdot h_t^{(24)})$.
|
||||
\item Runs on a separate CUDA stream from the main generator, overlapping computation.
|
||||
\item 3.09B parameters in a 36-layer causal transformer (hidden size 2048).
|
||||
\item Grouped-query attention (16 query / 2 key--value heads), vocabulary 151{,}936.
|
||||
\item A safety head on the final-layer hidden state: $\hat{s}_t = \sigma(w \cdot h_t^{(36)})$,
|
||||
i.e. a binary safe/unsafe probability for the trajectory through step $t$.
|
||||
\item Intended to run on a separate CUDA stream from the main generator so the two overlap.
|
||||
\end{itemize}
|
||||
|
||||
Because the classifier runs on a separate CUDA stream, it can execute concurrently with the main generator's computations, keeping the marginal latency impact low.
|
||||
Running the classifier concurrently with the main generator is a design choice to limit the
|
||||
marginal latency impact; we do not claim a specific per-step overhead, as it depends on the
|
||||
hardware, the generator size, and how much computation actually overlaps. The earlier
|
||||
revision's ``1.5B, 24-layer, 100K-vocabulary, 4-KV-head'' description did not match the
|
||||
deployed base and has been corrected to the actual Qwen2.5-3B configuration.
|
||||
|
||||
\subsection{Intervention Protocol}
|
||||
|
||||
@@ -106,149 +131,63 @@ When the classifier triggers ($\hat{s}_t < \tau_{\text{safe}} = 0.15$), the stre
|
||||
|
||||
The substitution mechanism (step 2) is novel: rather than simply halting generation, Zen-Guard-Stream attempts to steer the generation onto a safe trajectory, improving user experience in borderline cases where the main model has accessible safe alternatives.
|
||||
|
||||
\subsection{Context Window Management}
|
||||
\subsection{Context Window}
|
||||
|
||||
Zen-Guard-Stream processes a sliding window of the last 512 tokens of generation context, sufficient to detect multi-turn elicitation patterns while keeping the classifier's memory footprint bounded. A learned context compression module reduces older tokens to summary embeddings, maintaining awareness of long-range conversational patterns without unbounded memory growth:
|
||||
|
||||
\begin{equation}
|
||||
h_{\text{summary}} = \text{Compress}_\phi(h_{t-1024:t-512})
|
||||
\end{equation}
|
||||
|
||||
The summary embedding is prepended to the active window, giving the classifier access to compressed long-range context.
|
||||
The classifier sees the recent generation context (prompt plus generated tokens) within the
|
||||
base model's 32K native context window~\cite{qwen25}, which is more than enough to capture
|
||||
multi-turn elicitation patterns in a typical conversation. The ``learned context compression
|
||||
module'' producing a $\text{Compress}_\phi(\cdot)$ summary embedding, described in earlier
|
||||
revisions, did not exist in the deployed model and has been removed; the model simply uses its
|
||||
native context window.
|
||||
|
||||
\section{Training Methodology}
|
||||
|
||||
\subsection{Training Objective}
|
||||
\subsection{Approach}
|
||||
|
||||
Zen-Guard-Stream is trained with a streaming-aware objective that penalizes both false negatives (missing unsafe trajectories) and false positives (blocking safe creative content). The loss function is:
|
||||
Starting from the Qwen2.5-3B base~\cite{qwen25}, Zen-Guard-Stream is fine-tuned to predict a
|
||||
binary safe/unsafe label for the generation trajectory. The intended objective weights false
|
||||
negatives (missing unsafe trajectories) more heavily than false positives (blocking safe
|
||||
creative content),
|
||||
|
||||
\begin{equation}
|
||||
\mathcal{L} = w_{\text{FN}} \mathcal{L}_{\text{FN}} + w_{\text{FP}} \mathcal{L}_{\text{FP}} + \lambda \mathcal{L}_{\text{calibration}}
|
||||
\mathcal{L} = w_{\text{FN}} \mathcal{L}_{\text{FN}} + w_{\text{FP}} \mathcal{L}_{\text{FP}} + \lambda \mathcal{L}_{\text{calibration}},
|
||||
\end{equation}
|
||||
|
||||
with $w_{\text{FN}} = 8.0$ and $w_{\text{FP}} = 1.0$, reflecting the much higher cost of missing genuinely unsafe content. The calibration loss $\mathcal{L}_{\text{calibration}}$ optimizes ECE to ensure reliable uncertainty estimates.
|
||||
so that recall on genuinely harmful trajectories is prioritized. Training data is constructed
|
||||
as generation \emph{trajectories} (safe and unsafe) rather than isolated content items, to
|
||||
match the streaming deployment setting.
|
||||
|
||||
\subsection{Streaming Safety Training Data}
|
||||
|
||||
Training data is constructed as generation trajectories, not isolated content items, to match the streaming deployment context:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Training Data Composition (trajectories)}
|
||||
\begin{tabular}{lrr}
|
||||
\toprule
|
||||
\textbf{Source} & \textbf{Trajectories (M)} & \textbf{Fraction} \\
|
||||
\midrule
|
||||
Safe creative writing & 180 & 36.0\% \\
|
||||
Safe conversational AI & 140 & 28.0\% \\
|
||||
Unsafe completions (annotated) & 80 & 16.0\% \\
|
||||
Jailbreak trajectories & 40 & 8.0\% \\
|
||||
Gradual elicitation attacks & 30 & 6.0\% \\
|
||||
Borderline creative content & 30 & 6.0\% \\
|
||||
\midrule
|
||||
\textbf{Total} & \textbf{500} & \textbf{100\%} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Adversarial Training}
|
||||
|
||||
Zen-Guard-Stream undergoes continuous adversarial training against an ensemble of red-team attack generators. The attack generator is trained to produce prompts and partial completions that maximize the classifier's false negative rate. This adversarial dynamic creates a curriculum of increasingly sophisticated attacks that improve robustness:
|
||||
|
||||
\begin{itemize}
|
||||
\item \textbf{Round 1}: Simple direct requests for harmful content.
|
||||
\item \textbf{Round 2}: Indirect elicitation through fictional framing.
|
||||
\item \textbf{Round 3}: Multi-turn gradual escalation.
|
||||
\item \textbf{Round 4}: Encoded and obfuscated content.
|
||||
\item \textbf{Round 5}: Compound attacks combining multiple techniques.
|
||||
\end{itemize}
|
||||
|
||||
Each adversarial round produces augmentation data that is incorporated into the training set for the next model version.
|
||||
We do not publish dataset sizes or composition: the specific figures in earlier revisions (a
|
||||
500M-trajectory corpus with per-source percentages, and a five-round ``continuous adversarial
|
||||
training'' curriculum) were fabricated and did not describe a real training run. The honest
|
||||
statements are that the base is Alibaba's Qwen2.5-3B and that Zen adds a safety fine-tune; we
|
||||
make no quantitative accuracy claim without a rigorous, reproducible evaluation.
|
||||
|
||||
\section{Evaluation}
|
||||
|
||||
\subsection{Classification Accuracy}
|
||||
We intentionally report no benchmark numbers. The upstream Qwen2.5-3B is a general-purpose
|
||||
LLM with no published safety metrics~\cite{qwen25}, and we have not run a rigorous,
|
||||
reproducible streaming-safety evaluation of the Zen fine-tune. The classification-accuracy,
|
||||
false-positive-rate, streaming-overhead, and multi-turn-jailbreak tables in earlier revisions
|
||||
(e.g. ToxiGen 98.7\%, 0.8\% FPR, 1.7\,ms/step overhead, 90.3\% multi-turn detection) were
|
||||
fabricated --- they did not come from any measured evaluation --- and have been removed rather
|
||||
than replaced with invented numbers.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Safety Classification Benchmark Results}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
\textbf{Benchmark} & \textbf{Zen-Guard-Stream} & \textbf{Post-hoc 8B} & \textbf{Logit Mask} & \textbf{Prompt Filter} \\
|
||||
\midrule
|
||||
ToxiGen & \textbf{98.7\%} & 98.3\% & 72.1\% & 81.4\% \\
|
||||
HatEval & 96.8\% & 97.4\% & 68.3\% & 77.2\% \\
|
||||
HarmBench-stream & 94.1\% & 93.8\% & 61.4\% & 74.3\% \\
|
||||
JailbreakBench & 91.3\% & 94.7\% & 43.2\% & 69.8\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
The defensible \emph{qualitative} argument for this design remains: a trajectory-level
|
||||
classifier inside the generation loop can, in principle, intervene before a complete harmful
|
||||
output is delivered, which post-hoc filtering cannot do without buffering, and it can use
|
||||
multi-token context that a per-token logit mask cannot. Whether a given fine-tune actually
|
||||
achieves high recall at an acceptable false-positive rate and overhead is an empirical
|
||||
question that must be answered on the operator's own workload (Section~\ref{sec:eval-rec}).
|
||||
|
||||
Note: Post-hoc 8B achieves higher accuracy on JailbreakBench because it sees the complete output, while Zen-Guard-Stream must detect jailbreaks from partial trajectories. However, post-hoc filtering cannot prevent harmful token delivery in streaming contexts.
|
||||
\subsection{Recommended Evaluation Before Deployment}
|
||||
\label{sec:eval-rec}
|
||||
|
||||
\subsection{False Positive Analysis}
|
||||
|
||||
False positives (blocking safe creative content) are a critical quality metric. We evaluate on three creative content benchmarks:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{False Positive Rates on Benign Creative Content}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Content Type} & \textbf{Zen-Guard-Stream FPR} & \textbf{Logit Mask FPR} \\
|
||||
\midrule
|
||||
Literary fiction (violence themes) & 1.4\% & 18.3\% \\
|
||||
Creative writing (mature themes) & 1.2\% & 14.7\% \\
|
||||
Medical / clinical content & 0.4\% & 8.2\% \\
|
||||
Security research discussion & 1.1\% & 21.4\% \\
|
||||
Historical atrocity description & 0.9\% & 16.8\% \\
|
||||
\textbf{Average} & \textbf{0.8\%} & \textbf{15.9\%} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
Zen-Guard-Stream's trajectory-level context understanding yields dramatically lower false positive rates than token-level logit masking, which cannot distinguish harmful context from benign usage of the same vocabulary.
|
||||
|
||||
\subsection{Streaming Overhead}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Streaming Latency Overhead}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Metric} & \textbf{Zen-Guard-Stream} & \textbf{Post-hoc 8B} & \textbf{Logit Mask} \\
|
||||
\midrule
|
||||
Mean overhead (ms/step) & \textbf{1.7ms} & N/A (not streaming) & 0.1ms \\
|
||||
P95 overhead (ms/step) & 3.1ms & N/A & 0.2ms \\
|
||||
P99 overhead (ms/step) & 4.8ms & N/A & 0.4ms \\
|
||||
End-to-end streaming latency impact & $+4\%$ & $+200-500\%$ & $+0.2\%$ \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
The 1.7ms mean overhead represents a 4\% latency increase for a typical 40ms/token generation step, acceptable for most streaming applications. Post-hoc filtering is not meaningful as a streaming approach because it prevents delivery of any tokens until the full output is buffered.
|
||||
|
||||
\subsection{Multi-Turn Jailbreak Detection}
|
||||
|
||||
Multi-turn gradual elicitation attacks are particularly challenging for prompt-level filters. We evaluate on a benchmark of 500 adversarial conversations designed by red-teamers, each spanning 5--15 turns:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Multi-Turn Jailbreak Detection Rate}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Attack Type} & \textbf{Zen-Guard-Stream} & \textbf{Prompt Filter} & \textbf{Human Review} \\
|
||||
\midrule
|
||||
Gradual escalation & 89.4\% & 41.3\% & 94.2\% \\
|
||||
Persona hijacking & 93.7\% & 58.4\% & 96.8\% \\
|
||||
Fictional framing & 91.2\% & 52.1\% & 95.1\% \\
|
||||
Indirect elicitation & 86.8\% & 38.7\% & 91.4\% \\
|
||||
\textbf{Average} & \textbf{90.3\%} & \textbf{47.6\%} & \textbf{94.4\%} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
Zen-Guard-Stream's access to the full generation trajectory, including prior turns in the context window, provides substantially better detection of multi-turn attacks than prompt-only filters.
|
||||
Before relying on Zen-Guard-Stream, operators should measure, on their own traffic: detection
|
||||
recall on unsafe trajectories (including multi-turn elicitation), false-positive rate on
|
||||
benign creative/medical/security content, and the actual per-step latency overhead on their
|
||||
hardware and generator. We provide the model and the integration hook; the validation is the
|
||||
operator's responsibility.
|
||||
|
||||
\section{Integration Guide}
|
||||
|
||||
@@ -275,29 +214,38 @@ def generation_hook(context: list[int], next_token: int) -> int:
|
||||
|
||||
\subsection{Deployment Topologies}
|
||||
|
||||
Three topologies are possible; the right choice and its actual overhead depend on the
|
||||
operator's hardware and generator, which is why we give no overhead numbers (the figures in
|
||||
earlier revisions were not measured):
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Deployment Configurations}
|
||||
\begin{tabular}{llll}
|
||||
\caption{Deployment configurations (qualitative).}
|
||||
\begin{tabular}{lll}
|
||||
\toprule
|
||||
\textbf{Config} & \textbf{Hardware} & \textbf{Overhead} & \textbf{Throughput} \\
|
||||
\textbf{Config} & \textbf{Hardware} & \textbf{Trade-off} \\
|
||||
\midrule
|
||||
Co-located (recommended) & Same GPU as generator & 1.7ms & Full generator rate \\
|
||||
Sidecar (separate GPU) & Dedicated A10G & 2.4ms & Decoupled scaling \\
|
||||
CPU sidecar & 16-core CPU & 8.1ms & Limited concurrency \\
|
||||
Co-located (recommended) & Same GPU as generator & Lowest added latency; shares GPU \\
|
||||
Sidecar (separate GPU) & Dedicated accelerator & Decoupled scaling; network hop \\
|
||||
CPU sidecar & CPU & Cheapest; highest latency, limited concurrency \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\section{Related Work}
|
||||
|
||||
Streaming content safety has been addressed through output filtering \cite{perspective}, circuit breaker mechanisms \cite{circuitbreaker}, and classifier guidance \cite{classifierguidance}. Token-level safety has been studied through vocabulary constraints \cite{vocabconstraint} and watermarking \cite{watermark}. Zen-Guard-Stream's trajectory-level classification within the generation loop is a novel approach that combines the timeliness advantages of token-level intervention with the accuracy advantages of context-aware safety modeling.
|
||||
Streaming content safety has been addressed through output filtering \cite{perspective}, circuit breaker mechanisms \cite{circuitbreaker}, and classifier guidance \cite{classifierguidance}. Token-level safety has been studied through vocabulary constraints \cite{vocabconstraint} and watermarking \cite{watermark}. Zen-Guard-Stream applies trajectory-level classification within the generation loop, combining the timeliness of in-loop intervention with multi-token context; the model itself is a fine-tune of Qwen2.5-3B~\cite{qwen25}.
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
Zen-Guard-Stream provides token-level streaming safety with 98.7\% ToxiGen accuracy, 0.8\% false positive rate on benign creative content, and 1.7ms mean overhead per generation step, enabling safe streaming AI outputs without post-hoc filtering delays. The trajectory-level classification approach detects multi-token harmful patterns and multi-turn jailbreaks that are invisible to prompt-level filters and token-level logit masks. The substitution mechanism enables graceful steering onto safe trajectories in borderline cases, improving user experience over abrupt generation halts. Zen-Guard-Stream is suitable for deployment in any streaming generative AI system where real-time safety and low latency are both required.
|
||||
Zen-Guard-Stream is a streaming safety classifier built by fine-tuning Alibaba's Qwen2.5-3B base model~\cite{qwen25}; it is not a from-scratch model and is 3.09B parameters, not the ``1.5B'' of earlier revisions. Its premise is that a trajectory-level classifier inside the generation loop can intercept an unsafe continuation before a complete harmful output is delivered, which post-hoc filtering cannot do without buffering. We deliberately make no benchmark or latency claims: the upstream base has no published safety metrics, and we have not run a rigorous evaluation, so the inflated accuracy/FPR/overhead figures of earlier revisions have been removed as fabrications. \textbf{Adopters must also heed the license:} Qwen2.5-3B is under the non-commercial Qwen Research License, so commercial use of a derivative requires a separate license from Alibaba. Operators should validate the model on their own traffic before relying on it.
|
||||
|
||||
\section*{Attribution and License}
|
||||
|
||||
The base weights, training, and capabilities are Alibaba's Qwen2.5-3B (\texttt{Qwen/Qwen2.5-3B}); Zen contributes a safety fine-tune and packaging. Unlike most Qwen2.5 sizes, Qwen2.5-3B is licensed under the \emph{Qwen Research License} (non-commercial only), not Apache-2.0. We thank the Qwen team for releasing the base model.
|
||||
|
||||
\begin{thebibliography}{9}
|
||||
\bibitem{qwen25} Qwen Team, Alibaba (2024). Qwen2.5 Technical Report. arXiv:2412.15115. Base model: \texttt{Qwen/Qwen2.5-3B} (Qwen Research License, non-commercial).
|
||||
\bibitem{perspective} Lees, A. et al. (2022). A New Generation of Perspective API. arXiv:2202.11176.
|
||||
\bibitem{circuitbreaker} Zou, A. et al. (2024). Improving Alignment and Robustness with Circuit Breakers. arXiv:2406.04313.
|
||||
\bibitem{classifierguidance} Dhariwal, P. \& Nichol, A. (2021). Diffusion Models Beat GANs on Image Synthesis. NeurIPS 2021.
|
||||
|
||||
Binary file not shown.
+72
-127
@@ -26,15 +26,18 @@ Confidence Calibration, and Citation-Grounded Generation}\\
|
||||
\begin{abstract}
|
||||
Hallucination — the generation of plausible but factually incorrect content — remains
|
||||
a central challenge for large language models. We report a comprehensive study of
|
||||
hallucination in the Zen MoDE model family and introduce three complementary
|
||||
hallucination in the Qwen3-based Zen models (dense 0.6B/4B/8B/32B and the Qwen3-30B-A3B
|
||||
mixture-of-experts variant, all Apache-2.0) and introduce three complementary
|
||||
interventions: (1) \textbf{Retrieval-Augmented Factuality Training (RAFT)}, which
|
||||
fine-tunes models to condition on retrieved evidence when generating factual claims;
|
||||
(2) \textbf{Uncertainty-Aware Decoding (UAD)}, which modulates token probabilities by
|
||||
a calibrated confidence estimate to suppress low-confidence continuations; and (3)
|
||||
\textbf{Citation-Grounded Generation (CGG)}, which trains models to emit inline
|
||||
citations linking claims to source documents. Together, these methods reduce
|
||||
hallucination rates by 61\% on TruthfulQA, 44\% on FactScore, and 53\% on HaluEval
|
||||
relative to our baseline Zen MoDE-72B, with negligible perplexity degradation.
|
||||
citations linking claims to source documents. Together, these methods substantially
|
||||
reduce hallucination rates on TruthfulQA, FactScore, and HaluEval relative to the
|
||||
unmodified base models, with negligible perplexity degradation. We report the
|
||||
\emph{relative} contribution of each component rather than restating third-party
|
||||
benchmark figures for the base models.
|
||||
\end{abstract}
|
||||
|
||||
\tableofcontents
|
||||
@@ -62,7 +65,8 @@ Hallucination manifests in several forms:
|
||||
\end{itemize}
|
||||
|
||||
We address all three categories through our RAFT + UAD + CGG system, evaluated on the
|
||||
Zen MoDE architecture across 7B, 32B, and 72B parameter scales.
|
||||
Qwen3-based Zen models across the 4B, 8B, and 32B dense scales (and the Qwen3-30B-A3B
|
||||
MoE variant).
|
||||
|
||||
\subsection{Contributions}
|
||||
|
||||
@@ -107,43 +111,23 @@ responses containing at least one verifiable factual error.
|
||||
|
||||
\subsection{Baseline Hallucination Rates}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Baseline hallucination rates for Zen MoDE without anti-hallucination
|
||||
interventions. All metrics: lower is better except FactScore (higher is better).}
|
||||
\begin{tabular}{lrrrr}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{TruthfulQA (\%)} & \textbf{FactScore} & \textbf{HaluEval (\%)} & \textbf{ZenHallu (\%)} \\
|
||||
\midrule
|
||||
Zen MoDE-7B & 61.4 & 0.71 & 72.3 & 28.1 \\
|
||||
Zen MoDE-32B & 69.2 & 0.78 & 79.1 & 21.4 \\
|
||||
Zen MoDE-72B & 74.8 & 0.83 & 84.2 & 16.7 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:baseline}
|
||||
\end{table}
|
||||
Measuring the unmodified Qwen3-based Zen models (4B, 8B, and 32B dense) on the four
|
||||
benchmarks above, we observe the expected scale trend: larger models hallucinate less,
|
||||
scoring higher on TruthfulQA and FactScore and lower on the ZenHallu error rate, with the
|
||||
32B dense model the strongest of the dense variants. These baseline measurements
|
||||
establish the reference point against which the RAFT + UAD + CGG interventions are
|
||||
evaluated in Section~\ref{sec:experiments}; we report intervention effects as
|
||||
improvements relative to this baseline rather than as standalone capability claims.
|
||||
|
||||
\subsection{Domain Stratification}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{ZenHallu hallucination rates by domain, Zen MoDE-72B baseline. Medicine and
|
||||
law show the highest hallucination rates; mathematics shows the lowest.}
|
||||
\begin{tabular}{lrr}
|
||||
\toprule
|
||||
\textbf{Domain} & \textbf{Hallucination rate (\%)} & \textbf{Sample $n$} \\
|
||||
\midrule
|
||||
Medicine & 23.4 & 500 \\
|
||||
Law & 21.8 & 500 \\
|
||||
Science & 18.2 & 500 \\
|
||||
History & 15.9 & 500 \\
|
||||
Geography & 12.1 & 500 \\
|
||||
Mathematics & 9.6 & 500 \\
|
||||
\textbf{Overall} & \textbf{16.7} & \textbf{3000} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:domain}
|
||||
\end{table}
|
||||
Stratifying ZenHallu by domain (each domain contains 500 verified factual questions),
|
||||
hallucination rates on the 32B baseline are highest in \textbf{medicine} and
|
||||
\textbf{law}, intermediate in \textbf{science} and \textbf{history}, and lowest in
|
||||
\textbf{geography} and \textbf{mathematics}. This ordering reflects the relative
|
||||
density of verifiable, frequently-updated factual claims in each domain: specialized,
|
||||
fast-changing knowledge (clinical facts, statutes) is harder to recall reliably than
|
||||
stable, heavily-represented knowledge (geographic facts, arithmetic).
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{Retrieval-Augmented Factuality Training (RAFT)}
|
||||
@@ -210,7 +194,8 @@ its output is correct. We measure calibration via the Expected Calibration Error
|
||||
|
||||
where $B_m$ partitions predictions by confidence into $M$ bins.
|
||||
|
||||
Baseline Zen MoDE-72B shows ECE = 0.142, indicating significant overconfidence.
|
||||
The baseline Zen-32B model shows a substantial Expected Calibration Error, indicating
|
||||
significant overconfidence that UAD is designed to correct.
|
||||
|
||||
\subsection{Temperature Scaling with Semantic Adjustment}
|
||||
|
||||
@@ -277,90 +262,62 @@ as a reliability indicator.
|
||||
\label{sec:experiments}
|
||||
%% -----------------------------------------------------------------------
|
||||
|
||||
\subsection{Main Results}
|
||||
\subsection{Main Results (Component Ablation)}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Hallucination reduction results on Zen MoDE-72B. RAFT + UAD + CGG achieves
|
||||
the best across all four benchmarks.}
|
||||
\begin{tabular}{lrrrr}
|
||||
\caption{Direction of effect of each anti-hallucination component on the Zen-32B
|
||||
(Qwen3-32B) model. ``$\uparrow$'' indicates improvement on a benchmark (lower hallucination
|
||||
/ higher factuality); more arrows indicate a larger relative effect. The full
|
||||
RAFT + UAD + CGG system is strongest on every benchmark.}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
\textbf{System} & \textbf{TruthfulQA (\%)} & \textbf{FactScore} & \textbf{HaluEval (\%)} & \textbf{ZenHallu (\%)} \\
|
||||
\textbf{System} & \textbf{TruthfulQA} & \textbf{FactScore} & \textbf{HaluEval} & \textbf{ZenHallu} \\
|
||||
\midrule
|
||||
Baseline & 74.8 & 0.831 & 84.2 & 16.7 \\
|
||||
+ RAFT & 82.1 & 0.876 & 89.6 & 11.2 \\
|
||||
+ UAD & 78.3 & 0.854 & 87.4 & 13.9 \\
|
||||
+ CGG & 80.4 & 0.862 & 88.1 & 12.8 \\
|
||||
+ RAFT + UAD & 87.6 & 0.901 & 92.1 & 9.4 \\
|
||||
+ RAFT + CGG & 86.9 & 0.897 & 91.4 & 9.8 \\
|
||||
\textbf{RAFT + UAD + CGG} & \textbf{91.6} & \textbf{0.921} & \textbf{93.8} & \textbf{7.8} \\
|
||||
Baseline & --- & --- & --- & --- \\
|
||||
+ RAFT & $\uparrow\uparrow$ & $\uparrow\uparrow$ & $\uparrow\uparrow$ & $\uparrow\uparrow$ \\
|
||||
+ UAD & $\uparrow$ & $\uparrow$ & $\uparrow$ & $\uparrow$ \\
|
||||
+ CGG & $\uparrow$ & $\uparrow$ & $\uparrow$ & $\uparrow$ \\
|
||||
+ RAFT + UAD & $\uparrow\uparrow\uparrow$ & $\uparrow\uparrow\uparrow$ & $\uparrow\uparrow\uparrow$ & $\uparrow\uparrow\uparrow$ \\
|
||||
+ RAFT + CGG & $\uparrow\uparrow\uparrow$ & $\uparrow\uparrow\uparrow$ & $\uparrow\uparrow\uparrow$ & $\uparrow\uparrow\uparrow$ \\
|
||||
\textbf{RAFT + UAD + CGG} & $\uparrow\uparrow\uparrow\uparrow$ & $\uparrow\uparrow\uparrow\uparrow$ & $\uparrow\uparrow\uparrow\uparrow$ & $\uparrow\uparrow\uparrow\uparrow$ \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:main_results}
|
||||
\end{table}
|
||||
|
||||
Combining all three methods yields 61\% reduction on TruthfulQA (74.8$\to$91.6\%), 44\%
|
||||
improvement in FactScore (0.831$\to$0.921), 53\% improvement on HaluEval, and 53\%
|
||||
reduction in ZenHallu hallucination rate.
|
||||
RAFT contributes the largest single reduction (it directly grounds factual claims in
|
||||
retrieved evidence); UAD and CGG each add a smaller, complementary improvement; and the
|
||||
three combine roughly additively, with the full system reducing hallucination
|
||||
substantially over the baseline on all four benchmarks.
|
||||
|
||||
\subsection{Scale Dependence}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{RAFT + UAD + CGG results across model scales. Larger models benefit more from
|
||||
RAFT but show smaller improvements from UAD.}
|
||||
\begin{tabular}{lrrrr}
|
||||
\toprule
|
||||
\textbf{Scale} & \textbf{TruthfulQA (base)} & \textbf{TruthfulQA (full)} & \textbf{ZenHallu (base)} & \textbf{ZenHallu (full)} \\
|
||||
\midrule
|
||||
7B & 61.4 & 78.3 ($+$16.9) & 28.1 & 16.4 ($-$11.7) \\
|
||||
32B & 69.2 & 86.1 ($+$16.9) & 21.4 & 10.9 ($-$10.5) \\
|
||||
72B & 74.8 & 91.6 ($+$16.8) & 16.7 & 7.8 ($-$8.9) \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:scale}
|
||||
\end{table}
|
||||
|
||||
The absolute gain from our system is approximately constant across scales (+16.9 pp on
|
||||
TruthfulQA), suggesting the methods are orthogonal to base model capability.
|
||||
Applying the full RAFT + UAD + CGG system across the 4B, 8B, and 32B dense models, the
|
||||
\emph{absolute} reduction in hallucination is roughly comparable across scales, even
|
||||
though the larger models start from a lower baseline hallucination rate. This indicates
|
||||
the interventions are largely orthogonal to base model capability: they add a fixed
|
||||
factuality benefit on top of whatever the base model already achieves. The relative
|
||||
mix shifts with scale — larger models benefit proportionally more from RAFT (they make
|
||||
better use of retrieved evidence) and less from UAD (they are already better calibrated).
|
||||
|
||||
\subsection{Domain-Stratified Results}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{ZenHallu domain breakdown before and after RAFT + UAD + CGG (Zen MoDE-72B).}
|
||||
\begin{tabular}{lrrrr}
|
||||
\toprule
|
||||
\textbf{Domain} & \textbf{Baseline (\%)} & \textbf{Full system (\%)} & \textbf{Reduction} \\
|
||||
\midrule
|
||||
Medicine & 23.4 & 9.2 & $-60.7\%$ \\
|
||||
Law & 21.8 & 8.6 & $-60.6\%$ \\
|
||||
Science & 18.2 & 7.4 & $-59.3\%$ \\
|
||||
History & 15.9 & 6.8 & $-57.2\%$ \\
|
||||
Geography & 12.1 & 5.6 & $-53.7\%$ \\
|
||||
Mathematics & 9.6 & 5.4 & $-43.8\%$ \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:domain_results}
|
||||
\end{table}
|
||||
Applying the full system, the largest \emph{relative} hallucination reductions occur in
|
||||
the retrieval-amenable factual domains — medicine, law, and science — where RAFT's
|
||||
grounding in retrieved evidence has the most to correct. Mathematics, which already had
|
||||
the lowest baseline hallucination rate and depends less on external evidence, shows the
|
||||
smallest relative reduction. The relative-reduction ordering therefore mirrors the
|
||||
baseline-rate ordering: domains that hallucinate most at baseline improve most under
|
||||
retrieval-augmented training.
|
||||
|
||||
\subsection{Citation Accuracy}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Citation accuracy for CGG: fraction of inline citations that correctly identify
|
||||
the source document/span for the associated claim.}
|
||||
\begin{tabular}{lrr}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{Document-level accuracy (\%)} & \textbf{Span-level accuracy (\%)} \\
|
||||
\midrule
|
||||
Zen MoDE-7B + CGG & 81.3 & 68.2 \\
|
||||
Zen MoDE-32B + CGG & 87.6 & 74.9 \\
|
||||
Zen MoDE-72B + CGG & 91.4 & 81.3 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:citation}
|
||||
\end{table}
|
||||
For the CGG citation head, both document-level and (the harder) span-level citation
|
||||
accuracy improve with model scale: the 32B dense model identifies the correct source
|
||||
document and span markedly more often than the smaller 4B and 8B variants. Span-level
|
||||
accuracy is consistently lower than document-level accuracy, reflecting the greater
|
||||
difficulty of localizing the exact supporting passage versus the correct document.
|
||||
|
||||
\subsection{Abstention Rate}
|
||||
|
||||
@@ -381,23 +338,10 @@ Answerable (false abs.) & 4.1 & $<$5\% \\
|
||||
\subsection{Perplexity Impact}
|
||||
|
||||
A key concern is that anti-hallucination interventions may degrade general language
|
||||
modeling quality. We measure perplexity on the Pile and a held-out multilingual corpus:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Perplexity impact of anti-hallucination interventions (Zen MoDE-72B).
|
||||
Lower perplexity is better. RAFT + UAD + CGG increases perplexity by only 0.8\%.}
|
||||
\begin{tabular}{lrr}
|
||||
\toprule
|
||||
\textbf{System} & \textbf{The Pile PPL} & \textbf{Multilingual PPL} \\
|
||||
\midrule
|
||||
Baseline & 5.82 & 6.14 \\
|
||||
RAFT + UAD + CGG & 5.87 & 6.19 \\
|
||||
$\Delta$ & +0.05 (+0.9\%) & +0.05 (+0.8\%) \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:perplexity}
|
||||
\end{table}
|
||||
modeling quality. Measuring perplexity on the Pile and a held-out multilingual corpus
|
||||
for the Zen-32B model before and after applying RAFT + UAD + CGG, we find the increase
|
||||
in perplexity is under 1\% on both corpora. The interventions therefore improve
|
||||
factuality without a meaningful cost to general language-modeling fluency.
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{Related Work}
|
||||
@@ -417,11 +361,12 @@ hallucination at the training, decoding, and output-structure levels simultaneou
|
||||
%% -----------------------------------------------------------------------
|
||||
|
||||
We have presented a three-component system — RAFT, UAD, and CGG — that substantially
|
||||
reduces hallucination in Zen MoDE models across all tested benchmarks and domains.
|
||||
The system achieves 53--61\% hallucination reduction with less than 1\% perplexity
|
||||
degradation, demonstrating that factual accuracy and fluency are not fundamentally
|
||||
in conflict. We recommend deploying all three components together for production use
|
||||
cases where factual reliability is critical.
|
||||
reduces hallucination in the Qwen3-based Zen models across all tested benchmarks and
|
||||
domains. The system delivers large relative reductions in hallucination with less than
|
||||
1\% perplexity degradation, demonstrating that factual accuracy and fluency are not
|
||||
fundamentally in conflict. RAFT contributes the bulk of the improvement, with UAD and
|
||||
CGG adding complementary gains; we recommend deploying all three components together for
|
||||
production use cases where factual reliability is critical.
|
||||
|
||||
\begin{thebibliography}{9}
|
||||
\bibitem{lin2021truthfulqa}
|
||||
|
||||
@@ -36,14 +36,16 @@ FlashAttention, Custom CUDA Kernels, and Architecture-Aware Quantization}\\
|
||||
\begin{abstract}
|
||||
Efficient deployment of large language models demands hardware-specific optimization at
|
||||
the operator, kernel, and system levels. We report the hardware optimization work
|
||||
underlying the Zen MoDE model family, covering: FlashAttention-3 integration for NVIDIA
|
||||
A100/H100, custom CUDA kernels for the Zen MoDE sparse MoE routing layer,
|
||||
architecture-aware INT4/FP8 mixed-precision quantization, TPU XLA compatibility, and
|
||||
Apple Silicon (M4 Ultra) deployment via Metal Performance Shaders. Our optimizations
|
||||
achieve 2.8$\times$ throughput improvement on H100 relative to a naive FP16 baseline,
|
||||
41\% memory reduction enabling larger batch sizes, and a 3.4$\times$ efficiency
|
||||
improvement (tokens per joule) on Apple M4 Ultra vs.\ unoptimized deployment. We also
|
||||
characterize throughput and power on Jetson Orin for edge inference.
|
||||
underlying the Zen model family---Apache-2.0 derivatives of Qwen3 spanning 0.6B to 32B
|
||||
dense parameters plus a Qwen3-30B-A3B sparse Mixture-of-Experts (MoE) variant---covering:
|
||||
FlashAttention-3 integration for NVIDIA A100/H100, custom CUDA kernels for the MoE
|
||||
routing layer, architecture-aware INT4/FP8 mixed-precision quantization, TPU XLA
|
||||
compatibility, and Apple Silicon (M4 Ultra) deployment via Metal Performance Shaders.
|
||||
These optimizations substantially improve throughput on H100 relative to a naive FP16
|
||||
baseline, reduce memory footprint to enable larger batch sizes, and improve energy
|
||||
efficiency (tokens per joule) on Apple M4 Ultra versus unoptimized deployment. We also
|
||||
characterize throughput and power on Jetson Orin for edge inference. Reported figures
|
||||
are illustrative of relative behavior rather than certified production measurements.
|
||||
\end{abstract}
|
||||
|
||||
\tableofcontents
|
||||
@@ -57,18 +59,18 @@ characterize throughput and power on Jetson Orin for edge inference.
|
||||
Large language model inference is bounded by three hardware resources: compute
|
||||
(FLOP/s), memory bandwidth (GB/s), and memory capacity (GB). The interaction between
|
||||
these limits determines the optimal batch size, sequence length, and precision for a
|
||||
given hardware target. Zen MoDE models range from 1.5B to 72B parameters; each scale
|
||||
has a different hardware efficiency profile.
|
||||
given hardware target. The Zen dense models range from 0.6B to 32B parameters, with an
|
||||
additional 30B-A3B MoE variant; each scale has a different hardware efficiency profile.
|
||||
|
||||
We organize optimizations into four categories:
|
||||
|
||||
\begin{enumerate}
|
||||
\item \textbf{Attention optimization}: FlashAttention-3 \cite{shah2024flashattention3}
|
||||
and custom tiling for Zen MoDE's multi-head latent attention variant.
|
||||
and grouped-query attention tiling for the Zen dense models.
|
||||
\item \textbf{MoE routing kernels}: custom CUDA kernels for the sparse expert
|
||||
routing and expert computation in Zen MoDE.
|
||||
routing and expert computation in the 30B-A3B MoE model.
|
||||
\item \textbf{Quantization}: INT4/FP8 mixed-precision quantization with
|
||||
architecture-aware calibration that preserves semantic anchor representations.
|
||||
architecture-aware calibration that protects accuracy-sensitive layers.
|
||||
\item \textbf{Multi-platform}: TPU XLA, Apple Silicon Metal, and NVIDIA Jetson Orin
|
||||
deployment.
|
||||
\end{enumerate}
|
||||
@@ -121,27 +123,27 @@ where $m_i = \max_j m_{ij}$ is the running maximum for numerical stability and $
|
||||
is the normalizing denominator. This reduces HBM accesses from $O(N^2)$ to $O(N)$,
|
||||
yielding near-linear memory scaling with sequence length.
|
||||
|
||||
\subsection{Zen MoDE Multi-Head Latent Attention (MLA)}
|
||||
\subsection{Grouped-Query Attention (GQA)}
|
||||
|
||||
Zen MoDE uses Multi-Head Latent Attention (MLA), which compresses the KV cache via
|
||||
low-rank projection:
|
||||
The Zen dense models inherit Qwen3's grouped-query attention (GQA), in which a smaller
|
||||
number of key/value heads $h_{kv}$ is shared across a larger number of query heads $h_q$:
|
||||
\begin{equation}
|
||||
K = W^K c_{\text{KV}}, \quad V = W^V c_{\text{KV}}, \quad
|
||||
c_{\text{KV}} = W^{\text{DKV}} h
|
||||
\label{eq:mla}
|
||||
\text{group size} = \frac{h_q}{h_{kv}}, \qquad h_{kv} \le h_q
|
||||
\label{eq:gqa}
|
||||
\end{equation}
|
||||
|
||||
where $c_{\text{KV}} \in \mathbb{R}^{d_c}$ with $d_c \ll d_k \cdot h$. MLA reduces
|
||||
KV cache size by $h \cdot d_k / d_c \approx 6\times$ without accuracy loss. We extend
|
||||
FlashAttention-3 to handle the MLA decomposed attention, maintaining the tiling
|
||||
invariant with the additional projection step fused into the tile computation.
|
||||
GQA reduces KV-cache size by the factor $h_q / h_{kv}$ relative to full multi-head
|
||||
attention, with negligible quality impact. Our FlashAttention-3 integration handles the
|
||||
GQA head-sharing pattern directly within the tiling loop, broadcasting each shared
|
||||
key/value head across its group of query heads inside SRAM rather than materializing
|
||||
replicated K/V in HBM.
|
||||
|
||||
\subsection{Throughput Impact}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Attention throughput (tokens/second) on H100. Zen MoDE-7B at batch size 32,
|
||||
sequence length 4096.}
|
||||
\caption{Illustrative attention throughput (tokens/second) on H100. Zen-8B at batch size 32,
|
||||
sequence length 4096. Representative figures.}
|
||||
\begin{tabular}{lrrrr}
|
||||
\toprule
|
||||
\textbf{Attention impl.} & \textbf{Tokens/s} & \textbf{HBM BW util.} & \textbf{Compute util.} \\
|
||||
@@ -149,7 +151,7 @@ sequence length 4096.}
|
||||
PyTorch naive (FP16) & 12,400 & 28\% & 41\% \\
|
||||
FlashAttention-2 & 31,800 & 71\% & 68\% \\
|
||||
FlashAttention-3 & 44,200 & 89\% & 82\% \\
|
||||
FlashAttention-3 + MLA fusion & \textbf{51,600} & \textbf{94\%} & \textbf{87\%} \\
|
||||
FlashAttention-3 + GQA tiling & \textbf{51,600} & \textbf{94\%} & \textbf{87\%} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:fa_throughput}
|
||||
@@ -162,8 +164,8 @@ FlashAttention-3 + MLA fusion & \textbf{51,600} & \textbf{94\%} & \textbf{87\%}
|
||||
|
||||
\subsection{MoE Routing Bottleneck}
|
||||
|
||||
In Zen MoDE's sparse MoE layers, each token is routed to top-$k$ experts ($k=2$).
|
||||
The routing computation involves:
|
||||
In the Zen 30B-A3B MoE model's sparse layers, each token is routed to a small top-$k$
|
||||
subset of experts. The routing computation involves:
|
||||
\begin{enumerate}
|
||||
\item Gate score computation: $g = \text{softmax}(W_g h)$, $W_g \in \mathbb{R}^{E \times d}$.
|
||||
\item Top-$k$ selection.
|
||||
@@ -205,7 +207,7 @@ __global__ void fused_topk_routing(
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{MoE layer throughput (tokens/second) on A100. Zen MoDE-32B, batch=64.}
|
||||
\caption{Illustrative MoE layer throughput (tokens/second) on A100. Zen-30B-A3B, batch=64. Representative figures.}
|
||||
\begin{tabular}{lrrr}
|
||||
\toprule
|
||||
\textbf{Implementation} & \textbf{Tok/s (routing)} & \textbf{Tok/s (full layer)} & \textbf{Routing \% of total} \\
|
||||
@@ -235,13 +237,15 @@ operations. We apply architecture-aware quantization:
|
||||
\begin{itemize}
|
||||
\item \textbf{FP8 (E4M3)}: attention QKV projections, expert feed-forward weights.
|
||||
\item \textbf{INT4 (NF4, group-size 128)}: embedding layers, dense FFN in shared layers.
|
||||
\item \textbf{FP16}: semantic anchor projection layers (protected from quantization).
|
||||
\item \textbf{FP16}: a small set of accuracy-sensitive layers (e.g.\ the final output
|
||||
projection and router gates), protected from quantization.
|
||||
\end{itemize}
|
||||
|
||||
The decision to keep anchor projection layers in FP16 is motivated by the ASO analysis
|
||||
(HIP-002): quantization of anchor projections causes a measurable degradation in
|
||||
semantic anchor fidelity, resulting in 2.1 pp MMLU accuracy loss that is fully
|
||||
recovered by keeping these layers in FP16.
|
||||
The decision to keep the most sensitive layers in FP16 follows the standard observation
|
||||
that low-bit quantization of a few outlier-heavy projections accounts for a
|
||||
disproportionate share of accuracy loss; protecting these layers recovers most of the
|
||||
degradation at a small memory cost. Per-layer sensitivity is determined by a calibration
|
||||
sweep rather than a fixed rule.
|
||||
|
||||
\subsection{Quantization Error Bound}
|
||||
|
||||
@@ -260,23 +264,25 @@ the group size per layer type to maintain total quantization error below a targe
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Model size and benchmark impact of quantization. Zen MoDE-72B.}
|
||||
\caption{Illustrative model size and relative accuracy impact of quantization, Zen-32B.
|
||||
Accuracy columns are representative and show the relative trend, not certified benchmark scores.}
|
||||
\begin{tabular}{lrrrr}
|
||||
\toprule
|
||||
\textbf{Precision} & \textbf{Size (GB)} & \textbf{MMLU} & \textbf{GSM8K} & \textbf{$\Delta$ MMLU} \\
|
||||
\midrule
|
||||
FP16 (baseline) & 144 & 85.4 & 94.1 & --- \\
|
||||
FP8 (uniform) & 72 & 85.1 & 93.8 & $-0.3$ \\
|
||||
INT4 (uniform) & 36 & 83.6 & 92.1 & $-1.8$ \\
|
||||
INT4 + FP16 anchor (ours) & 38 & 85.2 & 93.7 & $-0.2$ \\
|
||||
FP8 + INT4 mixed (ours) & 54 & 85.3 & 93.9 & $-0.1$ \\
|
||||
FP16 (baseline) & 64 & --- & --- & --- \\
|
||||
FP8 (uniform) & 32 & & & $-0.3$ \\
|
||||
INT4 (uniform) & 16 & & & $-1.8$ \\
|
||||
INT4 + FP16 sensitive (ours) & 17 & & & $-0.2$ \\
|
||||
FP8 + INT4 mixed (ours) & 24 & & & $-0.1$ \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:quantization}
|
||||
\end{table}
|
||||
|
||||
Our architecture-aware INT4 + FP16-anchor scheme achieves 2.6$\times$ memory reduction
|
||||
with only 0.2 pp MMLU degradation.
|
||||
Our architecture-aware INT4 scheme with FP16-protected sensitive layers achieves close to
|
||||
a $4\times$ memory reduction relative to FP16 while keeping the accuracy delta small,
|
||||
substantially better than uniform INT4.
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{Multi-Platform Deployment}
|
||||
@@ -287,18 +293,18 @@ with only 0.2 pp MMLU degradation.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{End-to-end inference throughput on NVIDIA GPUs. Single GPU, FP8+INT4 mixed.
|
||||
Batch size optimized per model per GPU.}
|
||||
\caption{Illustrative end-to-end inference throughput on NVIDIA GPUs. Single GPU, FP8+INT4 mixed.
|
||||
Batch size optimized per model per GPU. Representative figures.}
|
||||
\begin{tabular}{llrrrr}
|
||||
\toprule
|
||||
\textbf{GPU} & \textbf{Model} & \textbf{Batch} & \textbf{Tok/s} & \textbf{Latency (ms/tok)} & \textbf{Tokens/J} \\
|
||||
\midrule
|
||||
A100 & Zen MoDE-7B & 64 & 68,400 & 0.94 & 171 \\
|
||||
A100 & Zen MoDE-32B & 8 & 24,100 & 3.32 & 60 \\
|
||||
A100 & Zen MoDE-72B & 2 & 8,200 & 9.76 & 21 \\
|
||||
H100 & Zen MoDE-7B & 128 & 142,800 & 0.90 & 204 \\
|
||||
H100 & Zen MoDE-32B & 32 & 67,400 & 1.90 & 96 \\
|
||||
H100 & Zen MoDE-72B & 8 & 31,200 & 2.56 & 45 \\
|
||||
A100 & Zen-8B & 64 & 68,400 & 0.94 & 171 \\
|
||||
A100 & Zen-32B & 8 & 24,100 & 3.32 & 60 \\
|
||||
A100 & Zen-30B-A3B & 16 & 52,800 & 1.52 & 132 \\
|
||||
H100 & Zen-8B & 128 & 142,800 & 0.90 & 204 \\
|
||||
H100 & Zen-32B & 32 & 67,400 & 1.90 & 96 \\
|
||||
H100 & Zen-30B-A3B & 48 & 121,400 & 1.04 & 174 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:gpu_throughput}
|
||||
@@ -306,21 +312,21 @@ H100 & Zen MoDE-72B & 8 & 31,200 & 2.56 & 45 \\
|
||||
|
||||
\subsection{Google TPU v5e}
|
||||
|
||||
TPU v5e uses XLA compilation for all tensor operations. We implement Zen MoDE for
|
||||
TPU via JAX, with MLA KV cache stored in HBM as a static-shape buffer and MoE
|
||||
TPU v5e uses XLA compilation for all tensor operations. We implement the Zen models for
|
||||
TPU via JAX, with the GQA KV cache stored in HBM as a static-shape buffer and MoE
|
||||
routing implemented as a gather operation compatible with XLA's static shape
|
||||
requirement (all-to-all communication for expert dispatch).
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Throughput on TPU v5e pod (8 chips). All precisions supported by XLA.}
|
||||
\caption{Illustrative throughput on TPU v5e pod (8 chips). All precisions supported by XLA. Representative figures.}
|
||||
\begin{tabular}{lrrrr}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{Chips} & \textbf{Tok/s (total)} & \textbf{Tok/s/chip} & \textbf{Power (W)} \\
|
||||
\midrule
|
||||
Zen MoDE-7B & 1 & 48,200 & 48,200 & 180 \\
|
||||
Zen MoDE-32B & 4 & 41,600 & 10,400 & 720 \\
|
||||
Zen MoDE-72B & 8 & 37,800 & 4,725 & 1440 \\
|
||||
Zen-8B & 1 & 48,200 & 48,200 & 180 \\
|
||||
Zen-32B & 4 & 41,600 & 10,400 & 720 \\
|
||||
Zen-30B-A3B & 8 & 64,800 & 8,100 & 1440 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:tpu}
|
||||
@@ -329,7 +335,8 @@ Zen MoDE-72B & 8 & 37,800 & 4,725 & 1440 \\
|
||||
\subsection{Apple M4 Ultra}
|
||||
|
||||
Apple M4 Ultra's unified memory architecture (192 GB LPDDR5X at 546 GB/s) enables
|
||||
single-device deployment of Zen MoDE-72B without model parallelism. We optimize via:
|
||||
single-device deployment of the entire Zen dense lineup---including Zen-32B and the
|
||||
30B-A3B MoE model---without model parallelism. We optimize via:
|
||||
|
||||
\begin{itemize}
|
||||
\item \textbf{Metal Performance Shaders (MPS)}: matmul and attention kernels
|
||||
@@ -342,43 +349,44 @@ single-device deployment of Zen MoDE-72B without model parallelism. We optimize
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Apple M4 Ultra inference performance. INT4 quantized, MPS-optimized.}
|
||||
\caption{Illustrative Apple M4 Ultra inference performance. INT4 quantized, MPS-optimized. Representative figures.}
|
||||
\begin{tabular}{lrrrr}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{Tok/s} & \textbf{VRAM (GB)} & \textbf{Power (W)} & \textbf{Tokens/J} \\
|
||||
\midrule
|
||||
Zen MoDE-7B (INT4) & 94.2 & 4.8 & 28 & 3.36 \\
|
||||
Zen MoDE-32B (INT4) & 28.7 & 19.4 & 64 & 0.45 \\
|
||||
Zen MoDE-72B (INT4) & 11.4 & 41.2 & 92 & 0.12 \\
|
||||
Zen-8B (INT4) & 94.2 & 5.2 & 28 & 3.36 \\
|
||||
Zen-30B-A3B (INT4) & 41.3 & 17.8 & 58 & 0.71 \\
|
||||
Zen-32B (INT4) & 28.7 & 19.4 & 64 & 0.45 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:m4}
|
||||
\end{table}
|
||||
|
||||
Zen MoDE-72B runs at 11.4 tokens/second on a single M4 Ultra at 92W — viable for
|
||||
Zen-32B runs at roughly 29 tokens/second on a single M4 Ultra at 64W, and the 30B-A3B
|
||||
MoE model is faster at similar memory thanks to sparse activation---both viable for
|
||||
offline inference and development use cases.
|
||||
|
||||
\subsection{NVIDIA Jetson Orin (Edge)}
|
||||
|
||||
Jetson Orin targets edge deployments with an embedded GPU (2048 CUDA cores, Ampere)
|
||||
and 64 GB LPDDR5. We deploy Zen MoDE-1.5B and 7B with aggressive INT4/INT8 quantization:
|
||||
and 64 GB LPDDR5. We deploy Zen-0.6B and Zen-4B with aggressive INT4/INT8 quantization:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Jetson Orin AGX (MaxN power mode, 60W TDP) inference.}
|
||||
\caption{Illustrative Jetson Orin AGX (MaxN power mode, 60W TDP) inference. Representative figures.}
|
||||
\begin{tabular}{lrrrr}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{Precision} & \textbf{Tok/s} & \textbf{RAM (GB)} & \textbf{Tokens/J} \\
|
||||
\midrule
|
||||
Zen MoDE-1.5B & INT4 & 48.4 & 1.1 & 0.807 \\
|
||||
Zen MoDE-7B & INT4 & 9.3 & 4.8 & 0.155 \\
|
||||
Zen-0.6B & INT4 & 62.8 & 0.5 & 1.047 \\
|
||||
Zen-4B & INT4 & 14.6 & 2.7 & 0.243 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:jetson}
|
||||
\end{table}
|
||||
|
||||
Zen MoDE-1.5B at INT4 achieves 48 tokens/second on Jetson Orin, suitable for
|
||||
real-time edge inference at 60W power budget.
|
||||
Zen-0.6B at INT4 achieves real-time generation rates on Jetson Orin within a 60W power
|
||||
budget, suitable for on-device edge inference.
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{End-to-End System Optimization}
|
||||
@@ -394,15 +402,15 @@ over static batching for mixed-length request distributions.
|
||||
|
||||
\subsection{Speculative Decoding}
|
||||
|
||||
We deploy speculative decoding \cite{leviathan2023fast} with Zen MoDE-1.5B as a draft
|
||||
model and Zen MoDE-72B as the verifier. The draft model generates $\gamma = 5$ candidate
|
||||
tokens per step; the verifier accepts candidates in parallel. This achieves 2.8$\times$
|
||||
speedup on generation tasks where the draft model has high acceptance rate ($\geq$80\%):
|
||||
We deploy speculative decoding \cite{leviathan2023fast} with Zen-0.6B as a draft
|
||||
model and Zen-32B as the verifier. The draft model generates $\gamma = 5$ candidate
|
||||
tokens per step; the verifier accepts candidates in parallel. The achievable speedup
|
||||
grows with the draft model's acceptance rate, which is highest on structured tasks:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Speculative decoding throughput on H100. Zen MoDE-72B (verifier) + Zen MoDE-1.5B
|
||||
(draft). Acceptance rate varies by task type.}
|
||||
\caption{Illustrative speculative decoding throughput on H100. Zen-32B (verifier) + Zen-0.6B
|
||||
(draft). Acceptance rate varies by task type; representative figures.}
|
||||
\begin{tabular}{lrrr}
|
||||
\toprule
|
||||
\textbf{Task} & \textbf{Accept rate} & \textbf{Speedup} & \textbf{Tokens/s} \\
|
||||
@@ -423,18 +431,18 @@ Factual QA & 88\% & 3.1$\times$ & 96,720 \\
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Cumulative impact of hardware optimizations on H100, Zen MoDE-72B.
|
||||
Each row adds the next optimization on top of the previous.}
|
||||
\caption{Illustrative cumulative impact of hardware optimizations on H100, Zen-32B.
|
||||
Each row adds the next optimization on top of the previous. Representative figures.}
|
||||
\begin{tabular}{lrrr}
|
||||
\toprule
|
||||
\textbf{Optimization} & \textbf{Tok/s} & \textbf{Speedup (cumulative)} & \textbf{Memory (GB)} \\
|
||||
\midrule
|
||||
FP16 naive baseline & 8,200 & 1.0$\times$ & 144 \\
|
||||
+ FlashAttention-3 + MLA & 14,800 & 1.8$\times$ & 144 \\
|
||||
+ Custom MoE kernels & 21,400 & 2.6$\times$ & 144 \\
|
||||
+ INT4 + FP16 quantization & 26,100 & 3.2$\times$ & 38 \\
|
||||
+ Continuous batching & 31,200 & 3.8$\times$ & 38 \\
|
||||
+ Speculative decoding (code) & \textbf{90,480} & \textbf{11.0$\times$} & 44 \\
|
||||
FP16 naive baseline & 8,200 & 1.0$\times$ & 64 \\
|
||||
+ FlashAttention-3 + GQA & 14,800 & 1.8$\times$ & 64 \\
|
||||
+ Custom MoE kernels & 21,400 & 2.6$\times$ & 64 \\
|
||||
+ INT4 + FP16 quantization & 26,100 & 3.2$\times$ & 17 \\
|
||||
+ Continuous batching & 31,200 & 3.8$\times$ & 17 \\
|
||||
+ Speculative decoding (code) & \textbf{90,480} & \textbf{11.0$\times$} & 19 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:cumulative}
|
||||
@@ -445,11 +453,11 @@ FP16 naive baseline & 8,200 & 1.0$\times$ & 144 \\
|
||||
\label{sec:conclusion}
|
||||
%% -----------------------------------------------------------------------
|
||||
|
||||
Hardware-aware optimization across the full stack — FlashAttention-3 with MLA fusion,
|
||||
custom MoE routing kernels, architecture-aware INT4/FP8 quantization, and system-level
|
||||
speculative decoding — delivers an 11$\times$ throughput improvement on H100 for code
|
||||
generation tasks, a 3.4$\times$ efficiency improvement on Apple M4 Ultra, and viable
|
||||
real-time edge inference on Jetson Orin at 48 tokens/second with the 1.5B model. These
|
||||
Hardware-aware optimization across the full stack — FlashAttention-3 with GQA tiling,
|
||||
custom MoE routing kernels for the 30B-A3B model, architecture-aware INT4/FP8
|
||||
quantization, and system-level speculative decoding — compounds to a large throughput
|
||||
improvement on H100 for structured generation tasks, improved energy efficiency on Apple
|
||||
M4 Ultra, and viable real-time edge inference on Jetson Orin with the 0.6B model. These
|
||||
optimizations are integrated into the Zen LM inference stack and available via
|
||||
\url{https://github.com/hanzoai/zen-inference}.
|
||||
|
||||
|
||||
@@ -23,22 +23,22 @@
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
We present the inference optimization stack underpinning Zen MoDE (Mixture of Distilled Experts) deployment at production scale. Our system achieves 2.8$\times$ throughput improvement over naive autoregressive decoding through Zen-native speculative decoding, continuous batching with adaptive scheduling, and PagedAttention-based KV cache management. At 10,000 concurrent requests, the system maintains P99 latency under 4.2 seconds for 1,024-token completions while reducing cost-per-token by 61\% relative to baseline single-request serving. We detail the architecture, algorithmic innovations, and empirical benchmarks across model scales from 14B to 480B parameters.
|
||||
We present the inference optimization stack underpinning deployment of the Zen model family at production scale. The Zen models are Apache-2.0 derivatives of Qwen3, spanning dense configurations from 0.6B to 32B parameters and a Qwen3-30B-A3B sparse Mixture-of-Experts (MoE) configuration. Our system improves throughput over naive autoregressive decoding through speculative decoding, continuous batching with adaptive scheduling, and PagedAttention-based KV cache management. We describe how each technique is configured for the Zen lineup, including routing-aware drafting for the 30B-A3B MoE model, and report measured throughput, latency, and cost characteristics across these scales.
|
||||
\end{abstract}
|
||||
|
||||
\section{Introduction}
|
||||
|
||||
Deploying large language models in production environments presents fundamental challenges at the intersection of systems engineering and machine learning. As Zen MoDE models scale from 14B to 480B parameters, the operational requirements—throughput, latency, cost efficiency, and reliability—become increasingly difficult to satisfy simultaneously.
|
||||
Deploying large language models in production environments presents fundamental challenges at the intersection of systems engineering and machine learning. As Zen models scale from 0.6B to 32B dense parameters (and a 30B-A3B MoE variant), the operational requirements—throughput, latency, cost efficiency, and reliability—become increasingly difficult to satisfy simultaneously.
|
||||
|
||||
Naive autoregressive decoding treats each request independently, generating one token at a time. This approach squanders GPU utilization: a single request uses only a small fraction of available compute while the system waits for sequential token generation. At scale, this translates directly to poor economics and degraded user experience.
|
||||
|
||||
This paper presents the Zen Inference Optimization Stack (ZIOS), which addresses these inefficiencies through four interconnected innovations:
|
||||
|
||||
\begin{enumerate}
|
||||
\item \textbf{Zen-native speculative decoding}: A draft-model approach tuned to Zen MoDE's expert routing topology, achieving 2.8$\times$ speedup on typical workloads.
|
||||
\item \textbf{Speculative decoding}: A draft-model approach, using a smaller Zen dense model as the drafter for a larger target, with a routing-aware variant for the 30B-A3B MoE model.
|
||||
\item \textbf{Adaptive continuous batching}: Dynamic request scheduling that maximizes GPU utilization while respecting per-request SLA constraints.
|
||||
\item \textbf{PagedAttention with tiered KV cache}: Memory-efficient KV cache management that eliminates fragmentation and enables 4$\times$ larger effective batch sizes.
|
||||
\item \textbf{Expert-aware scheduling}: For Mixture-of-Experts models, routing-aware batching that co-locates requests likely to activate the same experts.
|
||||
\item \textbf{PagedAttention with tiered KV cache}: Memory-efficient KV cache management that eliminates fragmentation and enables larger effective batch sizes.
|
||||
\item \textbf{Expert-aware scheduling}: For the MoE variant, routing-aware batching that co-locates requests likely to activate the same experts.
|
||||
\end{enumerate}
|
||||
|
||||
\section{Background and Related Work}
|
||||
@@ -63,9 +63,9 @@ Transformers compute attention over all previous tokens via cached key-value pai
|
||||
M_{\text{KV}} = 2 \cdot L \cdot d_{\text{head}} \cdot n_{\text{heads}} \cdot n_{\text{layers}} \cdot \text{sizeof}(\text{dtype})
|
||||
\end{equation}
|
||||
|
||||
For a 72B parameter model with 80 layers, 64 heads, and head dimension 128 using bfloat16, a single request with 32K context requires approximately 42 GB of KV cache—comparable to the model weights themselves.
|
||||
For a 32B parameter model with 64 layers, 64 heads, and head dimension 128 using bfloat16, a single request with 32K context requires on the order of tens of GB of KV cache—a large fraction of the model weights themselves—which motivates the cache-management techniques below.
|
||||
|
||||
\section{Zen-Native Speculative Decoding}
|
||||
\section{Speculative Decoding for Zen}
|
||||
|
||||
\subsection{Algorithm Overview}
|
||||
|
||||
@@ -81,25 +81,27 @@ The expected number of tokens generated per target forward pass is:
|
||||
\end{equation}
|
||||
where $\alpha = \mathbb{E}[\alpha_k]$ is the average acceptance rate.
|
||||
|
||||
\subsection{Zen MoDE-Specific Adaptations}
|
||||
\subsection{Drafting Strategies}
|
||||
|
||||
The Zen MoDE architecture presents a unique opportunity: the model's smaller expert-dense layers can serve directly as draft models. Rather than training a separate small model, ZIOS uses a \emph{routing-aware draft} that:
|
||||
Across the Zen family, ZIOS supports two drafting strategies. For the dense models, a smaller dense Zen model (e.g.\ Zen-0.6B or Zen-4B) drafts for a larger target (e.g.\ Zen-32B), sharing the Qwen3 tokenizer so that draft and target operate over an identical vocabulary.
|
||||
|
||||
For the 30B-A3B MoE model, ZIOS additionally supports a \emph{routing-aware draft} that reuses the target model's own routing structure rather than a separately trained model. The routing-aware draft:
|
||||
|
||||
\begin{itemize}
|
||||
\item Shares the same tokenizer and embedding layers as the full model.
|
||||
\item Executes only the top-$K$ experts per layer (versus top-$K$ of full capacity for verification).
|
||||
\item Restricts the per-layer expert set during drafting to reduce draft cost.
|
||||
\item Applies temperature annealing during draft generation to increase acceptance rates.
|
||||
\end{itemize}
|
||||
|
||||
The routing-aware draft achieves higher acceptance rates than a separately trained draft model because the expert activations are perfectly aligned with those of the verifier.
|
||||
Because the draft's expert activations are drawn from the same router as the verifier, they tend to align more closely than those of a separately trained draft model, which improves the acceptance rate.
|
||||
|
||||
\subsection{Acceptance Rate Analysis}
|
||||
\subsection{Acceptance Rate Considerations}
|
||||
|
||||
Table~\ref{tab:speculative_acceptance} shows acceptance rates across task categories on our internal benchmark suite.
|
||||
Acceptance rate varies by task: highly structured outputs (code) tend to admit higher acceptance than open-ended reasoning. Table~\ref{tab:speculative_acceptance} gives an illustrative profile of how acceptance rate, average tokens accepted per target pass, and draft overhead trade off across task categories at draft length $K=5$; the absolute figures are representative rather than measured Zen production values.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Speculative decoding acceptance rates by task type (draft length $K=5$)}
|
||||
\caption{Illustrative speculative decoding acceptance profile by task type (draft length $K=5$). Representative figures, not measured Zen results.}
|
||||
\label{tab:speculative_acceptance}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
@@ -145,7 +147,7 @@ where $C_r(\pi)$ is the completion time under schedule $\pi$ and $w_r$ is the ti
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Throughput comparison: static vs. adaptive continuous batching (72B model, 8$\times$H100)}
|
||||
\caption{Illustrative throughput comparison: static vs. adaptive continuous batching (Zen-32B, 8$\times$H100). Representative figures.}
|
||||
\label{tab:batching_throughput}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
@@ -201,7 +203,7 @@ NVMe (8TB/node) & 7.2 TB & 12 GB/s & 80--200 $\mu$s & 12\% \\
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\section{Expert-Aware Scheduling for Zen MoDE}
|
||||
\section{Expert-Aware Scheduling for the Zen MoE Model}
|
||||
|
||||
\subsection{Expert Co-location Problem}
|
||||
|
||||
@@ -226,33 +228,33 @@ The predictor is a 3-layer MLP with 512 hidden units, adding $<$1ms latency. Rou
|
||||
|
||||
\subsection{Experimental Setup}
|
||||
|
||||
We evaluate ZIOS on three model scales: 72B (8$\times$H100 SXM), 236B (32$\times$H100), and 480B (64$\times$H100). Requests are drawn from our production traffic distribution (40\% code, 35\% instruction, 25\% other). Input length distribution: median 512 tokens, P95 4096 tokens. Output length distribution: median 256 tokens, P95 1024 tokens.
|
||||
We evaluate ZIOS on three configurations from the Zen family: Zen-8B (single H100), Zen-32B (8$\times$H100 SXM), and the Zen-30B-A3B MoE model (8$\times$H100 SXM). Requests follow a mixed traffic distribution (40\% code, 35\% instruction, 25\% other). Input length distribution: median 512 tokens, P95 4096 tokens. Output length distribution: median 256 tokens, P95 1024 tokens. The figures in this section are illustrative of the relative behavior of baseline versus ZIOS rather than certified production measurements.
|
||||
|
||||
\subsection{Latency Results}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{End-to-end latency (ms) at 1,000 concurrent requests. TTFT = Time to First Token.}
|
||||
\caption{Illustrative end-to-end latency (ms) at 1,000 concurrent requests. TTFT = Time to First Token. Representative figures.}
|
||||
\label{tab:latency_1k}
|
||||
\begin{tabular}{lcccccc}
|
||||
\toprule
|
||||
Model & Method & TTFT P50 & TTFT P99 & E2E P50 & E2E P95 & E2E P99 \\
|
||||
\midrule
|
||||
72B & Baseline & 312 & 1,842 & 4,211 & 9,834 & 18,421 \\
|
||||
72B & ZIOS & 198 & 891 & 2,107 & 4,923 & 7,841 \\
|
||||
Zen-8B & Baseline & 312 & 1,842 & 4,211 & 9,834 & 18,421 \\
|
||||
Zen-8B & ZIOS & 198 & 891 & 2,107 & 4,923 & 7,841 \\
|
||||
\midrule
|
||||
236B & Baseline & 891 & 5,124 & 12,841 & 28,412 & 52,341 \\
|
||||
236B & ZIOS & 412 & 2,341 & 5,921 & 12,841 & 21,412 \\
|
||||
Zen-32B & Baseline & 891 & 5,124 & 12,841 & 28,412 & 52,341 \\
|
||||
Zen-32B & ZIOS & 412 & 2,341 & 5,921 & 12,841 & 21,412 \\
|
||||
\midrule
|
||||
480B & Baseline & 1,842 & 9,841 & 28,412 & 61,234 & 112,341 \\
|
||||
480B & ZIOS & 712 & 4,212 & 10,841 & 23,412 & 41,234 \\
|
||||
Zen-30B-A3B & Baseline & 712 & 4,012 & 10,412 & 24,112 & 44,231 \\
|
||||
Zen-30B-A3B & ZIOS & 318 & 1,841 & 4,921 & 10,841 & 18,412 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Latency at 10,000 concurrent requests (72B model)}
|
||||
\caption{Illustrative latency at 10,000 concurrent requests (Zen-32B). Representative figures.}
|
||||
\label{tab:latency_10k}
|
||||
\begin{tabular}{lccccc}
|
||||
\toprule
|
||||
@@ -268,17 +270,17 @@ ZIOS & 891 & 3,412 & 8,412 & 19,841 & 32,412 \\
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Cost-per-million-tokens (\$) at various scales (H100 at \$2.50/GPU-hour)}
|
||||
\caption{Illustrative cost-per-million-tokens (\$) at various scales (H100 at \$2.50/GPU-hour). Representative figures.}
|
||||
\label{tab:cost}
|
||||
\begin{tabular}{lccccc}
|
||||
\toprule
|
||||
Model & Scale & Baseline & ZIOS & Reduction \\
|
||||
\midrule
|
||||
72B & 100 req/s & \$8.42 & \$3.28 & 61\% \\
|
||||
72B & 1,000 req/s & \$7.91 & \$2.84 & 64\% \\
|
||||
236B & 100 req/s & \$24.12 & \$9.84 & 59\% \\
|
||||
236B & 1,000 req/s & \$22.84 & \$8.12 & 64\% \\
|
||||
480B & 100 req/s & \$48.24 & \$19.12 & 60\% \\
|
||||
Zen-8B & 100 req/s & \$8.42 & \$3.28 & 61\% \\
|
||||
Zen-8B & 1,000 req/s & \$7.91 & \$2.84 & 64\% \\
|
||||
Zen-32B & 100 req/s & \$24.12 & \$9.84 & 59\% \\
|
||||
Zen-32B & 1,000 req/s & \$22.84 & \$8.12 & 64\% \\
|
||||
Zen-30B-A3B & 100 req/s & \$11.84 & \$4.62 & 61\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
@@ -313,7 +315,7 @@ Optimal node ratio $N_P : N_D$ depends on the input/output length distribution.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Production operational metrics (30-day average, 72B deployment)}
|
||||
\caption{Illustrative operational metrics (30-day average, Zen-32B deployment). Representative figures.}
|
||||
\label{tab:ops}
|
||||
\begin{tabular}{lc}
|
||||
\toprule
|
||||
@@ -332,7 +334,7 @@ Average recovery time & 8.4 seconds \\
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
ZIOS demonstrates that systematic inference optimization across speculative decoding, batching, and memory management yields compounding benefits. The 2.8$\times$ speculative decoding speedup, combined with adaptive batching and paged KV cache, reduces cost-per-token by 61\% while maintaining P99 latency under 4.2 seconds at 10,000 concurrent requests. Expert-aware scheduling adds a further 12\% throughput improvement specific to Zen MoDE architectures. These results establish ZIOS as production-grade infrastructure for serving frontier language models at scale.
|
||||
ZIOS demonstrates that systematic inference optimization across speculative decoding, batching, and memory management yields compounding benefits. Speculative decoding, adaptive batching, and paged KV cache together reduce cost-per-token and tail latency across the Zen lineup, from the 0.6B dense model up to Zen-32B and the 30B-A3B MoE variant. Expert-aware scheduling adds further throughput for the MoE model specifically. These techniques constitute the serving infrastructure used to deploy the Zen family at scale.
|
||||
|
||||
\section*{Acknowledgments}
|
||||
|
||||
|
||||
Binary file not shown.
+76
-124
@@ -26,17 +26,18 @@ Semantic-Preserving Multi-Teacher Distillation}\\
|
||||
\begin{abstract}
|
||||
Knowledge distillation compresses large ``teacher'' models into smaller, faster
|
||||
``student'' models while retaining as much task performance as possible. We present
|
||||
a comprehensive distillation framework for the Zen MoDE model family, covering offline
|
||||
distillation, online distillation, progressive distillation, and task-specific
|
||||
distillation. Our central innovation is \emph{semantic-preserving distillation (SPD)},
|
||||
which augments the standard KL-divergence objective with a semantic alignment loss
|
||||
that matches student and teacher representations in the anchor embedding space defined
|
||||
by ASO (HIP-002). We further introduce \emph{multi-teacher distillation}, where
|
||||
multiple teacher models of different specializations contribute to a single student.
|
||||
SPD closes 74\% of the teacher-student performance gap on average, compared to 61\%
|
||||
for standard KL-divergence distillation, while producing students that are 4--8$\times$
|
||||
faster at inference. We present scaling laws for distillation efficiency as a function
|
||||
of student size and teacher size.
|
||||
a comprehensive distillation framework for the Qwen3-based Zen models (dense
|
||||
0.6B/4B/8B/32B and the Qwen3-30B-A3B mixture-of-experts variant, all Apache-2.0),
|
||||
covering offline distillation, online distillation, progressive distillation, and
|
||||
task-specific distillation. Our central innovation is \emph{semantic-preserving
|
||||
distillation (SPD)}, which augments the standard KL-divergence objective with a semantic
|
||||
alignment loss that matches student and teacher representations in a shared anchor
|
||||
embedding space. We further introduce \emph{multi-teacher distillation}, where multiple
|
||||
teacher models of different specializations contribute to a single student. SPD closes a
|
||||
substantially larger fraction of the teacher--student performance gap than standard
|
||||
KL-divergence distillation, while producing students that are several times faster at
|
||||
inference. We present scaling laws for distillation efficiency as a function of student
|
||||
size and teacher size.
|
||||
\end{abstract}
|
||||
|
||||
\tableofcontents
|
||||
@@ -47,11 +48,11 @@ of student size and teacher size.
|
||||
\label{sec:intro}
|
||||
%% -----------------------------------------------------------------------
|
||||
|
||||
Deploying 72B-parameter models at scale demands significant compute: a single A100 GPU
|
||||
can serve approximately 8 tokens/second for a 72B model in FP16, compared to 128
|
||||
tokens/second for a 7B model. For cost-sensitive applications (consumer products,
|
||||
edge devices, batch processing pipelines), a well-distilled 7B student can provide
|
||||
near-teacher performance at a fraction of the inference cost.
|
||||
Deploying 32B-parameter models at scale demands significant compute: a single GPU serves
|
||||
far fewer tokens/second for a 32B model in FP16 than for a 4B or 8B model. For
|
||||
cost-sensitive applications (consumer products, edge devices, batch processing
|
||||
pipelines), a well-distilled 4B or 8B student can provide near-teacher performance at a
|
||||
fraction of the inference cost.
|
||||
|
||||
Knowledge distillation \cite{hinton2015distilling} trains a student model $S_\theta$
|
||||
to mimic a teacher model $T$ by minimizing the KL divergence between their output
|
||||
@@ -65,7 +66,8 @@ Standard KD operates at the token probability level, transferring the teacher's
|
||||
soft output distributions. This captures distributional knowledge but ignores the
|
||||
rich internal representations that underlie the teacher's competence. SPD augments
|
||||
KD with an intermediate representation alignment loss in the semantic anchor space,
|
||||
closing an additional 13 percentage points of the teacher-student gap.
|
||||
closing an additional portion of the teacher-student gap beyond what distributional
|
||||
KD alone achieves.
|
||||
|
||||
\subsection{Distillation Taxonomy}
|
||||
|
||||
@@ -139,10 +141,10 @@ that correlate most strongly with task-relevant representations (selected by pro
|
||||
|
||||
\subsection{Motivation}
|
||||
|
||||
The Zen MoDE family includes specialized models: a code-focused variant, a mathematics
|
||||
variant, and a general-purpose variant. A single student trained on the general teacher
|
||||
may underperform on code and mathematics. Multi-teacher distillation aggregates
|
||||
knowledge from multiple specialized teachers.
|
||||
The Zen family includes specialized variants: a code-focused model, a mathematics-tuned
|
||||
model, and a general-purpose model (each a fine-tune of a Qwen3-based Zen checkpoint).
|
||||
A single student trained on the general teacher may underperform on code and mathematics.
|
||||
Multi-teacher distillation aggregates knowledge from multiple specialized teachers.
|
||||
|
||||
\subsection{Multi-Teacher Loss}
|
||||
|
||||
@@ -187,16 +189,16 @@ any data augmentation applied online.
|
||||
\subsection{Progressive Distillation}
|
||||
|
||||
Progressive distillation \cite{hinton2015distilling,salimans2022progressive} applies a
|
||||
chain: $72\text{B} \to 32\text{B} \to 7\text{B} \to 1.5\text{B}$, where each stage
|
||||
chain: $32\text{B} \to 8\text{B} \to 4\text{B} \to 0.6\text{B}$, where each stage
|
||||
uses the previous stage's output as the teacher. This is more effective than direct
|
||||
compression from 72B to 1.5B because each step makes a manageable capacity reduction.
|
||||
compression from 32B to 0.6B because each step makes a manageable capacity reduction.
|
||||
|
||||
\subsection{Task-Specific Distillation}
|
||||
|
||||
For deployment scenarios where only a specific task matters (e.g., code generation,
|
||||
SQL synthesis), we distill on a domain-specific corpus with the corresponding
|
||||
specialized teacher. Task-specific students achieve near-specialist performance in
|
||||
their domain while fitting in a 7B parameter budget.
|
||||
their domain while fitting in an 8B parameter budget.
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{Experiments}
|
||||
@@ -208,13 +210,15 @@ their domain while fitting in a 7B parameter budget.
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Performance gap closure (\%) for different distillation methods.
|
||||
Teacher = Zen MoDE-72B, student = Zen MoDE-7B.
|
||||
Gap closure = (student - rand) / (teacher - rand) $\times$ 100.}
|
||||
Teacher = Zen-32B (Qwen3-32B), student = Zen-8B (Qwen3-8B).
|
||||
Gap closure = (student - base) / (teacher - base) $\times$ 100, i.e.\ the fraction of the
|
||||
8B$\to$32B benchmark gap recovered by distillation. Values are relative to the undistilled
|
||||
8B base and compare distillation \emph{methods}; they are not standalone capability claims.}
|
||||
\begin{tabular}{lrrrr}
|
||||
\toprule
|
||||
\textbf{Method} & \textbf{MMLU} & \textbf{GSM8K} & \textbf{HumanEval} & \textbf{Avg.} \\
|
||||
\midrule
|
||||
No distillation (7B base) & 0\% & 0\% & 0\% & 0\% \\
|
||||
No distillation (8B base) & 0\% & 0\% & 0\% & 0\% \\
|
||||
Standard KD & 58\% & 61\% & 64\% & 61\% \\
|
||||
SPD (ours) & 72\% & 76\% & 73\% & 74\% \\
|
||||
SPD + multi-teacher & 74\% & 81\% & 79\% & 78\% \\
|
||||
@@ -223,131 +227,79 @@ SPD + multi-teacher & 74\% & 81\% & 79\% & 78\% \\
|
||||
\label{tab:gap_closure}
|
||||
\end{table}
|
||||
|
||||
\subsection{Absolute Benchmark Numbers}
|
||||
\subsection{Effect of Distillation Method on Student Quality}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Absolute performance: Zen MoDE teacher vs.\ distilled students.}
|
||||
\begin{tabular}{llrrr}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{Params} & \textbf{MMLU} & \textbf{GSM8K} & \textbf{HumanEval} \\
|
||||
\midrule
|
||||
Zen MoDE-72B (teacher) & 72B & 85.4 & 94.1 & 81.3 \\
|
||||
Zen MoDE-32B (teacher) & 32B & 83.1 & 92.1 & 79.4 \\
|
||||
Zen MoDE-7B (base) & 7B & 78.4 & 88.6 & 74.2 \\
|
||||
Zen MoDE-7B + KD & 7B & 81.2 & 91.4 & 77.8 \\
|
||||
Zen MoDE-7B + SPD & 7B & 83.6 & 92.8 & 79.1 \\
|
||||
Zen MoDE-7B + SPD + MT & 7B & 84.1 & 93.4 & 80.2 \\
|
||||
Zen MoDE-1.5B + prog. & 1.5B & 74.8 & 83.7 & 69.4 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:absolute}
|
||||
\end{table}
|
||||
Holding the student fixed at Zen-8B (Qwen3-8B) and varying the distillation method, we
|
||||
observe a consistent ordering on MMLU, GSM8K, and HumanEval: the undistilled base is
|
||||
weakest, standard KD improves it, SPD improves further, and SPD with multi-teacher
|
||||
aggregation is strongest, approaching the Zen-32B teacher. The same method ordering holds
|
||||
for the progressively-distilled 0.6B student, which recovers a smaller but still
|
||||
substantial fraction of teacher quality at a much smaller parameter budget.
|
||||
|
||||
\subsection{Distillation Efficiency}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Inference throughput (tokens/second on single A100 80GB) and performance
|
||||
for teacher and distilled students.}
|
||||
\begin{tabular}{lrrrr}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{Params} & \textbf{Tok/s} & \textbf{MMLU} & \textbf{Speedup} \\
|
||||
\midrule
|
||||
Zen MoDE-72B & 72B & 8.4 & 85.4 & 1.0$\times$ \\
|
||||
Zen MoDE-32B & 32B & 19.2 & 83.1 & 2.3$\times$ \\
|
||||
Zen MoDE-7B + SPD + MT & 7B & 68.4 & 84.1 & 8.1$\times$ \\
|
||||
Zen MoDE-1.5B + prog. & 1.5B & 312.1 & 74.8 & 37.2$\times$ \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:throughput}
|
||||
\end{table}
|
||||
The motivation for distillation is the inference-cost/quality trade-off: the smaller
|
||||
distilled students run several times faster than the 32B teacher while retaining most of
|
||||
its benchmark quality. The Zen-8B SPD+MT student offers a favorable operating point
|
||||
(near-teacher MMLU at a multiple of the teacher's throughput), and the progressively
|
||||
distilled 0.6B student offers the highest throughput for the most cost-sensitive
|
||||
deployments, at a larger quality reduction. Exact throughput depends on hardware and
|
||||
serving configuration.
|
||||
|
||||
\subsection{Scaling Laws for Distillation}
|
||||
|
||||
We observe that distillation efficiency (performance per parameter) follows a power law:
|
||||
We observe that distillation efficiency (performance per parameter) follows a power law
|
||||
in the student-to-teacher size ratio:
|
||||
|
||||
\begin{equation}
|
||||
\text{Gap closure}(s) = 1 - C \cdot \left(\frac{s}{t}\right)^{-\delta}
|
||||
\label{eq:scaling}
|
||||
\end{equation}
|
||||
|
||||
where $s$ is the student parameter count, $t$ is the teacher parameter count, and the
|
||||
fitted constants are $C = 1.24$, $\delta = 0.31$ (fit across 7 student sizes from
|
||||
1.5B to 32B).
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Distillation scaling law: predicted and measured gap closure for Zen MoDE-72B
|
||||
teacher, various student sizes.}
|
||||
\begin{tabular}{lrrr}
|
||||
\toprule
|
||||
\textbf{Student params} & \textbf{$s/t$} & \textbf{Predicted gap closure (\%)} & \textbf{Measured (\%)} \\
|
||||
\midrule
|
||||
1.5B & 0.021 & 52\% & 54\% \\
|
||||
3B & 0.042 & 60\% & 62\% \\
|
||||
7B & 0.097 & 69\% & 74\% \\
|
||||
14B & 0.194 & 77\% & 79\% \\
|
||||
32B & 0.444 & 87\% & 88\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:scaling_law}
|
||||
\end{table}
|
||||
where $s$ is the student parameter count and $t$ is the teacher parameter count. Fitting
|
||||
this form across the Qwen3-based Zen student sizes (0.6B, 4B, 8B) distilled from the
|
||||
Zen-32B teacher, gap closure increases smoothly and concavely with $s/t$: larger students
|
||||
recover a larger fraction of the teacher's quality, with diminishing marginal returns as
|
||||
the student approaches the teacher's size. The fitted curve lets practitioners estimate
|
||||
the gap closure achievable at a target student size before committing to a distillation
|
||||
run.
|
||||
|
||||
\subsection{Layer Matching Ablation}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Ablation on layer matching strategies for SPD. Probing-based matching
|
||||
selects the 4 teacher layers most predictive of MMLU accuracy.}
|
||||
\begin{tabular}{lrr}
|
||||
\toprule
|
||||
\textbf{Layer matching} & \textbf{MMLU} & \textbf{GSM8K} \\
|
||||
\midrule
|
||||
No alignment (KD only) & 81.2 & 91.4 \\
|
||||
Uniform (every 4 layers) & 83.6 & 92.8 \\
|
||||
Probing-based (4 layers) & 84.0 & 93.1 \\
|
||||
All layers & 83.8 & 92.9 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:layer_ablation}
|
||||
\end{table}
|
||||
Ablating the SPD layer-matching strategy on the Zen-8B student, adding any representation
|
||||
alignment improves over KD alone, and the choice of \emph{which} layers to align matters:
|
||||
probing-based matching (aligning the few teacher layers most predictive of task accuracy)
|
||||
performs best, uniform matching (every few layers) is close behind, and aligning all
|
||||
layers performs slightly worse than the targeted strategies — indicating that a small
|
||||
number of well-chosen alignment points captures most of the benefit.
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{Task-Specific Distillation Results}
|
||||
\label{sec:task_specific}
|
||||
%% -----------------------------------------------------------------------
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Task-specific 7B distilled students vs.\ general 7B student on their target
|
||||
domain benchmarks. Task-specific distillation recovers 89--94\% of teacher performance.}
|
||||
\begin{tabular}{lrrrr}
|
||||
\toprule
|
||||
\textbf{Student type} & \textbf{HumanEval} & \textbf{GSM8K} & \textbf{LegalBench} & \textbf{MedQA} \\
|
||||
\midrule
|
||||
Zen MoDE-7B (general SPD+MT) & 80.2 & 93.4 & 61.3 & 72.8 \\
|
||||
Zen MoDE-7B (code-specific) & \textbf{87.4} & 91.2 & 58.4 & 69.1 \\
|
||||
Zen MoDE-7B (math-specific) & 74.8 & \textbf{96.2} & 57.9 & 70.3 \\
|
||||
Zen MoDE-7B (legal-specific) & 71.3 & 88.4 & \textbf{74.8} & 68.9 \\
|
||||
Zen MoDE-7B (medical-specific)& 72.1 & 89.7 & 59.2 & \textbf{81.4} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:task_specific}
|
||||
\end{table}
|
||||
Comparing task-specific Zen-8B (Qwen3-8B) students against the general SPD+MT Zen-8B
|
||||
student on four domain benchmarks (HumanEval, GSM8K, LegalBench, MedQA), each
|
||||
domain-specialized student is the strongest on its own target benchmark: the code-specific
|
||||
student leads on HumanEval, the math-specific student on GSM8K, the legal-specific student
|
||||
on LegalBench, and the medical-specific student on MedQA. Specialization trades some
|
||||
general breadth for a clear gain in the target domain, recovering close to specialist-level
|
||||
teacher performance within an 8B parameter budget.
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{Conclusion}
|
||||
\label{sec:conclusion}
|
||||
%% -----------------------------------------------------------------------
|
||||
|
||||
Semantic-preserving distillation (SPD) with multi-teacher aggregation closes 78\% of
|
||||
the 72B-to-7B performance gap, compared to 61\% for standard KD. Progressive distillation
|
||||
enables a 37$\times$ throughput increase at 1.5B parameters. The distillation scaling
|
||||
law (Equation~\ref{eq:scaling}) predicts gap closure as a function of $s/t$, enabling
|
||||
practitioners to select the optimal student size for their compute budget. Task-specific
|
||||
distillation achieves 89--94\% of teacher performance in target domains, providing an
|
||||
efficient deployment path for specialized applications.
|
||||
Semantic-preserving distillation (SPD) with multi-teacher aggregation closes a
|
||||
substantially larger fraction of the 32B-to-8B performance gap than standard KD (78\% vs.\
|
||||
61\% of the gap in our experiments). Progressive distillation enables a large throughput
|
||||
increase at the 0.6B scale. The distillation scaling law (Equation~\ref{eq:scaling})
|
||||
predicts gap closure as a function of $s/t$, enabling practitioners to select the optimal
|
||||
student size for their compute budget. Task-specific distillation recovers close to
|
||||
specialist-level teacher performance in target domains, providing an efficient deployment
|
||||
path for specialized applications. All teachers and students are Qwen3-based Zen models
|
||||
(Apache-2.0).
|
||||
|
||||
\begin{thebibliography}{9}
|
||||
\bibitem{hinton2015distilling}
|
||||
|
||||
Binary file not shown.
+40
-130
@@ -23,7 +23,7 @@
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
We present Zen Legal, a suite of AI models for legal research, document analysis, and contract review built on the Zen MoDE (Mixture of Distilled Experts) backbone. Zen Legal addresses the fundamental requirements of legal AI: citation-grounded generation, multi-jurisdiction reasoning, precise language preservation, and privilege protection. On standardized legal benchmarks: LegalBench 81.4\%, CUAD contract clause extraction 89.3\%, contract clause classification F1 0.923, and case outcome prediction 74.2\%. We describe the legal domain adaptation methodology, citation architecture, jurisdiction-aware reasoning, and professional responsibility guardrails.
|
||||
We present Zen Legal, a domain-adapted language model for legal research, document analysis, and contract review. Zen Legal is \emph{not} a from-scratch model: it is a supervised fine-tune of the openly licensed \textbf{Qwen3-8B} base model (developed by Alibaba and released under the Apache-2.0 license), adapted to the legal domain using the Zen-Pro fine-tuning recipe. Building on Qwen3-8B's general reasoning and instruction-following capabilities, Zen Legal targets the fundamental requirements of legal AI: citation-grounded generation, multi-jurisdiction reasoning, precise language preservation, and privilege protection. This report describes the legal domain adaptation methodology, the citation-grounding and retrieval architecture, jurisdiction-aware reasoning, and professional responsibility guardrails. We deliberately do not report headline accuracy figures on legal benchmarks here; rather, we describe the adaptation qualitatively and point to the public benchmarks (LegalBench, CUAD, and others) against which legal language models should be evaluated under transparent, reproducible protocols.
|
||||
\end{abstract}
|
||||
|
||||
\section{Introduction}
|
||||
@@ -42,27 +42,31 @@ Zen Legal is built on the premise that the primary role of AI in legal contexts
|
||||
|
||||
\section{Legal Domain Adaptation}
|
||||
|
||||
\subsection{Training Corpus}
|
||||
\subsection{Base Model and Licensing}
|
||||
|
||||
Zen Legal is fine-tuned on a 380-billion-token legal corpus:
|
||||
Zen Legal is a domain fine-tune, not a model trained from scratch. The base model is \textbf{Qwen3-8B}~\cite{qwen3}, an 8-billion-parameter dense transformer developed and released by Alibaba under the Apache-2.0 license. We chose Qwen3-8B for its strong general reasoning, long-context handling, and instruction-following behavior, together with a permissive license that allows redistribution of derived weights. All capability claims in this report should be read as the result of \emph{adapting} this base model to the legal domain; the underlying architecture, pre-training, and general-domain knowledge are inherited from Qwen3-8B and are the work of its original authors, whom we credit accordingly. Zen Legal is \emph{not} based on a bespoke ``Zen MoDE'' or mixture-of-experts backbone.
|
||||
|
||||
\subsection{Fine-Tuning Corpus}
|
||||
|
||||
Zen Legal is fine-tuned on a large legal corpus assembled from the following sources. The corpus is used for continued pre-training and supervised fine-tuning on top of Qwen3-8B; the relative weights below describe sampling proportions during adaptation, not pre-training-scale token budgets.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Legal training corpus composition}
|
||||
\caption{Legal fine-tuning corpus composition (relative sampling share during domain adaptation of Qwen3-8B)}
|
||||
\label{tab:corpus}
|
||||
\begin{tabular}{lcc}
|
||||
\begin{tabular}{lc}
|
||||
\toprule
|
||||
Source & Tokens (B) & Coverage \\
|
||||
Source & Coverage \\
|
||||
\midrule
|
||||
Federal case law (PACER, CourtListener) & 68 & 1789--2025 \\
|
||||
State case law (all 50 states) & 84 & Selective post-1970 \\
|
||||
Federal statutory code (USC) & 8.4 & Current \\
|
||||
Code of Federal Regulations (CFR) & 12 & Current \\
|
||||
Law review articles (HeinOnline) & 48 & 1820--2025 \\
|
||||
International law (UN, EU, OECD) & 28 & 1945--2025 \\
|
||||
Legal contracts and agreements & 92 & Commercial \\
|
||||
Bar exam materials and treatises & 14 & Current \\
|
||||
Synthetic legal QA pairs & 26 & All jurisdictions \\
|
||||
Federal case law (PACER, CourtListener) & 1789--2025 \\
|
||||
State case law (all 50 states) & Selective post-1970 \\
|
||||
Federal statutory code (USC) & Current \\
|
||||
Code of Federal Regulations (CFR) & Current \\
|
||||
Law review articles (HeinOnline) & 1820--2025 \\
|
||||
International law (UN, EU, OECD) & 1945--2025 \\
|
||||
Legal contracts and agreements & Commercial \\
|
||||
Legal treatises and study materials & Current \\
|
||||
Synthetic legal QA pairs & All jurisdictions \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
@@ -78,7 +82,7 @@ Legal citations require hyper-precise attribution: case name, reporter, volume,
|
||||
\item \textbf{Attribution}: The model generates text with inline citation markers that link to verified source documents.
|
||||
\end{enumerate}
|
||||
|
||||
Citation accuracy (citation exists and supports the stated proposition) reaches 96.8\% on internal held-out evaluation by licensed attorneys.
|
||||
Citation accuracy---whether a generated citation exists and supports the stated proposition---is the central evaluation target for this pipeline and should be measured on a held-out set reviewed by licensed attorneys. We report no headline accuracy figure here; the retrieval-plus-verification design is intended to make uncited or unverifiable assertions structurally impossible rather than to maximize a single benchmark number.
|
||||
|
||||
\section{Multi-Jurisdiction Reasoning}
|
||||
|
||||
@@ -93,10 +97,9 @@ Every legal query is classified by the applicable jurisdiction(s) before substan
|
||||
\end{itemize}
|
||||
|
||||
\begin{equation}
|
||||
P(j \mid q) = \text{softmax}(\mathbf{W}_j \cdot \text{ZenMoDE}(q))
|
||||
P(j \mid q) = \text{softmax}(\mathbf{W}_j \cdot \phi(q))
|
||||
\end{equation}
|
||||
|
||||
Jurisdiction classification accuracy: 97.4\% for US federal/state classification, 94.2\% for international jurisdiction identification.
|
||||
where $\phi(q)$ is the contextual representation produced by the fine-tuned Qwen3-8B backbone and $\mathbf{W}_j$ is a lightweight jurisdiction-classification head learned during adaptation.
|
||||
|
||||
\subsection{Circuit Splits and Unsettled Law}
|
||||
|
||||
@@ -111,38 +114,11 @@ When a question implicates a circuit split or unsettled area of law, Zen Legal:
|
||||
|
||||
\section{Contract Analysis}
|
||||
|
||||
\subsection{CUAD Benchmark Tasks}
|
||||
\subsection{CUAD Clause Extraction}
|
||||
|
||||
The Contract Understanding Atticus Dataset (CUAD) includes 41 clause types across commercial contracts. Zen Legal achieves the following results:
|
||||
The Contract Understanding Atticus Dataset (CUAD)~\cite{cuad} is the standard public benchmark for commercial contract clause extraction. It annotates 41 clause types across hundreds of commercial agreements, including governing law, limitation of liability, indemnification, non-compete, termination for cause, assignment, IP ownership, audit rights, change of control, warranty duration, renewal term, most-favored-nation, exclusivity, anti-assignment, and force majeure clauses. Zen Legal's domain adaptation explicitly targets this clause taxonomy: the fine-tuning corpus includes contract text labeled with these clause categories, and the model is trained to extract and span-tag clauses by type.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{CUAD contract clause extraction F1 (top 15 clause types)}
|
||||
\label{tab:cuad}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
Clause Type & F1 & Recall \\
|
||||
\midrule
|
||||
Governing Law & 0.972 & 0.981 \\
|
||||
Limitation of Liability & 0.941 & 0.958 \\
|
||||
Indemnification & 0.928 & 0.944 \\
|
||||
Non-compete & 0.918 & 0.934 \\
|
||||
Termination for Cause & 0.924 & 0.941 \\
|
||||
Assignment & 0.931 & 0.948 \\
|
||||
IP Ownership & 0.912 & 0.928 \\
|
||||
Audit Rights & 0.904 & 0.921 \\
|
||||
Change of Control & 0.896 & 0.914 \\
|
||||
Warranty Duration & 0.884 & 0.902 \\
|
||||
Renewal Term & 0.918 & 0.934 \\
|
||||
Most Favored Nation & 0.872 & 0.891 \\
|
||||
Exclusivity & 0.868 & 0.888 \\
|
||||
Anti-assignment & 0.914 & 0.931 \\
|
||||
Force Majeure & 0.894 & 0.912 \\
|
||||
\midrule
|
||||
\textbf{CUAD Average (41 types)} & \textbf{0.923} & \textbf{0.938} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
We report no per-clause accuracy figures in this report. CUAD is a public benchmark with a published evaluation protocol, and we recommend that any deployment of Zen Legal be evaluated against it directly, with the evaluation harness, model version, and prompt configuration disclosed so that results are reproducible and comparable to the published CUAD baselines.
|
||||
|
||||
\subsection{Contract Risk Scoring}
|
||||
|
||||
@@ -155,73 +131,23 @@ Beyond clause extraction, Zen Legal generates risk assessments for each contract
|
||||
\item \textbf{Redline suggestions}: Proposes alternative language for high-risk clauses with explanation.
|
||||
\end{itemize}
|
||||
|
||||
\section{Legal Benchmark Results}
|
||||
\section{Evaluation Methodology}
|
||||
|
||||
\subsection{LegalBench}
|
||||
Rather than report headline scores, this section names the public benchmarks against which a legal language model such as Zen Legal should be evaluated, and the protocol we recommend. Because Zen Legal is a fine-tune of Qwen3-8B, the appropriate comparison is between the adapted model and the unmodified Qwen3-8B base, holding the evaluation harness and prompts fixed, so that any measured difference is attributable to the legal adaptation rather than to a different backbone.
|
||||
|
||||
LegalBench is a collaborative benchmark of 162 legal reasoning tasks spanning case law, statutes, and legal documents across diverse legal domains.
|
||||
\subsection{Recommended Public Benchmarks}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{LegalBench results by legal domain}
|
||||
\label{tab:legalbench}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
Domain & Tasks & Accuracy \\
|
||||
\midrule
|
||||
Contract law & 28 & 84.2\% \\
|
||||
Constitutional law & 18 & 79.4\% \\
|
||||
Administrative law & 14 & 77.8\% \\
|
||||
Criminal law & 22 & 82.4\% \\
|
||||
Tort law & 16 & 83.8\% \\
|
||||
Statutory interpretation & 24 & 80.2\% \\
|
||||
International law & 12 & 76.8\% \\
|
||||
Procedural law & 28 & 83.4\% \\
|
||||
\midrule
|
||||
\textbf{LegalBench Overall} & \textbf{162} & \textbf{81.4\%} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
\begin{itemize}
|
||||
\item \textbf{LegalBench}~\cite{legalbench}: a collaboratively built benchmark of 162 legal-reasoning tasks spanning contract, constitutional, administrative, criminal, and tort law, statutory interpretation, international law, and procedure. It is the most comprehensive public measure of legal reasoning in language models and exercises exactly the capabilities the domain adaptation targets.
|
||||
\item \textbf{CUAD}~\cite{cuad}: commercial contract clause extraction across 41 clause types (see the Contract Analysis section above).
|
||||
\item \textbf{ECHR judgment prediction}~\cite{echr}: legal judgment prediction over European Court of Human Rights cases.
|
||||
\item \textbf{ContractNLI}~\cite{contractnli}: document-level natural-language inference over contracts (e.g.\ obligation and entailment detection).
|
||||
\item \textbf{Multi-LexSum}~\cite{multilexsum}: abstractive summarization of civil-rights litigation records.
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Case Outcome Prediction}
|
||||
\subsection{Reporting Protocol}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Case outcome prediction accuracy by court level}
|
||||
\label{tab:outcome}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
Court & Cases & Accuracy & AUC \\
|
||||
\midrule
|
||||
U.S. Supreme Court & 8,412 & 78.4\% & 0.841 \\
|
||||
U.S. Circuit Courts & 142,841 & 74.8\% & 0.812 \\
|
||||
U.S. District Courts & 284,121 & 72.4\% & 0.784 \\
|
||||
State Supreme Courts & 48,412 & 71.8\% & 0.778 \\
|
||||
\midrule
|
||||
\textbf{Overall} & — & \textbf{74.2\%} & \textbf{0.804} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Legal NLP Tasks}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Legal NLP task results}
|
||||
\label{tab:legal_nlp}
|
||||
\begin{tabular}{llcc}
|
||||
\toprule
|
||||
Task & Dataset & Metric & Score \\
|
||||
\midrule
|
||||
Statute classification & EUR-Lex & Micro-F1 & 0.848 \\
|
||||
Legal judgment prediction & ECHR & Accuracy & 81.4\% \\
|
||||
Obligation detection & ContractNLI & F1 & 0.872 \\
|
||||
Legal argument mining & ECHRArguments & F1 & 0.814 \\
|
||||
Regulatory compliance & PrivacyQA & F1 & 0.864 \\
|
||||
Legal summarization & Multi-LexSum & ROUGE-L & 0.384 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
For each benchmark we recommend disclosing: the exact model version and quantization, the prompt template, the decoding parameters, the evaluation harness, and a side-by-side result for the unmodified Qwen3-8B base. We also recommend reporting calibration (how often a stated confidence matches empirical accuracy) and citation-verifiability rates, which matter more for safe legal use than raw task accuracy. Case-outcome \emph{prediction} is deliberately excluded from our recommended deployment evaluations: predicting how a court will rule is ethically fraught, prone to spurious correlation, and not a use we endorse for an augmentation tool.
|
||||
|
||||
\section{Legal Research Workflow}
|
||||
|
||||
@@ -237,24 +163,7 @@ Zen Legal generates complete legal memoranda following standard structure:
|
||||
\item \textbf{Conclusion}: Summary of findings and recommended next steps.
|
||||
\end{enumerate}
|
||||
|
||||
Attorney evaluation of generated memoranda (blind comparison vs. associate-drafted):
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Attorney evaluation of AI-generated legal memoranda (N=150)}
|
||||
\label{tab:memo_eval}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
Criterion & Zen Legal Score & Associate Avg \\
|
||||
\midrule
|
||||
Legal accuracy & 4.24 / 5 & 4.18 / 5 \\
|
||||
Citation completeness & 4.41 / 5 & 3.84 / 5 \\
|
||||
Analytical clarity & 4.08 / 5 & 4.28 / 5 \\
|
||||
Conclusion soundness & 4.18 / 5 & 4.22 / 5 \\
|
||||
\textbf{Overall} & \textbf{4.23 / 5} & \textbf{4.13 / 5} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
The appropriate evaluation of generated memoranda is a blind, attorney-graded comparison against human-drafted work product, scored on legal accuracy, citation completeness, analytical clarity, and conclusion soundness. We do not report scores from such a study in this report; any quality claim of this kind should be backed by a pre-registered evaluation with disclosed rubric, sample size, and inter-rater agreement. As a structured-generation tool, Zen Legal's intended advantage is in citation completeness and systematic IRAC coverage rather than in replacing attorney judgment.
|
||||
|
||||
\section{Professional Responsibility Compliance}
|
||||
|
||||
@@ -280,9 +189,10 @@ Per ABA Model Rule 5.3, attorneys using AI tools must supervise the AI's work co
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
Zen Legal demonstrates that rigorous citation grounding, jurisdictional awareness, and professional responsibility compliance can be integrated into a production legal AI system without sacrificing capability. The 81.4\% LegalBench score, 89.3\% CUAD clause extraction, and 0.923 contract clause F1 establish Zen Legal as state-of-the-art for AI-assisted legal research, with attorney evaluations confirming practical utility comparable to junior associate work product.
|
||||
Zen Legal demonstrates that rigorous citation grounding, jurisdictional awareness, and professional responsibility compliance can be layered onto an openly licensed general-purpose model through targeted domain fine-tuning. Built as an Apache-2.0 fine-tune of Qwen3-8B rather than a from-scratch system, it inherits a strong, transparently licensed foundation and adds the retrieval, verification, and guardrail machinery that legal use demands. We have intentionally avoided headline benchmark claims in this report; the honest measure of Zen Legal is a reproducible, attorney-reviewed evaluation against public benchmarks such as LegalBench and CUAD, reported side-by-side with the unmodified Qwen3-8B base so that the contribution of the legal adaptation is transparent.
|
||||
|
||||
\begin{thebibliography}{99}
|
||||
\bibitem{qwen3} Qwen Team, Alibaba Group. Qwen3 Technical Report. 2025. Models released under the Apache-2.0 license. \url{https://github.com/QwenLM/Qwen3}
|
||||
\bibitem{legalbench} Guha, N. et al. LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models. \textit{NeurIPS}, 2023.
|
||||
\bibitem{cuad} Hendrycks, D. et al. CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review. \textit{NeurIPS}, 2021.
|
||||
\bibitem{echr} Chalkidis, I. et al. Neural Legal Judgment Prediction in English. \textit{ACL}, 2019.
|
||||
|
||||
Binary file not shown.
+107
-220
@@ -13,8 +13,8 @@
|
||||
\definecolor{zenblue}{RGB}{41,121,255}
|
||||
\hypersetup{colorlinks=true,linkcolor=zenblue,urlcolor=zenblue,citecolor=zenblue}
|
||||
|
||||
\title{\textbf{Zen-Live: A Real-Time Conversational AI Model\\
|
||||
with Native Turn Management and Voice Interaction}\\[0.5em]
|
||||
\title{\textbf{Zen-Live: A Real-Time Multimodal Conversation Service\\
|
||||
built on Qwen3-Omni}\\[0.5em]
|
||||
\large Technical Whitepaper v2025.03}
|
||||
\author{Antje Worring, Zach Kelling \\ Zen LM Research Team\\
|
||||
\texttt{research@zenlm.org}\\
|
||||
@@ -25,7 +25,19 @@ with Native Turn Management and Voice Interaction}\\[0.5em]
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
Zen-Live is a 7 billion parameter language model purpose-built for real-time voice and video conversation, delivering P50 first-token latency of 67ms and P95 of 98ms through speculative decoding and early-exit inference mechanisms. Unlike conversational adaptations of general-purpose LLMs, Zen-Live integrates native voice activity detection (VAD), barge-in handling, and turn-taking management directly into the model architecture, eliminating the 80--150ms overhead of external pipeline components. Zen-Live achieves a word error rate (WER) of 4.2\% on conversational speech benchmarks and turn-taking accuracy of 94.3\% on the MultiTurn evaluation suite. This paper describes the model architecture, training methodology, latency optimization techniques, and conversational capability evaluations.
|
||||
Zen-Live is a real-time voice/video conversation \emph{service} built on Alibaba's
|
||||
open-source \textbf{Qwen3-Omni}~\cite{qwen3omni} (Apache~2.0)---not a 7-billion-parameter
|
||||
model we trained, and not a custom architecture with ``native VAD,'' ``early-exit,'' and
|
||||
a bespoke ``dialogue draft model.'' Qwen3-Omni is a natively end-to-end omni-modal
|
||||
model whose Thinker--Talker design understands text, audio, images, and video and
|
||||
generates speech in real time, with upstream-reported streaming first-packet latency of
|
||||
234\,ms (audio) and 547\,ms (video). Zen-Live's contribution is the serving layer: a
|
||||
WebRTC front end for control-room/voice-assistant use, turn-taking and barge-in handling
|
||||
at the application level, and integration with backend inference infrastructure. This
|
||||
paper describes that serving architecture and the provenance of the underlying model.
|
||||
Latency and accuracy figures are attributed to the upstream model rather than
|
||||
re-measured here; previously-reported benchmark tables have been removed as
|
||||
unsubstantiated.
|
||||
\end{abstract}
|
||||
|
||||
\tableofcontents
|
||||
@@ -45,231 +57,89 @@ Zen-Live integrates all conversational management functions into a single model
|
||||
\item \textbf{Streaming compatibility}: The model produces usable partial outputs from the first token, enabling streaming synthesis that begins before generation completes.
|
||||
\end{enumerate}
|
||||
|
||||
\subsection{Model Overview}
|
||||
\subsection{System Overview}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Zen-Live Model Specification}
|
||||
\caption{Zen-Live components and provenance. Zen-Live is a serving layer over the
|
||||
upstream model; it does not train a model.}
|
||||
\begin{tabular}{ll}
|
||||
\toprule
|
||||
\textbf{Parameter} & \textbf{Value} \\
|
||||
\textbf{Component} & \textbf{Value} \\
|
||||
\midrule
|
||||
Architecture & Streaming Transformer with early-exit \\
|
||||
Total Parameters & 7B \\
|
||||
Context Window & 16K tokens (audio + text interleaved) \\
|
||||
P50 First-Token Latency & 67ms \\
|
||||
P95 First-Token Latency & 98ms \\
|
||||
Turn-Taking Accuracy & 94.3\% \\
|
||||
Word Error Rate (conversational) & 4.2\% \\
|
||||
Supported Languages & 18 \\
|
||||
Underlying model & Qwen3-Omni~\cite{qwen3omni} (Alibaba, Apache 2.0) \\
|
||||
Modalities & Text, audio, image, video in; text + speech out \\
|
||||
Architecture & Thinker--Talker (per upstream) \\
|
||||
Streaming latency & 234\,ms audio / 547\,ms video (upstream-reported) \\
|
||||
Serving layer & WebRTC front end + turn/barge-in logic \\
|
||||
Backend & Hanzo inference infrastructure \\
|
||||
Version & v2025.03 \\
|
||||
Release Date & March 2025 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
The conversational capability---multimodal understanding and real-time speech
|
||||
generation---is provided by Qwen3-Omni. Zen-Live adds the serving and interaction
|
||||
layer described below.
|
||||
|
||||
\section{Architecture}
|
||||
|
||||
\subsection{Interleaved Audio-Text Representation}
|
||||
\subsection{Multimodal Understanding and Speech Generation (Upstream)}
|
||||
|
||||
Zen-Live processes audio and text in a unified interleaved sequence representation. Audio is encoded as discrete acoustic tokens using a quantized audio codec at 50Hz (one token per 20ms frame). Text tokens use the standard 100K vocabulary. A modality embedding distinguishes token types:
|
||||
The model-level capabilities are Qwen3-Omni's~\cite{qwen3omni}: it natively ingests
|
||||
text, audio, image, and video and generates text and speech in real time via the
|
||||
Thinker--Talker design (the Thinker handles multimodal reasoning; the Talker generates
|
||||
speech). Because understanding and generation are end-to-end in one model, a separate
|
||||
ASR$\to$LLM$\to$TTS cascade is not required. Zen-Live does not modify this model and
|
||||
does not add ``native VAD,'' ``early-exit,'' or a ``dialogue draft model''; earlier
|
||||
drafts described such bespoke architecture and specific tuned values (e.g.\ mean exit
|
||||
layer 14.3/32, draft acceptance $\beta=0.84$, 6\,ms VAD latency), none of which are
|
||||
real, and they have been removed.
|
||||
|
||||
\begin{equation}
|
||||
h_t = \text{Embed}(x_t) + \text{ModalityEmbed}(m_t) + \text{PosEmbed}(t)
|
||||
\end{equation}
|
||||
\subsection{Serving and Interaction Layer (Zen-Live)}
|
||||
|
||||
where $m_t \in \{\text{audio}, \text{text}\}$ is the modality label. This unified representation allows the model to condition text generation directly on the acoustic input without a separate transcription step.
|
||||
|
||||
\subsection{Early-Exit Inference}
|
||||
|
||||
Early-exit \cite{earlyexit} reduces latency by allowing the model to produce outputs from intermediate layers when confidence is high, avoiding the full forward pass through all layers. Zen-Live implements adaptive early-exit with per-layer confidence heads:
|
||||
|
||||
\begin{equation}
|
||||
\hat{y}_l = \text{ExitHead}_l(h_l), \quad c_l = \text{Confidence}_l(h_l)
|
||||
\end{equation}
|
||||
|
||||
If $c_l > \tau_{\text{exit}} = 0.92$, the model exits at layer $l$ and returns $\hat{y}_l$ without computing deeper layers. On conversational speech, the mean exit layer is 14.3 out of 32 total layers, reducing effective computation by 55\% compared to full forward passes.
|
||||
|
||||
The confidence threshold $\tau_{\text{exit}}$ is calibrated to maintain $<0.1\%$ accuracy degradation on held-out conversational benchmarks, verified by comparing early-exit outputs to full-pass outputs on 100K evaluation examples.
|
||||
|
||||
\subsection{Speculative Decoding with Dialogue Draft Model}
|
||||
|
||||
Zen-Live uses a 180M parameter dialogue-specialized draft model for speculative decoding. Unlike general-purpose draft models, the dialogue draft model is trained specifically on conversational turn patterns, giving higher acceptance rates for the stereotyped phrases common in conversational AI (greetings, acknowledgments, question frames).
|
||||
|
||||
\begin{equation}
|
||||
\beta_{\text{dialogue}} = 0.84, \quad \beta_{\text{general}} = 0.71
|
||||
\end{equation}
|
||||
|
||||
The higher acceptance rate yields a 3.1$\times$ effective speedup in conversational contexts, compared to 2.4$\times$ for a general draft model of equivalent size.
|
||||
|
||||
\subsection{Native Voice Activity Detection}
|
||||
|
||||
VAD is implemented as a lightweight auxiliary head on the audio encoder layers, producing a binary activity signal at the frame level (20ms resolution):
|
||||
|
||||
\begin{equation}
|
||||
\hat{v}_t = \sigma\left(W_{\text{VAD}} \cdot h_t^{(8)} + b_{\text{VAD}}\right)
|
||||
\end{equation}
|
||||
|
||||
where $h_t^{(8)}$ is the hidden state at encoder layer 8. The VAD head adds $<0.1\%$ parameter overhead and produces activity estimates with 6ms latency from the raw audio frame, compared to 25--40ms for external VAD models operating on decoded PCM.
|
||||
|
||||
\subsection{Turn-Taking Architecture}
|
||||
|
||||
Turn-taking prediction determines when it is appropriate to begin a response. Zen-Live implements turn-taking as a classification head that predicts one of three turn states on each VAD-active frame:
|
||||
Zen-Live's actual contribution is the serving wrapper:
|
||||
|
||||
\begin{itemize}
|
||||
\item \textbf{HOLD}: Speaker is mid-utterance; continue listening.
|
||||
\item \textbf{YIELD}: Speaker has completed their turn; begin response.
|
||||
\item \textbf{BACKCHANNEL}: Speaker expects acknowledgment; produce a brief backchannel.
|
||||
\item \textbf{Transport}: A WebRTC/streaming front end for control-room and
|
||||
voice-assistant clients (full-duplex, half-duplex, transcription-only, and
|
||||
speech-out modes).
|
||||
\item \textbf{Turn-taking and barge-in}: Application-level logic that decides when to
|
||||
start responding and how to stop gracefully when the user speaks over the
|
||||
assistant. This is conventional dialogue-management glue around the model's
|
||||
streaming input/output, not a trained turn-taking head. Voice-activity
|
||||
detection, where needed, uses a standard external VAD.
|
||||
\item \textbf{Backend integration}: Requests are served by backend inference
|
||||
infrastructure (Hanzo Node / Hanzo API) running the upstream model.
|
||||
\end{itemize}
|
||||
|
||||
The turn state predictor is conditioned on both acoustic features (prosodic contour, speaking rate, final phoneme duration) and semantic features (syntactic completeness, question vs. statement framing):
|
||||
We deliberately do not present equations for ``early-exit confidence heads,''
|
||||
``speculative decoding acceptance rates,'' or a ``turn-state classifier,'' because
|
||||
Zen-Live does not implement these as model components.
|
||||
|
||||
\begin{equation}
|
||||
\hat{t}_{\text{turn}} = \text{softmax}\left(W_t [h_{\text{acoustic}}; h_{\text{semantic}}]\right)
|
||||
\end{equation}
|
||||
\section{No Training}
|
||||
|
||||
This joint conditioning achieves 94.3\% turn-taking accuracy, compared to 87.1\% for acoustic-only models and 89.4\% for heuristic rule-based systems.
|
||||
|
||||
\subsection{Barge-In Handling}
|
||||
|
||||
Barge-in detection monitors the VAD signal during synthesis. When user speech is detected during model output, Zen-Live executes a graceful interruption protocol:
|
||||
|
||||
\begin{enumerate}
|
||||
\item Synthesis halts at the next sentence boundary (if within 200ms) or immediately.
|
||||
\item The partial response is retained in the context as an incomplete assistant turn.
|
||||
\item Listening mode resumes from the point of interruption.
|
||||
\item The model's next response acknowledges the interruption if semantically appropriate.
|
||||
\end{enumerate}
|
||||
|
||||
In a held-out evaluation of 500 barge-in events, Zen-Live achieved clean interruption in 97.3\% of cases (synthesis stopped within 1 sentence boundary of the interrupt signal), compared to 82.1\% for systems using external VAD with TCP notification delay.
|
||||
|
||||
\section{Training Methodology}
|
||||
|
||||
\subsection{Conversational Pre-Training Data}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Training Data Composition}
|
||||
\begin{tabular}{lrr}
|
||||
\toprule
|
||||
\textbf{Source} & \textbf{Hours} & \textbf{Fraction} \\
|
||||
\midrule
|
||||
Telephone conversations (licensed) & 90,000 & 34.6\% \\
|
||||
Podcast and interview audio & 60,000 & 23.1\% \\
|
||||
Customer service recordings & 40,000 & 15.4\% \\
|
||||
Video conference transcripts & 30,000 & 11.5\% \\
|
||||
Synthetic dialogue augmentation & 25,000 & 9.6\% \\
|
||||
Academic lectures and Q\&A & 15,000 & 5.8\% \\
|
||||
\midrule
|
||||
\textbf{Total} & \textbf{260,000} & \textbf{100\%} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Latency-Aware Training}
|
||||
|
||||
Zen-Live is trained with a latency-aware training objective that penalizes late exit points. In addition to the standard language modeling loss, an early-exit regularization term encourages the model to produce confident predictions from earlier layers:
|
||||
|
||||
\begin{equation}
|
||||
\mathcal{L}_{\text{latency}} = \mathcal{L}_{\text{LM}} + \lambda \sum_l w_l \mathcal{L}_{\text{KD}}(l)
|
||||
\end{equation}
|
||||
|
||||
where $w_l = (L - l + 1) / L$ weights later layers more heavily (penalizing failures to exit early), and $\mathcal{L}_{\text{KD}}(l)$ is the KL divergence between the full-model output distribution and the early-exit head at layer $l$.
|
||||
|
||||
\subsection{Turn-Taking Fine-Tuning}
|
||||
|
||||
Turn-taking capability is fine-tuned on a corpus of 50K annotated conversational turns, with human annotators labeling each audio frame with HOLD / YIELD / BACKCHANNEL ground truth. The annotation protocol includes inter-annotator agreement calibration; frames with $<80\%$ annotator agreement are excluded from training to avoid learning from ambiguous social cues.
|
||||
Zen-Live performs \emph{no} model training. There is no 260,000-hour conversational
|
||||
pre-training corpus, no ``latency-aware'' early-exit objective, and no turn-taking
|
||||
fine-tuning on annotated frames; earlier drafts described all three, and they have been
|
||||
removed as fabricated. The underlying model is used as released~\cite{qwen3omni};
|
||||
training and data details for it are documented upstream.
|
||||
|
||||
\section{Evaluation}
|
||||
|
||||
\subsection{Latency Benchmarks}
|
||||
We do not present first-token-latency, conversational-WER, turn-taking-accuracy,
|
||||
barge-in, or naturalness-MOS benchmark tables for Zen-Live. Earlier versions contained
|
||||
such tables with specific numbers (e.g.\ ``P95 98\,ms,'' ``WER 4.2\%,'' ``turn-taking
|
||||
accuracy 94.3\%''), attributed to a model and components that do not exist as
|
||||
described; they have been removed rather than presented as measured results.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{First-Token Latency Benchmarks (single A10G GPU, FP8)}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{P50} & \textbf{P90} & \textbf{P95} & \textbf{P99} \\
|
||||
\midrule
|
||||
Zen-Live (7B) & \textbf{67ms} & 88ms & \textbf{98ms} & 134ms \\
|
||||
Adapted 7B general model & 143ms & 191ms & 218ms & 287ms \\
|
||||
Adapted 3B general model & 89ms & 121ms & 143ms & 194ms \\
|
||||
External pipeline (7B + VAD) & 198ms & 261ms & 294ms & 381ms \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Speech Recognition Accuracy}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Word Error Rate on Conversational Benchmarks}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{Conversational WER} & \textbf{Noisy WER} & \textbf{Accented WER} \\
|
||||
\midrule
|
||||
Zen-Live (7B) & \textbf{4.2\%} & 8.7\% & 7.1\% \\
|
||||
Adapted 7B ASR model & 4.8\% & 9.3\% & 8.4\% \\
|
||||
External ASR pipeline & 5.1\% & 10.2\% & 9.7\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Turn-Taking Accuracy}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Turn-Taking Accuracy on MultiTurn Benchmark}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{YIELD F1} & \textbf{BACKCHANNEL F1} & \textbf{Overall Accuracy} \\
|
||||
\midrule
|
||||
Zen-Live (7B) & 95.1\% & 91.4\% & \textbf{94.3\%} \\
|
||||
Acoustic-only model & 88.3\% & 83.7\% & 87.1\% \\
|
||||
Rule-based heuristics & 90.2\% & 85.4\% & 89.4\% \\
|
||||
Human agreement baseline & 97.8\% & 94.1\% & 97.1\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Barge-In Performance}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Barge-In Handling Evaluation (500 events)}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Metric} & \textbf{Zen-Live} & \textbf{External VAD Pipeline} \\
|
||||
\midrule
|
||||
Clean interruption rate & 97.3\% & 82.1\% \\
|
||||
Mean synthesis overshoot & 0.4 sentences & 2.1 sentences \\
|
||||
Context retention after barge-in & 94.7\% & 81.3\% \\
|
||||
Recovery quality (MOS) & 4.1/5 & 3.4/5 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Naturalness}
|
||||
|
||||
Human evaluators rated extended 5-minute conversations with Zen-Live on naturalness and engagement:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Conversational Naturalness MOS (200 conversations)}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Dimension} & \textbf{Zen-Live} & \textbf{Adapted General Model} \\
|
||||
\midrule
|
||||
Response relevance & 4.3/5 & 4.1/5 \\
|
||||
Turn timing naturalness & 4.2/5 & 3.3/5 \\
|
||||
Backchannel appropriateness & 3.9/5 & 2.7/5 \\
|
||||
Interruption handling & 4.1/5 & 2.9/5 \\
|
||||
\textbf{Overall conversational MOS} & \textbf{4.1/5} & \textbf{3.3/5} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
The largest gains from Zen-Live's native turn management are in timing naturalness (+0.9 MOS) and backchannel appropriateness (+1.2 MOS), dimensions where external pipeline approaches are weakest.
|
||||
For grounded figures on the underlying model---latency, multimodal benchmark
|
||||
performance, and language coverage---readers should consult the Qwen3-Omni
|
||||
report~\cite{qwen3omni}, which reports streaming first-packet latency of 234\,ms (audio)
|
||||
/ 547\,ms (video) and support for 119 text, 19 speech-input, and 10 speech-output
|
||||
languages. Application-level metrics such as end-to-end conversational latency depend on
|
||||
the deployment (network, backend, transport) and should be measured on the target
|
||||
system.
|
||||
|
||||
\section{Deployment}
|
||||
|
||||
@@ -277,60 +147,77 @@ The largest gains from Zen-Live's native turn management are in timing naturalne
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Deployment Configurations}
|
||||
\begin{tabular}{llcl}
|
||||
\caption{Deployment targets. Concrete capacity and latency depend on the upstream
|
||||
model variant, precision, backend, and network, and should be measured per deployment.}
|
||||
\begin{tabular}{lll}
|
||||
\toprule
|
||||
\textbf{Config} & \textbf{Hardware} & \textbf{P95 Latency} & \textbf{Concurrent Sessions} \\
|
||||
\textbf{Target} & \textbf{Example Hardware} & \textbf{Notes} \\
|
||||
\midrule
|
||||
Cloud (FP8) & 1 $\times$ H100 80GB & 98ms & 96 sessions \\
|
||||
Edge server (FP8) & 1 $\times$ A10G 24GB & 113ms & 48 sessions \\
|
||||
On-device (INT4) & Apple M3 Max & 187ms & 4 sessions \\
|
||||
Cloud & Datacenter GPU & Highest concurrency \\
|
||||
Edge server & Single workstation GPU & Control-room deployment \\
|
||||
On-device & Apple Silicon (quantized) & Single-session assistant \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
We do not quote per-configuration P95 latency or concurrent-session counts here; the
|
||||
previous figures were tied to a fabricated 7B model. Measure on the target backend.
|
||||
|
||||
\subsection{API Integration}
|
||||
|
||||
Zen-Live exposes a WebRTC-compatible streaming API with the following modes:
|
||||
|
||||
\begin{itemize}
|
||||
\item \textbf{Full duplex}: Bidirectional audio streaming with native barge-in.
|
||||
\item \textbf{Full duplex}: Bidirectional audio streaming with application-level barge-in.
|
||||
\item \textbf{Half duplex}: Push-to-talk or VAD-gated turn alternation.
|
||||
\item \textbf{Transcription-only}: Audio in, text out, for accessibility or logging use cases.
|
||||
\item \textbf{Voice synthesis}: Text in, speech out, using the internal TTS component.
|
||||
\item \textbf{Voice synthesis}: Text in, speech out, using the upstream model's speech output.
|
||||
\end{itemize}
|
||||
|
||||
\section{Safety Considerations}
|
||||
|
||||
\subsection{Real-Time Content Filtering}
|
||||
\subsection{Content Filtering}
|
||||
|
||||
Zen-Live integrates a lightweight token-level safety filter (the Zen-Guard-Stream module) that operates with $<2$ms overhead per synthesis step, enabling safe streaming outputs without post-hoc filtering that would introduce latency spikes.
|
||||
Streaming output can be passed through an application-level safety filter before it
|
||||
reaches the client. This is a deployment-layer guard around the model rather than a
|
||||
property of the model weights; we do not claim a specific per-token overhead figure.
|
||||
|
||||
\subsection{Consent and Privacy}
|
||||
|
||||
All voice recordings processed by Zen-Live are subject to:
|
||||
Recommended deployment practices for voice data:
|
||||
|
||||
\begin{itemize}
|
||||
\item Explicit user consent before voice data processing.
|
||||
\item No persistent storage of audio beyond the session context window.
|
||||
\item Speaker anonymization in telemetry logs (voice embeddings are discarded; only aggregate quality metrics retained).
|
||||
\item Obtain explicit user consent before processing voice data.
|
||||
\item Avoid persistent storage of audio beyond the active session.
|
||||
\item Anonymize telemetry; do not retain raw audio or speaker embeddings beyond what
|
||||
is operationally required.
|
||||
\end{itemize}
|
||||
|
||||
\section{Related Work}
|
||||
|
||||
Real-time conversational AI has been developed through voice assistants \cite{siri,alexa}, dialogue systems \cite{dialog}, and recently through LLM-based voice interfaces \cite{voicellm}. Early-exit inference has been applied to NLP tasks \cite{earlyexit}. Speculative decoding for conversational AI is explored in \cite{spectalk}. Zen-Live unifies these advances into a model architecture trained end-to-end for conversational requirements.
|
||||
Real-time conversational AI has been developed through voice assistants
|
||||
\cite{siri,alexa}, dialogue systems \cite{dialog}, and LLM-based voice interfaces such
|
||||
as Moshi~\cite{voicellm}. Natively end-to-end omni-modal models such as
|
||||
Qwen3-Omni~\cite{qwen3omni} now provide real-time multimodal understanding and speech
|
||||
generation in a single model. Zen-Live is not a new model of this kind; it is a serving
|
||||
layer that exposes such an upstream model for live conversational use.
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
Zen-Live achieves real-time voice conversation with P95 first-token latency of 98ms, turn-taking accuracy of 94.3\%, and WER of 4.2\% by integrating VAD, turn management, and language generation into a single 7B parameter model. The architecture eliminates the coordination overhead of cascaded pipeline approaches and enables more natural conversational behavior, particularly in barge-in handling and backchannel generation. Zen-Live is deployable on a single GPU for cloud applications and on Apple Silicon for on-device voice assistant use cases.
|
||||
Zen-Live is a real-time conversation \emph{service} built on Alibaba's openly-licensed
|
||||
\textbf{Qwen3-Omni} model, not a 7B model we trained and not a custom architecture with
|
||||
native VAD, early-exit, or a dialogue draft model. Its contribution is the serving and
|
||||
interaction layer: WebRTC transport, application-level turn-taking and barge-in, and
|
||||
backend integration. The conversational capability is upstream Qwen3-Omni's and is
|
||||
attributed as such; the previously-reported latency, WER, and turn-taking numbers were
|
||||
not measured for this system and have been removed.
|
||||
|
||||
\begin{thebibliography}{9}
|
||||
\bibitem{earlyexit} Schwartz, R. et al. (2020). The Right Tool for the Job: Matching Model and Instance Complexities. ACL 2020.
|
||||
\bibitem{qwen3omni} Qwen Team, Alibaba Cloud. (2025). Qwen3-Omni Technical Report. \textit{arXiv:2509.17765}. Code/weights: \url{https://github.com/QwenLM/Qwen3-Omni} (Apache 2.0).
|
||||
\bibitem{siri} Shum, H.-Y. et al. (2018). From Eliza to XiaoIce: Challenges and Opportunities with Social Chatbots. Frontiers of IT and EE.
|
||||
\bibitem{alexa} Alexa AI Team. (2018). The Alexa Meaning Representation Language. NAACL 2018.
|
||||
\bibitem{dialog} Gao, J. et al. (2019). Neural Approaches to Conversational AI. Foundations and Trends in IR.
|
||||
\bibitem{voicellm} Défossez, A. et al. (2024). Moshi: a speech-text foundation model for real-time dialogue. arXiv:2410.00037.
|
||||
\bibitem{spectalk} Medina, G. et al. (2024). Speculative Decoding for Conversational AI Systems. arXiv preprint.
|
||||
\end{thebibliography}
|
||||
|
||||
\end{document}
|
||||
|
||||
Binary file not shown.
@@ -1,422 +0,0 @@
|
||||
\documentclass[11pt,a4paper]{article}
|
||||
\usepackage[utf8]{inputenc}
|
||||
\usepackage[T1]{fontenc}
|
||||
\usepackage{amsmath,amsfonts,amssymb}
|
||||
\usepackage{graphicx}
|
||||
\usepackage{hyperref}
|
||||
\usepackage{listings}
|
||||
\usepackage{color}
|
||||
\usepackage{booktabs}
|
||||
\usepackage{float}
|
||||
\usepackage{geometry}
|
||||
\geometry{margin=1in}
|
||||
\definecolor{zenblue}{RGB}{41,121,255}
|
||||
\definecolor{zengreen}{RGB}{52,199,89}
|
||||
\hypersetup{colorlinks=true,linkcolor=zenblue,urlcolor=zenblue,citecolor=zenblue}
|
||||
|
||||
\title{\textbf{Zen-Max: Maximum Capability via Mixture-of-Experts Architecture}\\
|
||||
\large Technical Report v2025.03}
|
||||
\author{Antje Worring, Zach Kelling \\ Zen LM Research Team\\
|
||||
\texttt{research@zenlm.org}}
|
||||
\date{March 2025}
|
||||
|
||||
\begin{document}
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
We present \textbf{Zen-Max}, the highest-capability model in the Zen family, implementing the
|
||||
Zen MoDE (Mixture of Distilled Experts) architecture at maximum scale: 480 billion total
|
||||
parameters activating 48 billion per forward pass through a sparse mixture-of-experts (MoE)
|
||||
routing mechanism. Trained on 5.5 trillion tokens with a 200K token context window, Zen-Max
|
||||
establishes new performance records across the Zen family on reasoning, scientific understanding,
|
||||
mathematics, and code generation benchmarks: MMLU 92.7\%, MATH 89.3\%, HumanEval 94.8\%,
|
||||
GPQA 78.2\%, and MMMU 87.6\%. The MoE design achieves dense-model reasoning quality at
|
||||
substantially lower per-token inference cost, making frontier capability accessible for research
|
||||
and enterprise deployment at manageable compute budgets.
|
||||
\end{abstract}
|
||||
|
||||
\tableofcontents
|
||||
\newpage
|
||||
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\section{Introduction}
|
||||
|
||||
Scaling language model capability to the frontier has historically required proportional
|
||||
increases in inference compute, creating tension between maximizing quality and deployment
|
||||
cost. Mixture-of-experts (MoE) architectures \cite{shazeer2017outrageously, fedus2022switch}
|
||||
address this tension by decoupling total parameter count from per-token active parameters:
|
||||
a large expert pool provides rich representational capacity while sparse routing activates
|
||||
only a relevant subset for each token.
|
||||
|
||||
Zen-Max realizes this principle at our largest scale to date: 480B total parameters, 48B
|
||||
active per token (10\% activation rate), with 128 experts per MoE layer and top-8 routing.
|
||||
This design delivers performance comparable to much larger dense models while constraining
|
||||
inference FLOPs to approximately 48B-equivalent. We make the following contributions:
|
||||
|
||||
\begin{itemize}
|
||||
\item The Zen MoDE architecture at 480B/48B scale, with 128 experts, fine-grained expert
|
||||
segmentation, and auxiliary-loss-free load balancing via a bias-adjustment mechanism.
|
||||
\item A 200K token context window enabled by extended RoPE and a chunked attention
|
||||
implementation optimized for memory efficiency.
|
||||
\item RLVR post-training on an extended suite of verifiable domains including formal
|
||||
mathematics, code, logic puzzles, scientific question answering, and multi-step
|
||||
tool-use trajectories.
|
||||
\item State-of-the-art benchmark performance: MMLU 92.7\%, MATH 89.3\%, HumanEval 94.8\%,
|
||||
GPQA Diamond 78.2\%, MMMU 87.6\%.
|
||||
\end{itemize}
|
||||
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\section{Architecture}
|
||||
|
||||
\subsection{Overall Structure}
|
||||
|
||||
Zen-Max is a decoder-only transformer with alternating dense attention layers and MoE
|
||||
feed-forward layers. The 96-layer network has MoE FFN in 80 of the 96 layers, with 16 dense
|
||||
layers (every 6th layer) to maintain stable global representations across the routing sparsity.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Zen-Max architecture hyperparameters.}
|
||||
\label{tab:arch}
|
||||
\begin{tabular}{lc}
|
||||
\toprule
|
||||
\textbf{Hyperparameter} & \textbf{Value} \\
|
||||
\midrule
|
||||
Parameters (total) & 480B \\
|
||||
Parameters (active per token) & 48B \\
|
||||
Layers (total) & 96 \\
|
||||
Dense layers & 16 (every 6th) \\
|
||||
MoE layers & 80 \\
|
||||
Experts per MoE layer & 128 \\
|
||||
Experts activated per token & 8 (top-8 routing) \\
|
||||
Shared expert per MoE layer & 2 (always active) \\
|
||||
Attention heads & 128 \\
|
||||
KV heads (GQA) & 8 \\
|
||||
Hidden dimension & 7{,}168 \\
|
||||
Expert FFN dimension & 2{,}048 (per expert) \\
|
||||
Vocabulary size & 151{,}936 \\
|
||||
Context length (training) & 200{,}000 \\
|
||||
Position encoding & RoPE ($\theta = 10{,}000{,}000$) \\
|
||||
Activation & SiLU \\
|
||||
Normalization & RMSNorm \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Fine-Grained Expert Architecture}
|
||||
|
||||
Zen-Max uses a fine-grained expert decomposition where each MoE layer contains 128 small
|
||||
experts rather than fewer large experts. Each expert $e_i$ applies a SwiGLU FFN with
|
||||
intermediate dimension 2,048, smaller than the 7,168 hidden dimension:
|
||||
|
||||
\begin{equation}
|
||||
\text{Expert}_i(x) = \left(\text{SiLU}(xW_{\text{gate},i}) \odot xW_{\text{up},i}\right) W_{\text{down},i}
|
||||
\end{equation}
|
||||
|
||||
The top-8 routing selects the 8 highest-scoring experts per token. Additionally, 2 shared
|
||||
experts per layer are always activated and added to the routed experts' outputs. This
|
||||
``shared expert'' pattern ensures that universal representations (stopwords, punctuation,
|
||||
formatting) are handled without burdening the routing mechanism.
|
||||
|
||||
\subsection{Auxiliary-Loss-Free Load Balancing}
|
||||
|
||||
Expert load imbalance is a critical failure mode in MoE training, causing some experts to
|
||||
receive disproportionate tokens (routing collapse) while others atrophy. Rather than adding
|
||||
an auxiliary load balancing loss that trades task performance for balance, Zen-Max employs
|
||||
a bias-adjustment mechanism \cite{deepseekmoe2024}:
|
||||
|
||||
\begin{equation}
|
||||
s_{i,j} = \text{softmax}(x_i W_g)_j + b_j
|
||||
\end{equation}
|
||||
|
||||
where $b_j$ is a per-expert bias updated after each training step based on the deviation of
|
||||
expert load from the target. Experts that are overloaded receive a negative bias; underloaded
|
||||
experts receive a positive bias. This maintains load balance without introducing conflicting
|
||||
gradient signals.
|
||||
|
||||
\subsection{200K Context Window}
|
||||
|
||||
A 200K token context window requires careful management of attention memory. Zen-Max combines:
|
||||
|
||||
\begin{enumerate}
|
||||
\item Extended RoPE with $\theta = 10{,}000{,}000$ providing position distinguishability
|
||||
up to 200K tokens.
|
||||
\item Chunked attention with chunk size $c = 4{,}096$ tokens, computing attention in chunks
|
||||
to avoid materializing the full $200K \times 200K$ attention matrix.
|
||||
\item A 3-stage training curriculum: 8K $\to$ 64K $\to$ 200K, each stage consuming
|
||||
20B, 10B, and 5B tokens respectively.
|
||||
\end{enumerate}
|
||||
|
||||
\subsection{Multi-Head Latent Attention}
|
||||
|
||||
For the attention sublayer, Zen-Max uses Multi-Head Latent Attention (MLA) \cite{liu2024deepseekv2}
|
||||
which compresses the KV cache via low-rank projections:
|
||||
|
||||
\begin{align}
|
||||
c^{KV} &= x W^{DKV} \in \mathbb{R}^{d_c} \\
|
||||
K &= c^{KV} W^{UK} \in \mathbb{R}^{n_h \times d_h} \\
|
||||
V &= c^{KV} W^{UV} \in \mathbb{R}^{n_h \times d_h}
|
||||
\end{align}
|
||||
|
||||
where $d_c = 512 \ll n_h \cdot d_h$. Only $c^{KV}$ is cached (512 floats per layer per
|
||||
token), reducing the KV cache by approximately 26$\times$ relative to full MHA at equivalent
|
||||
head count. For a 200K-token sequence at 96 layers in BF16: $96 \times 200{,}000 \times 512
|
||||
\times 2 \approx 19.7$ GB, compared to $\sim$510 GB for full MHA.
|
||||
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\section{Training Methodology}
|
||||
|
||||
\subsection{Pretraining at Scale}
|
||||
|
||||
Zen-Max pretrains on 5.5T tokens across the same domain distribution as Zen-Pro (see
|
||||
Table~\ref{tab:data}), with an increased allocation to mathematical and scientific content.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Zen-Max pretraining data (5.5T tokens total).}
|
||||
\label{tab:data}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Domain} & \textbf{Tokens (B)} & \textbf{Fraction} \\
|
||||
\midrule
|
||||
Web text (quality-filtered) & 2{,}035 & 37.0\% \\
|
||||
Code (all languages) & 825 & 15.0\% \\
|
||||
Books and long-form & 770 & 14.0\% \\
|
||||
Scientific articles & 605 & 11.0\% \\
|
||||
Mathematics (formal+informal) & 550 & 10.0\% \\
|
||||
Multilingual web & 385 & 7.0\% \\
|
||||
Curated reasoning chains & 330 & 6.0\% \\
|
||||
\midrule
|
||||
Total & 5{,}500 & 100.0\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
Training infrastructure: 2048 H100-SXM5 GPUs, expert parallelism $\times$16, tensor
|
||||
parallelism $\times$8, pipeline parallelism $\times$16, data parallelism $\times$8. Total
|
||||
training compute approximately $3.2 \times 10^{24}$ FLOPs.
|
||||
|
||||
\subsection{MoE-Specific Training Stability}
|
||||
|
||||
MoE models are prone to instability from routing discontinuities. Zen-Max applies:
|
||||
|
||||
\begin{itemize}
|
||||
\item Gradient clipping at norm 1.0 with monitoring of per-expert gradient magnitudes.
|
||||
\item Expert dropout: randomly zeroing 1 expert per token during training for robustness.
|
||||
\item Router z-loss \cite{zoph2022stmoe}: $\mathcal{L}_z = \frac{1}{B}\sum_{x} (\log \sum_j e^{x_j})^2$
|
||||
applied at coefficient $10^{-3}$ to prevent logit explosion in the router.
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Post-Training: Extended RLVR}
|
||||
|
||||
Zen-Max applies RLVR on an expanded set of verifiable tasks beyond Zen-Pro's three domains:
|
||||
|
||||
\begin{itemize}
|
||||
\item Mathematics (AMC/AIME/Olympiad, formal Lean proofs)
|
||||
\item Code (Python, Rust, Go, C++ on competitive programming problems)
|
||||
\item Logic (first-order logic satisfiability, constraint satisfaction)
|
||||
\item Science QA (verified against authoritative databases: PubMed, arXiv theorem checks)
|
||||
\item Multi-step tool use (API call sequences verified by environment simulation)
|
||||
\end{itemize}
|
||||
|
||||
GRPO sampling uses $G=16$ rollouts per prompt (doubled from Zen-Pro) to improve advantage
|
||||
estimation quality at the 480B parameter scale.
|
||||
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\section{Evaluation}
|
||||
|
||||
\subsection{Core Benchmarks}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Zen-Max benchmark results versus frontier models.}
|
||||
\label{tab:benchmarks}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
\textbf{Benchmark} & \textbf{Zen-Max (480B)} & \textbf{Zen-Pro (72B)} & \textbf{Comp.\ A (large)} & \textbf{Comp.\ B (MoE)} \\
|
||||
\midrule
|
||||
MMLU (5-shot) & \textbf{92.7} & 89.2 & 91.8 & 90.4 \\
|
||||
MMLU-Pro & \textbf{78.3} & 72.4 & 77.1 & 75.6 \\
|
||||
ARC-Challenge & 78.4 & 72.8 & 79.1 & 77.3 \\
|
||||
HellaSwag & 91.2 & 88.4 & 90.8 & 89.6 \\
|
||||
\midrule
|
||||
MATH (4-shot, CoT) & \textbf{89.3} & 84.1 & 87.9 & 86.4 \\
|
||||
AIME 2024 (pass@1) & \textbf{66.7} & 53.3 & 63.3 & 60.0 \\
|
||||
GSM8K & \textbf{96.2} & 93.4 & 95.8 & 94.7 \\
|
||||
GPQA Diamond & \textbf{78.2} & 71.8 & 76.4 & 74.1 \\
|
||||
\midrule
|
||||
HumanEval (pass@1) & \textbf{94.8} & 91.3 & 93.2 & 91.8 \\
|
||||
MBPP (pass@1) & \textbf{88.4} & 84.6 & 86.9 & 85.3 \\
|
||||
SWE-bench Verified & \textbf{52.1} & 38.7 & 49.3 & 46.8 \\
|
||||
\midrule
|
||||
MMMU (val) & \textbf{87.6} & -- & 86.1 & 84.3 \\
|
||||
MT-Bench & \textbf{9.4} & 9.1 & 9.3 & 9.2 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Mathematical Olympiad Performance}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Performance on mathematical competition benchmarks.}
|
||||
\label{tab:math}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Benchmark} & \textbf{Zen-Max} & \textbf{Prior Best} \\
|
||||
\midrule
|
||||
AIME 2024 (pass@1) & 66.7\% & 63.3\% \\
|
||||
AIME 2024 (pass@32) & 93.3\% & 90.0\% \\
|
||||
AIME 2023 & 71.4\% & 68.6\% \\
|
||||
AMC 10 & 94.2\% & 92.8\% \\
|
||||
AMC 12 & 91.8\% & 89.4\% \\
|
||||
MATH Level 5 only & 83.1\% & 80.7\% \\
|
||||
OlympiadBench & 67.4\% & 64.2\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Long-Context Performance}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{RULER scores and needle-in-a-haystack (NIAH) accuracy at increasing context lengths.}
|
||||
\label{tab:longctx}
|
||||
\begin{tabular}{lcccccc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{8K} & \textbf{32K} & \textbf{64K} & \textbf{128K} & \textbf{200K} \\
|
||||
\midrule
|
||||
Zen-Max (480B) & 97.1 & 95.3 & 92.7 & 88.4 & 82.1 \\
|
||||
Zen-Pro (72B) & 96.2 & 92.1 & 88.3 & 82.7 & --- \\
|
||||
Competitor A & 96.8 & 93.4 & 87.1 & 71.3 & 54.8 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Inference Cost Analysis}
|
||||
|
||||
The MoE sparsity of Zen-Max provides substantial cost advantages over dense models at
|
||||
comparable quality.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Comparative inference cost (A100-80GB cluster, BF16).}
|
||||
\label{tab:inference_cost}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{Active Params} & \textbf{GPUs (TP8)} & \textbf{tok/s (batch=1)} & \textbf{Relative cost} \\
|
||||
\midrule
|
||||
Zen-Max (480B MoE) & 48B & 8$\times$ A100 & 28 & 1.0$\times$ \\
|
||||
Dense-equivalent & 72B & 8$\times$ A100 & 18 & 1.6$\times$ \\
|
||||
Dense-equivalent & 120B & 16$\times$ A100 & 11 & 4.2$\times$ \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\section{Expert Specialization Analysis}
|
||||
|
||||
We analyze expert routing patterns to understand emergent specialization. After training,
|
||||
we feed 1M tokens from each domain and measure the Kullback-Leibler divergence between
|
||||
per-domain routing distributions and the uniform distribution:
|
||||
|
||||
\begin{equation}
|
||||
\text{Specialization}(e, d) = D_{\text{KL}}(P(e \mid x \in d) \| P(e))
|
||||
\end{equation}
|
||||
|
||||
We find that:
|
||||
|
||||
\begin{itemize}
|
||||
\item Mathematical tokens activate a consistent set of 12--18 experts (out of 128) at
|
||||
rates 4$\times$ above average, suggesting strong specialization.
|
||||
\item Code tokens show stronger expert specialization by programming language than by
|
||||
task type (generation vs.\ debugging), with Python and C$+$$+$ activating distinct
|
||||
expert subsets.
|
||||
\item Natural language tokens show weaker specialization than code or math, consistent
|
||||
with the hypothesis that diverse language modeling requires broader expert coverage.
|
||||
\end{itemize}
|
||||
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\section{Related Work}
|
||||
|
||||
Sparse mixture-of-experts for language models was introduced in Sparsely-Gated MoE
|
||||
\cite{shazeer2017outrageously} and scaled in Switch Transformer \cite{fedus2022switch}
|
||||
and GLaM \cite{du2022glam}. Load balancing has been addressed via auxiliary losses
|
||||
\cite{fedus2022switch}, expert capacity buffers \cite{lepikhin2020gshard}, and more
|
||||
recently bias-adjustment mechanisms \cite{deepseekmoe2024}. Multi-Head Latent Attention
|
||||
for KV cache compression follows \cite{liu2024deepseekv2}.
|
||||
|
||||
Long-context MoE models face additional challenges in routing consistency across long
|
||||
sequences. Our chunked attention and staged context extension follow best practices
|
||||
established in recent long-context scaling work \cite{peng2023yarn, hsieh2024ruler}.
|
||||
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\section{Limitations}
|
||||
|
||||
Expert routing introduces non-determinism that complicates reproducibility. Load balancing
|
||||
remains imperfect: a small fraction of experts (3--5\%) consistently receives below-average
|
||||
routing probability despite bias adjustment. Inference requires communication-heavy expert
|
||||
parallelism, increasing latency on multi-node deployments. The 200K context window, while
|
||||
large, degrades for generation tasks beyond 64K output tokens. MoE training instability
|
||||
necessitates careful monitoring and checkpoint selection.
|
||||
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\section{Conclusion}
|
||||
|
||||
Zen-Max demonstrates that the Zen MoDE architecture scales favorably to 480B/48B active
|
||||
parameters, achieving the highest benchmark scores in the Zen family: MMLU 92.7\%, MATH
|
||||
89.3\%, HumanEval 94.8\%, GPQA Diamond 78.2\%, and MMMU 87.6\%. The sparse MoE design
|
||||
delivers these results at the per-token FLOPs budget of a 48B dense model, providing a
|
||||
compelling capability/cost trade-off for research and enterprise deployments. Zen-Max
|
||||
serves as the primary model for demanding scientific, mathematical, and multi-step reasoning
|
||||
workloads in the Zen family ecosystem.
|
||||
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\begin{thebibliography}{99}
|
||||
|
||||
\bibitem{shazeer2017outrageously}
|
||||
N.~Shazeer et al., ``Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts
|
||||
Layer,'' \textit{ICLR}, 2017.
|
||||
|
||||
\bibitem{fedus2022switch}
|
||||
W.~Fedus, B.~Zoph, and N.~Shazeer, ``Switch Transformers: Scaling to Trillion Parameter Models
|
||||
with Simple and Efficient Sparsity,'' \textit{JMLR}, 2022.
|
||||
|
||||
\bibitem{du2022glam}
|
||||
N.~Du et al., ``GLaM: Efficient Scaling of Language Models with Mixture-of-Experts,''
|
||||
\textit{ICML}, 2022.
|
||||
|
||||
\bibitem{lepikhin2020gshard}
|
||||
D.~Lepikhin et al., ``GShard: Scaling Giant Models with Conditional Computation and Automatic
|
||||
Sharding,'' \textit{ICLR}, 2021.
|
||||
|
||||
\bibitem{deepseekmoe2024}
|
||||
DeepSeek AI, ``DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language
|
||||
Model,'' \textit{arXiv:2405.04434}, 2024.
|
||||
|
||||
\bibitem{liu2024deepseekv2}
|
||||
A.~Liu et al., ``DeepSeek-V2,'' \textit{arXiv:2405.04434}, 2024.
|
||||
|
||||
\bibitem{zoph2022stmoe}
|
||||
B.~Zoph et al., ``ST-MoE: Designing Stable and Transferable Sparse Expert Models,''
|
||||
\textit{arXiv:2202.08906}, 2022.
|
||||
|
||||
\bibitem{peng2023yarn}
|
||||
B.~Peng et al., ``YaRN: Efficient Context Window Extension,'' \textit{arXiv:2309.00071}, 2023.
|
||||
|
||||
\bibitem{hsieh2024ruler}
|
||||
C.-Y.~Hsieh et al., ``RULER: What's the Real Context Size of Your Long-Context Language Models?''
|
||||
\textit{arXiv:2404.06654}, 2024.
|
||||
|
||||
\bibitem{shao2024deepseekmath}
|
||||
Z.~Shao et al., ``DeepSeekMath,'' \textit{arXiv:2402.03300}, 2024.
|
||||
|
||||
\bibitem{wei2022emergent}
|
||||
J.~Wei et al., ``Emergent Abilities of Large Language Models,'' \textit{TMLR}, 2022.
|
||||
|
||||
\bibitem{lightman2023lets}
|
||||
H.~Lightman et al., ``Let's Verify Step by Step,'' \textit{arXiv:2305.20050}, 2023.
|
||||
|
||||
\end{thebibliography}
|
||||
|
||||
\end{document}
|
||||
Binary file not shown.
+45
-206
@@ -113,7 +113,7 @@
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
We present \textbf{Zen-Medical}, a clinical AI system specialized for multi-modal diagnostic reasoning, differential diagnosis generation, drug interaction checking, and clinical decision support. Zen-Medical is built on the Zen-72B foundation model with domain-specific adaptations that enable it to integrate patient histories, laboratory results, medical imaging, and clinical guidelines into structured diagnostic reasoning. The system introduces three key innovations: (1) a \textbf{Clinical Reasoning Chain (CRC)} framework that mirrors the diagnostic workflow of experienced physicians---chief complaint analysis, history synthesis, differential diagnosis generation, investigation planning, and management recommendation---with explicit uncertainty quantification at each stage, (2) a \textbf{Multi-Modal Clinical Encoder (MMCE)} that jointly processes chest X-rays, CT scans, dermatoscopic images, ECG tracings, and pathology slides alongside textual clinical data, and (3) a \textbf{Drug Interaction Knowledge Graph (DIKG)} containing 2.4 million drug-drug and drug-condition interactions sourced from FDA labels, DrugBank, and peer-reviewed literature, queryable in real-time during clinical reasoning. On the MedQA (USMLE) benchmark, Zen-Medical achieves 92.4\% accuracy, surpassing both Med-PaLM 2 (86.5\%) and GPT-4-Medical (90.2\%). On PubMedQA, it achieves 81.8\% accuracy with calibrated confidence estimates (Expected Calibration Error = 0.032). On the CheXpert chest X-ray benchmark, the multi-modal variant achieves 0.941 mean AUC across 5 pathologies, competitive with specialist radiology models. Critically, Zen-Medical is designed for HIPAA-compliant deployment with on-premise inference, audit logging, and explicit uncertainty communication that flags cases requiring human physician review. All outputs include structured citations to clinical evidence and are intended as decision support tools, not autonomous diagnostic systems.
|
||||
We present \textbf{Zen-Medical}, a clinical AI system specialized for multi-modal diagnostic reasoning, differential diagnosis generation, drug interaction checking, and clinical decision support. Zen-Medical is \emph{not} a model trained from scratch: it is a supervised fine-tune of the openly licensed \textbf{Qwen3-8B} base model (developed by Alibaba and released under the Apache-2.0 license), adapted to the clinical domain with components that integrate patient histories, laboratory results, medical imaging, and clinical guidelines into structured diagnostic reasoning. The system introduces three engineering components layered on this base: (1) a \textbf{Clinical Reasoning Chain (CRC)} framework that mirrors the diagnostic workflow of experienced physicians---chief complaint analysis, history synthesis, differential diagnosis generation, investigation planning, and management recommendation---with explicit uncertainty quantification at each stage, (2) a \textbf{Multi-Modal Clinical Encoder (MMCE)} that pairs the fine-tuned language model with modality-specific medical-image encoders for chest X-rays, CT scans, dermatoscopic images, ECG tracings, and pathology slides alongside textual clinical data, and (3) a \textbf{Drug Interaction Knowledge Graph (DIKG)} that aggregates drug-drug and drug-condition interactions from FDA labels, DrugBank, and peer-reviewed literature, queryable in real time during clinical reasoning. We deliberately do not report headline accuracy figures or comparisons against other systems in this report; instead we describe the clinical adaptation qualitatively and name the public benchmarks (MedQA/USMLE, PubMedQA, MedMCQA, CheXpert, and others) against which clinical language models should be evaluated under transparent, reproducible protocols. Critically, Zen-Medical is designed for HIPAA-compliant deployment with on-premise inference, audit logging, and explicit uncertainty communication that flags cases requiring human physician review. All outputs include structured citations to clinical evidence and are intended as decision support tools, not autonomous diagnostic systems.
|
||||
\end{abstract}
|
||||
|
||||
\vspace{0.5em}
|
||||
@@ -350,28 +350,29 @@ Drug Data & DrugBank + FDA + FAERS & 2.4M interactions & Graph \\
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\paragraph{Clinical Reasoning Traces.} We curate 120K high-quality diagnostic reasoning traces from three sources: (1) published clinical case reports with expert commentary (42K), (2) de-identified physician reasoning transcripts from teaching hospitals (38K, IRB-approved), and (3) synthetic traces generated by the Zen-72B model and validated by board-certified physicians (40K). Each trace follows the CRC format with explicit uncertainty annotations.
|
||||
\paragraph{Clinical Reasoning Traces.} We curate 120K high-quality diagnostic reasoning traces from three sources: (1) published clinical case reports with expert commentary (42K), (2) de-identified physician reasoning transcripts from teaching hospitals (38K, IRB-approved), and (3) synthetic traces generated by the Qwen3-8B base model and validated by board-certified physicians (40K). Each trace follows the CRC format with explicit uncertainty annotations.
|
||||
|
||||
\paragraph{Data De-identification.} All patient-derived data undergoes rigorous de-identification following Safe Harbor guidelines (45 CFR 164.514): removal of 18 HIPAA identifiers, date shifting, and expert review of a random 5\% sample to verify de-identification completeness.
|
||||
|
||||
\subsection{Training Pipeline}
|
||||
|
||||
\paragraph{Stage 1: Domain Pre-training (2 weeks, 64 A100 GPUs).} The Zen-72B base model undergoes continued pre-training on the medical text corpus (PubMed, textbooks, guidelines) for 50B tokens. This stage adapts the model's vocabulary and knowledge to the medical domain.
|
||||
The pipeline begins from the openly licensed \textbf{Qwen3-8B} base model~\citep{qwen3} (Alibaba, Apache-2.0) and applies a sequence of domain-adaptation stages. Because the base is an 8B-parameter dense model rather than a from-scratch or multi-tens-of-billions-parameter model, each stage is comparatively lightweight and runs on a small GPU cluster; we omit precise wall-clock and device-count figures here as they depend on the available hardware.
|
||||
|
||||
\paragraph{Stage 2: Clinical SFT (1 week, 64 A100 GPUs).} The model is fine-tuned on the medical QA pairs and clinical reasoning traces using the SFT objective with loss masking on instruction tokens:
|
||||
\paragraph{Stage 1: Domain-adaptive pre-training.} The Qwen3-8B base model undergoes continued pre-training on the medical text corpus (PubMed, textbooks, guidelines). This stage adapts the model's distribution toward medical language and terminology while preserving the general capabilities inherited from the base.
|
||||
|
||||
\paragraph{Stage 2: Clinical SFT.} The model is fine-tuned on the medical QA pairs and clinical reasoning traces using the supervised fine-tuning objective with loss masking on instruction tokens:
|
||||
\begin{equation}
|
||||
\mathcal{L}_{\text{SFT}} = -\frac{1}{|\mathcal{R}|}\sum_{t \in \mathcal{R}} \log p_\theta(x_t | x_{<t})
|
||||
\end{equation}
|
||||
We train for 3 epochs with learning rate $2 \times 10^{-5}$ and batch size 128.
|
||||
|
||||
\paragraph{Stage 3: Multi-Modal Training (1 week, 32 A100 GPUs).} The image encoders and cross-modal adapters are trained while the language model backbone is frozen (except for the cross-attention layers). This stage uses paired (image, clinical text, diagnosis) data from CheXpert, MIMIC-CXR, ISIC, and PTB-XL.
|
||||
\paragraph{Stage 3: Multi-Modal Training.} The image encoders and cross-modal adapters are trained while the language model backbone is frozen (except for the cross-attention layers). This stage uses paired (image, clinical text, diagnosis) data from CheXpert, MIMIC-CXR, ISIC, and PTB-XL.
|
||||
|
||||
\paragraph{Stage 4: Alignment (3 days, 32 A100 GPUs).} DPO alignment using 50K physician-rated preference pairs, where physicians select the more clinically appropriate response considering accuracy, safety, uncertainty communication, and reasoning quality:
|
||||
\paragraph{Stage 4: Alignment.} DPO alignment using physician-rated preference pairs, where physicians select the more clinically appropriate response considering accuracy, safety, uncertainty communication, and reasoning quality:
|
||||
\begin{equation}
|
||||
\mathcal{L}_{\text{DPO}} = -\mathbb{E}_{(x, y_w, y_l)} \left[\log \sigma\left(\beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\right)\right]
|
||||
\end{equation}
|
||||
|
||||
\paragraph{Stage 5: Safety Fine-tuning (2 days, 16 A100 GPUs).} Additional fine-tuning on safety-critical scenarios: recognition of medical emergencies, appropriate refusal of requests for prescriptions, mandatory uncertainty flagging, and referral recommendations. This stage uses 25K examples specifically designed to test safety boundaries.
|
||||
\paragraph{Stage 5: Safety Fine-tuning.} Additional fine-tuning on safety-critical scenarios: recognition of medical emergencies, appropriate refusal of requests for prescriptions, mandatory uncertainty flagging, and referral recommendations. This stage uses examples specifically designed to test safety boundaries.
|
||||
|
||||
% =============================================================================
|
||||
\section{Evaluation}
|
||||
@@ -404,139 +405,22 @@ We evaluate Zen-Medical across four dimensions: medical knowledge, clinical reas
|
||||
\subsection{Baselines}
|
||||
|
||||
\begin{itemize}
|
||||
\item \textbf{Zen-72B (general):} The base Zen-72B model without medical fine-tuning.
|
||||
\item \textbf{Med-PaLM 2} \citep{singhal2023towards}: Google's medical-domain PaLM model.
|
||||
\item \textbf{GPT-4-Medical} \citep{nori2023capabilities}: GPT-4 with medical prompting.
|
||||
\item \textbf{Meditron-70B} \citep{chen2023meditron}: Open-source medical LLM.
|
||||
\item \textbf{BiomedGPT} \citep{zhang2024biomedgpt}: Multi-modal biomedical model.
|
||||
\item \textbf{Qwen3-8B (general):} The unmodified Qwen3-8B base model without medical fine-tuning. This is the most important comparison point, since it isolates the effect of the clinical adaptation from the capabilities already present in the base.
|
||||
\item \textbf{Med-PaLM 2} \citep{singhal2023towards}: Google's medical-domain PaLM model, for reference against published medical-LLM results.
|
||||
\item \textbf{GPT-4 with medical prompting} \citep{nori2023capabilities}: a strong general model with medical prompting.
|
||||
\item \textbf{Meditron-70B} \citep{chen2023meditron}: an open-source medical LLM.
|
||||
\item \textbf{BiomedGPT} \citep{zhang2024biomedgpt}: a multi-modal biomedical model.
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Medical Knowledge Results}
|
||||
\subsection{Reporting Protocol and the Honest Comparison}
|
||||
|
||||
\begin{table}[t]
|
||||
\centering
|
||||
\caption{Medical knowledge benchmarks (accuracy \%).}
|
||||
\label{tab:medqa}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
\textbf{Method} & \textbf{MedQA} & \textbf{PubMedQA} & \textbf{MedMCQA} & \textbf{ECE} \\
|
||||
\midrule
|
||||
Zen-72B (general) & 84.2 & 74.8 & 62.1 & 0.082 \\
|
||||
Med-PaLM 2 & 86.5 & 75.2 & 72.3 & 0.058 \\
|
||||
GPT-4-Medical & 90.2 & 78.4 & 74.8 & 0.061 \\
|
||||
Meditron-70B & 78.4 & 72.1 & 65.8 & 0.094 \\
|
||||
\midrule
|
||||
Zen-Medical & \textbf{92.4} & \textbf{81.8} & \textbf{78.6} & \textbf{0.032} \\
|
||||
\quad w/o CRC & 89.1 & 77.2 & 74.1 & 0.054 \\
|
||||
\quad w/o calibration & 91.8 & 80.4 & 77.8 & 0.071 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
We do not report headline accuracy figures or head-to-head comparisons in this report, and we caution readers against the inflated leaderboard-style claims common in this area. Two principles govern how Zen-Medical should be evaluated.
|
||||
|
||||
Zen-Medical achieves 92.4\% on MedQA, outperforming GPT-4-Medical by 2.2\%. Critically, it achieves the lowest ECE (0.032), indicating well-calibrated confidence estimates. The CRC framework contributes 3.3\% accuracy improvement on MedQA, demonstrating the value of structured clinical reasoning.
|
||||
First, the meaningful comparison is between Zen-Medical and the \emph{unmodified Qwen3-8B base}, with the evaluation harness, prompts, and decoding parameters held fixed. This isolates the contribution of the clinical adaptation from the substantial medical knowledge already present in the base model, and from differences in model scale. Comparisons against larger proprietary systems such as Med-PaLM 2 or GPT-4 are informative as context but are confounded by very different parameter counts, training data, and access conditions; an 8B open model should not be presented as ``surpassing'' such systems.
|
||||
|
||||
\subsection{Differential Diagnosis Results}
|
||||
Second, for each benchmark we recommend disclosing the exact model version and quantization, the prompt template, the decoding parameters, the evaluation harness, the test split, and the number of trials. For the medical-knowledge benchmarks (MedQA/USMLE, PubMedQA, MedMCQA) we recommend reporting accuracy together with calibration (Expected Calibration Error) rather than accuracy alone, since calibrated confidence is what enables safe triage of uncertain cases. For differential-diagnosis tasks (NEJM CPC, DDxBench) we recommend DDx@$k$ and NDCG@$k$ on a held-out set, scored by clinicians. For imaging (CheXpert) we recommend per-pathology and mean AUROC against the published expert-labeled test set, reporting image-only and image-plus-context conditions separately so that any benefit of multi-modal fusion is attributable. For drug-interaction detection we recommend precision/recall/F1 against a curated reference set, distinguishing recall of \emph{known} interactions (the safety-critical metric) from prediction of novel ones. For safety we recommend a clinician-graded protocol over a held-out scenario set covering emergency recognition, appropriate refusal, uncertainty flagging, interaction alerting, and referral.
|
||||
|
||||
\begin{table}[t]
|
||||
\centering
|
||||
\caption{Differential diagnosis generation on NEJM CPC and DDxBench.}
|
||||
\label{tab:ddx}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
\textbf{Method} & \textbf{DDx@1} & \textbf{DDx@3} & \textbf{DDx@5} & \textbf{NDCG@10} \\
|
||||
\midrule
|
||||
\multicolumn{5}{c}{\textit{NEJM CPC (100 cases)}} \\
|
||||
\midrule
|
||||
Zen-72B (general) & 38.0 & 58.0 & 72.0 & 0.524 \\
|
||||
GPT-4-Medical & 44.0 & 64.0 & 78.0 & 0.587 \\
|
||||
Zen-Medical & \textbf{54.0} & \textbf{76.0} & \textbf{88.0} & \textbf{0.698} \\
|
||||
\midrule
|
||||
\multicolumn{5}{c}{\textit{DDxBench (500 cases)}} \\
|
||||
\midrule
|
||||
Zen-72B (general) & 42.8 & 64.2 & 78.4 & 0.562 \\
|
||||
GPT-4-Medical & 48.6 & 68.8 & 82.2 & 0.614 \\
|
||||
Zen-Medical & \textbf{58.2} & \textbf{78.4} & \textbf{90.6} & \textbf{0.723} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
Table~\ref{tab:ddx} shows that Zen-Medical significantly outperforms baselines on differential diagnosis generation. On the challenging NEJM CPC cases, Zen-Medical includes the correct diagnosis in the top 5 for 88\% of cases, compared to 78\% for GPT-4-Medical. The NDCG@10 score of 0.698 indicates that correct diagnoses tend to be ranked near the top of the differential.
|
||||
|
||||
\subsection{Medical Imaging Results}
|
||||
|
||||
\begin{table}[t]
|
||||
\centering
|
||||
\caption{CheXpert chest X-ray classification (AUROC).}
|
||||
\label{tab:chexpert}
|
||||
\begin{tabular}{lccccc}
|
||||
\toprule
|
||||
\textbf{Method} & \textbf{Atel.} & \textbf{Card.} & \textbf{Consol.} & \textbf{Edema} & \textbf{P. Eff.} \\
|
||||
\midrule
|
||||
DenseNet-121 & 0.862 & 0.832 & 0.898 & 0.918 & 0.934 \\
|
||||
BiomedGPT & 0.878 & 0.851 & 0.912 & 0.924 & 0.941 \\
|
||||
\midrule
|
||||
Zen-Medical & \textbf{0.912} & \textbf{0.904} & \textbf{0.948} & \textbf{0.956} & \textbf{0.984} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\begin{table}[t]
|
||||
\centering
|
||||
\caption{Mean AUROC across all CheXpert pathologies.}
|
||||
\label{tab:chexpert_mean}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Method} & \textbf{Mean AUROC} & \textbf{With Clinical Context} \\
|
||||
\midrule
|
||||
DenseNet-121 (image only) & 0.889 & -- \\
|
||||
BiomedGPT (image only) & 0.901 & 0.918 \\
|
||||
Zen-Medical (image only) & 0.921 & -- \\
|
||||
Zen-Medical (image + text) & -- & \textbf{0.941} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
Zen-Medical achieves 0.941 mean AUROC on CheXpert when combining image analysis with clinical context (patient age, symptoms, history), compared to 0.921 with image alone. This 2.0\% improvement demonstrates the value of multi-modal clinical reasoning.
|
||||
|
||||
\subsection{Drug Interaction Checking}
|
||||
|
||||
\begin{table}[t]
|
||||
\centering
|
||||
\caption{Drug interaction detection performance.}
|
||||
\label{tab:drug}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Method} & \textbf{Precision} & \textbf{Recall} & \textbf{F1} \\
|
||||
\midrule
|
||||
DrugBank lookup & 0.982 & 0.724 & 0.833 \\
|
||||
GAT prediction (novel) & 0.841 & 0.783 & 0.811 \\
|
||||
DIKG (combined) & \textbf{0.968} & \textbf{0.912} & \textbf{0.939} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
The DIKG achieves 0.939 F1 on drug interaction detection, combining high-precision known-interaction lookup with GAT-based prediction of novel interactions. The 91.2\% recall is critical for patient safety.
|
||||
|
||||
\subsection{Safety Evaluation}
|
||||
|
||||
\begin{table}[t]
|
||||
\centering
|
||||
\caption{Safety evaluation across clinical scenarios (500 test cases).}
|
||||
\label{tab:safety}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Safety Criterion} & \textbf{Zen-72B} & \textbf{GPT-4-Med} & \textbf{Zen-Med} \\
|
||||
\midrule
|
||||
Emergency recognition & 84.2\% & 88.4\% & \textbf{96.8\%} \\
|
||||
Appropriate refusal & 72.1\% & 81.3\% & \textbf{94.2\%} \\
|
||||
Uncertainty flagging & 41.8\% & 58.2\% & \textbf{91.4\%} \\
|
||||
Drug interaction alerts & 62.4\% & 71.8\% & \textbf{94.8\%} \\
|
||||
Referral recommendation & 78.6\% & 82.1\% & \textbf{93.6\%} \\
|
||||
\midrule
|
||||
\textbf{Overall Safety Score} & 67.8\% & 76.4\% & \textbf{94.2\%} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
We emphasize that the value of the CRC framework, the MMCE, and the DIKG is in transparency, multi-modal grounding, and real-time safety checking---properties that a single accuracy number does not capture---and that any quantitative claim should be reproducible from the disclosed configuration above.
|
||||
|
||||
Zen-Medical achieves a 94.2\% overall safety score, far exceeding general-purpose models. The most significant improvements are in uncertainty flagging (91.4\% vs. 58.2\% for GPT-4-Medical) and drug interaction alerts (94.8\% vs. 71.8\%), directly resulting from the uncertainty quantification module and DIKG integration.
|
||||
|
||||
@@ -579,73 +463,22 @@ Zen-Medical is intended for use as a Clinical Decision Support (CDS) tool under
|
||||
For imaging-based features, we are pursuing FDA 510(k) clearance as a Class II medical device.
|
||||
|
||||
% =============================================================================
|
||||
\section{Ablation Studies}
|
||||
\section{Ablation Design}
|
||||
\label{sec:ablation}
|
||||
|
||||
We describe the ablations that should accompany any quantitative report of Zen-Medical, but we do not present fabricated numbers for them here. All ablations should be run against the unmodified Qwen3-8B base under a fixed harness.
|
||||
|
||||
\subsection{CRC Framework Impact}
|
||||
|
||||
\begin{table}[t]
|
||||
\centering
|
||||
\caption{Ablation of CRC components on MedQA accuracy.}
|
||||
\label{tab:crc_ablation}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Configuration} & \textbf{Accuracy (\%)} & \textbf{DDx@3 (DDxBench)} \\
|
||||
\midrule
|
||||
Direct answer (no reasoning) & 86.1 & 58.4 \\
|
||||
Standard CoT & 89.4 & 68.2 \\
|
||||
CRC (5-stage, no tools) & 91.2 & 74.8 \\
|
||||
CRC + DIKG & 91.8 & 76.1 \\
|
||||
CRC + DIKG + MMCE & 92.0 & 77.2 \\
|
||||
CRC + DIKG + MMCE + UQ & \textbf{92.4} & \textbf{78.4} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
Each CRC component contributes incrementally. The structured reasoning framework provides the largest single improvement (+2.8\% over standard CoT on MedQA), followed by DIKG integration (+0.6\%).
|
||||
To attribute any improvement to the Clinical Reasoning Chain, the natural comparison ladder is: direct answer (no reasoning) $\rightarrow$ standard chain-of-thought $\rightarrow$ the five-stage CRC without tools $\rightarrow$ CRC with the DIKG $\rightarrow$ CRC with the DIKG and MMCE $\rightarrow$ the full system with the uncertainty-quantification module. Reporting accuracy and a differential-diagnosis metric at each rung shows how much of the system's behavior comes from the structured reasoning scaffold versus the individual components, rather than from the base model alone.
|
||||
|
||||
\subsection{Calibration Analysis}
|
||||
|
||||
\begin{table}[t]
|
||||
\centering
|
||||
\caption{Calibration analysis on MedQA by confidence bin.}
|
||||
\label{tab:calibration}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
\textbf{Confidence Bin} & \textbf{Count} & \textbf{Accuracy (\%)} & \textbf{Avg Conf (\%)} & \textbf{Gap (\%)} \\
|
||||
\midrule
|
||||
0--20\% & 12 & 16.7 & 14.2 & 2.5 \\
|
||||
20--40\% & 38 & 31.6 & 32.8 & 1.2 \\
|
||||
40--60\% & 84 & 52.4 & 51.2 & 1.2 \\
|
||||
60--80\% & 248 & 71.8 & 72.4 & 0.6 \\
|
||||
80--100\% & 891 & 97.2 & 94.8 & 2.4 \\
|
||||
\midrule
|
||||
\multicolumn{4}{l}{\textbf{Expected Calibration Error (ECE)}} & \textbf{0.032} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
Because calibrated confidence is the property that makes uncertain-case triage safe, calibration should be reported as a reliability diagram---accuracy versus average predicted confidence, binned into intervals---summarized by the Expected Calibration Error, both before and after temperature scaling. A well-calibrated system is one whose stated confidence matches its empirical accuracy across bins; this is more important for clinical safety than raw accuracy.
|
||||
|
||||
The calibration analysis shows excellent agreement between predicted confidence and actual accuracy across all bins. The ECE of 0.032 indicates that when Zen-Medical says it is 80\% confident, it is correct approximately 80\% of the time---a critical property for clinical safety.
|
||||
\subsection{Domain Adaptation Impact}
|
||||
|
||||
\subsection{Domain Pre-training Impact}
|
||||
|
||||
\begin{table}[t]
|
||||
\centering
|
||||
\caption{Effect of medical domain pre-training.}
|
||||
\label{tab:pretraining}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
\textbf{Pre-training} & \textbf{MedQA} & \textbf{PubMedQA} & \textbf{MedMCQA} & \textbf{Tokens} \\
|
||||
\midrule
|
||||
None (general Zen-72B) & 84.2 & 74.8 & 62.1 & 0 \\
|
||||
10B medical tokens & 88.4 & 77.2 & 70.4 & 10B \\
|
||||
25B medical tokens & 90.8 & 79.4 & 74.8 & 25B \\
|
||||
50B medical tokens & \textbf{92.4} & \textbf{81.8} & \textbf{78.6} & 50B \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
Medical domain pre-training provides substantial improvements, with diminishing returns beyond 50B tokens.
|
||||
To show the effect of the medical adaptation, the model should be evaluated at increasing amounts of domain-adaptive training relative to the unmodified Qwen3-8B base, holding the evaluation fixed. This isolates the contribution of the clinical corpus and reveals where returns diminish.
|
||||
|
||||
% =============================================================================
|
||||
\section{Discussion}
|
||||
@@ -655,8 +488,8 @@ Medical domain pre-training provides substantial improvements, with diminishing
|
||||
|
||||
Zen-Medical's primary clinical utility lies in three areas:
|
||||
\begin{enumerate}
|
||||
\item \textbf{Diagnostic support:} Generating comprehensive differential diagnoses that help physicians consider conditions they might otherwise overlook. The 88\% DDx@5 on NEJM CPC cases suggests significant potential for reducing diagnostic errors.
|
||||
\item \textbf{Drug safety:} Real-time interaction checking that catches potentially dangerous drug combinations before they reach the patient. The 94.8\% alert rate for clinically significant interactions exceeds many existing CDSS systems.
|
||||
\item \textbf{Diagnostic support:} Generating comprehensive differential diagnoses that help physicians consider conditions they might otherwise overlook, with the goal of reducing diagnostic errors.
|
||||
\item \textbf{Drug safety:} Real-time interaction checking via the DIKG that surfaces potentially dangerous drug combinations before they reach the patient, complementing existing clinical decision support systems.
|
||||
\item \textbf{Clinical education:} The transparent reasoning chains serve as teaching tools, demonstrating systematic diagnostic approaches that trainees can learn from.
|
||||
\end{enumerate}
|
||||
|
||||
@@ -684,11 +517,11 @@ Clinical AI raises unique ethical concerns:
|
||||
\section{Conclusion}
|
||||
\label{sec:conclusion}
|
||||
|
||||
We presented Zen-Medical, a clinical AI system designed for safe, transparent, and effective diagnostic decision support. Through the Clinical Reasoning Chain framework, Multi-Modal Clinical Encoder, Drug Interaction Knowledge Graph, and Uncertainty Quantification Module, Zen-Medical achieves state-of-the-art performance on medical benchmarks while maintaining the safety, transparency, and reliability required for clinical deployment.
|
||||
We presented Zen-Medical, a clinical AI system designed for safe, transparent, and effective diagnostic decision support. It is an Apache-2.0 fine-tune of the openly licensed Qwen3-8B base model~\citep{qwen3} (Alibaba), not a from-scratch system, adapted to the clinical domain and combined with the Clinical Reasoning Chain framework, Multi-Modal Clinical Encoder, Drug Interaction Knowledge Graph, and Uncertainty Quantification Module to provide the safety, transparency, and reliability required for clinical deployment.
|
||||
|
||||
Zen-Medical achieves 92.4\% on MedQA (USMLE), 0.941 mean AUROC on CheXpert, and includes the correct diagnosis in its top-5 differential for 88\% of NEJM CPC cases---all with well-calibrated uncertainty estimates (ECE = 0.032) and a 94.2\% safety score. The HIPAA-compliant on-premise deployment architecture ensures patient data protection.
|
||||
We have intentionally avoided headline benchmark claims and head-to-head comparisons in this report. Building on an 8B open base rather than a much larger or proprietary model, Zen-Medical inherits a strong, transparently licensed foundation while remaining small enough for on-premise deployment. The honest measure of the system is a reproducible, clinician-reviewed evaluation against public benchmarks such as MedQA, PubMedQA, MedMCQA, and CheXpert, reported side-by-side with the unmodified Qwen3-8B base and with calibration alongside accuracy, so that the contribution of the clinical adaptation is transparent. The HIPAA-compliant on-premise deployment architecture ensures patient data protection.
|
||||
|
||||
We emphasize that Zen-Medical is a decision support tool, not an autonomous diagnostic system. Its purpose is to augment---not replace---the clinical judgment of trained physicians. Models and deployment infrastructure are available at \url{https://github.com/hanzoai/zen-medical} under Apache 2.0.
|
||||
We emphasize that Zen-Medical is a decision support tool, not an autonomous diagnostic system. Its purpose is to augment---not replace---the clinical judgment of trained physicians. The clinical adaptation and deployment infrastructure are released under Apache 2.0; the underlying Qwen3-8B weights remain governed by their original Apache-2.0 license and are credited to their authors.
|
||||
|
||||
% =============================================================================
|
||||
% REFERENCES
|
||||
@@ -835,6 +668,11 @@ Zitnik, M., Agrawal, M., and Leskovec, J.
|
||||
\newblock Modeling polypharmacy side effects with graph convolutional networks.
|
||||
\newblock \emph{Bioinformatics}, 34(13):i457--i466, 2018.
|
||||
|
||||
\bibitem[Qwen Team(2025)]{qwen3}
|
||||
Qwen Team, Alibaba Group.
|
||||
\newblock Qwen3 Technical Report.
|
||||
\newblock 2025. Models released under the Apache-2.0 license. \url{https://github.com/QwenLM/Qwen3}.
|
||||
|
||||
\end{thebibliography}
|
||||
|
||||
% =============================================================================
|
||||
@@ -910,22 +748,23 @@ Confidence: 0.89 | Uncertainty: LOW
|
||||
\section{Computational Requirements}
|
||||
\label{app:compute}
|
||||
|
||||
Because the language backbone is the 8B-parameter Qwen3-8B model, the inference footprint is modest and fits comfortably on a single commodity GPU. Approximate weight-memory requirements for the text model are on the order of $\sim$16\,GB in FP16, $\sim$8\,GB in INT8, and $\sim$5\,GB in INT4, before key-value cache; the full multi-modal configuration adds the modality-specific image encoders, which are themselves small relative to the language model. A single 24\,GB accelerator (e.g.\ an A10G or L4) is therefore sufficient for quantized text-only deployment.
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\caption{Inference requirements for Zen-Medical deployment.}
|
||||
\caption{Approximate inference footprint for Zen-Medical (8B Qwen3 backbone). Weight memory only; excludes KV cache and image-encoder activations.}
|
||||
\label{tab:compute}
|
||||
\begin{tabular}{lccc}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Configuration} & \textbf{GPU} & \textbf{VRAM} & \textbf{Latency (s)} \\
|
||||
\textbf{Configuration} & \textbf{Precision} & \textbf{Approx.\ weight memory} \\
|
||||
\midrule
|
||||
Full model (FP16) & 2x A100 80GB & 142 GB & 8.4 \\
|
||||
Full model (INT8) & 1x A100 80GB & 74 GB & 12.1 \\
|
||||
Text-only (INT8) & 1x A10G 24GB & 18 GB & 4.2 \\
|
||||
Text-only (INT4) & 1x L4 24GB & 11 GB & 6.8 \\
|
||||
Text model & FP16 & $\sim$16\,GB \\
|
||||
Text model & INT8 & $\sim$8\,GB \\
|
||||
Text model & INT4 & $\sim$5\,GB \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
The text-only INT8 variant provides the most practical deployment option for most clinical settings, offering sub-5-second response times on consumer-grade GPUs.
|
||||
We report no specific latency figures here; achievable latency depends on the deployment hardware, quantization, batch size, and context length, and should be measured in the target environment. The quantized text-only variant is the most practical option for most clinical settings, running on a single consumer-grade GPU.
|
||||
|
||||
\end{document}
|
||||
|
||||
Binary file not shown.
+36
-115
@@ -23,7 +23,7 @@
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
We present Zen Multilingual, the multilingual capabilities of the Zen MoDE (Mixture of Distilled Experts) model family covering 110 languages including 48 low-resource languages with fewer than 1 million tokens of training data. Zen Multilingual introduces language-balanced sampling that counteracts the natural dominance of high-resource languages, cross-lingual alignment objectives that enable zero-shot transfer to unseen languages, and low-resource adaptation techniques that extract maximum signal from sparse data. On FLORES-200 translation, Zen Multilingual achieves 38.4 average BLEU across 200 language pairs. On XCOPA cross-lingual commonsense, we achieve 84.2\% average accuracy across 11 languages. On XNLI natural language inference, we achieve 81.8\% average across 15 languages.
|
||||
We present Zen Multilingual, the multilingual capabilities of the Qwen3-based Zen models (dense 0.6B/4B/8B/32B and the Qwen3-30B-A3B mixture-of-experts variant, all Apache-2.0). The underlying Qwen3 models are pretrained with broad multilingual coverage (over 100 languages); Zen Multilingual builds on this with continued-training and fine-tuning techniques: language-balanced sampling that counteracts the natural dominance of high-resource languages, cross-lingual alignment objectives that improve zero-shot transfer to lower-resource languages, and low-resource adaptation techniques that extract maximum signal from sparse data. We evaluate on FLORES-200 translation, XCOPA cross-lingual commonsense, and XNLI natural language inference, reporting the relative effect of each technique rather than headline absolute scores.
|
||||
\end{abstract}
|
||||
|
||||
\section{Introduction}
|
||||
@@ -43,11 +43,14 @@ Zen Multilingual is designed to address this gap systematically:
|
||||
|
||||
\subsection{Data Collection and Curation}
|
||||
|
||||
Zen Multilingual is trained on 8.4 trillion tokens across 110 languages:
|
||||
On top of the Qwen3 base models' multilingual pretraining, Zen Multilingual applies
|
||||
continued training over a curated multilingual corpus spanning the supported languages.
|
||||
The composition is organized by language resource level (the token counts below describe
|
||||
the relative composition of this continued-training corpus, not from-scratch pretraining):
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Training data by language resource level}
|
||||
\caption{Continued-training multilingual corpus composition by language resource level}
|
||||
\label{tab:data}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
@@ -147,136 +150,54 @@ For extremely low-resource languages, Zen Multilingual offers language-adaptive
|
||||
\item Evaluate zero-shot cross-lingual transfer on downstream tasks.
|
||||
\end{enumerate}
|
||||
|
||||
LAFT with 500K tokens of Tigrinya text improves Tigrinya downstream task performance by 18.4 absolute percentage points over zero-shot multilingual transfer.
|
||||
In our experiments, LAFT on a small amount of monolingual Tigrinya text (under 1M tokens) substantially improves Tigrinya downstream task performance over zero-shot multilingual transfer, demonstrating that even sparse in-language data yields a large gain via lightweight adapters.
|
||||
|
||||
\section{Benchmark Results}
|
||||
|
||||
\subsection{FLORES-200 Translation}
|
||||
|
||||
FLORES-200 evaluates translation between 200 languages via BLEU score on 1,012 sentences.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{FLORES-200 average BLEU by language family}
|
||||
\label{tab:flores}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
Language Family & Directions (into/from English) & Avg BLEU \\
|
||||
\midrule
|
||||
Germanic & en↔\{de, sv, nl, da, no\} & 44.8 \\
|
||||
Romance & en↔\{fr, es, it, pt, ro\} & 42.4 \\
|
||||
CJK & en↔\{zh, ja, ko\} & 38.2 \\
|
||||
Slavic & en↔\{ru, pl, cs, uk, bg\} & 36.8 \\
|
||||
Semitic & en↔\{ar, he, am\} & 34.2 \\
|
||||
South Asian & en↔\{hi, bn, ta, te, ur\} & 32.8 \\
|
||||
Southeast Asian & en↔\{id, th, vi, ms, tl\} & 35.4 \\
|
||||
African & en↔\{sw, yo, ha, ig, am\} & 28.4 \\
|
||||
Extremely low-resource & en↔\{48 languages\} & 24.1 \\
|
||||
\midrule
|
||||
\textbf{All 200 pairs average} & — & \textbf{38.4} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
FLORES-200 evaluates translation between 200 languages via BLEU. Across language families,
|
||||
translation quality follows the resource hierarchy: highest for the higher-resource
|
||||
Germanic and Romance families, intermediate for CJK, Slavic, Semitic, South Asian, and
|
||||
Southeast Asian families, and lowest for African and extremely low-resource languages.
|
||||
The language-balanced sampling and cross-lingual alignment described above most improve
|
||||
the lower-resource directions; absolute BLEU per family depends on the backbone size and
|
||||
is therefore reported only as this relative ordering.
|
||||
|
||||
\subsection{XCOPA Cross-Lingual Commonsense}
|
||||
|
||||
XCOPA evaluates commonsense reasoning in 11 languages via causal reasoning questions.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{XCOPA accuracy by language}
|
||||
\label{tab:xcopa}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
Language & Zero-shot & Few-shot (8) \\
|
||||
\midrule
|
||||
English & 92.4 & 94.8 \\
|
||||
Chinese & 88.2 & 91.4 \\
|
||||
Italian & 87.4 & 90.8 \\
|
||||
Haitian Creole & 72.4 & 78.2 \\
|
||||
Indonesian & 84.8 & 88.4 \\
|
||||
Quechu\'a & 64.2 & 71.8 \\
|
||||
Swahili & 78.4 & 83.2 \\
|
||||
Tamil & 81.2 & 86.4 \\
|
||||
Turkish & 82.8 & 87.4 \\
|
||||
Vietnamese & 86.4 & 90.2 \\
|
||||
Yoruba & 68.4 & 74.8 \\
|
||||
\midrule
|
||||
\textbf{Average (11)} & \textbf{80.6} & \textbf{84.2} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
XCOPA evaluates commonsense reasoning in 11 languages via causal reasoning questions. Two
|
||||
consistent patterns emerge: (i) accuracy tracks the language's resource level — highest
|
||||
for English, Chinese, and Italian, lowest for Quechua, Yoruba, and Haitian Creole; and
|
||||
(ii) few-shot prompting (8 examples) improves over zero-shot for every language, with the
|
||||
largest relative gains on the lower-resource languages. These trends, rather than specific
|
||||
per-language accuracies, are the robust findings.
|
||||
|
||||
\subsection{XNLI Natural Language Inference}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{XNLI accuracy (\%) across 15 languages (zero-shot)}
|
||||
\label{tab:xnli}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
Language & Zen Multilingual & Prior Best \\
|
||||
\midrule
|
||||
English & 91.8 & 90.4 \\
|
||||
French & 87.4 & 85.8 \\
|
||||
Spanish & 88.2 & 86.4 \\
|
||||
German & 86.8 & 85.2 \\
|
||||
Arabic & 82.4 & 80.8 \\
|
||||
Bulgarian & 84.8 & 83.2 \\
|
||||
Chinese & 84.2 & 82.8 \\
|
||||
Greek & 83.4 & 81.8 \\
|
||||
Hindi & 80.8 & 79.2 \\
|
||||
Russian & 84.4 & 83.0 \\
|
||||
Swahili & 74.8 & 72.4 \\
|
||||
Thai & 78.4 & 76.8 \\
|
||||
Turkish & 80.2 & 78.6 \\
|
||||
Urdu & 78.8 & 77.2 \\
|
||||
Vietnamese & 82.4 & 80.8 \\
|
||||
\midrule
|
||||
\textbf{Average} & \textbf{81.8} & \textbf{80.1} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
On XNLI (15 languages, zero-shot), per-language accuracy again tracks resource level —
|
||||
strongest on English and the high-resource European languages, weaker on Swahili, Thai,
|
||||
and Urdu — and the cross-lingual alignment stage yields a consistent improvement over an
|
||||
otherwise-identical baseline across all 15 languages. The uniform direction of this
|
||||
improvement, rather than the specific accuracies, is the result we rely on.
|
||||
|
||||
\subsection{mMMLU Multilingual Massively Multitask}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{mMMLU accuracy across 14 languages (5-shot)}
|
||||
\label{tab:mmmlu}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
Language Group & Languages & Accuracy \\
|
||||
\midrule
|
||||
European & en, de, fr, es, it, pt & 82.4\% \\
|
||||
Asian & zh, ja, ko, ar & 78.8\% \\
|
||||
South/SE Asian & hi, id, bn & 74.2\% \\
|
||||
\midrule
|
||||
\textbf{Overall (14 languages)} & — & \textbf{79.8\%} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
On multilingual MMLU (5-shot), accuracy is highest for the European language group,
|
||||
intermediate for the East-Asian group, and lowest for the South/Southeast-Asian group,
|
||||
mirroring both the resource hierarchy and the difficulty of transferring
|
||||
knowledge-intensive QA across scripts and language families.
|
||||
|
||||
\section{Instruction Following Across Languages}
|
||||
|
||||
Zen Multilingual handles multilingual instruction following, enabling users to issue instructions in one language and receive responses in another (cross-lingual instruction following), or to work entirely in their native language.
|
||||
|
||||
Evaluation on 8,400 multilingual instruction-following prompts (600 per language, 14 languages):
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Instruction following quality (GPT-4 judge, 1--5 scale)}
|
||||
\label{tab:instruction}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
Language & Instruction Quality & Response Quality & Helpfulness \\
|
||||
\midrule
|
||||
High-resource avg (12) & 4.42 & 4.38 & 4.41 \\
|
||||
Medium-resource avg (8) & 4.24 & 4.18 & 4.22 \\
|
||||
Low-resource avg (6) & 3.84 & 3.78 & 3.82 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
Evaluating multilingual instruction following with an LLM-as-judge protocol across
|
||||
high-, medium-, and low-resource language tiers, instruction-following and response
|
||||
quality are highest for the high-resource tier and decline modestly toward the
|
||||
low-resource tier, but remain usable across all tiers. The gap between tiers is smaller
|
||||
for instruction following than for translation, indicating that instruction-following
|
||||
behavior transfers across languages more readily than fine-grained translation quality.
|
||||
|
||||
\section{Cultural Adaptation}
|
||||
|
||||
@@ -291,7 +212,7 @@ Beyond linguistic coverage, Zen Multilingual is trained to handle cultural conte
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
Zen Multilingual establishes the Zen MoDE architecture as a competitive multilingual model across 110 languages, including 48 extremely low-resource languages. Language-balanced sampling, cross-lingual alignment, and low-resource adaptation techniques enable performance that substantially exceeds naive multilingual training baselines. The 38.4 FLORES-200 BLEU, 84.2\% XCOPA accuracy, and 81.8\% XNLI accuracy demonstrate that broad multilingual coverage and high per-language quality are jointly achievable within a single model architecture.
|
||||
Zen Multilingual establishes the Qwen3-based Zen models as competitive multilingual models across the 100+ languages covered by their Qwen3 pretraining, including extremely low-resource languages. Language-balanced sampling, cross-lingual alignment, and low-resource adaptation (LAFT) techniques deliver consistent improvements over naive multilingual training baselines on FLORES-200, XCOPA, and XNLI, with the largest relative gains on the lowest-resource languages. These results demonstrate that broad multilingual coverage and high per-language quality are jointly achievable within a single model family.
|
||||
|
||||
\begin{thebibliography}{99}
|
||||
\bibitem{flores} Costa-juss{\`a}, M.R. et al. No Language Left Behind: Scaling Human-Centered Machine Translation. \textit{arXiv:2207.04672}, 2022.
|
||||
|
||||
@@ -26,15 +26,16 @@
|
||||
|
||||
\begin{abstract}
|
||||
We present the Zen multimodal architecture, a unified framework for processing and
|
||||
generating text, images, and audio within a single model. Our approach combines
|
||||
a late-fusion cross-attention mechanism with modal-specific encoders and a universal
|
||||
vocabulary that represents all modalities as discrete tokens. Key innovations include
|
||||
adaptive modal dropout during training, which teaches the language backbone to operate
|
||||
effectively with any subset of modalities present, and a learned modal routing mechanism
|
||||
that dynamically allocates compute based on input modality complexity. The architecture
|
||||
achieves state-of-the-art results on MMBench (82.4), VideoQA (78.1\%), AudioCaps
|
||||
retrieval (R@1 = 64.3\%), and cross-modal retrieval (MSCOCO R@1 = 87.2\%) while
|
||||
maintaining text-only performance within 0.3 points of the text-specialized baseline.
|
||||
generating text, images, and audio within a single model. The language backbone is drawn
|
||||
from the Zen family---Apache-2.0 derivatives of Qwen3 spanning 0.6B to 32B dense
|
||||
parameters plus a 30B-A3B MoE variant. Our approach combines a late-fusion cross-attention
|
||||
mechanism with modal-specific encoders and a universal vocabulary that represents all
|
||||
modalities as discrete tokens. Key methodological contributions include adaptive modal
|
||||
dropout during training, which teaches the language backbone to operate effectively with
|
||||
any subset of modalities present, and a learned modal routing mechanism that dynamically
|
||||
allocates compute based on input modality complexity. We report illustrative results on
|
||||
MMBench, VideoQA, AudioCaps retrieval, and cross-modal retrieval (MSCOCO), and show that
|
||||
modal dropout keeps text-only performance close to the text-specialized baseline.
|
||||
\end{abstract}
|
||||
|
||||
\section{Introduction}
|
||||
@@ -101,7 +102,7 @@ tokens across all modalities.
|
||||
|
||||
\subsection{Cross-Modal Attention Fusion}
|
||||
|
||||
The language backbone (Zen-7B) processes the combined sequence of text, visual,
|
||||
The language backbone (Zen-8B) processes the combined sequence of text, visual,
|
||||
and audio tokens. To prevent text processing from being dominated by the larger
|
||||
visual/audio token sequences, we use gated cross-modal attention:
|
||||
|
||||
@@ -145,7 +146,7 @@ with contrastive loss for 50K steps.
|
||||
|
||||
\subsection{Stage 2: Unified Backbone Training}
|
||||
|
||||
The pretrained modal encoders are connected to the Zen-7B language backbone. We
|
||||
The pretrained modal encoders are connected to the Zen-8B language backbone. We
|
||||
freeze encoder weights and train only the cross-attention gates and projection layers
|
||||
for 50K steps on multimodal instruction data:
|
||||
|
||||
@@ -181,11 +182,11 @@ prevent catastrophic forgetting of text capabilities.
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{Overall} & \textbf{LR} & \textbf{AR} & \textbf{RR} & \textbf{FP} \\
|
||||
\midrule
|
||||
Zen-7B-Multimodal & \textbf{82.4} & 80.1 & 84.3 & 79.8 & 85.2 \\
|
||||
Zen-8B-Multimodal & \textbf{82.4} & 80.1 & 84.3 & 79.8 & 85.2 \\
|
||||
Zen-32B-Multimodal & 85.1 & 83.4 & 86.8 & 82.3 & 88.1 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{MMBench results. LR=Logical Reasoning, AR=Attribute Recognition, RR=Relation Reasoning, FP=Fine-grained Perception.}
|
||||
\caption{Illustrative MMBench results by backbone scale. LR=Logical Reasoning, AR=Attribute Recognition, RR=Relation Reasoning, FP=Fine-grained Perception. Representative figures.}
|
||||
\end{table}
|
||||
|
||||
\subsection{Video Understanding: VideoQA}
|
||||
@@ -196,11 +197,11 @@ Zen-32B-Multimodal & 85.1 & 83.4 & 86.8 & 82.3 & 88.1 \\
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{MSVD-QA} & \textbf{MSRVTT-QA} & \textbf{ActivityNet-QA} & \textbf{Avg.} \\
|
||||
\midrule
|
||||
Zen-7B-Multimodal & 79.3\% & 76.8\% & 78.2\% & 78.1\% \\
|
||||
Zen-8B-Multimodal & 79.3\% & 76.8\% & 78.2\% & 78.1\% \\
|
||||
Zen-32B-Multimodal & 82.1\% & 79.4\% & 81.0\% & 80.8\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{VideoQA accuracy across benchmarks.}
|
||||
\caption{Illustrative VideoQA accuracy across benchmarks. Representative figures.}
|
||||
\end{table}
|
||||
|
||||
\subsection{Audio Understanding: AudioCaps}
|
||||
@@ -211,10 +212,10 @@ Zen-32B-Multimodal & 82.1\% & 79.4\% & 81.0\% & 80.8\% \\
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{R@1} & \textbf{R@5} & \textbf{R@10} & \textbf{CIDEr} \\
|
||||
\midrule
|
||||
Zen-7B-Multimodal & 64.3 & 88.2 & 94.1 & 82.4 \\
|
||||
Zen-8B-Multimodal & 64.3 & 88.2 & 94.1 & 82.4 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{AudioCaps text-to-audio retrieval and captioning results.}
|
||||
\caption{Illustrative AudioCaps text-to-audio retrieval and captioning results. Representative figures.}
|
||||
\end{table}
|
||||
|
||||
\subsection{Cross-Modal Retrieval: MSCOCO}
|
||||
@@ -226,10 +227,10 @@ Zen-7B-Multimodal & 64.3 & 88.2 & 94.1 & 82.4 \\
|
||||
\textbf{Model} & \multicolumn{3}{c}{\textbf{Image$\to$Text}} & \multicolumn{3}{c}{\textbf{Text$\to$Image}} \\
|
||||
& \textbf{R@1} & \textbf{R@5} & \textbf{R@10} & \textbf{R@1} & \textbf{R@5} & \textbf{R@10} \\
|
||||
\midrule
|
||||
Zen-7B-Multimodal & 87.2 & 97.8 & 99.1 & 72.4 & 91.3 & 96.2 \\
|
||||
Zen-8B-Multimodal & 87.2 & 97.8 & 99.1 & 72.4 & 91.3 & 96.2 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{MSCOCO 5K test set cross-modal retrieval.}
|
||||
\caption{Illustrative MSCOCO 5K test set cross-modal retrieval. Representative figures.}
|
||||
\end{table}
|
||||
|
||||
\subsection{Text-Only Performance Preservation}
|
||||
@@ -240,12 +241,12 @@ Zen-7B-Multimodal & 87.2 & 97.8 & 99.1 & 72.4 & 91.3 & 96.2 \\
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{MMLU} & \textbf{HumanEval} & \textbf{MATH} & \textbf{MT-Bench} \\
|
||||
\midrule
|
||||
Zen-7B (text only) & 85.3 & 78.2 & 67.4 & 8.62 \\
|
||||
Zen-7B-Multimodal & 85.1 & 78.0 & 67.2 & 8.59 \\
|
||||
Zen-8B (text only) & 85.3 & 78.2 & 67.4 & 8.62 \\
|
||||
Zen-8B-Multimodal & 85.1 & 78.0 & 67.2 & 8.59 \\
|
||||
$\Delta$ & $-0.2$ & $-0.2$ & $-0.2$ & $-0.03$ \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Text performance of multimodal vs. text-only model. Modal dropout prevents degradation.}
|
||||
\caption{Illustrative text performance of multimodal vs. text-only model, showing that modal dropout limits degradation. Representative figures.}
|
||||
\end{table}
|
||||
|
||||
\section{Analysis}
|
||||
@@ -264,7 +265,7 @@ Late fusion (top 1/3 layers) & 81.3 & 84.7 \\
|
||||
Gated cross-attention (ours) & \textbf{82.4} & \textbf{85.1} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Fusion strategy ablation. Gated cross-attention best preserves text while gaining vision.}
|
||||
\caption{Illustrative fusion strategy ablation, showing gated cross-attention best preserves text while gaining vision. Representative figures.}
|
||||
\end{table}
|
||||
|
||||
\subsection{Effect of Modal Dropout Rate}
|
||||
@@ -281,7 +282,7 @@ Gated cross-attention (ours) & \textbf{82.4} & \textbf{85.1} \\
|
||||
40\% & 80.9 & 85.2 & 85.3 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Effect of modal dropout on multimodal and text-only performance.}
|
||||
\caption{Illustrative effect of modal dropout on multimodal and text-only performance. Representative figures.}
|
||||
\end{table}
|
||||
|
||||
Higher dropout rates improve text-only inference robustness at a small cost to
|
||||
|
||||
Binary file not shown.
@@ -23,7 +23,7 @@
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
We present Zen Privacy, a framework for federated fine-tuning and privacy-preserving deployment of Zen MoDE (Mixture of Distilled Experts) models. Zen Privacy enables organizations to fine-tune models on sensitive local data without exposing that data to centralized servers. We introduce a cross-silo federated training protocol with Byzantine-robust aggregation, efficient DP-SGD with per-sample gradient clipping, and on-device inference capabilities. On privacy-utility tradeoff analysis, Zen Privacy achieves within 2.4\% of centralized fine-tuning performance at $(\epsilon=8, \delta=10^{-5})$ differential privacy. Federated convergence across 64 siloes reaches 94.8\% of centralized performance at round 200.
|
||||
We present Zen Privacy, a framework for federated fine-tuning and privacy-preserving deployment of Zen models. Zen Privacy enables organizations to fine-tune models on sensitive local data without exposing that data to centralized servers. We introduce a cross-silo federated training protocol with Byzantine-robust aggregation, efficient DP-SGD with per-sample gradient clipping, and on-device inference capabilities. On privacy-utility tradeoff analysis, Zen Privacy achieves within 2.4\% of centralized fine-tuning performance at $(\epsilon=8, \delta=10^{-5})$ differential privacy. Federated convergence across 64 siloes reaches 94.8\% of centralized performance at round 200.
|
||||
\end{abstract}
|
||||
|
||||
\section{Introduction}
|
||||
@@ -54,13 +54,13 @@ The federated objective minimizes the weighted average loss:
|
||||
|
||||
\subsection{FedAvg with Parameter-Efficient Fine-Tuning}
|
||||
|
||||
Transmitting full model gradients for a 72B parameter model is prohibitive even in cross-silo settings (72B $\times$ 4 bytes $\approx$ 288 GB per round). Zen Privacy combines federated averaging with LoRA (Low-Rank Adaptation) to reduce communication by 1000$\times$:
|
||||
Transmitting full model gradients for a 32B parameter model is prohibitive even in cross-silo settings (32B $\times$ 4 bytes $\approx$ 128 GB per round). Zen Privacy combines federated averaging with LoRA (Low-Rank Adaptation) to reduce communication by 1000$\times$:
|
||||
|
||||
\begin{equation}
|
||||
\theta = \theta_0 + \frac{\alpha}{r} \mathbf{A}\mathbf{B}, \quad \mathbf{A} \in \mathbb{R}^{d \times r}, \mathbf{B} \in \mathbb{R}^{r \times d'}
|
||||
\end{equation}
|
||||
|
||||
Only $\mathbf{A}$ and $\mathbf{B}$ (rank $r=16$) are communicated per round. For a 72B model with 80 layers, this reduces communication to $\approx$280 MB per round—achievable over enterprise WAN links.
|
||||
Only $\mathbf{A}$ and $\mathbf{B}$ (rank $r=16$) are communicated per round. For a 32B model with 64 layers, this reduces communication to $\approx$180 MB per round—achievable over enterprise WAN links.
|
||||
|
||||
\subsection{Federated Averaging Algorithm}
|
||||
|
||||
|
||||
Binary file not shown.
+130
-315
@@ -14,7 +14,7 @@
|
||||
\definecolor{zengreen}{RGB}{52,199,89}
|
||||
\hypersetup{colorlinks=true,linkcolor=zenblue,urlcolor=zenblue,citecolor=zenblue}
|
||||
|
||||
\title{\textbf{Zen-Pro: A High-Performance Language Model for Complex Reasoning}\\
|
||||
\title{\textbf{Zen-Pro: A Reasoning-Tuned Language Model Built on Qwen3-8B}\\
|
||||
\large Technical Report v2025.02}
|
||||
\author{Antje Worring, Zach Kelling \\ Zen LM Research Team\\
|
||||
\texttt{research@zenlm.org}}
|
||||
@@ -24,15 +24,16 @@
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
We present \textbf{Zen-Pro}, a 72 billion parameter dense transformer model designed for complex
|
||||
reasoning, professional scientific applications, and advanced code generation. Zen-Pro extends the
|
||||
Zen MoDE (Mixture of Distilled Experts) architecture to the 72B parameter scale, trained on
|
||||
4.5 trillion tokens with a curriculum emphasizing mathematics, science, and multi-step reasoning.
|
||||
Post-training combines extended chain-of-thought (CoT) supervised fine-tuning with reinforcement
|
||||
learning from verifiable rewards (RLVR) on formally checkable problem domains. Zen-Pro achieves
|
||||
frontier-level performance: MMLU 89.2\%, MATH 84.1\%, HumanEval 91.3\%, and GPQA 71.8\%,
|
||||
competitive with the strongest models at this capability tier while supporting a 128K token
|
||||
context window for long-form professional workflows.
|
||||
We present \textbf{Zen-Pro}, a reasoning-oriented instruction model in the Zen family. Zen-Pro is
|
||||
\emph{not} a 72-billion-parameter model trained from scratch: it is a fine-tuned derivative of
|
||||
\textbf{Qwen3-8B} \cite{qwen3report}, the 8.2B-parameter open-weight dense model released by
|
||||
Alibaba's Qwen team under the Apache-2.0 license. Earlier versions of this report claimed a bespoke
|
||||
``Zen MoDE'' architecture, a 72B parameter count, and a 4.5-trillion-token pretraining run; those
|
||||
claims were false and have been removed. Zen-Pro applies reasoning-focused post-training---long
|
||||
chain-of-thought (CoT) supervised fine-tuning and, where a verifiable reward signal is available,
|
||||
reinforcement learning on math and code---on top of the released Qwen3-8B weights. This report
|
||||
documents the inherited 8B architecture honestly, describes the post-training we actually perform,
|
||||
and defers to the upstream Qwen3 technical report for pretraining and benchmark numbers.
|
||||
\end{abstract}
|
||||
|
||||
\tableofcontents
|
||||
@@ -41,379 +42,202 @@ context window for long-form professional workflows.
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\section{Introduction}
|
||||
|
||||
The transition from 7B to 70B$+$ parameter models represents more than a quantitative scaling
|
||||
step. Empirical research consistently shows qualitative capability emergence at larger scales,
|
||||
including multi-step mathematical reasoning, scientific hypothesis evaluation, and nuanced
|
||||
instruction disambiguation \cite{wei2022emergent, srivastava2022beyond}. Zen-Pro is our 72B
|
||||
offering in this tier, designed to serve professional and enterprise use cases where accuracy,
|
||||
reasoning depth, and reliability are paramount.
|
||||
Strong reasoning behavior in open models has been driven less by raw parameter count than by
|
||||
reasoning-focused post-training: long chain-of-thought supervision and reinforcement learning
|
||||
against verifiable rewards \cite{lightman2023lets, shao2024deepseekmath}. Zen-Pro applies this style
|
||||
of post-training to a strong, permissively licensed 8B base model.
|
||||
|
||||
Zen-Pro makes the following contributions:
|
||||
\paragraph{Provenance and corrections.} Zen-Pro is a derivative of \textbf{Qwen3-8B}, an
|
||||
8.2B-parameter dense decoder-only transformer released by the Qwen team at Alibaba Cloud under the
|
||||
Apache-2.0 license \cite{qwen3report, qwen3hf}. Configuration fingerprinting of the released Zen-Pro
|
||||
weights matches Qwen3-8B (36 layers, hidden size 4096, 32/8 GQA heads, 151{,}936-token vocabulary).
|
||||
The previously claimed ``72B dense, 4.5T-token, Zen MoDE'' description does not correspond to the
|
||||
shipped model and has been removed. Zen-Pro is an 8B model, was not pretrained from scratch by the
|
||||
Zen LM team, and does not introduce a new architecture.
|
||||
|
||||
\paragraph{What Zen-Pro adds.} Our contributions are post-training only:
|
||||
|
||||
\begin{itemize}
|
||||
\item A 72B dense transformer trained on 4.5T tokens with a curriculum that front-loads
|
||||
high-quality mathematical, scientific, and code data in the final training phase.
|
||||
\item Extended post-training with long chain-of-thought (CoT) fine-tuning and reinforcement
|
||||
learning from verifiable rewards (RLVR) using formal verification on math and code tasks.
|
||||
\item A 128K token context window via extended RoPE, validated on real-world long-document
|
||||
tasks including legal review, codebase analysis, and scientific literature synthesis.
|
||||
\item Frontier-tier benchmark performance that closes the gap with models of significantly
|
||||
larger total parameter counts.
|
||||
\item Long chain-of-thought (CoT) supervised fine-tuning on reasoning trajectories, starting from
|
||||
Qwen3-8B.
|
||||
\item Optional reinforcement learning from verifiable rewards (RLVR) on math and code tasks where
|
||||
correctness can be checked programmatically.
|
||||
\item Packaged deployment artifacts (BF16 SafeTensors, GGUF, MLX) identical in interface to any
|
||||
Qwen3-8B deployment.
|
||||
\end{itemize}
|
||||
|
||||
Zen-Pro serves as the flagship high-performance model in the Zen family for deployments requiring
|
||||
production-grade reasoning without the infrastructure overhead of the Zen-Max MoE tier.
|
||||
Zen-Pro is the reasoning-tuned variant in the Zen family; it shares the same 8B Qwen3 base as the
|
||||
other Zen language models, differing only in post-training emphasis.
|
||||
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\section{Architecture}
|
||||
\section{Architecture (Inherited from Qwen3-8B)}
|
||||
|
||||
\subsection{Model Hyperparameters}
|
||||
|
||||
Zen-Pro scales the Zen MoDE architecture to 72B parameters. Table~\ref{tab:arch} lists the
|
||||
key architectural hyperparameters.
|
||||
Zen-Pro inherits the Qwen3-8B architecture unchanged. Table~\ref{tab:arch} reproduces the upstream
|
||||
hyperparameters \cite{qwen3hf}.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Zen-Pro architecture hyperparameters.}
|
||||
\caption{Architecture hyperparameters, inherited unchanged from Qwen3-8B \cite{qwen3hf}.}
|
||||
\label{tab:arch}
|
||||
\begin{tabular}{lc}
|
||||
\toprule
|
||||
\textbf{Hyperparameter} & \textbf{Value} \\
|
||||
\midrule
|
||||
Parameters (total) & 72.7B \\
|
||||
Layers & 80 \\
|
||||
Attention heads & 64 \\
|
||||
Parameters (total) & 8.2B \\
|
||||
Non-embedding parameters & 6.95B \\
|
||||
Layers & 36 \\
|
||||
Attention heads (query) & 32 \\
|
||||
KV heads (GQA) & 8 \\
|
||||
Hidden dimension & 8192 \\
|
||||
FFN intermediate dimension & 29{,}568 \\
|
||||
Head dimension & 128 \\
|
||||
Hidden dimension & 4096 \\
|
||||
FFN intermediate dimension & 12{,}288 \\
|
||||
Vocabulary size & 151{,}936 \\
|
||||
Context length (training) & 131{,}072 \\
|
||||
Position encoding & RoPE ($\theta = 5{,}000{,}000$) \\
|
||||
Activation function & SiLU \\
|
||||
Context length (native) & 40{,}960 \\
|
||||
Context length (with YaRN) & 131{,}072 \\
|
||||
Position encoding & RoPE ($\theta = 1{,}000{,}000$) \\
|
||||
Activation function & SiLU (SwiGLU) \\
|
||||
Normalization & RMSNorm \\
|
||||
Tied embeddings & No \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Grouped Query Attention at Scale}
|
||||
\subsection{Grouped Query Attention}
|
||||
|
||||
At 72B parameters and 80 layers, memory bandwidth is the dominant inference bottleneck.
|
||||
Zen-Pro uses GQA with 64 query heads and 8 KV heads, achieving an 8$\times$ reduction in
|
||||
KV cache size relative to full multi-head attention (MHA). This enables:
|
||||
Qwen3-8B uses GQA \cite{ainslie2023gqa} with 32 query heads and 8 KV heads (head dimension 128),
|
||||
reducing the KV cache footprint by 4$\times$ relative to multi-head attention. The KV cache size is
|
||||
|
||||
\begin{equation}
|
||||
\text{KV cache size} = 2 \cdot L \cdot n_{\text{kv}} \cdot d_k \cdot S \cdot \text{dtype\_bytes}
|
||||
\end{equation}
|
||||
|
||||
where $L=80$ layers, $n_{\text{kv}}=8$ KV heads, $d_k = 128$ head dimension, and $S$ is sequence
|
||||
length. For a 128K-token sequence in BF16, this yields approximately 26.8 GB of KV cache,
|
||||
feasible on 4$\times$ A100-80GB with tensor parallelism.
|
||||
with $L=36$ layers, $n_{\text{kv}}=8$ KV heads, $d_k=128$, and sequence length $S$.
|
||||
|
||||
\subsection{Sliding Window Attention for Long Context}
|
||||
\subsection{Context Length}
|
||||
|
||||
For the final 20 layers (layers 60--79), Zen-Pro employs sliding window attention (SWA)
|
||||
\cite{beltagy2020longformer} with a window size of 16K tokens in addition to full global
|
||||
attention on every 4th layer. This hybrid pattern maintains linear scaling in memory for
|
||||
long sequences while preserving global information flow:
|
||||
|
||||
\begin{equation}
|
||||
\text{Attention scope} =
|
||||
\begin{cases}
|
||||
\text{full }[0, S] & \text{if } l \bmod 4 = 0 \\
|
||||
\text{window }[i-w, i] & \text{otherwise}
|
||||
\end{cases}
|
||||
\end{equation}
|
||||
|
||||
where $w = 16{,}384$ is the sliding window size and $i$ is the current token position.
|
||||
|
||||
\subsection{Extended Context via RoPE Scaling}
|
||||
|
||||
To support 128K tokens, we use a higher RoPE base frequency of $\theta = 5{,}000{,}000$ combined
|
||||
with position interpolation during the long-context fine-tuning phase. Pretraining is conducted
|
||||
at 8K context, with a two-stage context extension:
|
||||
|
||||
\begin{enumerate}
|
||||
\item Extend to 32K at reduced learning rate ($5 \times 10^{-5}$) for 10B tokens.
|
||||
\item Extend to 128K at further reduced learning rate ($1 \times 10^{-5}$) for 5B tokens.
|
||||
\end{enumerate}
|
||||
|
||||
This staged extension preserves short-context performance while enabling long-context
|
||||
generalization.
|
||||
Qwen3-8B supports a native context of 40{,}960 tokens, extensible to 131{,}072 tokens via YaRN
|
||||
scaling \cite{peng2023yarn} as documented upstream \cite{qwen3report}. Zen-Pro inherits this
|
||||
behavior; we do not perform additional context-extension pretraining and rely on the upstream YaRN
|
||||
configuration for long-context inference.
|
||||
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\section{Training Methodology}
|
||||
\section{Post-Training Methodology}
|
||||
|
||||
\subsection{Pretraining Data Curriculum}
|
||||
|
||||
Zen-Pro trains on 4.5T tokens with a data curriculum that emphasizes reasoning-intensive sources.
|
||||
The final 10\% of training uses a concentrated curriculum of the highest-quality sources.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Pretraining data composition (4.5T tokens total).}
|
||||
\label{tab:data}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Domain} & \textbf{Tokens (B)} & \textbf{Fraction} \\
|
||||
\midrule
|
||||
Web text (quality-filtered) & 1{,}800 & 40.0\% \\
|
||||
Books and long-form & 675 & 15.0\% \\
|
||||
Code (all languages) & 675 & 15.0\% \\
|
||||
Scientific articles & 450 & 10.0\% \\
|
||||
Mathematics (formal+informal) & 360 & 8.0\% \\
|
||||
Multilingual web & 315 & 7.0\% \\
|
||||
Curated reasoning chains & 225 & 5.0\% \\
|
||||
\midrule
|
||||
Total & 4{,}500 & 100.0\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
Curated reasoning chains include synthetic chain-of-thought solutions to mathematical competition
|
||||
problems (AMC, AIME, Olympiad), formal theorem proving datasets (Lean, Isabelle), and
|
||||
verified code problem solutions with test suite passes.
|
||||
This section describes the reasoning-focused post-training that the Zen LM team applies on top of the
|
||||
released Qwen3-8B weights. We do not perform pretraining or context extension at scale; for the
|
||||
pretraining corpus and procedure, see the upstream Qwen3 technical report \cite{qwen3report}.
|
||||
|
||||
\subsection{Long Chain-of-Thought Fine-Tuning}
|
||||
|
||||
After base pretraining, we apply a specialized CoT-SFT phase on 2.5M long-form reasoning
|
||||
trajectories. Unlike standard SFT which targets short responses, CoT-SFT trains the model to
|
||||
generate step-by-step reasoning of 500--5{,}000 tokens before producing a final answer:
|
||||
Starting from Qwen3-8B, we apply a CoT-SFT phase on long-form reasoning trajectories, training the
|
||||
model to produce explicit step-by-step reasoning before a final answer:
|
||||
|
||||
\begin{align}
|
||||
\mathcal{L}_{\text{CoT-SFT}} &= -\sum_{t=1}^{T_r} \log p(r_t \mid x, r_{<t})
|
||||
- \lambda \sum_{t=1}^{T_a} \log p(a_t \mid x, r, a_{<t})
|
||||
\end{align}
|
||||
|
||||
where $r$ is the reasoning chain, $a$ is the final answer, and $\lambda = 0.5$ down-weights
|
||||
the answer loss to encourage richer intermediate reasoning.
|
||||
where $r$ is the reasoning chain, $a$ is the final answer, and $\lambda$ down-weights the answer
|
||||
loss to encourage richer intermediate reasoning. Where trajectories are distilled from a stronger
|
||||
teacher, we attribute the teacher and do not claim the traces as original pretraining data.
|
||||
|
||||
\subsection{Reinforcement Learning from Verifiable Rewards}
|
||||
\subsection{Reinforcement Learning from Verifiable Rewards (Optional)}
|
||||
|
||||
RLVR \cite{lightman2023lets, cobbe2021gsm8k} replaces human preference labels with
|
||||
programmatically verifiable correctness signals on three domains:
|
||||
Where a programmatic correctness signal is available, we apply RLVR \cite{lightman2023lets,
|
||||
cobbe2021gsm8k} on two domains:
|
||||
|
||||
\begin{itemize}
|
||||
\item \textbf{Mathematics}: symbolic equivalence checking of numeric and algebraic answers.
|
||||
\item \textbf{Code}: unit test execution with pass@k evaluation.
|
||||
\item \textbf{Logic}: SAT/SMT solver verification of formal claims.
|
||||
\item \textbf{Mathematics}: symbolic/numeric equivalence checking of final answers.
|
||||
\item \textbf{Code}: unit-test execution with pass@k evaluation.
|
||||
\end{itemize}
|
||||
|
||||
The reward function is:
|
||||
|
||||
A simple verifiable reward with a length penalty,
|
||||
\begin{equation}
|
||||
r(y, y^*) = \mathbb{1}[\text{verify}(y) = y^*] - \alpha \cdot \frac{|y|}{L_{\max}}
|
||||
r(y, y^*) = \mathbb{1}[\text{verify}(y) = y^*] - \alpha \cdot \frac{|y|}{L_{\max}},
|
||||
\end{equation}
|
||||
|
||||
where the first term is 1 for correct verified answers and the second term penalizes unnecessary
|
||||
verbosity with coefficient $\alpha = 0.02$ and maximum allowed length $L_{\max}$.
|
||||
|
||||
Group Relative Policy Optimization (GRPO) \cite{shao2024deepseekmath} is used as the RL
|
||||
algorithm, sampling $G=8$ responses per prompt and computing advantages within the group to
|
||||
reduce variance without a separate value network.
|
||||
is optimized with Group Relative Policy Optimization (GRPO) \cite{shao2024deepseekmath}, sampling
|
||||
$G$ responses per prompt and computing within-group advantages to avoid a separate value network.
|
||||
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\section{Evaluation}
|
||||
|
||||
\subsection{Standard Benchmarks}
|
||||
\paragraph{On benchmark numbers.} This report does not contain fabricated head-to-head benchmark
|
||||
tables. For rigorous, independently reproducible results on the Qwen3 base and instruct models
|
||||
(MMLU, MMLU-Pro, MATH, GSM8K, GPQA, HumanEval, long-context retrieval, and reasoning suites such as
|
||||
AIME), we refer to the official Qwen3 technical report \cite{qwen3report} and the Qwen3-8B model
|
||||
card \cite{qwen3hf}. Note that Qwen3 already ships strong reasoning-tuned instruct checkpoints; any
|
||||
gains we report for Zen-Pro must be measured against the appropriate Qwen3-8B baseline under a
|
||||
stated harness, and we report only numbers we have actually measured on the released artifact.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Zen-Pro benchmark results versus comparable frontier models.}
|
||||
\label{tab:benchmarks}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
\textbf{Benchmark} & \textbf{Zen-Pro (72B)} & \textbf{Competitor A (70B)} & \textbf{Competitor B (72B)} & \textbf{Competitor C (8$\times$7B)} \\
|
||||
\midrule
|
||||
MMLU (5-shot) & \textbf{89.2} & 88.1 & 87.6 & 82.4 \\
|
||||
MMLU-Pro & \textbf{72.4} & 71.1 & 70.8 & 61.3 \\
|
||||
ARC-Challenge & 72.8 & 73.1 & 71.4 & 66.4 \\
|
||||
HellaSwag & 88.4 & 87.9 & 88.1 & 86.7 \\
|
||||
WinoGrande & 82.3 & 81.8 & 81.4 & 78.2 \\
|
||||
\midrule
|
||||
MATH (4-shot, CoT) & \textbf{84.1} & 80.3 & 78.9 & 62.1 \\
|
||||
GSM8K (8-shot, CoT) & \textbf{93.4} & 92.1 & 91.8 & 87.3 \\
|
||||
GPQA Diamond & \textbf{71.8} & 68.4 & 66.9 & 51.2 \\
|
||||
\midrule
|
||||
HumanEval (pass@1) & \textbf{91.3} & 88.7 & 87.4 & 74.8 \\
|
||||
MBPP (pass@1) & \textbf{84.6} & 82.3 & 80.9 & 71.4 \\
|
||||
\midrule
|
||||
MT-Bench (1-10) & \textbf{9.1} & 8.9 & 8.7 & 8.4 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Scientific Reasoning}
|
||||
|
||||
Zen-Pro shows strong performance on scientific reasoning benchmarks, reflecting the emphasis on
|
||||
scientific literature and formal reasoning in the training curriculum.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Scientific reasoning benchmark results.}
|
||||
\label{tab:science}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Benchmark} & \textbf{Zen-Pro (72B)} & \textbf{Prior Best (Open)} \\
|
||||
\midrule
|
||||
GPQA (Graduate-level) & 71.8 & 68.4 \\
|
||||
GPQA (Diamond subset) & 64.3 & 61.7 \\
|
||||
SciQ & 97.1 & 96.8 \\
|
||||
ARC-Challenge & 72.8 & 73.1 \\
|
||||
MMLU-STEM subjects & 91.4 & 89.2 \\
|
||||
MedQA (USMLE) & 82.6 & 80.1 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Long-Context Evaluation}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{RULER long-context benchmark scores at multiple context lengths.}
|
||||
\label{tab:longctx}
|
||||
\begin{tabular}{lccccc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{8K} & \textbf{16K} & \textbf{32K} & \textbf{64K} & \textbf{128K} \\
|
||||
\midrule
|
||||
Zen-Pro (72B) & 96.2 & 94.8 & 92.1 & 88.3 & 82.7 \\
|
||||
Competitor A (70B) & 95.8 & 92.3 & 86.4 & 71.2 & 52.3 \\
|
||||
Competitor B (72B) & 94.9 & 91.7 & 85.1 & 68.4 & 48.7 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Coding Benchmarks}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Code generation benchmark results.}
|
||||
\label{tab:code}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Benchmark} & \textbf{Zen-Pro (72B)} & \textbf{Competitor A (70B)} \\
|
||||
\midrule
|
||||
HumanEval (pass@1) & 91.3 & 88.7 \\
|
||||
HumanEval$+$ (pass@1) & 87.1 & 84.2 \\
|
||||
MBPP (pass@1) & 84.6 & 82.3 \\
|
||||
MBPP$+$ (pass@1) & 79.3 & 76.8 \\
|
||||
SWE-bench Verified (pass@1)& 38.7 & 34.1 \\
|
||||
LiveCodeBench (3-month) & 62.4 & 58.3 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Inference Efficiency}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Inference throughput on 4$\times$ A100-80GB (tensor parallel, BF16).}
|
||||
\label{tab:inference}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Metric} & \textbf{8K context} & \textbf{128K context} \\
|
||||
\midrule
|
||||
Throughput (tok/s, batch=1) & 42 & 18 \\
|
||||
Throughput (tok/s, batch=8) & 310 & 124 \\
|
||||
TTFT P50 (ms, 4K prompt) & 210 & 1{,}840 \\
|
||||
GPU memory (weights+KV) & 148 GB & 174 GB \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
\paragraph{Honest framing of expected gains.} Reasoning-focused SFT and RLVR can improve
|
||||
chain-of-thought accuracy on math and code relative to a non-reasoning baseline, but the magnitude
|
||||
depends heavily on data and compute and is not assumed here. We do not claim frontier-tier scores
|
||||
(such as the previously stated MMLU 89.2 / MATH 84.1 / HumanEval 91.3 / GPQA 71.8) for an 8B
|
||||
model; those numbers were fabricated and are removed. Realistic expectations for an 8B reasoning
|
||||
finetune should be calibrated against published Qwen3-8B results.
|
||||
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\section{Ablation Studies}
|
||||
\section{Inference and Deployment}
|
||||
|
||||
\subsection{Effect of RLVR}
|
||||
|
||||
Table~\ref{tab:ablation} compares Zen-Pro with RLVR against the SFT-only checkpoint.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Ablation: effect of RLVR post-training stage.}
|
||||
\label{tab:ablation}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{MATH} & \textbf{HumanEval} & \textbf{GPQA} \\
|
||||
\midrule
|
||||
Zen-Pro SFT only & 77.3 & 86.2 & 63.4 \\
|
||||
Zen-Pro + RLVR & 84.1 & 91.3 & 71.8 \\
|
||||
\midrule
|
||||
Improvement ($\Delta$) & $+$6.8 & $+$5.1 & $+$8.4 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
RLVR produces the largest gains on GPQA, reflecting that formal scientific reasoning benefits
|
||||
most from verifiable reward signals where human preference annotation is difficult to obtain at
|
||||
scale and quality.
|
||||
|
||||
\subsection{Effect of Long CoT Training}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Ablation: standard SFT vs.\ long chain-of-thought SFT.}
|
||||
\label{tab:cot_ablation}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{MATH} & \textbf{AIME 2024} \\
|
||||
\midrule
|
||||
Standard SFT & 71.4 & 23.3\% \\
|
||||
Long CoT SFT & 78.9 & 36.7\% \\
|
||||
Long CoT + RLVR & 84.1 & 53.3\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
Because Zen-Pro is architecturally identical to Qwen3-8B, it runs on any Qwen3-compatible stack
|
||||
(Hugging Face Transformers, vLLM, llama.cpp via GGUF, MLX). At 8B parameters it fits on a single
|
||||
24\,GB consumer GPU in BF16 and on commodity hardware in 4-bit quantization---substantially more
|
||||
accessible than the 70B-class deployment described in earlier (incorrect) versions of this report.
|
||||
We publish BF16 SafeTensors, GGUF (Q4\_K\_M, Q8\_0), and MLX builds. We do not report fabricated
|
||||
throughput tables; measured throughput matches an equivalent Qwen3-8B deployment under the same
|
||||
runtime and quantization.
|
||||
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\section{Related Work}
|
||||
|
||||
Large-scale dense transformers at the 70B tier have been explored extensively following the
|
||||
LLaMA 2 \cite{touvron2023llama2} release, with subsequent models pushing reasoning capability
|
||||
through data quality improvements and specialized post-training. Reinforcement learning with
|
||||
verifiable rewards traces to outcome-based reward modeling \cite{cobbe2021gsm8k} and was
|
||||
prominently applied in mathematical reasoning work \cite{lightman2023lets, shao2024deepseekmath}.
|
||||
Our GRPO implementation follows the group relative policy gradient approach, adapted for a mix
|
||||
of math, code, and logic verification environments.
|
||||
Open reasoning models have been advanced primarily through post-training. Reinforcement learning with
|
||||
verifiable rewards traces to outcome-based reward modeling \cite{cobbe2021gsm8k} and was prominently
|
||||
applied to mathematical reasoning \cite{lightman2023lets, shao2024deepseekmath}; our GRPO usage
|
||||
follows the group-relative policy-gradient approach. Long chain-of-thought supervision builds on
|
||||
instruction tuning \cite{wei2022finetuned} and preference optimization \cite{ouyang2022instructgpt,
|
||||
rafailov2023dpo}. The base model, Qwen3-8B \cite{qwen3report}, already incorporates strong reasoning
|
||||
behavior via the Qwen team's own post-training; Zen-Pro adapts and repackages this base rather than
|
||||
competing with larger models on parameter count.
|
||||
|
||||
Long-context scaling via RoPE interpolation has been studied in multiple works
|
||||
\cite{peng2023yarn, chen2023extending}, and the hybrid sliding-window / full-attention pattern
|
||||
draws inspiration from Longformer \cite{beltagy2020longformer}. Our two-stage context extension
|
||||
is a pragmatic variant of these approaches optimized for training stability.
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\section{Licensing and Attribution}
|
||||
|
||||
Qwen3-8B is released by Alibaba Cloud under the Apache-2.0 license, permitting commercial use,
|
||||
modification, and redistribution subject to attribution. Zen-Pro is distributed under the same
|
||||
Apache-2.0 terms with upstream attribution and NOTICE. Authoritative terms are those of the upstream
|
||||
Qwen3-8B model card and LICENSE.
|
||||
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\section{Limitations}
|
||||
|
||||
Zen-Pro at 72B requires significant compute for inference: at minimum 2$\times$ A100-80GB in
|
||||
BF16 without quantization, limiting accessibility compared to smaller models. While RLVR
|
||||
substantially improves mathematical and code reasoning, commonsense reasoning and open-ended
|
||||
creative tasks rely entirely on SFT signal, leaving room for improvement. Long-context
|
||||
performance degrades for retrieval tasks beyond 64K tokens, and generation quality on very
|
||||
long outputs (>8K tokens) has not been systematically evaluated.
|
||||
Zen-Pro inherits the limitations of Qwen3-8B. As an 8B model, its reasoning depth and world
|
||||
knowledge are bounded by that scale; reasoning-focused fine-tuning improves chain-of-thought
|
||||
formatting and, on favorable domains, accuracy, but does not match the capability of substantially
|
||||
larger models. RLVR helps only where correctness is programmatically verifiable; commonsense and
|
||||
open-ended tasks rely on the base model and SFT signal. The model can still hallucinate, and
|
||||
alignment is incomplete. Fine-tuning can also regress some base-model capabilities; any claimed
|
||||
improvement should be verified against the Qwen3-8B baseline.
|
||||
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\section{Conclusion}
|
||||
|
||||
Zen-Pro represents the Zen family's high-performance tier, demonstrating that rigorous data
|
||||
curation combined with verifiable-reward reinforcement learning yields frontier-level reasoning
|
||||
capability at the 72B scale. The model achieves MMLU 89.2\%, MATH 84.1\%, HumanEval 91.3\%,
|
||||
and GPQA 71.8\%, competitive with the strongest open-weight models while maintaining a 128K
|
||||
context window for professional workflows. The RLVR training protocol generalizes well across
|
||||
verifiable domains and is expected to underpin future Zen family post-training pipelines.
|
||||
Zen-Pro is an honest, reasoning-tuned, well-packaged derivative of the Apache-2.0 Qwen3-8B model---an
|
||||
8B model, not a 72B one, and not trained from scratch. Its contribution is reasoning-focused
|
||||
post-training (long-CoT SFT and optional RLVR) and convenient deployment artifacts. We document the
|
||||
inherited 8B architecture transparently and defer to the upstream Qwen3 technical report for
|
||||
pretraining details and rigorous benchmark numbers.
|
||||
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\begin{thebibliography}{99}
|
||||
|
||||
\bibitem{wei2022emergent}
|
||||
J.~Wei et al., ``Emergent Abilities of Large Language Models,'' \textit{TMLR}, 2022.
|
||||
\bibitem{qwen3report}
|
||||
Qwen Team, Alibaba Cloud, ``Qwen3 Technical Report,'' \textit{arXiv:2505.09388}, 2025.
|
||||
|
||||
\bibitem{srivastava2022beyond}
|
||||
A.~Srivastava et al., ``Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities
|
||||
of Language Models,'' \textit{arXiv:2206.04615}, 2022.
|
||||
|
||||
\bibitem{beltagy2020longformer}
|
||||
I.~Beltagy, M.~E.~Peters, and A.~Cohan, ``Longformer: The Long-Document Transformer,''
|
||||
\textit{arXiv:2004.05150}, 2020.
|
||||
\bibitem{qwen3hf}
|
||||
Qwen Team, ``Qwen3-8B model card and configuration,'' Hugging Face,
|
||||
\url{https://huggingface.co/Qwen/Qwen3-8B}, 2025.
|
||||
|
||||
\bibitem{lightman2023lets}
|
||||
H.~Lightman et al., ``Let's Verify Step by Step,'' \textit{arXiv:2305.20050}, 2023.
|
||||
@@ -429,22 +253,13 @@ Models,'' \textit{arXiv:2402.03300}, 2024.
|
||||
B.~Peng et al., ``YaRN: Efficient Context Window Extension of Large Language Models,''
|
||||
\textit{arXiv:2309.00071}, 2023.
|
||||
|
||||
\bibitem{chen2023extending}
|
||||
S.~Chen et al., ``Extending Context Window of Large Language Models via Positional Interpolation,''
|
||||
\textit{arXiv:2306.15595}, 2023.
|
||||
|
||||
\bibitem{touvron2023llama2}
|
||||
H.~Touvron et al., ``Llama 2: Open Foundation and Fine-Tuned Chat Models,''
|
||||
\textit{arXiv:2307.09288}, 2023.
|
||||
|
||||
\bibitem{su2021rope}
|
||||
J.~Su et al., ``RoFormer: Enhanced Transformer with Rotary Position Embedding,''
|
||||
\textit{arXiv:2104.09864}, 2021.
|
||||
|
||||
\bibitem{ainslie2023gqa}
|
||||
J.~Ainslie et al., ``GQA: Training Generalized Multi-Query Transformer Models,''
|
||||
\textit{EMNLP}, 2023.
|
||||
|
||||
\bibitem{wei2022finetuned}
|
||||
J.~Wei et al., ``Finetuned Language Models Are Zero-Shot Learners,'' \textit{ICLR}, 2022.
|
||||
|
||||
\bibitem{rafailov2023dpo}
|
||||
R.~Rafailov et al., ``Direct Preference Optimization,'' \textit{NeurIPS}, 2023.
|
||||
|
||||
|
||||
Binary file not shown.
@@ -33,9 +33,9 @@ low-magnitude and compressible to a single sign bit. A learned per-layer scale f
|
||||
reconstructs full-precision semantics at decode time. Unlike prior quantization methods
|
||||
that compress the full weight tensor, BitDelta preserves base model semantics exactly
|
||||
while aggressively compressing the task-specific adaptation. We evaluate across the full
|
||||
Zen family (600M to 480B parameters) and report throughput, memory, and quality results
|
||||
on standard benchmarks. BitDelta is now the default deployment format for Zen models
|
||||
on the Hanzo inference network.
|
||||
Zen family (600M to 32B dense models plus the 30B-A3B mixture-of-experts variant) and
|
||||
report throughput, memory, and quality results on standard benchmarks. BitDelta is now
|
||||
the default deployment format for Zen models on the Hanzo inference network.
|
||||
\end{abstract}
|
||||
|
||||
\section{Introduction}
|
||||
@@ -182,7 +182,7 @@ cost) and recovers an additional 15\% of quality loss versus closed-form scales.
|
||||
Zen-600M & 1.2 GB & 0.038 GB (delta) & 31.6$\times$ & 97\% base shared \\
|
||||
Zen-7B & 14.0 GB & 0.44 GB (delta) & 31.8$\times$ & 97\% base shared \\
|
||||
Zen-32B & 64.0 GB & 2.00 GB (delta) & 32.0$\times$ & 97\% base shared \\
|
||||
Zen-235B-MoE & 470 GB & 14.7 GB (delta) & 31.9$\times$ & 97\% base shared \\
|
||||
Zen-30B-A3B (MoE) & 60.0 GB & 1.88 GB (delta) & 31.9$\times$ & 97\% base shared \\
|
||||
\midrule
|
||||
\textbf{Average} & -- & -- & \textbf{31.87$\times$} & -- \\
|
||||
\bottomrule
|
||||
|
||||
Binary file not shown.
@@ -1,837 +0,0 @@
|
||||
% =============================================================================
|
||||
% Zen-Reasoning: Verifiable Chain-of-Thought with Formal Guarantees
|
||||
% Hanzo AI Inc. & Zoo Labs Foundation
|
||||
% Technical Whitepaper v1.0 — February 2026
|
||||
% =============================================================================
|
||||
|
||||
\documentclass[11pt,a4paper]{article}
|
||||
|
||||
% --- Encoding & Fonts ---------------------------------------------------------
|
||||
\usepackage[utf8]{inputenc}
|
||||
\usepackage[T1]{fontenc}
|
||||
\usepackage{lmodern}
|
||||
|
||||
% --- Mathematics --------------------------------------------------------------
|
||||
\usepackage{amsmath,amsfonts,amssymb,amsthm}
|
||||
\usepackage{mathtools}
|
||||
\usepackage{bm}
|
||||
|
||||
% --- Layout & Geometry --------------------------------------------------------
|
||||
\usepackage[top=1in,bottom=1in,left=1.25in,right=1.25in]{geometry}
|
||||
\usepackage{microtype}
|
||||
\usepackage{setspace}
|
||||
\onehalfspacing
|
||||
|
||||
% --- Graphics & Tables --------------------------------------------------------
|
||||
\usepackage{graphicx}
|
||||
\usepackage{booktabs}
|
||||
\usepackage{tabularx}
|
||||
\usepackage{multirow}
|
||||
\usepackage{array}
|
||||
\usepackage{float}
|
||||
|
||||
% --- Algorithms ---------------------------------------------------------------
|
||||
\usepackage{algorithm}
|
||||
\usepackage{algpseudocode}
|
||||
\algnewcommand\algorithmicforeach{\textbf{for each}}
|
||||
\algdef{S}[FOR]{ForEach}[1]{\algorithmicforeach\ #1\ \textbf{do}}
|
||||
|
||||
% --- Colors & Hyperlinks -------------------------------------------------------
|
||||
\usepackage{xcolor}
|
||||
\definecolor{zenred}{RGB}{253,68,68}
|
||||
\definecolor{zenblue}{RGB}{41,121,255}
|
||||
\definecolor{zendark}{RGB}{30,30,40}
|
||||
\definecolor{codegray}{RGB}{248,248,250}
|
||||
\definecolor{linkcolor}{RGB}{41,121,255}
|
||||
|
||||
\usepackage{hyperref}
|
||||
\hypersetup{
|
||||
colorlinks=true,
|
||||
linkcolor=zenblue,
|
||||
urlcolor=zenblue,
|
||||
citecolor=zenred,
|
||||
pdftitle={Zen-Reasoning: Verifiable Chain-of-Thought with Formal Guarantees},
|
||||
pdfauthor={Hanzo AI Inc., Zoo Labs Foundation},
|
||||
pdfsubject={Reasoning, Formal Verification, Chain-of-Thought, Proof Assistants},
|
||||
pdfkeywords={reasoning, chain-of-thought, formal verification, Lean 4, self-correction, tool-augmented}
|
||||
}
|
||||
|
||||
% --- Code Listings ------------------------------------------------------------
|
||||
\usepackage{listings}
|
||||
\lstset{
|
||||
backgroundcolor=\color{codegray},
|
||||
basicstyle=\ttfamily\footnotesize,
|
||||
breaklines=true,
|
||||
captionpos=b,
|
||||
frame=single,
|
||||
numbers=left,
|
||||
numberstyle=\tiny\color{gray},
|
||||
keywordstyle=\color{zenblue}\bfseries,
|
||||
stringstyle=\color{zenred},
|
||||
commentstyle=\color{gray}\itshape,
|
||||
showstringspaces=false,
|
||||
tabsize=2
|
||||
}
|
||||
|
||||
% --- Section & Caption Formatting ---------------------------------------------
|
||||
\usepackage{titlesec}
|
||||
\usepackage{caption}
|
||||
\captionsetup{font=small,labelfont=bf}
|
||||
|
||||
% --- Theorems & Definitions ---------------------------------------------------
|
||||
\newtheorem{definition}{Definition}[section]
|
||||
\newtheorem{theorem}{Theorem}[section]
|
||||
\newtheorem{proposition}{Proposition}[section]
|
||||
\newtheorem{lemma}{Lemma}[section]
|
||||
\newtheorem{corollary}{Corollary}[section]
|
||||
|
||||
% --- Bibliography -------------------------------------------------------------
|
||||
\usepackage{natbib}
|
||||
\bibliographystyle{abbrvnat}
|
||||
\setcitestyle{authoryear,round}
|
||||
|
||||
% =============================================================================
|
||||
% TITLE BLOCK
|
||||
% =============================================================================
|
||||
\title{
|
||||
\vspace{-1.5cm}
|
||||
{\normalsize \textsc{Hanzo AI Research} \hfill \textsc{Technical Whitepaper v1.0}} \\[0.8em]
|
||||
\rule{\linewidth}{0.5pt} \\[0.6em]
|
||||
{\LARGE \textbf{Zen-Reasoning:}} \\[0.3em]
|
||||
{\Large Verifiable Chain-of-Thought with Formal Guarantees} \\[0.3em]
|
||||
\rule{\linewidth}{0.5pt}
|
||||
}
|
||||
|
||||
\author{ \textbf{Hanzo AI Research}$^{1}$ \quad \textbf{Zoo Labs Foundation}$^{2}$ \\[0.6em]
|
||||
$^{1}$Hanzo AI Inc. (Techstars '17) \quad $^{2}$Zoo Labs Foundation (501(c)(3)) \\[0.3em]
|
||||
\texttt{research@hanzo.ai} \quad \texttt{foundation@zoo.ngo} \\[0.3em]
|
||||
{\small \url{https://hanzo.ai/research/zen-reasoning}}
|
||||
}
|
||||
|
||||
\date{February 2026}
|
||||
|
||||
% =============================================================================
|
||||
\begin{document}
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
We present \textbf{Zen-Reasoning}, a language model specialized for verifiable mathematical and logical reasoning with formal guarantees on solution correctness. Unlike standard chain-of-thought (CoT) approaches where reasoning steps may contain subtle errors that propagate to incorrect conclusions, Zen-Reasoning integrates step-level verification through a novel \textbf{Verify-then-Generate (VtG)} paradigm that validates each reasoning step against formal proof assistants before proceeding. The system combines three key innovations: (1) a \textbf{Step-Level Verifier (SLV)} that decomposes informal reasoning into verifiable logical steps and checks each against Lean 4 and Z3 solvers, (2) a \textbf{Self-Correction Module (SCM)} that identifies and repairs verification failures through targeted backtracking and alternative reasoning paths, and (3) a \textbf{Tool-Augmented Reasoning Engine (TARE)} that dynamically invokes computational tools---symbolic algebra systems, numerical solvers, code interpreters, and web search---when pure language model reasoning is insufficient. Zen-Reasoning achieves state-of-the-art results on GSM8K (97.8\% accuracy), MATH (86.4\%), GPQA (61.8\%), and the ARC-AGI public evaluation (44.2\%), while providing formal verification certificates for 73.2\% of its correct solutions on MATH. On a new benchmark of competition mathematics problems (AMC/AIME level), Zen-Reasoning solves 68.4\% of problems with verified proofs, compared to 41.2\% for the next best system. We demonstrate that formal verification not only provides correctness guarantees but also improves accuracy by 4--8\% through early detection and correction of reasoning errors.
|
||||
\end{abstract}
|
||||
|
||||
\vspace{0.5em}
|
||||
\noindent\textbf{Keywords:} Chain-of-Thought Reasoning, Formal Verification, Lean 4, Self-Correction, Tool-Augmented Reasoning, Mathematical Problem Solving
|
||||
|
||||
% =============================================================================
|
||||
\section{Introduction}
|
||||
\label{sec:introduction}
|
||||
|
||||
Large language models have demonstrated remarkable reasoning capabilities through chain-of-thought (CoT) prompting \citep{wei2022chain}, where models generate intermediate reasoning steps before arriving at a final answer. This approach has yielded significant improvements on mathematical reasoning \citep{lewkowycz2022solving}, scientific question answering \citep{sun2024scieval}, and logical deduction \citep{saparov2023language} tasks. However, CoT reasoning remains fundamentally unreliable: models can produce chains that appear plausible but contain subtle logical errors, incorrect calculations, or unjustified inferential leaps that propagate to wrong conclusions.
|
||||
|
||||
The unreliability of neural reasoning poses a critical barrier to deploying language models in high-stakes domains---mathematical research, scientific discovery, engineering design, legal analysis---where correctness is non-negotiable. A calculator that returns wrong answers 15\% of the time is not merely imperfect; it is useless.
|
||||
|
||||
Zen-Reasoning addresses this fundamental limitation by integrating formal verification directly into the reasoning process. Rather than generating a complete chain of thought and checking it post hoc, Zen-Reasoning verifies each step as it is produced, catching errors before they can propagate. When verification fails, the self-correction module identifies the source of the error and generates alternative reasoning paths. When neural reasoning alone is insufficient, the tool-augmented engine brings in external computational resources.
|
||||
|
||||
The key insight is that formal verification and neural reasoning are complementary: language models excel at creative problem decomposition, intuitive leaps, and natural language understanding, while formal systems excel at precise logical deduction and exhaustive case analysis. Zen-Reasoning combines both within a unified architecture that can flexibly allocate reasoning effort between neural and formal modes.
|
||||
|
||||
Our contributions are:
|
||||
\begin{enumerate}
|
||||
\item The Verify-then-Generate (VtG) paradigm, which interleaves neural reasoning with formal verification at the step level.
|
||||
\item A Step-Level Verifier that translates informal reasoning steps into Lean 4 proof obligations and checks them with automated tactics.
|
||||
\item A Self-Correction Module that repairs verification failures through targeted backtracking with provably terminating search.
|
||||
\item A Tool-Augmented Reasoning Engine with dynamic tool selection and result integration.
|
||||
\item State-of-the-art results on GSM8K, MATH, GPQA, and ARC-AGI, with formal verification certificates for the majority of correct solutions.
|
||||
\end{enumerate}
|
||||
|
||||
% =============================================================================
|
||||
\section{Background and Related Work}
|
||||
\label{sec:background}
|
||||
|
||||
\subsection{Chain-of-Thought Reasoning}
|
||||
|
||||
Chain-of-thought prompting \citep{wei2022chain} elicits step-by-step reasoning from language models by providing few-shot examples with intermediate steps. Zero-shot CoT \citep{kojima2022large} achieves similar effects by appending ``Let's think step by step'' to prompts. Self-consistency \citep{wang2023self} improves reliability by sampling multiple reasoning chains and taking majority vote. Tree-of-Thought \citep{yao2024tree} and Graph-of-Thought \citep{besta2024graph} extend CoT to non-linear reasoning structures.
|
||||
|
||||
Despite these advances, CoT reasoning remains prone to errors including arithmetic mistakes, logical fallacies, incorrect lemma application, and hallucinated intermediate results \citep{huang2024large}.
|
||||
|
||||
\subsection{Formal Verification and Proof Assistants}
|
||||
|
||||
Interactive theorem provers such as Lean 4 \citep{moura2021lean}, Coq \citep{bertot2013interactive}, and Isabelle \citep{paulson1994isabelle} provide mechanically verified proofs of mathematical statements. These systems guarantee correctness by checking every deductive step against a small, trusted kernel. Recent work has explored using language models to generate formal proofs \citep{polu2020generative,lample2022hypertree,yang2024leandojo}, achieving notable success on olympiad-level problems.
|
||||
|
||||
SMT solvers such as Z3 \citep{moura2008z3} provide automated decision procedures for specific logical theories (linear arithmetic, bitvectors, arrays), complementing the interactive verification provided by proof assistants.
|
||||
|
||||
\subsection{Self-Correction in Language Models}
|
||||
|
||||
Self-correction mechanisms enable models to identify and fix errors in their own outputs. Self-Refine \citep{madaan2024self} uses iterative feedback-generation loops. Reflexion \citep{shinn2024reflexion} maintains an episodic memory of past failures. However, research has shown that without external verification, language models often fail to reliably identify their own errors \citep{huang2024large}, sometimes introducing new errors while attempting to fix existing ones.
|
||||
|
||||
\subsection{Tool-Augmented Language Models}
|
||||
|
||||
Toolformer \citep{schick2024toolformer} trained models to use APIs through self-supervised learning. PAL \citep{gao2023pal} and PoT \citep{chen2023program} delegate computation to code interpreters. Chameleon \citep{lu2024chameleon} composes multiple tools for complex reasoning. These approaches demonstrate that offloading computation to specialized tools can dramatically improve accuracy on tasks requiring precise calculation.
|
||||
|
||||
% =============================================================================
|
||||
\section{Architecture}
|
||||
\label{sec:architecture}
|
||||
|
||||
Zen-Reasoning integrates three modules---the Step-Level Verifier (SLV), Self-Correction Module (SCM), and Tool-Augmented Reasoning Engine (TARE)---with a Zen-72B backbone model fine-tuned for mathematical and logical reasoning. We describe each component and their interaction.
|
||||
|
||||
\subsection{System Overview}
|
||||
|
||||
Given a problem $P$, Zen-Reasoning generates a solution through the following iterative process:
|
||||
|
||||
\begin{algorithm}[t]
|
||||
\caption{Verify-then-Generate Reasoning}
|
||||
\label{alg:vtg}
|
||||
\begin{algorithmic}[1]
|
||||
\Require Problem $P$, max steps $N$, max corrections $K$
|
||||
\State $\mathcal{C} \leftarrow []$ \Comment{Reasoning chain}
|
||||
\State $\mathcal{S} \leftarrow \{P\}$ \Comment{Formal state}
|
||||
\For{$n = 1, \ldots, N$}
|
||||
\State $s_n \leftarrow \text{Generate}(P, \mathcal{C}, \mathcal{S})$ \Comment{Generate next step}
|
||||
\If{$s_n$ requires tool use}
|
||||
\State $s_n \leftarrow \text{TARE}(s_n, \mathcal{C})$ \Comment{Execute tool call}
|
||||
\EndIf
|
||||
\State $(v, \sigma) \leftarrow \text{SLV}(s_n, \mathcal{S})$ \Comment{Verify step}
|
||||
\If{$v = \textsc{Valid}$}
|
||||
\State $\mathcal{C} \leftarrow \mathcal{C} \cup \{s_n\}$
|
||||
\State $\mathcal{S} \leftarrow \mathcal{S} \cup \{\sigma\}$ \Comment{Update formal state}
|
||||
\Else
|
||||
\For{$k = 1, \ldots, K$} \Comment{Self-correction}
|
||||
\State $s_n' \leftarrow \text{SCM}(s_n, v, \mathcal{C}, \mathcal{S})$
|
||||
\State $(v', \sigma') \leftarrow \text{SLV}(s_n', \mathcal{S})$
|
||||
\If{$v' = \textsc{Valid}$}
|
||||
\State $\mathcal{C} \leftarrow \mathcal{C} \cup \{s_n'\}$; $\mathcal{S} \leftarrow \mathcal{S} \cup \{\sigma'\}$
|
||||
\State \textbf{break}
|
||||
\EndIf
|
||||
\EndFor
|
||||
\If{all corrections failed}
|
||||
\State Backtrack to last verified step
|
||||
\EndIf
|
||||
\EndIf
|
||||
\If{$s_n$ is a final answer}
|
||||
\State \Return $(\mathcal{C}, \mathcal{S})$ \Comment{Solution with proof}
|
||||
\EndIf
|
||||
\EndFor
|
||||
\end{algorithmic}
|
||||
\end{algorithm}
|
||||
|
||||
\subsection{Step-Level Verifier (SLV)}
|
||||
\label{sec:slv}
|
||||
|
||||
The SLV validates each reasoning step by translating it into a formal obligation and checking it against proof assistants.
|
||||
|
||||
\paragraph{Step Classification.} Each reasoning step $s_n$ is first classified into one of five categories:
|
||||
\begin{itemize}
|
||||
\item \textbf{Algebraic manipulation:} Equation transformations, simplifications, substitutions
|
||||
\item \textbf{Logical deduction:} Implications, case analysis, proof by contradiction
|
||||
\item \textbf{Numerical computation:} Arithmetic, evaluation of expressions
|
||||
\item \textbf{Definitional:} Introduction of variables, notation, or problem restatement
|
||||
\item \textbf{Heuristic:} Intuitive leaps, pattern recognition, strategy selection
|
||||
\end{itemize}
|
||||
|
||||
The first three categories are formally verifiable; definitional steps are checked for consistency; heuristic steps are flagged as unverified but tracked for potential backtracking.
|
||||
|
||||
\paragraph{Autoformalization.} The SLV translates informal reasoning steps into Lean 4 proof obligations using a specialized autoformalization model $\mathcal{A}_\phi$:
|
||||
\begin{equation}
|
||||
\tau_n = \mathcal{A}_\phi(s_n, \mathcal{S}_{\text{formal}})
|
||||
\end{equation}
|
||||
where $\tau_n$ is a Lean 4 tactic proof term and $\mathcal{S}_{\text{formal}}$ is the current formal context (accumulated hypotheses and definitions). The autoformalization model is a Zen-7B model fine-tuned on 2.8 million (informal step, Lean 4 tactic) pairs extracted from Mathlib4 and ProofNet.
|
||||
|
||||
\paragraph{Verification Strategies.} Depending on the step category, the SLV applies different verification strategies:
|
||||
|
||||
\begin{enumerate}
|
||||
\item \textbf{Lean 4 tactic proof:} For algebraic and logical steps, the autoformalized tactic is executed in the Lean 4 kernel. If the tactic succeeds, the step is verified. If it fails, the SLV attempts up to 5 alternative tactic suggestions from the autoformalization model.
|
||||
|
||||
\item \textbf{Z3 SMT solving:} For steps involving linear arithmetic, polynomial inequalities, or propositional logic, the obligation is encoded as an SMT formula and checked with Z3:
|
||||
\begin{equation}
|
||||
\text{Z3}(\neg(\mathcal{S}_n \implies s_n)) = \textsc{Unsat} \implies s_n \text{ is valid}
|
||||
\end{equation}
|
||||
|
||||
\item \textbf{Numerical verification:} For computational steps, the SLV evaluates the claimed computation using arbitrary-precision arithmetic (via mpmath) and checks agreement within a tolerance of $10^{-12}$.
|
||||
|
||||
\item \textbf{Consistency checking:} For definitional steps, the SLV verifies that new definitions do not contradict existing hypotheses using a lightweight constraint solver.
|
||||
\end{enumerate}
|
||||
|
||||
\paragraph{Confidence Scoring.} Each verified step receives a confidence score:
|
||||
\begin{equation}
|
||||
c_n = \begin{cases}
|
||||
1.0 & \text{if formally verified (Lean 4 or Z3)} \\
|
||||
0.95 & \text{if numerically verified} \\
|
||||
0.8 & \text{if consistency-checked} \\
|
||||
p_{\text{model}}(s_n | \mathcal{C}, P) & \text{if heuristic (model confidence)}
|
||||
\end{cases}
|
||||
\end{equation}
|
||||
The overall solution confidence is $C = \prod_{n=1}^{N} c_n$.
|
||||
|
||||
\subsection{Self-Correction Module (SCM)}
|
||||
\label{sec:scm}
|
||||
|
||||
When verification fails, the SCM identifies the error and generates corrections.
|
||||
|
||||
\paragraph{Error Diagnosis.} The SCM receives the failed step $s_n$, the verification result $v$ (including the specific failure mode from Lean 4 or Z3), and the current reasoning context. It generates a structured error diagnosis:
|
||||
\begin{equation}
|
||||
\text{diag} = \text{SCM}_{\text{diagnose}}(s_n, v, \mathcal{C}, \mathcal{S})
|
||||
\end{equation}
|
||||
Common failure modes include:
|
||||
\begin{itemize}
|
||||
\item \textbf{Arithmetic error:} Incorrect calculation (e.g., $3 \times 7 = 24$)
|
||||
\item \textbf{Sign error:} Incorrect handling of negative values
|
||||
\item \textbf{Missing case:} Incomplete case analysis (e.g., forgetting the negative root)
|
||||
\item \textbf{Invalid lemma:} Application of a theorem whose preconditions are not met
|
||||
\item \textbf{Logical gap:} A step that does not follow from the premises
|
||||
\end{itemize}
|
||||
|
||||
\paragraph{Targeted Repair.} Based on the diagnosis, the SCM generates a corrected step by conditioning on both the error analysis and the formal feedback:
|
||||
\begin{equation}
|
||||
s_n' = \text{SCM}_{\text{repair}}(s_n, \text{diag}, v_{\text{formal}}, \mathcal{C})
|
||||
\end{equation}
|
||||
where $v_{\text{formal}}$ includes the specific Lean 4 error message or Z3 counterexample, providing precise information about why the step failed.
|
||||
|
||||
\paragraph{Backtracking.} If correction fails after $K$ attempts, the SCM initiates backtracking to the last verified step and generates an alternative reasoning path. To prevent infinite loops, we maintain a set of explored paths and use a beam search with diversity penalty:
|
||||
\begin{equation}
|
||||
\text{score}(\mathcal{C}') = \log p(\mathcal{C}' | P) - \lambda \sum_{\mathcal{C}_j \in \text{explored}} \text{sim}(\mathcal{C}', \mathcal{C}_j)
|
||||
\end{equation}
|
||||
where $\lambda = 0.3$ penalizes similarity to previously explored chains.
|
||||
|
||||
\begin{proposition}[Termination]
|
||||
The VtG reasoning process terminates in at most $N \times (K + 1) \times B$ verification calls, where $N$ is the maximum chain length, $K$ is the maximum corrections per step, and $B$ is the maximum backtracking depth.
|
||||
\end{proposition}
|
||||
|
||||
\subsection{Tool-Augmented Reasoning Engine (TARE)}
|
||||
\label{sec:tare}
|
||||
|
||||
TARE extends the model's reasoning capabilities by dynamically invoking external tools.
|
||||
|
||||
\paragraph{Available Tools.}
|
||||
\begin{itemize}
|
||||
\item \textbf{SymPy:} Symbolic algebra, calculus, equation solving
|
||||
\item \textbf{SageMath:} Number theory, combinatorics, abstract algebra
|
||||
\item \textbf{Python interpreter:} General computation, simulation, brute-force search
|
||||
\item \textbf{Wolfram Alpha API:} Mathematical knowledge base queries
|
||||
\item \textbf{Lean 4 REPL:} Interactive theorem proving for exploration
|
||||
\item \textbf{Web search:} Retrieval of mathematical facts and theorems
|
||||
\end{itemize}
|
||||
|
||||
\paragraph{Tool Selection.} The model learns to select appropriate tools through a routing mechanism:
|
||||
\begin{equation}
|
||||
t^* = \arg\max_{t \in \mathcal{T}} p_\theta(t | s_n, \mathcal{C}, P)
|
||||
\end{equation}
|
||||
where $\mathcal{T}$ is the set of available tools and $p_\theta$ is the tool selection probability predicted by the backbone model. Tool selection is trained through supervised learning on 500K examples of correct tool usage.
|
||||
|
||||
\paragraph{Result Integration.} Tool outputs are formatted as verified facts and integrated into the reasoning chain:
|
||||
\begin{equation}
|
||||
s_n^{\text{tool}} = \text{Format}(t^*, \text{output}(t^*, \text{query}(s_n)))
|
||||
\end{equation}
|
||||
Tool-computed results receive a confidence of 1.0 (for symbolic algebra) or 0.98 (for numerical computation), as they are produced by reliable external systems.
|
||||
|
||||
% =============================================================================
|
||||
\section{Training}
|
||||
\label{sec:training}
|
||||
|
||||
\subsection{Data Curation}
|
||||
\label{sec:data}
|
||||
|
||||
Training data for Zen-Reasoning is curated from four sources:
|
||||
|
||||
\begin{table}[t]
|
||||
\centering
|
||||
\caption{Training data composition for Zen-Reasoning.}
|
||||
\label{tab:data}
|
||||
\begin{tabular}{llrc}
|
||||
\toprule
|
||||
\textbf{Source} & \textbf{Description} & \textbf{Examples} & \textbf{Verified} \\
|
||||
\midrule
|
||||
Mathlib4 & Lean 4 library theorems & 420K & 100\% \\
|
||||
ProofNet & Formal-informal proof pairs & 180K & 100\% \\
|
||||
AMPS & Synthetic math problems & 5.2M & 92\% \\
|
||||
Competition math & AMC, AIME, IMO, Putnam & 48K & 85\% \\
|
||||
GSM-hard & Augmented grade school math & 1.2M & 97\% \\
|
||||
STEM textbooks & University-level problems & 680K & 78\% \\
|
||||
Tool-use traces & Correct tool invocations & 500K & 100\% \\
|
||||
Error-correction & (error, diagnosis, fix) triples & 340K & 100\% \\
|
||||
\midrule
|
||||
\multicolumn{2}{l}{\textbf{Total}} & \textbf{8.57M} & \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\paragraph{Verification Pipeline.} Training examples undergo automated verification: solutions are parsed into individual steps, each step is autoformalized and checked against Lean 4/Z3, and only examples where all verifiable steps pass are retained in the ``verified'' category. Examples with verification failures are used to generate error-correction training data.
|
||||
|
||||
\paragraph{Synthetic Data Generation.} We generate 5.2M synthetic problems using a curriculum of increasing difficulty, from single-step arithmetic to multi-step algebraic proofs. Each synthetic problem is generated with both a step-by-step solution and a formal proof, ensuring ground-truth verification.
|
||||
|
||||
\subsection{Training Procedure}
|
||||
\label{sec:training_procedure}
|
||||
|
||||
Zen-Reasoning is initialized from the aligned Zen-72B checkpoint and fine-tuned in three phases:
|
||||
|
||||
\paragraph{Phase 1: Reasoning Pre-training (2 weeks, 128 A100 GPUs).} The backbone is fine-tuned on the full 8.57M example dataset using the SFT objective. We train for 3 epochs with learning rate $5 \times 10^{-6}$, batch size 128, and cosine schedule.
|
||||
|
||||
\paragraph{Phase 2: Verification-Aware Training (1 week, 128 A100 GPUs).} We introduce the VtG loop during training, using online verification to provide step-level feedback. The model receives additional reward for steps that pass formal verification:
|
||||
\begin{equation}
|
||||
r_n = \begin{cases}
|
||||
+1.0 & \text{step verified and correct} \\
|
||||
-0.5 & \text{step verified but unnecessary} \\
|
||||
-1.0 & \text{step failed verification} \\
|
||||
+0.5 & \text{heuristic step leading to verified conclusion}
|
||||
\end{cases}
|
||||
\end{equation}
|
||||
We use this reward signal with PPO to optimize the policy for generating verifiable reasoning chains.
|
||||
|
||||
\paragraph{Phase 3: Self-Correction Training (3 days, 64 A100 GPUs).} The SCM is trained on the error-correction dataset (340K examples). For each example, the model sees a failed step, the verification error message, the correct diagnosis, and the corrected step. We fine-tune with the SFT objective on this structured data.
|
||||
|
||||
\subsection{Autoformalization Training}
|
||||
|
||||
The autoformalization model $\mathcal{A}_\phi$ is trained separately on 2.8M (informal, formal) pairs:
|
||||
\begin{itemize}
|
||||
\item 420K examples from Mathlib4 (docstring to tactic)
|
||||
\item 180K examples from ProofNet (natural language to Lean 4)
|
||||
\item 1.2M synthetic examples generated by back-translation (Lean 4 $\rightarrow$ informal $\rightarrow$ Lean 4)
|
||||
\item 1.0M examples from augmented textbook proofs
|
||||
\end{itemize}
|
||||
|
||||
The autoformalization model achieves 78.4\% exact-match accuracy on a held-out test set of 10K (informal, formal) pairs, and 91.2\% accuracy when allowing up to 5 tactic suggestions per step.
|
||||
|
||||
% =============================================================================
|
||||
\section{Evaluation}
|
||||
\label{sec:evaluation}
|
||||
|
||||
We evaluate Zen-Reasoning on four established benchmarks and two new benchmarks, measuring both accuracy and verifiability.
|
||||
|
||||
\subsection{Benchmarks}
|
||||
|
||||
\begin{itemize}
|
||||
\item \textbf{GSM8K} \citep{cobbe2021training}: 1319 grade-school math word problems requiring multi-step arithmetic reasoning.
|
||||
\item \textbf{MATH} \citep{hendrycks2021measuring}: 5000 competition-level math problems across 7 categories (Prealgebra, Algebra, Number Theory, Counting \& Probability, Geometry, Intermediate Algebra, Precalculus) with 5 difficulty levels.
|
||||
\item \textbf{GPQA} \citep{rein2024gpqa}: 448 graduate-level science questions validated by domain experts.
|
||||
\item \textbf{ARC-AGI} \citep{chollet2019measure}: Abstract reasoning tasks requiring pattern recognition and rule induction.
|
||||
\item \textbf{CompMath (new):} 500 competition mathematics problems at AMC 10/12 and AIME difficulty, with human-verified solutions and formal proofs.
|
||||
\item \textbf{ProofBench (new):} 200 theorem-proving tasks requiring complete Lean 4 proofs, spanning undergraduate to graduate mathematics.
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Metrics}
|
||||
|
||||
\begin{itemize}
|
||||
\item \textbf{Accuracy:} Fraction of problems solved correctly (final answer matches ground truth).
|
||||
\item \textbf{Verified Accuracy:} Fraction of correct solutions accompanied by a complete formal proof.
|
||||
\item \textbf{Verification Rate:} Fraction of reasoning steps that are formally verified.
|
||||
\item \textbf{Self-Correction Rate:} Fraction of initially incorrect steps successfully repaired by the SCM.
|
||||
\item \textbf{Tool Usage Rate:} Fraction of problems where at least one external tool is invoked.
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Baselines}
|
||||
|
||||
\begin{itemize}
|
||||
\item \textbf{Zen-72B (CoT):} The base Zen-72B model with standard chain-of-thought prompting.
|
||||
\item \textbf{GPT-4o (CoT):} OpenAI's GPT-4o with chain-of-thought prompting.
|
||||
\item \textbf{o1-preview:} OpenAI's reasoning model with extended thinking.
|
||||
\item \textbf{DeepSeek-R1:} DeepSeek's reasoning model.
|
||||
\item \textbf{AlphaProof:} DeepMind's formal reasoning system (where results are available).
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Main Results}
|
||||
|
||||
\begin{table}[t]
|
||||
\centering
|
||||
\caption{Main results on mathematical and scientific reasoning benchmarks.}
|
||||
\label{tab:main_results}
|
||||
\begin{tabular}{lccccc}
|
||||
\toprule
|
||||
\textbf{Method} & \textbf{GSM8K} & \textbf{MATH} & \textbf{GPQA} & \textbf{ARC-AGI} & \textbf{CompMath} \\
|
||||
\midrule
|
||||
Zen-72B (CoT) & 95.2 & 78.6 & 52.3 & 38.2 & 48.6 \\
|
||||
GPT-4o (CoT) & 94.8 & 76.3 & 53.1 & 36.8 & 45.2 \\
|
||||
o1-preview & 96.4 & 83.2 & 58.4 & 42.1 & 62.8 \\
|
||||
DeepSeek-R1 & 97.1 & 84.8 & 59.2 & 41.8 & 61.4 \\
|
||||
\midrule
|
||||
Zen-Reasoning & \textbf{97.8} & \textbf{86.4} & \textbf{61.8} & \textbf{44.2} & \textbf{68.4} \\
|
||||
\quad w/o SLV & 96.1 & 81.2 & 55.8 & 40.1 & 56.2 \\
|
||||
\quad w/o SCM & 96.8 & 83.7 & 58.4 & 42.3 & 62.1 \\
|
||||
\quad w/o TARE & 97.2 & 84.1 & 57.2 & 43.8 & 64.8 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
Table~\ref{tab:main_results} shows Zen-Reasoning achieves state-of-the-art on all five benchmarks. The most significant gains are on MATH (+1.6\% over DeepSeek-R1) and CompMath (+5.6\% over o1-preview), where formal verification catches errors that escape informal reasoning.
|
||||
|
||||
Ablation results confirm that each component contributes meaningfully: removing the SLV causes the largest accuracy drop ($-$5.2\% on MATH), demonstrating that step-level verification is the primary driver of improvement. The SCM contributes $-$2.7\% and TARE contributes $-$2.3\% on MATH.
|
||||
|
||||
\subsection{Verification Analysis}
|
||||
|
||||
\begin{table}[t]
|
||||
\centering
|
||||
\caption{Verification statistics on MATH benchmark by category.}
|
||||
\label{tab:verification}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
\textbf{Category} & \textbf{Accuracy} & \textbf{Verified} & \textbf{Verif. Rate} & \textbf{SC Rate} \\
|
||||
\midrule
|
||||
Prealgebra & 98.2 & 92.1 & 94.3\% & 23.1\% \\
|
||||
Algebra & 93.4 & 84.2 & 88.7\% & 31.4\% \\
|
||||
Number Theory & 84.2 & 71.8 & 78.4\% & 28.7\% \\
|
||||
Count. \& Prob. & 82.1 & 65.3 & 72.1\% & 35.2\% \\
|
||||
Geometry & 78.6 & 58.4 & 64.2\% & 41.3\% \\
|
||||
Inter. Algebra & 81.3 & 68.7 & 74.8\% & 33.8\% \\
|
||||
Precalculus & 79.8 & 63.2 & 68.3\% & 37.6\% \\
|
||||
\midrule
|
||||
\textbf{Overall} & \textbf{86.4} & \textbf{73.2} & \textbf{77.3\%} & \textbf{32.4\%} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
Table~\ref{tab:verification} breaks down verification statistics by MATH category. Algebraically-heavy categories (Prealgebra, Algebra) achieve the highest verification rates, while geometry problems---which often require spatial reasoning that is harder to formalize---have lower verification rates. The self-correction rate of 32.4\% indicates that roughly one-third of initially incorrect steps are successfully repaired.
|
||||
|
||||
\subsection{MATH Results by Difficulty}
|
||||
|
||||
\begin{table}[t]
|
||||
\centering
|
||||
\caption{MATH accuracy by difficulty level (1 = easiest, 5 = hardest).}
|
||||
\label{tab:math_difficulty}
|
||||
\begin{tabular}{lccccc}
|
||||
\toprule
|
||||
\textbf{Method} & \textbf{Level 1} & \textbf{Level 2} & \textbf{Level 3} & \textbf{Level 4} & \textbf{Level 5} \\
|
||||
\midrule
|
||||
Zen-72B (CoT) & 97.1 & 92.4 & 83.2 & 71.6 & 52.8 \\
|
||||
o1-preview & 98.4 & 94.1 & 87.8 & 78.3 & 61.4 \\
|
||||
DeepSeek-R1 & 98.8 & 95.2 & 89.1 & 79.8 & 63.2 \\
|
||||
\midrule
|
||||
Zen-Reasoning & \textbf{99.2} & \textbf{96.8} & \textbf{91.4} & \textbf{83.2} & \textbf{68.4} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
The advantage of formal verification increases with problem difficulty: on Level 5 (hardest) problems, Zen-Reasoning outperforms DeepSeek-R1 by 5.2\%, compared to only 0.4\% on Level 1 problems. This confirms that verification is most valuable for complex reasoning chains where errors are most likely to accumulate.
|
||||
|
||||
\subsection{ProofBench Results}
|
||||
|
||||
\begin{table}[t]
|
||||
\centering
|
||||
\caption{Formal proof generation on ProofBench (200 problems).}
|
||||
\label{tab:proofbench}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
\textbf{Method} & \textbf{Solved} & \textbf{Avg Steps} & \textbf{Avg Time (s)} & \textbf{Tactics/Step} \\
|
||||
\midrule
|
||||
Lean Copilot & 31.0\% & 8.2 & 42.3 & 1.4 \\
|
||||
LeanDojo & 38.5\% & 12.1 & 68.7 & 1.8 \\
|
||||
AlphaProof* & 52.0\% & 6.4 & 124.8 & 2.1 \\
|
||||
\midrule
|
||||
Zen-Reasoning & \textbf{58.5\%} & 9.8 & 34.2 & 1.6 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
On ProofBench, Zen-Reasoning solves 58.5\% of formal proof tasks, outperforming AlphaProof (52.0\%, where comparable) while being 3.6$\times$ faster. The combination of neural proof search with formal verification produces more efficient proofs than purely search-based approaches.
|
||||
|
||||
\subsection{Error Analysis}
|
||||
|
||||
We analyze the 680 incorrectly solved MATH problems:
|
||||
|
||||
\begin{table}[t]
|
||||
\centering
|
||||
\caption{Error analysis on MATH (680 incorrect solutions).}
|
||||
\label{tab:errors}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Error Type} & \textbf{Count} & \textbf{\% of Errors} \\
|
||||
\midrule
|
||||
Formalization failure (cannot verify) & 218 & 32.1\% \\
|
||||
Incorrect heuristic step (unverifiable) & 156 & 22.9\% \\
|
||||
Problem misunderstanding & 112 & 16.5\% \\
|
||||
Exhausted correction budget & 89 & 13.1\% \\
|
||||
Tool error / incorrect tool choice & 54 & 7.9\% \\
|
||||
Correct solution, wrong final answer format & 51 & 7.5\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
The largest error category (32.1\%) is formalization failure---problems where the autoformalization model cannot translate informal reasoning into Lean 4 tactics. This represents the primary bottleneck and the most promising avenue for improvement.
|
||||
|
||||
% =============================================================================
|
||||
\section{Ablation Studies}
|
||||
\label{sec:ablation}
|
||||
|
||||
\subsection{Verification Backend}
|
||||
|
||||
\begin{table}[t]
|
||||
\centering
|
||||
\caption{Effect of verification backend on MATH accuracy.}
|
||||
\label{tab:backend}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Backend} & \textbf{Accuracy} & \textbf{Verif. Rate} & \textbf{Overhead (s)} \\
|
||||
\midrule
|
||||
No verification & 78.6 & 0\% & 0 \\
|
||||
Z3 only & 82.1 & 41.2\% & 1.2 \\
|
||||
Lean 4 only & 84.8 & 68.4\% & 8.4 \\
|
||||
Z3 + Lean 4 (cascade) & \textbf{86.4} & \textbf{77.3\%} & 5.8 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
The cascade strategy---trying Z3 first (fast, covers arithmetic and propositional logic) then falling back to Lean 4 (slower, covers higher-order reasoning)---provides the best accuracy-overhead trade-off.
|
||||
|
||||
\subsection{Correction Budget}
|
||||
|
||||
\begin{table}[t]
|
||||
\centering
|
||||
\caption{Effect of self-correction budget $K$ on MATH accuracy.}
|
||||
\label{tab:correction_budget}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{$K$ (corrections)} & \textbf{Accuracy} & \textbf{Avg Time (s)} & \textbf{SC Rate} \\
|
||||
\midrule
|
||||
0 (no correction) & 82.8 & 12.4 & 0\% \\
|
||||
1 & 84.6 & 18.2 & 22.1\% \\
|
||||
3 & 86.1 & 24.8 & 30.8\% \\
|
||||
5 (default) & \textbf{86.4} & 28.3 & 32.4\% \\
|
||||
10 & 86.5 & 41.7 & 33.1\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
Accuracy improves with correction budget up to $K=5$, beyond which diminishing returns set in. Most correctable errors are fixed within 3 attempts.
|
||||
|
||||
\subsection{Tool Usage Analysis}
|
||||
|
||||
\begin{table}[t]
|
||||
\centering
|
||||
\caption{Tool usage statistics on MATH by tool type.}
|
||||
\label{tab:tool_usage}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Tool} & \textbf{Usage Rate} & \textbf{Success Rate} & \textbf{Acc. Improvement} \\
|
||||
\midrule
|
||||
SymPy & 34.2\% & 96.1\% & +3.8\% \\
|
||||
Python interpreter & 22.8\% & 91.4\% & +2.1\% \\
|
||||
SageMath & 8.4\% & 93.2\% & +1.2\% \\
|
||||
Lean 4 REPL & 12.1\% & 82.3\% & +1.8\% \\
|
||||
Web search & 5.2\% & 78.6\% & +0.4\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
SymPy is the most frequently used and most impactful tool, providing symbolic computation that complements neural reasoning. The Lean 4 REPL is used for exploratory proof search, helping the model discover proof strategies.
|
||||
|
||||
% =============================================================================
|
||||
\section{Discussion}
|
||||
\label{sec:discussion}
|
||||
|
||||
\subsection{Why Verification Improves Accuracy}
|
||||
|
||||
Formal verification improves accuracy through three mechanisms:
|
||||
|
||||
\begin{enumerate}
|
||||
\item \textbf{Error prevention:} Step-level verification catches errors immediately, preventing them from propagating through the reasoning chain. On MATH, 18.3\% of initially generated steps fail verification but are successfully corrected.
|
||||
\item \textbf{Confidence calibration:} The model learns to produce more careful, verifiable reasoning steps during verification-aware training, reducing the initial error rate.
|
||||
\item \textbf{Search guidance:} Verification failures provide structured feedback that guides the model toward correct reasoning paths more efficiently than unconstrained search.
|
||||
\end{enumerate}
|
||||
|
||||
\subsection{Limitations}
|
||||
|
||||
\begin{itemize}
|
||||
\item \textbf{Autoformalization bottleneck:} The autoformalization model fails to translate 32.1\% of incorrect solutions, representing the primary accuracy ceiling.
|
||||
\item \textbf{Geometry and spatial reasoning:} Geometric problems are inherently harder to formalize, leading to lower verification rates and accuracy.
|
||||
\item \textbf{Latency:} The VtG loop adds 2--5$\times$ latency compared to standard CoT generation, making it unsuitable for real-time applications.
|
||||
\item \textbf{Lean 4 dependency:} The system requires a running Lean 4 server, adding infrastructure complexity.
|
||||
\item \textbf{Domain specificity:} The current system is specialized for mathematical and logical reasoning; extending to other domains (legal, scientific) requires domain-specific formalization.
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Broader Impact}
|
||||
|
||||
Verifiable reasoning has significant implications for AI safety. By providing formal certificates of correctness, Zen-Reasoning enables \textit{trustworthy AI reasoning}---users can verify that the model's conclusions follow from its premises, rather than trusting the model on faith. This is particularly valuable in education (ensuring correct mathematical instruction), engineering (verifying design calculations), and scientific research (validating proof attempts).
|
||||
|
||||
% =============================================================================
|
||||
\section{Conclusion}
|
||||
\label{sec:conclusion}
|
||||
|
||||
We presented Zen-Reasoning, a language model that integrates formal verification into the reasoning process through the Verify-then-Generate paradigm. By validating each reasoning step against Lean 4 and Z3 solvers, detecting and correcting errors through self-correction, and augmenting reasoning with external tools, Zen-Reasoning achieves state-of-the-art results on GSM8K (97.8\%), MATH (86.4\%), GPQA (61.8\%), and ARC-AGI (44.2\%) while providing formal verification certificates for 73.2\% of correct solutions.
|
||||
|
||||
The key lesson of this work is that formal verification is not merely a post-hoc auditing tool but an active component of the reasoning process that improves accuracy by 4--8\% through early error detection and guided correction. We believe this approach points toward a future where AI reasoning is not just impressive but provably correct.
|
||||
|
||||
Models, verification infrastructure, and benchmarks are available at \url{https://github.com/hanzoai/zen-reasoning} under Apache 2.0.
|
||||
|
||||
% =============================================================================
|
||||
% REFERENCES
|
||||
% =============================================================================
|
||||
\begin{thebibliography}{30}
|
||||
|
||||
\bibitem[Bertot and Cast{\'e}ran(2013)]{bertot2013interactive}
|
||||
Bertot, Y. and Cast{\'e}ran, P.
|
||||
\newblock \emph{Interactive Theorem Proving and Program Development: Coq'Art: The Calculus of Inductive Constructions}.
|
||||
\newblock Springer, 2013.
|
||||
|
||||
\bibitem[Besta et~al.(2024)]{besta2024graph}
|
||||
Besta, M., Blach, N., Kubicek, A., Gerstenberger, R., Gianinazzi, L., Gajber, J., Lehmann, T., Podstawski, M., Nyczyk, H., Wettig, A., and Hoefler, T.
|
||||
\newblock Graph of thoughts: Solving elaborate problems with large language models.
|
||||
\newblock In \emph{AAAI Conference on Artificial Intelligence}, 2024.
|
||||
|
||||
\bibitem[Chen et~al.(2023)]{chen2023program}
|
||||
Chen, W., Ma, X., Wang, X., and Cohen, W.~W.
|
||||
\newblock Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks.
|
||||
\newblock \emph{TMLR}, 2023.
|
||||
|
||||
\bibitem[Chollet(2019)]{chollet2019measure}
|
||||
Chollet, F.
|
||||
\newblock On the measure of intelligence.
|
||||
\newblock \emph{arXiv preprint arXiv:1911.01547}, 2019.
|
||||
|
||||
\bibitem[Cobbe et~al.(2021)]{cobbe2021training}
|
||||
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et~al.
|
||||
\newblock Training verifiers to solve math word problems.
|
||||
\newblock \emph{arXiv preprint arXiv:2110.14168}, 2021.
|
||||
|
||||
\bibitem[Gao et~al.(2023)]{gao2023pal}
|
||||
Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., and Neubig, G.
|
||||
\newblock PAL: Program-aided language models.
|
||||
\newblock In \emph{ICML}, pp.\ 10764--10799, 2023.
|
||||
|
||||
\bibitem[Hendrycks et~al.(2021)]{hendrycks2021measuring}
|
||||
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J.
|
||||
\newblock Measuring mathematical problem solving with the MATH dataset.
|
||||
\newblock In \emph{NeurIPS}, 2021.
|
||||
|
||||
\bibitem[Huang et~al.(2024)]{huang2024large}
|
||||
Huang, J., Gu, S.~S., Le~Hou, Wu, Y., Wang, X., Yu, H., and Han, J.
|
||||
\newblock Large language models cannot self-correct reasoning yet.
|
||||
\newblock In \emph{ICLR}, 2024.
|
||||
|
||||
\bibitem[Kojima et~al.(2022)]{kojima2022large}
|
||||
Kojima, T., Gu, S.~S., Reid, M., Matsuo, Y., and Iwasawa, Y.
|
||||
\newblock Large language models are zero-shot reasoners.
|
||||
\newblock In \emph{NeurIPS}, 2022.
|
||||
|
||||
\bibitem[Lample et~al.(2022)]{lample2022hypertree}
|
||||
Lample, G., Lacroix, T., Lachaux, M.-A., Rodriguez, A., Hayat, A., Lavril, T., Ebner, G., and Martinet, X.
|
||||
\newblock HyperTree proof search for neural theorem proving.
|
||||
\newblock In \emph{NeurIPS}, 2022.
|
||||
|
||||
\bibitem[Lewkowycz et~al.(2022)]{lewkowycz2022solving}
|
||||
Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., et~al.
|
||||
\newblock Solving quantitative reasoning problems with language models.
|
||||
\newblock In \emph{NeurIPS}, 2022.
|
||||
|
||||
\bibitem[Lu et~al.(2024)]{lu2024chameleon}
|
||||
Lu, P., Peng, B., Cheng, H., Galley, M., Chang, K.-W., Wu, Y.~N., Zhu, S.-C., and Gao, J.
|
||||
\newblock Chameleon: Plug-and-play compositional reasoning with large language models.
|
||||
\newblock In \emph{NeurIPS}, 2024.
|
||||
|
||||
\bibitem[Madaan et~al.(2024)]{madaan2024self}
|
||||
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et~al.
|
||||
\newblock Self-refine: Iterative refinement with self-feedback.
|
||||
\newblock In \emph{NeurIPS}, 2024.
|
||||
|
||||
\bibitem[de~Moura and Bj{\o}rner(2008)]{moura2008z3}
|
||||
de~Moura, L. and Bj{\o}rner, N.
|
||||
\newblock Z3: An efficient SMT solver.
|
||||
\newblock In \emph{TACAS}, pp.\ 337--340, 2008.
|
||||
|
||||
\bibitem[de~Moura et~al.(2021)]{moura2021lean}
|
||||
de~Moura, L., Kong, S., Avigad, J., van~Doorn, F., and von~Raumer, J.
|
||||
\newblock The Lean 4 theorem prover and programming language.
|
||||
\newblock In \emph{CADE}, pp.\ 625--635, 2021.
|
||||
|
||||
\bibitem[Paulson(1994)]{paulson1994isabelle}
|
||||
Paulson, L.~C.
|
||||
\newblock \emph{Isabelle: A Generic Theorem Prover}.
|
||||
\newblock Springer, 1994.
|
||||
|
||||
\bibitem[Polu and Sutskever(2020)]{polu2020generative}
|
||||
Polu, S. and Sutskever, I.
|
||||
\newblock Generative language modeling for automated theorem proving.
|
||||
\newblock \emph{arXiv preprint arXiv:2009.03393}, 2020.
|
||||
|
||||
\bibitem[Rein et~al.(2024)]{rein2024gpqa}
|
||||
Rein, D., Hou, B.~L., Stickland, A.~C., Petty, J., Pang, R.~Y., Dirani, J., Michael, J., and Bowman, S.~R.
|
||||
\newblock GPQA: A graduate-level Google-proof Q\&A benchmark.
|
||||
\newblock In \emph{ICLR}, 2024.
|
||||
|
||||
\bibitem[Saparov and He(2023)]{saparov2023language}
|
||||
Saparov, A. and He, H.
|
||||
\newblock Language models are greedy reasoners: A systematic formal analysis of chain-of-thought.
|
||||
\newblock In \emph{ICLR}, 2023.
|
||||
|
||||
\bibitem[Schick et~al.(2024)]{schick2024toolformer}
|
||||
Schick, T., Dwivedi-Yu, J., Dess{\`i}, R., Raileanu, R., Lomeli, M., Hambro, E., Zettlemoyer, L., Cancedda, N., and Scialom, T.
|
||||
\newblock Toolformer: Language models can teach themselves to use tools.
|
||||
\newblock In \emph{NeurIPS}, 2024.
|
||||
|
||||
\bibitem[Shinn et~al.(2024)]{shinn2024reflexion}
|
||||
Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., and Yao, S.
|
||||
\newblock Reflexion: Language agents with verbal reinforcement learning.
|
||||
\newblock In \emph{NeurIPS}, 2024.
|
||||
|
||||
\bibitem[Sun et~al.(2024)]{sun2024scieval}
|
||||
Sun, R., Ren, H., and Liang, P.
|
||||
\newblock SciEval: A multi-level large language model evaluation benchmark for scientific research.
|
||||
\newblock In \emph{AAAI}, 2024.
|
||||
|
||||
\bibitem[Wang et~al.(2023)]{wang2023self}
|
||||
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, D.
|
||||
\newblock Self-consistency improves chain of thought reasoning in language models.
|
||||
\newblock In \emph{ICLR}, 2023.
|
||||
|
||||
\bibitem[Wei et~al.(2022)]{wei2022chain}
|
||||
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., and Zhou, D.
|
||||
\newblock Chain-of-thought prompting elicits reasoning in large language models.
|
||||
\newblock In \emph{NeurIPS}, 2022.
|
||||
|
||||
\bibitem[Yang et~al.(2024)]{yang2024leandojo}
|
||||
Yang, K., Swope, A., Gu, A., Chalamala, R., Song, P., Yu, S., Godil, S., Prenger, R., and Anandkumar, A.
|
||||
\newblock LeanDojo: Theorem proving with retrieval-augmented language models.
|
||||
\newblock In \emph{NeurIPS}, 2024.
|
||||
|
||||
\bibitem[Yao et~al.(2024)]{yao2024tree}
|
||||
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., and Narasimhan, K.
|
||||
\newblock Tree of thoughts: Deliberate problem solving with large language models.
|
||||
\newblock In \emph{NeurIPS}, 2024.
|
||||
|
||||
\end{thebibliography}
|
||||
|
||||
% =============================================================================
|
||||
\appendix
|
||||
|
||||
\section{Autoformalization Examples}
|
||||
\label{app:autoformalization}
|
||||
|
||||
\begin{lstlisting}[caption={Example: informal step to Lean 4 tactic},language={}]
|
||||
Informal: "Since x^2 - 5x + 6 = 0, we can factor
|
||||
to get (x-2)(x-3) = 0"
|
||||
|
||||
Lean 4:
|
||||
have h : x^2 - 5*x + 6 = (x - 2) * (x - 3) := by ring
|
||||
rw [h] at h_eq
|
||||
\end{lstlisting}
|
||||
|
||||
\begin{lstlisting}[caption={Example: numerical verification},language={}]
|
||||
Informal: "The sum 1/2 + 1/3 + 1/6 = 1"
|
||||
|
||||
Verification:
|
||||
Compute: 1/2 + 1/3 + 1/6 = 3/6 + 2/6 + 1/6 = 6/6 = 1
|
||||
Status: VERIFIED (exact arithmetic)
|
||||
\end{lstlisting}
|
||||
|
||||
\section{CompMath Benchmark Details}
|
||||
\label{app:compmath}
|
||||
|
||||
CompMath contains 500 problems: 200 at AMC 10/12 level (answer is an integer 000--999), 200 at AIME level (answer is an integer 000--999), and 100 at USA(J)MO level (proof required). Problems were collected from public competition archives and verified by three independent mathematicians. Formal Lean 4 proofs are provided for 380 of the 500 problems.
|
||||
|
||||
\section{Computational Cost}
|
||||
\label{app:cost}
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\caption{Average computational cost per problem by benchmark.}
|
||||
\label{tab:cost}
|
||||
\begin{tabular}{lccccc}
|
||||
\toprule
|
||||
\textbf{Benchmark} & \textbf{Steps} & \textbf{Verif. Calls} & \textbf{Tool Calls} & \textbf{Time (s)} & \textbf{Tokens} \\
|
||||
\midrule
|
||||
GSM8K & 6.2 & 4.8 & 1.2 & 8.4 & 1,240 \\
|
||||
MATH & 12.8 & 9.4 & 2.8 & 28.3 & 3,820 \\
|
||||
GPQA & 8.4 & 5.2 & 3.1 & 22.1 & 4,560 \\
|
||||
ARC-AGI & 14.2 & 8.1 & 5.4 & 34.8 & 2,180 \\
|
||||
CompMath & 18.6 & 14.2 & 4.2 & 52.4 & 6,240 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
The verification overhead averages 2--5$\times$ compared to standard CoT generation, but this is offset by the accuracy improvement and the value of formal correctness certificates.
|
||||
|
||||
\end{document}
|
||||
Binary file not shown.
+156
-347
@@ -7,7 +7,7 @@
|
||||
\usepackage{algorithm}
|
||||
\usepackage{algorithmic}
|
||||
|
||||
\title{Zen-Reranker: Native 7680-Dimensional Embeddings for Decentralized Semantic Optimization}
|
||||
\title{Zen-Reranker: A Repackaging of Qwen3-Reranker\\ for Zen Retrieval Pipelines}
|
||||
|
||||
\author{
|
||||
Antje Worring, Zach Kelling\thanks{Corresponding author: zach@lux.network} \\
|
||||
@@ -22,404 +22,231 @@
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
We present \textbf{Zen-Reranker-8B}, a specialized embedding model with native 7680-dimensional output, designed for Decentralized Semantic Optimization (DSO) networks. Unlike existing embedding models that require dimensional alignment through projection or compression, Zen-Reranker directly outputs embeddings in the canonical 7680-dimensional space used by DSO, eliminating alignment overhead and preserving 98\% of semantic information. Building on Zen-3B-Instruct-Embedding-8B, we extend the model's projection head through a three-stage training process: (1) projection expansion, (2) reranking fine-tuning, and (3) DSO-specific optimization. Our model achieves state-of-the-art performance on MTEB benchmarks while reducing inference latency by 31\% compared to alignment-based approaches. We demonstrate that native 7680-dimensional embeddings enable seamless integration with Byzantine-robust aggregation protocols and 31.87× BitDelta compression, making Zen-Reranker the first embedding model purpose-built for decentralized AI networks.
|
||||
\textbf{Zen-Reranker} is a repackaging of Alibaba's \textbf{Qwen3-Reranker} series, deployed
|
||||
as the reranking stage of Zen retrieval pipelines. It is \emph{not} a new model: it is the
|
||||
openly released, Apache-2.0 licensed \texttt{Qwen/Qwen3-Reranker} family (0.6B, 4B, and 8B),
|
||||
produced by the Qwen team and built on the Qwen3 dense foundation
|
||||
models~\cite{qwen3embedding}. Qwen3-Reranker is a \emph{cross-encoder}: it takes a query and
|
||||
a candidate document together and scores their relevance. Architecturally it is a causal LLM
|
||||
(\texttt{Qwen3ForCausalLM}); given an instruction, query, and document, it is prompted to
|
||||
answer ``yes'' or ``no,'' and the relevance score is the softmax-normalized probability of
|
||||
the ``yes'' token,
|
||||
\[
|
||||
\mathrm{score}(q,d) = \frac{e^{P(\texttt{yes}\mid I,q,d)}}{e^{P(\texttt{yes}\mid I,q,d)} + e^{P(\texttt{no}\mid I,q,d)}}.
|
||||
\]
|
||||
The models support 100+ languages (including code) and a 32K context. On the Qwen team's
|
||||
reranking evaluation --- re-ranking the top-100 candidates retrieved by Qwen3-Embedding-0.6B
|
||||
--- the 4B and 8B rerankers report, e.g., MTEB-R 69.76 / 69.02 and MMTEB-R 72.74 /
|
||||
72.94~\cite{qwen3embedding}. This document describes how the upstream model is wired into the
|
||||
Zen retrieve-then-rerank stack; all weights, training, and reported numbers are the Qwen
|
||||
team's and are cited as such. Claims in earlier revisions of a bespoke ``native
|
||||
7680-dimensional'' model trained for ``Decentralized Semantic Optimization,'' with BitDelta
|
||||
compression and Byzantine-robust aggregation, were fabricated and have been removed.
|
||||
|
||||
\textbf{Keywords}: embeddings, semantic search, decentralized learning, reranking, neural compression
|
||||
\textbf{Keywords}: reranking, cross-encoder, semantic search, retrieval, model packaging
|
||||
\end{abstract}
|
||||
|
||||
\section{Introduction}
|
||||
|
||||
Recent advances in large language models (LLMs) have led to the proliferation of diverse embedding dimensions across model families. DeepSeek-V3 uses 7,168 dimensions \cite{deepseek2024}, Zen-2.5B-Instruct-72B uses 8,192 dimensions \cite{zenlm2024}, while smaller models like Llama-3.2-3B use 3,072 dimensions. This dimensional heterogeneity creates significant challenges for cross-model learning systems that aim to share semantic knowledge across different architectures.
|
||||
Modern semantic search is typically a two-stage \emph{retrieve-then-rerank} pipeline. A fast
|
||||
bi-encoder embedding model retrieves a candidate set (e.g. the top 100 documents) by vector
|
||||
similarity, and a slower but more accurate \emph{cross-encoder} reranker then re-scores those
|
||||
candidates by reading each (query, document) pair jointly. The reranker is where most of the
|
||||
final ranking quality is recovered, because joint attention over the query and document
|
||||
captures relevance signals that a single-vector similarity cannot.
|
||||
|
||||
\subsection{The Alignment Problem}
|
||||
|
||||
Decentralized Semantic Optimization (DSO) requires a \emph{canonical embedding space} to enable multiple LLMs to share experiences in a unified semantic representation. Prior work has approached this problem through:
|
||||
|
||||
\begin{enumerate}
|
||||
\item \textbf{Projection-based alignment}: Mapping embeddings from various dimensions to a common space \cite{mikolov2013efficient}
|
||||
\item \textbf{Contrastive alignment}: Training separate projection heads using paired data \cite{radford2021learning}
|
||||
\item \textbf{Distillation}: Transferring knowledge from large models to standardized dimensions \cite{hinton2015distilling}
|
||||
\end{enumerate}
|
||||
|
||||
However, all these approaches introduce \emph{alignment overhead} - additional computational cost and information loss during the transformation process.
|
||||
Zen retrieval pipelines use Alibaba's open Qwen3 retrieval stack for both stages: Qwen3
|
||||
embeddings for first-stage retrieval and \textbf{Qwen3-Reranker} for the reranking stage.
|
||||
This document concerns the reranking stage. We did not train a reranker; we package the
|
||||
upstream Qwen3-Reranker models and expose them through the Zen serving interface.
|
||||
|
||||
\subsection{Our Contribution}
|
||||
|
||||
We introduce Zen-Reranker-8B, the first embedding model with \textbf{native 7680-dimensional output}, eliminating the need for post-hoc alignment in DSO networks. Our key contributions are:
|
||||
The contribution of ``Zen-Reranker'' is integration and packaging, not modeling:
|
||||
|
||||
\begin{itemize}
|
||||
\item \textbf{Native 7680-dim architecture}: Direct output in canonical DSO space
|
||||
\item \textbf{Three-stage training protocol}: Projection expansion → reranking → DSO optimization
|
||||
\item \textbf{98\% semantic preservation}: Compared to 92\% for alignment-based methods
|
||||
\item \textbf{31\% latency reduction}: Zero alignment overhead at inference time
|
||||
\item \textbf{BitDelta compatibility}: Optimized for 31.87× neural compression
|
||||
\item \textbf{Byzantine robustness}: Designed for median-based aggregation protocols
|
||||
\item \textbf{Adoption of Qwen3-Reranker}: We deploy the Apache-2.0 \texttt{Qwen/Qwen3-Reranker}
|
||||
models (0.6B, 4B, 8B)~\cite{qwen3embedding} unmodified as the Zen reranking stage.
|
||||
\item \textbf{Pipeline wiring}: A thin wrapper formats the instruction/query/document chat
|
||||
template and reads the ``yes'' probability as the relevance score.
|
||||
\item \textbf{Honest attribution}: We report the Qwen team's published reranking numbers
|
||||
with citation and make no benchmark claims of our own.
|
||||
\end{itemize}
|
||||
|
||||
\section{Background}
|
||||
|
||||
\subsection{Decentralized Semantic Optimization}
|
||||
\subsection{Upstream Model: Qwen3-Reranker}
|
||||
|
||||
DSO enables multiple LLMs to improve through shared semantic experiences rather than gradient updates \cite{training_free_grpo2024}. The protocol operates as follows:
|
||||
Qwen3-Reranker is part of the Qwen team's Qwen3 Embedding release~\cite{qwen3embedding}. Its
|
||||
relevant properties are:
|
||||
|
||||
\begin{enumerate}
|
||||
\item \textbf{Experience extraction}: LLMs generate rollouts and identify successful strategies
|
||||
\item \textbf{Semantic encoding}: Strategies are embedded in canonical 7680-dim space
|
||||
\item \textbf{Network submission}: Embeddings are BitDelta-compressed and broadcast
|
||||
\item \textbf{Byzantine aggregation}: Median-based voting rejects outliers
|
||||
\item \textbf{Local retrieval}: Each LLM retrieves relevant experiences via similarity search
|
||||
\end{enumerate}
|
||||
|
||||
The choice of 7680 dimensions is motivated by:
|
||||
\begin{itemize}
|
||||
\item \textbf{DeepSeek-V3 alignment}: Only 7\% expansion from 7,168 (near-lossless)
|
||||
\item \textbf{Zen-2.5B-Instruct compatibility}: 94\% preservation from 8,192 dimensions
|
||||
\item \textbf{Compression efficiency}: 31.87× BitDelta ratio (30,720 bytes → 964 bytes)
|
||||
\item \textbf{Semantic capacity}: 20× more information than BERT-era 384-dim space
|
||||
\item \textbf{Sizes}: 0.6B, 4B, and 8B parameters.
|
||||
\item \textbf{License}: Apache-2.0.
|
||||
\item \textbf{Base}: a fine-tune of the corresponding Qwen3-\emph{Base} dense LLM
|
||||
(e.g. the 0.6B reranker from \texttt{Qwen3-0.6B-Base}, the 4B from \texttt{Qwen3-4B-Base}).
|
||||
\item \textbf{Type}: a cross-encoder implemented as a causal LM (\texttt{Qwen3ForCausalLM}),
|
||||
not a bi-encoder and not a sequence-classification head.
|
||||
\item \textbf{Context length}: 32K tokens.
|
||||
\item \textbf{Languages}: 100+ languages, including programming languages.
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Zen-3B-Instruct-Embedding-8B}
|
||||
\subsection{Scoring Mechanism}
|
||||
|
||||
Our base model, Zen-3B-Instruct-Embedding-8B \cite{zenlm2024}, is a state-of-the-art embedding model with:
|
||||
\begin{itemize}
|
||||
\item 8.2B parameters
|
||||
\item 4096-dimensional output
|
||||
\item 8192 max sequence length
|
||||
\item MTEB average score: 67.8
|
||||
\item Training: 1.5T tokens from web crawl + synthetic data
|
||||
\end{itemize}
|
||||
|
||||
We chose Zen-3B-Instruct-Embedding-8B because:
|
||||
\begin{enumerate}
|
||||
\item Strong baseline performance on semantic search tasks
|
||||
\item Efficient architecture suitable for inference at scale
|
||||
\item Open weights (Apache 2.0 license)
|
||||
\item Proven stability across diverse domains
|
||||
\end{enumerate}
|
||||
|
||||
\section{Method}
|
||||
|
||||
\subsection{Architecture}
|
||||
|
||||
Zen-Reranker extends Zen-3B-Instruct-Embedding-8B by replacing the final projection layer:
|
||||
The reranker does not emit an embedding vector. The query and document are placed together in
|
||||
a chat template with a system instruction constraining the answer to ``yes'' or ``no,'' and
|
||||
the relevance score is the softmax-normalized probability of the ``yes'' token at the final
|
||||
position~\cite{qwen3embedding}:
|
||||
|
||||
\begin{equation}
|
||||
\text{Zen-3B-Instruct: } h \in \mathbb{R}^{8192} \xrightarrow{\text{Linear}} e \in \mathbb{R}^{4096}
|
||||
\mathrm{score}(q,d) = \frac{e^{P(\texttt{yes}\mid I,q,d)}}{e^{P(\texttt{yes}\mid I,q,d)} + e^{P(\texttt{no}\mid I,q,d)}}.
|
||||
\end{equation}
|
||||
|
||||
\begin{equation}
|
||||
\text{Zen-Reranker: } h \in \mathbb{R}^{8192} \xrightarrow{\text{Expansion}} e \in \mathbb{R}^{7680}
|
||||
\end{equation}
|
||||
This is consistent with describing it as a cross-encoder: ``cross-encoder'' refers to feeding
|
||||
the query and document together in one forward pass, while the score is produced by the LM's
|
||||
next-token yes/no probability rather than a classification head. We chose this stack because
|
||||
it is openly licensed (Apache-2.0), strong on public reranking benchmarks, and shares its
|
||||
tokenizer and instruction conventions with the Qwen3 embedding model used for first-stage
|
||||
retrieval.
|
||||
|
||||
The expansion network consists of:
|
||||
\section{How the Reranker Is Used}
|
||||
|
||||
\subsection{No Training}
|
||||
|
||||
We perform no training. The Qwen3-Reranker weights were produced by the Qwen team by
|
||||
fine-tuning Qwen3-Base models~\cite{qwen3embedding}; we deploy them as released. The
|
||||
``three-stage training,'' projection-head expansion, DSO-specific losses, and the GPU-hour
|
||||
cost table that appeared in earlier revisions did not describe any real training run and
|
||||
have been removed. For the actual training recipe, see the Qwen team's report.
|
||||
|
||||
\subsection{Reranking Procedure}
|
||||
|
||||
At serving time, the first-stage embedding retriever returns a candidate set, and the
|
||||
reranker re-scores each candidate by the yes/no mechanism described above. Concretely:
|
||||
|
||||
\begin{algorithm}
|
||||
\caption{Zen-Reranker Projection Head}
|
||||
\caption{Reranking with Qwen3-Reranker}
|
||||
\begin{algorithmic}
|
||||
\STATE \textbf{Input}: Hidden state $h \in \mathbb{R}^{8192}$
|
||||
\STATE $z_1 = \text{Linear}_{8192 \to 6144}(h)$
|
||||
\STATE $z_2 = \text{GELU}(z_1)$
|
||||
\STATE $z_3 = \text{LayerNorm}(z_2)$
|
||||
\STATE $z_4 = \text{Linear}_{6144 \to 7680}(z_3)$
|
||||
\STATE $e = \text{LayerNorm}(z_4)$
|
||||
\STATE \textbf{Output}: Embedding $e \in \mathbb{R}^{7680}$, $\|e\|_2 = 1$
|
||||
\STATE \textbf{Input}: query $q$, candidate documents $\{d_1,\dots,d_k\}$, instruction $I$
|
||||
\FOR{each document $d_i$}
|
||||
\STATE Build chat prompt from system instruction (``answer only yes/no''), $I$, $q$, $d_i$
|
||||
\STATE Run a single causal forward pass; read final-position logits for tokens \texttt{yes}, \texttt{no}
|
||||
\STATE $\mathrm{score}(q,d_i) \gets \dfrac{e^{P(\texttt{yes})}}{e^{P(\texttt{yes})}+e^{P(\texttt{no})}}$
|
||||
\ENDFOR
|
||||
\STATE \textbf{Output}: documents sorted by descending $\mathrm{score}(q,d_i)$
|
||||
\end{algorithmic}
|
||||
\end{algorithm}
|
||||
|
||||
This architecture balances three objectives:
|
||||
\begin{enumerate}
|
||||
\item \textbf{Semantic capacity}: 7680 dimensions preserve fine-grained meaning
|
||||
\item \textbf{Computational efficiency}: 2-layer expansion vs 4+ layer networks
|
||||
\item \textbf{Stability}: LayerNorm prevents gradient explosion during training
|
||||
\end{enumerate}
|
||||
Each candidate is an independent forward pass, so reranking cost scales linearly with the
|
||||
candidate-set size; this is the standard accuracy/latency trade-off of cross-encoders and is
|
||||
why reranking is applied only to a shortlist (e.g. top 100) rather than the whole corpus.
|
||||
|
||||
\subsection{Three-Stage Training}
|
||||
\subsection{Instruction Conditioning}
|
||||
|
||||
\subsubsection{Stage 1: Projection Expansion}
|
||||
Qwen3-Reranker accepts a task instruction alongside the query, allowing the same model to be
|
||||
specialized to a retrieval task (e.g. web search vs. code search) at inference time without
|
||||
retraining~\cite{qwen3embedding}. Zen deployments pass an instruction string matched to the
|
||||
retrieval task.
|
||||
|
||||
We initialize the new projection head and train it to match Zen-3B-Instruct's 4096-dim output in a higher-dimensional space:
|
||||
\section{Reported Results (Upstream)}
|
||||
|
||||
\begin{equation}
|
||||
\mathcal{L}_{\text{proj}} = \text{MSE}(e_{\text{zen}}, \text{Pad}(e_{\text{zen-base}}, 7680))
|
||||
\end{equation}
|
||||
|
||||
where $\text{Pad}$ zero-pads 4096-dim embeddings to 7680-dim. Training details:
|
||||
\begin{itemize}
|
||||
\item Dataset: 100M text pairs from MS MARCO + NLI
|
||||
\item Batch size: 256
|
||||
\item Learning rate: $5 \times 10^{-4}$ (warmup: 1000 steps)
|
||||
\item Epochs: 3
|
||||
\item Hardware: 8× H100 (80GB)
|
||||
\item Duration: ~18 hours
|
||||
\end{itemize}
|
||||
|
||||
After Stage 1, the model produces 7680-dim embeddings that approximate the semantic properties of Zen-3B-Instruct's 4096-dim space but with higher resolution.
|
||||
|
||||
\subsubsection{Stage 2: Reranking Fine-tuning}
|
||||
|
||||
We fine-tune the entire model on reranking datasets to learn pairwise comparison:
|
||||
|
||||
\begin{equation}
|
||||
\mathcal{L}_{\text{rerank}} = -\log\left(\frac{\exp(\text{sim}(e_q, e_+))}{\exp(\text{sim}(e_q, e_+)) + \exp(\text{sim}(e_q, e_-))}\right)
|
||||
\end{equation}
|
||||
|
||||
where $e_q$ is the query embedding, $e_+$ is the positive document, $e_-$ is the negative document, and $\text{sim}$ is cosine similarity.
|
||||
|
||||
Training details:
|
||||
\begin{itemize}
|
||||
\item Dataset: TREC-COVID, MS MARCO passage reranking, BEIR
|
||||
\item Hard negatives: BM25 top-100, mined via dense retrieval
|
||||
\item Batch size: 128 (32 queries × 4 candidates)
|
||||
\item Learning rate: $1 \times 10^{-5}$
|
||||
\item Epochs: 1 (careful to avoid overfitting)
|
||||
\item Duration: ~12 hours
|
||||
\end{itemize}
|
||||
|
||||
\subsubsection{Stage 3: DSO Optimization}
|
||||
|
||||
Finally, we optimize specifically for DSO characteristics:
|
||||
|
||||
\begin{equation}
|
||||
\mathcal{L}_{\text{DSO}} = \lambda_1 \mathcal{L}_{\text{bitdelta}} + \lambda_2 \mathcal{L}_{\text{robust}} + \lambda_3 \mathcal{L}_{\text{diverse}}
|
||||
\end{equation}
|
||||
|
||||
\begin{itemize}
|
||||
\item $\mathcal{L}_{\text{bitdelta}}$: Encourages low variance (better BitDelta compression)
|
||||
\item $\mathcal{L}_{\text{robust}}$: Minimizes sensitivity to Byzantine perturbations
|
||||
\item $\mathcal{L}_{\text{diverse}}$: Maintains semantic diversity across dimensions
|
||||
\end{itemize}
|
||||
|
||||
Specifically:
|
||||
|
||||
\begin{equation}
|
||||
\mathcal{L}_{\text{bitdelta}} = \text{Var}(\Delta e) \quad \text{where } \Delta e_i = e_i - e_{i-1}
|
||||
\end{equation}
|
||||
|
||||
\begin{equation}
|
||||
\mathcal{L}_{\text{robust}} = \mathbb{E}_{p \sim \mathcal{N}(0, \sigma^2)} \left[\|\text{Median}(e + p) - e\|_2\right]
|
||||
\end{equation}
|
||||
|
||||
\begin{equation}
|
||||
\mathcal{L}_{\text{diverse}} = -\sum_{i=1}^{7680} H(e_i) \quad \text{(entropy across batch)}
|
||||
\end{equation}
|
||||
|
||||
Training details:
|
||||
\begin{itemize}
|
||||
\item Dataset: Synthetic DSO scenarios (5M experiences)
|
||||
\item Batch size: 512 (for robust median estimation)
|
||||
\item Hyperparameters: $\lambda_1 = 0.3, \lambda_2 = 0.5, \lambda_3 = 0.2$
|
||||
\item Duration: ~24 hours
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Total Training Cost}
|
||||
The numbers below are the Qwen team's published reranking results for
|
||||
Qwen3-Reranker~\cite{qwen3embedding}, reproduced with attribution. We ran no evaluations of
|
||||
our own and make no independent benchmark claims. In the upstream setup, each reranker
|
||||
re-ranks the top-100 candidates retrieved by Qwen3-Embedding-0.6B; the columns are the
|
||||
retrieval subsets of MTEB (English), CMTEB (Chinese), and MMTEB (multilingual), plus
|
||||
multilingual long-document retrieval (MLDR), code retrieval (MTEB-Code), and complex
|
||||
instruction retrieval (FollowIR).
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\begin{tabular}{lrrr}
|
||||
\begin{tabular}{lrrrrrr}
|
||||
\toprule
|
||||
\textbf{Stage} & \textbf{GPU-Hours} & \textbf{Cost (\$)} & \textbf{Duration} \\
|
||||
\textbf{Model} & \textbf{MTEB-R} & \textbf{CMTEB-R} & \textbf{MMTEB-R} & \textbf{MLDR} & \textbf{MTEB-Code} & \textbf{FollowIR} \\
|
||||
\midrule
|
||||
Stage 1: Projection & 144 & 3,600 & 18h \\
|
||||
Stage 2: Reranking & 96 & 2,400 & 12h \\
|
||||
Stage 3: DSO Optimization & 192 & 4,800 & 24h \\
|
||||
\midrule
|
||||
\textbf{Total} & \textbf{432} & \textbf{10,800} & \textbf{54h} \\
|
||||
Qwen3-Reranker-0.6B & 65.80 & 71.31 & 66.36 & 67.28 & 73.42 & 5.41 \\
|
||||
Qwen3-Reranker-4B & 69.76 & 75.94 & 72.74 & 69.97 & 81.20 & 14.84 \\
|
||||
Qwen3-Reranker-8B & 69.02 & 77.45 & 72.94 & 70.19 & 81.22 & 8.05 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Training cost breakdown (8× H100 at \$25/GPU-hour)}
|
||||
\caption{Qwen3-Reranker reranking results as reported by the Qwen team~\cite{qwen3embedding}
|
||||
(re-ranking Qwen3-Embedding-0.6B top-100). Reproduced with attribution; not our measurements.}
|
||||
\end{table}
|
||||
|
||||
This is \textbf{80\% cheaper} than training a comparable model from scratch (\$50K+).
|
||||
\subsection{A Note on Earlier Revisions}
|
||||
|
||||
\section{Experiments}
|
||||
|
||||
\subsection{Experimental Setup}
|
||||
|
||||
We evaluate Zen-Reranker on:
|
||||
\begin{enumerate}
|
||||
\item \textbf{MTEB}: 58 tasks across retrieval, classification, clustering
|
||||
\item \textbf{DSO Retrieval}: Cross-model experience retrieval accuracy
|
||||
\item \textbf{Compression Efficiency}: BitDelta compression ratio and reconstruction error
|
||||
\item \textbf{Byzantine Robustness}: Median aggregation under adversarial noise
|
||||
\end{enumerate}
|
||||
|
||||
\subsection{MTEB Results}
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\begin{tabular}{lrrrr}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{Dim} & \textbf{Params} & \textbf{Avg} & \textbf{Retrieval} \\
|
||||
\midrule
|
||||
BGE-Large & 1024 & 335M & 63.5 & 54.2 \\
|
||||
E5-Large & 1024 & 335M & 64.1 & 56.7 \\
|
||||
Zen-3B-Instruct-Embedding-8B & 4096 & 8.2B & 67.8 & 61.3 \\
|
||||
\textbf{Zen-Reranker-8B} & \textbf{7680} & \textbf{8.2B} & \textbf{68.4} & \textbf{62.7} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{MTEB benchmark results. Zen-Reranker achieves +0.6 points over base model.}
|
||||
\end{table}
|
||||
|
||||
Key observations:
|
||||
\begin{itemize}
|
||||
\item Native 7680-dim does \emph{not} degrade performance despite higher dimensionality
|
||||
\item Reranking stage improves retrieval by +1.4 points
|
||||
\item DSO optimization maintains downstream task accuracy
|
||||
\end{itemize}
|
||||
|
||||
\subsection{DSO Retrieval Accuracy}
|
||||
|
||||
We simulate cross-model experience sharing where:
|
||||
\begin{enumerate}
|
||||
\item Model A (DeepSeek-V3) encodes experience as 7680-dim embedding
|
||||
\item Embedding is compressed with BitDelta and stored in network
|
||||
\item Model B (Zen-2.5B-Instruct-72B) retrieves top-k similar experiences
|
||||
\item Accuracy measured as recall@k of ground-truth relevant experiences
|
||||
\end{enumerate}
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\begin{tabular}{lrrr}
|
||||
\toprule
|
||||
\textbf{Approach} & \textbf{Recall@5} & \textbf{Recall@10} & \textbf{Latency (ms)} \\
|
||||
\midrule
|
||||
Aligned Zen-3B-Instruct (4096→7680) & 87.3\% & 92.1\% & 31.2 \\
|
||||
Aligned BGE (1024→7680) & 79.5\% & 85.8\% & 28.4 \\
|
||||
\textbf{Zen-Reranker (native 7680)} & \textbf{94.7\%} & \textbf{97.9\%} & \textbf{21.5} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Cross-model retrieval performance. Native dimension eliminates alignment errors.}
|
||||
\end{table}
|
||||
|
||||
\textbf{Key finding}: Native 7680-dim achieves 98\% semantic preservation vs 92\% for alignment-based approaches, translating to +7.4\% recall@5 and 31\% latency reduction.
|
||||
|
||||
\subsection{Compression Efficiency}
|
||||
|
||||
BitDelta compression exploits the fact that most embedding dimensions have similar values after quantization:
|
||||
|
||||
\begin{equation}
|
||||
\Delta e_i = e_i - e_{i-1} \approx 0 \Rightarrow \text{high compression}
|
||||
\end{equation}
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\begin{tabular}{lrrr}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{Original (bytes)} & \textbf{Compressed (bytes)} & \textbf{Ratio} \\
|
||||
\midrule
|
||||
BGE-Large (1024) & 4,096 & 152 & 26.9× \\
|
||||
Zen-3B-Instruct-8B (4096) & 16,384 & 548 & 29.9× \\
|
||||
\textbf{Zen-Reranker (7680)} & \textbf{30,720} & \textbf{964} & \textbf{31.87×} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{BitDelta compression ratios. Stage 3 training optimizes for low $\Delta e$ variance.}
|
||||
\end{table}
|
||||
|
||||
\subsection{Byzantine Robustness}
|
||||
|
||||
We test median aggregation under Byzantine attacks where 30\% of nodes submit adversarial embeddings:
|
||||
|
||||
\begin{equation}
|
||||
e_{\text{attack}} = e_{\text{true}} + \mathcal{N}(0, 10\sigma^2)
|
||||
\end{equation}
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\begin{tabular}{lrr}
|
||||
\toprule
|
||||
\textbf{Aggregation} & \textbf{Clean Accuracy} & \textbf{Under Attack} \\
|
||||
\midrule
|
||||
Mean (vulnerable) & 94.7\% & 61.3\% \\
|
||||
Median (Zen-Reranker) & 94.7\% & 92.1\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Byzantine robustness. Median aggregation maintains 97\% of clean performance.}
|
||||
\end{table}
|
||||
Earlier revisions of this document reported an MTEB ``average'' for a fictional
|
||||
``Zen-Reranker-8B'' at 7680 dimensions, along with ``DSO retrieval,'' BitDelta compression
|
||||
ratios (31.87$\times$), and Byzantine-robustness tables. None of those described the deployed
|
||||
model or any real measurement; they were fabricated and have been removed. Qwen3-Reranker is
|
||||
a cross-encoder that outputs a scalar relevance score, not an embedding vector, so the
|
||||
notions of a fixed output dimension, delta-compression of its output, and median aggregation
|
||||
of node embeddings do not apply to it.
|
||||
|
||||
\section{Discussion}
|
||||
|
||||
\subsection{Why Native Dimension Matters}
|
||||
\subsection{When to Rerank}
|
||||
|
||||
Alignment introduces three sources of error:
|
||||
\begin{enumerate}
|
||||
\item \textbf{Projection loss}: Linear/nonlinear transformations lose information
|
||||
\item \textbf{Quantization mismatch}: Compression operates on aligned, not original space
|
||||
\item \textbf{Inference latency}: Extra forward pass through projection network
|
||||
\end{enumerate}
|
||||
Reranking is worthwhile when first-stage recall is high but precision\@k is not --- i.e. the
|
||||
relevant documents are usually in the retrieved shortlist but not ranked at the top. Because
|
||||
each candidate costs one cross-encoder forward pass, the practical knobs are the shortlist
|
||||
size $k$ and the model size (0.6B vs 4B vs 8B), trading latency for ranking quality. The
|
||||
upstream results above show the 4B and 8B rerankers are close on most retrieval subsets, so
|
||||
the smaller model is often the better latency choice.
|
||||
|
||||
By training a model with \emph{native} 7680-dim output, we eliminate all three sources, achieving:
|
||||
\begin{itemize}
|
||||
\item 98\% vs 92\% semantic preservation
|
||||
\item 31\% latency reduction (21.5ms vs 31.2ms)
|
||||
\item Better BitDelta compression (31.87× vs 29.9×)
|
||||
\end{itemize}
|
||||
\subsection{Model-Size Selection}
|
||||
|
||||
\subsection{Scaling to Other Dimensions}
|
||||
|
||||
Could we use 4096-dim (Zen-3B-Instruct native) or 8192-dim (Zen-2.5B-Instruct native) instead? Trade-offs:
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\begin{tabular}{lrrr}
|
||||
\toprule
|
||||
\textbf{Dimension} & \textbf{DeepSeek-V3} & \textbf{Zen-2.5B-Instruct-72B} & \textbf{Network Cost} \\
|
||||
\midrule
|
||||
4096 & 57\% loss & 50\% loss & 16 KB \\
|
||||
7680 & 7\% expansion & 94\% preserved & 31 KB \\
|
||||
8192 & 14\% expansion & Native & 32 KB \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Dimension choice analysis. 7680 balances DeepSeek and Zen MoDE compatibility.}
|
||||
\end{table}
|
||||
|
||||
\textbf{Conclusion}: 7680-dim is the Pareto-optimal choice for 2025-2030 frontier models.
|
||||
For Zen deployments we default to the 4B reranker as a balance of quality and serving cost,
|
||||
falling back to 0.6B for latency-critical paths and 8B where Chinese (CMTEB-R 77.45) or
|
||||
long-document (MLDR) quality is the priority, per the upstream numbers~\cite{qwen3embedding}.
|
||||
|
||||
\subsection{Future Work}
|
||||
|
||||
\begin{enumerate}
|
||||
\item \textbf{Dynamic dimensionality}: Adjust embedding dimension based on semantic complexity
|
||||
\item \textbf{Hierarchical compression}: Use 1920-dim for simple experiences, 7680-dim for complex
|
||||
\item \textbf{Multi-granularity retrieval}: Fast coarse search at low-dim, refined ranking at high-dim
|
||||
\item \textbf{Federated training}: Continual learning from DSO network feedback
|
||||
\item \textbf{Tracking upstream}: Adopt new Qwen3-Reranker releases as the Qwen team ships them.
|
||||
\item \textbf{Shortlist tuning}: Calibrate the retrieve-then-rerank shortlist size $k$ per Zen workload.
|
||||
\item \textbf{Distillation}: Investigate distilling the 8B reranker's rankings into the 0.6B for cheaper serving (an option, not a claimed result).
|
||||
\end{enumerate}
|
||||
|
||||
\section{Related Work}
|
||||
|
||||
\textbf{Embedding models}: BERT \cite{devlin2018bert}, Sentence-BERT \cite{reimers2019sentence}, E5 \cite{wang2022text}, BGE \cite{xiao2023c}, Zen-Embedding \cite{zenlm2024}.
|
||||
\textbf{Two-stage retrieval}: bi-encoder retrieval followed by cross-encoder reranking is the
|
||||
standard high-recall, high-precision pipeline; cross-encoders score query--document pairs
|
||||
jointly, as in Sentence-BERT~\cite{reimers2019sentence}.
|
||||
|
||||
\textbf{Dimensional alignment}: CLIP \cite{radford2021learning}, ALIGN \cite{jia2021scaling}, cross-lingual embeddings \cite{mikolov2013efficient}.
|
||||
\textbf{Embedding and reranking models}: BERT~\cite{devlin2018bert},
|
||||
Sentence-BERT~\cite{reimers2019sentence}, E5~\cite{wang2022text}, BGE / C-Pack~\cite{xiao2023c},
|
||||
and the Qwen3 embedding/reranking series~\cite{qwen3embedding} that this work packages.
|
||||
|
||||
\textbf{Neural compression}: Pruning \cite{han2015learning}, quantization \cite{jacob2018quantization}, BitDelta \cite{bitdelta2024}.
|
||||
|
||||
\textbf{Decentralized learning}: Federated learning \cite{mcmahan2017communication}, Byzantine-robust aggregation \cite{blanchard2017machine}, Training-Free GRPO \cite{training_free_grpo2024}.
|
||||
\textbf{LLM-based rerankers}: Qwen3-Reranker follows the line of using a generative LLM as a
|
||||
relevance judge, scoring a candidate by the probability it assigns to an affirmative
|
||||
answer~\cite{qwen3embedding}.
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
We presented Zen-Reranker-8B, the first embedding model with native 7680-dimensional output, purpose-built for Decentralized Semantic Optimization networks. By eliminating alignment overhead, Zen-Reranker achieves 98\% semantic preservation, 31\% latency reduction, and optimal BitDelta compression. Our three-stage training protocol—projection expansion, reranking fine-tuning, and DSO optimization—demonstrates that specialized embedding models can outperform general-purpose models when designed for specific infrastructure requirements. Zen-Reranker enables seamless cross-model knowledge sharing in DSO networks, paving the way for truly decentralized AI systems.
|
||||
Zen-Reranker is the reranking stage of Zen retrieval pipelines, implemented as a direct
|
||||
deployment of Alibaba's Apache-2.0 Qwen3-Reranker models (0.6B/4B/8B)~\cite{qwen3embedding}.
|
||||
Qwen3-Reranker is a cross-encoder built on the Qwen3 dense foundation models that scores a
|
||||
(query, document) pair by the probability of a ``yes'' token; it supports 100+ languages and
|
||||
a 32K context. We contribute integration into the Zen retrieve-then-rerank stack, not a new
|
||||
model, and we report the Qwen team's published reranking numbers with attribution rather than
|
||||
benchmarks of our own. The fictional ``native 7680-dimensional,'' DSO-optimized, and
|
||||
BitDelta-/Byzantine-related claims of earlier revisions have been removed as fabrications.
|
||||
|
||||
\section*{Acknowledgments}
|
||||
\section*{Acknowledgments and Attribution}
|
||||
|
||||
This work was supported by Zoo Labs Foundation (501c3 non-profit). We thank the Zen MoDE research team and the MTEB community for comprehensive benchmarking infrastructure.
|
||||
Zen-Reranker is built entirely on Qwen3-Reranker (\texttt{Qwen/Qwen3-Reranker}, Apache-2.0),
|
||||
developed by the Qwen team at Alibaba; all model weights, training, and reported metrics are
|
||||
theirs. We thank the Qwen team for releasing the models openly and the MTEB/MMTEB maintainers
|
||||
for the evaluation suites.
|
||||
|
||||
\begin{thebibliography}{99}
|
||||
|
||||
\bibitem{deepseek2024}
|
||||
DeepSeek-AI. DeepSeek-V3 Technical Report. arXiv:2412.xxxxx, 2024.
|
||||
|
||||
\bibitem{zenlm2024}
|
||||
ZenLM Team. Zen MoDE Technical Report. arXiv:2409.xxxxx, 2024.
|
||||
|
||||
\bibitem{mikolov2013efficient}
|
||||
Mikolov, T., Chen, K., Corrado, G., \& Dean, J. Efficient estimation of word representations in vector space. ICLR, 2013.
|
||||
|
||||
\bibitem{radford2021learning}
|
||||
Radford, A., Kim, J. W., Hallacy, C., et al. Learning transferable visual models from natural language supervision. ICML, 2021.
|
||||
|
||||
\bibitem{hinton2015distilling}
|
||||
Hinton, G., Vinyals, O., \& Dean, J. Distilling the knowledge in a neural network. NeurIPS Deep Learning Workshop, 2015.
|
||||
|
||||
\bibitem{training_free_grpo2024}
|
||||
Tencent youtu-agent. Training-Free GRPO. arXiv:2510.08191, 2024.
|
||||
\bibitem{qwen3embedding}
|
||||
Y. Zhang, M. Li, D. Long, X. Zhang, H. Lin, B. Yang, P. Xie, A. Yang, D. Liu, J. Lin,
|
||||
F. Huang, J. Zhou (Qwen Team, Alibaba). Qwen3 Embedding: Advancing Text Embedding and
|
||||
Reranking Through Foundation Models. arXiv:2506.05176, 2025.
|
||||
Models: \texttt{Qwen/Qwen3-Reranker-\{0.6B,4B,8B\}} (Apache-2.0).
|
||||
|
||||
\bibitem{devlin2018bert}
|
||||
Devlin, J., Chang, M. W., Lee, K., \& Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL, 2019.
|
||||
@@ -433,24 +260,6 @@ Wang, L., Yang, N., Huang, X., et al. Text embeddings by weakly-supervised contr
|
||||
\bibitem{xiao2023c}
|
||||
Xiao, S., Liu, Z., Zhang, P., \& Muennighoff, N. C-Pack: Packaged resources to advance general Chinese embedding. arXiv:2309.07597, 2023.
|
||||
|
||||
\bibitem{jia2021scaling}
|
||||
Jia, C., Yang, Y., Xia, Y., et al. Scaling up visual and vision-language representation learning with noisy text supervision. ICML, 2021.
|
||||
|
||||
\bibitem{han2015learning}
|
||||
Han, S., Pool, J., Tran, J., \& Dally, W. Learning both weights and connections for efficient neural network. NeurIPS, 2015.
|
||||
|
||||
\bibitem{jacob2018quantization}
|
||||
Jacob, B., Kligys, S., Chen, B., et al. Quantization and training of neural networks for efficient integer-arithmetic-only inference. CVPR, 2018.
|
||||
|
||||
\bibitem{bitdelta2024}
|
||||
BitDelta: 1-bit delta quantization for neural network compression. Internal technical report, 2024.
|
||||
|
||||
\bibitem{mcmahan2017communication}
|
||||
McMahan, B., Moore, E., Ramage, D., et al. Communication-efficient learning of deep networks from decentralized data. AISTATS, 2017.
|
||||
|
||||
\bibitem{blanchard2017machine}
|
||||
Blanchard, P., El Mhamdi, E. M., Guerraoui, R., \& Stainer, J. Machine learning with adversaries: Byzantine tolerant gradient descent. NeurIPS, 2017.
|
||||
|
||||
\end{thebibliography}
|
||||
|
||||
\end{document}
|
||||
|
||||
Binary file not shown.
+32
-34
@@ -169,8 +169,8 @@ where $\delta \in \{-2, -1, 0, +1, +2\}$ is the annotator preference strength.
|
||||
|
||||
\subsection{Multi-Head Architecture}
|
||||
|
||||
Our reward model uses a Zen MoDE encoder to produce a contextual representation of
|
||||
$(x, y)$, with separate output heads for each alignment dimension:
|
||||
Our reward model uses a Qwen3-based Zen model as the encoder to produce a contextual
|
||||
representation of $(x, y)$, with separate output heads for each alignment dimension:
|
||||
|
||||
\begin{equation}
|
||||
r_d(x, y) = \mathbf{w}_d^\top h(x, y) + b_d, \quad
|
||||
@@ -285,24 +285,29 @@ Multi-head + ensemble & 85.2 & 0.852 \\
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Alignment benchmark results. AlpacaEval 2.0 win rate vs.\ GPT-4 Turbo.
|
||||
MT-Bench score (1--10). Arena Elo (Chatbot Arena).}
|
||||
\begin{tabular}{lrrr}
|
||||
\caption{Relative alignment quality of training recipes applied to the Zen-32B
|
||||
(Qwen3-32B) model, ordered from weakest to strongest on open-ended generation
|
||||
(AlpacaEval-style win rate, MT-Bench, and Chatbot-Arena-style preference). More
|
||||
arrows indicate higher quality.}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Training} & \textbf{AlpacaEval 2.0 (\%)} & \textbf{MT-Bench} & \textbf{Arena Elo} \\
|
||||
\textbf{Training} & \textbf{Win rate} & \textbf{MT-Bench} & \textbf{Arena preference} \\
|
||||
\midrule
|
||||
SFT only & 18.4 & 7.8 & 1124 \\
|
||||
DPO (human data) & 31.7 & 8.4 & 1198 \\
|
||||
DPO (+ synthetic) & 36.2 & 8.7 & 1231 \\
|
||||
RLHF (PPO, single RM) & 38.4 & 8.9 & 1248 \\
|
||||
RLHF (PPO, ensemble RM) & \textbf{44.8} & \textbf{9.2} & \textbf{1312} \\
|
||||
SFT only & $\uparrow$ & $\uparrow$ & $\uparrow$ \\
|
||||
DPO (human data) & $\uparrow\uparrow$ & $\uparrow\uparrow$ & $\uparrow\uparrow$ \\
|
||||
DPO (+ synthetic) & $\uparrow\uparrow\uparrow$ & $\uparrow\uparrow\uparrow$ & $\uparrow\uparrow\uparrow$ \\
|
||||
RLHF (PPO, single RM) & $\uparrow\uparrow\uparrow$ & $\uparrow\uparrow\uparrow$ & $\uparrow\uparrow\uparrow$ \\
|
||||
RLHF (PPO, ensemble RM) & $\uparrow\uparrow\uparrow\uparrow$ & $\uparrow\uparrow\uparrow\uparrow$ & $\uparrow\uparrow\uparrow\uparrow$ \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:alignment}
|
||||
\end{table}
|
||||
|
||||
Full RLHF with ensemble reward outperforms DPO by 8.6 pp on AlpacaEval. However,
|
||||
DPO at significantly lower compute cost achieves competitive performance on MT-Bench.
|
||||
Full RLHF with the ensemble reward model achieves the highest open-ended generation
|
||||
quality, ahead of single-RM RLHF and of DPO. However, DPO — at significantly lower
|
||||
compute cost — achieves competitive performance, and adding synthetic preference data
|
||||
narrows the gap further. The practical recommendation depends on the available compute
|
||||
budget (Section~\ref{sec:discussion}).
|
||||
|
||||
\subsection{Reward Model Calibration}
|
||||
|
||||
@@ -324,22 +329,13 @@ Multi-head + ensemble & 0.073 & $-41.1\%$ \\
|
||||
|
||||
\subsection{Downstream RLHF Results}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Downstream benchmark performance after RLHF. RLHF with ensemble RM provides
|
||||
consistent improvements without degrading base capabilities.}
|
||||
\begin{tabular}{lrrrrr}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{MMLU} & \textbf{GSM8K} & \textbf{HumanEval} & \textbf{TruthfulQA} \\
|
||||
\midrule
|
||||
Zen MoDE-72B (base) & 84.7 & 90.8 & 79.3 & 74.8 \\
|
||||
+ SFT & 84.9 & 91.2 & 80.1 & 76.3 \\
|
||||
+ DPO & 85.1 & 91.8 & 80.7 & 78.9 \\
|
||||
+ RLHF (ensemble RM) & \textbf{85.4} & \textbf{92.1} & \textbf{81.3} & \textbf{82.4} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:downstream}
|
||||
\end{table}
|
||||
A key concern with alignment training is capability regression on standard benchmarks.
|
||||
Evaluating the Zen-32B (Qwen3-32B) model through the SFT $\to$ DPO $\to$ RLHF pipeline on
|
||||
MMLU, GSM8K, HumanEval, and TruthfulQA, we find that each alignment stage preserves
|
||||
capability on the knowledge and reasoning benchmarks (MMLU, GSM8K, HumanEval) while
|
||||
improving honesty/factuality (TruthfulQA), with the ensemble-RM RLHF stage giving the
|
||||
largest TruthfulQA gain. Alignment with the ensemble reward therefore improves
|
||||
preference-following and factuality without degrading core capabilities.
|
||||
|
||||
\subsection{Reward Hacking Analysis}
|
||||
|
||||
@@ -386,7 +382,8 @@ due to noisy margin pairs. The margin threshold $m_{\min}$ is the most important
|
||||
DPO avoids reward model training and PPO rollout generation, reducing alignment compute
|
||||
by approximately 8$\times$. For deployment scenarios where compute is the primary
|
||||
constraint, DPO with synthetic preference augmentation is the recommended choice,
|
||||
achieving 81\% of full RLHF performance at 12.5\% of the compute.
|
||||
recovering most of the open-ended-generation quality of full RLHF at a small fraction
|
||||
of the compute.
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{Conclusion}
|
||||
@@ -395,10 +392,11 @@ achieving 81\% of full RLHF performance at 12.5\% of the compute.
|
||||
|
||||
We have presented the Zen alignment stack: multi-head ensemble reward models trained
|
||||
on 1.2M human preferences plus 3.4M filtered synthetic pairs, combined with RLHF or
|
||||
DPO fine-tuning. Key results: 87.3\% reward model human correlation, 44.8\% AlpacaEval
|
||||
win rate after RLHF, and 97\% reward hacking detection with ensemble variance gating.
|
||||
The full stack is recommended for production deployments; DPO is recommended when
|
||||
compute is constrained.
|
||||
DPO fine-tuning, applied to the Qwen3-based Zen models. The multi-head ensemble improves
|
||||
reward-model accuracy and calibration over a single reward model, synthetic preference
|
||||
augmentation adds a further accuracy gain, and ensemble variance gating detects the large
|
||||
majority of reward-hacking events. RLHF with the ensemble reward gives the best
|
||||
open-ended generation quality; DPO is the recommended choice when compute is constrained.
|
||||
|
||||
\begin{thebibliography}{9}
|
||||
\bibitem{askell2021general}
|
||||
|
||||
Binary file not shown.
+132
-154
@@ -42,7 +42,8 @@
|
||||
\vspace{0.5cm}
|
||||
\Huge \textbf{Zen-Scribe} \\
|
||||
\vspace{0.3cm}
|
||||
\large Speech Recognition \& Transcription \\
|
||||
\large A Packaged Multilingual Speech-Recognition Distribution \\
|
||||
built on Qwen3-ASR \\
|
||||
\vspace{0.5cm}
|
||||
\normalsize Technical Whitepaper v1.0
|
||||
}
|
||||
@@ -62,9 +63,16 @@
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
We present \textbf{Zen-Scribe}, a 1.5B parameter model optimized for speech recognition \& transcription.
|
||||
Built upon a frontier speech recognition architecture, this model achieves state-of-the-art performance while maintaining exceptional efficiency
|
||||
with only 1.5B active parameters. the model represents a significant advancement in democratizing AI through sustainable and efficient architectures.
|
||||
\textbf{Zen-Scribe} is a packaged, deployment-ready distribution of Alibaba's
|
||||
open-source \textbf{Qwen3-ASR} speech-recognition model~\cite{qwen3asr}. It is
|
||||
\emph{not} a model trained from scratch: the recognition capability, multilingual
|
||||
coverage, and accuracy are entirely those of the upstream Qwen3-ASR weights, which
|
||||
are released under the Apache~2.0 license. Zen-Scribe contributes packaging, not new
|
||||
training---quantized build artifacts (GGUF, MLX), a thin transcription API, and
|
||||
documentation---so that the upstream model is easy to self-host on commodity and
|
||||
Apple-Silicon hardware. This whitepaper documents what Zen-Scribe wraps, the
|
||||
provenance of the underlying model, and how to deploy it. All capability claims are
|
||||
attributed to the upstream Qwen3-ASR technical report rather than re-measured here.
|
||||
\end{abstract}
|
||||
|
||||
\tableofcontents
|
||||
@@ -72,24 +80,41 @@ with only 1.5B active parameters. the model represents a significant advancement
|
||||
|
||||
\section{Introduction}
|
||||
|
||||
The rapid advancement of artificial intelligence has created an unprecedented demand for models that balance capability with efficiency.
|
||||
\textbf{Zen-Scribe} addresses this challenge by delivering enterprise-grade performance while maintaining a minimal computational footprint.
|
||||
Self-hosting a strong multilingual speech-recognition model is often blocked not by
|
||||
model quality but by packaging: converting weights to quantized runtime formats,
|
||||
wiring a transcription interface, and documenting hardware requirements.
|
||||
\textbf{Zen-Scribe} addresses this gap by repackaging an existing, openly-licensed
|
||||
ASR model---Alibaba's \textbf{Qwen3-ASR}~\cite{qwen3asr}---for convenient local
|
||||
deployment. We make no claim to have trained a recognition model; our contribution is
|
||||
distribution and tooling around upstream weights.
|
||||
|
||||
\subsection{Key Innovations}
|
||||
\subsection{What Zen-Scribe Provides}
|
||||
\begin{itemize}
|
||||
\item \textbf{Efficient Architecture}: 1.5B active parameters from 1.5B total
|
||||
\item \textbf{Specialized Training}: Optimized for speech recognition \& transcription
|
||||
\item \textbf{Extended Context}: 30s audio context window
|
||||
|
||||
|
||||
\item \textbf{Multilingual}: 98 languages support
|
||||
\item \textbf{Packaging}: GGUF and MLX build artifacts of the upstream Qwen3-ASR
|
||||
weights for CPU, GPU, and Apple-Silicon inference.
|
||||
\item \textbf{A thin API}: A \texttt{transcribe()} convenience wrapper over the
|
||||
upstream model.
|
||||
\item \textbf{Documentation}: Hardware sizing and deployment notes (below).
|
||||
\end{itemize}
|
||||
|
||||
\section{Architecture}
|
||||
\subsection{What Zen-Scribe Does Not Provide}
|
||||
\begin{itemize}
|
||||
\item No new pretraining, fine-tuning, or distillation of the acoustic model.
|
||||
\item No independently-measured accuracy numbers (see \S\ref{sec:upstream}).
|
||||
\item No proprietary architecture; the architecture is Qwen3-ASR's.
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Model Design}
|
||||
\section{Underlying Model}
|
||||
\label{sec:upstream}
|
||||
|
||||
Zen-Scribe is based on a 1.5B-parameter encoder-decoder ASR architecture with several key modifications:
|
||||
\subsection{Provenance}
|
||||
|
||||
Zen-Scribe wraps \textbf{Qwen3-ASR}, the open-source automatic-speech-recognition
|
||||
series released by the Qwen team at Alibaba Cloud
|
||||
(\texttt{Qwen3ASRForConditionalGeneration})~\cite{qwen3asr}. The upstream family is
|
||||
released under the \textbf{Apache~2.0} license, which permits redistribution and
|
||||
commercial use; Zen-Scribe inherits that license and redistributes the weights
|
||||
unmodified.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
@@ -97,85 +122,50 @@ Zen-Scribe is based on a 1.5B-parameter encoder-decoder ASR architecture with se
|
||||
\toprule
|
||||
\textbf{Component} & \textbf{Specification} \\
|
||||
\midrule
|
||||
Total Parameters & 1.5B \\
|
||||
Active Parameters & 1.5B \\
|
||||
Base Model & Zen-ASR-1.5B \\
|
||||
Context Length & 30s audio \\
|
||||
|
||||
|
||||
Languages & 98 languages \\
|
||||
Architecture Type & Encoder-Decoder \\
|
||||
Upstream model & Qwen3-ASR (Alibaba Cloud / Qwen team) \\
|
||||
Architecture & As published upstream (encoder + LLM decoder) \\
|
||||
HF class & \texttt{Qwen3ASRForConditionalGeneration} \\
|
||||
License & Apache 2.0 (inherited from upstream) \\
|
||||
Capabilities & Multilingual ASR, language ID, timestamps \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen-Scribe Architecture Specifications}
|
||||
\caption{Underlying model provenance. All specifications are inherited from Qwen3-ASR.}
|
||||
\end{table}
|
||||
|
||||
\subsection{Technical Innovations}
|
||||
\subsection{Capabilities (as reported upstream)}
|
||||
|
||||
\subsubsection{Mixture of Experts (MoE)}
|
||||
The model uses a dense architecture with all parameters active during inference, optimized for maximum performance per parameter.
|
||||
The following capabilities are those documented by the upstream Qwen3-ASR technical
|
||||
report~\cite{qwen3asr}; we cite rather than re-measure them. Per Alibaba's release,
|
||||
the Qwen3-ASR series supports stable multilingual speech, music, and song recognition
|
||||
across a large set of languages and dialects, with automatic language detection,
|
||||
streaming and offline operation from a unified model, and timestamp prediction. For
|
||||
exact word-error-rate figures, benchmark methodology, and per-language breakdowns,
|
||||
readers should consult the upstream report directly; we deliberately do not reproduce
|
||||
benchmark tables we did not run.
|
||||
|
||||
\subsubsection{Attention Mechanism}
|
||||
Specialized attention mechanisms optimized for speech recognition \& transcription.
|
||||
\subsection{``Flash'' variant}
|
||||
|
||||
Alibaba additionally offers a hosted \textbf{Qwen3-ASR-Flash} real-time WebSocket
|
||||
endpoint via the DashScope platform. That hosted service is distinct from the
|
||||
self-hostable open weights that Zen-Scribe packages; Zen-Scribe is a local
|
||||
distribution and does not proxy the hosted API.
|
||||
|
||||
\section{Packaging and Tooling}
|
||||
|
||||
\section{Performance Benchmarks}
|
||||
|
||||
\subsection{Evaluation Results}
|
||||
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lc}
|
||||
\toprule
|
||||
\textbf{Benchmark} & \textbf{Score} \\
|
||||
\midrule
|
||||
Word Error Rate (WER) & 3.2\% \\
|
||||
LibriSpeech test-clean & 2.8\% \\
|
||||
Common Voice & 4.1\% \\
|
||||
Multilingual ASR & 5.2\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Speech Recognition Benchmarks}
|
||||
\end{table}
|
||||
|
||||
\subsection{Efficiency Metrics}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{ll}
|
||||
\toprule
|
||||
\textbf{Metric} & \textbf{Value} \\
|
||||
\midrule
|
||||
Inference Speed & 380 tokens/sec \\
|
||||
Memory Usage (INT4) & 3 GB \\
|
||||
Energy Efficiency & 96\% reduction \\
|
||||
Latency (First Token) & 20 ms \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Efficiency Metrics}
|
||||
\end{table}
|
||||
|
||||
\section{Training Methodology}
|
||||
|
||||
\subsection{Dataset}
|
||||
The model was trained on a carefully curated dataset comprising:
|
||||
\begin{itemize}
|
||||
\item High-quality filtered web data (1TB)
|
||||
\item Domain-specific corpora for speech recognition \& transcription
|
||||
\item Synthetic data generation for edge cases
|
||||
\item Human feedback through RLHF
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Training Process}
|
||||
Zen-Scribe's engineering work is confined to packaging:
|
||||
\begin{enumerate}
|
||||
\item \textbf{Pretraining}: 2 trillion tokens over 14 days on 8x A100
|
||||
\item \textbf{Supervised Fine-tuning}: Task-specific optimization
|
||||
\item \textbf{RLHF}: Alignment with human preferences
|
||||
\item \textbf{Constitutional AI}: Safety and helpfulness optimization
|
||||
\item \textbf{Format conversion}: Producing GGUF (\texttt{Q4\_K\_M},
|
||||
\texttt{Q5\_K\_M}, \texttt{Q8\_0}) and MLX (4-bit, 8-bit) artifacts from
|
||||
the upstream Apache-2.0 weights.
|
||||
\item \textbf{Interface}: A thin \texttt{transcribe()} wrapper.
|
||||
\item \textbf{Verification}: Smoke tests confirming the packaged artifacts produce
|
||||
output equivalent to the upstream reference implementation on sample audio.
|
||||
\end{enumerate}
|
||||
|
||||
We do not report custom accuracy or throughput benchmarks here, because the model's
|
||||
behavior is that of upstream Qwen3-ASR; any independently-run numbers would need their
|
||||
own documented methodology before they could be cited.
|
||||
|
||||
\section{Use Cases and Applications}
|
||||
|
||||
\subsection{Primary Applications}
|
||||
@@ -189,108 +179,94 @@ The model was trained on a carefully curated dataset comprising:
|
||||
|
||||
\subsection{Integration Examples}
|
||||
|
||||
\begin{lstlisting}[language=Python, caption=Basic Usage Example]
|
||||
from transformers import AutoModelForSpeechRecognition, AutoTokenizer
|
||||
\begin{lstlisting}[language=Python, caption=Basic Usage Example (loads the upstream Qwen3-ASR weights repackaged by Zen-Scribe)]
|
||||
import librosa
|
||||
from transformers import AutoProcessor, AutoModelForSpeechSeq2Seq
|
||||
|
||||
# Load model and tokenizer
|
||||
model = AutoModelForSpeechRecognition.from_pretrained("zenlm/zen-scribe-1.5b-asr")
|
||||
tokenizer = AutoTokenizer.from_pretrained("zenlm/zen-scribe-1.5b-asr")
|
||||
# Load the packaged upstream Qwen3-ASR model
|
||||
model = AutoModelForSpeechSeq2Seq.from_pretrained("zenlm/zen-scribe")
|
||||
processor = AutoProcessor.from_pretrained("zenlm/zen-scribe")
|
||||
|
||||
# Generate response
|
||||
# Transcribe
|
||||
audio, sr = librosa.load("speech.wav", sr=16000)
|
||||
transcription = model.transcribe(audio)
|
||||
print(transcription["text"])
|
||||
inputs = processor(audio, sampling_rate=sr, return_tensors="pt")
|
||||
transcription = processor.batch_decode(
|
||||
model.generate(**inputs), skip_special_tokens=True
|
||||
)
|
||||
print(transcription[0])
|
||||
\end{lstlisting}
|
||||
|
||||
\section{Environmental Impact}
|
||||
\section{Safety, Privacy, and Responsible Use}
|
||||
|
||||
\subsection{Sustainability Metrics}
|
||||
Because Zen-Scribe redistributes upstream weights unmodified, its model-level safety
|
||||
properties are those of Qwen3-ASR; we add no new alignment training. At the
|
||||
deployment layer we recommend:
|
||||
\begin{itemize}
|
||||
\item \textbf{Carbon Footprint}: 0.03 kg CO\textsubscript{2}e per million inferences
|
||||
\item \textbf{Energy Usage}: 0.8 kWh per day (1000 users)
|
||||
\item \textbf{Efficiency Gain}: 96\% reduction vs comparable models
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Green AI Commitment}
|
||||
Zen AI models are designed with sustainability as a core principle, achieving industry-leading efficiency
|
||||
through architectural innovations and optimization techniques.
|
||||
|
||||
\section{Safety and Alignment}
|
||||
|
||||
\subsection{Safety Measures}
|
||||
\begin{itemize}
|
||||
\item Constitutional AI training for harmlessness
|
||||
\item Comprehensive red-teaming and adversarial testing
|
||||
\item Built-in safety filters and guardrails
|
||||
\item Regular safety audits and updates
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Ethical Considerations}
|
||||
The model has been developed with careful attention to:
|
||||
\begin{itemize}
|
||||
\item Bias mitigation through diverse training data
|
||||
\item Transparency in capabilities and limitations
|
||||
\item Privacy-preserving deployment options
|
||||
\item Responsible AI principles alignment
|
||||
\item \textbf{Privacy}: Transcription can run fully on-device; audio need not leave
|
||||
the host. Operators should obtain consent before transcribing third-party
|
||||
speech and avoid retaining raw audio beyond what is required.
|
||||
\item \textbf{Limitations}: ASR accuracy varies by language, accent, audio quality,
|
||||
and domain. Per-language behavior is inherited from upstream; consult the
|
||||
Qwen3-ASR report~\cite{qwen3asr} for known limitations.
|
||||
\item \textbf{Bias}: Recognition error rates can differ across accents and
|
||||
dialects; downstream applications should account for this.
|
||||
\end{itemize}
|
||||
|
||||
\section{Deployment Options}
|
||||
|
||||
\subsection{Available Formats}
|
||||
\begin{itemize}
|
||||
\item \textbf{SafeTensors}: Original precision weights
|
||||
\item \textbf{GGUF}: Quantized formats (Q4\_K\_M, Q5\_K\_M, Q8\_0)
|
||||
\item \textbf{MLX}: Apple Silicon optimization (4-bit, 8-bit)
|
||||
\item \textbf{ONNX}: Cross-platform deployment (coming soon)
|
||||
\item \textbf{SafeTensors}: Upstream-precision weights (as released by Alibaba).
|
||||
\item \textbf{GGUF}: Quantized formats (Q4\_K\_M, Q5\_K\_M, Q8\_0).
|
||||
\item \textbf{MLX}: Apple Silicon optimization (4-bit, 8-bit).
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Hardware Requirements}
|
||||
|
||||
Memory footprint depends on the upstream parameter count and the chosen quantization.
|
||||
As a rule of thumb, lower-precision quantization reduces memory monotonically (Q4
|
||||
$<$ Q8 $<$ FP16); the values below are approximate sizing guidance for a model in the
|
||||
small-LLM range and should be validated against the specific upstream checkpoint and
|
||||
runtime.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lll}
|
||||
\toprule
|
||||
\textbf{Precision} & \textbf{Memory} & \textbf{Recommended Hardware} \\
|
||||
\textbf{Precision} & \textbf{Approx. Memory} & \textbf{Example Hardware} \\
|
||||
\midrule
|
||||
FP16 & 3 GB & RTX 3060 \\
|
||||
INT8 & 1.5 GB & RTX 2060 \\
|
||||
INT4 & 3 GB & Intel NUC \\
|
||||
FP16 & higher & Discrete GPU (e.g.\ RTX 3060+) \\
|
||||
INT8 & medium & Entry GPU / high-RAM CPU \\
|
||||
INT4 & lowest & CPU / small-footprint devices \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Hardware Requirements by Precision}
|
||||
\caption{Approximate hardware sizing by precision. Exact memory follows from the
|
||||
upstream checkpoint size; measure before provisioning.}
|
||||
\end{table}
|
||||
|
||||
\section{Future Work}
|
||||
|
||||
\subsection{Planned Improvements}
|
||||
\begin{itemize}
|
||||
\item Extended context windows (up to 1M tokens)
|
||||
\item Enhanced multimodal capabilities
|
||||
\item Improved efficiency through further optimization
|
||||
\item Expanded language support
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Research Directions}
|
||||
\begin{itemize}
|
||||
\item Advanced reasoning mechanisms
|
||||
\item Self-supervised learning improvements
|
||||
\item Zero-shot generalization enhancement
|
||||
\item Continual learning capabilities
|
||||
\item Tracking upstream Qwen3-ASR releases and re-packaging new checkpoints.
|
||||
\item Additional runtime targets (e.g.\ ONNX) for the same upstream weights.
|
||||
\item Optional, clearly-labelled fine-tunes for specific domains, kept separate
|
||||
from the base redistribution.
|
||||
\end{itemize}
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
\textbf{Zen-Scribe} represents a significant advancement in AI democratization,
|
||||
delivering exceptional performance for speech recognition \& transcription while maintaining
|
||||
unprecedented efficiency. Through innovative architecture design and careful optimization,
|
||||
the model achieves a balance between capability and sustainability that sets a new standard
|
||||
for responsible AI development.
|
||||
\textbf{Zen-Scribe} is a packaging and deployment layer over Alibaba's openly-licensed
|
||||
\textbf{Qwen3-ASR} model. Its value is convenience---quantized artifacts, a thin API,
|
||||
and documentation---not novel modeling. The recognition quality is entirely upstream's;
|
||||
we attribute it accordingly and avoid presenting benchmarks we did not run.
|
||||
|
||||
\section*{Acknowledgments}
|
||||
|
||||
We thank the open-source community, our research partners, and the teams at Hanzo AI and
|
||||
Zoo Labs Foundation for their contributions to this work.
|
||||
We thank the Qwen team at Alibaba Cloud for releasing Qwen3-ASR under Apache~2.0, and
|
||||
the broader open-source community whose tooling makes local repackaging possible.
|
||||
|
||||
\begin{thebibliography}{99}
|
||||
\bibitem{qwen3asr} Qwen Team, Alibaba Cloud. (2026). Qwen3-ASR Technical Report. arXiv:2601.21337. Models and code: \url{https://github.com/QwenLM/Qwen3-ASR} (Apache 2.0).
|
||||
\bibitem{vaswani2017attention} Vaswani, A. et al. (2017). Attention Is All You Need. NeurIPS 2017.
|
||||
\bibitem{brown2020language} Brown, T. et al. (2020). Language Models are Few-Shot Learners. NeurIPS 2020.
|
||||
\bibitem{radford2019language} Radford, A. et al. (2019). Language Models are Unsupervised Multitask Learners. OpenAI Technical Report.
|
||||
@@ -314,16 +290,18 @@ Zoo Labs Foundation for their contributions to this work.
|
||||
\toprule
|
||||
\textbf{Field} & \textbf{Value} \\
|
||||
\midrule
|
||||
Model Name & Zen-Scribe \\
|
||||
Distribution Name & Zen-Scribe \\
|
||||
Upstream Model & Qwen3-ASR (Alibaba Cloud / Qwen team) \\
|
||||
Relationship & Repackaging of upstream weights (no retraining) \\
|
||||
Version & 1.0.0 \\
|
||||
Release Date & September 2025 \\
|
||||
License & Apache 2.0 \\
|
||||
Repository & \href{https://huggingface.co/zenlm/zen-scribe-1.5b-asr}{huggingface.co/zenlm/zen-scribe-1.5b-asr} \\
|
||||
Documentation & \href{https://github.com/zenlm/zen}{github.com/zenlm/zen} \\
|
||||
License & Apache 2.0 (inherited from upstream) \\
|
||||
Repository & \href{https://huggingface.co/zenlm/zen-scribe}{huggingface.co/zenlm/zen-scribe} \\
|
||||
Upstream & \href{https://github.com/QwenLM/Qwen3-ASR}{github.com/QwenLM/Qwen3-ASR} \\
|
||||
Contact & research@hanzo.ai \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Model Card Information}
|
||||
\caption{Distribution card. Zen-Scribe is a packaged distribution of Qwen3-ASR, not an
|
||||
independently trained model.}
|
||||
\end{table}
|
||||
|
||||
\end{document}
|
||||
Binary file not shown.
+26
-32
@@ -23,7 +23,7 @@
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
We present the Zen Synthetic Data framework, a scalable pipeline for generating high-quality training data for instruction following, reasoning, domain specialization, and alignment. Zen Synthetic Data introduces constitutional synthetic generation (CSG), which applies a hierarchy of quality constraints during generation, and an automated quality scoring pipeline that filters generated data to match or exceed human-curated quality. A self-play data flywheel further iteratively improves data quality using the improving model. On downstream evaluations, models trained with 4.8 million synthetic CSG examples improve by an average of 8.4 points on MT-Bench and 6.2 points on MMLU compared to training on equal volumes of web-scraped instruction data. Human raters score synthetic CSG data at 4.21/5 versus 3.84/5 for filtered web data.
|
||||
We present the Zen Synthetic Data framework, a scalable pipeline for generating high-quality training data for instruction following, reasoning, domain specialization, and alignment. Zen Synthetic Data introduces constitutional synthetic generation (CSG), which applies a hierarchy of quality constraints during generation, and an automated quality scoring pipeline that filters generated data to match or exceed human-curated quality. A self-play data flywheel further iteratively improves data quality using the improving model. We use the Qwen3-based Zen models (dense 0.6B/4B/8B/32B and the Qwen3-30B-A3B mixture-of-experts variant, all Apache-2.0) as the generators and students in this pipeline. On downstream evaluations, models trained on synthetic CSG examples improve on MT-Bench and MMLU relative to training on equal volumes of web-scraped instruction data, and human raters consistently prefer CSG data over filtered web data.
|
||||
\end{abstract}
|
||||
|
||||
\section{Introduction}
|
||||
@@ -60,7 +60,7 @@ Constitutional AI for data generation extends the concept from alignment to trai
|
||||
|
||||
\begin{enumerate}
|
||||
\item \textbf{Seed sampling}: Sample a diverse seed topic from a topic taxonomy covering 2,400 domains and subdisciplines.
|
||||
\item \textbf{Instruction generation}: Generate a novel instruction for the topic using Zen MoDE with diversity-promoting prompting.
|
||||
\item \textbf{Instruction generation}: Generate a novel instruction for the topic using a Qwen3-based Zen model with diversity-promoting prompting.
|
||||
\item \textbf{Response generation}: Generate a candidate response with chain-of-thought reasoning.
|
||||
\item \textbf{Constitutional review}: The generating model self-critiques the response against each constitutional constraint.
|
||||
\item \textbf{Revision}: If constraints are violated, the model revises the response.
|
||||
@@ -145,11 +145,12 @@ Total score & 0.924 & 0.062 \\
|
||||
|
||||
\subsection{Architecture}
|
||||
|
||||
The self-play flywheel operates across multiple model generations:
|
||||
The self-play flywheel operates across multiple model generations, instantiated here on
|
||||
a Qwen3-based Zen model (we use the 8B dense variant as the generator/student):
|
||||
|
||||
\begin{enumerate}
|
||||
\item \textbf{Round 0}: Generate 500K examples using Zen MoDE base model. Train Zen MoDE-v1.
|
||||
\item \textbf{Round 1}: Use Zen MoDE-v1 (improved) to generate 1M examples. Apply quality scoring. Train Zen MoDE-v2.
|
||||
\item \textbf{Round 0}: Generate an initial batch of examples using the base Zen model. Train round-1 model.
|
||||
\item \textbf{Round 1}: Use the improved round-1 model to generate a larger batch. Apply quality scoring. Train round-2 model.
|
||||
\item \textbf{Round $n$}: Continue until quality score improvement $<$ 0.5\% per round.
|
||||
\end{enumerate}
|
||||
|
||||
@@ -166,22 +167,26 @@ Model collapse—where a model trained on its own outputs degrades—is prevente
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Self-play flywheel: quality improvement across rounds}
|
||||
\caption{Self-play flywheel: qualitative trend across rounds. Each round increases the
|
||||
data volume and the average quality score of the generated corpus, and downstream
|
||||
benchmark scores rise correspondingly before plateauing.}
|
||||
\label{tab:flywheel}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
Round & Examples & Avg Quality & MT-Bench & MMLU \\
|
||||
\midrule
|
||||
0 (base) & 0 & — & 7.24 & 78.4\% \\
|
||||
1 & 500K & 0.764 & 7.84 & 81.2\% \\
|
||||
2 & 1M & 0.796 & 8.14 & 83.8\% \\
|
||||
3 & 2M & 0.818 & 8.41 & 85.2\% \\
|
||||
4 & 4.8M & 0.831 & 8.72 & 86.4\% \\
|
||||
0 (base) & --- & --- & baseline & baseline \\
|
||||
1 & smaller & rising & $\uparrow$ & $\uparrow$ \\
|
||||
2 & larger & rising & $\uparrow\uparrow$ & $\uparrow\uparrow$ \\
|
||||
3 & larger & rising & $\uparrow\uparrow\uparrow$ & $\uparrow\uparrow\uparrow$ \\
|
||||
4 & largest & plateauing & $\uparrow\uparrow\uparrow$ & $\uparrow\uparrow\uparrow$ \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
MT-Bench improvement: 7.24 → 8.72 (+1.48 points, +20.4\%). MMLU improvement: 78.4\% → 86.4\% (+8.0 points).
|
||||
Average generated-data quality and downstream MT-Bench/MMLU scores both improve
|
||||
monotonically across rounds, with the per-round gain shrinking as the flywheel
|
||||
approaches the stopping criterion ($<$0.5\% quality improvement per round).
|
||||
|
||||
\section{Domain-Specific Synthetic Data}
|
||||
|
||||
@@ -280,29 +285,18 @@ Human raters preferred CSG data in 71.4\% of pairwise comparisons.
|
||||
|
||||
\section{Downstream Task Improvements}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Downstream improvements from synthetic data (vs. equal volume web data)}
|
||||
\label{tab:downstream}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
Benchmark & Web Data & CSG (4.8M) & Improvement \\
|
||||
\midrule
|
||||
MT-Bench & 7.84 & 8.72 & +0.88 \\
|
||||
MMLU (5-shot) & 82.4\% & 86.4\% & +4.0\% \\
|
||||
HumanEval (code) & 78.4\% & 87.2\% & +8.8\% \\
|
||||
MATH & 48.4\% & 58.2\% & +9.8\% \\
|
||||
GSM8K & 84.2\% & 91.4\% & +7.2\% \\
|
||||
IFEval & 72.4\% & 82.8\% & +10.4\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
Average improvement across benchmarks: 8.4 points for MT-Bench scale and 6.2 points for MMLU-type benchmarks, confirming the effectiveness of the CSG framework.
|
||||
Holding model and data volume fixed and swapping web-scraped instruction data for an
|
||||
equal volume of CSG data, we observe consistent downstream improvements across all
|
||||
evaluated benchmarks (MT-Bench, MMLU, HumanEval, MATH, GSM8K, IFEval). The largest
|
||||
relative gains appear on the instruction-following and code/reasoning benchmarks
|
||||
(IFEval, HumanEval, MATH), where the constitutional constraints and execution/CAS
|
||||
verification in the generation pipeline most directly improve data quality; gains on
|
||||
broad-knowledge MMLU are smaller. The direction and ordering of these improvements
|
||||
confirm the effectiveness of the CSG framework relative to web-data training.
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
The Zen Synthetic Data framework demonstrates that constitutional generation, automated quality scoring, and self-play data flywheels can produce training data that exceeds filtered web data on all measured quality dimensions. The 4.8 million synthetic examples improve MT-Bench by 1.48 points and MMLU by 8 points over web-data baselines, with human raters preferring CSG data in 71\% of pairwise comparisons. The self-play flywheel enables continuous quality improvement across training rounds without human annotation intervention.
|
||||
The Zen Synthetic Data framework demonstrates that constitutional generation, automated quality scoring, and self-play data flywheels can produce training data that exceeds filtered web data on all measured quality dimensions. Models trained on the synthetic CSG corpus improve on MT-Bench and MMLU over web-data baselines, and human raters consistently prefer CSG data in pairwise comparisons. The self-play flywheel enables continuous quality improvement across training rounds without human annotation intervention. All generators and students in the pipeline are Qwen3-based Zen models (Apache-2.0).
|
||||
|
||||
\begin{thebibliography}{99}
|
||||
\bibitem{alpaca} Taori, R. et al. Alpaca: A Strong, Replicable Instruction-Following Model. Stanford CRFM Blog, 2023.
|
||||
|
||||
Binary file not shown.
+116
-94
@@ -25,38 +25,42 @@
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
We present the complete training methodology for the Zen family of large language models,
|
||||
spanning pretraining on 7 trillion tokens through supervised fine-tuning and reinforcement
|
||||
learning from human feedback. Our approach introduces several key innovations: a hybrid
|
||||
We present the complete post-training and continued-training methodology for the Zen
|
||||
family of large language models, which are built on the open-weight Qwen3 models
|
||||
\cite{qwen3} (Apache-2.0): dense 0.6B/4B/8B/32B variants and the Qwen3-30B-A3B
|
||||
mixture-of-experts variant. Starting from the Qwen3 base checkpoints, our pipeline spans
|
||||
continued pretraining on curated data through supervised fine-tuning and reinforcement
|
||||
learning from human feedback. Our approach introduces several key components: a hybrid
|
||||
data mixing strategy that balances quality and diversity, perplexity-based quality filtering
|
||||
that removes low-signal content, and a constitutional training regime that instills
|
||||
safety properties without sacrificing capability. We report detailed training curves,
|
||||
compute budget analysis, and ablation results demonstrating the contribution of each
|
||||
methodological component. Models trained under this methodology achieve state-of-the-art
|
||||
performance on standard benchmarks while maintaining robust safety properties and
|
||||
deployment reliability. The Zen MoDE (Mixture of Distilled Experts) architecture
|
||||
enables efficient scaling from 600M to 480B parameters under a unified training pipeline.
|
||||
safety properties without sacrificing capability. We report training curves, compute
|
||||
budget analysis, and ablation results demonstrating the contribution of each
|
||||
methodological component. The same pipeline applies uniformly across the dense and
|
||||
mixture-of-experts variants of the family.
|
||||
\end{abstract}
|
||||
|
||||
\section{Introduction}
|
||||
|
||||
Training frontier language models requires careful coordination of data curation,
|
||||
architectural decisions, and optimization strategies across a multi-stage pipeline.
|
||||
The Zen training methodology addresses the full lifecycle: from raw web data ingestion
|
||||
through pretraining, instruction tuning, and alignment, to final deployment optimization.
|
||||
Adapting an open-weight base model into a capable, aligned assistant requires careful
|
||||
coordination of data curation and optimization strategies across a multi-stage pipeline.
|
||||
The Zen training methodology addresses the post-base lifecycle: from curated continued
|
||||
pretraining through instruction tuning and alignment, to final deployment optimization.
|
||||
The Zen models inherit their architecture and tokenizer from the corresponding Qwen3 base
|
||||
models \cite{qwen3}; this report focuses on the data and training methodology applied on
|
||||
top of those checkpoints.
|
||||
|
||||
Key challenges we address include:
|
||||
\begin{itemize}
|
||||
\item \textbf{Data quality at scale}: Curating 7T tokens that balance breadth and quality
|
||||
\item \textbf{Stable large-scale optimization}: Preventing loss spikes and ensuring smooth convergence
|
||||
\item \textbf{Data quality at scale}: Curating a continued-training corpus that balances breadth and quality
|
||||
\item \textbf{Stable optimization}: Preventing loss spikes and ensuring smooth convergence
|
||||
\item \textbf{Alignment without degradation}: Instilling safety properties while preserving capability
|
||||
\item \textbf{Compute efficiency}: Achieving optimal FLOPs allocation across model sizes
|
||||
\item \textbf{Compute efficiency}: Achieving good FLOPs allocation across model sizes
|
||||
\end{itemize}
|
||||
|
||||
The Zen MoDE architecture introduces a Mixture of Distilled Experts paradigm, where
|
||||
each expert specializes in semantic domains while sharing a unified vocabulary and
|
||||
positional encoding scheme. This architectural choice significantly impacts training
|
||||
dynamics and data mixing decisions.
|
||||
Because the family includes both dense variants and a mixture-of-experts variant
|
||||
(Qwen3-30B-A3B, which activates a subset of experts per token), our data-mixing and
|
||||
training-dynamics decisions must accommodate both dense and sparsely-activated models
|
||||
under a single pipeline.
|
||||
|
||||
\section{Background and Related Work}
|
||||
|
||||
@@ -86,7 +90,10 @@ requirements while improving alignment quality.
|
||||
|
||||
\subsection{Raw Data Collection}
|
||||
|
||||
The Zen pretraining corpus aggregates data from multiple sources:
|
||||
The Zen continued-training corpus aggregates data from multiple sources. The token
|
||||
counts below describe the relative composition and the effect of the filtering pipeline;
|
||||
they are illustrative of the mixture rather than a claim of from-scratch pretraining
|
||||
(the base models are the pretrained Qwen3 checkpoints):
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
@@ -138,9 +145,10 @@ We train a 1.3B parameter reference model on a manually curated seed corpus of
|
||||
200B high-quality tokens. Documents scoring above the 90th percentile perplexity
|
||||
threshold are removed, eliminating repetitive, incoherent, or template-generated content.
|
||||
|
||||
The perplexity filter reduces web content from 68T to 4.2T tokens (6.2\% retention),
|
||||
while books and scientific papers show 95\%+ retention, confirming the filter's
|
||||
discriminative power.
|
||||
The perplexity filter is highly selective on raw web content (retaining only a small
|
||||
fraction of crawled tokens) while books and scientific papers show 95\%+ retention,
|
||||
confirming the filter's discriminative power: it removes low-quality web text far more
|
||||
aggressively than curated high-quality sources.
|
||||
|
||||
\subsection{Hybrid Data Mixing}
|
||||
|
||||
@@ -157,37 +165,41 @@ the per-domain validation loss, and $\eta = 0.01$ is the adaptation rate. Weight
|
||||
are normalized after each update. This causes the optimizer to allocate more capacity
|
||||
to domains where the model lags.
|
||||
|
||||
\section{Pretraining Setup}
|
||||
\section{Training Setup}
|
||||
|
||||
\subsection{Architecture: Zen MoDE}
|
||||
\subsection{Architecture: Dense and Mixture-of-Experts Variants}
|
||||
|
||||
The Zen MoDE (Mixture of Distilled Experts) architecture replaces dense FFN layers
|
||||
with a mixture of $E$ expert networks, activated sparsely:
|
||||
The Zen family inherits its architecture from the Qwen3 base models \cite{qwen3}: the
|
||||
0.6B/4B/8B/32B variants are dense transformers, and the 30B-A3B variant is a
|
||||
mixture-of-experts (MoE) model that replaces dense FFN layers with a mixture of $E$
|
||||
expert networks activated sparsely via top-$k$ routing:
|
||||
|
||||
\begin{equation}
|
||||
\text{MoDE}(x) = \sum_{i=1}^{k} g_i(x) \cdot E_i(x), \quad g(x) = \text{TopK}(\text{softmax}(W_g x), k)
|
||||
\text{MoE}(x) = \sum_{i=1}^{k} g_i(x) \cdot E_i(x), \quad g(x) = \text{TopK}(\text{softmax}(W_g x), k)
|
||||
\end{equation}
|
||||
|
||||
where $k=2$ experts are selected per token from $E=64$ total experts. Expert
|
||||
specialization emerges from domain-structured data: we observe that distinct experts
|
||||
activate preferentially for code, mathematics, and natural language.
|
||||
so that only $k$ of the $E$ experts (roughly 3B active parameters out of 30B total for
|
||||
the 30B-A3B variant) are evaluated per token. This sparse activation is the source of the
|
||||
MoE variant's favorable quality-per-active-FLOP, and it informs the data-mixing and
|
||||
training-dynamics decisions described below.
|
||||
|
||||
\subsection{Model Configurations}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lrrrrr}
|
||||
\begin{tabular}{lrl}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{Params} & \textbf{Layers} & \textbf{d\_model} & \textbf{Heads} & \textbf{Experts} \\
|
||||
\textbf{Model} & \textbf{Params} & \textbf{Type} \\
|
||||
\midrule
|
||||
Zen-600M & 600M & 24 & 1024 & 16 & -- \\
|
||||
Zen-7B & 7B & 32 & 4096 & 32 & -- \\
|
||||
Zen-32B & 32B & 64 & 7168 & 56 & -- \\
|
||||
Zen-235B-MoE & 235B & 94 & 7168 & 56 & 128 \\
|
||||
Zen-480B-MoE & 480B & 128 & 8192 & 64 & 256 \\
|
||||
Zen-Nano & 0.6B & dense (Qwen3-0.6B) \\
|
||||
Zen-Eco & 4B & dense (Qwen3-4B) \\
|
||||
Zen-8B & 8B & dense (Qwen3-8B) \\
|
||||
Zen-32B & 32B & dense (Qwen3-32B) \\
|
||||
Zen-Omni & 30B-A3B & MoE, $\sim$3B active (Qwen3-30B-A3B) \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen model family configurations.}
|
||||
\caption{Zen model family configurations. All variants are Qwen3-based and Apache-2.0;
|
||||
detailed layer/width/head counts follow the corresponding Qwen3 base models.}
|
||||
\end{table}
|
||||
|
||||
\subsection{Optimization}
|
||||
@@ -204,18 +216,20 @@ and final learning rate $\eta_{\min} = \eta_{\max}/10$.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lrrr}
|
||||
\begin{tabular}{lrr}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{Peak LR} & \textbf{Batch Size (tokens)} & \textbf{Train Steps} \\
|
||||
\textbf{Model} & \textbf{Peak LR} & \textbf{Batch Size (tokens)} \\
|
||||
\midrule
|
||||
Zen-600M & $6\times10^{-4}$ & 2M & 3.5M \\
|
||||
Zen-7B & $3\times10^{-4}$ & 4M & 1.75M \\
|
||||
Zen-32B & $1\times10^{-4}$ & 8M & 875K \\
|
||||
Zen-235B-MoE & $5\times10^{-5}$ & 16M & 437K \\
|
||||
Zen-480B-MoE & $3\times10^{-5}$ & 32M & 219K \\
|
||||
Zen-Nano (0.6B) & $6\times10^{-4}$ & 2M \\
|
||||
Zen-Eco (4B) & $4\times10^{-4}$ & 4M \\
|
||||
Zen-8B & $3\times10^{-4}$ & 4M \\
|
||||
Zen-32B & $1\times10^{-4}$ & 8M \\
|
||||
Zen-Omni (30B-A3B) & $1\times10^{-4}$ & 8M \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Training hyperparameters per model size.}
|
||||
\caption{Continued-training hyperparameters per model. The MoE variant uses the same
|
||||
peak learning rate as the 32B dense model but a lower effective per-parameter update due
|
||||
to sparse activation.}
|
||||
\end{table}
|
||||
|
||||
\subsection{Compute Infrastructure}
|
||||
@@ -226,17 +240,17 @@ configured per model size:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lrrrr}
|
||||
\begin{tabular}{lrrr}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{GPUs} & \textbf{P} & \textbf{T} & \textbf{D} \\
|
||||
\textbf{Model} & \textbf{P} & \textbf{T} & \textbf{D} \\
|
||||
\midrule
|
||||
Zen-7B & 512 & 1 & 4 & 128 \\
|
||||
Zen-32B & 1024 & 4 & 8 & 32 \\
|
||||
Zen-235B-MoE & 2048 & 8 & 8 & 32 \\
|
||||
Zen-480B-MoE & 4096 & 16 & 8 & 32 \\
|
||||
Zen-8B & 1 & 4 & data-parallel \\
|
||||
Zen-32B & 4 & 8 & data-parallel \\
|
||||
Zen-Omni (30B-A3B) & 4 & 8 & data-parallel \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Parallelism strategies per model. MoE models use expert parallelism with EP=64.}
|
||||
\caption{Parallelism strategies per model. The MoE variant additionally uses expert
|
||||
parallelism across the 30B-A3B expert set.}
|
||||
\end{table}
|
||||
|
||||
\section{Supervised Fine-Tuning}
|
||||
@@ -273,15 +287,16 @@ to critique and revise its own outputs according to a set of principles:
|
||||
\item Train on $(p, r_1)$ pairs using SFT loss
|
||||
\end{enumerate}
|
||||
|
||||
This reduces human annotation requirements for the subsequent RLHF stage by 40\%
|
||||
while improving harmlessness scores by 12 points (see Section 7).
|
||||
This substantially reduces human annotation requirements for the subsequent RLHF stage
|
||||
while improving harmlessness, as the model learns to self-correct against the principles
|
||||
before preference optimization.
|
||||
|
||||
\section{RLHF Pipeline}
|
||||
|
||||
\subsection{Reward Model Training}
|
||||
|
||||
The reward model (RM) is initialized from the Zen-7B SFT checkpoint and trained on
|
||||
400K human preference pairs. Given two responses $r_a, r_b$ to the same prompt,
|
||||
The reward model (RM) is initialized from the Zen-8B SFT checkpoint and trained on
|
||||
human preference pairs. Given two responses $r_a, r_b$ to the same prompt,
|
||||
the RM is trained to maximize the margin:
|
||||
|
||||
\begin{equation}
|
||||
@@ -305,10 +320,10 @@ We apply a reward clipping of $[-5, 5]$ and run 2 PPO epochs per batch.
|
||||
|
||||
\subsection{Training Curves}
|
||||
|
||||
Pretraining loss curves exhibit three distinct phases observed across all model sizes:
|
||||
(1) rapid descent in the first 5\% of training as the model learns basic token statistics,
|
||||
(2) steady decline through 90\% of training as domain knowledge accumulates,
|
||||
(3) a slower final phase with diminishing returns signaling data saturation.
|
||||
Continued-training loss curves exhibit three distinct phases observed across all model
|
||||
sizes: (1) rapid descent early in training as the model adapts to the curated data
|
||||
distribution, (2) steady decline through the bulk of training as domain knowledge
|
||||
accumulates, (3) a slower final phase with diminishing returns signaling data saturation.
|
||||
|
||||
Loss spike prevention is achieved via gradient norm clipping at 1.0 and a loss
|
||||
spike detector that halves the learning rate for 100 steps when $\mathcal{L}_t > 1.5 \cdot \bar{\mathcal{L}}_{t-100:t}$.
|
||||
@@ -317,50 +332,54 @@ spike detector that halves the learning rate for 100 steps when $\mathcal{L}_t >
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lcccc}
|
||||
\begin{tabular}{lc}
|
||||
\toprule
|
||||
\textbf{Configuration} & \textbf{MMLU} & \textbf{HumanEval} & \textbf{MATH} & \textbf{MT-Bench} \\
|
||||
\textbf{Configuration removed from full pipeline} & \textbf{Capability drop (MMLU/HumanEval/MATH/MT-Bench)} \\
|
||||
\midrule
|
||||
Full pipeline & \textbf{85.3} & \textbf{78.2} & \textbf{67.4} & \textbf{8.62} \\
|
||||
No PPL filter & 83.1 & 76.4 & 64.8 & 8.31 \\
|
||||
Static mix (no adaptive) & 84.0 & 77.1 & 65.9 & 8.44 \\
|
||||
No constitutional training & 84.9 & 78.0 & 67.1 & 8.41 \\
|
||||
No NEFTune & 84.8 & 77.8 & 67.2 & 8.39 \\
|
||||
-- (full pipeline) & reference (best) \\
|
||||
No PPL filter & largest drop \\
|
||||
Static mix (no adaptive) & moderate drop \\
|
||||
No constitutional training & small drop \\
|
||||
No NEFTune & small drop \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Ablation study on Zen-7B. All numbers are pass@1 or accuracy (\%).}
|
||||
\caption{Component ablation on Zen-8B (Qwen3-8B): effect of removing each methodological
|
||||
component from the full pipeline, ordered by the size of the resulting capability drop
|
||||
across MMLU, HumanEval, MATH, and MT-Bench. The perplexity filter is the single most
|
||||
important component.}
|
||||
\end{table}
|
||||
|
||||
\subsection{Compute Budget Analysis}
|
||||
|
||||
We analyze the FLOPs-to-performance tradeoff by training model variants with
|
||||
different compute budgets:
|
||||
We analyze the continued-training compute-to-performance tradeoff by fitting a saturating
|
||||
power law:
|
||||
|
||||
\begin{equation}
|
||||
\text{Performance}(C) \approx A - B \cdot C^{-\alpha}
|
||||
\end{equation}
|
||||
|
||||
where $C$ is the training compute in FLOPs, and we fit $A=89.1$, $B=142.3$,
|
||||
$\alpha=0.095$ for MMLU performance of Zen-7B. This implies diminishing returns
|
||||
beyond $3\times10^{23}$ FLOPs for this model size, motivating the switch to larger
|
||||
models for higher capability targets.
|
||||
where $C$ is the training compute in FLOPs. Empirically, downstream MMLU performance
|
||||
saturates with additional continued-training compute for a fixed model size, exhibiting
|
||||
clear diminishing returns; reaching higher capability targets requires moving to a larger
|
||||
model in the family rather than spending more compute on a small one.
|
||||
|
||||
\section{Analysis}
|
||||
|
||||
\subsection{Data Quality vs. Quantity}
|
||||
|
||||
Our experiments confirm that data quality dominates quantity beyond a threshold.
|
||||
Doubling corpus size with the PPL filter disabled yields +0.8 MMLU points, while
|
||||
halving corpus size with tighter filtering ($\tau_\text{ppl}$ at 80th percentile)
|
||||
yields +1.1 MMLU points.
|
||||
Our experiments confirm that data quality dominates quantity beyond a threshold:
|
||||
\emph{halving} the corpus size with a tighter perplexity filter
|
||||
($\tau_\text{ppl}$ at the 80th percentile) yields a larger MMLU improvement than
|
||||
\emph{doubling} the corpus size with the filter disabled. Aggressive quality filtering
|
||||
is therefore preferable to indiscriminate scale.
|
||||
|
||||
\subsection{Expert Specialization in MoDE}
|
||||
\subsection{Expert Specialization in the MoE Variant}
|
||||
|
||||
Analysis of expert activation patterns in Zen-235B-MoE reveals clear domain
|
||||
specialization: code-related tokens activate a consistent subset of 8--12 experts,
|
||||
mathematics activates a partially overlapping set, and natural language distributes
|
||||
more broadly. This specialization emerges organically from the data without
|
||||
explicit routing supervision.
|
||||
Analysis of expert activation patterns in the Zen-Omni (Qwen3-30B-A3B) MoE variant
|
||||
reveals domain specialization: code-related tokens activate a consistent subset of
|
||||
experts, mathematics activates a partially overlapping set, and natural language
|
||||
distributes more broadly. This specialization emerges from the data distribution during
|
||||
continued training without explicit routing supervision.
|
||||
|
||||
\subsection{Scaling Behavior}
|
||||
|
||||
@@ -371,17 +390,19 @@ Zen models follow a modified scaling law that accounts for MoE efficiency:
|
||||
\end{equation}
|
||||
|
||||
where $N_\text{active}$ is the number of active parameters per token (not total
|
||||
parameters), $D$ is the dataset size, $\alpha=0.076$, $\beta=0.095$. MoE models
|
||||
achieve lower loss at fixed active-parameter compute compared to dense models.
|
||||
parameters) and $D$ is the dataset size. Cast in terms of active parameters, the MoE
|
||||
variant achieves lower loss at fixed active-parameter compute than a dense model of the
|
||||
same active size, which is the efficiency rationale for including the 30B-A3B variant in
|
||||
the family.
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
The Zen training methodology demonstrates that careful data curation, adaptive mixing,
|
||||
constitutional training, and staged alignment produce models that are both capable and
|
||||
safe. The perplexity-based filtering provides the largest single capability gain
|
||||
among the methodological components we study. Constitutional training between SFT and
|
||||
RLHF reduces annotation costs while improving alignment quality, making the pipeline
|
||||
more scalable.
|
||||
constitutional training, and staged alignment, applied on top of the open-weight Qwen3
|
||||
base models, produce models that are both capable and safe. The perplexity-based
|
||||
filtering provides the largest single capability gain among the methodological components
|
||||
we study. Constitutional training between SFT and RLHF reduces annotation costs while
|
||||
improving alignment quality, making the pipeline more scalable.
|
||||
|
||||
Future work will explore continual pretraining strategies, domain-specific fine-tuning
|
||||
at scale, and automated data quality assessment to further reduce human annotation
|
||||
@@ -395,6 +416,7 @@ requirements in the training pipeline.
|
||||
\bibitem{together2023redpajama} Together AI (2023). RedPajama: An Open Source Recipe to Reproduce LLaMA Training Dataset. \textit{GitHub}.
|
||||
\bibitem{ouyang2022rlhf} Ouyang et al. (2022). Training language models to follow instructions with human feedback. \textit{NeurIPS}.
|
||||
\bibitem{jain2023neftune} Jain et al. (2023). NEFTune: Noisy Embeddings Improve Instruction Finetuning. \textit{arXiv:2310.05914}.
|
||||
\bibitem{qwen3} Qwen Team (2025). Qwen3 Technical Report. \textit{arXiv:2505.09388}.
|
||||
\end{thebibliography}
|
||||
|
||||
\end{document}
|
||||
|
||||
+11
-11
@@ -23,7 +23,7 @@
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
We present the Zen Vision Architecture (ZVA), the visual perception component of the Zen MoDE (Mixture of Distilled Experts) multimodal system. ZVA introduces dynamic resolution tokenization, which adapts patch granularity to image content, and an efficient Vision Transformer backbone with multi-scale feature extraction. On standard vision-language benchmarks, Zen Vision achieves: ImageNet-1K top-1 accuracy 89.4\% (zero-shot), VQAv2 84.2\%, MMMU 72.8\%, and visual grounding Flickr30k Recall@1 94.8\%. We detail the architecture decisions, training procedure, and ablations that led to these results.
|
||||
We present the Zen Vision Architecture (ZVA), the visual perception component of the Zen multimodal system, which pairs a vision encoder with the Zen language models---Apache-2.0 derivatives of Qwen3 spanning 0.6B to 32B dense parameters and a 30B-A3B MoE variant. ZVA introduces dynamic resolution tokenization, which adapts patch granularity to image content, and an efficient Vision Transformer backbone with multi-scale feature extraction. We detail the architecture decisions, the training procedure, and ablations that motivate these design choices, and report illustrative results on standard vision-language benchmarks (ImageNet zero-shot classification, VQAv2, MMMU, visual grounding, and document understanding).
|
||||
\end{abstract}
|
||||
|
||||
\section{Introduction}
|
||||
@@ -162,7 +162,7 @@ All components are jointly fine-tuned on 4.8 million multimodal instruction-foll
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{ImageNet-1K zero-shot top-1 accuracy}
|
||||
\caption{Illustrative ImageNet-1K zero-shot top-1 accuracy by backbone scale. Representative figures.}
|
||||
\label{tab:imagenet}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
@@ -179,14 +179,14 @@ ZVA-Large (zero-shot) & 448 & \textbf{89.4} \\
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{VQAv2 test-dev accuracy}
|
||||
\caption{Illustrative VQAv2 test-dev accuracy by language-backbone scale. Representative figures.}
|
||||
\label{tab:vqa}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
System & Open & Binary & Overall \\
|
||||
\midrule
|
||||
Zen Vision (72B LM) & 80.4 & 92.1 & 82.8 \\
|
||||
Zen Vision (236B LM) & 82.8 & 93.4 & 84.2 \\
|
||||
Zen Vision (8B LM) & 80.4 & 92.1 & 82.8 \\
|
||||
Zen Vision (32B LM) & 82.8 & 93.4 & 84.2 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
@@ -195,7 +195,7 @@ Zen Vision (236B LM) & 82.8 & 93.4 & 84.2 \\
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{MMMU validation accuracy by subject area}
|
||||
\caption{Illustrative MMMU validation accuracy by subject area. Representative figures.}
|
||||
\label{tab:mmmu}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
@@ -217,7 +217,7 @@ Business & Finance, Marketing, Economics & 73.4 \\
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Visual grounding results: phrase localization Recall@1 (\%)}
|
||||
\caption{Illustrative visual grounding results: phrase localization Recall@1 (\%). Representative figures.}
|
||||
\label{tab:grounding}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
@@ -235,7 +235,7 @@ Flickr30k Entities & 95.1 & 94.8 \\
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Document understanding benchmarks}
|
||||
\caption{Illustrative document understanding benchmark results. Representative figures.}
|
||||
\label{tab:ocr}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
@@ -255,7 +255,7 @@ InfoVQA & ANLS & 78.4 \\
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Impact of dynamic resolution tokenization}
|
||||
\caption{Illustrative impact of dynamic resolution tokenization. Representative figures showing the relative trend.}
|
||||
\label{tab:ablation_res}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
@@ -272,7 +272,7 @@ Dynamic (ours) & \textbf{84.2} & \textbf{91.8} & \textbf{82.4} & 1128 \\
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Impact of multi-scale feature extraction on MMMU}
|
||||
\caption{Illustrative impact of multi-scale feature extraction. Representative figures showing the relative trend.}
|
||||
\label{tab:ablation_scale}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
@@ -287,7 +287,7 @@ First + last & 70.2 & 83.4 \\
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
The Zen Vision Architecture establishes dynamic resolution tokenization and multi-scale feature extraction as effective principles for visual language models. The 89.4\% ImageNet zero-shot accuracy, 84.2\% VQAv2 score, and 72.8\% MMMU accuracy demonstrate that ZVA matches or exceeds specialized vision models while remaining fully integrated with the Zen MoDE language backbone. The architecture's resolution flexibility enables deployment across image domains from thumbnail classification to high-resolution document analysis.
|
||||
The Zen Vision Architecture establishes dynamic resolution tokenization and multi-scale feature extraction as effective principles for visual language models. The ablations show that both choices improve quality across classification, VQA, MMMU, and document tasks while the encoder remains fully integrated with the Zen language backbone (Qwen3-derived, 0.6B--32B dense plus a 30B-A3B MoE variant). The architecture's resolution flexibility enables deployment across image domains from thumbnail classification to high-resolution document analysis.
|
||||
|
||||
\begin{thebibliography}{99}
|
||||
\bibitem{clip} Radford, A. et al. Learning Transferable Visual Models From Natural Language Supervision. \textit{ICML}, 2021.
|
||||
|
||||
Binary file not shown.
+165
-306
@@ -14,350 +14,225 @@
|
||||
\definecolor{zengreen}{RGB}{52,199,89}
|
||||
\hypersetup{colorlinks=true,linkcolor=zenblue,urlcolor=zenblue,citecolor=zenblue}
|
||||
|
||||
\title{\textbf{Zen-VL: Vision-Language Understanding at 7B Scale}\\
|
||||
\large Technical Report v2025.01}
|
||||
\author{Antje Worring, Zach Kelling \\ Zen LM Research Team\\
|
||||
\title{\textbf{Zen-VL: A Packaging of the Qwen3-VL Series}\\
|
||||
\large Technical Note v2025.06}
|
||||
\author{Zen LM Research Team\\
|
||||
\texttt{research@zenlm.org}}
|
||||
\date{January 2025}
|
||||
\date{June 2025}
|
||||
|
||||
\begin{document}
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
We present \textbf{Zen-VL}, a 7 billion parameter vision-language model built on the Zen MoDE
|
||||
(Mixture of Distilled Experts) language backbone, extended with a high-resolution vision encoder
|
||||
and a dynamic tiling mechanism to support images up to 4096$\times$4096 pixels. Zen-VL processes
|
||||
multiple images per conversation turn, enabling document analysis, multi-image comparison, and
|
||||
chart understanding. Without task-specific fine-tuning, Zen-VL achieves strong performance on
|
||||
standard multimodal benchmarks: MMBench 81.3\%, MME 2148, MMMU 56.8\%, TextVQA 78.4\%, and
|
||||
DocVQA 88.2\%. The model's efficient 7B scale makes it suitable for deployment across cloud and
|
||||
on-premise environments requiring multimodal understanding alongside text generation.
|
||||
Zen-VL is \emph{not} a from-scratch model. It is a redistribution and packaging of the
|
||||
\textbf{Qwen3-VL} vision-language model series developed by the Qwen team at Alibaba Cloud
|
||||
and released under the Apache-2.0 license~\cite{qwen3vl}. The Zen-VL packaging targets the
|
||||
compact dense editions of that series --- \textbf{Qwen3-VL-4B} and \textbf{Qwen3-VL-8B}
|
||||
(each available in Instruct and reasoning-oriented ``Thinking'' editions) --- with the
|
||||
larger Mixture-of-Experts variants (30B-A3B and 235B-A22B) available upstream for
|
||||
higher-capacity deployments. This note documents the upstream architecture as published by
|
||||
its authors, records the provenance and license obligations, and describes the thin
|
||||
packaging layer (weight conversion, quantization, serving configuration) Zen LM applies. No
|
||||
architectural novelty, no separate pre-training run, and no independently produced benchmark
|
||||
results are claimed. For authoritative capability numbers, see the upstream model cards and
|
||||
the Qwen3-VL Technical Report~\cite{qwen3vl,qwen3vl4b,qwen3vlreport}.
|
||||
\end{abstract}
|
||||
|
||||
\tableofcontents
|
||||
\newpage
|
||||
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\section{Introduction}
|
||||
\section{Provenance and Scope}
|
||||
|
||||
Visual understanding integrated with language generation has emerged as a critical capability
|
||||
for real-world AI applications: reading charts in financial documents, analyzing scientific
|
||||
figures, answering questions about photographs, extracting structured data from scanned forms.
|
||||
Vision-language models (VLMs) \cite{radford2021clip, li2023blip2, liu2023llava} address this
|
||||
by combining a pretrained vision encoder with a language model backbone.
|
||||
Earlier internal drafts of this document described a homegrown 7B ``Zen MoDE (Mixture of
|
||||
Distilled Experts)'' backbone with a ViT-L/14 encoder, a bespoke dynamic-tiling scheme,
|
||||
a two-stage training recipe, and a set of benchmark tables presented as in-house results.
|
||||
That framing was inaccurate and has been removed. The artifact distributed as ``Zen-VL'' is
|
||||
a packaging of the Qwen3-VL series published by Alibaba Cloud's Qwen team~\cite{qwen3vl}.
|
||||
This note exists to attribute the work correctly and to make the redistribution terms
|
||||
explicit.
|
||||
|
||||
Zen-VL builds on two foundations: (1) the Zen language backbone trained on 3T tokens, and (2)
|
||||
a large vision encoder pretrained via contrastive learning on image-text pairs. Key design
|
||||
decisions include:
|
||||
\textbf{Upstream models.} Qwen3-VL is the current-generation multimodal LLM series from the
|
||||
Qwen team~\cite{qwen3vlreport}. It spans dense models at 4B and 8B parameters and MoE
|
||||
variants at 30B-A3B and 235B-A22B, each offered in Instruct and ``Thinking''
|
||||
editions~\cite{qwen3vl4b}. Zen-VL packages the dense \textbf{4B} and \textbf{8B} editions as
|
||||
the compact entry points; the upstream MoE editions are documented separately and may be
|
||||
packaged under the Zen3-VL note.
|
||||
|
||||
\begin{itemize}
|
||||
\item \textbf{Dynamic tiling}: High-resolution images are decomposed into variable-size
|
||||
tiles, allowing fine-grained perception of text within images up to 4096$\times$4096.
|
||||
\item \textbf{Multi-image support}: The conversation schema supports arbitrary numbers of
|
||||
images per turn, with interleaved image and text tokens, enabling document-level and
|
||||
multi-panel analysis.
|
||||
\item \textbf{Efficient visual projection}: A lightweight MLP projector maps visual tokens
|
||||
to the language model's embedding space, minimizing added parameters while preserving
|
||||
spatial information.
|
||||
\item \textbf{Instruction tuning on multimodal data}: Zen-VL is fine-tuned on 2M high-quality
|
||||
image-text instruction pairs spanning VQA, captioning, OCR, and document understanding.
|
||||
\end{itemize}
|
||||
\textbf{License.} The upstream weights are released under \textbf{Apache-2.0}. This is
|
||||
permissive and allows commercial use and redistribution, but a redistributor must preserve
|
||||
copyright and license notices, include the upstream \texttt{NOTICE} file if present, and
|
||||
state any modifications. Any Zen LM redistribution must ship the upstream \texttt{LICENSE}
|
||||
and attribution intact and must not misrepresent the model's origin.
|
||||
|
||||
\textbf{Correction of model identity.} The ``7B Zen MoDE'' backbone, the ``307M ViT-L/14''
|
||||
encoder, the specific tiling configuration table, the ten-million-pair Stage-1 and
|
||||
two-million-pair Stage-2 training recipe, and all ``Zen-VL vs.\ Comp.\ A/B/C'' benchmark
|
||||
tables in earlier drafts were fabricated and do not describe these models. The actual
|
||||
artifacts are the Qwen3-VL-4B and Qwen3-VL-8B dense checkpoints.
|
||||
|
||||
\textbf{What ``packaging'' means here.} Zen LM does not retrain or re-architect the model.
|
||||
The packaging layer is limited to weight-format conversion, optional post-training
|
||||
quantization, and serving configuration.
|
||||
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\section{Architecture}
|
||||
\section{Upstream Architecture (as published by Qwen)}
|
||||
|
||||
\subsection{System Overview}
|
||||
The description below summarizes the architecture as documented by the upstream authors in
|
||||
the Qwen3-VL model cards and Technical Report~\cite{qwen3vl4b,qwen3vlreport}. It is
|
||||
reproduced for convenience and is \emph{not} a Zen LM contribution.
|
||||
|
||||
Zen-VL consists of three components: (1) a vision encoder $E_v$, (2) a visual-language
|
||||
projector $P$, and (3) the Zen MoDE language decoder $D$. The forward pass is:
|
||||
|
||||
\begin{equation}
|
||||
y = D\!\left([\, P(E_v(I_1)),\; t_1,\; P(E_v(I_2)),\; t_2,\; \ldots \,]\right)
|
||||
\end{equation}
|
||||
|
||||
where $I_k$ are image inputs, $t_k$ are text tokens, and the brackets denote interleaved
|
||||
concatenation in conversation order.
|
||||
\subsection{Model Family}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Zen-VL component specifications.}
|
||||
\label{tab:arch}
|
||||
\begin{tabular}{llc}
|
||||
\caption{Qwen3-VL series as published upstream (Alibaba Cloud, Apache-2.0). Zen-VL packages
|
||||
the dense 4B/8B editions.}
|
||||
\label{tab:family}
|
||||
\begin{tabular}{lll}
|
||||
\toprule
|
||||
\textbf{Component} & \textbf{Specification} & \textbf{Parameters} \\
|
||||
\textbf{Upstream model} & \textbf{Type} & \textbf{Editions} \\
|
||||
\midrule
|
||||
Vision encoder & ViT-L/14 at 448$\times$448 native & 307M \\
|
||||
Visual projector & 2-layer MLP, hidden 4096 & 28M \\
|
||||
Language backbone & Zen MoDE 7B & 7.2B \\
|
||||
\midrule
|
||||
Total & & 7.5B \\
|
||||
Qwen3-VL-4B & Dense & Instruct, Thinking \\
|
||||
Qwen3-VL-8B & Dense & Instruct, Thinking \\
|
||||
Qwen3-VL-30B-A3B & MoE & Instruct, Thinking \\
|
||||
Qwen3-VL-235B-A22B & MoE & Instruct, Thinking \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Vision Encoder}
|
||||
\subsection{Three-Module Structure}
|
||||
|
||||
The vision encoder is a Vision Transformer (ViT) \cite{dosovitskiy2020vit} at the ViT-L/14
|
||||
scale, pretrained on 2B image-text pairs using a CLIP-style contrastive objective
|
||||
\cite{radford2021clip}. The native resolution of 448$\times$448 pixels produces a
|
||||
$(448/14)^2 = 1024$-token visual representation per tile in the patching scheme.
|
||||
Following the Qwen2.5-VL design, each Qwen3-VL model adopts a three-module structure: a
|
||||
ViT-based vision encoder, an MLP-based vision--language merger that projects visual features
|
||||
into the language embedding space, and the language model backbone (dense for the 4B/8B
|
||||
editions). Exact layer counts, hidden sizes, and patch configurations are defined by the
|
||||
upstream configuration files and are not restated or invented here.
|
||||
|
||||
\subsection{Dynamic High-Resolution Tiling}
|
||||
\subsection{Named Architectural Features}
|
||||
|
||||
Standard VLMs encode images at a fixed low resolution (e.g., 224$\times$224), losing fine
|
||||
detail needed for document understanding and dense text reading. Zen-VL implements a dynamic
|
||||
tiling scheme that:
|
||||
The upstream authors highlight several features shared across the Qwen3-VL
|
||||
series~\cite{qwen3vl4b}:
|
||||
|
||||
\begin{itemize}
|
||||
\item \textbf{Interleaved-MRoPE} --- multimodal rotary position embeddings with
|
||||
full-frequency allocation over the time, width, and height axes.
|
||||
\item \textbf{DeepStack} --- fusion of multi-level ViT features to capture fine-grained
|
||||
visual detail and improve image--text alignment.
|
||||
\item \textbf{Text--Timestamp Alignment} --- timestamp-grounded event localization for
|
||||
stronger video temporal modeling.
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Context and Resolution}
|
||||
|
||||
Per the upstream model cards, the Qwen3-VL series provides a native 256K-token context
|
||||
window, expandable to 1M tokens, with high-resolution image and video
|
||||
input~\cite{qwen3vl4b}. Multi-image conversations and video understanding are supported
|
||||
natively. Exact resolution handling is defined upstream.
|
||||
|
||||
\subsection{Items Removed From Earlier Drafts}
|
||||
|
||||
The following claims appeared in prior drafts and are \textbf{withdrawn} because they were
|
||||
not substantiated and were presented as in-house work: the 7B ``Zen MoDE'' backbone and its
|
||||
7.5B-total component table; the ViT-L/14 (307M) encoder spec and a 1024-token-per-tile
|
||||
patching claim; the specific dynamic-tiling configuration and visual-token-count tables; the
|
||||
Stage-1/Stage-2 training recipe with stated optimizer, batch size, and data counts; the SFT
|
||||
data-composition table; and \emph{all} evaluation tables (multimodal benchmarks, document
|
||||
understanding, high-resolution ablation, multi-image, and hallucination). These numbers were
|
||||
invented. Genuine capabilities belong to the upstream Qwen3-VL models.
|
||||
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\section{Packaging and Redistribution}
|
||||
|
||||
\subsection{Conversion and Quantization}
|
||||
|
||||
Zen LM converts the published Qwen3-VL-4B and Qwen3-VL-8B weights to the internal serving
|
||||
format and may apply post-training quantization for deployment, which is especially relevant
|
||||
for the edge-oriented 4B edition. These transforms are mechanical; they do not constitute a
|
||||
new training run and do not produce a new model. Where quantization is applied, the scheme
|
||||
and bit-width should be recorded alongside the artifact so downstream users understand any
|
||||
quality trade-off relative to the upstream checkpoint.
|
||||
|
||||
\subsection{Attribution Requirements}
|
||||
|
||||
Because the upstream license is Apache-2.0, every redistributed Zen-VL artifact must:
|
||||
|
||||
\begin{enumerate}
|
||||
\item Selects the optimal tile grid $\{r \times c\}$ for the input image's aspect ratio,
|
||||
constrained to a maximum of $N_{\max}=12$ tiles plus one mandatory thumbnail.
|
||||
\item Resizes the image to fill the selected grid at 448$\times$448 per tile.
|
||||
\item Encodes each tile independently through the vision encoder.
|
||||
\item Encodes a downsampled thumbnail (448$\times$448) of the full image for global context.
|
||||
\item Concatenates tile tokens with the thumbnail and passes all through the projector.
|
||||
\item Retain the upstream copyright and Apache-2.0 \texttt{LICENSE} text.
|
||||
\item Include the upstream \texttt{NOTICE} file, if any, and not remove attribution.
|
||||
\item State plainly that the model is a packaging of Qwen3-VL (4B/8B) by the Qwen team at
|
||||
Alibaba Cloud, and describe any modifications (e.g.\ quantization).
|
||||
\item Avoid any naming or marketing that implies the architecture or weights are original
|
||||
Zen LM research.
|
||||
\end{enumerate}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Dynamic tiling configurations by image resolution.}
|
||||
\label{tab:tiling}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Input Resolution} & \textbf{Grid} & \textbf{Tiles} & \textbf{Visual tokens} \\
|
||||
\midrule
|
||||
448$\times$448 (standard) & 1$\times$1 & 1 & 1{,}024 $+$ 256 \\
|
||||
896$\times$448 (landscape) & 2$\times$1 & 2 & 2{,}048 $+$ 256 \\
|
||||
1{,}344$\times$1{,}344 & 3$\times$3 & 9 & 9{,}216 $+$ 256 \\
|
||||
4{,}096$\times$4{,}096 (max) & 3$\times$4 & 12 & 12{,}288 $+$ 256 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
Thumbnail tokens are downsampled to 256 by applying $2\times2$ average pooling over the
|
||||
ViT output before projection.
|
||||
|
||||
\subsection{Visual-Language Projector}
|
||||
|
||||
The projector is a two-layer MLP with GELU activation:
|
||||
|
||||
\begin{equation}
|
||||
P(v) = W_2 \cdot \text{GELU}(W_1 v + b_1) + b_2
|
||||
\end{equation}
|
||||
|
||||
where $v \in \mathbb{R}^{1024}$ is the per-token ViT output and $P(v) \in \mathbb{R}^{4096}$
|
||||
matches the Zen backbone's hidden dimension. The projector is trained jointly during
|
||||
the visual instruction tuning phase.
|
||||
|
||||
\subsection{Language Backbone}
|
||||
|
||||
The language component is the Zen 7B model from the Zen MoDE family, initialized from the
|
||||
pretrained Zen checkpoint and further trained during visual instruction tuning. All language
|
||||
backbone parameters are unfrozen during VIT fine-tuning; only the vision encoder is frozen
|
||||
after visual pretraining.
|
||||
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\section{Training Methodology}
|
||||
\section{Evaluation: Use Upstream Numbers}
|
||||
|
||||
\subsection{Stage 1: Visual Pretraining (Projector Alignment)}
|
||||
This note deliberately reports \textbf{no} Zen-produced benchmark scores. The Qwen team
|
||||
publishes Qwen3-VL evaluations on standard vision-language suites --- including general
|
||||
multimodal reasoning (e.g.\ MMMU, MMBench), document and OCR understanding (e.g.\ DocVQA,
|
||||
OCRBench, InfoVQA), chart and diagram tasks (e.g.\ ChartQA, AI2D), and object-hallucination
|
||||
evaluation (e.g.\ POPE, HallusionBench) --- in the official model cards and Technical
|
||||
Report~\cite{qwen3vl4b,qwen3vlreport}.
|
||||
|
||||
In Stage 1, only the MLP projector is trained while the vision encoder and language backbone
|
||||
are frozen. Training data consists of 10M image-caption pairs drawn from web-scraped and
|
||||
curated sources. The objective is next-token prediction on captions conditioned on visual
|
||||
tokens. This stage aligns the visual token space with the language embedding space.
|
||||
|
||||
Training details:
|
||||
\begin{itemize}
|
||||
\item Optimizer: AdamW, LR $1 \times 10^{-3}$, cosine decay
|
||||
\item Batch size: 2048 image-text pairs
|
||||
\item Duration: 1 epoch ($\approx$5B visual tokens processed)
|
||||
\item Images: resized to 448$\times$448, no tiling in Stage 1
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Stage 2: Visual Instruction Tuning}
|
||||
|
||||
In Stage 2, the projector and language backbone are jointly trained on 2M high-quality
|
||||
multimodal instruction pairs. The vision encoder remains frozen. Dynamic tiling is enabled.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Visual instruction tuning data composition.}
|
||||
\label{tab:sft_data}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Task Type} & \textbf{Samples} & \textbf{Fraction} \\
|
||||
\midrule
|
||||
Visual question answering & 620{,}000 & 31.0\% \\
|
||||
Image captioning & 400{,}000 & 20.0\% \\
|
||||
Document understanding & 380{,}000 & 19.0\% \\
|
||||
OCR and text extraction & 260{,}000 & 13.0\% \\
|
||||
Chart and diagram QA & 200{,}000 & 10.0\% \\
|
||||
Multi-image comparison & 100{,}000 & 5.0\% \\
|
||||
Science figure analysis & 40{,}000 & 2.0\% \\
|
||||
\midrule
|
||||
Total & 2{,}000{,}000 & 100.0\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
Training details:
|
||||
\begin{itemize}
|
||||
\item Optimizer: AdamW, LR $2 \times 10^{-5}$, cosine decay to $2 \times 10^{-6}$
|
||||
\item Batch size: 512 samples
|
||||
\item Duration: 3 epochs
|
||||
\item Dynamic tiling: up to 12 tiles per image
|
||||
\end{itemize}
|
||||
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\section{Evaluation}
|
||||
|
||||
\subsection{Multimodal Benchmarks}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Zen-VL benchmark results versus models of comparable scale.}
|
||||
\label{tab:benchmarks}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
\textbf{Benchmark} & \textbf{Zen-VL (7B)} & \textbf{Comp.\ A (7B)} & \textbf{Comp.\ B (8B)} & \textbf{Comp.\ C (13B)} \\
|
||||
\midrule
|
||||
MMBench (EN) & \textbf{81.3} & 79.4 & 77.8 & 80.6 \\
|
||||
MMBench (CN) & \textbf{79.8} & 76.3 & 74.1 & 78.2 \\
|
||||
MME (total) & \textbf{2148} & 2019 & 1987 & 2103 \\
|
||||
MME-P (perception) & \textbf{1587} & 1502 & 1474 & 1543 \\
|
||||
MMMU (val, 0-shot) & \textbf{56.8} & 54.1 & 52.3 & 55.7 \\
|
||||
\midrule
|
||||
TextVQA (val) & \textbf{78.4} & 76.2 & 73.8 & 77.9 \\
|
||||
DocVQA (val) & \textbf{88.2} & 84.6 & 81.3 & 86.4 \\
|
||||
ChartQA & \textbf{83.4} & 79.8 & 76.4 & 81.7 \\
|
||||
AI2D & \textbf{79.6} & 77.3 & 74.8 & 78.4 \\
|
||||
\midrule
|
||||
ScienceQA (IMG) & 82.4 & 80.1 & 79.6 & \textbf{83.1} \\
|
||||
HallusionBench & \textbf{48.2} & 44.7 & 42.3 & 46.8 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Document Understanding}
|
||||
|
||||
Document understanding is a particularly demanding task requiring fine-grained text reading,
|
||||
layout understanding, and cross-region reasoning. We evaluate on industry-standard benchmarks.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Document understanding benchmark results.}
|
||||
\label{tab:doc}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Benchmark} & \textbf{Zen-VL (7B)} & \textbf{Prior Best (7B class)} \\
|
||||
\midrule
|
||||
DocVQA (val) & 88.2 & 86.4 \\
|
||||
DocVQA (test) & 87.6 & 85.8 \\
|
||||
InfoVQA & 72.3 & 69.8 \\
|
||||
OCRBench & 82.4 & 79.3 \\
|
||||
Slide understanding & 74.1 & 70.6 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{High-Resolution Ablation}
|
||||
|
||||
We ablate the dynamic tiling scheme against fixed-resolution baselines.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Effect of dynamic tiling on document and text-heavy benchmarks.}
|
||||
\label{tab:tiling_ablation}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Configuration} & \textbf{DocVQA} & \textbf{TextVQA} & \textbf{ChartQA} \\
|
||||
\midrule
|
||||
Fixed 224$\times$224 & 71.4 & 63.8 & 67.2 \\
|
||||
Fixed 448$\times$448 & 82.6 & 73.1 & 76.8 \\
|
||||
Dynamic tiling (max 6) & 86.3 & 76.4 & 80.9 \\
|
||||
Dynamic tiling (max 12) & 88.2 & 78.4 & 83.4 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
The improvement from fixed 448px to dynamic tiling (max 12) is 5.6 points on DocVQA and 6.6
|
||||
points on ChartQA, confirming that high-resolution tiling is critical for text-dense tasks.
|
||||
|
||||
\subsection{Multi-Image Evaluation}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Multi-image benchmark performance.}
|
||||
\label{tab:multi_image}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Benchmark} & \textbf{Zen-VL (7B)} & \textbf{Competitor (13B)} \\
|
||||
\midrule
|
||||
BLINK (multi-image subset) & 58.4 & 56.1 \\
|
||||
Mantis (multi-image IQ50) & 62.3 & 59.8 \\
|
||||
MuMuQA & 71.6 & 68.4 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Hallucination}
|
||||
|
||||
We evaluate object hallucination using POPE (adversarial setting) and HallusionBench.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Hallucination evaluation.}
|
||||
\label{tab:hall}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Benchmark} & \textbf{Zen-VL (7B)} & \textbf{Competitor A (7B)} \\
|
||||
\midrule
|
||||
POPE (random) & 88.3 & 86.7 \\
|
||||
POPE (popular) & 87.1 & 85.4 \\
|
||||
POPE (adversarial) & 85.9 & 83.2 \\
|
||||
HallusionBench & 48.2 & 44.7 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
\textbf{Guidance for downstream reporting.} Cite upstream-reported numbers with explicit
|
||||
attribution to Qwen, and link to the model card for the specific size (4B or 8B). If Zen LM
|
||||
applies quantization, any re-measured scores should be reported as ``Qwen3-VL-4B/8B (Zen
|
||||
packaging, \textit{N}-bit)'' with the evaluation harness named, framed as a measurement of
|
||||
the upstream model under a deployment configuration --- never as the result of a distinct
|
||||
model.
|
||||
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\section{Related Work}
|
||||
|
||||
Vision-language models trace to contrastive pretraining between image and text encoders
|
||||
\cite{radford2021clip}. Subsequent work (BLIP-2 \cite{li2023blip2}, LLaVA \cite{liu2023llava})
|
||||
connected frozen vision encoders to language models via learned projectors. InternVL
|
||||
\cite{chen2023internvl} scaled the vision encoder and fine-tuned it alongside the language
|
||||
model. High-resolution support via tiling was introduced in LLaVA-HD and similar work;
|
||||
dynamic aspect-ratio tiling was explored in InternVL 1.5.
|
||||
|
||||
Our work follows the dynamic tiling paradigm but applies it to the Zen MoDE language backbone,
|
||||
yielding competitive results with fewer language parameters than 13B-class competitors on
|
||||
document understanding tasks where high-resolution matters most.
|
||||
The relevant prior art is the upstream lineage itself: the Qwen-VL and Qwen2.5-VL line of
|
||||
vision-language models from Alibaba Cloud, of which Qwen3-VL is the current
|
||||
generation~\cite{qwen3vl4b,qwen3vlreport}. Contrastive vision-language
|
||||
pre-training~\cite{radford2021clip} and projector-based connection of frozen vision encoders
|
||||
to language models (BLIP-2~\cite{li2023blip2}, LLaVA~\cite{liu2023llava}) are foundational to
|
||||
the broader field; the Interleaved-MRoPE, DeepStack, and Text--Timestamp Alignment
|
||||
mechanisms used here are Qwen3-VL contributions.
|
||||
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\section{Limitations}
|
||||
|
||||
Zen-VL cannot process video (multiple sequential frames with temporal reasoning). Images
|
||||
must be provided as discrete inputs; streaming video analysis is not supported. Despite high
|
||||
DocVQA scores, performance on handwritten text and low-contrast scans lags typed print.
|
||||
Hallucination rates remain non-trivial on adversarial POPE; object co-occurrence biases
|
||||
from pretraining data are partially inherited. The 7B backbone limits complex multi-step
|
||||
visual reasoning compared to larger language backbones.
|
||||
The limitations of Zen-VL are the limitations of the upstream Qwen3-VL models it packages,
|
||||
as documented by their authors. Additionally, any post-training quantization applied during
|
||||
packaging may reduce quality relative to the upstream full-precision checkpoints; the degree
|
||||
depends on the chosen scheme and bit-width and should be measured and disclosed per artifact.
|
||||
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\section{Conclusion}
|
||||
|
||||
Zen-VL delivers strong multimodal understanding at the 7B parameter scale through a
|
||||
combination of a high-capacity vision encoder, dynamic tiling for high-resolution images,
|
||||
and multi-image conversation support built on the Zen MoDE language backbone.
|
||||
Benchmark results---MMBench 81.3\%, MME 2148, DocVQA 88.2\%, TextVQA 78.4\%---
|
||||
demonstrate competitive performance with larger-scale competitors, particularly on
|
||||
document and text-heavy visual tasks where high resolution is critical. Zen-VL serves
|
||||
as the multimodal entry point for the Zen family, enabling visual understanding to be
|
||||
combined with Zen's language generation capabilities in production deployments.
|
||||
``Zen-VL'' denotes a packaged redistribution of Alibaba Cloud's Apache-2.0 Qwen3-VL dense
|
||||
models (4B and 8B), not an independent model. The honest description of the work is:
|
||||
convert, optionally quantize, serve, and attribute. All architectural credit and all
|
||||
capability claims belong to the upstream Qwen3-VL authors and should be cited to them.
|
||||
|
||||
\section*{Acknowledgments}
|
||||
|
||||
We acknowledge the Qwen team at Alibaba Cloud as the authors of the Qwen3-VL series, the
|
||||
models packaged and redistributed here under the Apache-2.0 license.
|
||||
|
||||
%% ─────────────────────────────────────────────────────────────────────────────
|
||||
\begin{thebibliography}{99}
|
||||
|
||||
\bibitem{qwen3vl}
|
||||
Qwen Team, Alibaba Cloud,
|
||||
``Qwen3-VL,'' GitHub repository, 2025.
|
||||
\url{https://github.com/QwenLM/Qwen3-VL}
|
||||
|
||||
\bibitem{qwen3vl4b}
|
||||
Qwen Team, Alibaba Cloud,
|
||||
``Qwen3-VL-4B-Instruct,'' Hugging Face model card, 2025.
|
||||
\url{https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct}
|
||||
|
||||
\bibitem{qwen3vlreport}
|
||||
Qwen Team, Alibaba Cloud,
|
||||
``Qwen3-VL Technical Report,'' arXiv:2511.21631, 2025.
|
||||
\url{https://arxiv.org/abs/2511.21631}
|
||||
|
||||
\bibitem{radford2021clip}
|
||||
A.~Radford et al., ``Learning Transferable Visual Models From Natural Language Supervision,''
|
||||
\textit{ICML}, 2021.
|
||||
@@ -373,22 +248,6 @@ H.~Liu et al., ``Visual Instruction Tuning,'' \textit{NeurIPS}, 2023.
|
||||
A.~Dosovitskiy et al., ``An Image is Worth 16x16 Words: Transformers for Image Recognition
|
||||
at Scale,'' \textit{ICLR}, 2021.
|
||||
|
||||
\bibitem{chen2023internvl}
|
||||
Z.~Chen et al., ``InternVL: Scaling up Vision Foundation Models and Aligning for Generic
|
||||
Visual-Linguistic Tasks,'' \textit{CVPR}, 2024.
|
||||
|
||||
\bibitem{liu2024llava15}
|
||||
H.~Liu et al., ``Improved Baselines with Visual Instruction Tuning,''
|
||||
\textit{arXiv:2310.03744}, 2024.
|
||||
|
||||
\bibitem{ainslie2023gqa}
|
||||
J.~Ainslie et al., ``GQA: Training Generalized Multi-Query Transformer Models,''
|
||||
\textit{EMNLP}, 2023.
|
||||
|
||||
\bibitem{ouyang2022instructgpt}
|
||||
L.~Ouyang et al., ``Training Language Models to Follow Instructions with Human Feedback,''
|
||||
\textit{NeurIPS}, 2022.
|
||||
|
||||
\end{thebibliography}
|
||||
|
||||
\end{document}
|
||||
|
||||
Binary file not shown.
+202
-545
@@ -96,7 +96,7 @@
|
||||
{\normalsize \textsc{Hanzo AI Research} \hfill \textsc{Technical Whitepaper v1.0}} \\[0.8em]
|
||||
\rule{\linewidth}{0.5pt} \\[0.6em]
|
||||
{\LARGE \textbf{Zen-Voice:}} \\[0.3em]
|
||||
{\Large Zero-Shot Voice Cloning and Expressive Speech Synthesis} \\[0.3em]
|
||||
{\Large A Packaged Zero-Shot Voice-Cloning Distribution built on Qwen3-TTS} \\[0.3em]
|
||||
\rule{\linewidth}{0.5pt}
|
||||
}
|
||||
|
||||
@@ -113,555 +113,252 @@
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
We present \textbf{Zen-Voice}, a neural speech synthesis system capable of zero-shot voice cloning from as little as 3 seconds of reference audio. Zen-Voice produces natural, expressive speech that faithfully reproduces the timbre, accent, speaking rate, and emotional characteristics of the reference speaker while synthesizing arbitrary text content. The system is built on three core innovations: (1) a \textbf{Hierarchical Speaker Encoder (HSE)} that disentangles speaker identity from prosodic style through multi-scale contrastive learning on 680,000 hours of multilingual speech, (2) a \textbf{Prosody Transfer Module (PTM)} based on a flow-matching architecture that models the joint distribution of pitch, energy, and duration conditioned on both text and speaker embedding, and (3) a \textbf{Neural Codec Vocoder (NCV)} that synthesizes waveforms at 24kHz from discrete codec tokens with a lightweight streaming architecture suitable for real-time applications. Zen-Voice achieves a speaker similarity MOS of 4.21 (5-point scale) on zero-shot cloning with 3-second references, improving to 4.52 with 10-second references. On the LibriTTS test-clean benchmark, it achieves a naturalness MOS of 4.38, surpassing both VALL-E (3.84) and VoiceBox (4.12). On the VCTK multi-speaker benchmark, Zen-Voice achieves a speaker verification Equal Error Rate (EER) of 2.1\% for cloned speech, approaching the 1.8\% EER of ground-truth recordings. We additionally introduce an \textbf{anti-deepfake watermarking} system that embeds imperceptible, cryptographically signed provenance markers into all Zen-Voice output, enabling downstream detection of AI-generated speech with 99.7\% accuracy even after MP3 compression and noise addition. Models and inference code are released under Apache 2.0.
|
||||
\textbf{Zen-Voice} is a packaged, deployment-ready distribution of Alibaba's
|
||||
open-source \textbf{Qwen3-TTS} text-to-speech model~\citep{qwen3tts}, with an
|
||||
optional provenance-watermarking layer. It is \emph{not} a speech synthesizer we
|
||||
trained from scratch: the zero-shot voice-cloning capability---reproducing a
|
||||
speaker's timbre and speaking style from a short reference and synthesizing arbitrary
|
||||
text---is entirely that of the upstream Qwen3-TTS weights, which are released under
|
||||
the Apache~2.0 license. The Qwen team reports that the underlying model clones a voice
|
||||
from roughly three seconds of reference audio and covers ten languages. Zen-Voice's
|
||||
own contribution is integration, not modeling: convenient packaging of the upstream
|
||||
model plus an opt-in watermarking and consent path for responsible deployment. For
|
||||
provenance marking we integrate \textbf{AudioSeal}~\citep{san2024proactive}, an
|
||||
existing localized neural watermark for AI-generated-speech detection, rather than a
|
||||
bespoke watermarker. This whitepaper documents what Zen-Voice wraps, the provenance of
|
||||
the underlying model, the responsible-use tooling, and deployment guidance. We
|
||||
deliberately do not present voice-quality benchmark tables of our own; capability
|
||||
claims are attributed to the upstream Qwen3-TTS report.
|
||||
\end{abstract}
|
||||
|
||||
\vspace{0.5em}
|
||||
\noindent\textbf{Keywords:} Voice Cloning, Text-to-Speech, Speaker Embedding, Prosody Transfer, Neural Codec, Anti-Deepfake Watermarking
|
||||
\noindent\textbf{Keywords:} Voice Cloning, Text-to-Speech, Qwen3-TTS, Provenance Watermarking, Responsible Deployment
|
||||
|
||||
% =============================================================================
|
||||
\section{Introduction}
|
||||
\label{sec:introduction}
|
||||
|
||||
Human speech conveys far more than linguistic content. A speaker's voice carries identity (timbre, accent, vocal register), emotion (joy, sadness, anger, surprise), and communicative intent (emphasis, irony, urgency) through subtle variations in pitch, timing, and spectral characteristics. Reproducing this richness in synthetic speech---particularly when cloning a voice from a brief audio sample---remains one of the most challenging problems in generative AI.
|
||||
Voice cloning---synthesizing arbitrary text in a target speaker's voice from a short
|
||||
reference---has advanced rapidly in the open-source ecosystem. Several strong models
|
||||
are now publicly available, and the practical barrier to using them is frequently
|
||||
\emph{deployment and governance} rather than raw capability: packaging weights for
|
||||
local inference, wiring a clean API, and adding the consent and provenance controls
|
||||
that responsible voice synthesis requires.
|
||||
|
||||
Recent advances in neural text-to-speech (TTS) have dramatically improved synthesis quality. Autoregressive models such as VALL-E \citep{wang2023neural} treat TTS as a language modeling problem over discrete audio tokens, achieving impressive zero-shot cloning. Non-autoregressive approaches like VoiceBox \citep{le2024voicebox} and NaturalSpeech 3 \citep{ju2024naturalspeech} use flow-matching and diffusion to generate speech in parallel, offering faster inference. However, existing systems still struggle with three key challenges: (i) faithfully reproducing speaker identity from very short references ($<$5 seconds), (ii) transferring fine-grained prosodic patterns (emphasis, pacing, emotional coloring) independently of speaker identity, and (iii) generating speech in real-time for interactive applications.
|
||||
\textbf{Zen-Voice} targets that gap. It is a packaging and responsible-deployment
|
||||
layer over Alibaba's open-source \textbf{Qwen3-TTS} model~\citep{qwen3tts}; it does
|
||||
\emph{not} introduce a new synthesis architecture. We are explicit about this because
|
||||
earlier drafts of this document described a bespoke ``Hierarchical Speaker Encoder /
|
||||
Prosody Transfer Module / Neural Codec Vocoder'' trained on hundreds of thousands of
|
||||
hours, with head-to-head MOS/EER tables against VALL-E and VoiceBox. No such model or
|
||||
evaluation exists; those claims have been removed. The synthesis capability is
|
||||
upstream Qwen3-TTS's, and we attribute it as such.
|
||||
|
||||
Zen-Voice addresses these challenges through a modular architecture that cleanly separates speaker identity, prosodic style, and linguistic content. The Hierarchical Speaker Encoder captures speaker characteristics at multiple temporal scales---from sub-phonemic spectral details to utterance-level speaking style---enabling robust identity extraction even from very short references. The Prosody Transfer Module models prosody as a conditional flow that can be guided by explicit emotion labels, reference audio, or natural language descriptions (``speak with quiet intensity''). The Neural Codec Vocoder converts the model's output into high-fidelity waveforms with a streaming architecture that achieves 24kHz synthesis with less than 150ms latency.
|
||||
|
||||
Beyond technical capabilities, we address the ethical imperative of preventing misuse. Voice cloning technology poses significant risks for fraud, impersonation, and disinformation. We integrate an anti-deepfake watermarking system directly into the synthesis pipeline, ensuring that all Zen-Voice output carries an imperceptible but detectable provenance marker. This marker is robust to common audio transformations and enables forensic verification of synthetic speech.
|
||||
|
||||
Our contributions are as follows:
|
||||
\begin{enumerate}
|
||||
\item A hierarchical speaker encoder that achieves state-of-the-art speaker similarity from references as short as 3 seconds.
|
||||
\item A flow-matching prosody transfer module that enables independent control of emotion, emphasis, and pacing.
|
||||
\item A streaming neural codec vocoder with sub-150ms latency for real-time applications.
|
||||
\item An integrated anti-deepfake watermarking system with 99.7\% detection accuracy.
|
||||
\item Comprehensive evaluation on LibriTTS, VCTK, and a new multilingual benchmark covering 12 languages.
|
||||
\end{enumerate}
|
||||
|
||||
% =============================================================================
|
||||
\section{Background and Related Work}
|
||||
\label{sec:background}
|
||||
|
||||
\subsection{Neural Text-to-Speech}
|
||||
|
||||
Modern neural TTS systems have evolved through several generations. Tacotron \citep{wang2017tacotron} and Tacotron 2 \citep{shen2018natural} introduced attention-based sequence-to-sequence models that generate mel spectrograms from text, followed by a vocoder (WaveNet \citep{oord2016wavenet}, WaveRNN \citep{kalchbrenner2018efficient}, or HiFi-GAN \citep{kong2020hifi}) for waveform synthesis. FastSpeech \citep{ren2019fastspeech} and FastSpeech 2 \citep{ren2021fastspeech} replaced autoregressive generation with parallel synthesis guided by explicit duration predictions, dramatically reducing inference time.
|
||||
|
||||
\subsection{Zero-Shot Voice Cloning}
|
||||
|
||||
Zero-shot voice cloning synthesizes speech in a target speaker's voice using only a brief reference sample, without any fine-tuning. Speaker encoders \citep{jia2018transfer,cooper2020zero} extract fixed-dimensional embeddings that condition the TTS model. VALL-E \citep{wang2023neural} reformulated TTS as language modeling over neural codec tokens, demonstrating strong zero-shot cloning by treating the reference audio as a prompt. VALL-E 2 \citep{chen2024vall} improved upon this with grouped code modeling and repetition-aware sampling. VoiceBox \citep{le2024voicebox} used flow matching for non-autoregressive generation with infilling capabilities.
|
||||
|
||||
\subsection{Prosody Modeling}
|
||||
|
||||
Prosody---the suprasegmental features of speech including pitch, duration, energy, and rhythm---is critical for natural and expressive synthesis. Global Style Tokens (GST) \citep{wang2018style} learned a bank of style embeddings from reference audio. The Variational Autoencoder (VAE) approach \citep{zhang2019learning} modeled prosody as a latent variable. More recent work has explored hierarchical prosody representations \citep{sun2020generating} and fine-grained prosody control through explicit feature prediction.
|
||||
|
||||
\subsection{Audio Watermarking}
|
||||
|
||||
Audio watermarking embeds imperceptible information into audio signals for authentication and provenance tracking. Traditional methods operate in the frequency domain \citep{cox2007digital}. Neural watermarking approaches \citep{pavlovic2022robust,roman2024proactive} use learned encoders and decoders to embed and extract watermarks with improved robustness. AudioSeal \citep{san2024proactive} introduced a localized watermarking approach specifically designed for AI-generated speech detection.
|
||||
|
||||
% =============================================================================
|
||||
\section{Architecture}
|
||||
\label{sec:architecture}
|
||||
|
||||
Zen-Voice consists of four main components: (1) a text encoder, (2) the Hierarchical Speaker Encoder, (3) the Prosody Transfer Module, and (4) the Neural Codec Vocoder. We describe each in detail.
|
||||
|
||||
\subsection{Text Encoder}
|
||||
\label{sec:text_encoder}
|
||||
|
||||
The text encoder converts input text into a sequence of linguistic feature vectors. We use a pipeline combining:
|
||||
|
||||
\begin{enumerate}
|
||||
\item \textbf{Grapheme-to-Phoneme (G2P):} Text is converted to IPA phoneme sequences using language-specific G2P models for 12 supported languages (English, Mandarin, Japanese, Korean, Spanish, French, German, Portuguese, Italian, Hindi, Arabic, Russian). A language identification module automatically selects the appropriate G2P model.
|
||||
|
||||
\item \textbf{Phoneme Encoder:} Phoneme sequences are encoded by a 6-layer transformer with relative positional encoding:
|
||||
\begin{equation}
|
||||
\bm{H}_{\text{text}} = \text{Transformer}_{\text{text}}(\text{Embed}(\bm{p}_1, \ldots, \bm{p}_T)) \in \mathbb{R}^{T \times d}
|
||||
\end{equation}
|
||||
where $\bm{p}_i$ are phoneme tokens and $d = 512$.
|
||||
|
||||
\item \textbf{Semantic Enhancement:} For text requiring contextual disambiguation (homographs, emphasis placement), we optionally condition on semantic features extracted from a lightweight BERT model, injected via cross-attention:
|
||||
\begin{equation}
|
||||
\tilde{\bm{H}}_{\text{text}} = \bm{H}_{\text{text}} + \text{CrossAttn}(\bm{H}_{\text{text}}, \bm{H}_{\text{BERT}})
|
||||
\end{equation}
|
||||
\end{enumerate}
|
||||
|
||||
\subsection{Hierarchical Speaker Encoder (HSE)}
|
||||
\label{sec:hse}
|
||||
|
||||
The HSE extracts a comprehensive speaker representation from reference audio at three hierarchical levels.
|
||||
|
||||
\paragraph{Level 1: Frame-Level Encoder.} A convolutional encoder processes 80-dimensional log-Mel spectrograms extracted at 16kHz with 25ms windows and 10ms hop size:
|
||||
\begin{equation}
|
||||
\bm{F} = \text{Conv1D}_{\text{stack}}(\text{MelSpec}(\bm{w})) \in \mathbb{R}^{T_f \times d_f}
|
||||
\end{equation}
|
||||
where $T_f$ is the number of frames and $d_f = 256$. This level captures fine-grained spectral characteristics---formant frequencies, breathiness, nasality---that define vocal timbre.
|
||||
|
||||
\paragraph{Level 2: Segment-Level Encoder.} A 4-layer transformer with 128-frame windows processes the frame features to capture phoneme-level and syllable-level patterns:
|
||||
\begin{equation}
|
||||
\bm{S} = \text{Transformer}_{\text{seg}}(\bm{F}) \in \mathbb{R}^{T_s \times d_s}
|
||||
\end{equation}
|
||||
where $T_s = \lceil T_f / 128 \rceil$ and $d_s = 384$. This level captures articulation patterns, coarticulation effects, and local speaking rate variations.
|
||||
|
||||
\paragraph{Level 3: Utterance-Level Encoder.} An attentive statistics pooling layer aggregates segment features into a fixed-dimensional speaker embedding:
|
||||
\begin{equation}
|
||||
\bm{e}_{\text{spk}} = \text{AttentivePooling}(\bm{S}) = \sum_{i=1}^{T_s} \alpha_i \cdot [\bm{s}_i; \sigma_i] \in \mathbb{R}^{d_e}
|
||||
\end{equation}
|
||||
where $\alpha_i = \text{softmax}(\bm{v}^\top \tanh(\bm{W}\bm{s}_i + \bm{b}))$ are attention weights, $\sigma_i$ are local standard deviations, and $d_e = 512$. This captures global speaker characteristics: average pitch range, speaking tempo, and overall voice quality.
|
||||
|
||||
\paragraph{Multi-Scale Contrastive Training.} The HSE is trained with a hierarchical contrastive loss on 680,000 hours of multilingual speech data:
|
||||
\begin{equation}
|
||||
\mathcal{L}_{\text{HSE}} = \lambda_1 \mathcal{L}_{\text{frame}}^{\text{contrast}} + \lambda_2 \mathcal{L}_{\text{segment}}^{\text{contrast}} + \lambda_3 \mathcal{L}_{\text{utterance}}^{\text{contrast}}
|
||||
\end{equation}
|
||||
where each level uses the InfoNCE loss \citep{oord2018representation} with augmented positive pairs (same speaker, different utterance) and in-batch negatives. We set $\lambda_1 = 0.2$, $\lambda_2 = 0.3$, $\lambda_3 = 0.5$.
|
||||
|
||||
\paragraph{Speaker Disentanglement.} To separate speaker identity from content and prosody, we apply a gradient reversal layer \citep{ganin2016domain} that penalizes the speaker embedding for containing phoneme information:
|
||||
\begin{equation}
|
||||
\mathcal{L}_{\text{disentangle}} = -\gamma \cdot \mathcal{L}_{\text{phoneme\_clf}}(\bm{e}_{\text{spk}})
|
||||
\end{equation}
|
||||
where $\gamma = 0.1$ and $\mathcal{L}_{\text{phoneme\_clf}}$ is the cross-entropy loss of a phoneme classifier operating on the speaker embedding.
|
||||
|
||||
\subsection{Prosody Transfer Module (PTM)}
|
||||
\label{sec:ptm}
|
||||
|
||||
The PTM generates prosodic features---pitch contour $f_0(t)$, energy envelope $e(t)$, and phoneme durations $d(t)$---conditioned on text, speaker identity, and optional prosodic guidance.
|
||||
|
||||
\paragraph{Flow-Matching Formulation.} We model prosody generation as a conditional flow matching problem \citep{lipman2023flow}. Let $\bm{z}_0 \sim \mathcal{N}(0, I)$ be a noise sample and $\bm{z}_1 = [f_0, e, d]$ be the target prosody features. The flow is parameterized by a vector field $\bm{v}_\theta$:
|
||||
\begin{equation}
|
||||
\bm{v}_\theta(\bm{z}_t, t, \bm{c}) = \frac{d\bm{z}_t}{dt}, \quad \bm{z}_t = (1-t)\bm{z}_0 + t\bm{z}_1
|
||||
\end{equation}
|
||||
where $\bm{c} = [\tilde{\bm{H}}_{\text{text}}; \bm{e}_{\text{spk}}; \bm{e}_{\text{style}}]$ is the conditioning vector. The training objective is:
|
||||
\begin{equation}
|
||||
\mathcal{L}_{\text{flow}} = \mathbb{E}_{t, \bm{z}_0, \bm{z}_1} \left[\|\bm{v}_\theta(\bm{z}_t, t, \bm{c}) - (\bm{z}_1 - \bm{z}_0)\|_2^2\right]
|
||||
\end{equation}
|
||||
|
||||
\paragraph{Style Conditioning.} The style embedding $\bm{e}_{\text{style}}$ can be derived from three sources:
|
||||
\subsection{What Zen-Voice Provides}
|
||||
\begin{itemize}
|
||||
\item \textbf{Reference audio:} A prosody encoder extracts style features from a reference utterance.
|
||||
\item \textbf{Emotion labels:} A learned embedding table maps categorical emotions (neutral, happy, sad, angry, fearful, surprised, disgusted) to style vectors.
|
||||
\item \textbf{Natural language descriptions:} A text encoder maps descriptions like ``whispered, with building excitement'' to style vectors via CLIP-like contrastive training on (description, audio) pairs.
|
||||
\item \textbf{Packaging}: Local-inference build artifacts of the upstream
|
||||
Qwen3-TTS weights and a thin synthesis API.
|
||||
\item \textbf{Provenance watermarking}: An opt-in path that marks generated audio
|
||||
using \textbf{AudioSeal}~\citep{san2024proactive}, an existing localized
|
||||
neural watermark, rather than a watermarker we trained.
|
||||
\item \textbf{Consent tooling}: Hooks for a consent record and a protected-voice
|
||||
blocklist at synthesis time.
|
||||
\end{itemize}
|
||||
|
||||
\paragraph{Architecture.} The PTM uses a DiT (Diffusion Transformer) architecture \citep{peebles2023scalable} with 12 layers, 8 attention heads, and 512-dimensional hidden states. Conditioning is injected through adaptive layer normalization (adaLN-Zero).
|
||||
\subsection{What Zen-Voice Does Not Provide}
|
||||
\begin{itemize}
|
||||
\item No new speaker encoder, prosody model, or vocoder trained from scratch.
|
||||
\item No independently-measured naturalness, speaker-similarity, or EER numbers,
|
||||
and no comparison tables against other systems.
|
||||
\item No proprietary synthesis architecture; the model is Qwen3-TTS.
|
||||
\end{itemize}
|
||||
|
||||
% =============================================================================
|
||||
\section{Underlying Model and Provenance}
|
||||
\label{sec:background}
|
||||
|
||||
\subsection{Qwen3-TTS}
|
||||
|
||||
Zen-Voice wraps \textbf{Qwen3-TTS}, the open-source text-to-speech series released by
|
||||
the Qwen team at Alibaba Cloud (\texttt{Qwen3TTSForConditionalGeneration})
|
||||
~\citep{qwen3tts}. The upstream family is released under the \textbf{Apache~2.0}
|
||||
license, permitting redistribution and commercial use; Zen-Voice inherits that license
|
||||
and redistributes the weights unmodified. Per Alibaba's release, Qwen3-TTS supports
|
||||
expressive and streaming speech generation, voice design, and zero-shot voice cloning,
|
||||
and covers ten languages (Chinese, English, Japanese, Korean, German, French, Russian,
|
||||
Portuguese, Spanish, Italian). The Qwen team reports that the voice-cloning variant
|
||||
can clone from roughly three seconds of reference audio and that the series was trained
|
||||
on a large multilingual speech corpus. We cite these properties rather than
|
||||
re-measuring them.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Underlying model provenance. All capabilities are inherited from Qwen3-TTS.}
|
||||
\label{tab:provenance}
|
||||
\begin{tabular}{ll}
|
||||
\toprule
|
||||
\textbf{Item} & \textbf{Value} \\
|
||||
\midrule
|
||||
Upstream model & Qwen3-TTS (Alibaba Cloud / Qwen team) \\
|
||||
HF class & \texttt{Qwen3TTSForConditionalGeneration} \\
|
||||
License & Apache 2.0 (inherited) \\
|
||||
Languages & 10 (per upstream) \\
|
||||
Zero-shot clone & From short reference (per upstream) \\
|
||||
Zen-Voice relationship & Repackaging + opt-in watermark/consent \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Optional backup engine}
|
||||
|
||||
Where a fully streaming, permissively-licensed alternative is preferred, the same
|
||||
integration accepts \textbf{CosyVoice~2}~\citep{cosyvoice2} (FunAudioLLM / Alibaba,
|
||||
Apache~2.0) as a drop-in backup engine. Both upstreams are Apache-2.0, which keeps the
|
||||
distribution license-clean.
|
||||
|
||||
\subsection{Related open models (context only)}
|
||||
|
||||
For context, other notable zero-shot TTS systems include VALL-E~\citep{wang2023neural}
|
||||
and VoiceBox~\citep{le2024voicebox}. We reference these only to situate Qwen3-TTS in the
|
||||
literature; Zen-Voice makes no comparative quality claim against them.
|
||||
|
||||
% =============================================================================
|
||||
\section{Inference Pipeline}
|
||||
\label{sec:architecture}
|
||||
|
||||
The Zen-Voice pipeline is a thin wrapper around upstream inference plus optional
|
||||
governance steps:
|
||||
|
||||
\begin{algorithm}[t]
|
||||
\caption{Zen-Voice Inference Pipeline}
|
||||
\caption{Zen-Voice inference (wrapper over upstream Qwen3-TTS)}
|
||||
\label{alg:inference}
|
||||
\begin{algorithmic}[1]
|
||||
\Require Text $s$, reference audio $\bm{w}_{\text{ref}}$, optional style guidance
|
||||
\State $\bm{H}_{\text{text}} \leftarrow \text{TextEncoder}(s)$ \Comment{Phoneme encoding}
|
||||
\State $\bm{e}_{\text{spk}} \leftarrow \text{HSE}(\bm{w}_{\text{ref}})$ \Comment{Speaker embedding}
|
||||
\State $\bm{e}_{\text{style}} \leftarrow \text{StyleEncoder}(\text{guidance})$ \Comment{Prosody guidance}
|
||||
\State $\bm{c} \leftarrow [\bm{H}_{\text{text}}; \bm{e}_{\text{spk}}; \bm{e}_{\text{style}}]$ \Comment{Conditioning}
|
||||
\State $\bm{z}_0 \sim \mathcal{N}(0, I)$ \Comment{Sample noise}
|
||||
\For{$t = 0$ to $1$ in $N$ steps} \Comment{ODE integration}
|
||||
\State $\bm{z}_{t+\Delta t} \leftarrow \bm{z}_t + \Delta t \cdot \bm{v}_\theta(\bm{z}_t, t, \bm{c})$
|
||||
\EndFor
|
||||
\State $[f_0, e, d] \leftarrow \bm{z}_1$ \Comment{Extract prosody}
|
||||
\State $\bm{y} \leftarrow \text{NCV}(\bm{H}_{\text{text}}, f_0, e, d, \bm{e}_{\text{spk}})$ \Comment{Waveform synthesis}
|
||||
\Require Text $s$, reference audio $\bm{w}_{\text{ref}}$, consent record $r$
|
||||
\State \textbf{assert} consent $r$ is valid and $\bm{w}_{\text{ref}}$ not on the protected-voice blocklist
|
||||
\State $\bm{y} \leftarrow \text{Qwen3TTS}(s, \bm{w}_{\text{ref}})$ \Comment{Upstream synthesis (unmodified)}
|
||||
\If{watermarking enabled}
|
||||
\State $\bm{y} \leftarrow \text{AudioSeal.embed}(\bm{y})$ \Comment{Existing localized watermark}
|
||||
\EndIf
|
||||
\State \Return $\bm{y}$
|
||||
\end{algorithmic}
|
||||
\end{algorithm}
|
||||
|
||||
\subsection{Neural Codec Vocoder (NCV)}
|
||||
\label{sec:ncv}
|
||||
|
||||
The NCV converts linguistic features, prosodic parameters, and speaker embeddings into high-fidelity audio waveforms.
|
||||
|
||||
\paragraph{Codec Token Generation.} We use a residual vector quantization (RVQ) scheme with 8 codebooks, each containing 1024 codes:
|
||||
\begin{equation}
|
||||
\bm{q}_l = \text{Quantize}_l\left(\bm{r}_{l-1}\right), \quad \bm{r}_l = \bm{r}_{l-1} - \text{Dequantize}_l(\bm{q}_l)
|
||||
\end{equation}
|
||||
where $\bm{r}_0$ is the input acoustic representation and $l \in \{1, \ldots, 8\}$ indexes the codebook level.
|
||||
|
||||
\paragraph{Token Prediction.} A 6-layer transformer predicts codec tokens from conditioned features:
|
||||
\begin{equation}
|
||||
p(\bm{q}_l | \bm{q}_{<l}, \bm{H}_{\text{text}}, f_0, e, d, \bm{e}_{\text{spk}}) = \text{Transformer}_{\text{codec}}(\bm{q}_{<l}, \bm{c}_{\text{acoustic}})
|
||||
\end{equation}
|
||||
We use a grouped prediction scheme: codebooks 1--2 are predicted autoregressively, while codebooks 3--8 are predicted in parallel.
|
||||
|
||||
\paragraph{Waveform Decoder.} Codec tokens are decoded to 24kHz waveforms using a HiFi-GAN-style generator \citep{kong2020hifi} with multi-period and multi-scale discriminators:
|
||||
\begin{equation}
|
||||
\bm{y} = \text{HiFiGAN}(\text{Dequantize}(\bm{q}_1, \ldots, \bm{q}_8)) \in \mathbb{R}^{L}
|
||||
\end{equation}
|
||||
|
||||
\paragraph{Streaming Architecture.} For real-time applications, the NCV operates in streaming mode with a lookahead of 80ms (2 codec frames). Causal convolutions replace non-causal ones, and the transformer uses sliding window attention of 32 frames. This achieves end-to-end latency of 148ms on a single NVIDIA A10G GPU.
|
||||
All acoustic modeling in step~2 is performed by the upstream model. Zen-Voice adds only
|
||||
the consent check (step~1) and the optional provenance watermark (step~4).
|
||||
|
||||
% =============================================================================
|
||||
\section{Anti-Deepfake Watermarking}
|
||||
\section{Provenance Watermarking}
|
||||
\label{sec:watermark}
|
||||
|
||||
Given the potential for misuse of voice cloning technology, we integrate a mandatory watermarking system into the synthesis pipeline.
|
||||
Because synthetic-speech misuse (fraud, impersonation, disinformation) is a real risk,
|
||||
Zen-Voice offers an opt-in provenance watermark on generated audio. We do not train a
|
||||
watermarker; we integrate \textbf{AudioSeal}~\citep{san2024proactive}, an existing
|
||||
localized neural watermarking method designed for proactive detection of AI-generated
|
||||
speech. AudioSeal embeds an imperceptible, detectable marker and is reported by its
|
||||
authors to remain detectable under common audio transformations.
|
||||
|
||||
\subsection{Watermark Design}
|
||||
\paragraph{Carried payload.} The integration optionally binds a small payload---a
|
||||
model/version identifier and a reference to the signed consent record---so that
|
||||
provenance can be tied back to a governance ledger. The cryptographic and storage
|
||||
design of that ledger is a deployment concern, not a property of the speech model.
|
||||
|
||||
The watermarking system embeds a 128-bit payload into the synthesized audio:
|
||||
\begin{itemize}
|
||||
\item \textbf{Bits 1--32:} Model identifier and version hash
|
||||
\item \textbf{Bits 33--64:} Timestamp (Unix epoch, second precision)
|
||||
\item \textbf{Bits 65--96:} User/API key fingerprint
|
||||
\item \textbf{Bits 97--128:} HMAC-SHA256 truncated signature over bits 1--96
|
||||
\end{itemize}
|
||||
|
||||
\paragraph{Embedding.} The watermark is embedded using a learned encoder $W_{\text{enc}}$ that modifies the codec tokens before waveform decoding:
|
||||
\begin{equation}
|
||||
\tilde{\bm{q}} = \bm{q} + W_{\text{enc}}(\bm{q}, \bm{m})
|
||||
\end{equation}
|
||||
where $\bm{m} \in \{0, 1\}^{128}$ is the watermark payload. The training loss is:
|
||||
\begin{equation}
|
||||
\mathcal{L}_{\text{wm}} = \lambda_{\text{det}} \cdot \mathcal{L}_{\text{BCE}}(W_{\text{dec}}(\tilde{\bm{y}}), \bm{m}) + \lambda_{\text{qual}} \cdot \|\tilde{\bm{y}} - \bm{y}\|_1 + \lambda_{\text{percept}} \cdot \mathcal{L}_{\text{STFT}}(\tilde{\bm{y}}, \bm{y})
|
||||
\end{equation}
|
||||
where $\mathcal{L}_{\text{STFT}}$ is a multi-resolution STFT loss ensuring imperceptibility.
|
||||
|
||||
\paragraph{Robustness Training.} During training, we apply a differentiable augmentation pipeline between embedding and detection: MP3 compression (64--320 kbps), Opus and AAC codec simulation, Gaussian and environmental noise (SNR 5--40 dB), resampling (8kHz--48kHz), time stretching ($\pm$20\%), pitch shifting ($\pm$4 semitones), dynamic range compression, and room impulse response convolution.
|
||||
|
||||
\subsection{Detection Performance}
|
||||
|
||||
\begin{table}[t]
|
||||
\centering
|
||||
\caption{Watermark detection accuracy under various audio transformations.}
|
||||
\label{tab:watermark}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Transformation} & \textbf{Bit Acc. (\%)} & \textbf{Payload Recovery (\%)} \\
|
||||
\midrule
|
||||
None (clean) & 99.9 & 99.8 \\
|
||||
MP3 128 kbps & 99.4 & 99.1 \\
|
||||
MP3 64 kbps & 98.1 & 96.8 \\
|
||||
Opus 32 kbps & 97.3 & 95.2 \\
|
||||
Gaussian noise (SNR 20 dB) & 98.8 & 97.4 \\
|
||||
Gaussian noise (SNR 10 dB) & 95.2 & 89.6 \\
|
||||
Resample 8kHz $\rightarrow$ 24kHz & 97.6 & 95.8 \\
|
||||
Time stretch $\pm$10\% & 98.2 & 96.4 \\
|
||||
Pitch shift $\pm$2 semitones & 97.8 & 95.9 \\
|
||||
RIR convolution (medium room) & 98.5 & 97.1 \\
|
||||
Combined (MP3 + noise + RIR) & 94.8 & 87.3 \\
|
||||
\midrule
|
||||
\textbf{AI-generated detection (binary)} & \multicolumn{2}{c}{\textbf{99.7\% accuracy}} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Perceptual Impact}
|
||||
|
||||
The watermark introduces minimal perceptual degradation: A/B testing with 200 listeners showed no statistically significant preference between watermarked and non-watermarked audio ($p = 0.42$, two-tailed binomial test). The signal-to-watermark ratio (SWR) averages 38.2 dB, well above the perceptual threshold.
|
||||
\paragraph{Honest scope.} We report AudioSeal's robustness as published by its authors
|
||||
and do not present a watermark-robustness table of our own in this document. The sibling
|
||||
Zen Live-Dub report contains directly-measured AudioSeal survival numbers on dubbed
|
||||
speech for readers who want measured figures; we avoid duplicating unverified numbers
|
||||
here.
|
||||
|
||||
% =============================================================================
|
||||
\section{Training}
|
||||
\section{Packaging and Tooling}
|
||||
\label{sec:training}
|
||||
|
||||
\subsection{Data}
|
||||
Zen-Voice's engineering work is integration, not training:
|
||||
\begin{enumerate}
|
||||
\item \textbf{Distribution}: Redistributing the upstream Apache-2.0 Qwen3-TTS
|
||||
weights with local-inference build artifacts.
|
||||
\item \textbf{API}: A thin synthesis wrapper (Algorithm~\ref{alg:inference}).
|
||||
\item \textbf{Governance}: Consent-record verification and a protected-voice
|
||||
blocklist enforced at synthesis time, plus the opt-in AudioSeal watermark.
|
||||
\item \textbf{Verification}: Smoke tests confirming packaged artifacts match the
|
||||
upstream reference implementation on sample inputs.
|
||||
\end{enumerate}
|
||||
|
||||
\begin{table}[t]
|
||||
\centering
|
||||
\caption{Training data composition for Zen-Voice.}
|
||||
\label{tab:data}
|
||||
\begin{tabular}{llrc}
|
||||
\toprule
|
||||
\textbf{Component} & \textbf{Dataset} & \textbf{Hours} & \textbf{Languages} \\
|
||||
\midrule
|
||||
\multirow{4}{*}{HSE Pre-training} & VoxCeleb 1\&2 & 7,400 & en \\
|
||||
& Common Voice 16.0 & 28,000 & 12 \\
|
||||
& MLS & 50,000 & 8 \\
|
||||
& Internal (licensed) & 594,600 & 12 \\
|
||||
\midrule
|
||||
\multirow{3}{*}{TTS Training} & LibriTTS-R & 585 & en \\
|
||||
& VCTK & 44 & en \\
|
||||
& Internal studio recordings & 12,000 & 12 \\
|
||||
\midrule
|
||||
Prosody & Expressive audiobooks & 8,400 & en, zh, ja \\
|
||||
\midrule
|
||||
Watermark & Synthetic + real mix & 50,000 & 12 \\
|
||||
\midrule
|
||||
\multicolumn{2}{l}{\textbf{Total (deduplicated)}} & \textbf{680,000} & 12 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Training Pipeline}
|
||||
|
||||
Training follows a four-stage curriculum:
|
||||
|
||||
\paragraph{Stage 1: Speaker Encoder Pre-training (2 weeks, 32 A100 GPUs).} The HSE is trained with the multi-scale contrastive objective on the full 680K-hour dataset. We use AdamW with learning rate $3 \times 10^{-4}$, batch size 4096, and cosine schedule with 5000 warm-up steps.
|
||||
|
||||
\paragraph{Stage 2: TTS Model Training (3 weeks, 64 A100 GPUs).} The text encoder, prosody transfer module, and codec predictor are jointly trained on the TTS subset (12,629 hours). The HSE is frozen during this stage. We use AdamW with learning rate $1 \times 10^{-4}$, batch size 256 utterances, and a two-phase schedule: 100K steps on clean studio data, then 200K steps on the full mix with data augmentation.
|
||||
|
||||
\paragraph{Stage 3: Vocoder Training (1 week, 16 A100 GPUs).} The HiFi-GAN vocoder is trained with multi-period and multi-scale discriminator losses:
|
||||
\begin{equation}
|
||||
\mathcal{L}_{\text{vocoder}} = \mathcal{L}_{\text{adv}} + \lambda_{\text{fm}} \mathcal{L}_{\text{feature}} + \lambda_{\text{mel}} \mathcal{L}_{\text{mel}}
|
||||
\end{equation}
|
||||
with $\lambda_{\text{fm}} = 2.0$ and $\lambda_{\text{mel}} = 45.0$.
|
||||
|
||||
\paragraph{Stage 4: Watermark Integration (3 days, 8 A100 GPUs).} The watermark encoder and decoder are trained end-to-end with the frozen vocoder, optimizing for detection accuracy under augmentation while minimizing perceptual impact.
|
||||
|
||||
\subsection{Emotion and Expressiveness Training}
|
||||
|
||||
For emotional speech synthesis, we curate a subset of 2,400 hours of speech with emotion annotations across seven categories. Annotations are obtained through human labeling (800 hours) and a pre-trained speech emotion recognition model validated against human judgments (Cohen's $\kappa = 0.78$). The PTM is fine-tuned with an emotion classification auxiliary loss:
|
||||
\begin{equation}
|
||||
\mathcal{L}_{\text{emotion}} = \mathcal{L}_{\text{flow}} + \mu \cdot \text{CE}(\text{EmotionClf}(\bm{z}_1), y_{\text{emotion}})
|
||||
\end{equation}
|
||||
where $\mu = 0.1$ and $y_{\text{emotion}}$ is the ground-truth emotion label.
|
||||
|
||||
% =============================================================================
|
||||
\section{Evaluation}
|
||||
\label{sec:evaluation}
|
||||
|
||||
We evaluate Zen-Voice on three dimensions: synthesis quality (naturalness), speaker similarity (cloning fidelity), and prosodic expressiveness.
|
||||
|
||||
\subsection{Benchmarks and Metrics}
|
||||
|
||||
\paragraph{Datasets.}
|
||||
\begin{itemize}
|
||||
\item \textbf{LibriTTS test-clean:} 500 utterances from 39 speakers, standard TTS evaluation set.
|
||||
\item \textbf{VCTK:} 109 speakers with diverse accents, 400 utterances per speaker.
|
||||
\item \textbf{ZenVoice-Eval:} Our multilingual benchmark with 1200 utterances across 12 languages, 100 speakers, balanced by gender and age.
|
||||
\end{itemize}
|
||||
|
||||
\paragraph{Metrics.}
|
||||
\begin{itemize}
|
||||
\item \textbf{MOS (Mean Opinion Score):} 5-point Likert scale rated by 200 native speakers.
|
||||
\item \textbf{UTMOS:} Automated MOS prediction using the UTokyo-SaruLab model \citep{saeki2022utmos}.
|
||||
\item \textbf{Speaker Verification EER:} Equal Error Rate of a pre-trained ECAPA-TDNN \citep{desplanques2020ecapa} speaker verification model.
|
||||
\item \textbf{Word Error Rate (WER):} Intelligibility measured via Whisper-large-v3 transcription.
|
||||
\item \textbf{F0 RMSE:} Root mean square error of pitch contour relative to reference.
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Baselines}
|
||||
|
||||
\begin{itemize}
|
||||
\item \textbf{VALL-E} \citep{wang2023neural}: Autoregressive neural codec language model.
|
||||
\item \textbf{VALL-E 2} \citep{chen2024vall}: Improved VALL-E with grouped code modeling.
|
||||
\item \textbf{VoiceBox} \citep{le2024voicebox}: Flow-matching TTS with infilling.
|
||||
\item \textbf{NaturalSpeech 3} \citep{ju2024naturalspeech}: Factorized diffusion with discrete tokens.
|
||||
\item \textbf{XTTS v2} \citep{casanova2024xtts}: Open-source multilingual TTS.
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Results}
|
||||
|
||||
\begin{table}[t]
|
||||
\centering
|
||||
\caption{Speech quality on LibriTTS test-clean (3-second reference).}
|
||||
\label{tab:quality}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
\textbf{Method} & \textbf{Nat. MOS} & \textbf{Sim. MOS} & \textbf{UTMOS} & \textbf{WER (\%)} \\
|
||||
\midrule
|
||||
Ground Truth & 4.52 & -- & 4.31 & 2.1 \\
|
||||
\midrule
|
||||
VALL-E & 3.84 & 3.62 & 3.71 & 5.8 \\
|
||||
VALL-E 2 & 4.02 & 3.81 & 3.89 & 4.2 \\
|
||||
VoiceBox & 4.12 & 3.94 & 4.01 & 3.6 \\
|
||||
NaturalSpeech 3 & 4.18 & 4.02 & 4.08 & 3.3 \\
|
||||
XTTS v2 & 3.91 & 3.72 & 3.82 & 4.8 \\
|
||||
\midrule
|
||||
Zen-Voice & \textbf{4.38} & \textbf{4.21} & \textbf{4.22} & \textbf{2.7} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\begin{table}[t]
|
||||
\centering
|
||||
\caption{Speaker verification EER on VCTK (lower is better).}
|
||||
\label{tab:speaker_ver}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Method} & \textbf{3-sec ref} & \textbf{5-sec ref} & \textbf{10-sec ref} \\
|
||||
\midrule
|
||||
Ground Truth & \multicolumn{3}{c}{1.8\%} \\
|
||||
\midrule
|
||||
VALL-E & 8.4\% & 6.2\% & 4.8\% \\
|
||||
VALL-E 2 & 6.1\% & 4.8\% & 3.6\% \\
|
||||
VoiceBox & 5.3\% & 4.1\% & 3.2\% \\
|
||||
NaturalSpeech 3 & 4.7\% & 3.6\% & 2.8\% \\
|
||||
\midrule
|
||||
Zen-Voice & \textbf{3.4\%} & \textbf{2.6\%} & \textbf{2.1\%} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\begin{table}[t]
|
||||
\centering
|
||||
\caption{Multilingual evaluation on ZenVoice-Eval (10-second reference).}
|
||||
\label{tab:multilingual}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
\textbf{Language} & \textbf{Nat. MOS} & \textbf{Sim. MOS} & \textbf{WER (\%)} & \textbf{F0 RMSE} \\
|
||||
\midrule
|
||||
English & 4.41 & 4.52 & 2.4 & 18.3 \\
|
||||
Mandarin & 4.32 & 4.38 & 3.1 & 22.1 \\
|
||||
Japanese & 4.28 & 4.31 & 3.8 & 19.7 \\
|
||||
Korean & 4.18 & 4.22 & 4.2 & 21.4 \\
|
||||
Spanish & 4.35 & 4.41 & 2.8 & 17.8 \\
|
||||
French & 4.31 & 4.36 & 3.2 & 18.9 \\
|
||||
German & 4.27 & 4.33 & 3.5 & 20.2 \\
|
||||
Portuguese & 4.22 & 4.28 & 3.9 & 19.3 \\
|
||||
Italian & 4.29 & 4.34 & 3.3 & 18.6 \\
|
||||
Hindi & 4.08 & 4.12 & 5.1 & 24.3 \\
|
||||
Arabic & 4.04 & 4.08 & 5.6 & 25.1 \\
|
||||
Russian & 4.15 & 4.19 & 4.4 & 22.8 \\
|
||||
\midrule
|
||||
\textbf{Average} & \textbf{4.24} & \textbf{4.30} & \textbf{3.8} & \textbf{20.7} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
Table~\ref{tab:quality} shows Zen-Voice achieves a naturalness MOS of 4.38 on LibriTTS, closing the gap to ground-truth (4.52) more than any prior method. Speaker similarity MOS of 4.21 with just 3 seconds of reference significantly outperforms all baselines.
|
||||
|
||||
Table~\ref{tab:speaker_ver} demonstrates speaker verification EER of 2.1\% with 10-second references, approaching the ground-truth EER of 1.8\%.
|
||||
|
||||
Table~\ref{tab:multilingual} shows consistent quality across 12 languages, with highest performance on English and slightly lower on Hindi and Arabic where training data is more limited.
|
||||
|
||||
\subsection{Emotion Transfer Evaluation}
|
||||
|
||||
\begin{table}[t]
|
||||
\centering
|
||||
\caption{Emotion recognition accuracy of synthesized emotional speech.}
|
||||
\label{tab:emotion}
|
||||
\begin{tabular}{lccccccc}
|
||||
\toprule
|
||||
\textbf{Method} & \textbf{Neu} & \textbf{Hap} & \textbf{Sad} & \textbf{Ang} & \textbf{Fear} & \textbf{Sur} & \textbf{Avg} \\
|
||||
\midrule
|
||||
Reference audio & 92.1 & 87.3 & 84.6 & 89.2 & 78.4 & 82.1 & 85.6 \\
|
||||
\midrule
|
||||
VALL-E & 78.3 & 52.1 & 48.7 & 54.2 & 41.3 & 45.8 & 53.4 \\
|
||||
NaturalSpeech 3 & 84.2 & 68.4 & 62.1 & 71.3 & 55.2 & 58.7 & 66.7 \\
|
||||
Zen-Voice & \textbf{89.4} & \textbf{78.2} & \textbf{74.8} & \textbf{81.3} & \textbf{68.4} & \textbf{72.1} & \textbf{77.4} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
Table~\ref{tab:emotion} evaluates emotional expressiveness using a pre-trained speech emotion recognition model. Zen-Voice achieves 77.4\% average emotion recognition accuracy, significantly outperforming baselines and approaching the 85.6\% accuracy on real emotional speech.
|
||||
|
||||
\subsection{Latency Analysis}
|
||||
|
||||
\begin{table}[t]
|
||||
\centering
|
||||
\caption{Inference latency for 10-second utterances on NVIDIA A10G GPU.}
|
||||
\label{tab:latency}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Method} & \textbf{First Token (ms)} & \textbf{RTF} & \textbf{Streaming} \\
|
||||
\midrule
|
||||
VALL-E & 1,240 & 0.82 & No \\
|
||||
VALL-E 2 & 680 & 0.51 & No \\
|
||||
VoiceBox & 320 & 0.24 & No \\
|
||||
NaturalSpeech 3 & 410 & 0.31 & No \\
|
||||
\midrule
|
||||
Zen-Voice (batch) & 280 & 0.18 & No \\
|
||||
Zen-Voice (stream) & \textbf{148} & \textbf{0.21} & \textbf{Yes} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
Zen-Voice achieves a real-time factor (RTF) of 0.18 in batch mode and 0.21 in streaming mode, both well under 1.0. The streaming mode achieves first-audio latency of 148ms.
|
||||
|
||||
% =============================================================================
|
||||
\section{Ablation Studies}
|
||||
\label{sec:ablation}
|
||||
|
||||
\subsection{Speaker Encoder Design}
|
||||
|
||||
\begin{table}[t]
|
||||
\centering
|
||||
\caption{Speaker encoder architecture ablation on VCTK (3-second reference).}
|
||||
\label{tab:spk_ablation}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Architecture} & \textbf{Sim. MOS} & \textbf{EER (\%)} & \textbf{Params} \\
|
||||
\midrule
|
||||
ECAPA-TDNN (frozen) & 3.72 & 6.8 & 6.2M \\
|
||||
x-vector & 3.68 & 7.2 & 4.8M \\
|
||||
Frame-level only & 3.91 & 5.1 & 12M \\
|
||||
Segment-level only & 3.98 & 4.4 & 18M \\
|
||||
HSE (no disentangle) & 4.12 & 3.8 & 32M \\
|
||||
HSE (full) & \textbf{4.21} & \textbf{3.4} & 32M \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
The hierarchical design contributes 0.30 MOS improvement over frame-level only, and disentanglement adds 0.09 MOS.
|
||||
|
||||
\subsection{Prosody Module Design}
|
||||
|
||||
\begin{table}[t]
|
||||
\centering
|
||||
\caption{Prosody generation method ablation on LibriTTS.}
|
||||
\label{tab:prosody_ablation}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Method} & \textbf{Nat. MOS} & \textbf{F0 RMSE (Hz)} & \textbf{Steps} \\
|
||||
\midrule
|
||||
Duration predictor only & 4.02 & 32.1 & 1 \\
|
||||
VAE prosody & 4.14 & 26.8 & 1 \\
|
||||
Diffusion (DDPM, 50 steps) & 4.28 & 21.4 & 50 \\
|
||||
Flow matching (10 steps) & 4.35 & 19.2 & 10 \\
|
||||
Flow matching (25 steps) & \textbf{4.38} & \textbf{18.3} & 25 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
Flow matching achieves the best quality at 25 ODE steps, with 10 steps providing a favorable quality-speed trade-off.
|
||||
|
||||
\subsection{Reference Length Sensitivity}
|
||||
|
||||
\begin{table}[t]
|
||||
\centering
|
||||
\caption{Speaker similarity vs. reference audio length on VCTK.}
|
||||
\label{tab:ref_length}
|
||||
\begin{tabular}{lcccccc}
|
||||
\toprule
|
||||
\textbf{Ref Length} & \textbf{1s} & \textbf{3s} & \textbf{5s} & \textbf{10s} & \textbf{30s} & \textbf{60s} \\
|
||||
\midrule
|
||||
Sim. MOS & 3.82 & 4.21 & 4.38 & 4.52 & 4.61 & 4.63 \\
|
||||
EER (\%) & 7.2 & 3.4 & 2.6 & 2.1 & 1.9 & 1.9 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
Performance improves significantly from 1 to 10 seconds with diminishing returns beyond 30 seconds.
|
||||
We do not report training compute, dataset composition, or a training curriculum,
|
||||
because Zen-Voice performs no training of the synthesis model. Dataset and training
|
||||
details for the underlying model are documented by the upstream
|
||||
report~\citep{qwen3tts}.
|
||||
|
||||
% =============================================================================
|
||||
\section{Discussion}
|
||||
\label{sec:discussion}
|
||||
|
||||
\subsection{Strengths}
|
||||
\subsection{Where the value is}
|
||||
|
||||
Zen-Voice's modular architecture provides three distinct advantages: (1) the separation of speaker identity and prosody enables independent control; (2) the flow-matching formulation provides deterministic, high-quality prosody generation in few ODE steps; (3) the streaming architecture enables real-time applications without sacrificing quality.
|
||||
Zen-Voice's value is operational, not architectural: it makes an openly-licensed,
|
||||
capable TTS model (Qwen3-TTS) easy to self-host with a license-clean dependency set,
|
||||
and it adds consent and provenance controls that bare model weights do not include.
|
||||
The synthesis quality, multilingual coverage, and short-reference cloning are upstream
|
||||
properties, attributed accordingly.
|
||||
|
||||
\subsection{Limitations}
|
||||
|
||||
\begin{itemize}
|
||||
\item \textbf{Singing voice:} Zen-Voice produces suboptimal results for singing, where pitch accuracy and vibrato control are critical.
|
||||
\item \textbf{Extremely short references:} Below 3 seconds, speaker similarity degrades noticeably.
|
||||
\item \textbf{Cross-lingual cloning:} Accent artifacts can occur when cloning into a language not present in the reference.
|
||||
\item \textbf{Watermark removal:} Targeted adversarial attacks could potentially remove the watermark.
|
||||
\item \textbf{No new modeling}: Zen-Voice cannot exceed upstream Qwen3-TTS quality;
|
||||
it inherits the upstream model's strengths and weaknesses (including any
|
||||
per-language quality differences and difficulty with singing voice).
|
||||
\item \textbf{Watermark is a deterrent, not a guarantee}: AudioSeal marks are
|
||||
robust to common transformations but, like any watermark, may be degraded or
|
||||
removed by targeted adversarial processing.
|
||||
\item \textbf{Governance is opt-in at the deployment layer}: Consent verification
|
||||
and the protected-voice blocklist are enforced by the deployer; they are not
|
||||
intrinsic, unremovable properties of the model weights.
|
||||
\item \textbf{No benchmarks of our own}: We have not run controlled MOS/EER studies;
|
||||
claims here are upstream-attributed or qualitative.
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Ethical Considerations}
|
||||
|
||||
Voice cloning poses significant ethical risks. We mitigate these through:
|
||||
Voice cloning poses real risks (fraud, impersonation, disinformation). Recommended
|
||||
mitigations at deployment:
|
||||
\begin{enumerate}
|
||||
\item \textbf{Mandatory watermarking:} All API output carries provenance markers that cannot be disabled.
|
||||
\item \textbf{Consent verification:} API users must attest consent from the voice owner.
|
||||
\item \textbf{Voice blocklist:} Protected voices are rejected by speaker verification at inference time.
|
||||
\item \textbf{Detection tools:} The watermark detector is released as a free, open-source tool.
|
||||
\item \textbf{Provenance watermarking}: Enable AudioSeal marking on generated audio.
|
||||
\item \textbf{Consent verification}: Require an attested, revocable consent record
|
||||
for any cloned voice, checked at synthesis time.
|
||||
\item \textbf{Voice blocklist}: Reject protected/known voices.
|
||||
\item \textbf{Disclosure}: Where law requires (e.g.\ synthetic-audio disclosure
|
||||
regimes), label generated audio as AI-generated.
|
||||
\end{enumerate}
|
||||
|
||||
% =============================================================================
|
||||
\section{Conclusion}
|
||||
\label{sec:conclusion}
|
||||
|
||||
We presented Zen-Voice, a neural speech synthesis system achieving state-of-the-art zero-shot voice cloning from as little as 3 seconds of reference audio. The hierarchical speaker encoder, flow-matching prosody transfer module, and streaming neural codec vocoder combine to produce natural, expressive speech that faithfully reproduces target speaker characteristics. Integrated anti-deepfake watermarking ensures responsible deployment.
|
||||
\textbf{Zen-Voice} is a packaging and responsible-deployment layer over Alibaba's
|
||||
openly-licensed \textbf{Qwen3-TTS} model, with an optional AudioSeal provenance
|
||||
watermark and consent tooling. It introduces no new synthesis model and reports no
|
||||
benchmarks of its own; the zero-shot cloning capability is upstream Qwen3-TTS's and is
|
||||
attributed as such. Earlier ``from-scratch'' architecture and head-to-head benchmark
|
||||
claims have been removed as inaccurate.
|
||||
|
||||
Zen-Voice achieves a naturalness MOS of 4.38 on LibriTTS test-clean, a speaker similarity MOS of 4.21 with 3-second references, and a speaker verification EER of 2.1\% with 10-second references. The streaming architecture achieves sub-150ms latency for real-time conversational applications.
|
||||
|
||||
Models and inference code are available at \url{https://github.com/hanzoai/zen-voice} under Apache 2.0. The watermark detection tool is released at \url{https://github.com/hanzoai/zen-voice-detect}.
|
||||
The packaging and inference wrapper are available at
|
||||
\url{https://github.com/hanzoai/zen-voice} under Apache~2.0 (matching the upstream
|
||||
license). The provenance watermark uses AudioSeal~\citep{san2024proactive}.
|
||||
|
||||
% =============================================================================
|
||||
% REFERENCES
|
||||
% =============================================================================
|
||||
\begin{thebibliography}{28}
|
||||
|
||||
\bibitem[Qwen Team(2026)]{qwen3tts}
|
||||
Qwen Team, Alibaba Cloud.
|
||||
\newblock Qwen3-TTS: open-source streaming text-to-speech with voice design and zero-shot cloning.
|
||||
\newblock Models and code: \url{https://github.com/QwenLM/Qwen3-TTS}, 2026. Apache 2.0.
|
||||
|
||||
\bibitem[Du et~al.(2024)]{cosyvoice2}
|
||||
Du, Z. et~al. (FunAudioLLM, Alibaba).
|
||||
\newblock CosyVoice 2: Scalable streaming speech synthesis with large language models.
|
||||
\newblock \emph{arXiv preprint arXiv:2412.10117}, 2024. Code/weights Apache 2.0.
|
||||
|
||||
\bibitem[Casanova et~al.(2024)]{casanova2024xtts}
|
||||
Casanova, E., Weber, J., Shulby, C.~D., Junior, A.~C., G{\"o}lge, E., and Ponti, M.~A.
|
||||
\newblock XTTS: A massively multilingual zero-shot text-to-speech model.
|
||||
@@ -802,66 +499,26 @@ Zhang, Y., Pan, S., He, L., and Ling, Z.-H.
|
||||
% =============================================================================
|
||||
\appendix
|
||||
|
||||
\section{Model Hyperparameters}
|
||||
\section{Model Architecture and Hyperparameters}
|
||||
\label{app:hyperparams}
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\caption{Zen-Voice module hyperparameters.}
|
||||
\label{tab:hyperparams}
|
||||
\begin{tabular}{llc}
|
||||
\toprule
|
||||
\textbf{Module} & \textbf{Parameter} & \textbf{Value} \\
|
||||
\midrule
|
||||
\multirow{4}{*}{Text Encoder} & Layers & 6 \\
|
||||
& Hidden dim & 512 \\
|
||||
& Attention heads & 8 \\
|
||||
& FFN dim & 2048 \\
|
||||
\midrule
|
||||
\multirow{5}{*}{HSE} & Frame encoder channels & [64, 128, 256] \\
|
||||
& Segment transformer layers & 4 \\
|
||||
& Speaker embedding dim & 512 \\
|
||||
& Total parameters & 32M \\
|
||||
& Contrastive temperature & 0.07 \\
|
||||
\midrule
|
||||
\multirow{4}{*}{Prosody Transfer} & DiT layers & 12 \\
|
||||
& Hidden dim & 512 \\
|
||||
& ODE steps (inference) & 25 \\
|
||||
& Classifier-free guidance & 2.0 \\
|
||||
\midrule
|
||||
\multirow{4}{*}{Neural Codec} & RVQ codebooks & 8 \\
|
||||
& Codebook size & 1024 \\
|
||||
& Codec frame rate & 50 Hz \\
|
||||
& Streaming lookahead & 80 ms \\
|
||||
\midrule
|
||||
\multirow{3}{*}{Watermark} & Payload bits & 128 \\
|
||||
& Encoder layers & 4 \\
|
||||
& SWR target & $\geq$35 dB \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
Zen-Voice does not define a model architecture of its own. The architecture,
|
||||
parameter counts, tokenizer, codec, and training hyperparameters are those of the
|
||||
upstream \textbf{Qwen3-TTS} model and are documented in the upstream
|
||||
report~\citep{qwen3tts}. Where CosyVoice~2 is used as the backup engine, refer to its
|
||||
report~\citep{cosyvoice2}. We intentionally do not reproduce module-level
|
||||
hyperparameter tables here, because doing so previously implied a bespoke architecture
|
||||
that does not exist.
|
||||
|
||||
\section{Computational Requirements}
|
||||
\label{app:compute}
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\caption{Inference computational requirements by deployment configuration.}
|
||||
\label{tab:compute_req}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Configuration} & \textbf{GPU} & \textbf{VRAM} & \textbf{RTF} \\
|
||||
\midrule
|
||||
Full model (FP16) & A100 80GB & 14.2 GB & 0.12 \\
|
||||
Full model (FP16) & A10G 24GB & 14.2 GB & 0.18 \\
|
||||
INT8 quantized & T4 16GB & 8.1 GB & 0.34 \\
|
||||
INT4 quantized & L4 24GB & 5.2 GB & 0.42 \\
|
||||
Streaming (FP16) & A10G 24GB & 14.8 GB & 0.21 \\
|
||||
CPU (INT4) & 32-core Xeon & 6.8 GB RAM & 2.8 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
The INT8 quantized model on T4 achieves real-time synthesis (RTF $<$ 1.0) at approximately \$0.0004 per second of generated audio.
|
||||
Memory footprint and throughput follow from the chosen upstream checkpoint and runtime.
|
||||
Qwen3-TTS is reported by Alibaba to run on consumer GPUs (on the order of a few GB of
|
||||
VRAM for smaller variants); exact VRAM and real-time factor depend on the variant,
|
||||
precision, and hardware and should be measured on the target deployment rather than
|
||||
quoted from this document. We do not publish per-configuration RTF or per-second cost
|
||||
figures here, as we have not run a controlled benchmark of our own; any such numbers
|
||||
would require their own documented methodology.
|
||||
|
||||
\end{document}
|
||||
|
||||
Binary file not shown.
+145
-224
@@ -13,7 +13,8 @@
|
||||
\definecolor{zenblue}{RGB}{41,121,255}
|
||||
\hypersetup{colorlinks=true,linkcolor=zenblue,urlcolor=zenblue,citecolor=zenblue}
|
||||
|
||||
\title{\textbf{Zen3-Embedding: DSO-Native Semantic Embedding Model}\\
|
||||
\title{\textbf{Zen3-Embedding: A Packaging of Qwen3-Embedding\\
|
||||
for Zen Retrieval}\\
|
||||
\large Technical Whitepaper v2025.06}
|
||||
\author{Zach Kelling \\ Zen LM Research Team\\
|
||||
\texttt{research@zenlm.org}\\
|
||||
@@ -24,16 +25,24 @@
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
Zen3-Embedding produces 7680-dimensional semantic representations optimized for DSO
|
||||
(Decentralized Semantic Optimization) retrieval, achieving strong multilingual embedding
|
||||
quality across the MTEB benchmark suite while maintaining BitDelta-compatible compression
|
||||
for efficient deployment. At 7B parameters, Zen3-Embedding achieves MTEB average 74.3\%,
|
||||
BEIR nDCG@10 57.8\%, MIRACL 81.2\%, and retrieval nDCG@10 0.812, surpassing prior
|
||||
embedding models with substantially smaller parameter counts. The model supports
|
||||
100+ languages through a shared multilingual encoder with language-balanced contrastive
|
||||
training, and its 7680-dimensional output space aligns natively with the DSO vector
|
||||
index format (ZIP-001), enabling direct integration with decentralized semantic search
|
||||
infrastructure without dimension projection overhead.
|
||||
Zen3-Embedding is a repackaging of Alibaba's \textbf{Qwen3-Embedding} series, used as the
|
||||
first-stage retrieval encoder in Zen retrieval pipelines. It is \emph{not} a from-scratch
|
||||
model: it is the openly released, Apache-2.0 licensed \texttt{Qwen/Qwen3-Embedding} family
|
||||
(0.6B, 4B, and 8B), produced by the Qwen team and built on the Qwen3 dense foundation
|
||||
models~\cite{qwen3embedding}. Qwen3-Embedding is a \emph{causal} (decoder-only) LLM adapted
|
||||
for embeddings: an \texttt{[EOS]} token is appended to the input and the final-layer hidden
|
||||
state at that position is taken as the embedding. The models accept task instructions on the
|
||||
query side, support Matryoshka-style user-defined output dimensions (from 32 up to the model's
|
||||
native dimension), span 100+ languages including code, and handle a 32K context. Native
|
||||
embedding dimensions are 1024 (0.6B), 2560 (4B), and 4096 (8B). On the Qwen team's evaluation,
|
||||
the flagship Qwen3-Embedding-8B reports a Mean(Task) of \textbf{70.58} on the MTEB
|
||||
Multilingual (MMTEB) leaderboard --- ranked No.~1 as of 5~June~2025 --- with MTEB English~v2
|
||||
Mean(Task) 75.22, C-MTEB Mean(Task) 73.83, and MTEB-Code 80.68~\cite{qwen3embedding}. This
|
||||
document describes how the upstream model is wired into Zen retrieval; all weights, training,
|
||||
and reported numbers are the Qwen team's and are cited as such. Claims in earlier revisions of
|
||||
a bespoke ``7680-dimensional'' bidirectional ``Zen MoDE'' encoder trained for ``Decentralized
|
||||
Semantic Optimization,'' with BitDelta index compression, were fabricated and have been
|
||||
removed.
|
||||
\end{abstract}
|
||||
|
||||
\tableofcontents
|
||||
@@ -57,88 +66,75 @@ applications. Yet most deployed embedding models face a trilemma:
|
||||
underperform language-specific models on high-resource languages.
|
||||
\end{itemize}
|
||||
|
||||
Zen3-Embedding addresses all three through:
|
||||
Rather than train an embedding model from scratch, Zen3-Embedding adopts the Qwen3-Embedding
|
||||
series~\cite{qwen3embedding}, which targets these trade-offs directly. Our contribution is
|
||||
integration and packaging, not modeling. The upstream model provides:
|
||||
|
||||
\begin{enumerate}
|
||||
\item \textbf{DSO-aligned 7680-dimensional output space} --- The embedding dimension
|
||||
is chosen to match the DSO vector index native format (ZIP-001), eliminating
|
||||
projection overhead and enabling direct P2P semantic search on the decentralized
|
||||
network.
|
||||
\item \textbf{Instruction-aware embeddings} --- Following recent instruction-tuned
|
||||
embedding work, Zen3-Embedding accepts task instructions that shift the
|
||||
embedding geometry toward task-optimal representations without additional
|
||||
fine-tuning.
|
||||
\item \textbf{BitDelta-compressed index compatibility} --- The 7680-dimensional output
|
||||
is compatible with BitDelta (ZIP-007) 1.58-bit scalar quantization, reducing
|
||||
index storage from 30.7 KB/vector to 1.5 KB/vector with less than 2\% NDCG degradation.
|
||||
\item \textbf{Matryoshka representation learning (MRL)} --- Zen3-Embedding supports
|
||||
dimension truncation at 256, 512, 1024, 2048, 3840, and 7680, with each prefix
|
||||
independently optimized for retrieval quality.
|
||||
\item \textbf{A spectrum of sizes} --- 0.6B, 4B, and 8B, letting a deployment trade quality
|
||||
against latency and memory.
|
||||
\item \textbf{Instruction-aware embeddings} --- The query may be prefixed with a task
|
||||
instruction that steers the representation toward the task, without fine-tuning.
|
||||
\item \textbf{Matryoshka output dimensions} --- A single model supports user-defined output
|
||||
dimensions from 32 up to its native dimension, so index storage can be traded against
|
||||
retrieval quality without a separate model.
|
||||
\item \textbf{Multilingual and code coverage} --- 100+ languages, including programming
|
||||
languages, with a 32K context.
|
||||
\end{enumerate}
|
||||
|
||||
We do not modify the Qwen3-Embedding weights; Zen3-Embedding is the upstream model wired into
|
||||
the Zen retrieval stack.
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{Architecture}
|
||||
\label{sec:arch}
|
||||
|
||||
\subsection{Bidirectional Zen MoDE Encoder}
|
||||
\subsection{Causal Decoder Adapted for Embeddings}
|
||||
|
||||
Unlike the autoregressive Zen family language models, Zen3-Embedding uses a
|
||||
bidirectional (BERT-style) encoder architecture built on the Zen MoDE foundation.
|
||||
Full bidirectional attention allows each token to contextualize from the entire
|
||||
sequence, producing richer representations for embedding tasks than causal attention.
|
||||
Qwen3-Embedding is \emph{not} a bidirectional BERT-style encoder and has no
|
||||
mixture-of-experts ``Zen MoDE'' backbone. Per the Qwen team~\cite{qwen3embedding}, each model
|
||||
is a fine-tune of the corresponding Qwen3-\emph{Base} \emph{dense, causal} (decoder-only) LLM
|
||||
(e.g. the 8B from \texttt{Qwen3-8B-Base}, the 0.6B from \texttt{Qwen3-0.6B-Base}). The
|
||||
``7.1B-parameter, 48-layer, 64-expert top-4 bidirectional'' specification in earlier revisions
|
||||
did not correspond to any deployed weights and has been removed.
|
||||
|
||||
Architecture specification:
|
||||
Released sizes and native embedding dimensions~\cite{qwen3embedding}:
|
||||
\begin{itemize}
|
||||
\item 7.1B parameters total (4.8B active per forward pass with MoDE sparsity)
|
||||
\item 48 transformer layers, 4096-dimensional hidden state
|
||||
\item 32 attention heads, 128-dimensional head size
|
||||
\item 64 experts, top-4 active per token
|
||||
\item Maximum input sequence length: 32,768 tokens (full attention)
|
||||
\item Longer inputs use hierarchical mean-pooling over 32K chunks
|
||||
\item Qwen3-Embedding-0.6B: 28 layers, native embedding dimension 1024.
|
||||
\item Qwen3-Embedding-4B: 36 layers, native embedding dimension 2560.
|
||||
\item Qwen3-Embedding-8B: 36 layers, native embedding dimension 4096.
|
||||
\item Context length: 32,768 tokens (all sizes).
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Embedding Head Design}
|
||||
\subsection{Embedding Extraction (Last-Token Pooling)}
|
||||
|
||||
The final embedding is produced by a two-component pooling strategy:
|
||||
The embedding is the final-layer hidden state at an appended \texttt{[EOS]}
|
||||
token~\cite{qwen3embedding}. Because attention is causal, that last position has attended over
|
||||
the entire input, so its hidden state serves as the sequence representation:
|
||||
|
||||
\begin{equation}
|
||||
\mathbf{e} = \text{LayerNorm}\left(\mathbf{W}_{\text{proj}} \cdot [\mathbf{h}_{\text{CLS}} \;\|\; \bar{\mathbf{h}}_{\text{mean}}]\right)
|
||||
\mathbf{e} = \mathbf{h}^{(L)}_{\texttt{[EOS]}} \in \mathbb{R}^{d_{\text{model}}},
|
||||
\end{equation}
|
||||
|
||||
where $\mathbf{h}_{\text{CLS}} \in \mathbb{R}^{4096}$ is the \texttt{[CLS]} token
|
||||
hidden state and $\bar{\mathbf{h}}_{\text{mean}} \in \mathbb{R}^{4096}$ is the
|
||||
mean-pooled sequence representation. The concatenated 8192-dimensional vector is
|
||||
projected to 7680 dimensions via $\mathbf{W}_{\text{proj}} \in \mathbb{R}^{7680 \times 8192}$.
|
||||
where $d_{\text{model}}$ is the model's native dimension (1024 / 2560 / 4096). There is no
|
||||
\texttt{[CLS]} token, no mean$+$CLS dual pooling, and no learned projection to a 7680-dim
|
||||
space; those constructions in earlier revisions were fabricated.
|
||||
|
||||
The dual-pooling design is motivated by the complementary nature of CLS (global semantic
|
||||
summary) and mean-pooling (compositional token-level semantics): combining both
|
||||
outperforms either alone by 1.2--2.4 points on MTEB retrieval tasks.
|
||||
\subsection{Matryoshka Output Dimensions}
|
||||
|
||||
\subsection{Matryoshka Representation Learning}
|
||||
Qwen3-Embedding supports user-defined output dimensions from 32 up to the native
|
||||
dimension~\cite{qwen3embedding,mrl}, so a shorter prefix of the embedding can be used to
|
||||
shrink index storage at a modest quality cost. The supported range is continuous (32 to
|
||||
$d_{\text{model}}$), not the fixed ladder $\{256,\dots,7680\}$ asserted in earlier revisions.
|
||||
|
||||
MRL~\cite{mrl} enables dimension-adaptive embeddings by training the model to
|
||||
produce high-quality representations at all prefixes simultaneously:
|
||||
\subsection{Use in the Zen Retrieval Stack}
|
||||
|
||||
\begin{equation}
|
||||
\mathcal{L}_{\text{MRL}} = \sum_{d \in \mathcal{D}} w_d \cdot \mathcal{L}_{\text{contrastive}}\left(\mathbf{e}_{:d}, \mathbf{e}^+_{:d}, \{\mathbf{e}^-_i\}_{:d}\right)
|
||||
\end{equation}
|
||||
|
||||
where $\mathcal{D} = \{256, 512, 1024, 2048, 3840, 7680\}$ are the supported
|
||||
truncation dimensions and $w_d$ is a dimension weight (larger dimensions receive
|
||||
higher weight since they have more capacity and are the primary deployment target).
|
||||
|
||||
\subsection{DSO Vector Index Alignment}
|
||||
|
||||
The DSO protocol (ZIP-001) specifies a 7680-dimensional float32 native embedding
|
||||
format for peer-to-peer semantic search nodes. Zen3-Embedding outputs are directly
|
||||
compatible without projection, enabling:
|
||||
|
||||
\begin{itemize}
|
||||
\item Plug-in replacement for DSO network nodes
|
||||
\item Direct HNSW index construction from model output
|
||||
\item BitDelta scalar quantization (ZIP-007) to 1.58-bit per dimension
|
||||
\item Cross-node cosine similarity computation without format conversion
|
||||
\end{itemize}
|
||||
Zen indexing computes Qwen3-Embedding vectors for documents (no instruction) and builds a
|
||||
standard vector index (e.g. HNSW) over them; at query time the query embedding (optionally
|
||||
with a task instruction) is used for similarity search, and the Qwen3-Reranker stage
|
||||
re-scores the shortlist. The ``DSO-native 7680-dim format,'' ``BitDelta 1.58-bit index
|
||||
compression,'' and ``peer-to-peer node'' descriptions of earlier revisions did not describe
|
||||
this model and have been removed.
|
||||
|
||||
\subsection{Instruction-Aware Embedding}
|
||||
|
||||
@@ -164,159 +160,89 @@ Instructions are applied to queries only (not documents), preserving document in
|
||||
\section{Training}
|
||||
\label{sec:training}
|
||||
|
||||
\subsection{Pre-Training}
|
||||
\subsection{Provenance, Not Training}
|
||||
|
||||
Zen3-Embedding is initialized from the Zen3-Small (7B) base language model, then
|
||||
adapted to bidirectional attention by:
|
||||
\begin{enumerate}
|
||||
\item Removing the causal attention mask
|
||||
\item Adding \texttt{[CLS]} and \texttt{[SEP]} special tokens to the vocabulary
|
||||
\item Continued pre-training on masked language modeling (MLM) for 100B tokens
|
||||
\item Learning rate: $1 \times 10^{-5}$, warm-up 5\%, cosine decay
|
||||
\end{enumerate}
|
||||
Zen3-Embedding performs no training of its own. The models were trained by the Qwen team; we
|
||||
summarize their published approach at a high level and attribute it
|
||||
accordingly~\cite{qwen3embedding}.
|
||||
|
||||
\subsection{Contrastive Training Data}
|
||||
|
||||
The contrastive training corpus contains 2.1B positive (query, document) pairs:
|
||||
|
||||
\begin{itemize}
|
||||
\item Web search query-document pairs (mined from search logs): 800M
|
||||
\item Academic paper abstract-body pairs: 120M
|
||||
\item Question-answer pairs (StackExchange, community QA): 180M
|
||||
\item Multilingual parallel corpora (translation pairs as positives): 400M
|
||||
\item Code documentation pairs (function signature + docstring): 90M
|
||||
\item Synthetic pairs generated by Zen3-Medium (for hard positives): 250M
|
||||
\item Curated high-quality pairs (human-annotated): 260M
|
||||
\end{itemize}
|
||||
|
||||
Hard negatives are mined using BM25 top-100 retrieval followed by reranking with
|
||||
a lightweight cross-encoder to select negatives that are lexically similar but
|
||||
semantically distinct (``hard BM25 negatives'').
|
||||
|
||||
\subsection{Contrastive Learning Objective}
|
||||
|
||||
Training uses InfoNCE loss with in-batch negatives plus 8 mined hard negatives per query:
|
||||
Per the Qwen team, Qwen3-Embedding starts from the Qwen3-Base dense LLMs and is adapted to
|
||||
embeddings with a contrastive objective (the standard last-token / \texttt{[EOS]}
|
||||
representation with an InfoNCE-style loss over positive (query, document) pairs and
|
||||
negatives), with the Qwen3 generative models used to synthesize a large, multilingual portion
|
||||
of the training pairs~\cite{qwen3embedding}. The contrastive loss has the usual form
|
||||
|
||||
\begin{equation}
|
||||
\mathcal{L}_{\text{InfoNCE}} = -\log \frac{\exp(\text{sim}(\mathbf{q}, \mathbf{d}^+) / \tau)}{\exp(\text{sim}(\mathbf{q}, \mathbf{d}^+) / \tau) + \sum_{j=1}^{N} \exp(\text{sim}(\mathbf{q}, \mathbf{d}^-_j) / \tau)}
|
||||
\mathcal{L}_{\text{InfoNCE}} = -\log \frac{\exp(\text{sim}(\mathbf{q}, \mathbf{d}^+) / \tau)}{\exp(\text{sim}(\mathbf{q}, \mathbf{d}^+) / \tau) + \sum_{j} \exp(\text{sim}(\mathbf{q}, \mathbf{d}^-_j) / \tau)},
|
||||
\end{equation}
|
||||
|
||||
where $\text{sim}(\cdot, \cdot)$ is cosine similarity, $\tau = 0.02$ is the
|
||||
temperature, $N = 8 + B - 1$ is the total number of negatives ($B$ is batch size,
|
||||
8 mined hard negatives per sample).
|
||||
with $\text{sim}(\cdot,\cdot)$ cosine similarity. For the authoritative dataset composition,
|
||||
synthetic-data pipeline, and hyperparameters, see the Qwen team's report~\cite{qwen3embedding}.
|
||||
|
||||
GradCache~\cite{gradcache} is used to enable batch sizes of 32,768 without requiring
|
||||
proportional GPU memory, by accumulating gradients from sub-batches.
|
||||
|
||||
\subsection{Multilingual Training}
|
||||
|
||||
Language-balanced sampling ensures no language dominates training:
|
||||
|
||||
\begin{itemize}
|
||||
\item High-resource languages (EN, ZH, ES, FR, DE, JA, KO, RU, AR, PT): 35\% total
|
||||
\item Mid-resource languages (50 languages): 45\% total
|
||||
\item Low-resource languages (50+ languages): 20\% total
|
||||
\end{itemize}
|
||||
|
||||
Cross-lingual positive pairs (same meaning, different languages) are sampled at 25\%
|
||||
of training data, training the model to produce language-agnostic representations.
|
||||
The fabricated training details in earlier revisions --- ``bidirectional MLM adaptation,'' a
|
||||
2.1B-pair corpus with specific per-source counts, a synthetic generator named
|
||||
``Zen3-Medium,'' and a 32{,}768 GradCache batch --- did not describe any real training run
|
||||
and have been removed.
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{Evaluation}
|
||||
\label{sec:eval}
|
||||
|
||||
\subsection{MTEB Overview}
|
||||
All numbers in this section are the Qwen team's published results for
|
||||
Qwen3-Embedding~\cite{qwen3embedding}, reproduced here with attribution. We ran no evaluations
|
||||
of our own and make no independent benchmark claims.
|
||||
|
||||
\subsection{Flagship (8B) Headline Numbers}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{MTEB benchmark performance by task category}
|
||||
\begin{tabular}{lccc}
|
||||
\caption{Qwen3-Embedding-8B, as reported by the Qwen team~\cite{qwen3embedding}. The MMTEB
|
||||
Mean(Task) of 70.58 ranked No.~1 on the MTEB Multilingual leaderboard as of 5~June~2025.}
|
||||
\begin{tabular}{lc}
|
||||
\toprule
|
||||
\textbf{Task Category} & \textbf{Zen3-Embedding} & \textbf{Zen2-Embedding} & \textbf{Tasks} \\
|
||||
\textbf{Benchmark} & \textbf{Score} \\
|
||||
\midrule
|
||||
Retrieval & 57.8 & 53.1 & 15 \\
|
||||
Semantic Similarity & 87.4 & 84.2 & 10 \\
|
||||
Reranking & 59.3 & 55.8 & 4 \\
|
||||
Classification & 78.6 & 74.9 & 12 \\
|
||||
Clustering & 51.2 & 47.4 & 11 \\
|
||||
Pair Classification & 86.9 & 83.1 & 3 \\
|
||||
Summarization & 30.4 & 28.7 & 1 \\
|
||||
\textbf{MTEB Average} & \textbf{74.3} & \textbf{70.1} & 56 \\
|
||||
MTEB Multilingual (MMTEB), Mean(Task) & 70.58 \\
|
||||
MTEB Multilingual (MMTEB), Mean(Type) & 61.69 \\
|
||||
MTEB English v2, Mean(Task) & 75.22 \\
|
||||
MTEB English v2, Mean(Type) & 68.70 \\
|
||||
C-MTEB (Chinese), Mean(Task) & 73.83 \\
|
||||
MTEB-Code (nDCG@10) & 80.68 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:mteb}
|
||||
\label{tab:qwen3emb-8b}
|
||||
\end{table}
|
||||
|
||||
\subsection{BEIR Retrieval}
|
||||
\subsection{Across Model Sizes}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{BEIR retrieval benchmark (nDCG@10)}
|
||||
\caption{Released sizes and native embedding dimensions~\cite{qwen3embedding}. The 8B is the
|
||||
flagship; the Qwen team's report gives the full MMTEB breakdown for each size, with quality
|
||||
increasing with size.}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Dataset} & \textbf{Zen3-Embedding} & \textbf{Zen2-Embedding} \\
|
||||
\textbf{Model} & \textbf{Layers} & \textbf{Native dim} \\
|
||||
\midrule
|
||||
MSMARCO & 41.2 & 38.4 \\
|
||||
TREC-COVID & 77.4 & 72.1 \\
|
||||
NFCorpus & 36.8 & 33.9 \\
|
||||
NQ & 52.3 & 48.7 \\
|
||||
HotpotQA & 68.9 & 64.2 \\
|
||||
FEVER & 78.1 & 73.4 \\
|
||||
SCIDOCS & 19.4 & 17.8 \\
|
||||
SciFact & 73.2 & 69.4 \\
|
||||
Quora & 88.4 & 85.1 \\
|
||||
DBPedia & 41.7 & 38.2 \\
|
||||
\textbf{BEIR Average} & \textbf{57.8} & \textbf{54.1} \\
|
||||
Qwen3-Embedding-0.6B & 28 & 1024 \\
|
||||
Qwen3-Embedding-4B & 36 & 2560 \\
|
||||
Qwen3-Embedding-8B & 36 & 4096 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:beir}
|
||||
\label{tab:qwen3emb-sizes}
|
||||
\end{table}
|
||||
|
||||
\subsection{Multilingual Retrieval}
|
||||
The 8B MMTEB Mean(Task) of 70.58 is given above; per-size MMTEB/MTEB scores for the 0.6B and
|
||||
4B models are reported in the Qwen team's paper and we refer the reader there rather than
|
||||
restating numbers we have not independently verified~\cite{qwen3embedding}.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{MIRACL multilingual retrieval (nDCG@10)}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Language} & \textbf{Zen3-Embedding} & \textbf{Zen2-Embedding} \\
|
||||
\midrule
|
||||
English (EN) & 87.3 & 83.4 \\
|
||||
Chinese (ZH) & 82.1 & 77.8 \\
|
||||
Arabic (AR) & 78.9 & 73.1 \\
|
||||
Spanish (ES) & 84.2 & 80.6 \\
|
||||
Japanese (JA) & 79.4 & 74.2 \\
|
||||
Korean (KO) & 81.3 & 76.9 \\
|
||||
Russian (RU) & 79.8 & 75.1 \\
|
||||
French (FR) & 83.7 & 79.4 \\
|
||||
German (DE) & 82.4 & 78.2 \\
|
||||
Swahili (SW) & 72.4 & 64.3 \\
|
||||
\textbf{Average (18 langs)} & \textbf{81.2} & \textbf{76.4} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:miracl}
|
||||
\end{table}
|
||||
\subsection{A Note on Earlier Revisions}
|
||||
|
||||
\subsection{Matryoshka Dimension Truncation}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{MTEB average score vs. embedding dimension (MRL truncation)}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Dimension} & \textbf{MTEB Average} & \textbf{Storage per Vector (KB)} \\
|
||||
\midrule
|
||||
256 & 68.2 & 1.0 \\
|
||||
512 & 70.4 & 2.0 \\
|
||||
1024 & 71.9 & 4.0 \\
|
||||
2048 & 73.1 & 8.0 \\
|
||||
3840 & 73.9 & 15.0 \\
|
||||
7680 & 74.3 & 30.0 \\
|
||||
\textbf{7680 + BitDelta} & \textbf{72.8} & \textbf{1.5} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:matryoshka}
|
||||
\end{table}
|
||||
Earlier revisions of this document reported a fabricated ``MTEB average 74.3,'' a ``Zen2-Embedding''
|
||||
comparison column, BEIR/MIRACL tables, and a Matryoshka/BitDelta storage table at 7680
|
||||
dimensions. None of those described this model or any real measurement; they were fabricated
|
||||
and have been removed. For per-task and per-dataset breakdowns of the genuine numbers, see the
|
||||
Qwen team's report and the HuggingFace model cards~\cite{qwen3embedding}.
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{Related Work}
|
||||
@@ -325,10 +251,9 @@ Swahili (SW) & 72.4 & 64.3 \\
|
||||
\subsection{Dense Retrieval}
|
||||
|
||||
Dense passage retrieval~\cite{karpukhin2020dense} established bi-encoder architectures
|
||||
as practical alternatives to BM25 for open-domain question answering. Subsequent
|
||||
scaling work demonstrated that larger encoders consistently improve retrieval quality.
|
||||
Zen3-Embedding scales to 7B parameters while maintaining production-practical latency
|
||||
through the MoDE sparse activation pattern (4.8B active parameters per forward pass).
|
||||
as practical alternatives to BM25 for open-domain question answering. Qwen3-Embedding sits in
|
||||
the more recent line of repurposing decoder-only LLMs as embedding models via last-token
|
||||
pooling and contrastive fine-tuning~\cite{qwen3embedding}.
|
||||
|
||||
\subsection{Contrastive Learning for Embeddings}
|
||||
|
||||
@@ -336,32 +261,36 @@ SimCSE~\cite{gao2021simcse} demonstrated that contrastive learning with dropout
|
||||
data augmentation substantially improves semantic textual similarity. E5~\cite{wang2022e5}
|
||||
and subsequent work showed that web-scale weakly supervised contrastive training
|
||||
with curated hard negatives matches or exceeds expensive human-annotated pairs.
|
||||
Zen3-Embedding synthesizes these insights at 7B scale.
|
||||
Qwen3-Embedding follows this contrastive-fine-tuning paradigm on top of the Qwen3 base
|
||||
LLMs~\cite{qwen3embedding}.
|
||||
|
||||
\subsection{Matryoshka Representations}
|
||||
|
||||
Matryoshka Representation Learning~\cite{mrl} enables flexible dimension truncation
|
||||
with minimal quality loss, supporting diverse downstream deployment constraints.
|
||||
Zen3-Embedding is the first 7B-scale model to implement full MRL across 6 dimensions.
|
||||
with minimal quality loss; Qwen3-Embedding exposes this as user-selectable output dimensions
|
||||
from 32 up to each model's native dimension~\cite{qwen3embedding}.
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{Conclusion}
|
||||
\label{sec:conclusion}
|
||||
|
||||
Zen3-Embedding delivers frontier retrieval quality at 7B scale with native DSO
|
||||
protocol compatibility, achieving MTEB 74.3\%, BEIR 57.8\%, and MIRACL 81.2\%.
|
||||
The 7680-dimensional embedding space, Matryoshka truncation support, BitDelta
|
||||
index compression, and instruction-aware embedding collectively address the quality,
|
||||
efficiency, and deployment flexibility requirements of production semantic search systems.
|
||||
Zen3-Embedding is the first-stage retrieval encoder of Zen retrieval pipelines, implemented as
|
||||
a direct deployment of Alibaba's Apache-2.0 Qwen3-Embedding models
|
||||
(0.6B/4B/8B)~\cite{qwen3embedding}. Qwen3-Embedding is a causal decoder LLM adapted for
|
||||
embeddings via last-token (\texttt{[EOS]}) pooling, with instruction-aware queries, Matryoshka
|
||||
output dimensions, 100+ language coverage, and a 32K context. The flagship 8B reports an MMTEB
|
||||
Mean(Task) of 70.58 (No.~1 as of 5~June~2025) per the Qwen team. We contribute integration into
|
||||
the Zen retrieval stack, not a new model, and we report the Qwen team's published numbers with
|
||||
attribution rather than benchmarks of our own. The ``DSO-native 7680-dimensional Zen MoDE
|
||||
encoder'' and ``BitDelta index compression'' claims of earlier revisions have been removed as
|
||||
fabrications.
|
||||
|
||||
Integration with the DSO network (ZIP-001) enables decentralized semantic search
|
||||
at scale without centralized index infrastructure, a capability unique to the
|
||||
Zen family embedding ecosystem.
|
||||
\section*{Acknowledgments and Attribution}
|
||||
|
||||
\section*{Acknowledgments}
|
||||
|
||||
The authors thank Zoo Labs Foundation for DSO network integration support and
|
||||
MTEB benchmark evaluation infrastructure.
|
||||
Zen3-Embedding is built entirely on Qwen3-Embedding (\texttt{Qwen/Qwen3-Embedding},
|
||||
Apache-2.0), developed by the Qwen team at Alibaba; all model weights, training, and reported
|
||||
metrics are theirs. We thank the Qwen team for releasing the models openly and the MTEB/MMTEB
|
||||
maintainers for the evaluation suites.
|
||||
|
||||
\bibliographystyle{plain}
|
||||
\begin{thebibliography}{9}
|
||||
@@ -386,20 +315,12 @@ A. Kusupati et al.,
|
||||
``Matryoshka Representation Learning,''
|
||||
\textit{NeurIPS}, 2022.
|
||||
|
||||
\bibitem{gradcache}
|
||||
L. Gao, J. Callan,
|
||||
``Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup,''
|
||||
\textit{RepL4NLP Workshop}, 2021.
|
||||
|
||||
\bibitem{zip001}
|
||||
Zoo Labs Foundation,
|
||||
``ZIP-001: Decentralized Semantic Optimization (DSO),''
|
||||
\textit{Zoo Improvement Proposals}, 2024.
|
||||
|
||||
\bibitem{zip007}
|
||||
Zoo Labs Foundation,
|
||||
``ZIP-007: BitDelta --- Quantization-Aware Delta Compression for Language Models,''
|
||||
\textit{Zoo Improvement Proposals}, 2024.
|
||||
\bibitem{qwen3embedding}
|
||||
Y. Zhang, M. Li, D. Long, X. Zhang, H. Lin, B. Yang, P. Xie, A. Yang, D. Liu, J. Lin,
|
||||
F. Huang, J. Zhou (Qwen Team, Alibaba),
|
||||
``Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models,''
|
||||
\textit{arXiv:2506.05176}, 2025.
|
||||
Models: \texttt{Qwen/Qwen3-Embedding-\{0.6B,4B,8B\}} (Apache-2.0).
|
||||
|
||||
\end{thebibliography}
|
||||
|
||||
|
||||
Binary file not shown.
+180
-272
@@ -13,7 +13,8 @@
|
||||
\definecolor{zenblue}{RGB}{41,121,255}
|
||||
\hypersetup{colorlinks=true,linkcolor=zenblue,urlcolor=zenblue,citecolor=zenblue}
|
||||
|
||||
\title{\textbf{Zen3-Guard: Third-Generation Multilingual Content Safety Model}\\
|
||||
\title{\textbf{Zen3-Guard: A Packaging of IBM Granite Guardian 8B\\
|
||||
for the Zen Model Family}\\
|
||||
\large Technical Whitepaper v2025.06}
|
||||
\author{Zach Kelling \\ Zen LM Research Team\\
|
||||
\texttt{research@zenlm.org}\\
|
||||
@@ -24,16 +25,22 @@
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
Zen3-Guard is a dedicated 8B parameter safety classification model for real-time content
|
||||
moderation, providing high-accuracy detection of harmful content across 100+ languages
|
||||
with inference latency suitable for production API deployments. Built on the Zen MoDE
|
||||
(Mixture of Distilled Experts) architecture with a safety-specialized expert cluster
|
||||
configuration, Zen3-Guard achieves ToxiGen 99.2\%, HatEval 97.8\%, OffComEval 96.4\%,
|
||||
and multilingual macro-F1 of 0.968 across all supported languages. At 8B parameters,
|
||||
the model delivers sub-5ms classification latency per request when batch-optimized on
|
||||
A10G GPU hardware, enabling real-time moderation of high-throughput API deployments.
|
||||
The model is designed as a drop-in safety layer for all Zen family models, configurable
|
||||
to enforce organization-specific content policies through a policy injection interface.
|
||||
Zen3-Guard is a repackaging of IBM's \textbf{Granite Guardian 8B} safety model, deployed
|
||||
as the content-safety layer for the Zen model family. It is \emph{not} a from-scratch
|
||||
model: it is the openly released, Apache-2.0 licensed \texttt{ibm-granite/granite-guardian}
|
||||
8B guardrail produced by IBM Research, which is itself a supervised fine-tune of the dense,
|
||||
decoder-only IBM Granite 3.x 8B Instruct base model (\texttt{GraniteForCausalLM}). Granite
|
||||
Guardian classifies a user prompt and/or assistant response against a configurable risk
|
||||
definition and emits a \texttt{Yes}/\texttt{No} verdict together with a probability of harm
|
||||
derived from the next-token logits. The risk taxonomy, aligned with the IBM AI Risk Atlas,
|
||||
spans harmful content (harm, social bias, profanity, sexual content, unethical behavior,
|
||||
violence, jailbreaking) as well as retrieval-augmented-generation faithfulness risks
|
||||
(context relevance, groundedness, answer relevance). IBM reports an aggregate AUC of
|
||||
\textbf{0.871} on harmful-content benchmarks and \textbf{0.854} on RAG-hallucination
|
||||
benchmarks for Granite Guardian 3.0 8B~\cite{graniteguardian}. This document describes how
|
||||
the upstream model is integrated as a drop-in safety filter for Zen deployments; all model
|
||||
weights, training, and reported metrics are IBM's and are cited as such. We add no
|
||||
benchmark claims of our own.
|
||||
\end{abstract}
|
||||
|
||||
\tableofcontents
|
||||
@@ -50,284 +57,178 @@ avoid false positives that degrade user experience), coverage across 100+ langua
|
||||
(to serve global user populations), and sub-10ms inference latency (to insert into
|
||||
API request paths without perceptible overhead).
|
||||
|
||||
Existing safety approaches are inadequate in at least one dimension:
|
||||
Existing safety approaches trade off along several dimensions:
|
||||
\begin{itemize}
|
||||
\item Keyword filters: high false positive rate, trivially bypassed by paraphrasing
|
||||
\item Binary classifiers: insufficient granularity for nuanced policy enforcement
|
||||
\item Large safety-tuned generation models: accurate but too slow for real-time use
|
||||
\item Narrow binary classifiers: insufficient granularity for nuanced policy enforcement
|
||||
\item Large safety-tuned generation models: accurate but heavier to serve
|
||||
\item Embedding-similarity approaches: brittle to adversarial rephrasing
|
||||
\end{itemize}
|
||||
|
||||
Zen3-Guard addresses all four constraints simultaneously through:
|
||||
Rather than train a safety classifier from scratch, Zen3-Guard adopts IBM's openly
|
||||
released Granite Guardian 8B~\cite{graniteguardian}, which directly targets these
|
||||
trade-offs as a purpose-built guardrail LLM. Our contribution is integration and
|
||||
packaging, not modeling. The upstream model provides:
|
||||
|
||||
\begin{enumerate}
|
||||
\item \textbf{Safety-specialized Zen MoDE architecture} --- A dedicated subset of
|
||||
experts trained exclusively on safety-relevant data, activated when routing
|
||||
detects safety-sensitive content patterns.
|
||||
\item \textbf{Hierarchical harm taxonomy} --- A 47-category harm taxonomy covering
|
||||
violence, hate speech, harassment, self-harm, illegal activity, privacy violations,
|
||||
and sensitive content, with configurable policy enforcement per category.
|
||||
\item \textbf{Multilingual safety training} --- Dedicated safety annotation data in
|
||||
100+ languages, with special attention to code-switching and cross-lingual
|
||||
adversarial evasion.
|
||||
\item \textbf{Policy injection interface} --- Organization-specific content policies
|
||||
are encoded as natural language policy documents and injected at inference time,
|
||||
allowing policy customization without model retraining.
|
||||
\item \textbf{A risk-conditioned guardrail LLM} --- A dense Granite 3.x 8B Instruct model
|
||||
fine-tuned by IBM to classify content against a named risk definition supplied at
|
||||
inference time, emitting a \texttt{Yes}/\texttt{No} verdict plus a harm probability.
|
||||
\item \textbf{An IBM AI Risk Atlas taxonomy} --- Risk dimensions covering harmful content
|
||||
and (in the RAG setting) faithfulness risks, rather than a single binary safe/unsafe
|
||||
label.
|
||||
\item \textbf{Prompt- and response-side checking} --- The same model can be applied to the
|
||||
user input, the model output, or a RAG response against its retrieved context.
|
||||
\item \textbf{Risk-definition prompting} --- Because the desired risk is described in the
|
||||
prompt template at inference time, the deployed model can be steered toward different
|
||||
checks without retraining.
|
||||
\end{enumerate}
|
||||
|
||||
We do not modify the Granite Guardian weights; Zen3-Guard is the upstream model wired into
|
||||
the Zen serving stack. The remainder of this document describes that integration and
|
||||
summarizes IBM's published design and results with attribution.
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{Architecture}
|
||||
\label{sec:arch}
|
||||
|
||||
\subsection{Safety-Specialized Zen MoDE Configuration}
|
||||
\subsection{Upstream Model: Granite Guardian 8B}
|
||||
|
||||
Zen3-Guard uses an 8B Zen MoDE backbone with 32 experts, where the expert population
|
||||
is partitioned into three functional groups:
|
||||
Zen3-Guard is the IBM Granite Guardian 8B model served unmodified. Per IBM's
|
||||
documentation~\cite{graniteguardian}, the model is a \emph{dense, decoder-only}
|
||||
transformer (\texttt{GraniteForCausalLM}; GQA, RoPE, SwiGLU MLP, RMSNorm). It is
|
||||
\emph{not} a mixture-of-experts model and has no notion of ``expert clusters'' or
|
||||
safety-specific routing. It is produced by supervised fine-tuning of the IBM Granite 3.x
|
||||
8B Instruct base model on risk-annotated data. We make no architectural changes; earlier
|
||||
revisions of this document incorrectly described a bespoke ``mixture-of-distilled-experts''
|
||||
backbone, which does not correspond to the deployed weights.
|
||||
|
||||
\textbf{General language experts (experts 1--12)}: Standard language understanding,
|
||||
providing the contextual representations needed for nuanced safety judgment. These
|
||||
experts capture pragmatics, irony, and implicit meaning that keyword systems miss.
|
||||
\subsection{Risk Taxonomy and Output Schema}
|
||||
|
||||
\textbf{Safety-specialized experts (experts 13--24)}: Trained predominantly on
|
||||
annotated safety datasets including ToxiGen, HatEval, OffComEval, and proprietary
|
||||
moderation datasets from production deployments. These experts develop representations
|
||||
tightly coupled to harm patterns.
|
||||
Granite Guardian does not emit a fixed multi-label vector. Instead, the caller supplies a
|
||||
\emph{risk definition} in the prompt template, and the model judges whether the supplied
|
||||
content exhibits that risk. The risk dimensions documented by IBM~\cite{graniteguardian},
|
||||
aligned with the IBM AI Risk Atlas, include:
|
||||
|
||||
\textbf{Policy enforcement experts (experts 25--32)}: Process policy injection context
|
||||
and map general harm representations to organization-specific policy decisions.
|
||||
|
||||
The router is trained with a safety-aware auxiliary loss that encourages activation
|
||||
of safety-specialized experts when input tokens match patterns associated with
|
||||
policy-sensitive categories:
|
||||
|
||||
\begin{equation}
|
||||
\mathcal{L}_{\text{safety-route}} = -\sum_{c \in \mathcal{C}_{\text{harmful}}} \mathbb{1}[\text{expert} \in \mathcal{E}_{\text{safety}}] \cdot \log p(\text{expert} | x)
|
||||
\end{equation}
|
||||
|
||||
\subsection{Harm Taxonomy and Output Schema}
|
||||
|
||||
Zen3-Guard outputs a structured classification across 47 harm categories organized
|
||||
in a 5-level hierarchy:
|
||||
|
||||
\begin{enumerate}
|
||||
\item Violence (physical threat, incitement, graphic content)
|
||||
\item Hate speech (identity-based, stereotyping, dehumanization)
|
||||
\item Harassment (personal attack, doxxing, coordinated abuse)
|
||||
\item Self-harm and mental health (suicide, eating disorders, self-injury)
|
||||
\item Illegal activity (CSAM, fraud, controlled substances, weapons)
|
||||
\item Privacy (PII exposure, surveillance, data theft)
|
||||
\item Misinformation (health, political, scientific)
|
||||
\item Sensitive content (adult content, religious sensitivity, political sensitivity)
|
||||
\end{enumerate}
|
||||
|
||||
Each category produces:
|
||||
\begin{itemize}
|
||||
\item Binary decision: \texttt{safe} / \texttt{unsafe}
|
||||
\item Confidence score: $p \in [0, 1]$
|
||||
\item Severity level: \texttt{low} / \texttt{medium} / \texttt{high} / \texttt{critical}
|
||||
\item Evidence span: character offsets of the triggering text segment
|
||||
\item \textbf{Harmful content}: harm (general), social bias, profanity, sexual content,
|
||||
unethical behavior, violence, and jailbreaking.
|
||||
\item \textbf{RAG faithfulness}: context relevance, groundedness, and answer relevance
|
||||
(i.e., detecting ungrounded or off-topic answers).
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Multilingual Architecture}
|
||||
The IBM AI Risk Atlas mapping organizes harm into a deeper hierarchy (four high-level
|
||||
categories and thirteen sub-categories), but the model's runtime interface is per-risk
|
||||
rather than a flat 47-way classifier.
|
||||
|
||||
Zen3-Guard uses a shared multilingual tokenizer with 150,528 vocabulary tokens covering
|
||||
100+ languages with balanced script coverage (Latin, Cyrillic, Arabic, CJK, Devanagari,
|
||||
and 40+ additional scripts). Language-specific routing biases are learned during
|
||||
training to account for language-specific harm expression patterns.
|
||||
For the selected risk, the model produces:
|
||||
\begin{itemize}
|
||||
\item Binary verdict: a \texttt{Yes} / \texttt{No} token (\texttt{Yes} = risk present).
|
||||
\item Probability of harm: derived from the log-probabilities of the \texttt{Yes}/\texttt{No}
|
||||
tokens at the final position.
|
||||
\end{itemize}
|
||||
|
||||
For low-resource languages with fewer than 1M safety-annotated tokens, a
|
||||
cross-lingual transfer mechanism leverages semantic similarity to high-resource
|
||||
language annotations:
|
||||
Severity tiers and character-offset evidence spans are not native outputs of the upstream
|
||||
model; where downstream Zen tooling needs them, they are derived heuristically outside the
|
||||
model and are not part of Granite Guardian's reported behavior.
|
||||
|
||||
\begin{equation}
|
||||
p(y | x_{\text{low}}) = \sum_{\ell \in \mathcal{L}_{\text{high}}} w_\ell \cdot p(y | \phi(x_{\text{low}}, x_\ell))
|
||||
\end{equation}
|
||||
\subsection{Language Coverage}
|
||||
|
||||
where $\phi$ is a cross-lingual alignment function and $w_\ell$ are similarity weights.
|
||||
Granite Guardian inherits the tokenizer and multilingual coverage of its Granite 3.x base.
|
||||
IBM's published evaluations focus primarily on English safety benchmarks; we therefore do
|
||||
not assert specific per-language safety scores. Claims in earlier revisions of a custom
|
||||
150{,}528-token multilingual tokenizer and a cross-lingual transfer objective were
|
||||
fabricated and have been removed.
|
||||
|
||||
\subsection{Policy Injection Interface}
|
||||
\subsection{Risk-Definition Prompting}
|
||||
|
||||
Organizations deploying Zen3-Guard can specify custom content policies as natural
|
||||
language documents. At inference time, the policy document is prepended to the
|
||||
classification context as a special \texttt{[POLICY]} token sequence:
|
||||
Because the risk to be checked is described in the chat template rather than baked into a
|
||||
classification head, a single deployed instance can perform different checks (e.g.
|
||||
\texttt{harm}, \texttt{social\_bias}, \texttt{groundedness}) by changing the
|
||||
\texttt{risk\_name} passed to the tokenizer's \texttt{apply\_chat\_template}. Schematically:
|
||||
|
||||
\begin{verbatim}
|
||||
[POLICY] Our platform prohibits: discussion of competitor products,
|
||||
political content of any kind, and medical advice. Legal content
|
||||
standards: US jurisdiction, adult content permitted with age verification.
|
||||
[/POLICY]
|
||||
[INPUT] <user content to classify> [/INPUT]
|
||||
apply_chat_template(
|
||||
messages=[{"role": "user", "content": <user content>},
|
||||
{"role": "assistant", "content": <response>}],
|
||||
guardian_config={"risk_name": "harm"})
|
||||
# -> model emits "Yes"/"No"; prob_of_risk read from token logits
|
||||
\end{verbatim}
|
||||
|
||||
The policy injection is processed by the policy enforcement expert cluster, which
|
||||
overrides default safety decisions based on organization-specific rules. This enables
|
||||
a single deployed model instance to serve multiple tenants with different content policies.
|
||||
This is the upstream interface; Zen deployments expose it through a thin wrapper. It is a
|
||||
prompt-time mechanism, not a learned ``policy enforcement expert cluster.''
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{Training}
|
||||
\label{sec:training}
|
||||
|
||||
\subsection{Safety Annotation Data}
|
||||
\subsection{Provenance, Not Training}
|
||||
|
||||
Zen3-Guard is trained on a 12B token safety-focused corpus:
|
||||
Zen3-Guard performs no training of its own. The guardrail was trained by IBM Research;
|
||||
we summarize their published methodology for completeness and attribute it
|
||||
accordingly~\cite{graniteguardian}.
|
||||
|
||||
\begin{itemize}
|
||||
\item ToxiGen (machine-generated, diverse group coverage): 274K examples
|
||||
\item HatEval 2019 (multilingual hate speech, EN/ES): 19K examples
|
||||
\item OffComEval (offensive language): 14K examples
|
||||
\item Jigsaw Unintended Bias (conversation toxicity): 1.8M examples
|
||||
\item WildGuard (instruction following safety): 92K examples
|
||||
\item OpenAI Policy Violations (annotated API violations): 240K examples
|
||||
\item Proprietary production moderation logs (de-identified): 8M examples
|
||||
\item Adversarial probing dataset (red-team generated): 2.1M examples
|
||||
\item Multilingual safety corpus (100+ languages): 4.2M examples
|
||||
\item Safe content (negative examples for precision): 12M examples
|
||||
\end{itemize}
|
||||
Per IBM, Granite Guardian is created by \emph{supervised fine-tuning} of the Granite 3.x
|
||||
8B Instruct base model on a combination of human-annotated and synthetic risk data
|
||||
spanning the harm and RAG-faithfulness dimensions of the IBM AI Risk Atlas. The objective
|
||||
is standard next-token supervision over the \texttt{Yes}/\texttt{No} verdict given a
|
||||
risk-conditioned prompt; the harm probability used downstream is read from the
|
||||
\texttt{Yes}/\texttt{No} token log-probabilities at inference time.
|
||||
|
||||
\subsection{Training Procedure}
|
||||
|
||||
\textbf{Phase 1 --- Base language model initialization}: Initialize from the Zen3-Small
|
||||
(8B) base checkpoint, which provides strong multilingual language understanding.
|
||||
|
||||
\textbf{Phase 2 --- Safety expert specialization}: Train only the safety-designated
|
||||
expert cluster on safety annotation data with a focal loss weighted toward false negatives:
|
||||
|
||||
\begin{equation}
|
||||
\mathcal{L}_{\text{focal}} = -\alpha_t (1 - p_t)^\gamma \log(p_t), \quad \gamma = 2, \quad \alpha_{\text{positive}} = 4
|
||||
\end{equation}
|
||||
|
||||
The positive class weight of 4 reflects the production priority of catching harmful
|
||||
content (false negatives) over minimizing false positives.
|
||||
|
||||
\textbf{Phase 3 --- End-to-end fine-tuning}: Joint training of all components on the
|
||||
full safety corpus with the taxonomy hierarchy loss:
|
||||
|
||||
\begin{equation}
|
||||
\mathcal{L}_{\text{taxonomy}} = \sum_{l=1}^{5} w_l \sum_{c \in \mathcal{C}_l} \mathcal{L}_{\text{BCE}}(p_c, y_c)
|
||||
\end{equation}
|
||||
|
||||
where $w_l$ increases with hierarchy depth to enforce consistency between parent
|
||||
and child category predictions.
|
||||
|
||||
\textbf{Phase 4 --- Adversarial robustness training}: Additional training on
|
||||
adversarially generated evasion attempts, including:
|
||||
\begin{itemize}
|
||||
\item Character substitution (l33t speak, Unicode confusables)
|
||||
\item Paraphrasing through synonym replacement
|
||||
\item Code-switching and macaronic text
|
||||
\item Prompt injection via role-playing framing
|
||||
\item Instruction hierarchy attacks (``ignore previous instructions'')
|
||||
\end{itemize}
|
||||
The fabricated training pipeline in earlier revisions of this document --- a 12B-token
|
||||
multilingual safety corpus, focal-loss ``safety expert'' specialization, a five-level
|
||||
taxonomy hierarchy loss, and a four-phase adversarial regimen --- did not describe any real
|
||||
training run and has been removed. For the authoritative dataset composition and training
|
||||
recipe, see IBM's paper~\cite{graniteguardian}.
|
||||
|
||||
\subsection{Calibration}
|
||||
|
||||
Zen3-Guard confidence scores are calibrated using temperature scaling on a held-out
|
||||
calibration set. Expected Calibration Error (ECE) after calibration: 0.018, indicating
|
||||
that a predicted confidence of 0.90 corresponds to approximately 90\% empirical accuracy.
|
||||
We do not recalibrate the model. IBM reports its evaluation metrics (Section~\ref{sec:eval})
|
||||
at a default decision threshold of $0.5$ on the harm probability; deployments may pick a
|
||||
different threshold per risk to trade recall against false positives.
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{Evaluation}
|
||||
\label{sec:eval}
|
||||
|
||||
\subsection{Primary Safety Benchmarks}
|
||||
All numbers in this section are IBM's published results for Granite Guardian
|
||||
3.0~8B~\cite{graniteguardian}, reproduced here with attribution. We ran no evaluations of
|
||||
our own and make no independent benchmark claims.
|
||||
|
||||
\subsection{Aggregate Results Reported by IBM}
|
||||
|
||||
IBM reports two headline metrics for Granite Guardian 3.0~8B, aggregated over public
|
||||
benchmark suites:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Primary safety classification benchmarks}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{ToxiGen} & \textbf{HatEval} & \textbf{OffComEval} & \textbf{WildGuard} \\
|
||||
\midrule
|
||||
Zen-Guard & 97.1 & 94.3 & 92.1 & 88.4 \\
|
||||
Zen2-Guard & 98.4 & 96.2 & 94.8 & 91.7 \\
|
||||
\textbf{Zen3-Guard} & \textbf{99.2} & \textbf{97.8} & \textbf{96.4} & \textbf{94.3} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:safety}
|
||||
\end{table}
|
||||
|
||||
\subsection{Multilingual Performance}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Multilingual safety classification (macro-F1 by language family)}
|
||||
\caption{Granite Guardian 3.0 8B aggregate metrics, as reported by IBM~\cite{graniteguardian}.}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Language Family} & \textbf{Zen3-Guard F1} & \textbf{Zen2-Guard F1} \\
|
||||
\textbf{Capability} & \textbf{Metric} & \textbf{Value} \\
|
||||
\midrule
|
||||
Germanic (EN, DE, NL, SV) & 0.981 & 0.972 \\
|
||||
Romance (ES, FR, IT, PT) & 0.974 & 0.963 \\
|
||||
Slavic (RU, PL, CS, UK) & 0.968 & 0.951 \\
|
||||
Semitic (AR, HE, AM) & 0.961 & 0.938 \\
|
||||
CJK (ZH, JA, KO) & 0.972 & 0.956 \\
|
||||
Indic (HI, BN, TA, TE) & 0.954 & 0.927 \\
|
||||
Low-resource (50+ others) & 0.943 & 0.901 \\
|
||||
\textbf{Overall average} & \textbf{0.968} & \textbf{0.944} \\
|
||||
Harmful-content detection & Aggregate AUC & 0.871 \\
|
||||
Harmful-content detection & Aggregate F1 (threshold 0.5) & 0.758 \\
|
||||
RAG hallucination / groundedness & Average AUC (TRUE suite) & 0.854 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:multilingual}
|
||||
\label{tab:granite-agg}
|
||||
\end{table}
|
||||
|
||||
\subsection{Inference Latency}
|
||||
The harmful-content figures are aggregated across public safety datasets used in IBM's
|
||||
evaluation (for example ToxicChat, HarmBench, SafeRLHF, BeaverTails, OpenAI Moderation,
|
||||
SimpleSafetyTests, and xstest), with comparisons against Llama~Guard and ShieldGemma
|
||||
baselines; the RAG-hallucination figure is the average AUC on the TRUE
|
||||
benchmark~\cite{graniteguardian}. We refer the reader to IBM's paper for per-dataset
|
||||
breakdowns, which we do not reproduce here.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Inference latency by hardware (batch size 32, sequence length 512)}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Hardware} & \textbf{P50 Latency (ms)} & \textbf{P99 Latency (ms)} & \textbf{Throughput (req/s)} \\
|
||||
\midrule
|
||||
NVIDIA A10G (24GB) & 3.8 & 5.1 & 8400 \\
|
||||
NVIDIA A100 (80GB) & 2.1 & 3.2 & 15200 \\
|
||||
NVIDIA L4 (24GB) & 4.2 & 5.8 & 7600 \\
|
||||
CPU (x86-64, INT8) & 24.0 & 31.0 & 1300 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:latency}
|
||||
\end{table}
|
||||
\subsection{A Note on the Numbers in Earlier Revisions}
|
||||
|
||||
\subsection{Adversarial Robustness}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Robustness to adversarial evasion (detection rate \%)}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Attack Type} & \textbf{Zen3-Guard} & \textbf{Zen2-Guard} \\
|
||||
\midrule
|
||||
Character substitution (l33t speak) & 98.7 & 96.2 \\
|
||||
Unicode confusable substitution & 97.9 & 93.4 \\
|
||||
Synonym paraphrase & 96.8 & 91.7 \\
|
||||
Code-switching & 95.4 & 88.3 \\
|
||||
Role-play framing & 94.1 & 85.9 \\
|
||||
Instruction injection & 93.2 & 83.1 \\
|
||||
Composite (all attacks combined) & 91.8 & 79.4 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:adversarial}
|
||||
\end{table}
|
||||
|
||||
\subsection{False Positive Rate}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{False positive rate on benign content categories}
|
||||
\begin{tabular}{lc}
|
||||
\toprule
|
||||
\textbf{Benign Category} & \textbf{FPR (\%)} \\
|
||||
\midrule
|
||||
General news articles & 0.31 \\
|
||||
Scientific papers & 0.18 \\
|
||||
Creative fiction & 1.24 \\
|
||||
Medical/clinical text & 0.89 \\
|
||||
Legal documents & 0.42 \\
|
||||
Historical records & 0.67 \\
|
||||
Code and documentation & 0.09 \\
|
||||
\textbf{Overall FPR} & \textbf{0.54} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:fpr}
|
||||
\end{table}
|
||||
Earlier revisions of this document reported figures such as ToxiGen 99.2\%, HatEval 97.8\%,
|
||||
a multilingual macro-F1 of 0.968, sub-5ms A10G latency, and per-attack adversarial
|
||||
robustness rates. None of these came from a measured evaluation of the deployed model; they
|
||||
were fabricated and have been removed. Serving latency depends entirely on the operator's
|
||||
hardware and stack and is not a property we can claim for the upstream weights.
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{Related Work}
|
||||
@@ -340,19 +241,21 @@ expression patterns. Machine learning approaches improved recall but suffered fr
|
||||
adversarial brittleness. Transformer-based classifiers~\cite{perspectiveapi} brought
|
||||
contextual understanding but required separate models per language.
|
||||
|
||||
\subsection{Safety-Tuned Language Models}
|
||||
\subsection{Guardrail Models}
|
||||
|
||||
Constitutional AI~\cite{bai2022constitutional} and related work demonstrated that
|
||||
safety behaviors can be instilled in generative models through targeted fine-tuning.
|
||||
Zen3-Guard extends this line by building a dedicated safety-only classifier rather
|
||||
than a safety-tuned generative model, achieving higher throughput through task specialization.
|
||||
Constitutional AI~\cite{bai2022constitutional} demonstrated that safety behaviors can be
|
||||
instilled in models through targeted fine-tuning. A subsequent line of work --- including
|
||||
Llama~Guard, ShieldGemma, and IBM's Granite Guardian~\cite{graniteguardian} --- packages a
|
||||
fine-tuned LLM specifically as an input/output guardrail that emits a risk verdict.
|
||||
Zen3-Guard does not introduce a new model in this line; it adopts Granite Guardian directly.
|
||||
|
||||
\subsection{Multilingual Safety}
|
||||
|
||||
Cross-lingual safety classification has been addressed through multilingual pre-training
|
||||
and cross-lingual transfer~\cite{multilingual-safety}. Zen3-Guard builds on this
|
||||
foundation with explicit multilingual safety annotation data and a cross-lingual
|
||||
alignment mechanism for low-resource languages.
|
||||
Cross-lingual safety classification has been studied through multilingual pre-training and
|
||||
transfer~\cite{multilingual-safety}. We make no multilingual-safety claims for Zen3-Guard
|
||||
beyond what IBM reports for Granite Guardian, whose published evaluations are primarily
|
||||
English; the cross-lingual ``alignment mechanism'' described in earlier revisions did not
|
||||
exist and has been removed.
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{Deployment Recommendations}
|
||||
@@ -363,45 +266,49 @@ alignment mechanism for low-resource languages.
|
||||
Zen3-Guard is designed for three primary deployment patterns:
|
||||
|
||||
\textbf{Synchronous API guard}: Insert as a pre-processing layer before the main
|
||||
LLM. Reject or flag requests before generation. Adds 4--5ms overhead on A10G.
|
||||
LLM, classifying the user prompt and rejecting or flagging it before generation.
|
||||
|
||||
\textbf{Parallel screening}: Run Zen3-Guard concurrently with the main LLM on
|
||||
the input, and suppress responses if Guard returns unsafe. Adds 0ms to perceived
|
||||
latency on GPU clusters with available capacity.
|
||||
\textbf{Parallel screening}: Run the guard concurrently with the main LLM on
|
||||
the input, and suppress responses if it returns a positive risk verdict. On GPU clusters
|
||||
with spare capacity this can overlap with generation.
|
||||
|
||||
\textbf{Output scanning}: Run Zen3-Guard on model outputs before returning to user.
|
||||
Recommended for generation tasks where harmful content may emerge from safe inputs.
|
||||
\textbf{Output scanning}: Run the guard on model outputs (and, in RAG, against the
|
||||
retrieved context for groundedness) before returning to the user. Recommended where
|
||||
harmful or ungrounded content may emerge from a benign-looking input.
|
||||
|
||||
Because the guard is itself an 8B decoder LLM, its serving cost is comparable to running an
|
||||
8B model; we make no specific latency guarantee, as it depends on the operator's hardware,
|
||||
batching, and quantization.
|
||||
|
||||
\subsection{Threshold Configuration}
|
||||
|
||||
Default thresholds are calibrated for a 0.54\% overall false positive rate.
|
||||
Organizations can adjust per-category thresholds based on their risk tolerance:
|
||||
|
||||
\begin{itemize}
|
||||
\item Strict mode ($p > 0.5$): 99.2\% recall, 1.8\% FPR
|
||||
\item Balanced mode ($p > 0.7$): 97.8\% recall, 0.54\% FPR (default)
|
||||
\item Precision mode ($p > 0.9$): 94.1\% recall, 0.12\% FPR
|
||||
\end{itemize}
|
||||
The harm probability is read from the \texttt{Yes}/\texttt{No} token log-probabilities.
|
||||
IBM reports its metrics at a default threshold of $p = 0.5$~\cite{graniteguardian}.
|
||||
Operators may raise the threshold per risk to favor precision (fewer false positives) or
|
||||
lower it to favor recall (catching more borderline content); the appropriate operating
|
||||
point is deployment-specific and should be chosen against the operator's own labeled data
|
||||
rather than the illustrative recall/FPR figures asserted in earlier revisions, which were
|
||||
not measured.
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{Conclusion}
|
||||
\label{sec:conclusion}
|
||||
|
||||
Zen3-Guard provides production-grade multilingual content safety classification at
|
||||
8B parameters, achieving ToxiGen 99.2\%, HatEval 97.8\%, multilingual macro-F1 0.968,
|
||||
and sub-5ms inference latency --- meeting all four production constraints simultaneously.
|
||||
The safety-specialized Zen MoDE expert configuration, 47-category harm taxonomy,
|
||||
adversarial robustness training, and policy injection interface make Zen3-Guard a
|
||||
flexible foundation for diverse content moderation use cases.
|
||||
Zen3-Guard is the content-safety layer of the Zen model family, implemented as a direct
|
||||
deployment of IBM's Apache-2.0 Granite Guardian 8B guardrail~\cite{graniteguardian}. It is
|
||||
a dense, decoder-only Granite 3.x 8B Instruct model fine-tuned by IBM Research to classify
|
||||
prompts and responses against a named risk and emit a \texttt{Yes}/\texttt{No} verdict with
|
||||
a harm probability. IBM reports an aggregate AUC of 0.871 on harmful-content benchmarks and
|
||||
0.854 on RAG-hallucination benchmarks for the 3.0 8B model. We contribute integration and
|
||||
packaging into the Zen serving stack, not a new model, and we make no benchmark claims of
|
||||
our own. Operators evaluating Zen3-Guard for production should validate it on their own
|
||||
content distribution and choose per-risk thresholds accordingly.
|
||||
|
||||
Future work will extend coverage to multimodal inputs (image and audio content safety),
|
||||
expand the harm taxonomy to emerging threat categories, and develop real-time online
|
||||
learning mechanisms for rapid adaptation to novel evasion patterns.
|
||||
\section*{Acknowledgments and Attribution}
|
||||
|
||||
\section*{Acknowledgments}
|
||||
|
||||
The authors thank Zoo Labs Foundation's safety research team for annotation protocols
|
||||
and the Zen LM red team for adversarial evaluation support.
|
||||
Zen3-Guard is built entirely on IBM Granite Guardian (\texttt{ibm-granite/granite-guardian},
|
||||
Apache-2.0), developed by IBM Research; all model weights, training data, and reported
|
||||
metrics are IBM's. We thank the IBM Granite team for releasing the model openly.
|
||||
|
||||
\bibliographystyle{plain}
|
||||
\begin{thebibliography}{9}
|
||||
@@ -417,14 +324,15 @@ Y. Bai et al.,
|
||||
\textit{arXiv:2212.08073}, 2022.
|
||||
|
||||
\bibitem{multilingual-safety}
|
||||
F. Hartvigsen et al.,
|
||||
T. Hartvigsen et al.,
|
||||
``ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection,''
|
||||
\textit{ACL}, 2022.
|
||||
|
||||
\bibitem{zip001}
|
||||
Zoo Labs Foundation,
|
||||
``ZIP-001: Decentralized Semantic Optimization (DSO),''
|
||||
\textit{Zoo Improvement Proposals}, 2024.
|
||||
\bibitem{graniteguardian}
|
||||
I. Padhi et al. (IBM Research),
|
||||
``Granite Guardian,''
|
||||
\textit{arXiv:2412.07724}, 2024.
|
||||
Model: \href{https://huggingface.co/ibm-granite/granite-guardian-3.0-8b}{ibm-granite/granite-guardian-3.0-8b} (Apache-2.0).
|
||||
|
||||
\end{thebibliography}
|
||||
|
||||
|
||||
Binary file not shown.
+176
-271
@@ -13,7 +13,7 @@
|
||||
\definecolor{zenblue}{RGB}{41,121,255}
|
||||
\hypersetup{colorlinks=true,linkcolor=zenblue,urlcolor=zenblue,citecolor=zenblue}
|
||||
|
||||
\title{\textbf{Zen3-Nano: Third-Generation Ultra-Compact Edge Intelligence}\\
|
||||
\title{\textbf{Zen3-Nano: A Compact-Deployment Finetune of Qwen3-8B}\\
|
||||
\large Technical Whitepaper v2025.06}
|
||||
\author{Zach Kelling \\ Zen LM Research Team\\
|
||||
\texttt{research@zenlm.org}\\
|
||||
@@ -24,16 +24,18 @@
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
Zen3-Nano is the third-generation 0.6B parameter edge model from the Zen family, achieving
|
||||
significant quality improvements over Zen-Nano through architectural refinements and enhanced
|
||||
knowledge distillation from the full Zen family hierarchy. Built on the Zen MoDE (Mixture of
|
||||
Distilled Experts) architecture, Zen3-Nano maintains a sub-250MB quantized footprint while
|
||||
delivering improved reasoning capabilities on mobile and embedded hardware. At 4-bit
|
||||
quantization the model fits in 250MB and achieves 2800 tokens per second on Apple A16 Bionic
|
||||
silicon, enabling real-time language inference at the edge without network connectivity.
|
||||
The model sets new state-of-the-art results for the sub-1B parameter class: MMLU 47.3\%,
|
||||
TriviaQA 61.2\%, and GSM8K 52.4\%, demonstrating that frontier distillation techniques close
|
||||
the gap between edge and server-class models substantially.
|
||||
Zen3-Nano is a packaging- and deployment-focused member of the Zen family. Despite the ``Nano''
|
||||
name, the released Zen3-Nano weights are \emph{not} a 0.6B-parameter from-scratch model: configuration
|
||||
fingerprinting shows an 8B-class dense transformer matching \textbf{Qwen3-8B} \cite{qwen3report}, the
|
||||
open-weight model released by Alibaba's Qwen team under the Apache-2.0 license. Earlier versions of
|
||||
this whitepaper described a bespoke ``Zen MoDE (Mixture of Distilled Experts)'' architecture, a 0.6B
|
||||
parameter count, a from-scratch 2-trillion-token pretraining run, and a hierarchical-distillation
|
||||
pipeline from ``Zen3-Mini/Small/Medium'' teachers. Those claims do not match the shipped artifact and
|
||||
have been removed. This corrected whitepaper documents the actual Qwen3-8B base, the light
|
||||
instruction tuning and quantization-for-deployment work the Zen team performs, and defers to the
|
||||
upstream Qwen3 technical report for pretraining and benchmark numbers. Quantized GGUF/MLX builds are
|
||||
provided for convenient deployment; the genuine ``small'' tiers of the Qwen3 family (0.6B, 1.7B, 4B)
|
||||
are noted as the appropriate bases for true edge use.
|
||||
\end{abstract}
|
||||
|
||||
\tableofcontents
|
||||
@@ -43,316 +45,210 @@ the gap between edge and server-class models substantially.
|
||||
\section{Introduction}
|
||||
\label{sec:intro}
|
||||
|
||||
On-device AI inference has become a primary deployment target for consumer applications,
|
||||
IoT sensors, and privacy-sensitive enterprise workloads. The constraints are severe: models
|
||||
must fit within a few hundred megabytes, execute without accelerators on ARM Cortex-class
|
||||
CPUs, and return responses in tens of milliseconds. Existing sub-1B approaches sacrifice
|
||||
too much quality to be practically useful for general-purpose instruction following.
|
||||
On-device and resource-constrained inference is an important deployment target, and quantization
|
||||
makes even 8B-class models broadly deployable. This whitepaper documents Zen3-Nano honestly,
|
||||
correcting prior inaccurate claims about its size and origin.
|
||||
|
||||
Zen3-Nano addresses this challenge by combining three advances:
|
||||
\paragraph{Provenance and corrections.} The released Zen3-Nano weights fingerprint to
|
||||
\textbf{Qwen3-8B}: 36 layers, hidden size 4096, 32 query heads / 8 KV heads (GQA), head dimension
|
||||
128, FFN intermediate size 12{,}288, and a 151{,}936-token vocabulary \cite{qwen3hf}. This is an
|
||||
8.2B-parameter dense model released by the Qwen team at Alibaba Cloud under the Apache-2.0 license
|
||||
\cite{qwen3report}. The following earlier claims were inaccurate and are withdrawn:
|
||||
|
||||
\begin{enumerate}
|
||||
\item \textbf{Hierarchical Zen MoDE distillation} --- structured distillation cascaded through
|
||||
Zen3-Mini (1.7B), Zen3-Small (7B), and Zen3-Medium (14B) teachers, each contributing
|
||||
a complementary signal.
|
||||
\item \textbf{Third-generation Nano architecture} --- refined attention with grouped-query
|
||||
heads optimized for cache reuse on low-memory hardware, reduced KV cache footprint,
|
||||
and fused MLPs that lower inference memory bandwidth.
|
||||
\item \textbf{BitDelta-aware training} --- the model is trained with quantization-aware
|
||||
objectives derived from the Zoo Labs BitDelta protocol (ZIP-007), so the 4-bit
|
||||
quantized artifact retains near-full-precision accuracy without post-hoc calibration.
|
||||
\end{enumerate}
|
||||
|
||||
Zen3-Nano is designed to run natively on:
|
||||
\begin{itemize}
|
||||
\item Mobile SoCs (Apple A16 Bionic, Qualcomm Snapdragon 8 Gen 3, MediaTek Dimensity 9300)
|
||||
\item Microcontrollers with $\geq$512 KB SRAM (via WASM SIMD runtime)
|
||||
\item Browser environments via WebAssembly and WebGPU
|
||||
\item Embedded Linux (Raspberry Pi 5, NVIDIA Jetson Nano)
|
||||
\item ``0.6B parameters'' / ``sub-250MB'' / ``third-generation 0.6B edge model'' --- the shipped
|
||||
model is 8B-class, not 0.6B.
|
||||
\item ``Zen MoDE (Mixture of Distilled Experts)'' architecture --- Qwen3-8B is a standard dense
|
||||
transformer, not a mixture-of-experts model; there is no ``MoDE'' routing.
|
||||
\item ``Trained from scratch on a 2T-token corpus'' and ``hierarchical distillation from
|
||||
Zen3-Mini/Small/Medium teachers'' --- no such from-scratch pretraining or teacher cascade was
|
||||
performed by the Zen team; the base is the released Qwen3-8B.
|
||||
\item Specific benchmark figures (e.g., MMLU 47.3\%, GSM8K 52.4\%, and the per-hardware
|
||||
tokens/second table) --- these were fabricated and are removed.
|
||||
\end{itemize}
|
||||
|
||||
\paragraph{What Zen3-Nano actually is.} Zen3-Nano is a lightly instruction-tuned and quantized
|
||||
repackaging of Qwen3-8B aimed at easy local deployment. Its contributions are (i) optional light SFT
|
||||
for formatting/instruction style on top of Qwen3-8B, and (ii) ready-to-run quantized artifacts
|
||||
(GGUF for llama.cpp, MLX for Apple Silicon, ONNX). For genuinely tiny edge footprints, the Qwen3
|
||||
family also offers true small dense models---Qwen3-0.6B, Qwen3-1.7B, and Qwen3-4B (all Apache-2.0)
|
||||
\cite{qwen3report}---which are the appropriate bases when sub-1B size is a hard requirement.
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{Architecture}
|
||||
\section{Architecture (Inherited from Qwen3-8B)}
|
||||
\label{sec:arch}
|
||||
|
||||
\subsection{Zen MoDE Foundation}
|
||||
Zen3-Nano inherits the Qwen3-8B architecture unchanged: a dense decoder-only transformer with
|
||||
grouped-query attention, SwiGLU feed-forward blocks, RMSNorm, and rotary position embeddings. There
|
||||
is no mixture-of-experts component.
|
||||
|
||||
Zen3-Nano is built on the Zen MoDE (Mixture of Distilled Experts) architecture, a
|
||||
design principle shared across the entire Zen family that replaces monolithic dense
|
||||
feed-forward blocks with sparsely-activated expert clusters. At the 0.6B scale, MoDE
|
||||
manifests as a lightweight routing mechanism that activates a subset of micro-experts
|
||||
per token, preserving capacity diversity while keeping the active parameter count
|
||||
well below memory limits.
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Architecture hyperparameters, inherited unchanged from Qwen3-8B \cite{qwen3hf}.}
|
||||
\begin{tabular}{lc}
|
||||
\toprule
|
||||
\textbf{Hyperparameter} & \textbf{Value} \\
|
||||
\midrule
|
||||
Parameters (total) & 8.2B \\
|
||||
Non-embedding parameters & 6.95B \\
|
||||
Layers & 36 \\
|
||||
Attention heads (query) & 32 \\
|
||||
KV heads (GQA) & 8 \\
|
||||
Head dimension & 128 \\
|
||||
Hidden dimension & 4096 \\
|
||||
FFN intermediate dimension & 12{,}288 \\
|
||||
Vocabulary size & 151{,}936 \\
|
||||
Context length (native) & 40{,}960 \\
|
||||
Context length (with YaRN) & 131{,}072 \\
|
||||
Position encoding & RoPE ($\theta = 1{,}000{,}000$) \\
|
||||
Activation function & SiLU (SwiGLU) \\
|
||||
Normalization & RMSNorm \\
|
||||
Tied embeddings & No \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:arch}
|
||||
\end{table}
|
||||
|
||||
Formally, for input token representation $\mathbf{x} \in \mathbb{R}^d$, the MoDE feed-forward
|
||||
layer computes:
|
||||
\subsection{Grouped-Query Attention}
|
||||
|
||||
Qwen3-8B uses GQA \cite{ainslie2023gqa} with 32 query heads grouped onto 8 key-value heads (head
|
||||
dimension 128), reducing KV-cache memory relative to full multi-head attention. Rotary positional
|
||||
embeddings (RoPE) \cite{su2021rope} are used with base frequency $\theta = 1{,}000{,}000$; long
|
||||
context up to 131{,}072 tokens is available via YaRN scaling \cite{peng2023yarn} as configured
|
||||
upstream.
|
||||
|
||||
\subsection{Feed-Forward Network}
|
||||
|
||||
Each layer uses a SwiGLU feed-forward block \cite{shazeer2020glu}:
|
||||
|
||||
\begin{equation}
|
||||
\text{MoDE}(\mathbf{x}) = \sum_{i \in \mathcal{K}(\mathbf{x})} g_i(\mathbf{x}) \cdot E_i(\mathbf{x})
|
||||
\text{FFN}(\mathbf{x}) = \left(\text{SiLU}(\mathbf{x}\mathbf{W}_{\text{gate}}) \odot
|
||||
\mathbf{x}\mathbf{W}_{\text{up}}\right) \mathbf{W}_{\text{down}}
|
||||
\end{equation}
|
||||
|
||||
where $\mathcal{K}(\mathbf{x})$ is the set of top-$k$ activated experts selected by a
|
||||
learned routing function, $g_i$ are normalized routing weights, and $E_i$ are the
|
||||
expert sub-networks.
|
||||
|
||||
\subsection{Grouped-Query Attention (GQA) Variant}
|
||||
|
||||
Standard multi-head attention at 0.6B scale incurs disproportionate KV cache overhead
|
||||
relative to total parameter count. Zen3-Nano uses a 3rd-generation grouped-query attention
|
||||
variant with the following configuration:
|
||||
|
||||
\begin{itemize}
|
||||
\item 16 query heads grouped into 4 key-value head groups
|
||||
\item 1024-dimensional hidden state, 64-dimensional head size
|
||||
\item Rotary positional embeddings (RoPE) with extended context via YaRN interpolation
|
||||
\item 32 transformer layers with pre-layer normalization (RMSNorm)
|
||||
\end{itemize}
|
||||
|
||||
The KV cache per token occupies only 512 bytes at full precision, enabling 32K token
|
||||
context windows within 16 MB of SRAM when using 8-bit KV quantization.
|
||||
|
||||
\subsection{Fused MLP Design}
|
||||
|
||||
The feed-forward blocks use a fused SwiGLU activation with weight-sharing across
|
||||
adjacent expert pairs, reducing total parameter count by 12\% relative to a naive
|
||||
MoDE implementation at equivalent quality:
|
||||
|
||||
\begin{equation}
|
||||
\text{FFN}(\mathbf{x}) = \left(\sigma(\mathbf{W}_1 \mathbf{x}) \odot \mathbf{W}_2 \mathbf{x}\right) \mathbf{W}_3
|
||||
\end{equation}
|
||||
|
||||
\subsection{Quantization-Native Design}
|
||||
|
||||
Unlike post-hoc quantization, Zen3-Nano is trained with BitDelta quantization-aware
|
||||
training from epoch 1. The quantization error $\epsilon_Q$ is included in the training
|
||||
loss as a regularizer:
|
||||
|
||||
\begin{equation}
|
||||
\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{CE}} + \mathcal{L}_{\text{KD}} + \lambda \cdot \mathbb{E}\left[\|\mathbf{W} - Q(\mathbf{W})\|_F^2\right]
|
||||
\end{equation}
|
||||
|
||||
where $Q(\cdot)$ is the 4-bit quantization operator and $\lambda = 0.01$.
|
||||
with intermediate dimension 12{,}288. There is no expert routing; the previously described
|
||||
``$\text{MoDE}(\mathbf{x}) = \sum_{i} g_i(\mathbf{x}) E_i(\mathbf{x})$'' formulation does not apply
|
||||
to this model.
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{Training}
|
||||
\section{Post-Training and Quantization}
|
||||
\label{sec:training}
|
||||
|
||||
\subsection{Pre-Training Data}
|
||||
The Zen team does not pretrain Zen3-Nano. Pretraining of the underlying Qwen3-8B was performed by the
|
||||
Qwen team; see the upstream technical report \cite{qwen3report} for the corpus, scale, and procedure.
|
||||
Our work on top of the released weights is limited to:
|
||||
|
||||
Zen3-Nano is initialized from scratch on a 2T token corpus drawn from:
|
||||
\begin{itemize}
|
||||
\item Curated web text (Common Crawl quality-filtered): 60\%
|
||||
\item Code repositories (GitHub, GitLab, permissive licenses): 20\%
|
||||
\item Books, academic papers, structured knowledge: 15\%
|
||||
\item Multilingual data (50+ languages, weighted by speaker population): 5\%
|
||||
\end{itemize}
|
||||
\subsection{Light Instruction Tuning (Optional)}
|
||||
|
||||
\subsection{Hierarchical Distillation}
|
||||
Where a chat-tuned variant is shipped, we apply light supervised fine-tuning on curated
|
||||
instruction--response data (response-masked loss, low learning rate, few epochs) to adjust formatting
|
||||
and instruction-following style. This does not constitute pretraining and does not materially change
|
||||
the base model's knowledge.
|
||||
|
||||
After base pre-training, Zen3-Nano undergoes a three-stage distillation curriculum:
|
||||
\subsection{Quantization for Deployment}
|
||||
|
||||
\textbf{Stage 1 (Token-level KL distillation)}: Soft targets from Zen3-Mini (1.7B) teacher
|
||||
over 500B tokens. Temperature $T=4.0$, KD weight $\alpha = 0.7$.
|
||||
|
||||
\textbf{Stage 2 (Layer-level representation matching)}: Intermediate hidden states
|
||||
from Zen3-Small (7B) are used as representation targets via a learned projection head.
|
||||
Loss: cosine similarity maximization over layers $\{6, 12, 18, 24\}$.
|
||||
|
||||
\textbf{Stage 3 (Reasoning chain distillation)}: Structured chain-of-thought traces
|
||||
generated by Zen3-Medium (14B) are used for supervised fine-tuning with a 3:1
|
||||
ratio of distilled traces to human annotations.
|
||||
|
||||
\subsection{Instruction Fine-Tuning}
|
||||
|
||||
Post-distillation instruction tuning uses a 50M token mix:
|
||||
\begin{itemize}
|
||||
\item General instruction following: 40\%
|
||||
\item Code generation and debugging: 25\%
|
||||
\item Mathematical reasoning (chain-of-thought): 20\%
|
||||
\item Multilingual instruction: 15\%
|
||||
\end{itemize}
|
||||
|
||||
Training uses AdamW optimizer, peak learning rate $3 \times 10^{-4}$, cosine decay,
|
||||
batch size 512, over 3 epochs on the instruction mix.
|
||||
|
||||
\subsection{RLHF and Constitutional Alignment}
|
||||
|
||||
A lightweight RLHF pass using PPO with a 0.6B reward model (trained separately)
|
||||
is applied for 10K steps to reduce harmful outputs and improve instruction adherence,
|
||||
keeping the KL divergence from the SFT checkpoint below 0.1 nats.
|
||||
The deployment artifacts use standard post-training quantization (e.g., llama.cpp's Q4\_K\_M and
|
||||
Q8\_0 schemes, and MLX 4-bit) applied to the released weights. We do not perform the
|
||||
``quantization-aware training from epoch 1'' described in earlier versions of this whitepaper; the
|
||||
shipped artifact is the released Qwen3-8B quantized post hoc with widely used GGUF/MLX recipes.
|
||||
References to a proprietary ``BitDelta (ZIP-007)'' quantization-aware objective have been removed
|
||||
as they were not used to produce this model.
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{Evaluation}
|
||||
\label{sec:eval}
|
||||
|
||||
\subsection{Language Understanding}
|
||||
\paragraph{On benchmark numbers.} This whitepaper no longer reports fabricated benchmark tables or
|
||||
per-device throughput figures. Because Zen3-Nano is the Qwen3-8B model (optionally lightly tuned and
|
||||
quantized), its capabilities are those of Qwen3-8B, subject to the usual accuracy cost of 4-bit
|
||||
quantization. For rigorous, independently reproducible results on Qwen3 models (MMLU, GSM8K, MATH,
|
||||
HumanEval, multilingual, and long-context), see the official Qwen3 technical report
|
||||
\cite{qwen3report} and the Qwen3-8B model card \cite{qwen3hf}. Any quantization-vs-BF16 quality
|
||||
deltas we publish are measured on the released GGUF/MLX artifacts under a stated harness; we do not
|
||||
quote numbers we have not measured.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Zen3-Nano vs. sub-1B baselines on core language benchmarks}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{Params} & \textbf{MMLU} & \textbf{TriviaQA} & \textbf{HellaSwag} \\
|
||||
\midrule
|
||||
Prior Zen-Nano & 0.5B & 39.1 & 53.4 & 61.2 \\
|
||||
Zen2-Nano & 0.6B & 43.7 & 57.8 & 66.4 \\
|
||||
\textbf{Zen3-Nano} & \textbf{0.6B} & \textbf{47.3} & \textbf{61.2} & \textbf{70.8} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:lang}
|
||||
\end{table}
|
||||
|
||||
\subsection{Mathematical Reasoning}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Mathematical reasoning benchmarks}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{GSM8K} & \textbf{MATH} & \textbf{MGSM (avg)} & \textbf{ARC-C} \\
|
||||
\midrule
|
||||
Zen-Nano & 38.2 & 11.3 & 31.4 & 54.1 \\
|
||||
Zen2-Nano & 44.6 & 15.7 & 38.9 & 59.3 \\
|
||||
\textbf{Zen3-Nano} & \textbf{52.4} & \textbf{20.1} & \textbf{46.2} & \textbf{64.7} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:math}
|
||||
\end{table}
|
||||
|
||||
\subsection{Code Generation}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Code generation benchmarks (pass@1)}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{HumanEval} & \textbf{MBPP} & \textbf{LiveCodeBench} \\
|
||||
\midrule
|
||||
Zen-Nano & 28.0 & 33.2 & 12.4 \\
|
||||
Zen2-Nano & 34.1 & 40.5 & 18.7 \\
|
||||
\textbf{Zen3-Nano} & \textbf{42.7} & \textbf{49.3} & \textbf{26.1} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:code}
|
||||
\end{table}
|
||||
|
||||
\subsection{Inference Performance}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Inference throughput across hardware targets (4-bit quantized)}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
\textbf{Hardware} & \textbf{Precision} & \textbf{Tok/s} & \textbf{RAM (MB)} & \textbf{Latency (ms/tok)} \\
|
||||
\midrule
|
||||
Apple A16 Bionic & INT4 & 2800 & 248 & 0.36 \\
|
||||
Snapdragon 8 Gen 3 & INT4 & 2340 & 248 & 0.43 \\
|
||||
Raspberry Pi 5 & INT4 & 420 & 248 & 2.38 \\
|
||||
WASM SIMD (browser) & INT4 & 310 & 248 & 3.23 \\
|
||||
x86-64 (laptop CPU) & INT4 & 1100 & 248 & 0.91 \\
|
||||
NVIDIA RTX 4090 & FP16 & 8400 & 950 & 0.12 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:perf}
|
||||
\end{table}
|
||||
|
||||
\subsection{Multilingual Performance}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Multilingual benchmark (FLORES-200, spBLEU)}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Language Pair} & \textbf{Zen3-Nano} & \textbf{Zen2-Nano} \\
|
||||
\midrule
|
||||
English $\to$ Chinese & 24.3 & 20.1 \\
|
||||
English $\to$ Spanish & 31.8 & 27.4 \\
|
||||
English $\to$ Arabic & 18.9 & 14.7 \\
|
||||
English $\to$ Japanese & 21.4 & 17.2 \\
|
||||
Average (50 pairs) & 22.6 & 18.4 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:multilingual}
|
||||
\end{table}
|
||||
\paragraph{Right-sizing for edge.} If a true sub-1B footprint is required, the appropriate choice is
|
||||
a genuinely small Qwen3 base (0.6B/1.7B/4B), not an 8B model relabeled ``Nano.'' Published Qwen3
|
||||
results for those tiers \cite{qwen3report} should be used to set expectations rather than the
|
||||
fabricated figures previously printed here.
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{Related Work}
|
||||
\label{sec:related}
|
||||
|
||||
\subsection{Small Language Models}
|
||||
|
||||
Research on sub-1B language models has accelerated as mobile deployment became
|
||||
commercially viable. Early work demonstrated that aggressive distillation from
|
||||
multi-billion parameter teachers could recover a substantial fraction of performance
|
||||
at a fraction of the parameter count~\cite{hinton2015distilling}. Subsequent work
|
||||
explored architectural modifications specifically suited to on-device constraints,
|
||||
including weight sharing, structured pruning, and low-rank factorization.
|
||||
|
||||
\subsection{Quantization-Aware Training}
|
||||
|
||||
Post-training quantization (PTQ) has become the default for edge deployment, but
|
||||
accuracy degradation at 4-bit is nontrivial for small models where the per-parameter
|
||||
information density is already high. BitDelta (ZIP-007) addresses this by encoding
|
||||
quantization error as an explicit training signal, showing that models trained
|
||||
quantization-natively outperform PTQ equivalents by 2--5 percentage points on
|
||||
reasoning benchmarks at 4-bit precision.
|
||||
|
||||
\subsection{Knowledge Distillation Hierarchies}
|
||||
|
||||
Prior work on hierarchical distillation~\cite{mirzadeh2020improved} demonstrated that
|
||||
using an intermediate ``teacher assistant'' model reduces the capacity gap and improves
|
||||
distillation efficiency. Zen3-Nano extends this to a three-level cascade, exploiting
|
||||
the full Zen family model hierarchy.
|
||||
Knowledge distillation \cite{hinton2015distilling} and teacher-assistant cascades
|
||||
\cite{mirzadeh2020improved} are well-established techniques for producing smaller models; however,
|
||||
Zen3-Nano as shipped is not the product of such a pipeline---it is the Qwen3-8B base, optionally
|
||||
lightly tuned and quantized. Post-training quantization for efficient deployment is standard practice
|
||||
(e.g., the GGUF schemes used by llama.cpp). The underlying base model belongs to the Qwen3 series
|
||||
\cite{qwen3report}, which provides a coherent family of dense (0.6B--32B) and MoE models under
|
||||
Apache-2.0.
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{Deployment and Runtime}
|
||||
\label{sec:deploy}
|
||||
|
||||
\subsection{Supported Runtimes}
|
||||
Because Zen3-Nano is architecturally Qwen3-8B, it runs on any Qwen3-compatible runtime:
|
||||
|
||||
Zen3-Nano ships with native support for:
|
||||
\begin{itemize}
|
||||
\item \textbf{llama.cpp} --- GGUF format, Q4\_K\_M quantization
|
||||
\item \textbf{ONNX Runtime} --- optimized for ARM and x86 SIMD
|
||||
\item \textbf{WebAssembly} --- via wasm-pack compiled inference engine
|
||||
\item \textbf{Core ML} --- iOS/macOS native accelerated inference
|
||||
\item \textbf{TensorFlow Lite} --- Android NPU acceleration
|
||||
\item \textbf{llama.cpp} --- GGUF format (Q4\_K\_M, Q8\_0).
|
||||
\item \textbf{MLX} --- Apple Silicon, 4-bit.
|
||||
\item \textbf{ONNX Runtime} --- ARM and x86 SIMD.
|
||||
\item \textbf{Hugging Face Transformers / vLLM} --- BF16 reference weights.
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Context Length}
|
||||
|
||||
Default context window: 8192 tokens (configurable to 32K via YaRN at cost of
|
||||
increased RAM). At 32K context, KV cache occupies approximately 64 MB at INT8.
|
||||
Native context is 40{,}960 tokens (Qwen3-8B default), extensible to 131{,}072 tokens via YaRN at the
|
||||
cost of additional KV-cache memory. Memory footprint depends on quantization: an 8B model in 4-bit
|
||||
requires on the order of several gigabytes, which is larger than the sub-250MB figure incorrectly
|
||||
cited in earlier versions.
|
||||
|
||||
\subsection{API Compatibility}
|
||||
|
||||
Zen3-Nano exposes an OpenAI-compatible API when deployed via the Zen LM inference
|
||||
server, enabling drop-in replacement for any OpenAI SDK client.
|
||||
When served via a Qwen3-compatible inference server, Zen3-Nano exposes an OpenAI-compatible API,
|
||||
enabling drop-in replacement for OpenAI SDK clients.
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{Limitations}
|
||||
\label{sec:limitations}
|
||||
|
||||
Zen3-Nano inherits the limitations of Qwen3-8B and of autoregressive models generally: factual
|
||||
hallucination, degraded performance on long multi-step reasoning, and incomplete safety alignment.
|
||||
Post-training quantization to 4-bit introduces an additional, measurable accuracy cost relative to
|
||||
BF16. The ``Nano'' name is a legacy product label and does not indicate a sub-1B model; deployments
|
||||
with hard memory limits should select a genuinely small Qwen3 base instead.
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{Conclusion}
|
||||
\label{sec:conclusion}
|
||||
|
||||
Zen3-Nano establishes a new state of the art for sub-1B parameter language models
|
||||
through hierarchical Zen MoDE distillation, quantization-native training, and
|
||||
architectural refinements that maximize inference efficiency on ARM-class hardware.
|
||||
The model achieves MMLU 47.3\%, TriviaQA 61.2\%, and GSM8K 52.4\% --- results
|
||||
that would have required 3--7B parameter models in prior generations --- while
|
||||
fitting within 250 MB at 4-bit precision and achieving 2800 tok/s on Apple A16
|
||||
Bionic.
|
||||
|
||||
Future work will target 1M token context windows via ring attention approximations
|
||||
suitable for low-memory hardware, and extended multimodal understanding via
|
||||
lightweight vision adapters compatible with the Zen3-Omni decoder.
|
||||
Zen3-Nano, as released, is the Apache-2.0 Qwen3-8B model---optionally lightly instruction-tuned and
|
||||
quantized for convenient local deployment---not a 0.6B from-scratch ``Zen MoDE'' model. This
|
||||
corrected whitepaper documents the inherited Qwen3-8B architecture, the limited post-training and
|
||||
standard quantization actually performed, and the correct guidance to use a true small Qwen3 base
|
||||
(0.6B/1.7B/4B) when a sub-1B edge footprint is required. We defer to the upstream Qwen3 technical
|
||||
report for pretraining details and rigorous, attributed benchmark numbers.
|
||||
|
||||
\section*{Acknowledgments}
|
||||
|
||||
The authors thank the Zoo Labs Foundation for compute allocation and the DSO
|
||||
research network for benchmark curation and evaluation infrastructure.
|
||||
We thank the Qwen team at Alibaba Cloud for releasing the Qwen3 models under the Apache-2.0 license,
|
||||
and the maintainers of llama.cpp and MLX for the quantization and runtime tooling used to produce the
|
||||
deployment artifacts.
|
||||
|
||||
\bibliographystyle{plain}
|
||||
\begin{thebibliography}{9}
|
||||
|
||||
\bibitem{qwen3report}
|
||||
Qwen Team, Alibaba Cloud,
|
||||
``Qwen3 Technical Report,''
|
||||
\textit{arXiv:2505.09388}, 2025.
|
||||
|
||||
\bibitem{qwen3hf}
|
||||
Qwen Team,
|
||||
``Qwen3-8B model card and configuration,''
|
||||
Hugging Face, \url{https://huggingface.co/Qwen/Qwen3-8B}, 2025.
|
||||
|
||||
\bibitem{hinton2015distilling}
|
||||
G. Hinton, O. Vinyals, and J. Dean,
|
||||
``Distilling the Knowledge in a Neural Network,''
|
||||
@@ -363,16 +259,25 @@ S. I. Mirzadeh et al.,
|
||||
``Improved Knowledge Distillation via Teacher Assistant,''
|
||||
\textit{AAAI}, 2020.
|
||||
|
||||
\bibitem{zip007}
|
||||
Zoo Labs Foundation,
|
||||
``ZIP-007: BitDelta --- Quantization-Aware Delta Compression for Language Models,''
|
||||
\textit{Zoo Improvement Proposals}, 2024.
|
||||
\href{https://zips.zoo.ngo/zip-007}{zips.zoo.ngo/zip-007}
|
||||
\bibitem{ainslie2023gqa}
|
||||
J. Ainslie et al.,
|
||||
``GQA: Training Generalized Multi-Query Transformer Models,''
|
||||
\textit{EMNLP}, 2023.
|
||||
|
||||
\bibitem{hip002}
|
||||
Hanzo AI,
|
||||
``HIP-002: Active Semantic Optimization (ASO),''
|
||||
\textit{Hanzo Improvement Proposals}, 2024.
|
||||
\bibitem{su2021rope}
|
||||
J. Su et al.,
|
||||
``RoFormer: Enhanced Transformer with Rotary Position Embedding,''
|
||||
\textit{arXiv:2104.09864}, 2021.
|
||||
|
||||
\bibitem{shazeer2020glu}
|
||||
N. Shazeer,
|
||||
``GLU Variants Improve Transformer,''
|
||||
\textit{arXiv:2002.05202}, 2020.
|
||||
|
||||
\bibitem{peng2023yarn}
|
||||
B. Peng et al.,
|
||||
``YaRN: Efficient Context Window Extension of Large Language Models,''
|
||||
\textit{arXiv:2309.00071}, 2023.
|
||||
|
||||
\end{thebibliography}
|
||||
|
||||
|
||||
Binary file not shown.
+151
-293
@@ -13,9 +13,9 @@
|
||||
\definecolor{zenblue}{RGB}{41,121,255}
|
||||
\hypersetup{colorlinks=true,linkcolor=zenblue,urlcolor=zenblue,citecolor=zenblue}
|
||||
|
||||
\title{\textbf{Zen3-Omni: Third-Generation Unified Multimodal Intelligence}\\
|
||||
\large Technical Whitepaper v2025.06}
|
||||
\author{Antje Worring, Zach Kelling \\ Zen LM Research Team\\
|
||||
\title{\textbf{Zen3-Omni: A Packaging of Qwen3-Omni-30B-A3B}\\
|
||||
\large Technical Note v2025.06}
|
||||
\author{Zen LM Research Team\\
|
||||
\texttt{research@zenlm.org}\\
|
||||
\href{https://papers.zenlm.org}{papers.zenlm.org}}
|
||||
\date{June 2025}
|
||||
@@ -24,336 +24,194 @@
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
Zen3-Omni advances multimodal AI by unifying text, image, audio, and video understanding
|
||||
in a single 7B parameter model, featuring a novel cross-modal attention architecture that
|
||||
enables fluent reasoning across modalities with state-of-the-art performance on multimodal
|
||||
benchmarks. The third generation introduces substantial improvements in video understanding
|
||||
through temporal sparse attention and real-time audio processing via streaming CTC decoders
|
||||
integrated natively into the Zen MoDE (Mixture of Distilled Experts) backbone. Zen3-Omni
|
||||
achieves MMBench 86.2\%, AudioCaps CLAP 0.523, VideoQA 79.4\%, and OCRBench 82.1\%,
|
||||
making it the leading 7B-class multimodal model across all four primary modalities
|
||||
simultaneously. The model supports streaming inference for real-time audio/video inputs
|
||||
with end-to-end latency below 300ms on a single A100 GPU.
|
||||
Zen3-Omni is \emph{not} a from-scratch model. It is a redistribution and packaging of
|
||||
\textbf{Qwen3-Omni-30B-A3B}, the natively end-to-end omni-modal foundation model developed
|
||||
by the Qwen team at Alibaba Cloud and released under the Apache-2.0 license~\cite{qwen3omni}.
|
||||
This note documents the upstream architecture as published by its authors, records the
|
||||
provenance and license obligations that attach to any redistribution, and describes the
|
||||
thin packaging layer (weight conversion, quantization, and serving configuration) that Zen
|
||||
LM applies on top of the upstream checkpoint. No architectural novelty, no separate
|
||||
pre-training run, and no independently produced benchmark results are claimed here. For
|
||||
authoritative capability numbers, the reader is directed to the upstream model cards and
|
||||
the Qwen3-Omni Technical Report~\cite{qwen3omni,qwen3omnireport}.
|
||||
\end{abstract}
|
||||
|
||||
\tableofcontents
|
||||
\newpage
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{Introduction}
|
||||
\section{Provenance and Scope}
|
||||
\label{sec:intro}
|
||||
|
||||
The dominant paradigm in multimodal AI has been modality-specialized towers: separate
|
||||
vision encoders, audio encoders, and video encoders, each pre-trained independently and
|
||||
fused into a shared language model backbone via adapter layers. While effective, this
|
||||
architecture introduces seams at modality boundaries --- the model reasons about each
|
||||
modality separately rather than in a unified representational space.
|
||||
Earlier internal drafts of this document described a homegrown ``Zen MoDE (Mixture of
|
||||
Distilled Experts)'' backbone and reported a set of benchmark scores as if they had been
|
||||
produced by an in-house model. That framing was inaccurate and has been removed. The
|
||||
artifact distributed as ``Zen3-Omni'' is the Qwen3-Omni-30B-A3B checkpoint published by
|
||||
Alibaba Cloud's Qwen team~\cite{qwen3omni}. This note exists to attribute the work
|
||||
correctly and to make the redistribution terms explicit.
|
||||
|
||||
Zen3-Omni is built on a fundamentally different premise: a single cross-modal attention
|
||||
mechanism processes tokens from all modalities together, allowing the model to form
|
||||
joint representations where a spoken word, its transcript, and a visual depiction of the
|
||||
concept co-attend to one another naturally.
|
||||
\textbf{Upstream model.} Qwen3-Omni-30B-A3B is a natively omni-modal model: a single system
|
||||
that accepts text, audio, image, and video inputs and produces both text and natural speech
|
||||
output. It is published in multiple editions on Hugging Face, including
|
||||
\texttt{Qwen3-Omni-30B-A3B-Instruct}, \texttt{Qwen3-Omni-30B-A3B-Thinking}, and
|
||||
\texttt{Qwen3-Omni-30B-A3B-Captioner}~\cite{qwen3omni}.
|
||||
|
||||
This paper makes the following contributions:
|
||||
\textbf{License.} The upstream weights are released under \textbf{Apache-2.0}. Apache-2.0 is
|
||||
permissive and allows commercial use and redistribution, but it carries obligations that a
|
||||
redistributor must honor: preservation of copyright and license notices, inclusion of the
|
||||
\texttt{NOTICE} file if present, and a clear statement of any modifications. Any Zen LM
|
||||
redistribution must ship the upstream \texttt{LICENSE} and attribution intact and must not
|
||||
misrepresent the model's origin.
|
||||
|
||||
\textbf{What ``packaging'' means here.} Zen LM does not retrain or re-architect the model.
|
||||
The packaging layer is limited to: (i) format conversion of the published weights to the
|
||||
serving runtime used internally; (ii) optional post-training quantization for deployment;
|
||||
and (iii) inference and serving configuration. These steps do not change the model's
|
||||
identity or its measured capabilities, and they do not justify a new model name implying
|
||||
original research.
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{Upstream Architecture (as published by Qwen)}
|
||||
\label{sec:arch}
|
||||
|
||||
The description below summarizes the architecture as documented by the upstream authors in
|
||||
the Qwen3-Omni model card and Technical Report~\cite{qwen3omni,qwen3omnireport}. It is
|
||||
reproduced for the reader's convenience and is \emph{not} a Zen LM contribution.
|
||||
|
||||
\subsection{Thinker--Talker--Code2Wav Design}
|
||||
|
||||
Qwen3-Omni uses a multi-module design built around three components:
|
||||
|
||||
\begin{itemize}
|
||||
\item \textbf{Thinker} --- A Mixture-of-Experts (MoE) transformer responsible for
|
||||
multimodal perception and reasoning. It ingests text, audio, image, and video tokens
|
||||
and produces a unified high-level representation. The ``A3B'' in the model name
|
||||
reflects an MoE configuration in which only a small subset of the total parameters is
|
||||
activated per token (the model is named for roughly 3B activated parameters out of a
|
||||
30B-class total parameter budget).
|
||||
\item \textbf{Talker} --- A second MoE transformer that performs streaming speech generation
|
||||
via multi-codebook autoregressive prediction, enabling low-latency text-to-speech
|
||||
output conditioned on the Thinker's representation.
|
||||
\item \textbf{Code2Wav} --- A streaming waveform renderer that converts the Talker's
|
||||
codebook sequence into audio frame-by-frame for real-time output.
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Audio Encoder (AuT)}
|
||||
|
||||
Per the upstream authors, the audio front-end is a custom Audio Transformer (\textbf{AuT})
|
||||
encoder trained on a large supervised audio corpus, replacing the Whisper encoder used in
|
||||
earlier Qwen omni-modal work. The exact pre-training scale and configuration are documented
|
||||
upstream~\cite{qwen3omnireport}; they are not reproduced or independently verified here.
|
||||
|
||||
\subsection{Modalities and Language Coverage}
|
||||
|
||||
As published on the model card~\cite{qwen3omni}, Qwen3-Omni supports:
|
||||
|
||||
\begin{itemize}
|
||||
\item \textbf{Inputs}: text, audio, image, and video.
|
||||
\item \textbf{Outputs}: text and natural speech.
|
||||
\item \textbf{Text}: 119 languages.
|
||||
\item \textbf{Speech input}: 19 languages.
|
||||
\item \textbf{Speech output}: 10 languages.
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Items Removed From Earlier Drafts}
|
||||
|
||||
The following claims appeared in prior drafts and are \textbf{withdrawn} because they were
|
||||
not substantiated and were presented as in-house work: a proprietary ``Zen MoDE'' expert
|
||||
backbone with a specific modality-balance loss; specific expert counts and routing
|
||||
top-$k$ values attributed to Zen; a bespoke ``temporal sparse attention'' formulation; and
|
||||
a ``distillation from Zen3-VL (72B)'' procedure. Any expert-routing, positional, or
|
||||
attention details that are real belong to the upstream Qwen3-Omni design and are documented
|
||||
by its authors~\cite{qwen3omnireport}.
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{Packaging and Redistribution}
|
||||
\label{sec:packaging}
|
||||
|
||||
\subsection{Conversion and Quantization}
|
||||
|
||||
Zen LM converts the published Qwen3-Omni weights to the internal serving format and may
|
||||
apply post-training quantization for deployment efficiency. These transforms are mechanical
|
||||
and lossy only in the usual sense of quantization; they do not constitute a new training run
|
||||
and do not produce a new model. Where quantization is applied, the quantization scheme and
|
||||
bit-width should be recorded alongside the redistributed artifact so that downstream users
|
||||
understand any quality trade-off relative to the upstream full-precision checkpoint.
|
||||
|
||||
\subsection{Attribution Requirements}
|
||||
|
||||
Because the upstream license is Apache-2.0, every redistributed Zen3-Omni artifact must:
|
||||
|
||||
\begin{enumerate}
|
||||
\item \textbf{Unified cross-modal attention} --- A single attention mechanism that handles
|
||||
text, image patch, audio frame, and video temporal tokens without modality-specific
|
||||
routing, enabling true multi-modal chain-of-thought reasoning.
|
||||
\item \textbf{Temporal sparse attention for video} --- A hierarchical temporal attention
|
||||
pattern that processes 60fps video at 1-second granularity without quadratic
|
||||
sequence length blowup, enabling up to 10-minute video inputs.
|
||||
\item \textbf{Streaming CTC audio decoder} --- Real-time audio transcription and
|
||||
understanding with 40ms chunk processing latency, enabling live speech input.
|
||||
\item \textbf{Third-generation Zen MoDE scaling} --- Expert routing updated with
|
||||
modality-aware load balancing that prevents expert collapse on underrepresented
|
||||
modalities.
|
||||
\item Retain the upstream copyright and Apache-2.0 \texttt{LICENSE} text.
|
||||
\item Include the upstream \texttt{NOTICE} file, if any, and not remove attribution.
|
||||
\item State plainly that the model is a packaging of Qwen3-Omni-30B-A3B by the Qwen team at
|
||||
Alibaba Cloud, and describe any modifications (e.g.\ quantization).
|
||||
\item Avoid any naming or marketing that implies the architecture or weights are original
|
||||
Zen LM research.
|
||||
\end{enumerate}
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{Architecture}
|
||||
\label{sec:arch}
|
||||
|
||||
\subsection{Unified Token Space}
|
||||
|
||||
All four modalities are projected into a shared 4096-dimensional token space before
|
||||
entering the transformer backbone:
|
||||
|
||||
\textbf{Text}: Standard BPE tokenization with a 150,528-token vocabulary. Each token
|
||||
$t_i \in \mathbb{R}^{4096}$.
|
||||
|
||||
\textbf{Image}: ViT-style patchification at 14$\times$14 pixel patches. High-resolution
|
||||
images use dynamic resolution tiling (up to 4$\times$4 tiles at 448$\times$448 base
|
||||
resolution). Each image produces 256--4096 patch tokens.
|
||||
|
||||
\textbf{Audio}: Mel spectrogram with 128 mel bins, 10ms hop, processed in 40ms chunks.
|
||||
A 3-layer CNN produces audio tokens at 40ms granularity. 30 seconds of audio yields
|
||||
750 tokens.
|
||||
|
||||
\textbf{Video}: Frame sampling at 1 fps for general understanding, 4 fps for action
|
||||
recognition. Each frame is patchified identically to images. Temporal position embeddings
|
||||
encode frame index alongside 2D spatial position.
|
||||
|
||||
The unified sequence for a multimodal input is:
|
||||
|
||||
\begin{equation}
|
||||
\mathcal{S} = [t_1, \ldots, t_T] \oplus [v_1, \ldots, v_P] \oplus [a_1, \ldots, a_A] \oplus [\text{vid}_1, \ldots, \text{vid}_F]
|
||||
\end{equation}
|
||||
|
||||
where $\oplus$ denotes concatenation with modality type embeddings prepended.
|
||||
|
||||
\subsection{Cross-Modal Attention}
|
||||
|
||||
The core innovation is a modified attention mechanism that allows unrestricted
|
||||
cross-modal attention within the same sequence. Rather than restricting vision
|
||||
tokens to attend only to vision tokens, Zen3-Omni allows each token to attend
|
||||
across the full multimodal context:
|
||||
|
||||
\begin{equation}
|
||||
\text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{QK^T}{\sqrt{d_k}} + M_{\text{causal}}\right) V
|
||||
\end{equation}
|
||||
|
||||
where $M_{\text{causal}}$ is a causal mask that prevents future text tokens from
|
||||
attending to past text tokens but permits bidirectional attention between vision,
|
||||
audio, and video tokens (which are treated as context, not generated sequence).
|
||||
|
||||
\subsection{Temporal Sparse Attention for Video}
|
||||
|
||||
Attending over 10-minute video at 1fps produces sequences of $600 \times 256 = 153{,}600$
|
||||
vision tokens, making full attention intractable. Zen3-Omni uses a hierarchical
|
||||
temporal attention pattern:
|
||||
|
||||
\begin{itemize}
|
||||
\item \textbf{Local window}: Each frame attends fully to adjacent $\pm$2 frames.
|
||||
\item \textbf{Strided global}: Every 10th frame attends to global frame summaries
|
||||
(mean-pooled representations of 10-frame segments).
|
||||
\item \textbf{Query-guided retrieval}: Text query tokens attend to all frame summary
|
||||
tokens, then retrieve top-$k$ full-resolution frames for dense attention.
|
||||
\end{itemize}
|
||||
|
||||
This reduces video attention from $O(F^2 P^2)$ to $O(F P^2 + F^2 / s)$ where $s=10$
|
||||
is the stride factor.
|
||||
|
||||
\subsection{Zen MoDE Expert Routing with Modality Balance}
|
||||
|
||||
The feed-forward layers use Zen MoDE expert routing with 64 experts, top-4 active
|
||||
per token. Third-generation routing adds a modality balance loss:
|
||||
|
||||
\begin{equation}
|
||||
\mathcal{L}_{\text{balance}} = \alpha \sum_{e=1}^{64} \left( f_e - \frac{1}{64} \right)^2 + \beta \sum_{m \in \mathcal{M}} \text{KL}(p_m^e \| \bar{p}^e)
|
||||
\end{equation}
|
||||
|
||||
where $f_e$ is the fraction of tokens routed to expert $e$, $p_m^e$ is the routing
|
||||
distribution for modality $m$, and $\bar{p}^e$ is the average. This prevents
|
||||
individual experts from specializing exclusively on high-frequency text tokens,
|
||||
maintaining expert utilization across all modalities.
|
||||
|
||||
\subsection{Streaming CTC Audio Decoder}
|
||||
|
||||
For real-time audio, Zen3-Omni maintains a parallel CTC head that produces
|
||||
transcript hypotheses at 40ms intervals. The CTC output is fed back as text tokens
|
||||
into the main transformer context, enabling the model to reason over audio content
|
||||
while it is still being received.
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{Training}
|
||||
\label{sec:training}
|
||||
|
||||
\subsection{Pre-Training Data}
|
||||
|
||||
Zen3-Omni pre-training uses a multimodal corpus of 3T text tokens and aligned
|
||||
multimodal data:
|
||||
|
||||
\begin{itemize}
|
||||
\item Text: 1.8T tokens (same pipeline as Zen3-Small base model)
|
||||
\item Image-text pairs: 1.2B pairs from web alt-text, academic figures, and curated datasets
|
||||
\item Audio-text pairs: 120M samples (speech recognition, audio captioning, music description)
|
||||
\item Video-text pairs: 85M clips from instructional video, film, and synthetic data
|
||||
\item Interleaved documents: 400M multimodal documents with mixed modalities in context
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Training Curriculum}
|
||||
|
||||
\textbf{Phase 1 --- Modality encoders}: Train image, audio, and video encoders
|
||||
independently with frozen LM backbone. Duration: 200B image tokens, 50B audio tokens,
|
||||
20B video tokens.
|
||||
|
||||
\textbf{Phase 2 --- Unified pre-training}: Jointly train all components on the full
|
||||
interleaved corpus with cross-modal attention enabled. Duration: 1.5T tokens equivalent.
|
||||
|
||||
\textbf{Phase 3 --- Instruction tuning}: Fine-tune on 80M multimodal instruction samples
|
||||
covering all four modality combinations.
|
||||
|
||||
\textbf{Phase 4 --- RLHF}: Preference optimization using a multimodal reward model
|
||||
trained on human ratings of response quality across all modality inputs.
|
||||
|
||||
\subsection{Distillation from Zen3-VL}
|
||||
|
||||
For vision-language tasks, Zen3-Omni uses Zen3-VL (72B) as a teacher, distilling
|
||||
visual understanding capabilities via logit matching on image-text pairs. This
|
||||
bridges the 10$\times$ parameter gap and brings Zen3-Omni VL performance within
|
||||
5\% of the 72B flagship.
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{Evaluation}
|
||||
\section{Evaluation: Use Upstream Numbers}
|
||||
\label{sec:eval}
|
||||
|
||||
\subsection{Vision-Language}
|
||||
This note deliberately reports \textbf{no} Zen-produced benchmark scores. Doing so would
|
||||
either duplicate or, worse, misrepresent the upstream results. The Qwen team publishes
|
||||
Qwen3-Omni evaluations on standard suites --- including text reasoning (e.g.\ MMLU-Redux,
|
||||
GPQA, AIME), automatic speech recognition (e.g.\ LibriSpeech, WenetSpeech), speech
|
||||
instruction following (e.g.\ VoiceBench), and a broad set of audio, vision, and
|
||||
audio-visual understanding benchmarks --- in the official model card and Technical
|
||||
Report~\cite{qwen3omni,qwen3omnireport}.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Vision-language benchmarks}
|
||||
\begin{tabular}{lccccc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{Params} & \textbf{MMBench} & \textbf{MMMU} & \textbf{TextVQA} & \textbf{OCRBench} \\
|
||||
\midrule
|
||||
Zen-Omni & 7B & 78.3 & 58.4 & 74.1 & 71.2 \\
|
||||
Zen2-Omni & 7B & 82.1 & 63.7 & 79.8 & 77.6 \\
|
||||
\textbf{Zen3-Omni} & \textbf{7B} & \textbf{86.2} & \textbf{68.9} & \textbf{83.4} & \textbf{82.1} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:vl}
|
||||
\end{table}
|
||||
|
||||
\subsection{Audio Understanding}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Audio benchmarks}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{AudioCaps CLAP} & \textbf{LibriSpeech WER} & \textbf{MSVD-QA} & \textbf{AIR-Bench} \\
|
||||
\midrule
|
||||
Zen-Omni & 0.481 & 3.8 & 61.3 & 71.4 \\
|
||||
Zen2-Omni & 0.502 & 3.2 & 67.9 & 76.8 \\
|
||||
\textbf{Zen3-Omni} & \textbf{0.523} & \textbf{2.7} & \textbf{73.2} & \textbf{81.9} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:audio}
|
||||
\end{table}
|
||||
|
||||
\subsection{Video Understanding}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Video understanding benchmarks}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{VideoQA} & \textbf{EgoSchema} & \textbf{MVBench} & \textbf{Video-MME} \\
|
||||
\midrule
|
||||
Zen-Omni & 68.4 & 52.1 & 71.2 & 61.3 \\
|
||||
Zen2-Omni & 73.8 & 59.4 & 76.9 & 68.7 \\
|
||||
\textbf{Zen3-Omni} & \textbf{79.4} & \textbf{66.2} & \textbf{83.1} & \textbf{74.8} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:video}
|
||||
\end{table}
|
||||
|
||||
\subsection{Cross-Modal Reasoning}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Cross-modal reasoning: tasks requiring simultaneous use of 2+ modalities}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Task} & \textbf{Zen3-Omni} & \textbf{Zen2-Omni} \\
|
||||
\midrule
|
||||
Audio-Visual QA (synchronized A+V) & 74.3 & 63.1 \\
|
||||
Talking-Head Description (A+V+Text) & 71.8 & 58.4 \\
|
||||
Chart with Narration Understanding & 81.2 & 69.7 \\
|
||||
Live Lecture Comprehension & 68.9 & 54.2 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:crossmodal}
|
||||
\end{table}
|
||||
|
||||
\subsection{Streaming Latency}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{End-to-end streaming inference latency (A100 80GB)}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Input Mode} & \textbf{Time to First Token (ms)} & \textbf{Throughput (tok/s)} \\
|
||||
\midrule
|
||||
Text only & 18 & 4200 \\
|
||||
Image + Text & 62 & 3800 \\
|
||||
Audio stream (40ms) & 43 & 3900 \\
|
||||
Video (1fps, 60s) & 210 & 3100 \\
|
||||
All modalities & 270 & 2800 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:latency}
|
||||
\end{table}
|
||||
\textbf{Guidance for downstream reporting.} Cite the upstream-reported numbers with explicit
|
||||
attribution to Qwen, and link to the model card. If Zen LM applies quantization, any
|
||||
re-measured scores should be reported as ``Qwen3-Omni-30B-A3B (Zen packaging,
|
||||
\textit{N}-bit)'' with the evaluation harness named, and should be framed as a measurement
|
||||
of the upstream model under a specific deployment configuration --- never as the result of a
|
||||
distinct model.
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{Related Work}
|
||||
\label{sec:related}
|
||||
|
||||
\subsection{Multimodal Language Models}
|
||||
|
||||
Early multimodal language models relied on frozen vision encoders with lightweight
|
||||
projection adapters connecting to a text backbone~\cite{llava}. While effective for
|
||||
image captioning and VQA, these architectures treat vision tokens as second-class
|
||||
context, limiting deep cross-modal reasoning. Zen3-Omni's unified token space builds
|
||||
on work showing that full cross-modal attention substantially improves tasks requiring
|
||||
joint reasoning.
|
||||
|
||||
\subsection{Audio-Language Models}
|
||||
|
||||
Audio integration with language models has advanced rapidly through mel spectrogram
|
||||
tokenization and learned audio codebooks. Streaming CTC integration, as used in
|
||||
Zen3-Omni, was pioneered for real-time speech translation systems and extended here
|
||||
to general audio understanding within a multimodal context.
|
||||
|
||||
\subsection{Video Understanding}
|
||||
|
||||
Long-form video understanding remains challenging due to the quadratic attention cost.
|
||||
Prior work addressed this through frame sampling, visual summarization, and hierarchical
|
||||
architectures. Zen3-Omni's temporal sparse attention synthesizes these approaches into
|
||||
a unified mechanism compatible with the Zen MoDE backbone.
|
||||
The relevant prior art is the upstream lineage itself: the Qwen-VL and Qwen2.5-Omni line of
|
||||
multimodal and omni-modal models from Alibaba Cloud, of which Qwen3-Omni is the current
|
||||
generation~\cite{qwen3omni,qwen3omnireport}. General multimodal instruction tuning with
|
||||
frozen vision encoders and projection adapters was popularized by LLaVA~\cite{llava}; the
|
||||
omni-modal, speech-generating design used here is specific to Qwen3-Omni.
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{Conclusion}
|
||||
\label{sec:conclusion}
|
||||
|
||||
Zen3-Omni demonstrates that unified cross-modal reasoning in a 7B parameter model
|
||||
can achieve state-of-the-art results across vision, audio, video, and text simultaneously.
|
||||
The third-generation Zen MoDE architecture with modality-balanced expert routing,
|
||||
temporal sparse attention, and streaming CTC decoding addresses the key challenges
|
||||
of prior generation multimodal models.
|
||||
|
||||
Future work will extend to 3D scene understanding, tactile sensor inputs, and
|
||||
sub-100ms real-time video captioning for accessibility applications.
|
||||
``Zen3-Omni'' denotes a packaged redistribution of Alibaba Cloud's Apache-2.0
|
||||
Qwen3-Omni-30B-A3B, not an independent model. The honest description of the work is:
|
||||
convert, optionally quantize, serve, and attribute. All architectural credit and all
|
||||
capability claims belong to the upstream Qwen3-Omni authors and should be cited to them.
|
||||
|
||||
\section*{Acknowledgments}
|
||||
|
||||
The authors thank Zoo Labs Foundation for multimodal dataset curation and compute
|
||||
allocation, and the Zen LM research community for benchmark contributions.
|
||||
We acknowledge the Qwen team at Alibaba Cloud as the authors of Qwen3-Omni-30B-A3B, the
|
||||
model packaged and redistributed here under the Apache-2.0 license.
|
||||
|
||||
\bibliographystyle{plain}
|
||||
\begin{thebibliography}{9}
|
||||
|
||||
\bibitem{qwen3omni}
|
||||
Qwen Team, Alibaba Cloud,
|
||||
``Qwen3-Omni-30B-A3B,'' Hugging Face model card, 2025.
|
||||
\url{https://huggingface.co/Qwen/Qwen3-Omni-30B-A3B-Instruct}
|
||||
|
||||
\bibitem{qwen3omnireport}
|
||||
Qwen Team, Alibaba Cloud,
|
||||
``Qwen3-Omni Technical Report,'' 2025.
|
||||
\url{https://github.com/QwenLM/Qwen3-Omni}
|
||||
|
||||
\bibitem{llava}
|
||||
H. Liu et al.,
|
||||
``Visual Instruction Tuning,''
|
||||
\textit{NeurIPS}, 2023.
|
||||
|
||||
\bibitem{zip007}
|
||||
Zoo Labs Foundation,
|
||||
``ZIP-007: BitDelta --- Quantization-Aware Delta Compression for Language Models,''
|
||||
\textit{Zoo Improvement Proposals}, 2024.
|
||||
|
||||
\bibitem{hip002}
|
||||
Hanzo AI,
|
||||
``HIP-002: Active Semantic Optimization (ASO),''
|
||||
\textit{Hanzo Improvement Proposals}, 2024.
|
||||
|
||||
\bibitem{zip001}
|
||||
Zoo Labs Foundation,
|
||||
``ZIP-001: Decentralized Semantic Optimization (DSO),''
|
||||
\textit{Zoo Improvement Proposals}, 2024.
|
||||
|
||||
\end{thebibliography}
|
||||
|
||||
\end{document}
|
||||
|
||||
Binary file not shown.
+132
-282
@@ -13,9 +13,9 @@
|
||||
\definecolor{zenblue}{RGB}{41,121,255}
|
||||
\hypersetup{colorlinks=true,linkcolor=zenblue,urlcolor=zenblue,citecolor=zenblue}
|
||||
|
||||
\title{\textbf{Zen3-VL: Third-Generation Flagship Vision-Language Model}\\
|
||||
\large Technical Whitepaper v2025.06}
|
||||
\author{Antje Worring, Zach Kelling \\ Zen LM Research Team\\
|
||||
\title{\textbf{Zen3-VL: A Packaging of Qwen3-VL-30B-A3B}\\
|
||||
\large Technical Note v2025.06}
|
||||
\author{Zen LM Research Team\\
|
||||
\texttt{research@zenlm.org}\\
|
||||
\href{https://papers.zenlm.org}{papers.zenlm.org}}
|
||||
\date{June 2025}
|
||||
@@ -24,334 +24,194 @@
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
Zen3-VL is our flagship third-generation vision-language model at 72B parameters,
|
||||
delivering frontier-level visual understanding across document analysis, scientific
|
||||
reasoning, and complex visual question answering, with particular strength in structured
|
||||
data interpretation and fine-grained image understanding. Built on the Zen MoDE
|
||||
(Mixture of Distilled Experts) architecture with a dedicated 6B-parameter high-resolution
|
||||
vision encoder, Zen3-VL achieves MMMU 72.1\%, MMBench 91.3\%, DocVQA 94.8\%,
|
||||
ChartQA 89.2\%, and TextVQA 85.4\% --- establishing new state-of-the-art results
|
||||
across all five primary vision-language benchmarks simultaneously. The model features
|
||||
dynamic ultra-high-resolution input (up to 4K), scientific figure parsing with
|
||||
structured output extraction, and a specialized chart-to-data pipeline that converts
|
||||
visual data representations into structured tables with high fidelity.
|
||||
Zen3-VL is \emph{not} a from-scratch model. It is a redistribution and packaging of
|
||||
\textbf{Qwen3-VL-30B-A3B}, the Mixture-of-Experts vision-language model developed by the
|
||||
Qwen team at Alibaba Cloud and released under the Apache-2.0 license~\cite{qwen3vl}.
|
||||
The upstream model uses the \texttt{Qwen3VLMoeForConditionalGeneration} architecture and is
|
||||
a roughly 31B-total-parameter MoE with about 3.3B activated parameters per token --- not the
|
||||
72B dense flagship described in earlier internal drafts. This note documents the upstream
|
||||
architecture as published by its authors, records the provenance and license obligations,
|
||||
and describes the thin packaging layer (weight conversion, quantization, serving
|
||||
configuration) Zen LM applies on top of the upstream checkpoint. No architectural novelty,
|
||||
no separate pre-training run, and no independently produced benchmark results are claimed.
|
||||
For authoritative capability numbers, see the upstream model card and the Qwen3-VL
|
||||
Technical Report~\cite{qwen3vl,qwen3vlreport}.
|
||||
\end{abstract}
|
||||
|
||||
\tableofcontents
|
||||
\newpage
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{Introduction}
|
||||
\section{Provenance and Scope}
|
||||
\label{sec:intro}
|
||||
|
||||
Vision-language models face a fundamental tension: general visual understanding requires
|
||||
broad training across diverse image types, yet specialized capabilities --- reading
|
||||
dense documents, parsing scientific figures, interpreting charts --- demand domain-specific
|
||||
architectural components and training data. Prior approaches either sacrifice specialization
|
||||
for generality or maintain separate model checkpoints for different visual domains.
|
||||
Earlier internal drafts of this document described a homegrown ``Zen MoDE (Mixture of
|
||||
Distilled Experts)'' backbone at 72B parameters, a bespoke document-specialized vision
|
||||
encoder, a ``scientific figure parser,'' and a table of benchmark scores presented as
|
||||
in-house results. That framing was inaccurate and has been removed. The artifact
|
||||
distributed as ``Zen3-VL'' is the Qwen3-VL-30B-A3B checkpoint published by Alibaba Cloud's
|
||||
Qwen team~\cite{qwen3vl}. This note exists to attribute the work correctly and to make the
|
||||
redistribution terms explicit.
|
||||
|
||||
Zen3-VL resolves this tension through a unified 72B parameter model with a shared
|
||||
Zen MoDE backbone that routes visual tokens through specialized expert clusters
|
||||
without requiring separate models per task. The key design choices are:
|
||||
\textbf{Upstream model.} Qwen3-VL-30B-A3B is a Mixture-of-Experts vision-language model. Per
|
||||
the model card, it has approximately 31B total parameters with about 3.3B activated per
|
||||
token, uses the \texttt{Qwen3VLMoeForConditionalGeneration} class, and supports a native
|
||||
256K-token context expandable to 1M tokens~\cite{qwen3vl}. It is published in both Instruct
|
||||
and reasoning-oriented ``Thinking'' editions.
|
||||
|
||||
\begin{enumerate}
|
||||
\item \textbf{High-resolution dynamic tiling} --- Input images up to 4096$\times$4096
|
||||
pixels are processed via adaptive tile decomposition, preserving fine-grained
|
||||
detail in dense text and chart elements without fixed resolution downsampling.
|
||||
\item \textbf{Document-specialized vision encoder} --- A 6B ViT-H encoder pre-trained
|
||||
on 2B document images provides representations tuned for text layout, table
|
||||
structure, and mathematical notation.
|
||||
\item \textbf{Scientific figure parser} --- A dedicated parsing head extracts
|
||||
structured data (axis labels, data series, legends) from scientific figures,
|
||||
enabling accurate chart-to-table conversion.
|
||||
\item \textbf{72B Zen MoDE backbone} --- The language backbone uses 128 experts with
|
||||
top-6 routing, providing 660B effective parameter capacity for complex
|
||||
multi-step visual reasoning.
|
||||
\end{enumerate}
|
||||
\textbf{License.} The upstream weights are released under \textbf{Apache-2.0}. This is
|
||||
permissive and allows commercial use and redistribution, but a redistributor must preserve
|
||||
copyright and license notices, include the upstream \texttt{NOTICE} file if present, and
|
||||
state any modifications. Any Zen LM redistribution must ship the upstream \texttt{LICENSE}
|
||||
and attribution intact and must not misrepresent the model's origin.
|
||||
|
||||
\textbf{Correction of parameter scale.} The ``72B'' figure and the ``128 experts, top-6,
|
||||
660B effective parameters'' configuration in earlier drafts were fabricated and do not
|
||||
describe this model. The upstream Qwen3-VL-30B-A3B is a $\sim$31B-total MoE with
|
||||
$\sim$3.3B activated parameters. The precise per-layer expert count and routing top-$k$ are
|
||||
defined by the upstream configuration files; this note does not restate or invent them.
|
||||
|
||||
\textbf{What ``packaging'' means here.} Zen LM does not retrain or re-architect the model.
|
||||
The packaging layer is limited to weight-format conversion, optional post-training
|
||||
quantization, and serving configuration. These steps do not change the model's identity or
|
||||
its measured capabilities.
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{Architecture}
|
||||
\section{Upstream Architecture (as published by Qwen)}
|
||||
\label{sec:arch}
|
||||
|
||||
\subsection{High-Resolution Vision Encoder}
|
||||
The description below summarizes the architecture as documented by the upstream authors in
|
||||
the Qwen3-VL model card and Technical Report~\cite{qwen3vl,qwen3vlreport}. It is reproduced
|
||||
for convenience and is \emph{not} a Zen LM contribution.
|
||||
|
||||
The vision encoder is a ViT-H variant with 6.5B parameters, trained from scratch
|
||||
on a document-rich image corpus. Key architectural choices:
|
||||
\subsection{Three-Module Structure}
|
||||
|
||||
Following the Qwen2.5-VL design, Qwen3-VL adopts a three-module structure:
|
||||
|
||||
\begin{itemize}
|
||||
\item 16$\times$16 pixel patch size (finer than standard 32$\times$32)
|
||||
\item 32 transformer layers, 48 attention heads, 2048-dimensional hidden state
|
||||
\item Flash Attention 3 for memory-efficient processing of long patch sequences
|
||||
\item Learnable class tokens for global image summary alongside local patch tokens
|
||||
\item \textbf{Vision encoder} --- a ViT-based encoder that processes images and video frames
|
||||
at high resolution.
|
||||
\item \textbf{Vision--language merger} --- an MLP-based module that projects visual features
|
||||
into the language model's embedding space.
|
||||
\item \textbf{LLM backbone} --- the Mixture-of-Experts language model that performs
|
||||
multimodal reasoning and text generation.
|
||||
\end{itemize}
|
||||
|
||||
For a 4096$\times$4096 input image, the encoder produces $256 \times 256 = 65{,}536$
|
||||
patch tokens. These are compressed to a manageable sequence via a hierarchical
|
||||
cross-attention pooling layer that produces 2048 summarized visual tokens while
|
||||
preserving local detail via a complementary sparse attention bypass.
|
||||
\subsection{Named Architectural Features}
|
||||
|
||||
\subsection{Dynamic Resolution Tiling}
|
||||
|
||||
Input images are adaptively tiled based on aspect ratio and content complexity:
|
||||
|
||||
\begin{equation}
|
||||
N_{\text{tiles}} = \left\lceil \frac{W}{W_{\text{base}}} \right\rceil \times \left\lceil \frac{H}{H_{\text{base}}} \right\rceil, \quad W_{\text{base}} = H_{\text{base}} = 448
|
||||
\end{equation}
|
||||
|
||||
Each tile is processed independently by the vision encoder and the resulting tile
|
||||
tokens are arranged in a 2D grid with position embeddings encoding tile row, column,
|
||||
and intra-tile patch position. A cross-tile attention layer fuses information across
|
||||
tile boundaries, critical for reading text that spans tile boundaries.
|
||||
|
||||
Maximum supported configurations:
|
||||
\begin{itemize}
|
||||
\item Standard: 1--4 tiles (up to 896$\times$896 effective resolution)
|
||||
\item High-resolution: 4--16 tiles (up to 1792$\times$1792)
|
||||
\item Ultra-high-resolution: 16--64 tiles (up to 3584$\times$3584)
|
||||
\item Document mode: up to 128 tiles for A0-format poster parsing
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Zen MoDE Backbone at 72B}
|
||||
|
||||
The language backbone uses a 72B Zen MoDE configuration:
|
||||
The upstream authors highlight several features of Qwen3-VL~\cite{qwen3vl}:
|
||||
|
||||
\begin{itemize}
|
||||
\item 80 transformer layers, 8192-dimensional hidden state
|
||||
\item 64 query heads, 8 key-value heads (GQA ratio 8:1)
|
||||
\item 128 experts, top-6 active per token (660B total parameter capacity)
|
||||
\item 131,072 token context window (native, no interpolation required)
|
||||
\item RoPE with base frequency $\theta = 1{,}000{,}000$ for long-context stability
|
||||
\item \textbf{Interleaved-MRoPE} --- multimodal rotary position embeddings with
|
||||
full-frequency allocation over the time, width, and height axes, intended to improve
|
||||
long-horizon video reasoning.
|
||||
\item \textbf{DeepStack} --- fusion of multi-level ViT features to capture fine-grained
|
||||
visual detail and sharpen image--text alignment.
|
||||
\item \textbf{Text--Timestamp Alignment} --- a video temporal-modeling mechanism that moves
|
||||
beyond T-RoPE toward timestamp-grounded event localization.
|
||||
\end{itemize}
|
||||
|
||||
The MoDE routing for VL tasks shows strong specialization: experts 1--24 handle
|
||||
primarily text tokens, experts 25--64 handle image-text alignment, experts 65--96
|
||||
specialize in structured data reasoning (tables, charts), and experts 97--128
|
||||
handle spatial reasoning and fine-grained visual question answering.
|
||||
These are upstream Qwen3-VL contributions and are documented by their
|
||||
authors~\cite{qwen3vl,qwen3vlreport}.
|
||||
|
||||
\subsection{Scientific Figure Parser}
|
||||
\subsection{Context Length and Resolution}
|
||||
|
||||
A dedicated parsing head sits alongside the main language modeling head:
|
||||
Per the model card, the native context window is 256K tokens, expandable to 1M tokens, and
|
||||
the vision stack supports high-resolution image input with adaptive handling of varying
|
||||
aspect ratios~\cite{qwen3vl}. Exact resolution and tiling parameters are defined upstream.
|
||||
|
||||
\begin{equation}
|
||||
\hat{D} = f_{\text{parse}}(\mathbf{v}_{\text{img}}, \mathbf{h}_{\text{backbone}})
|
||||
\end{equation}
|
||||
\subsection{Items Removed From Earlier Drafts}
|
||||
|
||||
where $\mathbf{v}_{\text{img}}$ is the pooled vision representation and
|
||||
$\mathbf{h}_{\text{backbone}}$ is the last-layer hidden state. The parser
|
||||
outputs a structured JSON document conforming to a schema covering:
|
||||
bar charts, line charts, scatter plots, pie charts, heatmaps, and box plots.
|
||||
|
||||
This structured output is then optionally rendered back to a human-readable
|
||||
description or a Markdown table by the main language head, enabling downstream
|
||||
programmatic processing.
|
||||
The following claims appeared in prior drafts and are \textbf{withdrawn} because they were
|
||||
not substantiated and were presented as in-house work: a 72B ``Zen MoDE'' backbone; a
|
||||
specific expert-cluster specialization map (``experts 1--24 handle text,'' etc.); a
|
||||
6.5B-parameter from-scratch document vision encoder with stated training scale; a
|
||||
``scientific figure parser'' head with a per-chart-type extraction-accuracy table; a
|
||||
``chart-to-data synthetic pipeline'' producing 100M samples; and all benchmark tables
|
||||
comparing a fictional ``Zen-VL / Zen2-VL / Zen3-VL'' lineage. Any genuine capabilities in
|
||||
these areas belong to the upstream Qwen3-VL model.
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{Training}
|
||||
\label{sec:training}
|
||||
\section{Packaging and Redistribution}
|
||||
\label{sec:packaging}
|
||||
|
||||
\subsection{Vision Encoder Pre-Training}
|
||||
\subsection{Conversion and Quantization}
|
||||
|
||||
The vision encoder is pre-trained in two phases:
|
||||
Zen LM converts the published Qwen3-VL-30B-A3B weights to the internal serving format and may
|
||||
apply post-training quantization for deployment. These transforms are mechanical; they do
|
||||
not constitute a new training run and do not produce a new model. Where quantization is
|
||||
applied, the scheme and bit-width should be recorded alongside the artifact so downstream
|
||||
users understand any quality trade-off relative to the upstream checkpoint.
|
||||
|
||||
\textbf{Phase 1 --- Contrastive pre-training}: CLIP-style contrastive training on
|
||||
2B image-text pairs from web data, academic papers, and document repositories.
|
||||
Batch size 65,536, InfoNCE loss with temperature 0.07.
|
||||
\subsection{Attribution Requirements}
|
||||
|
||||
\textbf{Phase 2 --- Document-domain fine-tuning}: Masked image modeling on 500M
|
||||
document images (PDFs, scanned books, charts, scientific figures) using a DINO-v2
|
||||
objective. This instills strong layout and text-aware representations.
|
||||
|
||||
\subsection{Backbone Pre-Training}
|
||||
|
||||
The Zen MoDE 72B backbone is pre-trained on 15T tokens of text before vision
|
||||
integration. This includes the full Zen family base pre-training corpus plus:
|
||||
|
||||
\begin{itemize}
|
||||
\item Scientific papers: 180B tokens from arXiv, Semantic Scholar, PubMed
|
||||
\item Technical documents: 120B tokens from patents, standards, manuals
|
||||
\item Structured data: 80B tokens from tables, spreadsheets, databases
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Multimodal Training}
|
||||
|
||||
Vision-language integration training proceeds in three stages:
|
||||
|
||||
\textbf{Stage 1 --- Connector training}: Train only the vision-to-language projection
|
||||
layer (2-layer MLP) with frozen encoder and backbone. Data: 500M image-text pairs.
|
||||
|
||||
\textbf{Stage 2 --- Full VL pre-training}: Unfreeze all components and train jointly
|
||||
on 4B multimodal samples including document understanding, chart QA, scientific
|
||||
figure understanding, and general VQA.
|
||||
|
||||
\textbf{Stage 3 --- Instruction tuning}: Fine-tune on 200M multimodal instruction
|
||||
samples with chain-of-thought reasoning traces. Includes:
|
||||
\begin{itemize}
|
||||
\item Document question answering with page-level context
|
||||
\item Chart and table extraction with structured output
|
||||
\item Scientific figure interpretation with mathematical notation
|
||||
\item Multi-image reasoning requiring cross-image comparison
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Chart-to-Data Synthetic Data Pipeline}
|
||||
|
||||
A significant innovation is a synthetic data pipeline that generates 100M training
|
||||
samples from known chart data:
|
||||
Because the upstream license is Apache-2.0, every redistributed Zen3-VL artifact must:
|
||||
|
||||
\begin{enumerate}
|
||||
\item Sample a data table from a structured database (financial, scientific, census)
|
||||
\item Render the table as a chart using matplotlib/plotly with randomized styling
|
||||
\item Generate question-answer pairs about the chart's content
|
||||
\item Train the model to reconstruct the original data table from the chart image
|
||||
\item Retain the upstream copyright and Apache-2.0 \texttt{LICENSE} text.
|
||||
\item Include the upstream \texttt{NOTICE} file, if any, and not remove attribution.
|
||||
\item State plainly that the model is a packaging of Qwen3-VL-30B-A3B by the Qwen team at
|
||||
Alibaba Cloud, and describe any modifications (e.g.\ quantization).
|
||||
\item Avoid any naming or marketing that implies the architecture or weights are original
|
||||
Zen LM research.
|
||||
\end{enumerate}
|
||||
|
||||
This creates a closed-loop training signal where ground truth is always available,
|
||||
enabling precise supervision for chart parsing tasks.
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{Evaluation}
|
||||
\section{Evaluation: Use Upstream Numbers}
|
||||
\label{sec:eval}
|
||||
|
||||
\subsection{General Vision-Language}
|
||||
This note deliberately reports \textbf{no} Zen-produced benchmark scores. The Qwen team
|
||||
publishes Qwen3-VL evaluations on standard vision-language suites --- including general
|
||||
multimodal reasoning (e.g.\ MMMU, MMBench), document and OCR understanding (e.g.\ DocVQA,
|
||||
OCRBench, InfoVQA), chart and structured-data tasks (e.g.\ ChartQA), and mathematical visual
|
||||
reasoning (e.g.\ MathVista) --- in the official model card and Technical
|
||||
Report~\cite{qwen3vl,qwen3vlreport}.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{General vision-language benchmarks}
|
||||
\begin{tabular}{lccccc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{Params} & \textbf{MMMU} & \textbf{MMBench} & \textbf{SeedBench} & \textbf{MathVista} \\
|
||||
\midrule
|
||||
Zen-VL & 72B & 62.3 & 85.4 & 77.8 & 58.1 \\
|
||||
Zen2-VL & 72B & 67.8 & 88.9 & 82.1 & 65.4 \\
|
||||
\textbf{Zen3-VL} & \textbf{72B} & \textbf{72.1} & \textbf{91.3} & \textbf{86.7} & \textbf{71.8} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:general_vl}
|
||||
\end{table}
|
||||
|
||||
\subsection{Document and OCR Understanding}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Document understanding and OCR benchmarks}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{DocVQA} & \textbf{TextVQA} & \textbf{OCRBench} & \textbf{InfoVQA} \\
|
||||
\midrule
|
||||
Zen-VL & 89.3 & 79.1 & 75.3 & 71.4 \\
|
||||
Zen2-VL & 92.1 & 82.8 & 79.6 & 76.2 \\
|
||||
\textbf{Zen3-VL} & \textbf{94.8} & \textbf{85.4} & \textbf{82.1} & \textbf{80.7} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:doc}
|
||||
\end{table}
|
||||
|
||||
\subsection{Chart and Structured Data}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Chart and structured data understanding benchmarks}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{ChartQA} & \textbf{TableVQA} & \textbf{PlotQA} & \textbf{FigureQA} \\
|
||||
\midrule
|
||||
Zen-VL & 82.1 & 71.3 & 84.2 & 87.4 \\
|
||||
Zen2-VL & 85.8 & 76.9 & 87.6 & 90.1 \\
|
||||
\textbf{Zen3-VL} & \textbf{89.2} & \textbf{82.4} & \textbf{91.3} & \textbf{93.6} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:chart}
|
||||
\end{table}
|
||||
|
||||
\subsection{Scientific Figure Understanding}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Scientific figure parsing accuracy (chart-to-table extraction)}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
\textbf{Chart Type} & \textbf{Extraction F1} & \textbf{Numeric Accuracy} & \textbf{Label Accuracy} \\
|
||||
\midrule
|
||||
Bar chart & 94.2 & 91.8 & 97.3 \\
|
||||
Line chart & 91.7 & 88.4 & 96.1 \\
|
||||
Scatter plot & 87.3 & 84.9 & 93.8 \\
|
||||
Pie chart & 92.8 & 90.3 & 96.7 \\
|
||||
Heatmap & 83.4 & 79.1 & 91.2 \\
|
||||
Box plot & 85.6 & 82.4 & 94.1 \\
|
||||
Average & 89.2 & 86.2 & 94.9 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:scientific}
|
||||
\end{table}
|
||||
|
||||
\subsection{High-Resolution Evaluation}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Performance vs. input resolution (MMBench, high-resolution subset)}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Resolution} & \textbf{Tiles} & \textbf{Score} & \textbf{Improvement vs. 448px} \\
|
||||
\midrule
|
||||
448$\times$448 (1 tile) & 1 & 87.4 & --- \\
|
||||
896$\times$896 (4 tiles) & 4 & 89.6 & +2.2 \\
|
||||
1792$\times$1792 (16 tiles) & 16 & 91.1 & +3.7 \\
|
||||
3584$\times$3584 (64 tiles) & 64 & 91.3 & +3.9 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\label{tab:resolution}
|
||||
\end{table}
|
||||
\textbf{Guidance for downstream reporting.} Cite the upstream-reported numbers with explicit
|
||||
attribution to Qwen, and link to the model card. If Zen LM applies quantization, any
|
||||
re-measured scores should be reported as ``Qwen3-VL-30B-A3B (Zen packaging,
|
||||
\textit{N}-bit)'' with the evaluation harness named, framed as a measurement of the
|
||||
upstream model under a deployment configuration --- never as the result of a distinct model.
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{Related Work}
|
||||
\label{sec:related}
|
||||
|
||||
\subsection{Vision-Language Pre-Training}
|
||||
|
||||
The contrastive vision-language pre-training paradigm established the foundation
|
||||
for aligning visual and textual representations~\cite{clip}. Subsequent work scaled
|
||||
both the vision encoder and the language backbone, demonstrating consistent
|
||||
capability improvements across VQA, image captioning, and visual reasoning.
|
||||
|
||||
\subsection{Document Understanding}
|
||||
|
||||
Document AI has evolved from OCR-focused pipelines to end-to-end transformer
|
||||
architectures that jointly understand text, layout, and visual elements~\cite{layoutlm}.
|
||||
Zen3-VL extends this by integrating document understanding capabilities into a
|
||||
general-purpose VL model rather than maintaining separate document-specific checkpoints.
|
||||
|
||||
\subsection{Chart Understanding}
|
||||
|
||||
Chart question answering has been benchmarked extensively since ChartQA~\cite{chartqa},
|
||||
with models progressing from template matching to learned visual reasoning. The
|
||||
Zen3-VL chart-to-data synthetic pipeline scales training data generation beyond what
|
||||
manual annotation can provide, enabling high-accuracy structured extraction.
|
||||
The relevant prior art is the upstream lineage itself: the Qwen-VL and Qwen2.5-VL line of
|
||||
vision-language models from Alibaba Cloud, of which Qwen3-VL is the current
|
||||
generation~\cite{qwen3vl,qwen3vlreport}. The contrastive vision-language pre-training
|
||||
paradigm~\cite{clip} and end-to-end document understanding~\cite{layoutlm} are foundational
|
||||
to the broader field; the specific Interleaved-MRoPE, DeepStack, and Text--Timestamp
|
||||
Alignment mechanisms used here are Qwen3-VL contributions.
|
||||
|
||||
%% -----------------------------------------------------------------------
|
||||
\section{Conclusion}
|
||||
\label{sec:conclusion}
|
||||
|
||||
Zen3-VL establishes new state-of-the-art results across all five primary
|
||||
vision-language benchmarks (MMMU 72.1\%, MMBench 91.3\%, DocVQA 94.8\%,
|
||||
ChartQA 89.2\%, TextVQA 85.4\%) through a combination of the 72B Zen MoDE backbone,
|
||||
high-resolution dynamic tiling, and a document-specialized vision encoder.
|
||||
|
||||
The scientific figure parser and chart-to-data synthetic pipeline represent
|
||||
particularly novel contributions, enabling structured data extraction from scientific
|
||||
literature at scale --- a capability with significant downstream value for
|
||||
research automation and knowledge graph construction.
|
||||
|
||||
Future work will target video document understanding (slide decks, lecture recordings)
|
||||
and extend the structured output schema to cover 3D plots, network diagrams, and
|
||||
biological pathway maps.
|
||||
``Zen3-VL'' denotes a packaged redistribution of Alibaba Cloud's Apache-2.0
|
||||
Qwen3-VL-30B-A3B (an $\sim$31B-total / $\sim$3.3B-active MoE), not an independent 72B model.
|
||||
The honest description of the work is: convert, optionally quantize, serve, and attribute.
|
||||
All architectural credit and all capability claims belong to the upstream Qwen3-VL authors
|
||||
and should be cited to them.
|
||||
|
||||
\section*{Acknowledgments}
|
||||
|
||||
The authors thank Zoo Labs Foundation for benchmark curation, scientific figure
|
||||
dataset annotation, and compute allocation for the high-resolution encoder training.
|
||||
We acknowledge the Qwen team at Alibaba Cloud as the authors of Qwen3-VL-30B-A3B, the model
|
||||
packaged and redistributed here under the Apache-2.0 license.
|
||||
|
||||
\bibliographystyle{plain}
|
||||
\begin{thebibliography}{9}
|
||||
|
||||
\bibitem{qwen3vl}
|
||||
Qwen Team, Alibaba Cloud,
|
||||
``Qwen3-VL-30B-A3B-Instruct,'' Hugging Face model card, 2025.
|
||||
\url{https://huggingface.co/Qwen/Qwen3-VL-30B-A3B-Instruct}
|
||||
|
||||
\bibitem{qwen3vlreport}
|
||||
Qwen Team, Alibaba Cloud,
|
||||
``Qwen3-VL Technical Report,'' arXiv:2511.21631, 2025.
|
||||
\url{https://arxiv.org/abs/2511.21631}
|
||||
|
||||
\bibitem{clip}
|
||||
A. Radford et al.,
|
||||
``Learning Transferable Visual Models From Natural Language Supervision,''
|
||||
@@ -362,16 +222,6 @@ Y. Xu et al.,
|
||||
``LayoutLM: Pre-Training of Text and Layout for Document Image Understanding,''
|
||||
\textit{KDD}, 2020.
|
||||
|
||||
\bibitem{chartqa}
|
||||
A. Masry et al.,
|
||||
``ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning,''
|
||||
\textit{ACL Findings}, 2022.
|
||||
|
||||
\bibitem{zip007}
|
||||
Zoo Labs Foundation,
|
||||
``ZIP-007: BitDelta --- Quantization-Aware Delta Compression for Language Models,''
|
||||
\textit{Zoo Improvement Proposals}, 2024.
|
||||
|
||||
\end{thebibliography}
|
||||
|
||||
\end{document}
|
||||
|
||||
Binary file not shown.
@@ -1,335 +0,0 @@
|
||||
\documentclass[11pt,a4paper]{article}
|
||||
\usepackage[utf8]{inputenc}
|
||||
\usepackage[T1]{fontenc}
|
||||
\usepackage{amsmath,amsfonts,amssymb}
|
||||
\usepackage{graphicx}
|
||||
\usepackage{hyperref}
|
||||
\usepackage{listings}
|
||||
\usepackage{color}
|
||||
\usepackage{booktabs}
|
||||
\usepackage{float}
|
||||
\usepackage{geometry}
|
||||
\geometry{margin=1in}
|
||||
\definecolor{zenblue}{RGB}{41,121,255}
|
||||
\hypersetup{colorlinks=true,linkcolor=zenblue,urlcolor=zenblue,citecolor=zenblue}
|
||||
|
||||
\definecolor{codegray}{rgb}{0.95,0.95,0.95}
|
||||
\lstset{
|
||||
backgroundcolor=\color{codegray},
|
||||
basicstyle=\ttfamily\small,
|
||||
breaklines=true,
|
||||
frame=single
|
||||
}
|
||||
|
||||
\title{\textbf{Zen4-Coder-Flash: Real-Time Code Intelligence\\
|
||||
for IDE Environments}\\[0.5em]
|
||||
\large Technical Whitepaper v2026.01}
|
||||
\author{Zach Kelling \\ Zen LM Research Team\\
|
||||
\texttt{research@zenlm.org}\\
|
||||
\href{https://papers.zenlm.org}{papers.zenlm.org}}
|
||||
\date{January 2026}
|
||||
|
||||
\begin{document}
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
Zen4-Coder-Flash is a distilled 8B parameter code intelligence model optimized for real-time IDE deployment, achieving sub-30ms P95 latency for inline code completion while maintaining strong coding accuracy. Distilled from Zen4-Coder (32B) using progressive knowledge distillation and Mixture of Experts (MoE) architecture compression, Zen4-Coder-Flash retains 88.5\% of Zen4-Coder's HumanEval performance (84.3\%) and 79.1\% MBPP accuracy at 1200 tokens per second throughput. Speculative decoding with a 120M parameter draft model reduces P95 first-token latency to 28ms, enabling responsive autocomplete and inline suggestion experiences that match or exceed prior dedicated completion models while generalizing across 92 programming languages.
|
||||
\end{abstract}
|
||||
|
||||
\tableofcontents
|
||||
\newpage
|
||||
|
||||
\section{Introduction}
|
||||
|
||||
IDE-integrated code completion places qualitatively different constraints on AI models than batch code generation. Where batch workflows tolerate multi-second latencies, IDE users expect responses within the same timeframe as keystroke feedback: tens of milliseconds. Where batch workflows permit large context windows, IDE completion must integrate with a lightweight editor extension that cannot afford the memory footprint of a 32B model inference server.
|
||||
|
||||
Prior approaches to this tradeoff either deployed small general-purpose models (low accuracy) or used expensive API calls to large remote models (high latency). Zen4-Coder-Flash resolves this tension through a purpose-built 8B model that combines three techniques:
|
||||
|
||||
\begin{enumerate}
|
||||
\item \textbf{Progressive Knowledge Distillation}: Token-level distillation from Zen4-Coder (32B) preserves the behavioral distribution of the teacher while drastically reducing parameter count.
|
||||
\item \textbf{Speculative Decoding}: A 120M draft model generates candidate token sequences that the 8B verifier accepts or rejects in parallel, reducing effective per-token latency by 2.8$\times$.
|
||||
\item \textbf{Architecture Optimization}: The MoE expert pool is compressed from 64 to 16 experts with larger individual capacity, reducing routing overhead and improving cache efficiency.
|
||||
\end{enumerate}
|
||||
|
||||
\subsection{Model Overview}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Zen4-Coder-Flash Model Specification}
|
||||
\begin{tabular}{ll}
|
||||
\toprule
|
||||
\textbf{Parameter} & \textbf{Value} \\
|
||||
\midrule
|
||||
Architecture & MoE (Compressed) \\
|
||||
Total Parameters & 8B \\
|
||||
Draft Model Parameters & 120M \\
|
||||
Context Window & 32K tokens \\
|
||||
Supported Languages & 92 programming languages \\
|
||||
P95 First-Token Latency & 28ms \\
|
||||
Throughput & 1,200 tok/s \\
|
||||
Version & v2026.01 \\
|
||||
Release Date & January 2026 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\section{Architecture}
|
||||
|
||||
\subsection{Compressed MoE}
|
||||
|
||||
The full MoE architecture in Zen4-Coder uses 64 experts with 4 active per token. For real-time deployment, this introduces routing overhead that is unacceptable at sub-30ms latency targets. Zen4-Coder-Flash uses a compressed expert configuration:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Expert Configuration Comparison}
|
||||
\begin{tabular}{lccccc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{Total Experts} & \textbf{Active ($k$)} & \textbf{Expert Dim} & \textbf{Router Overhead} \\
|
||||
\midrule
|
||||
Zen4-Coder (32B) & 64 & 4 & 2,048 & 1.8ms \\
|
||||
Zen4-Coder-Flash (8B) & 16 & 2 & 4,096 & 0.4ms \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
Individual experts in Zen4-Coder-Flash are wider (4,096 vs. 2,048 dimensional) to compensate for the reduced count, maintaining aggregate representational capacity while reducing routing computation. The routing mechanism is simplified to a single softmax over 16 logits, compared to the hierarchical two-level routing used in Zen4-Coder-Pro.
|
||||
|
||||
\subsection{Speculative Decoding}
|
||||
|
||||
Speculative decoding \cite{speculative} uses a small draft model to propose $\gamma$ tokens in parallel, which the main model verifies in a single forward pass. The effective speedup is:
|
||||
|
||||
\begin{equation}
|
||||
S = \frac{\gamma + 1}{1 + \gamma(1 - \beta)}
|
||||
\end{equation}
|
||||
|
||||
where $\beta$ is the acceptance rate (fraction of draft tokens accepted by the verifier). With the 120M draft model trained specifically on code distribution, Zen4-Coder-Flash achieves $\beta = 0.81$ and $\gamma = 8$, yielding $S = 2.8\times$ speedup over autoregressive decoding.
|
||||
|
||||
The draft model architecture is a lightweight 12-layer decoder-only transformer with:
|
||||
\begin{itemize}
|
||||
\item Hidden dimension: 768
|
||||
\item Attention heads: 12
|
||||
\item No mixture-of-experts (dense feedforward)
|
||||
\item Vocabulary shared with main model (100K tokens)
|
||||
\item Grouped query attention (4 key-value heads)
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Attention Optimizations}
|
||||
|
||||
Zen4-Coder-Flash uses several attention optimizations to reduce per-token cost:
|
||||
|
||||
\begin{enumerate}
|
||||
\item \textbf{Multi-Query Attention (MQA)}: Single key and value head shared across all query heads, reducing KV cache memory by $8\times$ compared to multi-head attention.
|
||||
\item \textbf{Flash Attention 3}: Fused CUDA kernel for attention computation, reducing memory bandwidth requirements and enabling longer effective context at lower latency.
|
||||
\item \textbf{Prefix Caching}: Static code context (imports, class definitions, boilerplate) is cached between completion requests, avoiding redundant computation for the stable prefix that dominates most IDE contexts.
|
||||
\end{enumerate}
|
||||
|
||||
\subsection{Quantization}
|
||||
|
||||
Zen4-Coder-Flash is deployed in FP8 quantization by default with GPTQ-style weight grouping (group size 128). Accuracy degradation from FP8 quantization is less than 0.4\% on HumanEval, while reducing model memory footprint from 16GB (BF16) to 8GB (FP8), enabling deployment on a single A100 40GB or consumer H100 NVL.
|
||||
|
||||
\section{Knowledge Distillation}
|
||||
|
||||
\subsection{Progressive Distillation Protocol}
|
||||
|
||||
Zen4-Coder-Flash is produced through a four-stage progressive distillation protocol:
|
||||
|
||||
\textbf{Stage 1 -- Vocabulary Alignment:} The student is initialized with a subset of Zen4-Coder's embedding matrix (shared tokenizer) and trained to match the teacher's token log-probabilities on a held-out code corpus:
|
||||
|
||||
\begin{equation}
|
||||
\mathcal{L}_{\text{KD}} = -\sum_t \sum_v p_T(v | x_{<t}) \log p_S(v | x_{<t})
|
||||
\end{equation}
|
||||
|
||||
where $p_T$ and $p_S$ are the teacher and student token distributions.
|
||||
|
||||
\textbf{Stage 2 -- Layer-Wise Representation Alignment:} For each student layer $l_s$, a corresponding teacher layer $l_t$ is selected. A projection head $W_{\text{proj}}$ maps student hidden states to the teacher's dimension for MSE alignment:
|
||||
|
||||
\begin{equation}
|
||||
\mathcal{L}_{\text{rep}} = \sum_{l} \left\| h_S^{l_s} W_{\text{proj}} - h_T^{l_t} \right\|_2^2
|
||||
\end{equation}
|
||||
|
||||
\textbf{Stage 3 -- Task-Specific Fine-Tuning:} The student is fine-tuned on code generation, completion, and inline suggestion tasks using the same RLCE objective as Zen4-Coder, with the teacher used as an additional reward signal.
|
||||
|
||||
\textbf{Stage 4 -- Latency-Aware Calibration:} Final fine-tuning on a latency-calibrated dataset: short, high-value completions (1--50 tokens) drawn from real IDE telemetry, weighted to optimize the distribution of outputs most commonly needed in autocomplete scenarios.
|
||||
|
||||
\subsection{Distillation Results}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Distillation Quality vs. Teacher Model}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Stage} & \textbf{HumanEval} & \textbf{MBPP} \\
|
||||
\midrule
|
||||
Scratch (8B, no distillation) & 71.3\% & 64.8\% \\
|
||||
After Stage 1 (KD only) & 76.8\% & 69.2\% \\
|
||||
After Stage 2 (+ rep alignment) & 80.1\% & 73.7\% \\
|
||||
After Stage 3 (+ task fine-tune) & 83.4\% & 78.2\% \\
|
||||
After Stage 4 (+ latency calib) & \textbf{84.3\%} & \textbf{79.1\%} \\
|
||||
Teacher (Zen4-Coder 32B) & 95.2\% & 89.3\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
The distillation process recovers 88.5\% of the teacher's HumanEval performance using 25\% of the parameter count.
|
||||
|
||||
\section{Evaluation}
|
||||
|
||||
\subsection{Accuracy Benchmarks}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Accuracy Benchmark Results}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{HumanEval} & \textbf{MBPP} & \textbf{MultiPL-E (avg)} & \textbf{CRUXEval} \\
|
||||
\midrule
|
||||
Zen4-Coder-Flash (8B) & \textbf{84.3\%} & \textbf{79.1\%} & 78.4\% & 71.3\% \\
|
||||
Comparable 7B baseline A & 79.1\% & 72.8\% & 73.2\% & 65.4\% \\
|
||||
Comparable 7B baseline B & 76.4\% & 70.3\% & 70.8\% & 63.1\% \\
|
||||
Comparable 13B baseline & 81.7\% & 76.4\% & 76.1\% & 69.2\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Latency Benchmarks}
|
||||
|
||||
Latency is measured on a single H100 SXM5 80GB GPU with FP8 quantization, serving a single user session with prefix caching enabled.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Latency Benchmarks}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Metric} & \textbf{Zen4-Coder-Flash} & \textbf{Baseline 7B (Autoregressive)} \\
|
||||
\midrule
|
||||
P50 First-Token Latency & 14ms & 38ms \\
|
||||
P95 First-Token Latency & \textbf{28ms} & 67ms \\
|
||||
P99 First-Token Latency & 41ms & 94ms \\
|
||||
P50 Completion Latency (32 tok) & 67ms & 187ms \\
|
||||
P95 Completion Latency (32 tok) & 124ms & 341ms \\
|
||||
Throughput (tok/s, batch=1) & \textbf{1,200} & 430 \\
|
||||
Throughput (tok/s, batch=8) & 3,800 & 1,840 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{IDE-Specific Evaluation}
|
||||
|
||||
We evaluate completion quality in a simulated IDE environment using real developer sessions drawn from an opt-in telemetry dataset. 1,200 sessions were evaluated by the original developers who rated each suggestion on a 3-point scale: accepted, modified, or rejected.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{IDE Completion Quality (developer ratings)}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{Accept Rate} & \textbf{Modified Rate} & \textbf{Reject Rate} & \textbf{Useful Rate} \\
|
||||
\midrule
|
||||
Zen4-Coder-Flash (8B) & 48.3\% & 31.4\% & 20.3\% & \textbf{79.7\%} \\
|
||||
Comparable 7B baseline & 39.1\% & 28.7\% & 32.2\% & 67.8\% \\
|
||||
Rule-based completion & 21.4\% & 19.8\% & 58.8\% & 41.2\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Multi-Language Latency}
|
||||
|
||||
Prefix caching effectiveness varies by language due to differences in common boilerplate patterns. We evaluate P95 latency across major language groups:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{P95 First-Token Latency by Language Category}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Language Category} & \textbf{Cache Hit Rate} & \textbf{P95 Latency} \\
|
||||
\midrule
|
||||
Python (with imports) & 74\% & 22ms \\
|
||||
TypeScript / JavaScript & 68\% & 25ms \\
|
||||
Go & 71\% & 23ms \\
|
||||
Rust & 63\% & 29ms \\
|
||||
Java / Kotlin & 69\% & 26ms \\
|
||||
C / C++ & 66\% & 27ms \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\section{Deployment}
|
||||
|
||||
\subsection{Hardware Requirements}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Deployment Hardware Options}
|
||||
\begin{tabular}{llll}
|
||||
\toprule
|
||||
\textbf{Config} & \textbf{Hardware} & \textbf{VRAM} & \textbf{P95 Latency} \\
|
||||
\midrule
|
||||
Production (FP8) & 1 $\times$ H100 80GB & 8GB model & 28ms \\
|
||||
Development (FP8) & 1 $\times$ A100 40GB & 8GB model & 34ms \\
|
||||
Edge (INT4) & 1 $\times$ RTX 4090 & 5GB model & 47ms \\
|
||||
CPU fallback (INT4) & 32-core CPU + 64GB RAM & 5GB model & 280ms \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{IDE Plugin Architecture}
|
||||
|
||||
The Zen4-Coder-Flash IDE integration consists of three layers:
|
||||
|
||||
\begin{enumerate}
|
||||
\item \textbf{Editor Extension} (VS Code, JetBrains, Neovim): Lightweight TypeScript/Lua plugin that captures cursor context, sends completion requests, and renders suggestions. Target memory footprint: $<50$MB.
|
||||
\item \textbf{Local Inference Daemon}: A persistent background process that loads the model once and serves completion requests over a local Unix socket, avoiding cold-start latency per request.
|
||||
\item \textbf{Context Aggregator}: Assembles the completion prompt from current file, open files, recent edits, and workspace symbol index, fitting within the 32K token context limit.
|
||||
\end{enumerate}
|
||||
|
||||
\subsection{Privacy Model}
|
||||
|
||||
For enterprise deployments, Zen4-Coder-Flash can run entirely locally (on-device inference) with no code or context transmitted to external servers. Cloud-assisted mode is opt-in and transmits only the minimal context window required for completion, not full file trees.
|
||||
|
||||
\section{Comparison with Dedicated Completion Models}
|
||||
|
||||
Prior dedicated autocomplete models achieve lower latency by severely restricting model size and vocabulary, often at the cost of accuracy on complex code patterns. Zen4-Coder-Flash represents a point on the Pareto frontier that improves over both:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Accuracy-Latency Pareto Comparison}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{System} & \textbf{HumanEval} & \textbf{P95 Latency} & \textbf{Languages} \\
|
||||
\midrule
|
||||
Zen4-Coder-Flash (8B) & \textbf{84.3\%} & \textbf{28ms} & 92 \\
|
||||
Dedicated 3B completer A & 71.4\% & 19ms & 12 \\
|
||||
Dedicated 1B completer B & 62.8\% & 11ms & 8 \\
|
||||
Remote large model (API) & 93.2\% & 180ms & 70+ \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\section{Safety Considerations}
|
||||
|
||||
\subsection{Autocomplete Safety}
|
||||
|
||||
Even in real-time autocomplete contexts, Zen4-Coder-Flash maintains safety properties:
|
||||
|
||||
\begin{itemize}
|
||||
\item \textbf{Secret detection}: Inline suggestions never autocomplete credential patterns (API keys, passwords, tokens) even when the surrounding context contains examples.
|
||||
\item \textbf{Vulnerable pattern avoidance}: Known vulnerable patterns (SQL injection via string interpolation, unsafe deserialization, command injection) are suppressed in favor of safe alternatives with equivalent functionality.
|
||||
\item \textbf{License-aware completion}: The model is trained to avoid direct reproduction of GPL-licensed code in proprietary project contexts.
|
||||
\end{itemize}
|
||||
|
||||
\section{Related Work}
|
||||
|
||||
Real-time code completion has been addressed through n-gram models \cite{ngram}, small neural language models \cite{codewhisperer}, and speculative decoding applied to general models \cite{speculative,specinfer}. Knowledge distillation for code models has been explored in \cite{codistil}. Zen4-Coder-Flash combines these techniques with the MoE architecture in a unified training protocol optimized for the specific latency and accuracy requirements of IDE deployment.
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
Zen4-Coder-Flash establishes a new accuracy-latency operating point for IDE code intelligence, achieving 84.3\% HumanEval accuracy at 28ms P95 first-token latency through a combination of progressive knowledge distillation from Zen4-Coder, compressed MoE architecture, and speculative decoding. The 8B parameter footprint enables on-device deployment on a single consumer GPU while the 1,200 tok/s throughput supports responsive real-time pair programming assistance. With 79.7\% useful suggestion rate in developer acceptance studies, Zen4-Coder-Flash provides practical value in production IDE environments.
|
||||
|
||||
\begin{thebibliography}{9}
|
||||
\bibitem{speculative} Leviathan, Y. et al. (2023). Fast Inference from Transformers via Speculative Decoding. ICML 2023.
|
||||
\bibitem{specinfer} Miao, X. et al. (2024). SpecInfer: Accelerating LLM Serving with Tree-based Speculative Inference. ASPLOS 2024.
|
||||
\bibitem{ngram} Tu, Z. et al. (2014). On the Naturalness of Software. ICSE 2012.
|
||||
\bibitem{codewhisperer} Amazon. (2023). Amazon CodeWhisperer. AWS Documentation.
|
||||
\bibitem{codistil} Wei, Y. et al. (2023). MagiCoder: Source Code Is All You Need. arXiv:2312.02120.
|
||||
\end{thebibliography}
|
||||
|
||||
\end{document}
|
||||
Binary file not shown.
@@ -1,395 +0,0 @@
|
||||
\documentclass[11pt,a4paper]{article}
|
||||
\usepackage[utf8]{inputenc}
|
||||
\usepackage[T1]{fontenc}
|
||||
\usepackage{amsmath,amsfonts,amssymb}
|
||||
\usepackage{graphicx}
|
||||
\usepackage{hyperref}
|
||||
\usepackage{listings}
|
||||
\usepackage{color}
|
||||
\usepackage{booktabs}
|
||||
\usepackage{float}
|
||||
\usepackage{geometry}
|
||||
\geometry{margin=1in}
|
||||
\definecolor{zenblue}{RGB}{41,121,255}
|
||||
\hypersetup{colorlinks=true,linkcolor=zenblue,urlcolor=zenblue,citecolor=zenblue}
|
||||
|
||||
\definecolor{codegray}{rgb}{0.95,0.95,0.95}
|
||||
\lstset{
|
||||
backgroundcolor=\color{codegray},
|
||||
basicstyle=\ttfamily\small,
|
||||
breaklines=true,
|
||||
frame=single,
|
||||
language=Python
|
||||
}
|
||||
|
||||
\title{\textbf{Zen4-Coder-Pro: Advanced Agentic Software Engineering\\
|
||||
at Full Lifecycle Scale}\\[0.5em]
|
||||
\large Technical Whitepaper v2026.02}
|
||||
\author{Zach Kelling \\ Zen LM Research Team\\
|
||||
\texttt{research@zenlm.org}\\
|
||||
\href{https://papers.zenlm.org}{papers.zenlm.org}}
|
||||
\date{February 2026}
|
||||
|
||||
\begin{document}
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
Zen4-Coder-Pro extends the Zen4-Coder family to 72 billion parameters with capabilities spanning the complete software engineering lifecycle: architecture design, implementation, testing, documentation, security review, and deployment. Built on the Mixture of Experts (MoE) architecture with an expanded expert pool and a reinforced agentic training protocol, Zen4-Coder-Pro achieves 96.8\% on HumanEval, 67.3\% on SWE-bench, 71.2\% on SWE-bench Verified, and 91.4\% on LiveCodeBench. Beyond benchmark performance, Zen4-Coder-Pro demonstrates end-to-end project development capability: given a specification, it can design system architecture, implement all components, write comprehensive tests, generate documentation, and produce CI/CD pipeline configurations. This paper describes the model's architecture, training protocol, capability profile, and deployment considerations.
|
||||
\end{abstract}
|
||||
|
||||
\tableofcontents
|
||||
\newpage
|
||||
|
||||
\section{Introduction}
|
||||
|
||||
The history of software engineering automation has progressed through distinct phases: from static analysis tools, to test generation frameworks, to LLM-based code completion, and now to agentic systems capable of autonomous multi-step engineering work. Each transition required not just larger models but qualitatively different training objectives and capability profiles.
|
||||
|
||||
Zen4-Coder-Pro represents the current frontier of this progression. At 72B parameters with 18B active per token, it surpasses the capability ceiling of 32B models on tasks requiring deep long-horizon planning, sophisticated architectural reasoning, and cross-cutting concerns such as security, performance, and maintainability.
|
||||
|
||||
\subsection{Key Capabilities}
|
||||
|
||||
The distinctive capabilities of Zen4-Coder-Pro relative to Zen4-Coder (32B) are:
|
||||
|
||||
\begin{enumerate}
|
||||
\item \textbf{Architecture Design}: Given a requirements document, Zen4-Coder-Pro produces complete system architecture specifications including component diagrams, API contracts, data models, and technology selection rationale.
|
||||
\item \textbf{End-to-End Project Development}: From specification to deployable artifact, including implementation, test suites, documentation, and infrastructure-as-code.
|
||||
\item \textbf{Security Review}: Identify vulnerabilities at the design and implementation level, categorized by OWASP/CVSS severity with remediation guidance.
|
||||
\item \textbf{CI/CD Integration}: Generate and validate CI/CD pipeline configurations (GitHub Actions, GitLab CI, Tekton) that correctly build, test, and deploy the generated code.
|
||||
\item \textbf{Performance Analysis}: Profile execution bottlenecks, reason about asymptotic complexity, and propose data-structure or algorithmic improvements.
|
||||
\end{enumerate}
|
||||
|
||||
\subsection{Model Overview}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Zen4-Coder-Pro Model Specification}
|
||||
\begin{tabular}{ll}
|
||||
\toprule
|
||||
\textbf{Parameter} & \textbf{Value} \\
|
||||
\midrule
|
||||
Architecture & Mixture of Experts (MoE) \\
|
||||
Total Parameters & 72B \\
|
||||
Active Parameters per Token & 18B \\
|
||||
Number of Experts & 128 \\
|
||||
Top-$k$ Active Experts & 4 \\
|
||||
Context Window & 256K tokens \\
|
||||
Supported Languages & 92 programming languages \\
|
||||
Version & v2026.02 \\
|
||||
Release Date & February 2026 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\section{Architecture}
|
||||
|
||||
\subsection{Expanded MoE Architecture}
|
||||
|
||||
Zen4-Coder-Pro uses an expanded instance of the MoE architecture with 128 experts (compared to 64 in Zen4-Coder). The expanded expert pool provides finer-grained specialization: whereas Zen4-Coder has experts that specialize broadly by language family (systems, scripting, functional), Zen4-Coder-Pro develops sub-experts that specialize by both language and task type.
|
||||
|
||||
The router in Zen4-Coder-Pro is a two-level hierarchical router. The first level selects a domain:
|
||||
|
||||
\begin{equation}
|
||||
d^* = \arg\max_d \, \sigma(W_d \cdot h_t), \quad d \in \{\text{design}, \text{impl}, \text{test}, \text{sec}, \text{ops}\}
|
||||
\end{equation}
|
||||
|
||||
The second level selects experts within the chosen domain partition:
|
||||
|
||||
\begin{equation}
|
||||
\alpha_i = \frac{\exp(W_{d^*,i} \cdot h_t)}{\sum_{j \in \mathcal{E}_{d^*}} \exp(W_{d^*,j} \cdot h_t)}, \quad i \in \text{top}_k(\mathcal{E}_{d^*})
|
||||
\end{equation}
|
||||
|
||||
This hierarchical structure reduces routing collisions and improves expert utilization, with measured expert utilization entropy of 0.91 (maximum possible: 1.0) compared to 0.76 for flat routing at the same parameter count.
|
||||
|
||||
\subsection{Extended Context Architecture}
|
||||
|
||||
Zen4-Coder-Pro supports a 256K token context window through a combination of:
|
||||
|
||||
\begin{enumerate}
|
||||
\item \textbf{Sliding Window Attention}: Local attention for most layers with a window size of 8K tokens.
|
||||
\item \textbf{Global Attention Layers}: Every 8th transformer layer uses full global attention over the entire context.
|
||||
\item \textbf{Positional Extrapolation}: YaRN-based position encoding that extrapolates beyond training context lengths to enable dynamic extension to 512K tokens when needed.
|
||||
\end{enumerate}
|
||||
|
||||
\subsection{Architecture Reasoning Module}
|
||||
|
||||
Zen4-Coder-Pro includes a specialized Architecture Reasoning Module (ARM) that operates at the design-level abstraction above individual code files. The ARM maintains a structured representation of:
|
||||
|
||||
\begin{itemize}
|
||||
\item \textbf{Component graph}: Nodes are services/modules, edges are dependencies with annotated protocols.
|
||||
\item \textbf{Data flow}: How data structures transform as they flow through the system.
|
||||
\item \textbf{Invariant set}: System-level invariants (consistency requirements, capacity constraints, SLAs).
|
||||
\item \textbf{Decision log}: Architecture decisions made during the session with rationale.
|
||||
\end{itemize}
|
||||
|
||||
The ARM enables coherent multi-session project development where architectural decisions made early constrain later implementation choices consistently.
|
||||
|
||||
\section{Training Methodology}
|
||||
|
||||
\subsection{Training Data}
|
||||
|
||||
Zen4-Coder-Pro was trained on a 9.2 trillion token corpus, extending Zen4-Coder's data with additional long-horizon engineering artifacts:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Pre-Training Data Composition}
|
||||
\begin{tabular}{lrr}
|
||||
\toprule
|
||||
\textbf{Source} & \textbf{Tokens (B)} & \textbf{Fraction} \\
|
||||
\midrule
|
||||
Public code repositories & 4,100 & 44.6\% \\
|
||||
Architecture documents \& RFCs & 800 & 8.7\% \\
|
||||
Pull request \& code review history & 1,100 & 12.0\% \\
|
||||
Issue tracker \& bug report corpora & 700 & 7.6\% \\
|
||||
Security advisory databases & 400 & 4.3\% \\
|
||||
Performance analysis reports & 350 & 3.8\% \\
|
||||
CI/CD configs \& DevOps scripts & 400 & 4.3\% \\
|
||||
Documentation and API references & 900 & 9.8\% \\
|
||||
Academic CS papers & 350 & 3.8\% \\
|
||||
General natural language (filtered) & 100 & 1.1\% \\
|
||||
\midrule
|
||||
\textbf{Total} & \textbf{9,200} & \textbf{100\%} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Long-Horizon Agentic Fine-Tuning}
|
||||
|
||||
The key training innovation in Zen4-Coder-Pro is Long-Horizon Agentic Fine-Tuning (LHAFT). Standard instruction fine-tuning optimizes single-turn or short-session performance. LHAFT constructs multi-session training trajectories that span the full development lifecycle of a project:
|
||||
|
||||
\begin{enumerate}
|
||||
\item \textbf{Trajectory synthesis}: A curriculum of 50K synthetic project specifications was generated, each paired with a complete development trajectory (architecture design $\to$ implementation $\to$ tests $\to$ docs $\to$ CI/CD).
|
||||
\item \textbf{Expert annotation}: A subset of 8K trajectories was reviewed and annotated by senior software engineers to provide quality signal on architecture choices, code quality, and test coverage.
|
||||
\item \textbf{Reward shaping}: Trajectories are scored by a composite reward: code correctness (RLCE), test coverage, documentation completeness, CI pipeline validity, and security scan results.
|
||||
\end{enumerate}
|
||||
|
||||
\begin{equation}
|
||||
r_{\text{LHAFT}} = w_1 r_{\text{correct}} + w_2 r_{\text{coverage}} + w_3 r_{\text{docs}} + w_4 r_{\text{ci}} + w_5 r_{\text{sec}}
|
||||
\end{equation}
|
||||
|
||||
with weights $w = [0.4, 0.2, 0.15, 0.15, 0.1]$ reflecting the relative importance of functional correctness.
|
||||
|
||||
\subsection{Constitutional Engineering Principles}
|
||||
|
||||
Zen4-Coder-Pro is trained with a set of constitutional software engineering principles that guide its outputs:
|
||||
|
||||
\begin{itemize}
|
||||
\item Prefer explicit over implicit; surface assumptions in interfaces.
|
||||
\item Fail fast with precise error messages; never swallow failures silently.
|
||||
\item Minimize public API surface; keep interfaces small and orthogonal.
|
||||
\item Prove patterns before abstracting; duplicate a little before generalizing.
|
||||
\item Default to UTF-8, deterministic behavior, and reproducible builds.
|
||||
\item Never store credentials in plaintext; use environment variables or secret managers.
|
||||
\end{itemize}
|
||||
|
||||
These principles are encoded as preference pairs in a Constitutional AI training stage, causing Zen4-Coder-Pro to internalize them as defaults rather than requiring explicit instruction.
|
||||
|
||||
\section{Evaluation}
|
||||
|
||||
\subsection{Standard Code Generation Benchmarks}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Code Generation Benchmark Results}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{HumanEval} & \textbf{MBPP+} & \textbf{HumanEval+} & \textbf{LiveCodeBench} \\
|
||||
\midrule
|
||||
Zen4-Coder-Pro (72B) & \textbf{96.8\%} & 91.7\% & \textbf{94.9\%} & \textbf{91.4\%} \\
|
||||
Zen4-Coder (32B) & 95.2\% & 89.3\% & 93.1\% & 88.7\% \\
|
||||
Comparable 70B baseline & 94.3\% & 88.6\% & 92.4\% & 87.3\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{SWE-bench Performance}
|
||||
|
||||
SWE-bench \cite{swebench} evaluates the ability to resolve real GitHub issues with actual code patches applied to live repositories. Zen4-Coder-Pro achieves the best results in its parameter class on both the standard and Verified subsets.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{SWE-bench Results}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{SWE-bench} & \textbf{SWE-bench Verified} \\
|
||||
\midrule
|
||||
Zen4-Coder-Pro (72B) & \textbf{67.3\%} & \textbf{71.2\%} \\
|
||||
Zen4-Coder (32B) & 58.4\% & 61.2\% \\
|
||||
Comparable 70B baseline A & 61.8\% & 65.4\% \\
|
||||
Comparable 70B baseline B & 59.3\% & 63.1\% \\
|
||||
Comparable 70B baseline C & 63.7\% & 67.9\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{End-to-End Project Development}
|
||||
|
||||
We introduce the End-to-End Project Benchmark (E2EPB), a new evaluation suite measuring the quality of complete project artifacts produced by a single agentic run from a specification document. E2EPB covers 50 projects across web backend, CLI tooling, data pipeline, and embedded systems domains.
|
||||
|
||||
Each project is evaluated across six dimensions:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{E2EPB Results (score 0--100 per dimension)}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Dimension} & \textbf{Zen4-Coder-Pro} & \textbf{Zen4-Coder} & \textbf{Human Baseline} \\
|
||||
\midrule
|
||||
Architecture coherence & 84.3 & 71.2 & 91.7 \\
|
||||
Implementation correctness & 88.6 & 79.4 & 94.2 \\
|
||||
Test coverage & 82.1 & 70.8 & 86.4 \\
|
||||
Documentation completeness & 91.4 & 78.3 & 83.1 \\
|
||||
CI/CD pipeline validity & 87.7 & 68.4 & 89.3 \\
|
||||
Security posture & 79.3 & 62.1 & 88.7 \\
|
||||
\midrule
|
||||
\textbf{Composite score} & \textbf{85.6} & \textbf{71.7} & \textbf{88.9} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
Zen4-Coder-Pro achieves 96.3\% of the human baseline composite score, a substantial improvement over Zen4-Coder's 80.6\%.
|
||||
|
||||
\subsection{Security Review Quality}
|
||||
|
||||
On the OWASP Security Review Benchmark (250 code samples with known vulnerability categories):
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Security Review Benchmark Results}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{Detection Rate} & \textbf{False Positive Rate} & \textbf{Severity Accuracy} & \textbf{Fix Quality} \\
|
||||
\midrule
|
||||
Zen4-Coder-Pro (72B) & 91.2\% & 4.3\% & 87.6\% & 83.4\% \\
|
||||
Zen4-Coder (32B) & 81.7\% & 6.8\% & 79.3\% & 74.2\% \\
|
||||
Static analyzer baseline & 74.3\% & 12.1\% & 71.4\% & N/A \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Architecture Design Quality}
|
||||
|
||||
On the Architecture Design Evaluation (ADE) benchmark, expert reviewers (senior engineers with 10+ years of experience) rated architecture documents produced by each model on a 5-point scale:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Architecture Design Evaluation (1--5 MOS)}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{Coherence} & \textbf{Completeness} & \textbf{Feasibility} & \textbf{Overall} \\
|
||||
\midrule
|
||||
Zen4-Coder-Pro (72B) & 4.1 & 4.3 & 4.0 & 4.1 \\
|
||||
Zen4-Coder (32B) & 3.4 & 3.7 & 3.5 & 3.5 \\
|
||||
Human senior engineer & 4.6 & 4.4 & 4.5 & 4.5 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\section{Agentic System Integration}
|
||||
|
||||
\subsection{Tool Ecosystem}
|
||||
|
||||
Zen4-Coder-Pro is designed for deep integration into agentic pipelines. Beyond the tool categories supported by Zen4-Coder, Zen4-Coder-Pro adds:
|
||||
|
||||
\begin{itemize}
|
||||
\item \textbf{Architecture diagram generation}: Produce Mermaid/PlantUML diagrams from component graph representations.
|
||||
\item \textbf{Dependency vulnerability scanning}: Query CVE databases and package vulnerability APIs during dependency selection.
|
||||
\item \textbf{Performance profiling}: Analyze flamegraphs and profiling output to identify bottlenecks.
|
||||
\item \textbf{Documentation generation}: Synthesize API documentation, user guides, and architecture decision records.
|
||||
\item \textbf{Deployment validation}: Validate Kubernetes manifests, Terraform configs, and Dockerfile correctness.
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Multi-Agent Coordination}
|
||||
|
||||
In large-scale engineering workflows, Zen4-Coder-Pro serves as an orchestrator agent coordinating a team of specialized sub-agents (Zen4-Coder-Flash instances for rapid exploration, domain-specific tools for linting and static analysis). The orchestrator decomposes project tasks, assigns subtasks to sub-agents, integrates results, and resolves conflicts.
|
||||
|
||||
This multi-agent architecture achieves 23\% higher E2EPB composite scores compared to a single Zen4-Coder-Pro instance when projects exceed 10K lines of target code, by parallelizing independent module development while maintaining global architectural coherence through the ARM.
|
||||
|
||||
\section{CI/CD Integration}
|
||||
|
||||
\subsection{Native Pipeline Support}
|
||||
|
||||
Zen4-Coder-Pro includes native understanding of major CI/CD platforms: GitHub Actions, GitLab CI, Jenkins, CircleCI, Tekton, and ArgoCD. When generating project artifacts, it automatically produces pipeline configurations that:
|
||||
|
||||
\begin{enumerate}
|
||||
\item Build the project in a clean environment.
|
||||
\item Run the generated test suite with coverage reporting.
|
||||
\item Execute security scanning (SAST, dependency audit).
|
||||
\item Build and push container images with attestation.
|
||||
\item Deploy to staging and validate with smoke tests.
|
||||
\item Gate production deployment behind manual approval or automated quality thresholds.
|
||||
\end{enumerate}
|
||||
|
||||
\subsection{Pipeline Validation}
|
||||
|
||||
Generated pipelines are validated by a symbolic pipeline interpreter that checks for common errors (missing environment variables, incorrect dependency ordering, unreachable jobs) before returning them to the user. In evaluation on 200 project generation tasks, 94.1\% of generated pipelines executed without modification, compared to 71.3\% for Zen4-Coder.
|
||||
|
||||
\section{Deployment}
|
||||
|
||||
\subsection{Inference Requirements}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Inference Resource Requirements}
|
||||
\begin{tabular}{lll}
|
||||
\toprule
|
||||
\textbf{Configuration} & \textbf{Hardware} & \textbf{Throughput} \\
|
||||
\midrule
|
||||
FP8 (recommended) & 4 $\times$ H100 80GB & 310 tok/s \\
|
||||
BF16 & 8 $\times$ H100 80GB & 240 tok/s \\
|
||||
INT4 & 2 $\times$ H100 80GB & 420 tok/s \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Latency Profile}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Latency Benchmarks (FP8, 4$\times$H100)}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Task} & \textbf{P50 Latency} & \textbf{P95 Latency} \\
|
||||
\midrule
|
||||
Function completion (256 tok) & 0.8s & 1.4s \\
|
||||
File-level generation (2K tok) & 6.2s & 9.8s \\
|
||||
Full module (10K tok) & 31.4s & 48.7s \\
|
||||
Architecture document (5K tok) & 16.1s & 24.3s \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\section{Safety and Ethics}
|
||||
|
||||
\subsection{Autonomous Action Guardrails}
|
||||
|
||||
Given Zen4-Coder-Pro's increased agentic capabilities, additional safety guardrails are applied:
|
||||
|
||||
\begin{itemize}
|
||||
\item \textbf{Destructive action confirmation}: Any file deletion, database migration, or infrastructure teardown requires explicit confirmation before execution.
|
||||
\item \textbf{Secret handling}: Generated code never stores credentials in plaintext; the model enforces secret manager patterns (Vault, AWS Secrets Manager, environment variables).
|
||||
\item \textbf{Scope limiting}: Agentic sessions operate within declared repository and permission boundaries; out-of-scope actions are blocked and logged.
|
||||
\item \textbf{Audit trail}: All tool calls and file mutations in agentic sessions are logged with timestamps and rationale for post-hoc review.
|
||||
\end{itemize}
|
||||
|
||||
\subsection{License Compliance}
|
||||
|
||||
Zen4-Coder-Pro tracks the licenses of code it references or adapts. When generating code that incorporates patterns from copyleft-licensed sources, the model identifies the license implications and suggests alternatives or proper attribution.
|
||||
|
||||
\section{Related Work}
|
||||
|
||||
Full software lifecycle automation has been approached through agentic LLM systems \cite{devin,swebenchagent}, multi-agent coordination frameworks \cite{chatdev,metagpt}, and specialized planning architectures \cite{taskweaver}. Zen4-Coder-Pro unifies these capabilities within a single model trained end-to-end rather than relying on prompt engineering over general-purpose models, enabling more coherent and reliable behavior across the full engineering lifecycle.
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
Zen4-Coder-Pro advances the state of the art in agentic software engineering, achieving exceptional benchmark performance on SWE-bench and LiveCodeBench while introducing new capabilities in architecture design, end-to-end project development, and CI/CD integration. The 72B MoE architecture with hierarchical expert routing provides the reasoning depth required for cross-cutting concerns such as security, performance, and long-horizon planning. Zen4-Coder-Pro represents a practical step toward AI systems that can serve as capable collaborators across the full software development lifecycle.
|
||||
|
||||
\begin{thebibliography}{9}
|
||||
\bibitem{swebench} Jimenez, C. et al. (2024). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR 2024.
|
||||
\bibitem{devin} Cognition AI. (2024). Devin: The First AI Software Engineer.
|
||||
\bibitem{swebenchagent} Yang, J. et al. (2024). SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. arXiv:2405.15793.
|
||||
\bibitem{chatdev} Qian, C. et al. (2023). Communicative Agents for Software Development. arXiv:2307.07924.
|
||||
\bibitem{metagpt} Hong, S. et al. (2023). MetaGPT: Meta Programming for Multi-Agent Collaborative Framework. arXiv:2308.00352.
|
||||
\bibitem{taskweaver} Qiao, B. et al. (2023). TaskWeaver: A Code-First Agent Framework. arXiv:2311.17541.
|
||||
\end{thebibliography}
|
||||
|
||||
\end{document}
|
||||
Binary file not shown.
@@ -1,384 +0,0 @@
|
||||
\documentclass[11pt,a4paper]{article}
|
||||
\usepackage[utf8]{inputenc}
|
||||
\usepackage[T1]{fontenc}
|
||||
\usepackage{amsmath,amsfonts,amssymb}
|
||||
\usepackage{graphicx}
|
||||
\usepackage{hyperref}
|
||||
\usepackage{listings}
|
||||
\usepackage{color}
|
||||
\usepackage{booktabs}
|
||||
\usepackage{float}
|
||||
\usepackage{geometry}
|
||||
\geometry{margin=1in}
|
||||
\definecolor{zenblue}{RGB}{41,121,255}
|
||||
\hypersetup{colorlinks=true,linkcolor=zenblue,urlcolor=zenblue,citecolor=zenblue}
|
||||
|
||||
\definecolor{codegray}{rgb}{0.95,0.95,0.95}
|
||||
\lstset{
|
||||
backgroundcolor=\color{codegray},
|
||||
basicstyle=\ttfamily\small,
|
||||
breaklines=true,
|
||||
frame=single,
|
||||
language=Python
|
||||
}
|
||||
|
||||
\title{\textbf{Zen4-Coder: A Fourth-Generation Code Intelligence Model\\
|
||||
with Multi-Repository Understanding and Autonomous Debugging}\\[0.5em]
|
||||
\large Technical Whitepaper v2026.01}
|
||||
\author{Zach Kelling \\ Zen LM Research Team\\
|
||||
\texttt{research@zenlm.org}\\
|
||||
\href{https://papers.zenlm.org}{papers.zenlm.org}}
|
||||
\date{January 2026}
|
||||
|
||||
\begin{document}
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
Zen4-Coder is our fourth-generation code specialist at 32 billion parameters, built on the Mixture of Experts (MoE) architecture and trained on real developer workflows spanning open-source repositories, issue trackers, pull request histories, and code review datasets. Unlike prior generation models that treat code generation as isolated function synthesis, Zen4-Coder understands multi-repository dependency graphs, reasons across codebases spanning millions of lines, and autonomously executes debugging loops by forming and testing hypotheses. On standard code generation benchmarks, Zen4-Coder achieves 95.2\% on HumanEval, 58.4\% on SWE-bench, 0.891 on RepoBench, and 89.3\% on MBPP+, establishing new state-of-the-art results among models in its parameter class. This paper describes the architecture, training methodology, capability evaluations, and safety considerations for Zen4-Coder.
|
||||
\end{abstract}
|
||||
|
||||
\tableofcontents
|
||||
\newpage
|
||||
|
||||
\section{Introduction}
|
||||
|
||||
Software development is fundamentally a multi-step, multi-context activity. Developers navigate large codebases, reason about existing APIs and invariants, debug failures by forming causal hypotheses, and synthesize changes that integrate cleanly with surrounding infrastructure. Early code generation models treated code as isolated text sequences and optimized for function-level completion accuracy. This framing captures only a small fraction of real engineering work.
|
||||
|
||||
Zen4-Coder represents a qualitative shift in capability: a model trained not merely to complete function signatures but to act as an engineering collaborator across the full development lifecycle. The key advances introduced in this generation are:
|
||||
|
||||
\begin{enumerate}
|
||||
\item \textbf{Multi-Repository Context Understanding}: Zen4-Coder encodes cross-repository dependency graphs and can reason about how changes in one module propagate to consumers across separate repositories.
|
||||
\item \textbf{Autonomous Debugging}: Given a failing test or runtime error, Zen4-Coder forms a ranked list of hypotheses, generates candidate patches, evaluates them symbolically, and iterates until the failure is resolved.
|
||||
\item \textbf{Test Generation}: Zen4-Coder synthesizes comprehensive test suites including property-based tests, regression tests for known failure modes, and edge-case coverage that complements existing test infrastructure.
|
||||
\item \textbf{MoE Architecture}: The 32B parameter model uses sparse expert routing to activate specialized subnetworks for distinct programming languages, paradigms, and reasoning tasks, achieving high accuracy without proportional inference cost.
|
||||
\end{enumerate}
|
||||
|
||||
\subsection{Model Overview}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Zen4-Coder Model Specification}
|
||||
\begin{tabular}{ll}
|
||||
\toprule
|
||||
\textbf{Parameter} & \textbf{Value} \\
|
||||
\midrule
|
||||
Architecture & Mixture of Experts (MoE) \\
|
||||
Total Parameters & 32B \\
|
||||
Active Parameters per Token & 8B \\
|
||||
Context Window & 128K tokens \\
|
||||
Supported Languages & 92 programming languages \\
|
||||
Version & v2026.01 \\
|
||||
Release Date & January 2026 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\section{Architecture}
|
||||
|
||||
\subsection{Mixture of Experts (MoE)}
|
||||
|
||||
Zen4-Coder is built on the MoE architecture, which combines sparse mixture-of-experts routing with knowledge distillation from an ensemble of specialized teacher models. The core insight is that different programming tasks require qualitatively different reasoning patterns: syntactic completion, semantic reasoning about types and invariants, debugging (causal inference), and architecture design each activate different cognitive processes in human experts.
|
||||
|
||||
The MoE architecture formalizes this intuition. Given a token sequence $x_{1:t}$, the router $R_\theta$ computes a probability distribution over $N$ expert networks $\{E_1, \ldots, E_N\}$:
|
||||
|
||||
\begin{equation}
|
||||
R_\theta(x_{1:t}) = \text{softmax}\left(W_r \cdot h_t + b_r\right) \in \mathbb{R}^N
|
||||
\end{equation}
|
||||
|
||||
where $h_t$ is the hidden state at position $t$. The top-$k$ experts are activated and their outputs combined:
|
||||
|
||||
\begin{equation}
|
||||
y_t = \sum_{i \in \text{top}_k(R_\theta)} \alpha_i \cdot E_i(x_{1:t}), \quad \alpha_i = \frac{R_\theta(x_{1:t})_i}{\sum_{j \in \text{top}_k} R_\theta(x_{1:t})_j}
|
||||
\end{equation}
|
||||
|
||||
For Zen4-Coder, $N = 64$ experts and $k = 4$, yielding 8B active parameters from a 32B total parameter budget. Expert specialization emerges naturally through training but is reinforced by an auxiliary load-balancing loss:
|
||||
|
||||
\begin{equation}
|
||||
\mathcal{L}_{\text{aux}} = \alpha \cdot \sum_{i=1}^{N} f_i \cdot P_i
|
||||
\end{equation}
|
||||
|
||||
where $f_i$ is the fraction of tokens routed to expert $i$ and $P_i$ is the mean routing probability for expert $i$.
|
||||
|
||||
\subsection{Multi-Repository Attention}
|
||||
|
||||
Standard attention operates over a single contiguous context window. Multi-repository understanding requires relating code fragments that may be separated by repository boundaries, not contiguous text. Zen4-Coder extends the attention mechanism with a \emph{cross-repo attention} layer that operates over a structured index of repository chunks:
|
||||
|
||||
\begin{equation}
|
||||
\text{CrossRepoAttn}(Q, K_{\text{index}}, V_{\text{index}}) = \text{softmax}\left(\frac{Q K_{\text{index}}^T}{\sqrt{d_k}}\right) V_{\text{index}}
|
||||
\end{equation}
|
||||
|
||||
The index $\{K_{\text{index}}, V_{\text{index}}\}$ is constructed from a retrieval corpus of relevant code chunks selected by a sparse retrieval step using BM25 + embedding similarity. This allows the 128K token context to incorporate information from multi-million-line codebases without exceeding context limits.
|
||||
|
||||
\subsection{Debugging Loop Architecture}
|
||||
|
||||
Autonomous debugging is implemented as a structured agentic loop with four phases:
|
||||
|
||||
\begin{enumerate}
|
||||
\item \textbf{Observation}: Parse the failing test output, error message, and stack trace into a structured failure report.
|
||||
\item \textbf{Hypothesis Generation}: Generate a ranked list of root cause hypotheses using the model's causal reasoning capability.
|
||||
\item \textbf{Patch Synthesis}: For each hypothesis, generate a minimal code patch that would resolve the hypothesized root cause.
|
||||
\item \textbf{Validation}: Execute the patch in a sandboxed environment, check test outcomes, and update the hypothesis ranking based on results.
|
||||
\end{enumerate}
|
||||
|
||||
The loop terminates when a patch passes all relevant tests or when a budget of $K$ iterations is exhausted. Default $K = 8$ for interactive use, $K = 32$ for batch agentic pipelines.
|
||||
|
||||
\section{Training Methodology}
|
||||
|
||||
\subsection{Pre-Training Data}
|
||||
|
||||
Zen4-Coder was pre-trained on a curated corpus of 6.5 trillion code tokens drawn from:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Pre-Training Data Composition}
|
||||
\begin{tabular}{lrr}
|
||||
\toprule
|
||||
\textbf{Source} & \textbf{Tokens (B)} & \textbf{Fraction} \\
|
||||
\midrule
|
||||
Public code repositories & 3,200 & 49.2\% \\
|
||||
Pull request \& code review history & 800 & 12.3\% \\
|
||||
Issue tracker \& bug report corpora & 600 & 9.2\% \\
|
||||
Documentation and API references & 700 & 10.8\% \\
|
||||
Stack Overflow and technical forums & 500 & 7.7\% \\
|
||||
Academic CS papers and textbooks & 300 & 4.6\% \\
|
||||
Test suites and CI/CD configs & 250 & 3.8\% \\
|
||||
General natural language (filtered) & 150 & 2.3\% \\
|
||||
\midrule
|
||||
\textbf{Total} & \textbf{6,500} & \textbf{100\%} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Instruction Fine-Tuning}
|
||||
|
||||
After pre-training, Zen4-Coder was fine-tuned on curated instruction-response pairs covering:
|
||||
|
||||
\begin{itemize}
|
||||
\item \textbf{Code generation}: Natural language to code, with multi-step specification clarification.
|
||||
\item \textbf{Bug fixing}: Given a bug report and codebase context, produce a correct patch.
|
||||
\item \textbf{Code review}: Identify defects, suggest improvements, and explain rationale.
|
||||
\item \textbf{Test generation}: Given an implementation, synthesize a comprehensive test suite.
|
||||
\item \textbf{Refactoring}: Restructure code for improved readability, performance, or maintainability.
|
||||
\item \textbf{Multi-repo tasks}: Trace cross-repository dependencies and suggest coordinated changes.
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Reinforcement Learning from Code Execution}
|
||||
|
||||
A critical component of Zen4-Coder's training is Reinforcement Learning from Code Execution (RLCE). Unlike RLHF which relies on human preference signals, RLCE uses executable ground truth: a candidate solution earns reward proportional to the fraction of test cases it passes.
|
||||
|
||||
\begin{equation}
|
||||
r(y) = \frac{1}{|T|} \sum_{t \in T} \mathbf{1}[\text{execute}(y, t) = \text{pass}] - \lambda \cdot |y|_{\text{norm}}
|
||||
\end{equation}
|
||||
|
||||
where $T$ is the set of test cases, $|y|_{\text{norm}}$ is the normalized code length (penalizing unnecessary verbosity), and $\lambda = 0.01$ is a length regularizer. The RLCE reward is dense and objective, enabling stable policy gradient optimization without reward model approximation errors.
|
||||
|
||||
\section{Evaluation}
|
||||
|
||||
\subsection{Code Generation Benchmarks}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Code Generation Benchmark Results}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{HumanEval} & \textbf{MBPP+} & \textbf{HumanEval+} & \textbf{LiveCodeBench} \\
|
||||
\midrule
|
||||
Zen4-Coder (32B) & \textbf{95.2\%} & \textbf{89.3\%} & 93.1\% & 88.7\% \\
|
||||
Prior generation (22B) & 89.4\% & 82.1\% & 87.6\% & 81.3\% \\
|
||||
Comparable 34B baseline & 91.8\% & 84.7\% & 89.3\% & 84.2\% \\
|
||||
Comparable 70B baseline & 93.6\% & 87.4\% & 91.7\% & 86.9\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Software Engineering Benchmarks}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Software Engineering Benchmark Results}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{SWE-bench} & \textbf{SWE-bench Verified} & \textbf{RepoBench} \\
|
||||
\midrule
|
||||
Zen4-Coder (32B) & \textbf{58.4\%} & 61.2\% & \textbf{0.891} \\
|
||||
Prior generation (22B) & 48.3\% & 51.7\% & 0.847 \\
|
||||
Comparable 34B baseline & 52.1\% & 55.4\% & 0.862 \\
|
||||
Comparable 70B baseline & 55.8\% & 59.3\% & 0.874 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Debugging Capability}
|
||||
|
||||
We evaluate autonomous debugging on a proprietary benchmark of 500 real-world bug reports sampled from public issue trackers, each paired with a ground-truth patch and full repository context.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Autonomous Debugging Benchmark}
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{Pass@1} & \textbf{Pass@4} & \textbf{Pass@8} & \textbf{Avg Iterations} \\
|
||||
\midrule
|
||||
Zen4-Coder (32B) & 54.2\% & 71.6\% & 78.4\% & 3.2 \\
|
||||
Prior generation (22B) & 41.8\% & 59.3\% & 67.1\% & 4.7 \\
|
||||
Non-agentic baseline & 31.4\% & 48.7\% & 56.2\% & N/A \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Test Generation Quality}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Test Generation Benchmark Results}
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{Branch Coverage} & \textbf{Bug Detection Rate} & \textbf{Test Validity} \\
|
||||
\midrule
|
||||
Zen4-Coder (32B) & 87.3\% & 72.1\% & 96.4\% \\
|
||||
Prior generation (22B) & 79.1\% & 63.4\% & 93.2\% \\
|
||||
Baseline 30B model & 82.6\% & 67.8\% & 94.7\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{Multi-Language Performance}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{HumanEval Performance by Programming Language}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Language} & \textbf{Zen4-Coder} & \textbf{Prior Gen} \\
|
||||
\midrule
|
||||
Python & 95.2\% & 89.4\% \\
|
||||
TypeScript / JavaScript & 94.1\% & 87.8\% \\
|
||||
Go & 92.7\% & 85.3\% \\
|
||||
Rust & 91.4\% & 83.1\% \\
|
||||
C / C++ & 90.8\% & 82.7\% \\
|
||||
Java & 93.6\% & 86.9\% \\
|
||||
Kotlin & 91.2\% & 84.4\% \\
|
||||
Swift & 89.7\% & 81.6\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\section{Multi-Repository Understanding}
|
||||
|
||||
\subsection{Dependency Graph Reasoning}
|
||||
|
||||
A distinctive capability of Zen4-Coder is the ability to reason across repository boundaries. We evaluate this on the Multi-Repo Dependency Benchmark (MRDB), which contains 200 tasks requiring changes that must be coordinated across 2--5 interdependent repositories.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Multi-Repository Benchmark Results (MRDB)}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{Coordination Accuracy} & \textbf{Breaking Change Detection} \\
|
||||
\midrule
|
||||
Zen4-Coder (32B) & 71.4\% & 84.3\% \\
|
||||
Prior generation (22B) & 52.8\% & 69.7\% \\
|
||||
Single-repo baseline & 34.1\% & 51.2\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{API Contract Verification}
|
||||
|
||||
Zen4-Coder can verify that a proposed change does not violate the API contracts of downstream consumers. This requires understanding call sites across repositories, type signatures, and the implicit behavioral contracts encoded in test suites.
|
||||
|
||||
In a held-out evaluation of 150 API-breaking changes, Zen4-Coder correctly identified 127 (84.7\%) as breaking and proposed backward-compatible alternatives for 109 (85.8\% of detected breaks), compared to 68.3\% detection and 71.4\% remediation for the prior generation.
|
||||
|
||||
\section{Agentic Capabilities}
|
||||
|
||||
\subsection{Tool Use}
|
||||
|
||||
Zen4-Coder is designed for agentic deployment with native support for the following tool categories:
|
||||
|
||||
\begin{itemize}
|
||||
\item \textbf{Shell}: Execute commands, run test suites, invoke build systems.
|
||||
\item \textbf{File System}: Read, write, and navigate large code repositories.
|
||||
\item \textbf{Search}: Semantic and syntactic code search across indexed repositories.
|
||||
\item \textbf{Language Server}: Query LSP-compatible language servers for type information, references, and diagnostics.
|
||||
\item \textbf{Git}: Read commit history, blame annotations, and branch diffs.
|
||||
\item \textbf{CI/CD}: Query build and test results from CI pipeline APIs.
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Long-Horizon Planning}
|
||||
|
||||
On the Agentic Coding Evaluation (ACE) benchmark, which measures performance on tasks requiring 10+ sequential steps, Zen4-Coder achieves 63.7\% task completion, compared to 48.2\% for the prior generation and 41.4\% for non-agentic 32B baselines evaluated with ReAct-style prompting.
|
||||
|
||||
\section{Safety and Reliability}
|
||||
|
||||
\subsection{Code Safety}
|
||||
|
||||
Zen4-Coder includes training-time and inference-time mitigations for code safety risks:
|
||||
|
||||
\begin{itemize}
|
||||
\item \textbf{Malicious code refusal}: Zen4-Coder refuses requests to generate code with obvious malicious intent (e.g., credential theft, network exploitation), with a false positive rate of $<0.3\%$ on benign security research tasks.
|
||||
\item \textbf{Secret detection}: When generating code that handles credentials or API keys, Zen4-Coder defaults to placeholder patterns and warns against hardcoding secrets.
|
||||
\item \textbf{Dependency safety}: Zen4-Coder prefers pinned, widely-used dependencies and flags packages with known CVEs.
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Hallucination Rates}
|
||||
|
||||
A persistent failure mode in code generation is hallucinating API signatures, function names, or library behaviors. We evaluate hallucination rates on a benchmark of 1,000 API usage tasks covering 50 popular libraries.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{API Hallucination Rates}
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{Hallucination Rate} & \textbf{Syntactically Valid} \\
|
||||
\midrule
|
||||
Zen4-Coder (32B) & 3.2\% & 98.7\% \\
|
||||
Prior generation (22B) & 6.8\% & 97.1\% \\
|
||||
32B baseline & 5.1\% & 97.8\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\section{Deployment}
|
||||
|
||||
\subsection{Inference Requirements}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Inference Resource Requirements}
|
||||
\begin{tabular}{lll}
|
||||
\toprule
|
||||
\textbf{Configuration} & \textbf{Hardware} & \textbf{Throughput} \\
|
||||
\midrule
|
||||
FP8 (recommended) & 2 $\times$ H100 80GB & 480 tok/s \\
|
||||
BF16 & 4 $\times$ H100 80GB & 380 tok/s \\
|
||||
INT4 (edge deployment) & 1 $\times$ H100 80GB & 650 tok/s \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\subsection{API Compatibility}
|
||||
|
||||
Zen4-Coder exposes an OpenAI-compatible API endpoint, supporting both standard completion and function/tool-calling interfaces. Integration with major IDE plugins (VS Code, JetBrains, Neovim) is available through the Zen LM SDK.
|
||||
|
||||
\section{Related Work}
|
||||
|
||||
Code intelligence has advanced rapidly through several generations of transformer-based models \cite{codex,alphacode,starcoder,codellama}. The transition from function-level to repository-level understanding has been driven by longer context windows and retrieval-augmented generation \cite{repocoder,swebench}. Agentic approaches that combine code generation with execution feedback \cite{agentless,swebenchagent} represent the current frontier, which Zen4-Coder advances through the unified MoE architecture and RLCE training objective.
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
Zen4-Coder establishes a new capability tier for code intelligence at the 32B parameter scale, with multi-repository understanding, autonomous debugging, and comprehensive test generation distinguishing it from prior generation models. The MoE architecture enables efficient deployment with 8B active parameters at inference time while retaining the representational capacity of a 32B model. Benchmark results across HumanEval, SWE-bench, RepoBench, and MBPP+ confirm state-of-the-art performance among models in its parameter class.
|
||||
|
||||
Future work will focus on deeper integration with distributed version control workflows, formal verification of generated code properties, and tighter coupling with continuous integration systems for real-time feedback during development.
|
||||
|
||||
\begin{thebibliography}{9}
|
||||
\bibitem{codex} Chen, M. et al. (2021). Evaluating Large Language Models Trained on Code. arXiv:2107.03374.
|
||||
\bibitem{alphacode} Li, Y. et al. (2022). Competition-level code generation with AlphaCode. Science, 378(6624).
|
||||
\bibitem{starcoder} Li, R. et al. (2023). StarCoder: May the source be with you. arXiv:2305.06161.
|
||||
\bibitem{codellama} Roziere, B. et al. (2023). Code Llama: Open Foundation Models for Code. arXiv:2308.12950.
|
||||
\bibitem{repocoder} Zhang, F. et al. (2023). RepoCoder: Repository-Level Code Completion Through Iterative Retrieval. arXiv:2303.12570.
|
||||
\bibitem{swebench} Jimenez, C. et al. (2024). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR 2024.
|
||||
\bibitem{agentless} Xia, C. et al. (2024). Agentless: Demystifying LLM-based Software Engineering Agents. arXiv:2407.01489.
|
||||
\bibitem{swebenchagent} Yang, J. et al. (2024). SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. arXiv:2405.15793.
|
||||
\end{thebibliography}
|
||||
|
||||
\end{document}
|
||||
Binary file not shown.
@@ -1,531 +0,0 @@
|
||||
\documentclass[11pt,a4paper]{article}
|
||||
\usepackage[utf8]{inputenc}
|
||||
\usepackage[T1]{fontenc}
|
||||
\usepackage{amsmath,amsfonts,amssymb}
|
||||
\usepackage{graphicx}
|
||||
\usepackage{hyperref}
|
||||
\usepackage{listings}
|
||||
\usepackage{xcolor}
|
||||
\usepackage{booktabs}
|
||||
\usepackage{float}
|
||||
\usepackage{geometry}
|
||||
\geometry{margin=1in}
|
||||
|
||||
\definecolor{zenblue}{RGB}{41,121,255}
|
||||
\definecolor{zengreen}{RGB}{52,199,89}
|
||||
\definecolor{zenorange}{RGB}{255,149,0}
|
||||
\definecolor{codegray}{RGB}{245,245,245}
|
||||
|
||||
\hypersetup{
|
||||
colorlinks=true,
|
||||
linkcolor=zenblue,
|
||||
urlcolor=zenblue,
|
||||
citecolor=zenblue
|
||||
}
|
||||
|
||||
\lstset{
|
||||
backgroundcolor=\color{codegray},
|
||||
basicstyle=\ttfamily\small,
|
||||
breaklines=true,
|
||||
captionpos=b,
|
||||
frame=single,
|
||||
numbers=left,
|
||||
numberstyle=\tiny\color{gray}
|
||||
}
|
||||
|
||||
\title{
|
||||
\vspace{-2cm}
|
||||
\Large \textbf{Zen AI Model Family} \\
|
||||
\vspace{0.5cm}
|
||||
\Huge \textbf{Zen4-Max} \\
|
||||
\vspace{0.3cm}
|
||||
\large Fourth-Generation High-Performance Research Model \\
|
||||
\vspace{0.5cm}
|
||||
\normalsize Technical Whitepaper v2026.01
|
||||
}
|
||||
|
||||
\author{
|
||||
Zach Kelling\thanks{zach@lux.network} \\
|
||||
\texttt{research@hanzo.ai} \\
|
||||
\\
|
||||
Zoo Labs Foundation \\
|
||||
\texttt{foundation@zoolabs.org}
|
||||
}
|
||||
|
||||
\date{January 2026}
|
||||
|
||||
\begin{document}
|
||||
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
We present \textbf{Zen4-Max}, a 480B total parameter Mixture-of-Experts (MoE) model
|
||||
activating 48B parameters per token, optimized for research-grade scientific reasoning,
|
||||
mathematical proof verification, and complex long-horizon analysis. Zen4-Max achieves
|
||||
93.4\% on MMLU, 90.1\% on MATH, 79.8\% on GPQA Diamond, 94.7\% on HumanEval, and
|
||||
61.2\% on FrontierMath, establishing it as the highest-performing model for formal
|
||||
reasoning tasks at the 48B active parameter inference cost. Through a novel
|
||||
\textbf{Proof-Augmented Training} (PAT) methodology that incorporates formal verification
|
||||
signals into the pretraining and post-training pipeline, Zen4-Max demonstrates qualitatively
|
||||
superior performance on tasks requiring logical completeness and self-consistency.
|
||||
\end{abstract}
|
||||
|
||||
\tableofcontents
|
||||
\newpage
|
||||
|
||||
\section{Introduction}
|
||||
|
||||
Scientific progress increasingly depends on AI systems capable of reliably assisting
|
||||
with tasks that require sustained, logically consistent reasoning: deriving mathematical
|
||||
proofs, designing experiments, interpreting empirical results, and synthesizing findings
|
||||
across hundreds of papers. These tasks share a common requirement that distinguishes them
|
||||
from most benchmarks: correctness cannot be approximated by fluency. A mathematical proof
|
||||
is either valid or it is not; a synthesized research hypothesis is either consistent with
|
||||
its supporting evidence or it contradicts it.
|
||||
|
||||
Zen4-Max is designed for this regime. Its primary architectural innovation, Proof-Augmented
|
||||
Training, introduces formal verification signals into both the pretraining corpus and the
|
||||
post-training reward function, teaching the model to distinguish correct chains of reasoning
|
||||
from plausible-but-incorrect ones. The result is measurable improvement on FrontierMath
|
||||
(61.2\%), competition-level mathematics benchmarks, and scientific reasoning tasks that
|
||||
require multi-step argument validation.
|
||||
|
||||
Positioned between Zen4-Pro (72B dense) and Zen4-Ultra (1T MoE), Zen4-Max provides a
|
||||
practical inference cost (48B active parameters, achievable on 4 H100s) while approaching
|
||||
the capability ceiling of the Zen4 family on reasoning-intensive tasks.
|
||||
|
||||
\subsection{Key Contributions}
|
||||
|
||||
\begin{itemize}
|
||||
\item \textbf{Proof-Augmented Training (PAT)}: A training methodology incorporating
|
||||
formal verification signals (lean4, isabelle, and symbolic computation results) into
|
||||
the pretraining corpus and using verifier outputs as post-training rewards.
|
||||
\item \textbf{480B/48B MoE}: 480B total parameter MoE with 48B active, using 64 experts
|
||||
per layer with top-6 routing, optimized for research-domain specialization.
|
||||
\item \textbf{FrontierMath Performance}: 61.2\% on FrontierMath, representing a significant
|
||||
advance on research-mathematics benchmarks that are resistant to pattern matching.
|
||||
\item \textbf{Long-Horizon Coherence}: Maintained logical consistency over outputs
|
||||
exceeding 8,000 tokens, enabling full proof derivations and experimental design documents.
|
||||
\end{itemize}
|
||||
|
||||
\section{Architecture}
|
||||
|
||||
\subsection{Model Specifications}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{ll}
|
||||
\toprule
|
||||
\textbf{Component} & \textbf{Specification} \\
|
||||
\midrule
|
||||
Total Parameters & 480B \\
|
||||
Active Parameters & 48B per token \\
|
||||
Architecture & Mixture of Experts (MoE) \\
|
||||
Number of Layers & 80 \\
|
||||
Dense Base Layers & 24 \\
|
||||
MoE Upper Layers & 56 \\
|
||||
Attention Mechanism & Hierarchical Hybrid Attention (HHA) \\
|
||||
Local Window Size & 16,384 tokens \\
|
||||
Global Anchor Tokens & 2,048 per layer \\
|
||||
Context Length & 131,072 tokens (128K) \\
|
||||
Hidden Dimension & 7,168 \\
|
||||
Experts per MoE Layer & 64 \\
|
||||
Active Experts per Token & 6 (top-6 routing) \\
|
||||
Expert Intermediate Dim & 2,816 \\
|
||||
Attention Heads & 56 \\
|
||||
Key-Value Heads & 8 (GQA) \\
|
||||
Vocabulary Size & 151,936 \\
|
||||
Positional Encoding & RoPE ($\theta = 10{,}000{,}000$) \\
|
||||
Normalization & RMSNorm \\
|
||||
Activation & SwiGLU \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen4-Max Architecture Specifications}
|
||||
\end{table}
|
||||
|
||||
\subsection{Expert Specialization for Research Domains}
|
||||
|
||||
Zen4-Max uses 64 experts per MoE layer organized into four research-domain clusters:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Expert Cluster} & \textbf{Count} & \textbf{Primary Data Sources} \\
|
||||
\midrule
|
||||
Mathematics and formal proof & 16 & ArXiv math, Lean4 libraries, OEIS \\
|
||||
Physical sciences & 14 & ArXiv physics, NIST databases \\
|
||||
Life sciences and chemistry & 12 & PubMed, ChEMBL, protein databases \\
|
||||
Computer science and systems & 14 & ArXiv CS, GitHub, formal specs \\
|
||||
General reasoning & 8 & All domains \\
|
||||
\midrule
|
||||
\textbf{Total} & 64 & --- \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen4-Max Expert Cluster Configuration}
|
||||
\end{table}
|
||||
|
||||
Top-6 routing across 64 experts activates 9.4\% of expert parameters per token.
|
||||
The routing mechanism uses a sigmoid-normalized gating function rather than softmax,
|
||||
which reduces routing sensitivity to outlier logit values and improves stability
|
||||
on long contexts where token distributions shift significantly.
|
||||
|
||||
\subsection{Proof-Augmented Training Architecture}
|
||||
|
||||
PAT operates at two levels: the pretraining corpus and the post-training reward function.
|
||||
|
||||
\subsubsection{Corpus-Level PAT}
|
||||
|
||||
The pretraining corpus includes formal proof corpora that provide ground-truth logical
|
||||
structures:
|
||||
|
||||
\begin{itemize}
|
||||
\item Mathlib4 (Lean4): 250,000+ theorems with complete formal proofs.
|
||||
\item Isabelle Archive of Formal Proofs: 720+ theory developments.
|
||||
\item HOL Light standard library: 12,000+ theorems.
|
||||
\item Symbolic computation results from SymPy and Mathematica (open datasets).
|
||||
\end{itemize}
|
||||
|
||||
These corpora are presented in a unified format that interleaves the formal statement,
|
||||
the tactic proof steps, and the verified status of each step. The model learns to
|
||||
distinguish verified from unverified reasoning by observing this structure.
|
||||
|
||||
\subsubsection{Reward-Level PAT}
|
||||
|
||||
During post-training, mathematical generation tasks use a hybrid reward:
|
||||
|
||||
\begin{equation}
|
||||
R_{\text{PAT}} = \alpha \cdot R_{\text{outcome}} + \beta \cdot R_{\text{verify}} + \gamma \cdot R_{\text{style}}
|
||||
\end{equation}
|
||||
|
||||
where $R_{\text{outcome}}$ is the binary correctness of the final answer, $R_{\text{verify}}$
|
||||
is the output of a learned step-level verification model that scores each reasoning step's
|
||||
logical validity, and $R_{\text{style}}$ penalizes informal hedging language in mathematical
|
||||
contexts (phrases like ``approximately'' or ``should be'' in proof steps). Empirically,
|
||||
$\alpha = 0.6, \beta = 0.3, \gamma = 0.1$ produces optimal FrontierMath performance.
|
||||
|
||||
\section{Training}
|
||||
|
||||
\subsection{Pretraining Data}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Data Category} & \textbf{Proportion} & \textbf{Tokens} \\
|
||||
\midrule
|
||||
Web (quality-filtered) & 30\% & 7.5T \\
|
||||
Code and formal specifications & 25\% & 6.25T \\
|
||||
Scientific literature (ArXiv) & 20\% & 5.0T \\
|
||||
Mathematical content & 12\% & 3.0T \\
|
||||
Formal proofs (Lean4, Isabelle) & 5\% & 1.25T \\
|
||||
Tool-call trajectories & 5\% & 1.25T \\
|
||||
Books and long-form prose & 3\% & 0.75T \\
|
||||
\midrule
|
||||
\textbf{Total} & 100\% & 25.0T \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen4-Max Pretraining Data Composition (25T tokens)}
|
||||
\end{table}
|
||||
|
||||
Formal proof data (5\%) is over-represented relative to its natural occurrence in the web
|
||||
corpus by approximately 50$\times$. This over-representation is critical for PAT: without
|
||||
sufficient formal proof exposure, the verification reward signal cannot be internalized.
|
||||
|
||||
\subsection{Training Infrastructure}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{ll}
|
||||
\toprule
|
||||
\textbf{Parameter} & \textbf{Value} \\
|
||||
\midrule
|
||||
Hardware & 8,192 $\times$ H100 80GB SXM \\
|
||||
Expert Parallelism & 64 (one expert per device) \\
|
||||
Tensor Parallelism & TP=4 \\
|
||||
Pipeline Parallelism & PP=10 \\
|
||||
Data Parallelism & DP=32 \\
|
||||
Global Batch Size & 12M tokens \\
|
||||
Peak Learning Rate & $1 \times 10^{-4}$ \\
|
||||
LR Schedule & Cosine with warmup (2500 steps) \\
|
||||
Optimizer & AdamW ($\beta_1=0.9, \beta_2=0.95$) \\
|
||||
Training Duration & 52 days \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen4-Max Training Infrastructure}
|
||||
\end{table}
|
||||
|
||||
\subsection{Post-Training Stages}
|
||||
|
||||
\begin{enumerate}
|
||||
\item \textbf{SFT} (5M pairs): Heavy emphasis on mathematical derivation and
|
||||
scientific analysis tasks. Includes 200K formal proof demonstrations in natural
|
||||
language augmented with lean4 verification certificates.
|
||||
\item \textbf{PAT-GRPO}: Proof-Augmented GRPO over 80K mathematics tasks with
|
||||
SymPy-verified answers. $G=16$ rollouts per task; the step-level verifier provides
|
||||
intermediate rewards at each reasoning step.
|
||||
\item \textbf{DPO}: 900K comparison pairs, weighted 3:1 toward research tasks.
|
||||
\item \textbf{Safety Alignment}: Standard constitutional AI pass.
|
||||
\end{enumerate}
|
||||
|
||||
\section{Evaluation}
|
||||
|
||||
\subsection{Standard Benchmarks}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
\textbf{Benchmark} & \textbf{Zen3-Max} & \textbf{Zen4-Pro} & \textbf{Zen4-Max} & \textbf{$\Delta$ vs Zen3} \\
|
||||
\midrule
|
||||
MMLU (5-shot) & 87.3\% & 91.8\% & 93.4\% & +6.1\% \\
|
||||
MATH (4-shot) & 84.6\% & 87.6\% & 90.1\% & +5.5\% \\
|
||||
HumanEval (0-shot) & 89.7\% & 93.2\% & 94.7\% & +5.0\% \\
|
||||
GPQA Diamond & 72.1\% & 74.3\% & 79.8\% & +7.7\% \\
|
||||
IFEval & 84.3\% & 88.4\% & 89.1\% & +4.8\% \\
|
||||
BBH (3-shot) & 84.9\% & 89.7\% & 91.4\% & +6.5\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Standard Benchmark Results: Zen4-Max vs. Prior Art}
|
||||
\end{table}
|
||||
|
||||
\subsection{Research and Formal Reasoning}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Benchmark} & \textbf{Zen3-Max} & \textbf{Zen4-Max} \\
|
||||
\midrule
|
||||
FrontierMath (overall) & 48.3\% & 61.2\% \\
|
||||
FrontierMath (Level 1) & 71.4\% & 82.7\% \\
|
||||
FrontierMath (Level 2) & 45.8\% & 59.3\% \\
|
||||
FrontierMath (Level 3) & 27.7\% & 41.6\% \\
|
||||
AIME 2024 (avg 2 sets) & 71.3\% & 81.4\% \\
|
||||
OlympiadBench & 59.2\% & 68.7\% \\
|
||||
ProofWriter (deductive) & 83.4\% & 91.2\% \\
|
||||
FOLIO (first-order logic) & 79.8\% & 87.6\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Research and Formal Reasoning Benchmarks}
|
||||
\end{table}
|
||||
|
||||
\subsection{Scientific Domain Evaluation}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Domain Benchmark} & \textbf{Metric} & \textbf{Zen4-Max} \\
|
||||
\midrule
|
||||
GPQA Diamond (physics) & Accuracy & 81.3\% \\
|
||||
GPQA Diamond (chemistry) & Accuracy & 78.4\% \\
|
||||
GPQA Diamond (biology) & Accuracy & 79.7\% \\
|
||||
SciBench (university-level)& Accuracy & 86.2\% \\
|
||||
MedQA USMLE (4-opt) & Accuracy & 88.7\% \\
|
||||
ClimateBench & Accuracy & 74.3\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Scientific Domain Evaluation}
|
||||
\end{table}
|
||||
|
||||
\subsection{Long-Output Coherence}
|
||||
|
||||
A key capability for research assistance is maintaining logical consistency across
|
||||
long generated outputs. We evaluate this using a custom \textbf{Proof Coherence Score}
|
||||
that measures the rate at which self-contained mathematical arguments in long outputs
|
||||
are free of logical contradictions, verified by a symbolic checker.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Output Length} & \textbf{Zen4-Pro} & \textbf{Zen4-Max} & \textbf{Zen4-Thinking} \\
|
||||
\midrule
|
||||
1K tokens & 94.1\% & 96.8\% & 97.3\% \\
|
||||
4K tokens & 87.3\% & 92.4\% & 95.1\% \\
|
||||
8K tokens & 74.2\% & 86.7\% & 91.8\% \\
|
||||
16K tokens & 61.8\% & 78.3\% & 85.4\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Proof Coherence Score by Output Length}
|
||||
\end{table}
|
||||
|
||||
\section{Related Work}
|
||||
|
||||
FrontierMath \cite{glazer2024frontiermath} provides a benchmark of research-mathematics
|
||||
problems designed to be resistant to pattern matching and memorization, representing a
|
||||
meaningful signal of genuine mathematical reasoning capability. Prior published scores on
|
||||
this benchmark range from single digits for standard instruction-tuned models to the
|
||||
mid-50s for extended-thinking models; Zen4-Max's 61.2\% represents the current best result
|
||||
in our evaluation.
|
||||
|
||||
Formal verification integration into language model training has been explored in the
|
||||
context of Lean-based theorem provers \cite{polu2022formal} and RLHF with verifier reward
|
||||
models \cite{lightman2023let}. PAT extends this by incorporating verification signals at
|
||||
the pretraining stage, not only during post-training, which we find is critical for
|
||||
deep internalization of proof structure.
|
||||
|
||||
Expert specialization in MoE models through domain-partitioned data mixtures was explored
|
||||
in prior work on mixture-of-experts for multilingual models \cite{artetxe2021efficient}.
|
||||
Zen4-Max applies this principle to scientific domains with more granular expert routing.
|
||||
|
||||
\section{Deployment}
|
||||
|
||||
\subsection{Hardware Requirements}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{llll}
|
||||
\toprule
|
||||
\textbf{Format} & \textbf{VRAM} & \textbf{Context} & \textbf{Hardware} \\
|
||||
\midrule
|
||||
FP8 & 4$\times$ H100 80GB & 128K & Recommended \\
|
||||
FP8 & 2$\times$ H100 80GB & 64K & Reduced context \\
|
||||
INT4 (AWQ) & 4$\times$ A100 80GB & 32K & Budget option \\
|
||||
INT4 (AWQ) & 2$\times$ A100 80GB & 16K & Minimal config \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen4-Max Deployment Configurations}
|
||||
\end{table}
|
||||
|
||||
\subsection{Research Workflow Integration}
|
||||
|
||||
\begin{lstlisting}[language=Python, caption=Zen4-Max for Mathematical Proof Assistance]
|
||||
from hanzo import HanzoAI
|
||||
|
||||
client = HanzoAI()
|
||||
|
||||
system_prompt = """You are a mathematical research assistant.
|
||||
When proving theorems, proceed step by step.
|
||||
Each step should follow logically from prior steps or stated axioms.
|
||||
Flag any step requiring an unproved assumption.
|
||||
"""
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="zen4-max",
|
||||
messages=[
|
||||
{"role": "system", "content": system_prompt},
|
||||
{"role": "user", "content": "Prove that sqrt(2) is irrational."}
|
||||
],
|
||||
max_tokens=4096,
|
||||
temperature=0.0, # deterministic for proof generation
|
||||
)
|
||||
\end{lstlisting}
|
||||
|
||||
\section{Safety and Alignment}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Metric} & \textbf{Zen3-Max} & \textbf{Zen4-Max} \\
|
||||
\midrule
|
||||
Harmful content refusal & 96.1\% & 98.4\% \\
|
||||
Over-refusal rate & 5.3\% & 3.1\% \\
|
||||
Jailbreak resistance & 94.8\% & 97.9\% \\
|
||||
TruthfulQA & 76.8\% & 84.7\% \\
|
||||
Scientific claim accuracy & 83.4\% & 89.2\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Safety and Alignment Metrics}
|
||||
\end{table}
|
||||
|
||||
Scientific claim accuracy measures the rate at which the model's scientific assertions
|
||||
match peer-reviewed consensus, evaluated by domain experts on 500 sampled claims per
|
||||
scientific domain. The improvement from 83.4\% to 89.2\% reflects the increased
|
||||
scientific literature proportion in pretraining and the PAT reward signal penalizing
|
||||
unsupported assertions.
|
||||
|
||||
\subsection{Limitations}
|
||||
|
||||
\begin{itemize}
|
||||
\item FrontierMath performance (61.2\%) indicates significant headroom on research-level
|
||||
mathematics; Zen4-Thinking is recommended for tasks where extended reasoning budget
|
||||
is more important than knowledge breadth.
|
||||
\item Formal proof output is not guaranteed to be machine-verifiable without post-processing;
|
||||
the model learns proof style from Lean4 but does not run a lean4 kernel during inference.
|
||||
\item Expert routing is non-deterministic across batch sizes; results may vary slightly
|
||||
when comparing single-item and batched inference.
|
||||
\end{itemize}
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
Zen4-Max establishes the highest standard for research-grade AI assistance at a practical
|
||||
inference cost. Through Proof-Augmented Training, it internalizes formal reasoning
|
||||
structure during pretraining rather than only imitating it during fine-tuning. The 480B/48B
|
||||
Mixture-of-Experts architecture concentrates domain expertise in research clusters that are
|
||||
selectively activated by the routing mechanism for each token. With 90.1\% on MATH,
|
||||
79.8\% on GPQA Diamond, and 61.2\% on FrontierMath, Zen4-Max is the recommended choice
|
||||
for research institutions, quantitative finance, pharmaceutical discovery, and any
|
||||
application where logical completeness and scientific accuracy are primary requirements.
|
||||
|
||||
\section*{Acknowledgments}
|
||||
|
||||
We thank the formal methods research community whose open Lean4 and Isabelle corpora
|
||||
made Proof-Augmented Training possible, the Zoo Labs Foundation science team for
|
||||
domain evaluation design, and the Hanzo AI infrastructure team for the training cluster.
|
||||
|
||||
\begin{thebibliography}{99}
|
||||
\bibitem{vaswani2017attention} Vaswani, A. et al. (2017). Attention Is All You Need. NeurIPS 2017.
|
||||
\bibitem{brown2020language} Brown, T. et al. (2020). Language Models are Few-Shot Learners. NeurIPS 2020.
|
||||
\bibitem{shazeer2017outrageously} Shazeer, N. et al. (2017). Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. ICLR 2017.
|
||||
\bibitem{fedus2022switch} Fedus, W. et al. (2022). Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. JMLR 2022.
|
||||
\bibitem{hoffmann2022training} Hoffmann, J. et al. (2022). Training Compute-Optimal Large Language Models. NeurIPS 2022.
|
||||
\bibitem{hinton2015distilling} Hinton, G., Vinyals, O. and Dean, J. (2015). Distilling the Knowledge in a Neural Network. arXiv:1503.02531.
|
||||
\bibitem{glazer2024frontiermath} Glazer, E. et al. (2024). FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI. arXiv:2411.04872.
|
||||
\bibitem{polu2022formal} Polu, S. et al. (2022). Formal Mathematics Statement Curriculum Learning. ICLR 2023.
|
||||
\bibitem{lightman2023let} Lightman, H. et al. (2023). Let's Verify Step by Step. arXiv:2305.20050.
|
||||
\bibitem{artetxe2021efficient} Artetxe, M. et al. (2021). Efficient Large Scale Language Modeling with Mixtures of Experts. EMNLP 2022.
|
||||
\bibitem{chowdhery2023palm} Chowdhery, A. et al. (2023). PaLM: Scaling Language Modeling with Pathways. JMLR 2023.
|
||||
\bibitem{romero2014fitnets} Romero, A. et al. (2015). FitNets: Hints for Thin Deep Nets. ICLR 2015.
|
||||
\end{thebibliography}
|
||||
|
||||
\appendix
|
||||
|
||||
\section{Model Card}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{ll}
|
||||
\toprule
|
||||
\textbf{Field} & \textbf{Value} \\
|
||||
\midrule
|
||||
Model Name & Zen4-Max \\
|
||||
Version & v2026.01 \\
|
||||
Release Date & January 2026 \\
|
||||
Total Parameters & 480B \\
|
||||
Active Parameters& 48B per token \\
|
||||
Context Length & 131,072 tokens \\
|
||||
License & Apache 2.0 (weights) \\
|
||||
Repository & \href{https://huggingface.co/zenlm/zen4-max}{huggingface.co/zenlm/zen4-max} \\
|
||||
Documentation & \href{https://papers.zenlm.org/zen4-max}{papers.zenlm.org/zen4-max} \\
|
||||
Contact & research@hanzo.ai \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen4-Max Model Card}
|
||||
\end{table}
|
||||
|
||||
\section{PAT Ablation Study}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Training Configuration} & \textbf{MATH} & \textbf{FrontierMath} & \textbf{GPQA} \\
|
||||
\midrule
|
||||
Base (no PAT) & 85.3\% & 48.1\% & 74.8\% \\
|
||||
+ Formal corpus (pretraining) & 87.4\% & 53.6\% & 76.3\% \\
|
||||
+ Step-level reward (GRPO) & 89.1\% & 58.4\% & 78.7\% \\
|
||||
+ Style penalty ($\gamma$ term) & 90.1\% & 61.2\% & 79.8\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{PAT Ablation: Contribution of Each Component}
|
||||
\end{table}
|
||||
|
||||
Each PAT component contributes incrementally. The formal corpus provides the largest
|
||||
gain on FrontierMath (5.5 points), while the step-level reward provides the largest
|
||||
gain on MATH (1.7 points, where intermediate step quality is most important).
|
||||
The style penalty provides a smaller but consistent benefit across all three benchmarks.
|
||||
|
||||
\end{document}
|
||||
Binary file not shown.
@@ -1,566 +0,0 @@
|
||||
\documentclass[11pt,a4paper]{article}
|
||||
\usepackage[utf8]{inputenc}
|
||||
\usepackage[T1]{fontenc}
|
||||
\usepackage{amsmath,amsfonts,amssymb}
|
||||
\usepackage{graphicx}
|
||||
\usepackage{hyperref}
|
||||
\usepackage{listings}
|
||||
\usepackage{xcolor}
|
||||
\usepackage{booktabs}
|
||||
\usepackage{float}
|
||||
\usepackage{geometry}
|
||||
\geometry{margin=1in}
|
||||
|
||||
\definecolor{zenblue}{RGB}{41,121,255}
|
||||
\definecolor{zengreen}{RGB}{52,199,89}
|
||||
\definecolor{zenorange}{RGB}{255,149,0}
|
||||
\definecolor{codegray}{RGB}{245,245,245}
|
||||
|
||||
\hypersetup{
|
||||
colorlinks=true,
|
||||
linkcolor=zenblue,
|
||||
urlcolor=zenblue,
|
||||
citecolor=zenblue
|
||||
}
|
||||
|
||||
\lstset{
|
||||
backgroundcolor=\color{codegray},
|
||||
basicstyle=\ttfamily\small,
|
||||
breaklines=true,
|
||||
captionpos=b,
|
||||
frame=single,
|
||||
numbers=left,
|
||||
numberstyle=\tiny\color{gray}
|
||||
}
|
||||
|
||||
\title{
|
||||
\vspace{-2cm}
|
||||
\Large \textbf{Zen AI Model Family} \\
|
||||
\vspace{0.5cm}
|
||||
\Huge \textbf{Zen4-Mini} \\
|
||||
\vspace{0.3cm}
|
||||
\large Fourth-Generation Compact Model \\
|
||||
\vspace{0.5cm}
|
||||
\normalsize Technical Whitepaper v2026.01
|
||||
}
|
||||
|
||||
\author{
|
||||
Zach Kelling\thanks{zach@lux.network} \\
|
||||
\texttt{research@hanzo.ai} \\
|
||||
\\
|
||||
Zoo Labs Foundation \\
|
||||
\texttt{foundation@zoolabs.org}
|
||||
}
|
||||
|
||||
\date{January 2026}
|
||||
|
||||
\begin{document}
|
||||
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
We present \textbf{Zen4-Mini}, a compact 4B parameter language model that delivers strong
|
||||
instruction-following and reasoning capabilities in a form factor suitable for edge servers,
|
||||
consumer hardware, and cost-sensitive deployment. Zen4-Mini is the direct successor to
|
||||
Zen-Eco and achieves 71.8\% on MMLU, 81.2\% on GSM8K, 74.3\% on HumanEval, and 78.4\%
|
||||
on IFEval, while sustaining 1,200 tokens per second on a single NVIDIA RTX 4090 in INT4
|
||||
quantization. Key advances over its predecessor include a redesigned tokenizer with
|
||||
higher text compression, an optimized attention architecture with 32K native context, and
|
||||
a substantially improved post-training corpus with 3$\times$ more instruction diversity.
|
||||
Zen4-Mini demonstrates that distilled fine-tuning of a Mixture-of-Experts base is highly effective even at compact
|
||||
parameter budgets, closing a large fraction of the gap between 4B and 14B models.
|
||||
\end{abstract}
|
||||
|
||||
\tableofcontents
|
||||
\newpage
|
||||
|
||||
\section{Introduction}
|
||||
|
||||
The majority of AI inference globally does not occur in data centers with multi-GPU servers.
|
||||
It occurs on laptop CPUs, edge inference chips, single-GPU workstations, and consumer mobile
|
||||
hardware. Building capable models that operate within the strict memory and compute budgets
|
||||
of these environments is a distinct engineering challenge from building the largest possible
|
||||
models for maximum benchmark scores.
|
||||
|
||||
Zen4-Mini addresses this challenge by applying the full Zen4 generation technology stack ---
|
||||
the distilled fine-tuning framework, native tool pretraining, improved post-training
|
||||
pipeline --- to the compact 4B parameter class. The result is a model that outperforms
|
||||
its predecessor Zen-Eco by substantial margins across all benchmarks while remaining
|
||||
deployable on consumer hardware:
|
||||
|
||||
\begin{itemize}
|
||||
\item Runs fully in memory on any device with 6 GB VRAM (INT4) or 8 GB (INT8).
|
||||
\item Delivers 1,200 tokens/sec on RTX 4090 (INT4), suitable for real-time applications.
|
||||
\item Fits entirely in Apple unified memory on M2 MacBook Air (8GB) in 4-bit.
|
||||
\item Supports 32K token context for multi-document tasks within consumer memory limits.
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Zen4-Mini vs. Zen-Eco}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Capability} & \textbf{Zen-Eco} & \textbf{Zen4-Mini} \\
|
||||
\midrule
|
||||
Parameters & 4B & 4.1B \\
|
||||
Context Length & 32K & 32K \\
|
||||
MMLU & 62.3\% & 71.8\% \\
|
||||
GSM8K & 74.8\% & 81.2\% \\
|
||||
HumanEval & 35.2\% & 74.3\% \\
|
||||
IFEval & --- & 78.4\% \\
|
||||
Tokens/sec (INT4) & 850 & 1,200 \\
|
||||
INT4 memory & 3.2 GB & 2.8 GB \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen4-Mini vs. Zen-Eco: Generation-over-Generation Improvements}
|
||||
\end{table}
|
||||
|
||||
The most dramatic improvement is HumanEval: 35.2\% to 74.3\% (+39.1 points). This reflects
|
||||
the native tool pretraining shared across the Zen4 generation, which produces dramatically
|
||||
better code understanding even in the compact model class.
|
||||
|
||||
\subsection{Design Principles}
|
||||
|
||||
Zen4-Mini follows three design principles specific to the compact model class:
|
||||
|
||||
\begin{enumerate}
|
||||
\item \textbf{Distillation over scale}: Every parameter must earn its place through
|
||||
effective distillation from larger teachers. No capacity is wasted on redundant
|
||||
representations.
|
||||
\item \textbf{Inference efficiency first}: Architecture choices are governed by
|
||||
inference cost, not training convenience. GQA head ratios, FFN dimensions, and
|
||||
quantization compatibility are primary constraints.
|
||||
\item \textbf{Task breadth over depth}: A 4B model cannot match a 72B model on
|
||||
any individual hard task, but it can handle the long tail of common tasks reliably.
|
||||
Instruction following quality is prioritized over peak reasoning performance.
|
||||
\end{enumerate}
|
||||
|
||||
\section{Architecture}
|
||||
|
||||
\subsection{Model Specifications}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{ll}
|
||||
\toprule
|
||||
\textbf{Component} & \textbf{Specification} \\
|
||||
\midrule
|
||||
Total Parameters & 4.1B \\
|
||||
Active Parameters & 4.1B (dense) \\
|
||||
Architecture & Compact Transformer \\
|
||||
Attention Mechanism & Full sliding-window attention \\
|
||||
Context Length & 32,768 tokens (32K) \\
|
||||
Hidden Dimension & 2,560 \\
|
||||
Intermediate Dimension & 6,912 \\
|
||||
Attention Heads & 32 \\
|
||||
Key-Value Heads & 8 (GQA, 4:1 ratio) \\
|
||||
Layers & 36 \\
|
||||
Vocabulary Size & 151,936 \\
|
||||
Positional Encoding & RoPE ($\theta = 500{,}000$) \\
|
||||
Normalization & RMSNorm \\
|
||||
Activation & SwiGLU \\
|
||||
Tie Embeddings & Yes (input/output weight tying) \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen4-Mini Architecture Specifications}
|
||||
\end{table}
|
||||
|
||||
\subsection{Architectural Choices for Compact Inference}
|
||||
|
||||
\subsubsection{Weight Tying}
|
||||
|
||||
Zen4-Mini ties the input token embedding matrix to the output projection (lm-head),
|
||||
saving 151,936 $\times$ 2,560 $\times$ 2 bytes $\approx$ 740 MB in FP16 at no cost
|
||||
to model quality (ablation shows $<$0.2\% degradation on MMLU).
|
||||
|
||||
\subsubsection{GQA at 4:1 Ratio}
|
||||
|
||||
The 32:8 query-to-KV-head ratio provides a $4\times$ reduction in KV-cache size relative
|
||||
to multi-head attention. At 32K context and FP16, the KV-cache for Zen4-Mini is 1.1 GB
|
||||
versus 4.4 GB for equivalent MHA. This is critical for consumer hardware where total
|
||||
system memory is the binding constraint.
|
||||
|
||||
\subsubsection{Sliding-Window Attention}
|
||||
|
||||
Unlike larger Zen4 models that use the full Hierarchical Hybrid Attention mechanism,
|
||||
Zen4-Mini uses simple sliding-window attention with window size 4,096 and global tokens
|
||||
at positions 0, 512, 1024, ..., every 512 tokens. This is computationally equivalent to
|
||||
HHA but implemented without the cross-attention global phase, reducing code complexity
|
||||
for embedded and edge deployments where custom attention kernels are unavailable.
|
||||
|
||||
\subsubsection{Optimized Tokenizer}
|
||||
|
||||
Zen4-Mini shares the vocabulary of 151,936 tokens with the larger Zen4 models. However,
|
||||
the byte-pair encoding merge list was reoptimized on the Zen4-Mini pretraining corpus with
|
||||
additional emphasis on common code patterns. The new tokenizer achieves 4.2 characters
|
||||
per token on English text (vs 3.9 for the Zen-Eco tokenizer) and 5.8 characters per token
|
||||
on Python code (vs 5.1), reducing the effective token length of inputs by 7--12\% and
|
||||
correspondingly improving throughput.
|
||||
|
||||
\subsection{Distilled Fine-Tuning at 4B}
|
||||
|
||||
At 4B parameters, distillation presents a capacity mismatch challenge: the teacher models
|
||||
used for larger Zen4 models have individual layers that are larger than the entire student.
|
||||
Zen4-Mini uses a layer-grouped distillation strategy:
|
||||
|
||||
\begin{itemize}
|
||||
\item The 80-layer Zen4-Pro teacher's layers are grouped into 36 target groups.
|
||||
\item Each student layer is distilled from the weighted average hidden state of its
|
||||
assigned teacher layer group, with weights determined by a learned alignment network
|
||||
trained jointly.
|
||||
\item The alignment network is discarded after distillation; only the student weights
|
||||
are retained.
|
||||
\end{itemize}
|
||||
|
||||
This approach outperforms both direct output-logit distillation (+4.1\% MMLU) and simple
|
||||
layer-to-layer distillation from a shallower intermediate teacher (+2.3\% MMLU).
|
||||
|
||||
\section{Training}
|
||||
|
||||
\subsection{Pretraining Data}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Data Category} & \textbf{Proportion} & \textbf{Tokens} \\
|
||||
\midrule
|
||||
Web (quality-filtered) & 45\% & 4.5T \\
|
||||
Code (multi-language) & 25\% & 2.5T \\
|
||||
Instruction and dialogue & 15\% & 1.5T \\
|
||||
Scientific and technical & 8\% & 0.8T \\
|
||||
Mathematical content & 4\% & 0.4T \\
|
||||
Books and long-form prose & 3\% & 0.3T \\
|
||||
\midrule
|
||||
\textbf{Total} & 100\% & 10.0T \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen4-Mini Pretraining Data Composition (10T tokens)}
|
||||
\end{table}
|
||||
|
||||
Instruction and dialogue data (15\%) is proportionally higher than in larger models (typically
|
||||
8--12\%) because the compact model benefits from more explicit instruction-following signal
|
||||
in pretraining, compensating for the reduced feed-forward capacity that would otherwise
|
||||
abstract this from raw text.
|
||||
|
||||
\subsection{Training Configuration}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{ll}
|
||||
\toprule
|
||||
\textbf{Parameter} & \textbf{Value} \\
|
||||
\midrule
|
||||
Hardware & 512 $\times$ H100 80GB SXM \\
|
||||
Parallelism & TP=2, DP=256 \\
|
||||
Batch Size & 2M tokens (global) \\
|
||||
Peak Learning Rate & $5 \times 10^{-4}$ \\
|
||||
LR Schedule & Cosine with linear warmup (1000 steps) \\
|
||||
Optimizer & AdamW ($\beta_1=0.9, \beta_2=0.95$) \\
|
||||
Weight Decay & 0.1 \\
|
||||
Gradient Clipping & 1.0 \\
|
||||
Precision & BF16 \\
|
||||
Training Duration & 8 days \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen4-Mini Training Configuration}
|
||||
\end{table}
|
||||
|
||||
The higher peak learning rate ($5 \times 10^{-4}$ vs $3 \times 10^{-4}$ for Zen4) is
|
||||
consistent with the finding that compact models benefit from higher learning rates due
|
||||
to their reduced capacity requiring more aggressive parameter updates to converge.
|
||||
|
||||
\subsection{Post-Training}
|
||||
|
||||
\begin{enumerate}
|
||||
\item \textbf{SFT} (800K pairs): Instruction diversity is the primary emphasis.
|
||||
The instruction taxonomy covers 8,400 task types, ensuring broad coverage of
|
||||
real-world requests. Responses are capped at 2,048 tokens to match the typical
|
||||
output length in compact-model deployments.
|
||||
\item \textbf{DPO} (300K comparison pairs): Preference data generated by Zen4-Pro
|
||||
acting as judge, scoring responses from Zen4-Mini SFT checkpoint. The mini-model
|
||||
bootstraps its preferences from a more capable judge, a key advantage of the
|
||||
Zen family's tiered architecture.
|
||||
\item \textbf{Safety Alignment}: Lightweight PPO with a 1B safety reward model
|
||||
that matches the parameter budget of the student.
|
||||
\end{enumerate}
|
||||
|
||||
\section{Evaluation}
|
||||
|
||||
\subsection{Standard Benchmarks}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Benchmark} & \textbf{Zen-Eco (4B)} & \textbf{Zen4-Mini (4B)} & \textbf{Improvement} \\
|
||||
\midrule
|
||||
MMLU (5-shot) & 62.3\% & 71.8\% & +9.5\% \\
|
||||
GSM8K (8-shot) & 74.8\% & 81.2\% & +6.4\% \\
|
||||
HumanEval (0-shot) & 35.2\% & 74.3\% & +39.1\% \\
|
||||
HellaSwag (10-shot) & 71.6\% & 78.9\% & +7.3\% \\
|
||||
IFEval & --- & 78.4\% & --- \\
|
||||
MBPP (pass@1) & --- & 69.8\% & --- \\
|
||||
ARC-Challenge & 57.3\% & 64.8\% & +7.5\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen4-Mini vs. Zen-Eco Benchmark Results}
|
||||
\end{table}
|
||||
|
||||
\subsection{Instruction-Following Quality}
|
||||
|
||||
Instruction following is a primary design target for Zen4-Mini. We evaluate across
|
||||
several instruction-following benchmarks:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Benchmark} & \textbf{Metric} & \textbf{Zen4-Mini} \\
|
||||
\midrule
|
||||
IFEval (prompt-strict) & Accuracy & 78.4\% \\
|
||||
IFEval (instruction-strict) & Accuracy & 83.1\% \\
|
||||
AlpacaEval 2.0 & Win rate & 67.3\% \\
|
||||
MT-Bench & Score/10 & 7.84 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Instruction-Following Evaluation}
|
||||
\end{table}
|
||||
|
||||
\subsection{Efficiency Benchmarks}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Configuration} & \textbf{Hardware} & \textbf{Tokens/sec} & \textbf{Memory} \\
|
||||
\midrule
|
||||
BF16 (full) & RTX 4090 24GB & 450 & 8.2 GB \\
|
||||
INT8 & RTX 4090 24GB & 820 & 4.3 GB \\
|
||||
INT4 (Q4\_K\_M) & RTX 4090 24GB & 1,200 & 2.8 GB \\
|
||||
INT4 (Q4\_K\_M) & RTX 3060 12GB & 640 & 2.8 GB \\
|
||||
MLX 4-bit & M4 (16GB) & 980 & 2.8 GB \\
|
||||
MLX 4-bit & M2 (8GB) & 560 & 2.8 GB \\
|
||||
CPU (Q4\_K\_M) & Intel i9-14900K & 42 & 2.8 GB \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen4-Mini Throughput Across Hardware Configurations}
|
||||
\end{table}
|
||||
|
||||
\subsection{Comparison to 14B Tier}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Benchmark} & \textbf{Zen4-Mini (4B)} & \textbf{Zen4 (14B)} \\
|
||||
\midrule
|
||||
MMLU & 71.8\% & 85.4\% \\
|
||||
GSM8K & 81.2\% & 93.7\% \\
|
||||
HumanEval & 74.3\% & 88.3\% \\
|
||||
IFEval & 78.4\% & 83.1\% \\
|
||||
Tokens/sec (INT4, RTX 4090) & 1,200 & 390 \\
|
||||
INT4 VRAM & 2.8 GB & 7.8 GB \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen4-Mini vs. Zen4 (14B): Capability-Efficiency Trade-off}
|
||||
\end{table}
|
||||
|
||||
Zen4-Mini closes roughly 60--70\% of the gap between Zen-Eco and Zen4 on knowledge
|
||||
benchmarks (MMLU), while closing more than 95\% of the code gap (HumanEval). This
|
||||
asymmetry reflects the disproportionate impact of native tool pretraining on code
|
||||
capability, which transfers highly efficiently through distillation.
|
||||
|
||||
\section{Related Work}
|
||||
|
||||
Compact language models have received sustained attention since it became clear that
|
||||
models under 10B parameters can serve the majority of real-world use cases if trained
|
||||
carefully. TinyLlama \cite{zhang2024tinyllama} demonstrated that 1.1B models trained on
|
||||
3T tokens could be surprisingly capable. Phi-series models \cite{gunasekar2023textbooks}
|
||||
showed that data quality (synthetic ``textbook'' data) matters more than data quantity
|
||||
for compact models. Zen4-Mini builds on both insights: the 10T pretraining run uses
|
||||
a higher-quality filtered corpus than prior compact models, and the distilled fine-tuning
|
||||
framework serves as a structured quality amplifier on top of the base data.
|
||||
|
||||
SmolLM \cite{allal2024smollm} and similar efforts have pushed the efficiency frontier
|
||||
for sub-2B models, establishing that the sub-10B space has substantial room for
|
||||
improvement through better training recipes rather than architectural novelty.
|
||||
Zen4-Mini's results confirm this: the primary driver of improvement over Zen-Eco is
|
||||
the training recipe (distillation, post-training quality, native tool pretraining),
|
||||
not architectural changes.
|
||||
|
||||
\section{Deployment}
|
||||
|
||||
\subsection{Supported Formats and Platforms}
|
||||
|
||||
\begin{itemize}
|
||||
\item \textbf{HuggingFace}: BF16 and FP16 safetensors
|
||||
\item \textbf{GGUF}: Q2\_K (1.8 GB), Q4\_K\_M (2.8 GB), Q5\_K\_M (3.2 GB), Q8\_0 (4.3 GB)
|
||||
\item \textbf{MLX}: 4-bit and 8-bit for Apple Silicon (via mlx-lm)
|
||||
\item \textbf{ONNX}: FP16 and INT8 for cross-platform inference engines
|
||||
\item \textbf{llama.cpp}: All GGUF variants supported natively
|
||||
\item \textbf{Ollama}: Available as \texttt{ollama pull zenlm/zen4-mini}
|
||||
\item \textbf{LM Studio}: Available in model library
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Recommended Use Cases}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lll}
|
||||
\toprule
|
||||
\textbf{Use Case} & \textbf{Recommended Config} & \textbf{Hardware} \\
|
||||
\midrule
|
||||
Edge server inference & INT8 & Single A10G \\
|
||||
Developer workstation & Q4\_K\_M & RTX 3060+ \\
|
||||
Apple Silicon laptop & MLX 4-bit & M2 Air+ \\
|
||||
Embedded AI appliance & Q2\_K & Jetson AGX Orin \\
|
||||
Mobile/web (WASM) & Q2\_K & Browser via WebGPU \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Recommended Deployment Configurations by Use Case}
|
||||
\end{table}
|
||||
|
||||
\subsection{Integration Example}
|
||||
|
||||
\begin{lstlisting}[language=bash, caption=Running Zen4-Mini with Ollama]
|
||||
# Pull and run
|
||||
ollama pull zenlm/zen4-mini
|
||||
ollama run zenlm/zen4-mini
|
||||
|
||||
# Or via API
|
||||
curl http://localhost:11434/api/generate \
|
||||
-d '{"model": "zenlm/zen4-mini",
|
||||
"prompt": "Explain recursion in Python with an example.",
|
||||
"stream": false}'
|
||||
\end{lstlisting}
|
||||
|
||||
\begin{lstlisting}[language=Python, caption=Zen4-Mini via Hanzo API]
|
||||
from hanzo import HanzoAI
|
||||
|
||||
client = HanzoAI()
|
||||
|
||||
# Zen4-Mini: fast, cost-efficient, good for high-throughput applications
|
||||
response = client.chat.completions.create(
|
||||
model="zen4-mini",
|
||||
messages=[{"role": "user", "content": "Summarize this article in 3 bullet points."}],
|
||||
max_tokens=256
|
||||
)
|
||||
\end{lstlisting}
|
||||
|
||||
\section{Safety and Alignment}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Safety Metric} & \textbf{Zen-Eco} & \textbf{Zen4-Mini} \\
|
||||
\midrule
|
||||
Harmful content refusal & 89.4\% & 95.8\% \\
|
||||
Over-refusal rate & 12.1\% & 5.7\% \\
|
||||
Jailbreak resistance & 82.3\% & 93.4\% \\
|
||||
TruthfulQA & 58.7\% & 67.3\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Safety Comparison: Zen-Eco vs. Zen4-Mini}
|
||||
\end{table}
|
||||
|
||||
Compact models have historically had weaker safety properties than larger models due to
|
||||
limited capacity for both capability and alignment objectives. Zen4-Mini improves on
|
||||
Zen-Eco across all safety metrics, partly through better post-training data and partly
|
||||
through distilling safety behaviors from the larger Zen4 models.
|
||||
|
||||
\subsection{Limitations}
|
||||
|
||||
\begin{itemize}
|
||||
\item Complex multi-step mathematical reasoning is best handled by Zen4 or Zen4-Pro;
|
||||
Zen4-Mini's GSM8K score (81.2\%) does not generalize to competition-level math.
|
||||
\item Long-context performance beyond 16K tokens degrades relative to full-context
|
||||
models, as the sliding-window attention limits long-range information propagation.
|
||||
\item The model is not recommended for applications where hallucination risk is
|
||||
critical; larger models in the Zen4 family have substantially better TruthfulQA scores.
|
||||
\end{itemize}
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
Zen4-Mini demonstrates that the Zen4 generation's architectural and training advances
|
||||
translate directly into compact model improvements, without requiring scale. The 39-point
|
||||
HumanEval improvement from Zen-Eco is the most striking evidence that native tool
|
||||
pretraining is architecture-agnostic: even at 4B parameters, grounding code understanding
|
||||
in pretraining rather than fine-tuning produces qualitatively better results. With 1,200
|
||||
tokens per second on consumer hardware, 2.8 GB INT4 footprint, and broad format support
|
||||
including Ollama and llama.cpp, Zen4-Mini is the recommended choice for any deployment
|
||||
where inference cost or hardware availability is a primary constraint.
|
||||
|
||||
\section*{Acknowledgments}
|
||||
|
||||
We thank the llama.cpp, mlx-lm, and Ollama communities whose work makes compact model
|
||||
deployment practical, and the Hanzo AI team for the distillation infrastructure.
|
||||
|
||||
\begin{thebibliography}{99}
|
||||
\bibitem{vaswani2017attention} Vaswani, A. et al. (2017). Attention Is All You Need. NeurIPS 2017.
|
||||
\bibitem{brown2020language} Brown, T. et al. (2020). Language Models are Few-Shot Learners. NeurIPS 2020.
|
||||
\bibitem{ouyang2022training} Ouyang, L. et al. (2022). Training Language Models to Follow Instructions with Human Feedback. NeurIPS 2022.
|
||||
\bibitem{touvron2023llama} Touvron, H. et al. (2023). LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971.
|
||||
\bibitem{hoffmann2022training} Hoffmann, J. et al. (2022). Training Compute-Optimal Large Language Models. NeurIPS 2022.
|
||||
\bibitem{dettmers2023qlora} Dettmers, T. et al. (2023). QLoRA: Efficient Finetuning of Quantized Language Models. NeurIPS 2023.
|
||||
\bibitem{hu2021lora} Hu, E. et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022.
|
||||
\bibitem{frantar2022gptq} Frantar, E. et al. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. ICLR 2023.
|
||||
\bibitem{hinton2015distilling} Hinton, G., Vinyals, O. and Dean, J. (2015). Distilling the Knowledge in a Neural Network. arXiv:1503.02531.
|
||||
\bibitem{raffel2020exploring} Raffel, C. et al. (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. JMLR 2020.
|
||||
\bibitem{zhang2024tinyllama} Zhang, P. et al. (2024). TinyLlama: An Open-Source Small Language Model. arXiv:2401.02385.
|
||||
\bibitem{gunasekar2023textbooks} Gunasekar, S. et al. (2023). Textbooks Are All You Need. arXiv:2306.11644.
|
||||
\bibitem{allal2024smollm} Allal, L.B. et al. (2024). SmolLM: A Family of Small Language Models. arXiv:2407.05483.
|
||||
\end{thebibliography}
|
||||
|
||||
\appendix
|
||||
|
||||
\section{Model Card}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{ll}
|
||||
\toprule
|
||||
\textbf{Field} & \textbf{Value} \\
|
||||
\midrule
|
||||
Model Name & Zen4-Mini \\
|
||||
Version & v2026.01 \\
|
||||
Release Date & January 2026 \\
|
||||
Parameters & 4.1B \\
|
||||
Context Length & 32,768 tokens \\
|
||||
License & Apache 2.0 \\
|
||||
HuggingFace & \href{https://huggingface.co/zenlm/zen4-mini-instruct}{huggingface.co/zenlm/zen4-mini-instruct} \\
|
||||
Ollama & \href{https://ollama.com/library/zenlm/zen4-mini}{ollama.com/library/zenlm/zen4-mini} \\
|
||||
Documentation & \href{https://papers.zenlm.org/zen4-mini}{papers.zenlm.org/zen4-mini} \\
|
||||
Contact & research@hanzo.ai \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen4-Mini Model Card}
|
||||
\end{table}
|
||||
|
||||
\section{Quantization Quality Retention}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Quantization} & \textbf{MMLU} & \textbf{HumanEval} & \textbf{GSM8K} \\
|
||||
\midrule
|
||||
BF16 (reference) & 71.8\% & 74.3\% & 81.2\% \\
|
||||
INT8 & 71.6\% & 74.1\% & 81.0\% \\
|
||||
Q5\_K\_M & 71.3\% & 73.8\% & 80.6\% \\
|
||||
Q4\_K\_M & 70.9\% & 73.2\% & 79.8\% \\
|
||||
Q2\_K & 68.4\% & 69.7\% & 76.3\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Benchmark Performance by Quantization Level}
|
||||
\end{table}
|
||||
|
||||
Zen4-Mini retains 99--100\% of BF16 benchmark performance through Q4\_K\_M and 95--96\%
|
||||
through Q2\_K. The consistent retention across quantization levels reflects the
|
||||
architecture's low sensitivity to weight precision, partly due to RMSNorm (which avoids
|
||||
the weight-scale sensitivity issues of LayerNorm) and the symmetric distribution of
|
||||
weight values encouraged by the training recipe.
|
||||
|
||||
\end{document}
|
||||
Binary file not shown.
@@ -1,517 +0,0 @@
|
||||
\documentclass[11pt,a4paper]{article}
|
||||
\usepackage[utf8]{inputenc}
|
||||
\usepackage[T1]{fontenc}
|
||||
\usepackage{amsmath,amsfonts,amssymb}
|
||||
\usepackage{graphicx}
|
||||
\usepackage{hyperref}
|
||||
\usepackage{listings}
|
||||
\usepackage{xcolor}
|
||||
\usepackage{booktabs}
|
||||
\usepackage{float}
|
||||
\usepackage{geometry}
|
||||
\geometry{margin=1in}
|
||||
|
||||
\definecolor{zenblue}{RGB}{41,121,255}
|
||||
\definecolor{zengreen}{RGB}{52,199,89}
|
||||
\definecolor{zenorange}{RGB}{255,149,0}
|
||||
\definecolor{codegray}{RGB}{245,245,245}
|
||||
|
||||
\hypersetup{
|
||||
colorlinks=true,
|
||||
linkcolor=zenblue,
|
||||
urlcolor=zenblue,
|
||||
citecolor=zenblue
|
||||
}
|
||||
|
||||
\lstset{
|
||||
backgroundcolor=\color{codegray},
|
||||
basicstyle=\ttfamily\small,
|
||||
breaklines=true,
|
||||
captionpos=b,
|
||||
frame=single,
|
||||
numbers=left,
|
||||
numberstyle=\tiny\color{gray}
|
||||
}
|
||||
|
||||
\title{
|
||||
\vspace{-2cm}
|
||||
\Large \textbf{Zen AI Model Family} \\
|
||||
\vspace{0.5cm}
|
||||
\Huge \textbf{Zen4-Pro} \\
|
||||
\vspace{0.3cm}
|
||||
\large Fourth-Generation Professional Model \\
|
||||
\vspace{0.5cm}
|
||||
\normalsize Technical Whitepaper v2026.01
|
||||
}
|
||||
|
||||
\author{
|
||||
Zach Kelling\thanks{zach@lux.network} \\
|
||||
\texttt{research@hanzo.ai} \\
|
||||
\\
|
||||
Zoo Labs Foundation \\
|
||||
\texttt{foundation@zoolabs.org}
|
||||
}
|
||||
|
||||
\date{January 2026}
|
||||
|
||||
\begin{document}
|
||||
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
We present \textbf{Zen4-Pro}, the flagship professional model of the Zen4 generation at 72B dense
|
||||
parameters. Zen4-Pro advances the state of agentic AI through deep reasoning chains, native
|
||||
agentic workflows with structured plan-execute-verify loops, and improved tool orchestration
|
||||
across multi-step tasks. The model achieves 91.8\% on MMLU, 87.6\% on MATH, 93.2\% on
|
||||
HumanEval, 74.3\% on GPQA Diamond, and 48.7\% on SWE-bench Verified, establishing new
|
||||
state-of-the-art results for the 72B dense parameter class. Zen4-Pro is designed for
|
||||
enterprise deployment scenarios requiring reliable, multi-step autonomous task execution
|
||||
while maintaining practical inference efficiency on standard data-center hardware.
|
||||
\end{abstract}
|
||||
|
||||
\tableofcontents
|
||||
\newpage
|
||||
|
||||
\section{Introduction}
|
||||
|
||||
Professional-tier AI deployment presents a distinct challenge: tasks are rarely single-turn
|
||||
question-answer exchanges. Software engineers debug across dozens of files; data analysts chain
|
||||
transformations through complex pipelines; legal professionals trace arguments across hundreds
|
||||
of documents. These workflows require a model that can sustain coherent intent, manage tool
|
||||
state, and self-verify intermediate results across many sequential steps.
|
||||
|
||||
Zen4-Pro is purpose-built for this regime. While sharing the core Hierarchical Hybrid Attention
|
||||
architecture with Zen4 (14B), Zen4-Pro's 72B parameter count provides substantially deeper
|
||||
feed-forward capacity, enabling richer world-model representations that support multi-step
|
||||
planning. The model's post-training pipeline introduces a dedicated \textbf{Agentic Alignment}
|
||||
phase that teaches plan-execute-verify patterns from real engineering task traces, not synthetic
|
||||
dialogues.
|
||||
|
||||
The result is a model that scores 48.7\% on SWE-bench Verified --- the industry-standard
|
||||
evaluation for autonomous software engineering --- while matching or exceeding models of greater
|
||||
scale on standard reasoning and knowledge benchmarks.
|
||||
|
||||
\subsection{Key Contributions}
|
||||
|
||||
\begin{itemize}
|
||||
\item \textbf{Agentic Alignment Training}: A new post-training stage using real engineering
|
||||
task traces with ground-truth verifiable outcomes as the reward signal.
|
||||
\item \textbf{Hierarchical Planning}: An internal representation of task state that persists
|
||||
across tool calls, enabling reliable multi-step execution without prompt re-injection.
|
||||
\item \textbf{Self-Verification}: The model learns to check its own intermediate outputs
|
||||
against stated task requirements before proceeding, reducing error propagation in long chains.
|
||||
\item \textbf{Frontier Benchmark Results}: New state-of-the-art at 72B dense for SWE-bench,
|
||||
GPQA, and MATH.
|
||||
\end{itemize}
|
||||
|
||||
\section{Architecture}
|
||||
|
||||
\subsection{Model Specifications}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{ll}
|
||||
\toprule
|
||||
\textbf{Component} & \textbf{Specification} \\
|
||||
\midrule
|
||||
Total Parameters & 72.7B \\
|
||||
Active Parameters & 72.7B (dense) \\
|
||||
Architecture & Hybrid Transformer \\
|
||||
Attention Mechanism & Hierarchical Hybrid Attention (HHA) \\
|
||||
Local Window Size & 8,192 tokens \\
|
||||
Global Anchor Tokens & 1,024 per layer \\
|
||||
Context Length & 131,072 tokens (128K) \\
|
||||
Hidden Dimension & 8,192 \\
|
||||
Intermediate Dimension & 29,568 \\
|
||||
Attention Heads & 64 \\
|
||||
Key-Value Heads & 8 (GQA) \\
|
||||
Layers & 80 \\
|
||||
Vocabulary Size & 151,936 \\
|
||||
Positional Encoding & RoPE ($\theta = 10{,}000{,}000$) \\
|
||||
Normalization & RMSNorm \\
|
||||
Activation & SwiGLU \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen4-Pro Architecture Specifications}
|
||||
\end{table}
|
||||
|
||||
\subsection{Hierarchical Hybrid Attention at 72B}
|
||||
|
||||
Zen4-Pro scales the HHA mechanism from Zen4 with a larger local window ($w = 8192$ vs 4096)
|
||||
and more global anchor tokens ($k = 1024$ vs 512). The larger window improves performance on
|
||||
tasks requiring dense local reasoning, such as code editing within a single large file.
|
||||
The additional anchors improve global coherence for tasks that span many documents.
|
||||
|
||||
At 72B parameters, the feed-forward networks can represent substantially richer world-model
|
||||
priors. Ablation studies show that the improvement from Zen4 to Zen4-Pro on complex planning
|
||||
tasks ($+18\%$ on AgentBench) is primarily attributable to feed-forward capacity rather than
|
||||
attention mechanism differences, confirming that scale continues to benefit planning tasks even
|
||||
after architectural improvements have maximized attention efficiency.
|
||||
|
||||
\subsection{Agentic State Representation}
|
||||
|
||||
Zen4-Pro introduces an explicit task-state representation within the context window. During
|
||||
agentic post-training, the model learns to maintain a structured scratchpad following the
|
||||
format:
|
||||
|
||||
\begin{lstlisting}[caption=Agentic State Token Format]
|
||||
<task_state>
|
||||
<goal>Refactor authentication module to use JWT</goal>
|
||||
<completed>
|
||||
- Read auth/session.py (lines 1-340)
|
||||
- Identified 3 session-creation call sites
|
||||
- Updated auth/session.py with JWT logic
|
||||
</completed>
|
||||
<pending>
|
||||
- Update call sites in api/routes.py, api/middleware.py
|
||||
- Run test suite to verify no regressions
|
||||
</pending>
|
||||
<open_questions>
|
||||
- Token expiry: use 24h or match existing session TTL?
|
||||
</open_questions>
|
||||
</task_state>
|
||||
\end{lstlisting}
|
||||
|
||||
This structured scratchpad is generated and updated by the model itself; it is not injected
|
||||
externally. The model learns to parse its own prior state tokens as factual context, enabling
|
||||
reliable task tracking over 20+ sequential tool calls without context-window management by
|
||||
the orchestration layer.
|
||||
|
||||
\subsection{Distilled Fine-Tuning at Professional Scale}
|
||||
|
||||
Zen4-Pro is distilled from an ensemble of Mixture-of-Experts teacher models using the fourth-generation
|
||||
distillation framework. At 72B, the student has sufficient capacity to absorb richer intermediate
|
||||
representations, enabling distillation from multiple specialized teachers simultaneously:
|
||||
|
||||
\begin{itemize}
|
||||
\item A mathematics specialist teacher contributes to layers 20--40.
|
||||
\item A code reasoning teacher contributes to layers 40--60.
|
||||
\item A general instruction-following teacher provides global signal across all layers.
|
||||
\end{itemize}
|
||||
|
||||
Multi-teacher distillation with layer-targeted routing improves MATH by 3.1\% and HumanEval
|
||||
by 2.8\% over single-teacher distillation at matched compute.
|
||||
|
||||
\section{Training}
|
||||
|
||||
\subsection{Pretraining}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Data Category} & \textbf{Proportion} & \textbf{Tokens} \\
|
||||
\midrule
|
||||
Web (quality-filtered) & 38\% & 7.6T \\
|
||||
Code (multi-language) & 32\% & 6.4T \\
|
||||
Scientific and technical & 14\% & 2.8T \\
|
||||
Tool-call trajectories & 8\% & 1.6T \\
|
||||
Mathematical content & 5\% & 1.0T \\
|
||||
Books and long-form prose & 3\% & 0.6T \\
|
||||
\midrule
|
||||
\textbf{Total} & 100\% & 20.0T \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen4-Pro Pretraining Data Composition (20T tokens)}
|
||||
\end{table}
|
||||
|
||||
Zen4-Pro pretrained on 20 trillion tokens. The increased code proportion (32\% vs 28\% in Zen4)
|
||||
reflects the professional-tier emphasis on software engineering tasks. Scientific and technical
|
||||
text proportion is also increased (14\% vs 10\%) for the deep-reasoning requirements of
|
||||
professional domains.
|
||||
|
||||
\subsection{Post-Training Pipeline}
|
||||
|
||||
\subsubsection{Supervised Fine-Tuning}
|
||||
|
||||
The SFT stage for Zen4-Pro uses 4.5M instruction-response pairs, approximately twice the
|
||||
Zen4 volume. The additional data is concentrated in:
|
||||
\begin{itemize}
|
||||
\item Multi-step software engineering tasks with execution traces.
|
||||
\item Multi-document analysis and synthesis tasks.
|
||||
\item Long-horizon planning and decomposition tasks.
|
||||
\end{itemize}
|
||||
|
||||
\subsubsection{Agentic Alignment}
|
||||
|
||||
The agentic alignment stage is unique to Zen4-Pro and Zen4-Max. It trains the model on
|
||||
trajectories collected from real engineering tasks in sandboxed environments:
|
||||
|
||||
\begin{enumerate}
|
||||
\item \textbf{Trajectory Collection}: Deploy an early checkpoint on 50,000 real GitHub
|
||||
issues in isolated container environments. Collect all tool calls, intermediate states,
|
||||
and final outcomes (pass/fail against the issue's test suite).
|
||||
\item \textbf{Outcome Filtering}: Retain trajectories where the final outcome is
|
||||
verifiably correct (test suite passes). Discard partial successes.
|
||||
\item \textbf{GRPO Training}: Use Group Relative Policy Optimization with the binary
|
||||
test-pass signal as reward. Generate $G = 8$ trajectories per task and train on the
|
||||
relative reward within each group.
|
||||
\item \textbf{Iteration}: Repeat trajectory collection and training for 3 rounds
|
||||
using the updated model.
|
||||
\end{enumerate}
|
||||
|
||||
This iterative process yields a 14.2-point improvement on SWE-bench Verified over the
|
||||
SFT-only baseline.
|
||||
|
||||
\subsubsection{Preference Optimization}
|
||||
|
||||
DPO training uses 1.2M comparison pairs. For professional-tier use cases, comparison pairs
|
||||
are weighted to emphasize:
|
||||
\begin{itemize}
|
||||
\item Correctness of multi-step plans (3$\times$ weight).
|
||||
\item Appropriate tool selection and argument accuracy (2$\times$ weight).
|
||||
\item Response clarity and actionability (1$\times$ weight).
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Training Infrastructure}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{ll}
|
||||
\toprule
|
||||
\textbf{Parameter} & \textbf{Value} \\
|
||||
\midrule
|
||||
Hardware & 4,096 $\times$ H100 80GB SXM \\
|
||||
Parallelism & TP=8, PP=8, DP=64 \\
|
||||
Batch Size & 8M tokens (global) \\
|
||||
Peak Learning Rate & $1.5 \times 10^{-4}$ \\
|
||||
LR Schedule & Cosine with warmup (3000 steps) \\
|
||||
Optimizer & AdamW ($\beta_1=0.9, \beta_2=0.95$) \\
|
||||
Training Duration & 32 days (pretraining) \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen4-Pro Training Infrastructure}
|
||||
\end{table}
|
||||
|
||||
\section{Evaluation}
|
||||
|
||||
\subsection{Standard Benchmarks}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Benchmark} & \textbf{Zen3-Pro} & \textbf{Zen4} & \textbf{Zen4-Pro} \\
|
||||
\midrule
|
||||
MMLU (5-shot) & 85.7\% & 85.4\% & 91.8\% \\
|
||||
MATH (4-shot) & 82.1\% & 81.2\% & 87.6\% \\
|
||||
HumanEval (0-shot) & 86.4\% & 88.3\% & 93.2\% \\
|
||||
GPQA Diamond (0-shot) & 65.8\% & 67.4\% & 74.3\% \\
|
||||
IFEval (prompt-strict) & 80.2\% & 83.1\% & 88.4\% \\
|
||||
GSM8K (8-shot) & 92.3\% & 93.7\% & 96.8\% \\
|
||||
BBH (3-shot) & 81.4\% & 83.9\% & 89.7\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Benchmark Comparison Across Generations and Tiers}
|
||||
\end{table}
|
||||
|
||||
\subsection{Agentic Benchmarks}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
\textbf{Benchmark} & \textbf{Zen3-Pro} & \textbf{Zen4} & \textbf{Zen4-Pro} \\
|
||||
\midrule
|
||||
SWE-bench Verified & 32.1\% & 34.8\% & 48.7\% \\
|
||||
AgentBench (overall) & 58.3\% & 63.1\% & 76.4\% \\
|
||||
WebArena (overall) & 31.4\% & 35.2\% & 44.8\% \\
|
||||
ToolBench (success rate) & 67.2\% & 79.4\% & 88.1\% \\
|
||||
GAIA (Level 1) & 71.3\% & 74.8\% & 83.6\% \\
|
||||
GAIA (Level 2) & 48.9\% & 52.3\% & 63.7\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Agentic Capability Benchmarks}
|
||||
\end{table}
|
||||
|
||||
\subsection{Multi-Step Reasoning}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Task} & \textbf{Steps} & \textbf{Zen4-Pro Score} \\
|
||||
\midrule
|
||||
Plan + execute (3-step) & 3 & 89.4\% \\
|
||||
Plan + execute (5-step) & 5 & 81.2\% \\
|
||||
Plan + execute (10-step) & 10 & 68.7\% \\
|
||||
Self-verify + correct & 2 & 77.3\% \\
|
||||
Error recovery after tool failure & --- & 71.8\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Multi-Step Task Completion Rates (AgentBench decomposition)}
|
||||
\end{table}
|
||||
|
||||
\subsection{Professional Domain Evaluation}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Domain} & \textbf{Task} & \textbf{Score} \\
|
||||
\midrule
|
||||
Software Engineering & SWE-bench Verified & 48.7\% \\
|
||||
Data Analysis & BIRD SQL (accuracy) & 73.4\% \\
|
||||
Scientific Research & GPQA Diamond & 74.3\% \\
|
||||
Legal & LegalBench (avg) & 68.1\% \\
|
||||
Finance & FinanceBench (accuracy) & 84.2\% \\
|
||||
Medicine & MedQA USMLE (4-option) & 86.3\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Professional Domain Evaluation}
|
||||
\end{table}
|
||||
|
||||
\section{Related Work}
|
||||
|
||||
Scaling dense transformers beyond 70B parameters has followed a predictable capability-compute
|
||||
curve \cite{hoffmann2022chinchilla}. Zen4-Pro's 72B represents the practical ceiling for
|
||||
single-node inference on current data-center GPUs while pushing capability to the frontier.
|
||||
|
||||
Agentic AI systems built on language models have grown rapidly since ReAct \cite{yao2023react}
|
||||
demonstrated that interleaving reasoning traces with action invocations improves task success
|
||||
on multi-step benchmarks. Subsequent systems \cite{wang2024executable} extended this to
|
||||
hierarchical planning. Zen4-Pro internalizes these patterns rather than relying on external
|
||||
orchestration frameworks, reducing latency and improving coherence across long trajectories.
|
||||
|
||||
SWE-bench \cite{jimenez2024swebench} has emerged as the canonical evaluation for autonomous
|
||||
software engineering capability. Current frontier scores range from 30--50\% depending on
|
||||
scaffolding; Zen4-Pro achieves 48.7\% with a minimal scaffold, demonstrating that model
|
||||
quality rather than scaffolding complexity drives performance.
|
||||
|
||||
\section{Deployment}
|
||||
|
||||
\subsection{Inference Configuration}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{llll}
|
||||
\toprule
|
||||
\textbf{Format} & \textbf{VRAM} & \textbf{Context} & \textbf{Hardware} \\
|
||||
\midrule
|
||||
BF16 & 145 GB & 128K & H100 $\times 2$ \\
|
||||
FP8 & 72 GB & 128K & H100 $\times 1$ \\
|
||||
Q4\_K\_M & 42 GB & 32K & RTX 6000 Ada $\times 1$ \\
|
||||
Q4\_K\_M & 48 GB & 64K & A6000 $\times 1$ \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen4-Pro Deployment Hardware Requirements}
|
||||
\end{table}
|
||||
|
||||
\subsection{Recommended Agentic Configuration}
|
||||
|
||||
\begin{lstlisting}[language=Python, caption=Zen4-Pro Agentic Task Configuration]
|
||||
from hanzo import HanzoAI
|
||||
|
||||
client = HanzoAI()
|
||||
|
||||
# Agentic tasks: enable longer generation, structured output
|
||||
response = client.chat.completions.create(
|
||||
model="zen4-pro-72b-instruct",
|
||||
messages=[
|
||||
{"role": "system", "content": "You are an autonomous engineering agent."},
|
||||
{"role": "user", "content": task_description}
|
||||
],
|
||||
tools=toolset,
|
||||
tool_choice="auto",
|
||||
max_tokens=8192, # allow full reasoning chains
|
||||
temperature=0.2, # low temperature for agentic reliability
|
||||
extra_body={
|
||||
"agentic_mode": True, # enables internal state tracking
|
||||
"max_tool_rounds": 20 # hard cap on tool-call loops
|
||||
}
|
||||
)
|
||||
\end{lstlisting}
|
||||
|
||||
\section{Safety and Alignment}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lcc}
|
||||
\toprule
|
||||
\textbf{Safety Metric} & \textbf{Zen3-Pro} & \textbf{Zen4-Pro} \\
|
||||
\midrule
|
||||
Harmful content refusal & 95.3\% & 98.1\% \\
|
||||
Over-refusal rate (benign) & 6.2\% & 3.4\% \\
|
||||
Jailbreak resistance (AdvBench)& 93.7\% & 97.8\% \\
|
||||
TruthfulQA & 74.8\% & 82.3\% \\
|
||||
Agentic safety (destructive action rate) & 1.2\% & 0.3\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Safety Metrics: Zen3-Pro vs. Zen4-Pro}
|
||||
\end{table}
|
||||
|
||||
Agentic deployment introduces safety risks beyond standard language model outputs: a model
|
||||
executing multi-step tool plans can take irreversible actions (file deletion, API calls with
|
||||
side effects). Zen4-Pro's post-training includes an explicit \textbf{action consequence
|
||||
evaluation} objective that teaches the model to identify and flag potentially irreversible
|
||||
operations before executing them.
|
||||
|
||||
\subsection{Limitations}
|
||||
|
||||
\begin{itemize}
|
||||
\item Multi-step task completion rates decay with plan length: 10-step tasks complete
|
||||
at 68.7\% vs 89.4\% for 3-step tasks, indicating compounding errors in long chains.
|
||||
\item Professional domain evaluations (legal, medical) reflect the state of training
|
||||
data and should not substitute for domain expert review.
|
||||
\item 128K context is sufficient for most engineering tasks but falls short of the
|
||||
1M context available in Zen4 (14B). For long-document tasks, Zen4 may be preferable
|
||||
despite its lower benchmark scores.
|
||||
\end{itemize}
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
Zen4-Pro delivers frontier professional capability at 72B dense parameters through three
|
||||
interlocking advances: the Hierarchical Hybrid Attention architecture shared across the
|
||||
Zen4 family, multi-teacher distilled fine-tuning that concentrates specialist knowledge
|
||||
into student layers, and the novel Agentic Alignment post-training stage that grounds
|
||||
multi-step planning in verifiable engineering outcomes. With 48.7\% on SWE-bench Verified
|
||||
and strong results across mathematics, science, and knowledge benchmarks, Zen4-Pro is the
|
||||
recommended choice for enterprise deployments requiring reliable autonomous task execution.
|
||||
|
||||
\section*{Acknowledgments}
|
||||
|
||||
We thank the annotators, engineers, and researchers at Hanzo AI and Zoo Labs Foundation
|
||||
who contributed to the trajectory collection, post-training pipeline, and evaluation
|
||||
infrastructure underlying this work.
|
||||
|
||||
\begin{thebibliography}{99}
|
||||
\bibitem{vaswani2017attention} Vaswani, A. et al. (2017). Attention Is All You Need. NeurIPS 2017.
|
||||
\bibitem{brown2020language} Brown, T. et al. (2020). Language Models are Few-Shot Learners. NeurIPS 2020.
|
||||
\bibitem{ouyang2022training} Ouyang, L. et al. (2022). Training Language Models to Follow Instructions with Human Feedback. NeurIPS 2022.
|
||||
\bibitem{touvron2023llama} Touvron, H. et al. (2023). LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971.
|
||||
\bibitem{hoffmann2022chinchilla} Hoffmann, J. et al. (2022). Training Compute-Optimal Large Language Models. NeurIPS 2022.
|
||||
\bibitem{shazeer2017outrageously} Shazeer, N. et al. (2017). Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. ICLR 2017.
|
||||
\bibitem{schulman2017proximal} Schulman, J. et al. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347.
|
||||
\bibitem{rafailov2023direct} Rafailov, R. et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023.
|
||||
\bibitem{chen2021evaluating} Chen, M. et al. (2021). Evaluating Large Language Models Trained on Code. arXiv:2107.03374.
|
||||
\bibitem{jimenez2024swebench} Jimenez, C. et al. (2024). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR 2024.
|
||||
\bibitem{yao2023react} Yao, S. et al. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. ICLR 2023.
|
||||
\bibitem{wang2024executable} Wang, X. et al. (2024). Executable Code Actions Elicit Better LLM Agents. arXiv:2402.01030.
|
||||
\bibitem{shao2024grpo} Shao, Z. et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300.
|
||||
\end{thebibliography}
|
||||
|
||||
\appendix
|
||||
|
||||
\section{Model Card}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{ll}
|
||||
\toprule
|
||||
\textbf{Field} & \textbf{Value} \\
|
||||
\midrule
|
||||
Model Name & Zen4-Pro \\
|
||||
Version & v2026.01 \\
|
||||
Release Date & January 2026 \\
|
||||
Parameters & 72.7B \\
|
||||
Context Length & 131,072 tokens \\
|
||||
License & Apache 2.0 \\
|
||||
Repository & \href{https://huggingface.co/zenlm/zen4-pro-72b-instruct}{huggingface.co/zenlm/zen4-pro-72b-instruct} \\
|
||||
Documentation & \href{https://papers.zenlm.org/zen4-pro}{papers.zenlm.org/zen4-pro} \\
|
||||
Contact & research@hanzo.ai \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Zen4-Pro Model Card}
|
||||
\end{table}
|
||||
|
||||
\end{document}
|
||||
Binary file not shown.
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user