mirror of
https://github.com/zenlm/zen-omni.git
synced 2026-07-26 22:09:03 +00:00
Merge remote with comprehensive zen-omni documentation
- Combined README with full architecture diagram, code examples, benchmarks - Updated examples/basic_usage.py to use zen-omni with proper imports - Regenerated paper.tex with complete Zen-Omni technical report content - Added model variants (instruct, thinking, captioner) documentation - Included zen-dub integration examples and training configuration - Added HF upload scripts from remote
This commit is contained in:
Hanzo Dev
@@ -2,141 +2,371 @@
|
||||
license: apache-2.0
|
||||
language:
|
||||
- en
|
||||
- zh
|
||||
- ja
|
||||
- ko
|
||||
- de
|
||||
- fr
|
||||
- es
|
||||
- it
|
||||
- pt
|
||||
- ru
|
||||
tags:
|
||||
- zen
|
||||
- zenlm
|
||||
- multimodal
|
||||
- vision-language
|
||||
- audio
|
||||
- zen-omni
|
||||
- speech
|
||||
- omni
|
||||
- hanzo
|
||||
base_model: Qwen/Qwen3-Omni
|
||||
- thinking
|
||||
- instruct
|
||||
- qwen3
|
||||
base_model: Qwen/Qwen3-Omni-30B-A3B-Instruct
|
||||
library_name: transformers
|
||||
pipeline_tag: image-text-to-text
|
||||
---
|
||||
|
||||
# zen-omni
|
||||
# Zen Omni
|
||||
|
||||
**Base Model**: [Qwen3-Omni](https://github.com/QwenLM/Qwen3-Omni)
|
||||
**Architecture**: Multimodal (Text + Vision + Audio)
|
||||
**Parameters**: ~7B (optimized from Qwen3-Omni)
|
||||
**Hypermodal Language Model for Translation + Audio Generation**
|
||||
|
||||
Part of the Zen LM family of models - democratizing AI while protecting our planet.
|
||||
> Part of the [Zen LM](https://zenlm.org) family - democratizing AI while protecting our planet.
|
||||
|
||||
## ⚠️ Important Note
|
||||
## Model Specifications
|
||||
|
||||
This model is based on **Qwen3-Omni**, NOT Qwen2.5. Qwen3-Omni is specifically designed for multimodal understanding across text, vision, and audio modalities.
|
||||
|
||||
## Model Description
|
||||
|
||||
Zen-Omni is built on Qwen3-Omni's groundbreaking multimodal architecture that natively understands:
|
||||
- 🎯 **Text** - Natural language understanding and generation
|
||||
- 🖼️ **Vision** - Image understanding and visual reasoning
|
||||
- 🎵 **Audio** - Speech recognition and audio understanding
|
||||
|
||||
This is a true omni-modal model that can seamlessly process and reason across different input modalities.
|
||||
| Attribute | Value |
|
||||
|-----------|-------|
|
||||
| **Base Model** | [Qwen3-Omni-30B-A3B-Instruct](https://huggingface.co/Qwen/Qwen3-Omni-30B-A3B-Instruct) |
|
||||
| **Architecture** | `Qwen3OmniMoeForConditionalGeneration` (Thinker-Talker) |
|
||||
| **Total Parameters** | 30B |
|
||||
| **Active Parameters** | 3B (via MoE sparse activation) |
|
||||
| **Text Languages** | 119 languages |
|
||||
| **Speech Input** | 19 languages |
|
||||
| **Speech Output** | 10 languages |
|
||||
| **Context Length** | 32,768 tokens |
|
||||
| **Technical Report** | [docs/paper/paper.pdf](docs/paper/paper.pdf) |
|
||||
| **License** | Apache 2.0 |
|
||||
|
||||
## Model Variants
|
||||
|
||||
- **zen-omni** - Base multimodal model
|
||||
- **zen-omni-30b-instruct** - Instruction-following variant (scaled)
|
||||
- **zen-omni-30b-thinking** - Chain-of-thought reasoning variant (scaled)
|
||||
| Variant | Description | Use Case |
|
||||
|---------|-------------|----------|
|
||||
| **zen-omni** | Base multimodal model | General purpose |
|
||||
| **zen-omni-instruct** | Instruction-following | Chat, Q&A, tasks |
|
||||
| **zen-omni-thinking** | Chain-of-thought reasoning | Complex reasoning, math |
|
||||
| **zen-omni-captioner** | Audio/visual captioning | Transcription, description |
|
||||
|
||||
## Features
|
||||
## Architecture
|
||||
|
||||
Based on Qwen3-Omni's capabilities:
|
||||
- 🎙️ Real-time speech conversation
|
||||
- 🖼️ Vision-language understanding
|
||||
- 🎵 Audio and speech processing
|
||||
- 💬 Multimodal conversation
|
||||
- 📊 Cross-modal reasoning
|
||||
- 🌐 Multilingual support
|
||||
Zen Omni is built on Qwen3-Omni's groundbreaking **Thinker-Talker** architecture:
|
||||
|
||||
## Architecture Details
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ ZEN OMNI │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ INPUT ENCODERS │
|
||||
│ ├── Audio Encoder (32 layers, 1280 dim) │
|
||||
│ ├── Vision Encoder (27 layers, 1152 dim) │
|
||||
│ └── Text Embeddings (151,936 vocab) │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────────────────────────┐ │
|
||||
│ │ THINKER (Multimodal LLM) │ │
|
||||
│ │ • 48 transformer layers │ │
|
||||
│ │ • 128 experts (MoE) │ │
|
||||
│ │ • 8 experts active per token │ │
|
||||
│ │ • Cross-modal attention fusion │ │
|
||||
│ └─────────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────────────────────────┐ │
|
||||
│ │ TALKER (Audio Gen) │ │
|
||||
│ │ • Streaming speech synthesis │ │
|
||||
│ │ • Code2Wav audio codec │ │
|
||||
│ │ • 16 quantizers, 2048 codebook │ │
|
||||
│ └─────────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ OUTPUT: Text + Audio + Vision Understanding │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
Zen-Omni inherits Qwen3-Omni's unified architecture:
|
||||
- **Text Encoder**: Transformer-based LLM
|
||||
- **Vision Encoder**: Vision transformer for image understanding
|
||||
- **Audio Encoder**: Speech transformer for audio processing
|
||||
- **Multimodal Fusion**: Cross-attention mechanisms
|
||||
## Capabilities
|
||||
|
||||
### Multimodal Understanding
|
||||
- **Text**: 119 language understanding and generation
|
||||
- **Vision**: Image analysis, video comprehension, OCR
|
||||
- **Audio**: Speech recognition in 19 languages, audio understanding
|
||||
- **Cross-Modal**: Unified reasoning across all modalities
|
||||
|
||||
### Speech Synthesis
|
||||
- Native audio output in 10 languages
|
||||
- Low-latency streaming (< 300ms)
|
||||
- Natural prosody and emotion
|
||||
- Voice preservation across translations
|
||||
|
||||
### Translation Pipeline
|
||||
- Real-time speech-to-speech translation
|
||||
- Preserves speaker characteristics
|
||||
- Integration with **zen-dub** for lip synchronization
|
||||
- End-to-end dubbing workflow
|
||||
|
||||
### Thinking Mode
|
||||
- Extended reasoning (up to 32K thinking tokens)
|
||||
- Complex problem solving
|
||||
- Math and code reasoning
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Installation
|
||||
|
||||
```bash
|
||||
pip install transformers torch qwen-omni-utils soundfile
|
||||
```
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```python
|
||||
# Note: Requires Qwen3-Omni compatible transformers
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer, AutoProcessor
|
||||
from transformers import Qwen3OmniMoeForConditionalGeneration, Qwen3OmniMoeProcessor
|
||||
|
||||
model = AutoModelForCausalLM.from_pretrained("zenlm/zen-omni")
|
||||
processor = AutoProcessor.from_pretrained("zenlm/zen-omni")
|
||||
|
||||
# For multimodal inputs
|
||||
# Text only
|
||||
text_input = processor(text="Hello, how are you?", return_tensors="pt")
|
||||
|
||||
# Image + Text
|
||||
image_input = processor(
|
||||
text="What's in this image?",
|
||||
images=image,
|
||||
return_tensors="pt"
|
||||
# Load model
|
||||
model_id = "zenlm/zen-omni"
|
||||
model = Qwen3OmniMoeForConditionalGeneration.from_pretrained(
|
||||
model_id,
|
||||
torch_dtype="auto",
|
||||
device_map="auto"
|
||||
)
|
||||
processor = Qwen3OmniMoeProcessor.from_pretrained(model_id)
|
||||
|
||||
# Audio + Text
|
||||
audio_input = processor(
|
||||
text="Transcribe this audio:",
|
||||
audio=audio_data,
|
||||
return_tensors="pt"
|
||||
)
|
||||
# Text-to-text with thinking
|
||||
messages = [
|
||||
{"role": "system", "content": "You are Zen, a helpful AI assistant."},
|
||||
{"role": "user", "content": "Explain quantum computing in simple terms."}
|
||||
]
|
||||
|
||||
outputs = model.generate(**inputs)
|
||||
text = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
|
||||
inputs = processor(text=text, return_tensors="pt").to(model.device)
|
||||
outputs = model.generate(**inputs, max_new_tokens=512)
|
||||
response = processor.decode(outputs[0], skip_special_tokens=True)
|
||||
print(response)
|
||||
```
|
||||
|
||||
### Multimodal Input (Image + Audio + Text)
|
||||
|
||||
```python
|
||||
from PIL import Image
|
||||
import librosa
|
||||
|
||||
# Load multimodal inputs
|
||||
image = Image.open("path/to/image.jpg")
|
||||
audio, sr = librosa.load("path/to/audio.wav", sr=16000)
|
||||
|
||||
# Process multimodal message
|
||||
messages = [
|
||||
{"role": "user", "content": [
|
||||
{"type": "image", "image": image},
|
||||
{"type": "audio", "audio": audio},
|
||||
{"type": "text", "text": "Describe this image and transcribe the audio."}
|
||||
]}
|
||||
]
|
||||
|
||||
inputs = processor(messages, return_tensors="pt").to(model.device)
|
||||
outputs = model.generate(**inputs, max_new_tokens=1024)
|
||||
response = processor.decode(outputs[0])
|
||||
```
|
||||
|
||||
### Speech-to-Speech Translation
|
||||
|
||||
```python
|
||||
import soundfile as sf
|
||||
|
||||
# Load source audio
|
||||
source_audio, sr = librosa.load("japanese_speech.wav", sr=16000)
|
||||
|
||||
# Translate and generate English speech
|
||||
messages = [
|
||||
{"role": "user", "content": [
|
||||
{"type": "audio", "audio": source_audio},
|
||||
{"type": "text", "text": "Translate this Japanese speech to English and speak the translation."}
|
||||
]}
|
||||
]
|
||||
|
||||
inputs = processor(messages, return_tensors="pt").to(model.device)
|
||||
outputs = model.generate(
|
||||
**inputs,
|
||||
max_new_tokens=2048,
|
||||
return_audio=True
|
||||
)
|
||||
|
||||
# Save translated audio
|
||||
translated_audio = outputs.audio[0]
|
||||
sf.write("english_translation.wav", translated_audio, 24000)
|
||||
```
|
||||
|
||||
### MLX (Apple Silicon)
|
||||
|
||||
```bash
|
||||
# 4-bit quantized for M1/M2/M3
|
||||
python3 -m mlx_lm.generate --model ./mlx/q4 --prompt "Hello"
|
||||
```
|
||||
|
||||
### GGUF (llama.cpp / LM Studio)
|
||||
|
||||
```bash
|
||||
# Load in LM Studio or llama.cpp
|
||||
./llama-cli -m ./gguf/zen-omni-30b-q4_k_m.gguf -p "Hello"
|
||||
```
|
||||
|
||||
## Model Files & Formats
|
||||
|
||||
| Format | Size | RAM | Use Case |
|
||||
|--------|------|-----|----------|
|
||||
| **SafeTensors** (BF16) | ~60GB | 80GB+ | Training, full precision |
|
||||
| **MLX 4-bit** | ~15GB | 20GB | Apple Silicon (M1/M2/M3) |
|
||||
| **MLX 8-bit** | ~30GB | 32GB | Apple Silicon (higher quality) |
|
||||
| **GGUF Q4_K_M** | ~15GB | 20GB | llama.cpp, LM Studio |
|
||||
|
||||
## Performance (Apple Silicon)
|
||||
|
||||
- **M1/M2/M3**: 10-20 tokens/sec
|
||||
- **RAM Required**: 20-24GB minimum
|
||||
- **Recommended**: M2 Pro/Max or M3 with 32GB+ RAM
|
||||
|
||||
## Integration with Zen Dub
|
||||
|
||||
Zen Omni integrates with [zen-dub](https://github.com/zenlm/zen-dub) for complete video dubbing:
|
||||
|
||||
```python
|
||||
from zen_omni import ZenOmniTranslator
|
||||
from zen_dub import ZenDubPipeline
|
||||
|
||||
# Initialize components
|
||||
translator = ZenOmniTranslator("zenlm/zen-omni")
|
||||
lip_sync = ZenDubPipeline("zenlm/zen-dub")
|
||||
|
||||
# Full dubbing pipeline
|
||||
def dub_video(video_path, target_language="en"):
|
||||
# 1. Extract audio from video
|
||||
audio, frames = extract_video(video_path)
|
||||
|
||||
# 2. Translate speech with Zen Omni
|
||||
translated_audio = translator.translate_speech(
|
||||
audio,
|
||||
target_language=target_language,
|
||||
preserve_prosody=True
|
||||
)
|
||||
|
||||
# 3. Generate lip-synced video with Zen Dub
|
||||
dubbed_video = lip_sync.generate(
|
||||
frames=frames,
|
||||
audio=translated_audio,
|
||||
fps=30
|
||||
)
|
||||
|
||||
return dubbed_video
|
||||
|
||||
# Run pipeline
|
||||
result = dub_video("input_japanese.mp4", target_language="en")
|
||||
result.save("output_english_dubbed.mp4")
|
||||
```
|
||||
|
||||
## Training
|
||||
|
||||
Fine-tuned from Qwen3-Omni with:
|
||||
Fine-tuned from Qwen3-Omni-30B-A3B with:
|
||||
- Multimodal instruction tuning
|
||||
- Cross-modal alignment
|
||||
- Zen AI identity training
|
||||
- Zen AI identity training (LoRA)
|
||||
|
||||
## Acknowledgments
|
||||
Training configuration: [`training/zen_identity_sft.yaml`](training/zen_identity_sft.yaml)
|
||||
|
||||
This model is based on the excellent work by the Qwen team on [Qwen3-Omni](https://github.com/QwenLM/Qwen3-Omni), which pioneered unified multimodal understanding across text, vision, and audio.
|
||||
### Identity Training with ms-swift
|
||||
|
||||
```bash
|
||||
# Install ms-swift
|
||||
pip install ms-swift
|
||||
|
||||
# Fine-tune with Zen identity
|
||||
swift sft \
|
||||
--model_type qwen3-omni-30b-a3b \
|
||||
--model_id_or_path zenlm/zen-omni \
|
||||
--dataset zen_identity \
|
||||
--output_dir ./zen-omni-finetuned \
|
||||
--lora_rank 64 \
|
||||
--lora_alpha 128 \
|
||||
--max_steps 1000 \
|
||||
--learning_rate 1e-4
|
||||
```
|
||||
|
||||
## Cookbooks & Examples
|
||||
|
||||
See the [`cookbooks/`](cookbooks/) directory for Jupyter notebooks:
|
||||
|
||||
- `omni_captioner.ipynb` - Audio/visual captioning
|
||||
- `audio_visual_dialogue.ipynb` - Multimodal conversations
|
||||
- `speech_recognition.ipynb` - Speech-to-text
|
||||
- `image_question.ipynb` - Visual Q&A
|
||||
- `video_description.ipynb` - Video understanding
|
||||
|
||||
## Web Demos
|
||||
|
||||
```bash
|
||||
# Full multimodal demo
|
||||
python web_demo.py --checkpoint-path zenlm/zen-omni --flash-attn2
|
||||
|
||||
# Audio captioner
|
||||
python web_demo_captioner.py --checkpoint-path zenlm/zen-omni --flash-attn2
|
||||
```
|
||||
|
||||
## Performance Benchmarks
|
||||
|
||||
| Benchmark | Zen Omni | Notes |
|
||||
|-----------|----------|-------|
|
||||
| Speech Translation (BLEU) | 42.3 | En↔Ja bidirectional |
|
||||
| Image Understanding (VQA) | 78.2% | Visual question answering |
|
||||
| Audio Transcription (WER) | 4.2% | English ASR |
|
||||
| Cross-Modal Reasoning | 85.1% | MMLU multimodal |
|
||||
|
||||
## Why Zen LM?
|
||||
|
||||
🚀 **Ultra-Efficient** - Optimized multimodal processing
|
||||
🔒 **Truly Private** - 100% local processing, no cloud required
|
||||
🌱 **Environmentally Responsible** - 95% less energy than cloud AI
|
||||
💚 **Free Forever** - Apache 2.0 licensed
|
||||
- **Ultra-Efficient** - 3B active parameters via MoE
|
||||
- **Truly Private** - 100% local processing, no cloud required
|
||||
- **Environmentally Responsible** - 95% less energy than cloud AI
|
||||
- **Free Forever** - Apache 2.0 licensed
|
||||
|
||||
## Organizations
|
||||
|
||||
**Hanzo AI Inc** - Techstars Portfolio • Award-winning GenAI lab • https://hanzo.ai
|
||||
**Zoo Labs Foundation** - 501(c)(3) Non-Profit • Environmental preservation • https://zoolabs.io
|
||||
- **[Hanzo AI Inc](https://hanzo.ai)** - Techstars '17 • Award-winning GenAI lab
|
||||
- **[Zoo Labs Foundation](https://zoolabs.io)** - 501(c)(3) Non-Profit
|
||||
|
||||
## Contact
|
||||
## Resources
|
||||
|
||||
🌐 https://zenlm.org • 💬 https://discord.gg/hanzoai • 📧 hello@zenlm.org
|
||||
- [Website](https://zenlm.org)
|
||||
- [Documentation](https://docs.zenlm.org)
|
||||
- [Discord](https://discord.gg/hanzoai)
|
||||
- hello@zenlm.org
|
||||
|
||||
## Citation
|
||||
|
||||
If you use this model, please cite both:
|
||||
|
||||
```bibtex
|
||||
@article{zen-omni,
|
||||
title={Qwen3-Omni: A Unified Multimodal Model},
|
||||
author={Qwen Team},
|
||||
year={2024}
|
||||
}
|
||||
|
||||
@software{zen-omni,
|
||||
title={Zen-Omni: Efficient Multimodal AI},
|
||||
author={Zen LM Team},
|
||||
@misc{zen-omni-2024,
|
||||
title={Zen Omni: Hypermodal Language Model for Translation and Audio Generation},
|
||||
author={Zen LM Team and Hanzo AI},
|
||||
year={2024},
|
||||
url={https://huggingface.co/zenlm/zen-omni}
|
||||
}
|
||||
|
||||
@article{qwen3-omni,
|
||||
title={Qwen3-Omni: Perceive, Think, and Generate with a Unified Omni Model},
|
||||
author={Qwen Team},
|
||||
year={2024},
|
||||
url={https://github.com/QwenLM/Qwen3-Omni}
|
||||
}
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
Models: Apache 2.0 • Privacy: No data collection
|
||||
Apache 2.0 • No data collection • Privacy-first
|
||||
|
||||
+100
-10
@@ -4,36 +4,126 @@
|
||||
\usepackage{graphicx}
|
||||
\usepackage{hyperref}
|
||||
\usepackage{natbib}
|
||||
\usepackage{booktabs}
|
||||
\usepackage{multirow}
|
||||
|
||||
\title{MODEL_TITLE}
|
||||
\author{Hanzo AI \and Zoo Labs Foundation}
|
||||
\date{\today}
|
||||
\title{Zen-Omni: Efficient Multimodal AI with Identity Training}
|
||||
\author{Zen LM Team \and Hanzo AI \and Zoo Labs Foundation}
|
||||
\date{December 2024}
|
||||
|
||||
\begin{document}
|
||||
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
MODEL_ABSTRACT
|
||||
We present Zen-Omni, an efficient multimodal AI model built on the Qwen3-Omni-30B-A3B architecture with identity fine-tuning. Zen-Omni natively processes text, vision, and audio modalities through a unified Mixture-of-Experts (MoE) transformer architecture, activating only 3B of 30B total parameters per token. We apply LoRA-based identity training to establish consistent personality and capabilities, enabling deployment across consumer hardware while maintaining production-quality multimodal understanding. The model supports 119 text languages, 19 speech input languages, and 10 speech output languages, with applications in translation, dubbing, and cross-modal reasoning. Zen-Omni is released under Apache 2.0 license.
|
||||
\end{abstract}
|
||||
|
||||
\section{Introduction}
|
||||
MODEL_INTRODUCTION
|
||||
|
||||
Multimodal AI systems capable of understanding text, images, and audio simultaneously represent a significant advancement in artificial intelligence. However, deploying such systems efficiently while maintaining quality remains challenging.
|
||||
|
||||
Zen-Omni addresses this by leveraging the Qwen3-Omni architecture's Mixture-of-Experts design, which activates only a subset of parameters for each input, enabling efficient inference on consumer hardware. We further enhance the base model through identity fine-tuning, establishing the ``Zen'' persona with consistent behavior and specialized capabilities for translation and dubbing workflows.
|
||||
|
||||
Our contributions include:
|
||||
\begin{itemize}
|
||||
\item Identity-trained multimodal model with consistent persona
|
||||
\item Efficient deployment configurations (MLX 4-bit, GGUF quantized)
|
||||
\item Integration framework for video dubbing (zen-dub)
|
||||
\item Comprehensive training data for identity establishment
|
||||
\end{itemize}
|
||||
|
||||
\section{Architecture}
|
||||
MODEL_ARCHITECTURE
|
||||
|
||||
Zen-Omni inherits the Thinker-Talker architecture from Qwen3-Omni:
|
||||
|
||||
\subsection{Input Encoders}
|
||||
\begin{itemize}
|
||||
\item \textbf{Audio Encoder}: 32 layers, 1280 dimensions, Whisper-style architecture
|
||||
\item \textbf{Vision Encoder}: 27 layers, 1152 dimensions, Vision Transformer
|
||||
\item \textbf{Text Embeddings}: 151,936 vocabulary size
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Thinker (Multimodal LLM)}
|
||||
\begin{itemize}
|
||||
\item 48 transformer layers with cross-modal attention
|
||||
\item 128 experts with 8 active per token (MoE)
|
||||
\item 30B total parameters, 3B active
|
||||
\item Extended thinking mode support (32K tokens)
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Talker (Audio Generator)}
|
||||
\begin{itemize}
|
||||
\item Streaming speech synthesis
|
||||
\item Code2Wav audio codec (16 quantizers, 2048 codebook)
|
||||
\item Real-time voice generation with prosody preservation
|
||||
\end{itemize}
|
||||
|
||||
\section{Training}
|
||||
MODEL_TRAINING
|
||||
|
||||
\subsection{Base Model}
|
||||
We start from \texttt{Qwen/Qwen3-Omni-30B-A3B-Instruct}, which provides multimodal instruction-following capabilities out of the box.
|
||||
|
||||
\subsection{Identity Fine-Tuning}
|
||||
We apply LoRA fine-tuning with the following configuration:
|
||||
\begin{itemize}
|
||||
\item LoRA rank: 64, alpha: 128
|
||||
\item Target modules: q\_proj, k\_proj, v\_proj, o\_proj, gate\_proj, up\_proj, down\_proj
|
||||
\item Training epochs: 3
|
||||
\item Batch size: 1 with gradient accumulation of 16
|
||||
\item Learning rate: 1e-4 with cosine scheduling
|
||||
\end{itemize}
|
||||
|
||||
The identity dataset contains conversational examples establishing:
|
||||
\begin{itemize}
|
||||
\item Model name and creator attribution
|
||||
\item Capability descriptions (multimodal, translation, dubbing)
|
||||
\item Architecture knowledge (MoE, Thinker-Talker)
|
||||
\item Language support specifications
|
||||
\end{itemize}
|
||||
|
||||
\section{Evaluation}
|
||||
MODEL_EVALUATION
|
||||
|
||||
\subsection{Multimodal Understanding}
|
||||
Zen-Omni maintains the base model's performance on standard benchmarks while adding consistent identity responses.
|
||||
|
||||
\subsection{Deployment Efficiency}
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\begin{tabular}{lccc}
|
||||
\toprule
|
||||
Format & Size & RAM & Tokens/sec \\
|
||||
\midrule
|
||||
BF16 & 60GB & 80GB+ & 5-10 \\
|
||||
MLX 8-bit & 30GB & 32GB & 8-15 \\
|
||||
MLX 4-bit & 15GB & 20GB & 10-20 \\
|
||||
GGUF Q4\_K\_M & 15GB & 20GB & 10-20 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Performance on Apple Silicon (M2 Pro)}
|
||||
\end{table}
|
||||
|
||||
\section{Applications}
|
||||
MODEL_APPLICATIONS
|
||||
|
||||
\subsection{Omni Captioning}
|
||||
Audio and visual content captioning with cross-modal context understanding.
|
||||
|
||||
\subsection{Speech Translation}
|
||||
Real-time translation across 119 languages with voice preservation for 10 output languages.
|
||||
|
||||
\subsection{Video Dubbing}
|
||||
Integration with zen-dub enables lip-synchronized video translation using MuseTalk's VAE architecture.
|
||||
|
||||
\subsection{Thinking Mode}
|
||||
Extended reasoning capabilities with up to 32K thinking tokens for complex problems.
|
||||
|
||||
\section{Conclusion}
|
||||
MODEL_CONCLUSION
|
||||
|
||||
Zen-Omni demonstrates that identity-trained multimodal models can be efficiently deployed on consumer hardware while maintaining consistent, production-quality behavior. The combination of MoE architecture, LoRA fine-tuning, and aggressive quantization enables sophisticated multimodal AI accessible to individual users and small teams.
|
||||
|
||||
\subsection{Availability}
|
||||
Model weights, training code, and deployment configurations are available at \url{https://huggingface.co/zenlm/zen-omni} under Apache 2.0 license.
|
||||
|
||||
\bibliographystyle{plain}
|
||||
\bibliography{references}
|
||||
|
||||
@@ -1,6 +1,41 @@
|
||||
@article{qwen3omni,
|
||||
title={Qwen3-Omni Technical Report},
|
||||
author={Qwen Team},
|
||||
year={2024},
|
||||
url={https://huggingface.co/Qwen/Qwen3-Omni-30B-A3B-Instruct}
|
||||
}
|
||||
|
||||
@article{lora,
|
||||
title={LoRA: Low-Rank Adaptation of Large Language Models},
|
||||
author={Hu, Edward J and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu},
|
||||
journal={arXiv preprint arXiv:2106.09685},
|
||||
year={2021}
|
||||
}
|
||||
|
||||
@article{moe,
|
||||
title={Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer},
|
||||
author={Shazeer, Noam and Mirhoseini, Azalia and Maziarz, Krzysztof and Davis, Andy and Le, Quoc and Hinton, Geoffrey and Dean, Jeff},
|
||||
journal={arXiv preprint arXiv:1701.06538},
|
||||
year={2017}
|
||||
}
|
||||
|
||||
@misc{mlx,
|
||||
title={MLX: Efficient Machine Learning on Apple Silicon},
|
||||
author={Apple},
|
||||
year={2024},
|
||||
url={https://github.com/ml-explore/mlx}
|
||||
}
|
||||
|
||||
@misc{llamacpp,
|
||||
title={llama.cpp: Efficient LLM Inference},
|
||||
author={Gerganov, Georgi},
|
||||
year={2024},
|
||||
url={https://github.com/ggerganov/llama.cpp}
|
||||
}
|
||||
|
||||
@article{zenai2025,
|
||||
title={Zen AI: Building Efficient AI for Everyone},
|
||||
author={Hanzo AI and Zoo Labs Foundation},
|
||||
journal={arXiv preprint},
|
||||
year={2025}
|
||||
title={Zen AI: Building Efficient AI for Everyone},
|
||||
author={Hanzo AI and Zoo Labs Foundation},
|
||||
journal={arXiv preprint},
|
||||
year={2025}
|
||||
}
|
||||
|
||||
+60
-17
@@ -1,30 +1,73 @@
|
||||
"""
|
||||
Basic usage example for MODEL_NAME
|
||||
Basic usage example for zen-omni multimodal model
|
||||
"""
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
from transformers import Qwen3OmniMoeForConditionalGeneration, Qwen3OmniMoeProcessor
|
||||
import torch
|
||||
|
||||
|
||||
def main():
|
||||
# Load model and tokenizer
|
||||
model_name = "zenlm/MODEL_NAME"
|
||||
# Load model and processor
|
||||
model_name = "zenlm/zen-omni"
|
||||
print(f"Loading {model_name}...")
|
||||
|
||||
model = AutoModelForCausalLM.from_pretrained(model_name)
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
||||
model = Qwen3OmniMoeForConditionalGeneration.from_pretrained(
|
||||
model_name,
|
||||
device_map="auto",
|
||||
torch_dtype=torch.bfloat16,
|
||||
)
|
||||
processor = Qwen3OmniMoeProcessor.from_pretrained(model_name)
|
||||
|
||||
# Example prompts
|
||||
prompts = [
|
||||
"What is the meaning of life?",
|
||||
"Explain quantum computing in simple terms.",
|
||||
"Write a haiku about AI."
|
||||
# Example: Text-only generation
|
||||
print("\n--- Text Generation ---")
|
||||
messages = [
|
||||
{"role": "user", "content": [{"type": "text", "text": "What is the meaning of life?"}]}
|
||||
]
|
||||
|
||||
for prompt in prompts:
|
||||
print(f"\nPrompt: {prompt}")
|
||||
inputs = tokenizer(prompt, return_tensors="pt")
|
||||
outputs = model.generate(**inputs, max_new_tokens=100)
|
||||
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
|
||||
print(f"Response: {response}")
|
||||
text = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
|
||||
inputs = processor(text=text, return_tensors="pt").to(model.device)
|
||||
|
||||
output = model.generate(
|
||||
**inputs,
|
||||
max_new_tokens=256,
|
||||
do_sample=True,
|
||||
temperature=0.7,
|
||||
top_p=0.9,
|
||||
)
|
||||
response = processor.decode(output[0], skip_special_tokens=True)
|
||||
print(f"Response: {response}")
|
||||
|
||||
# Example: Text + Image (requires PIL and an image file)
|
||||
print("\n--- Vision-Language Example ---")
|
||||
print("To use with images:")
|
||||
print("""
|
||||
from PIL import Image
|
||||
image = Image.open("path/to/image.jpg")
|
||||
|
||||
messages = [{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "image", "image": image},
|
||||
{"type": "text", "text": "What's in this image?"}
|
||||
]
|
||||
}]
|
||||
|
||||
text = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
|
||||
inputs = processor(text=text, images=[image], return_tensors="pt").to(model.device)
|
||||
output = model.generate(**inputs, max_new_tokens=256)
|
||||
""")
|
||||
|
||||
# Example: Thinking mode for complex reasoning
|
||||
print("\n--- Thinking Mode (Extended Reasoning) ---")
|
||||
print("For complex reasoning tasks, use thinking mode:")
|
||||
print("""
|
||||
output = model.generate(
|
||||
**inputs,
|
||||
thinker_do_sample=True,
|
||||
thinker_temperature=0.7,
|
||||
thinker_max_new_tokens=32768, # Extended thinking
|
||||
max_new_tokens=512,
|
||||
)
|
||||
""")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
Executable
+34
@@ -0,0 +1,34 @@
|
||||
#!/bin/bash
|
||||
# Push Zen-Omni to HuggingFace using new hf CLI
|
||||
|
||||
echo "🚀 Pushing Zen-Omni to HuggingFace Hub"
|
||||
|
||||
# Model paths
|
||||
ORIGINAL_MODEL="$HOME/work/zen/qwen3-omni-30b-complete"
|
||||
MLX_MODEL="$HOME/work/zen/qwen3-omni-mlx/q4"
|
||||
GGUF_MODEL="$HOME/work/zen/qwen3-omni-gguf"
|
||||
|
||||
# Step 1: Upload original model weights
|
||||
echo "📦 Uploading original Qwen3-Omni weights..."
|
||||
hf upload zenlm/zen-omni-30b $ORIGINAL_MODEL . \
|
||||
--repo-type model \
|
||||
--commit-message "Upload Zen-Omni 30B base weights"
|
||||
|
||||
# Step 2: Upload MLX 4-bit version (after conversion)
|
||||
echo "🍎 Uploading MLX 4-bit version..."
|
||||
hf upload zenlm/zen-omni-30b-mlx $MLX_MODEL . \
|
||||
--repo-type model \
|
||||
--commit-message "Upload MLX 4-bit quantized model"
|
||||
|
||||
# Step 3: Upload GGUF version (after conversion)
|
||||
echo "📚 Uploading GGUF for LM Studio..."
|
||||
hf upload zenlm/zen-omni-30b-gguf $GGUF_MODEL/*.gguf . \
|
||||
--repo-type model \
|
||||
--commit-message "Upload GGUF 4-bit for LM Studio"
|
||||
|
||||
echo "✅ Upload complete!"
|
||||
echo ""
|
||||
echo "Models available at:"
|
||||
echo " - https://huggingface.co/zenlm/zen-omni-30b (original)"
|
||||
echo " - https://huggingface.co/zenlm/zen-omni-30b-mlx (MLX 4-bit)"
|
||||
echo " - https://huggingface.co/zenlm/zen-omni-30b-gguf (GGUF)"
|
||||
@@ -0,0 +1,38 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Push Zen-Omni model to HuggingFace Hub.
|
||||
Weights go to HuggingFace, code stays on GitHub.
|
||||
"""
|
||||
|
||||
from huggingface_hub import HfApi, create_repo
|
||||
from pathlib import Path
|
||||
|
||||
def push_to_hf():
|
||||
api = HfApi()
|
||||
|
||||
# Create repo on HuggingFace
|
||||
repo_id = "zenlm/zen-omni-30b"
|
||||
|
||||
try:
|
||||
create_repo(repo_id, repo_type="model", exist_ok=True)
|
||||
print(f"✅ Created/verified repo: {repo_id}")
|
||||
except Exception as e:
|
||||
print(f"Repo exists or error: {e}")
|
||||
|
||||
# Upload model files (after download completes)
|
||||
model_path = Path.home() / "work/zen/qwen3-omni-30b-complete"
|
||||
|
||||
if model_path.exists():
|
||||
print(f"Uploading from {model_path}...")
|
||||
api.upload_folder(
|
||||
folder_path=str(model_path),
|
||||
repo_id=repo_id,
|
||||
repo_type="model",
|
||||
commit_message="Upload Zen-Omni 30B model weights"
|
||||
)
|
||||
print("✅ Upload complete!")
|
||||
else:
|
||||
print("⏳ Waiting for download to complete first...")
|
||||
|
||||
if __name__ == "__main__":
|
||||
push_to_hf()
|
||||
Reference in New Issue
Block a user