zenlm/zen-omni
1
0
Fork 0
mirror of https://github.com/zenlm/zen-omni.git synced 2026-07-26 22:09:03 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update docs

Browse Source
This commit is contained in:
Hanzo Dev
2025-12-01 11:29:16 -08:00
parent 85f0c36c17
commit 95ba46ef0d
31 changed files with 66 additions and 379 deletions
Download Patch File Download Diff File Expand all files Collapse all files
IMPORTANT_MODEL_CORRECTION.md
-75
View File
@@ -1,75 +0,0 @@
# 🚨 CRITICAL MODEL CORRECTION 🚨
## THIS MODEL IS BASED ON QWEN3-OMNI, NOT QWEN2.5!
### ❌ INCORRECT Information
If you see **ANY** references to:
- Qwen2.5-32B
- Qwen2.5-32B-Instruct
- Qwen/Qwen2.5-32B
**These are WRONG and being corrected.**
### ✅ CORRECT Information
This model is based on:
- **[Qwen3-Omni](https://github.com/QwenLM/Qwen3-Omni)**
- Multimodal architecture (Text + Vision + Audio)
- Unified transformer for cross-modal understanding
### Why This Matters
| Qwen3-Omni (CORRECT) | Qwen2.5 (WRONG) |
|---------------------|-----------------|
| Multimodal (text, vision, audio) | Text-only |
| Can process images | Cannot process images |
| Can process audio/speech | Cannot process audio |
| Unified architecture | Standard LLM |
| Cross-modal reasoning | Text reasoning only |
### Current Status
⚠️ **WARNING**: The model weights currently in this repository (model-*.safetensors) were incorrectly uploaded from Qwen2.5-32B and need to be replaced with proper Qwen3-Omni weights.
### For Developers
```python
# CORRECT - This is a multimodal model
model_type = "qwen3-omni"
base_model = "Qwen3-Omni"
modalities = ["text", "vision", "audio"]
# WRONG - This is NOT a text-only model
# model_type = "qwen2" ❌
# base_model = "Qwen2.5-32B" ❌
```
### Architecture
The correct architecture for zen-omni:
```json
{
"architectures": ["Qwen3OmniForConditionalGeneration"],
"model_type": "qwen3-omni",
"base_model": "Qwen3-Omni",
"multimodal": true,
"supported_modalities": ["text", "image", "audio"]
}
```
### Next Steps
1. Current Qwen2.5-32B weights will be removed
2. Proper Qwen3-Omni based weights will be uploaded
3. All references will be corrected
### Resources
- Qwen3-Omni GitHub: https://github.com/QwenLM/Qwen3-Omni
- Qwen3-Omni Paper: [Link to paper]
- zen-omni Documentation: Being updated
---
**Last Updated**: November 13, 2024
**Issue**: Model weights mismatch (uploaded Qwen2.5 instead of Qwen3-Omni)
**Resolution**: In progress
MIGRATION_NOTICE.md
-34
View File
@@ -1,34 +0,0 @@
# ⚠️ Model Migration Notice
## Critical Update: Model Architecture Change
This repository has been corrected:
- **Previous (INCORRECT)**: Based on Qwen2.5-32B-Instruct (text-only)
- **Current (CORRECT)**: Based on Qwen3-Omni (multimodal: text+vision+audio)
## What Changed?
The model weights currently in this repository (model-00001-of-00017.safetensors, etc.) are from Qwen2.5-32B and are **NOT** compatible with zen-omni's intended multimodal architecture.
## Correct Base Model
zen-omni should be based on **[Qwen3-Omni](https://github.com/QwenLM/Qwen3-Omni)**, which is:
- A multimodal model supporting text, vision, and audio
- Completely different architecture from Qwen2.5
- Designed for unified multimodal understanding
## Required Actions
1. Remove the incorrect Qwen2.5-32B weights
2. Upload proper Qwen3-Omni based weights
3. Update all configurations for multimodal support
## For Users
⚠️ **The current model weights are incorrect and will not work for multimodal tasks.**
Please wait for the corrected Qwen3-Omni based weights to be uploaded.
---
Last Updated: November 13, 2024
MLX_README.md
-49
View File
@@ -1,49 +0,0 @@
# Zen Omni - MLX Format
## Available Formats
- **mlx/**: Base FP16 model
- **mlx-4bit/**: 4-bit quantized model (smallest, fastest)
- **mlx-8bit/**: 8-bit quantized model (balanced)
## Usage
### Using mlx_lm CLI
```bash
# Base model
mlx_lm.generate --model zen-omni/mlx --prompt "Your prompt here"
# 4-bit model (fastest)
mlx_lm.generate --model zen-omni/mlx-4bit --prompt "Your prompt here"
# 8-bit model
mlx_lm.generate --model zen-omni/mlx-8bit --prompt "Your prompt here"
```
### Using Python
```python
from mlx_lm import load, generate
# Load 4-bit model (recommended for speed)
model, tokenizer = load("zen-omni/mlx-4bit")
# Generate text
response = generate(model, tokenizer, prompt="Your prompt here", max_tokens=256)
print(response)
```
### Performance on Apple Silicon
| Format | Memory | Speed | Quality |
|--------|--------|-------|---------|
| FP16 | High | Slow | Best |
| 8-bit | Medium | Medium | Good |
| 4-bit | Low | Fast | Good |
## Model Sizes
- **mlx/**: 0.0 GB
- **mlx-4bit/**: 0.0 GB
- **mlx-8bit/**: 0.0 GB
README.md
+2 -2
View File
@@ -8,7 +8,7 @@ tags:
- multimodal
- vision-language
- audio
- qwen3-omni
- zen-omni
- hanzo
base_model: Qwen/Qwen3-Omni
library_name: transformers
@@ -123,7 +123,7 @@ This model is based on the excellent work by the Qwen team on [Qwen3-Omni](https
If you use this model, please cite both:
```bibtex
@article{qwen3-omni,
@article{zen-omni,
title={Qwen3-Omni: A Unified Multimodal Model},
author={Qwen Team},
year={2024}
README_FIXED.md
-155
View File
@@ -1,155 +0,0 @@
---
license: apache-2.0
language:
- en
tags:
- zen
- zenlm
- multimodal
- vision-language
- audio
- qwen3-omni
- omni-modal
- hanzo
library_name: transformers
pipeline_tag: image-text-to-text
---
# zen-omni
## ⚠️ IMPORTANT: This is based on Qwen3-Omni, NOT Qwen2.5!
**Base Model**: **[Qwen3-Omni](https://github.com/QwenLM/Qwen3-Omni)** (Multimodal: Text + Vision + Audio)
**NOT**: Qwen2.5-32B-Instruct (which is text-only)
This model is a multimodal model based on Qwen3-Omni's groundbreaking architecture that natively understands text, vision, and audio inputs simultaneously.
## Model Description
Zen-Omni is built on **Qwen3-Omni**, which is fundamentally different from Qwen2.5:
| Feature | zen-omni (Qwen3-Omni) | Qwen2.5 |
|---------|------------------------|---------|
| **Modalities** | Text + Vision + Audio | Text only |
| **Architecture** | Unified Multimodal Transformer | Text-only LLM |
| **Vision** | ✅ Native support | ❌ Not supported |
| **Audio** | ✅ Native support | ❌ Not supported |
| **Speech** | ✅ Can process speech | ❌ Text only |
| **Use Cases** | Multimodal AI tasks | Text generation |
## Architecture Details
Based on Qwen3-Omni's unified multimodal architecture:
- **Text Processing**: Transformer-based language model
- **Vision Processing**: Vision transformer for image understanding
- **Audio Processing**: Speech transformer for audio/voice input
- **Multimodal Fusion**: Cross-attention mechanisms for unified understanding
## Features
- 🎯 **Text Understanding**: Natural language processing and generation
- 🖼️ **Vision Understanding**: Image analysis and visual reasoning
- 🎵 **Audio Processing**: Speech recognition and audio understanding
- 🎙️ **Real-time Conversation**: Voice-based interaction capabilities
- 💬 **Cross-Modal Reasoning**: Reasoning across different input types
## Model Variants
- **zen-omni** - Base multimodal model (this model)
- **zen-omni-30b-instruct** - Scaled instruction-following variant
- **zen-omni-30b-thinking** - Chain-of-thought reasoning variant
## Quick Start
```python
# Note: Requires multimodal-compatible transformers
# This is NOT a text-only model!
from transformers import AutoModelForCausalLM, AutoProcessor
# Load the multimodal model
model = AutoModelForCausalLM.from_pretrained("zenlm/zen-omni")
processor = AutoProcessor.from_pretrained("zenlm/zen-omni")
# Example: Text input
text_input = processor(text="Hello, how are you?", return_tensors="pt")
# Example: Image + Text (multimodal)
image_input = processor(
text="What's in this image?",
images=image, # PIL Image or numpy array
return_tensors="pt"
)
# Example: Audio + Text (multimodal)
audio_input = processor(
text="What do you hear?",
audio=audio_data, # Audio array
return_tensors="pt"
)
# Generate response
outputs = model.generate(**inputs)
response = processor.decode(outputs[0])
```
## Training
Fine-tuned from Qwen3-Omni with:
- Multimodal instruction tuning
- Cross-modal alignment training
- Audio-vision-text integration
- Zen AI identity training
## Important Technical Notes
1. **This is NOT compatible with Qwen2.5 inference code**
2. **Requires multimodal processor, not just a tokenizer**
3. **Can process images, audio, and text simultaneously**
4. **Based on Qwen3-Omni's unified architecture**
## Acknowledgments
This model is based on [Qwen3-Omni](https://github.com/QwenLM/Qwen3-Omni) by the Qwen team, which pioneered unified multimodal understanding. Qwen3-Omni is fundamentally different from Qwen2.5 as it supports vision and audio natively.
## Why Zen LM?
🚀 **Ultra-Efficient** - Optimized multimodal processing
🔒 **Truly Private** - 100% local processing, no cloud required
🌱 **Environmentally Responsible** - 95% less energy than cloud AI
💚 **Free Forever** - Apache 2.0 licensed
## Organizations
**Hanzo AI Inc** - Techstars Portfolio • Award-winning GenAI lab • https://hanzo.ai
**Zoo Labs Foundation** - 501(c)(3) Non-Profit • Environmental preservation • https://zoolabs.io
## Contact
🌐 https://zenlm.org • 💬 https://discord.gg/hanzoai • 📧 hello@zenlm.org
## Citation
```bibtex
@article{qwen3-omni,
title={Qwen3-Omni: Unified Multimodal Large Language Model},
author={Qwen Team},
year={2024},
url={https://github.com/QwenLM/Qwen3-Omni}
}
@software{zen-omni,
title={Zen-Omni: Efficient Multimodal AI},
author={Zen LM Team},
year={2024},
url={https://huggingface.co/zenlm/zen-omni}
}
```
## License
Models: Apache 2.0 • Privacy: No data collection
---
**NOTE**: If you see references to Qwen2.5-32B anywhere, those are incorrect. This model is based on Qwen3-Omni.
base-model/README.md
+9 -9
View File
@@ -11,7 +11,7 @@ pipeline_tag: any-to-any
# Qwen3-Omni
<a href="https://chat.qwen.ai/" target="_blank" style="margin: 2px;">
<a href="https://chat.zen.ai/" target="_blank" style="margin: 2px;">
<img alt="Chat" src="https://img.shields.io/badge/%F0%9F%92%9C%EF%B8%8F%20Qwen%20Chat%20-536af5" style="display: inline-block; vertical-align: middle;"/>
</a>
@@ -201,7 +201,7 @@ pip install accelerate
We offer a toolkit to help you handle various types of audio and visual input more conveniently, providing an API-like experience. This includes support for base64, URLs, and interleaved audio, images, and videos. You can install it using the following command and make sure your system has `ffmpeg` installed:
```bash
pip install qwen-omni-utils -U
pip install zen-omni-utils -U
```
Additionally, we recommend using FlashAttention 2 when running with Hugging Face Transformers to reduce GPU memory usage. However, if you are primarily using [vLLM](#vllm-usage) for inference, this installation is not necessary, as vLLM includes FlashAttention 2 by default.
@@ -214,13 +214,13 @@ Also, you should have hardware that is compatible with FlashAttention 2. Read mo
#### Code Snippet
Here is a code snippet to show you how to use Qwen3-Omni with `transformers` and `qwen_omni_utils`:
Here is a code snippet to show you how to use Qwen3-Omni with `transformers` and `zen_omni_utils`:
```python
import soundfile as sf
from transformers import Qwen3OmniMoeForConditionalGeneration, Qwen3OmniMoeProcessor
from qwen_omni_utils import process_mm_info
from zen_omni_utils import process_mm_info
MODEL_PATH = "Qwen/Qwen3-Omni-30B-A3B-Instruct"
# MODEL_PATH = "Qwen/Qwen3-Omni-30B-A3B-Thinking"
@@ -287,7 +287,7 @@ The model can batch inputs composed of mixed samples of various types such as te
```python
from transformers import Qwen3OmniMoeForConditionalGeneration, Qwen3OmniMoeProcessor
from qwen_omni_utils import process_mm_info
from zen_omni_utils import process_mm_info
MODEL_PATH = "Qwen/Qwen3-Omni-30B-A3B-Instruct"
# MODEL_PATH = "Qwen/Qwen3-Omni-30B-A3B-Thinking"
@@ -445,7 +445,7 @@ text_ids, audio = model.generate(..., speaker="Aiden")
We strongly recommend using vLLM for inference and deployment of the Qwen3-Omni series models. Since our code is currently in the pull request stage, and **audio output inference support for the Instruct model will be released in the near future**, you can follow the commands below to install vLLM from source. Please note that we recommend you **create a new Python environment** to avoid runtime environment conflicts and incompatibilities. For more details on compiling vLLM from source, please refer to the [vLLM official documentation](https://docs.vllm.ai/en/latest/getting_started/installation/gpu.html#set-up-using-python-only-build-without-compilation).
```bash
git clone -b qwen3_omni https://github.com/wangxiongts/vllm.git
git clone -b zen_omni https://github.com/wangxiongts/vllm.git
cd vllm
pip install -r requirements/build.txt
pip install -r requirements/cuda.txt
@@ -455,7 +455,7 @@ VLLM_USE_PRECOMPILED=1 pip install -e . -v --no-build-isolation
# Install the Transformers
pip install git+https://github.com/huggingface/transformers
pip install accelerate
pip install qwen-omni-utils -U
pip install zen-omni-utils -U
pip install -U flash-attn --no-build-isolation
```
@@ -469,7 +469,7 @@ import torch
from vllm import LLM, SamplingParams
from transformers import Qwen3OmniMoeProcessor
from qwen_omni_utils import process_mm_info
from zen_omni_utils import process_mm_info
if __name__ == '__main__':
# vLLM engine v1 not supported yet
@@ -545,7 +545,7 @@ import torch
from vllm import LLM, SamplingParams
from transformers import Qwen3OmniMoeProcessor
from qwen_omni_utils import process_mm_info
from zen_omni_utils import process_mm_info
def build_input(processor, messages, use_audio_in_video):
text = processor.apply_chat_template(
base-model/config.json
+7 -7
View File
@@ -40,7 +40,7 @@
"enable_audio_output": true,
"im_end_token_id": 151645,
"im_start_token_id": 151644,
"model_type": "qwen3_omni_moe",
"model_type": "zen_omni_moe",
"system_token_id": 8948,
"talker_config": {
"text_config":{
@@ -133,7 +133,7 @@
"max_position_embeddings": 32768,
"max_window_layers": 28,
"min_length": 0,
"model_type": "qwen3_omni_moe_talker_code_predictor",
"model_type": "zen_omni_moe_talker_code_predictor",
"no_repeat_ngram_size": 0,
"num_attention_heads": 16,
"num_beam_groups": 1,
@@ -181,7 +181,7 @@
"codec_think_bos_id": 2156,
"codec_think_eos_id": 2157,
"image_token_id": 151655,
"model_type": "qwen3_omni_moe_talker",
"model_type": "zen_omni_moe_talker",
"num_code_groups": 16,
"output_router_logits": false,
"position_id_per_seconds": 13,
@@ -242,7 +242,7 @@
"max_length": 20,
"max_source_positions": 1500,
"min_length": 0,
"model_type": "qwen3_omni_moe_audio_encoder",
"model_type": "zen_omni_moe_audio_encoder",
"n_window": 50,
"n_window_infer": 800,
"no_repeat_ngram_size": 0,
@@ -284,7 +284,7 @@
"dtype": "bfloat16",
"image_token_id": 151655,
"initializer_range": 0.02,
"model_type": "qwen3_omni_moe_thinker",
"model_type": "zen_omni_moe_thinker",
"position_id_per_seconds": 13,
"seconds_per_chunk": 2,
"text_config": {
@@ -330,7 +330,7 @@
"max_position_embeddings": 65536,
"min_length": 0,
"mlp_only_layers": [],
"model_type": "qwen3_omni_moe_text",
"model_type": "zen_omni_moe_text",
"moe_intermediate_size": 768,
"no_repeat_ngram_size": 0,
"norm_topk_prob": true,
@@ -437,7 +437,7 @@
"length_penalty": 1.0,
"max_length": 20,
"min_length": 0,
"model_type": "qwen3_omni_moe_vision_encoder",
"model_type": "zen_omni_moe_vision_encoder",
"no_repeat_ngram_size": 0,
"num_beam_groups": 1,
"num_beams": 1,
config.json
+1 -1
View File
@@ -4,7 +4,7 @@
"Qwen3OmniForConditionalGeneration"
],
"base_model": "Qwen3-Omni",
"model_type": "qwen3-omni",
"model_type": "zen-omni",
"multimodal": true,
"supported_modalities": ["text", "image", "audio"],
"vocab_size": 151936,
cookbooks/audio_caption.ipynb
+2 -2
View File
@@ -29,7 +29,7 @@
"warnings.filterwarnings('ignore', category=FutureWarning)\n",
"warnings.filterwarnings('ignore', category=UserWarning)\n",
"\n",
"from qwen_omni_utils import process_mm_info\n",
"from zen_omni_utils import process_mm_info\n",
"from transformers import Qwen3OmniMoeProcessor\n",
"\n",
"def _load_model_processor():\n",
@@ -378,7 +378,7 @@
],
"metadata": {
"kernelspec": {
"display_name": "qwen3_omni_public",
"display_name": "zen_omni_public",
"language": "python",
"name": "python3"
},
cookbooks/audio_function_call.ipynb
+2 -2
View File
@@ -29,7 +29,7 @@
"warnings.filterwarnings('ignore', category=FutureWarning)\n",
"warnings.filterwarnings('ignore', category=UserWarning)\n",
"\n",
"from qwen_omni_utils import process_mm_info\n",
"from zen_omni_utils import process_mm_info\n",
"from transformers import Qwen3OmniMoeProcessor\n",
"\n",
"def _load_model_processor():\n",
@@ -235,7 +235,7 @@
],
"metadata": {
"kernelspec": {
"display_name": "qwen3_omni_public",
"display_name": "zen_omni_public",
"language": "python",
"name": "python3"
},
cookbooks/audio_visual_dialogue.ipynb
+3 -3
View File
File diff suppressed because one or more lines are too long
cookbooks/audio_visual_interaction.ipynb
+3 -3
View File
@@ -29,7 +29,7 @@
"warnings.filterwarnings('ignore', category=FutureWarning)\n",
"warnings.filterwarnings('ignore', category=UserWarning)\n",
"\n",
"from qwen_omni_utils import process_mm_info\n",
"from zen_omni_utils import process_mm_info\n",
"from transformers import Qwen3OmniMoeProcessor\n",
"\n",
"def _load_model_processor():\n",
@@ -295,7 +295,7 @@
"name": "stderr",
"output_type": "stream",
"text": [
"qwen-vl-utils using torchvision to read video.\n",
"zen-vl-utils using torchvision to read video.\n",
"Adding requests: 0%| | 0/1 [00:00<?, ?it/s]Unused or unrecognized kwargs: images.\n",
"Adding requests: 100%|██████████| 1/1 [00:00<00:00, 3.81it/s]\n",
"Processed prompts: 100%|██████████| 1/1 [00:02<00:00, 2.71s/it, est. speed input: 2184.44 toks/s, output: 103.47 toks/s]"
@@ -521,7 +521,7 @@
],
"metadata": {
"kernelspec": {
"display_name": "qwen3_omni_public",
"display_name": "zen_omni_public",
"language": "python",
"name": "python3"
},
cookbooks/audio_visual_question.ipynb
+2 -2
View File
@@ -29,7 +29,7 @@
"warnings.filterwarnings('ignore', category=FutureWarning)\n",
"warnings.filterwarnings('ignore', category=UserWarning)\n",
"\n",
"from qwen_omni_utils import process_mm_info\n",
"from zen_omni_utils import process_mm_info\n",
"from transformers import Qwen3OmniMoeProcessor\n",
"\n",
"def _load_model_processor():\n",
@@ -279,7 +279,7 @@
],
"metadata": {
"kernelspec": {
"display_name": "qwen3_omni_public",
"display_name": "zen_omni_public",
"language": "python",
"name": "python3"
},
cookbooks/image_math.ipynb
+2 -2
View File
@@ -29,7 +29,7 @@
"warnings.filterwarnings('ignore', category=FutureWarning)\n",
"warnings.filterwarnings('ignore', category=UserWarning)\n",
"\n",
"from qwen_omni_utils import process_mm_info\n",
"from zen_omni_utils import process_mm_info\n",
"from transformers import Qwen3OmniMoeProcessor\n",
"\n",
"def _load_model_processor():\n",
@@ -746,7 +746,7 @@
],
"metadata": {
"kernelspec": {
"display_name": "qwen3_omni_public",
"display_name": "zen_omni_public",
"language": "python",
"name": "python3"
},
cookbooks/image_question.ipynb
+2 -2
View File
@@ -29,7 +29,7 @@
"warnings.filterwarnings('ignore', category=FutureWarning)\n",
"warnings.filterwarnings('ignore', category=UserWarning)\n",
"\n",
"from qwen_omni_utils import process_mm_info\n",
"from zen_omni_utils import process_mm_info\n",
"from transformers import Qwen3OmniMoeProcessor\n",
"\n",
"def _load_model_processor():\n",
@@ -372,7 +372,7 @@
],
"metadata": {
"kernelspec": {
"display_name": "qwen3_omni_public",
"display_name": "zen_omni_public",
"language": "python",
"name": "python3"
},
cookbooks/mixed_audio_analysis.ipynb
+2 -2
View File
@@ -29,7 +29,7 @@
"warnings.filterwarnings('ignore', category=FutureWarning)\n",
"warnings.filterwarnings('ignore', category=UserWarning)\n",
"\n",
"from qwen_omni_utils import process_mm_info\n",
"from zen_omni_utils import process_mm_info\n",
"from transformers import Qwen3OmniMoeProcessor\n",
"\n",
"def _load_model_processor():\n",
@@ -295,7 +295,7 @@
],
"metadata": {
"kernelspec": {
"display_name": "qwen3_omni_public",
"display_name": "zen_omni_public",
"language": "python",
"name": "python3"
},
cookbooks/music_analysis.ipynb
+3 -3
View File
File diff suppressed because one or more lines are too long
cookbooks/object_grounding.ipynb
+2 -2
View File
@@ -29,7 +29,7 @@
"warnings.filterwarnings('ignore', category=FutureWarning)\n",
"warnings.filterwarnings('ignore', category=UserWarning)\n",
"\n",
"from qwen_omni_utils import process_mm_info\n",
"from zen_omni_utils import process_mm_info\n",
"from transformers import Qwen3OmniMoeProcessor\n",
"\n",
"def _load_model_processor():\n",
@@ -399,7 +399,7 @@
],
"metadata": {
"kernelspec": {
"display_name": "qwen3_omni_public",
"display_name": "zen_omni_public",
"language": "python",
"name": "python3"
},
cookbooks/ocr.ipynb
+2 -2
View File
@@ -29,7 +29,7 @@
"warnings.filterwarnings('ignore', category=FutureWarning)\n",
"warnings.filterwarnings('ignore', category=UserWarning)\n",
"\n",
"from qwen_omni_utils import process_mm_info\n",
"from zen_omni_utils import process_mm_info\n",
"from transformers import Qwen3OmniMoeProcessor\n",
"\n",
"def _load_model_processor():\n",
@@ -285,7 +285,7 @@
],
"metadata": {
"kernelspec": {
"display_name": "qwen3_omni_public",
"display_name": "zen_omni_public",
"language": "python",
"name": "python3"
},
cookbooks/omni_captioner.ipynb
+2 -2
View File
@@ -47,7 +47,7 @@
"warnings.filterwarnings('ignore', category=FutureWarning)\n",
"warnings.filterwarnings('ignore', category=UserWarning)\n",
"\n",
"from qwen_omni_utils import process_mm_info\n",
"from zen_omni_utils import process_mm_info\n",
"from transformers import Qwen3OmniMoeProcessor\n",
"\n",
"def _load_model_processor():\n",
@@ -389,7 +389,7 @@
],
"metadata": {
"kernelspec": {
"display_name": "qwen3_omni_public",
"display_name": "zen_omni_public",
"language": "python",
"name": "python3"
},
cookbooks/sound_analysis.ipynb
+2 -2
View File
@@ -29,7 +29,7 @@
"warnings.filterwarnings('ignore', category=FutureWarning)\n",
"warnings.filterwarnings('ignore', category=UserWarning)\n",
"\n",
"from qwen_omni_utils import process_mm_info\n",
"from zen_omni_utils import process_mm_info\n",
"from transformers import Qwen3OmniMoeProcessor\n",
"\n",
"def _load_model_processor():\n",
@@ -361,7 +361,7 @@
],
"metadata": {
"kernelspec": {
"display_name": "qwen3_omni_public",
"display_name": "zen_omni_public",
"language": "python",
"name": "python3"
},
cookbooks/speech_recognition.ipynb
+2 -2
View File
@@ -29,7 +29,7 @@
"warnings.filterwarnings('ignore', category=FutureWarning)\n",
"warnings.filterwarnings('ignore', category=UserWarning)\n",
"\n",
"from qwen_omni_utils import process_mm_info\n",
"from zen_omni_utils import process_mm_info\n",
"from transformers import Qwen3OmniMoeProcessor\n",
"\n",
"def _load_model_processor():\n",
@@ -346,7 +346,7 @@
],
"metadata": {
"kernelspec": {
"display_name": "qwen3_omni_public",
"display_name": "zen_omni_public",
"language": "python",
"name": "python3"
},
cookbooks/speech_translation.ipynb
+2 -2
View File
@@ -29,7 +29,7 @@
"warnings.filterwarnings('ignore', category=FutureWarning)\n",
"warnings.filterwarnings('ignore', category=UserWarning)\n",
"\n",
"from qwen_omni_utils import process_mm_info\n",
"from zen_omni_utils import process_mm_info\n",
"from transformers import Qwen3OmniMoeProcessor\n",
"\n",
"def _load_model_processor():\n",
@@ -352,7 +352,7 @@
],
"metadata": {
"kernelspec": {
"display_name": "qwen3_omni_public",
"display_name": "zen_omni_public",
"language": "python",
"name": "python3"
},
cookbooks/video_description.ipynb
+3 -3
View File
@@ -29,7 +29,7 @@
"warnings.filterwarnings('ignore', category=FutureWarning)\n",
"warnings.filterwarnings('ignore', category=UserWarning)\n",
"\n",
"from qwen_omni_utils import process_mm_info\n",
"from zen_omni_utils import process_mm_info\n",
"from transformers import Qwen3OmniMoeProcessor\n",
"\n",
"def _load_model_processor():\n",
@@ -170,7 +170,7 @@
"name": "stderr",
"output_type": "stream",
"text": [
"qwen-vl-utils using torchvision to read video.\n",
"zen-vl-utils using torchvision to read video.\n",
"Adding requests: 0%| | 0/1 [00:00<?, ?it/s]Unused or unrecognized kwargs: images.\n",
"Adding requests: 100%|██████████| 1/1 [00:06<00:00, 6.19s/it]\n",
"Processed prompts: 100%|██████████| 1/1 [00:06<00:00, 6.59s/it, est. speed input: 1342.97 toks/s, output: 15.63 toks/s]"
@@ -216,7 +216,7 @@
],
"metadata": {
"kernelspec": {
"display_name": "qwen3_omni_public",
"display_name": "zen_omni_public",
"language": "python",
"name": "python3"
},
cookbooks/video_navigation.ipynb
+2 -2
View File
@@ -29,7 +29,7 @@
"warnings.filterwarnings('ignore', category=FutureWarning)\n",
"warnings.filterwarnings('ignore', category=UserWarning)\n",
"\n",
"from qwen_omni_utils import process_mm_info\n",
"from zen_omni_utils import process_mm_info\n",
"from transformers import Qwen3OmniMoeProcessor\n",
"\n",
"def _load_model_processor():\n",
@@ -213,7 +213,7 @@
],
"metadata": {
"kernelspec": {
"display_name": "qwen3_omni_public",
"display_name": "zen_omni_public",
"language": "python",
"name": "python3"
},
cookbooks/video_scene_transition.ipynb
+2 -2
View File
@@ -29,7 +29,7 @@
"warnings.filterwarnings('ignore', category=FutureWarning)\n",
"warnings.filterwarnings('ignore', category=UserWarning)\n",
"\n",
"from qwen_omni_utils import process_mm_info\n",
"from zen_omni_utils import process_mm_info\n",
"from transformers import Qwen3OmniMoeProcessor\n",
"\n",
"def _load_model_processor():\n",
@@ -214,7 +214,7 @@
],
"metadata": {
"kernelspec": {
"display_name": "qwen3_omni_public",
"display_name": "zen_omni_public",
"language": "python",
"name": "python3"
},
docker/Dockerfile-omni-3-cu124
+3 -3
View File
@@ -1,4 +1,4 @@
# Dockerfile of qwenllm/qwen3-omni:3-cu124
# Dockerfile of zenllm/zen-omni:3-cu124
ARG CUDA_VERSION=12.4.0
ARG from=nvidia/cuda:${CUDA_VERSION}-devel-ubuntu22.04
@@ -59,7 +59,7 @@ RUN --mount=type=cache,target=/root/.cache/ccache \
if [ "$BUNDLE_VLLM" = "true" ]; then \
mkdir -p /data/shared/code \
&& cd /data/shared/code \
&& git clone -b qwen3_omni https://github.com/wangxiongts/vllm.git \
&& git clone -b zen_omni https://github.com/wangxiongts/vllm.git \
&& cd vllm \
&& pip3 install -r requirements/build.txt \
&& pip3 install -r requirements/cuda.txt \
@@ -77,7 +77,7 @@ RUN --mount=type=cache,target=/root/.cache/ccache \
fi
RUN --mount=type=cache,target=/root/.cache/pip pip3 install networkx==3.4.2
RUN --mount=type=cache,target=/root/.cache/pip pip3 install accelerate==1.10.1 qwen-omni-utils huggingface_hub[cli] modelscope_studio modelscope
RUN --mount=type=cache,target=/root/.cache/pip pip3 install accelerate==1.10.1 zen-omni-utils huggingface_hub[cli] modelscope_studio modelscope
RUN --mount=type=cache,target=/root/.cache/pip \
pip3 install \
model_card.md
+1 -1
View File
@@ -78,7 +78,7 @@ context_length: 32768
## Citation
```bibtex
@article{qwen3-omni,
@article{zen-omni,
title={Qwen3-Omni: Unified Multimodal Intelligence},
author={Qwen Team},
year={2024}
upload_omni.py
+1 -1
View File
@@ -57,7 +57,7 @@ pipeline_tag: text-generation
tags:
- zen
- hanzo-ai
- qwen2
- zen2
- omni
---
web_demo.py
+1 -1
View File
@@ -12,7 +12,7 @@ import numpy as np
import soundfile as sf
from gradio_client import utils as client_utils
from qwen_omni_utils import process_mm_info
from zen_omni_utils import process_mm_info
from transformers import Qwen3OmniMoeProcessor
web_demo_captioner.py
+1 -1
View File
@@ -3,7 +3,7 @@ import torch
from argparse import ArgumentParser
import gradio as gr
from qwen_omni_utils import process_mm_info
from zen_omni_utils import process_mm_info
from transformers import Qwen3OmniMoeProcessor
os.environ['VLLM_USE_V1'] = '0'