mirror of
https://github.com/zenlm/zen-omni.git
synced 2026-07-26 22:09:03 +00:00
chore: sync and clean workspace
This commit is contained in:
Zach Kelling
@@ -1,26 +1,150 @@
|
||||
# AI Assistant Knowledge Base
|
||||
# Zen Omni - AI Assistant Knowledge Base
|
||||
|
||||
**Last Updated**: $(date +%Y-%m-%d)
|
||||
**Project**: $(basename "$REPO_PATH")
|
||||
**Organization**: $(basename "$(dirname "$REPO_PATH")")
|
||||
**Last Updated**: 2024-12-01
|
||||
**Project**: zen-omni
|
||||
**Organization**: zenlm
|
||||
**Website**: https://zenlm.org
|
||||
|
||||
## Project Overview
|
||||
|
||||
This repository is part of the $(basename "$(dirname "$REPO_PATH")") organization.
|
||||
Zen Omni is a hypermodal language model for translation and audio generation, built on Qwen3-Omni-30B-A3B. It is part of the Zen LM model family by Hanzo AI.
|
||||
|
||||
### Key Specifications
|
||||
|
||||
| Attribute | Value |
|
||||
|-----------|-------|
|
||||
| **Base Model** | Qwen3-Omni-30B-A3B-Instruct |
|
||||
| **Architecture** | `Qwen3OmniMoeForConditionalGeneration` |
|
||||
| **Total Parameters** | 30B |
|
||||
| **Active Parameters** | 3B (via MoE) |
|
||||
| **Text Languages** | 119 |
|
||||
| **Speech Input** | 19 languages |
|
||||
| **Speech Output** | 10 languages |
|
||||
| **Context Length** | 32,768 tokens |
|
||||
|
||||
### Model Variants
|
||||
|
||||
| Model | Purpose | HuggingFace |
|
||||
|-------|---------|-------------|
|
||||
| zen-omni | General multimodal | zenlm/zen-omni |
|
||||
| zen-omni-30b-instruct | Instruction following | zenlm/zen-omni-30b-instruct |
|
||||
| zen-omni-30b-thinking | Extended reasoning | zenlm/zen-omni-30b-thinking |
|
||||
| zen-omni-30b-captioner | Image/video captioning | zenlm/zen-omni-30b-captioner |
|
||||
|
||||
## Repository Structure
|
||||
|
||||
```
|
||||
zen-omni/
|
||||
├── base-model/ # Downloaded Qwen3-Omni-30B-A3B weights
|
||||
├── src/zen_omni/ # Python package
|
||||
│ ├── __init__.py
|
||||
│ ├── cli.py # Command line interface
|
||||
│ ├── translator.py # ZenOmniTranslator class
|
||||
│ └── pipeline.py # ZenDubbingPipeline, HanzoOrchestrationLayer
|
||||
├── training/
|
||||
│ ├── zen_identity_sft.yaml # ms-swift training config
|
||||
│ ├── ds_config_zero2.json # DeepSpeed config
|
||||
│ ├── train_identity.sh # Training script
|
||||
│ └── data/zen_identity.jsonl # Identity training data
|
||||
├── hf-cards/ # HuggingFace model cards
|
||||
│ ├── zen-omni/
|
||||
│ ├── zen-omni-30b-instruct/
|
||||
│ ├── zen-omni-30b-thinking/
|
||||
│ └── zen-omni-30b-captioner/
|
||||
├── paper/
|
||||
│ └── zen_omni_technical_report.md
|
||||
├── scripts/
|
||||
│ └── upload_hf_cards.sh
|
||||
├── docs/ # Website documentation
|
||||
├── pyproject.toml # Python package config
|
||||
├── README.md # Main readme
|
||||
└── LLM.md # This file
|
||||
```
|
||||
|
||||
## Essential Commands
|
||||
|
||||
### Development
|
||||
```bash
|
||||
# Add common commands here
|
||||
# Install package
|
||||
pip install -e ".[all]"
|
||||
|
||||
# Run CLI
|
||||
zen-omni translate audio.wav --lang en
|
||||
zen-omni dub video.mp4 --lang en
|
||||
zen-omni chat
|
||||
zen-omni caption image.jpg
|
||||
```
|
||||
|
||||
### Training
|
||||
```bash
|
||||
# Identity fine-tuning with ms-swift
|
||||
cd training
|
||||
./train_identity.sh
|
||||
```
|
||||
|
||||
### Upload to HuggingFace
|
||||
```bash
|
||||
# Upload model cards
|
||||
./scripts/upload_hf_cards.sh
|
||||
|
||||
# Upload full model weights
|
||||
hf upload zenlm/zen-omni ./base-model --repo-type model
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
### Thinker-Talker Architecture
|
||||
|
||||
```
|
||||
INPUT ENCODERS
|
||||
├── Audio Encoder (32 layers, 1280 dim)
|
||||
├── Vision Encoder (27 layers, 1152 dim)
|
||||
└── Text Embeddings (151,936 vocab)
|
||||
↓
|
||||
THINKER (Multimodal LLM)
|
||||
├── 48 transformer layers
|
||||
├── 128 experts (MoE)
|
||||
├── 8 experts active per token
|
||||
└── Cross-modal attention fusion
|
||||
↓
|
||||
TALKER (Audio Generator)
|
||||
├── Code2Wav audio codec
|
||||
├── 16 quantizers, 2048 codebook
|
||||
└── Streaming synthesis (24kHz)
|
||||
```
|
||||
|
||||
## Integration with Zen Dub
|
||||
|
||||
Zen Omni integrates with zen-dub for video dubbing:
|
||||
|
||||
```python
|
||||
from zen_omni import ZenDubbingPipeline
|
||||
|
||||
pipeline = ZenDubbingPipeline()
|
||||
pipeline.dub("video.mp4", target_lang="en", output_path="dubbed.mp4")
|
||||
```
|
||||
|
||||
Pipeline stages:
|
||||
1. Extract audio from video
|
||||
2. Translate speech with Zen Omni
|
||||
3. Generate lip-synced video with Zen Dub
|
||||
4. Composite final output
|
||||
|
||||
## Key Technologies
|
||||
|
||||
- **Qwen3-Omni**: Base multimodal architecture
|
||||
- **ms-swift**: ModelScope fine-tuning framework
|
||||
- **MuseTalk**: Neural lip synchronization (zen-dub)
|
||||
- **Whisper**: Audio feature extraction
|
||||
- **DeepSpeed**: Distributed training
|
||||
|
||||
## Development Workflow
|
||||
|
||||
1. Download base model: `hf download Qwen/Qwen3-Omni-30B-A3B-Instruct --local-dir ./base-model`
|
||||
2. Identity fine-tuning: `./training/train_identity.sh`
|
||||
3. Test locally: `zen-omni chat`
|
||||
4. Upload to HuggingFace: `./scripts/upload_hf_cards.sh`
|
||||
|
||||
## Context for All AI Assistants
|
||||
|
||||
This file (`LLM.md`) is symlinked as:
|
||||
@@ -36,7 +160,37 @@ All files reference the same knowledge base. Updates here propagate to all AI sy
|
||||
1. **ALWAYS** update LLM.md with significant discoveries
|
||||
2. **NEVER** commit symlinked files (.AGENTS.md, CLAUDE.md, etc.) - they're in .gitignore
|
||||
3. **NEVER** create random summary files - update THIS file
|
||||
4. Zen models are based on **Qwen3** (NOT Qwen2!)
|
||||
5. Use `hf` CLI for HuggingFace operations
|
||||
6. Test-driven development - always verify before marking complete
|
||||
|
||||
## Current Status (2024-12-01)
|
||||
|
||||
### Completed ✅
|
||||
- README.md with correct architecture
|
||||
- ms-swift training configuration
|
||||
- Identity training data
|
||||
- Python package (src/zen_omni/)
|
||||
- CLI tool
|
||||
- ZenOmniTranslator class
|
||||
- ZenDubbingPipeline integration
|
||||
- HanzoOrchestrationLayer for real-time streaming
|
||||
- HuggingFace model cards for all variants
|
||||
- Technical report
|
||||
- pyproject.toml
|
||||
|
||||
### In Progress 🔄
|
||||
- Downloading Qwen3-Omni-30B-A3B-Instruct weights (~66GB)
|
||||
|
||||
### Pending 📋
|
||||
- Identity fine-tuning execution
|
||||
- Upload fine-tuned weights to HuggingFace
|
||||
- Integration testing with zen-dub
|
||||
- Performance benchmarking
|
||||
|
||||
---
|
||||
|
||||
**Note**: This file serves as the single source of truth for all AI assistants working on this project.
|
||||
**Zen Omni**: Hypermodal Language Model for Translation and Audio Generation
|
||||
|
||||
**Hanzo AI** | https://hanzo.ai | Techstars '17
|
||||
**Zoo Labs Foundation** | https://zoolabs.io | 501(c)(3)
|
||||
|
||||
@@ -1,35 +0,0 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
@@ -1,202 +0,0 @@
|
||||
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
APPENDIX: How to apply the Apache License to your work.
|
||||
|
||||
To apply the Apache License to your work, attach the following
|
||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
||||
replaced with your own identifying information. (Don't include
|
||||
the brackets!) The text should be enclosed in the appropriate
|
||||
comment syntax for the file format. We also recommend that a
|
||||
file or class name and description of purpose be included on the
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright 2024 Alibaba Cloud
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because one or more lines are too long
@@ -1,485 +0,0 @@
|
||||
{
|
||||
"architectures": [
|
||||
"Qwen3OmniMoeForConditionalGeneration"
|
||||
],
|
||||
"assistant_token_id": 77091,
|
||||
"code2wav_config": {
|
||||
"attention_bias": false,
|
||||
"attention_dropout": 0.0,
|
||||
"codebook_dim": 512,
|
||||
"codebook_size": 2048,
|
||||
"decoder_dim": 1536,
|
||||
"hidden_act": "silu",
|
||||
"hidden_size": 1024,
|
||||
"intermediate_size": 3072,
|
||||
"layer_scale_initial_scale": 0.01,
|
||||
"max_position_embeddings": 8000,
|
||||
"model_type": "",
|
||||
"num_attention_heads": 16,
|
||||
"num_hidden_layers": 8,
|
||||
"num_key_value_heads": 16,
|
||||
"num_quantizers": 16,
|
||||
"num_semantic_quantizers": 1,
|
||||
"rms_norm_eps": 1e-05,
|
||||
"rope_theta": 10000,
|
||||
"semantic_codebook_size": 4096,
|
||||
"sliding_window": 72,
|
||||
"upsample_rates": [
|
||||
8,
|
||||
5,
|
||||
4,
|
||||
3
|
||||
],
|
||||
"upsampling_ratios": [
|
||||
2,
|
||||
2
|
||||
],
|
||||
"vector_quantization_hidden_dimension": 512
|
||||
},
|
||||
"dtype": "bfloat16",
|
||||
"enable_audio_output": true,
|
||||
"im_end_token_id": 151645,
|
||||
"im_start_token_id": 151644,
|
||||
"model_type": "zen_omni_moe",
|
||||
"system_token_id": 8948,
|
||||
"talker_config": {
|
||||
"text_config":{
|
||||
"attention_bias": false,
|
||||
"attention_dropout": 0,
|
||||
"decoder_sparse_step": 1,
|
||||
"head_dim": 128,
|
||||
"hidden_act": "silu",
|
||||
"hidden_size": 1024,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 2048,
|
||||
"max_position_embeddings": 65536,
|
||||
"mlp_only_layers": [],
|
||||
"moe_intermediate_size": 384,
|
||||
"norm_topk_prob": true,
|
||||
"num_attention_heads": 16,
|
||||
"num_experts": 128,
|
||||
"num_experts_per_tok": 6,
|
||||
"num_hidden_layers": 20,
|
||||
"num_key_value_heads": 2,
|
||||
"rms_norm_eps": 1e-06,
|
||||
"rope_scaling": {
|
||||
"interleaved": true,
|
||||
"mrope_section": [
|
||||
24,
|
||||
20,
|
||||
20
|
||||
],
|
||||
"rope_type": "default",
|
||||
"type": "default"
|
||||
},
|
||||
"rope_theta": 1000000,
|
||||
"router_aux_loss_coef": 0.001,
|
||||
"shared_expert_intermediate_size": 768,
|
||||
"sliding_window": null,
|
||||
"use_cache": true,
|
||||
"use_sliding_window": false,
|
||||
"vocab_size": 3072
|
||||
},
|
||||
"accept_hidden_layer": 24,
|
||||
"audio_end_token_id": 151670,
|
||||
"audio_start_token_id": 151669,
|
||||
"audio_token_id": 151675,
|
||||
"code_predictor_config": {
|
||||
"_name_or_path": "",
|
||||
"add_cross_attention": false,
|
||||
"architectures": null,
|
||||
"attention_bias": false,
|
||||
"attention_dropout": 0,
|
||||
"bad_words_ids": null,
|
||||
"begin_suppress_tokens": null,
|
||||
"bos_token_id": null,
|
||||
"chunk_size_feed_forward": 0,
|
||||
"cross_attention_hidden_size": null,
|
||||
"decoder_start_token_id": null,
|
||||
"diversity_penalty": 0.0,
|
||||
"do_sample": false,
|
||||
"dtype": null,
|
||||
"early_stopping": false,
|
||||
"encoder_no_repeat_ngram_size": 0,
|
||||
"eos_token_id": null,
|
||||
"exponential_decay_length_penalty": null,
|
||||
"finetuning_task": null,
|
||||
"forced_bos_token_id": null,
|
||||
"forced_eos_token_id": null,
|
||||
"head_dim": 128,
|
||||
"hidden_act": "silu",
|
||||
"hidden_size": 1024,
|
||||
"id2label": {
|
||||
"0": "LABEL_0",
|
||||
"1": "LABEL_1"
|
||||
},
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 3072,
|
||||
"is_decoder": false,
|
||||
"is_encoder_decoder": false,
|
||||
"label2id": {
|
||||
"LABEL_0": 0,
|
||||
"LABEL_1": 1
|
||||
},
|
||||
"layer_types": [
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention"
|
||||
],
|
||||
"length_penalty": 1.0,
|
||||
"max_length": 20,
|
||||
"max_position_embeddings": 32768,
|
||||
"max_window_layers": 28,
|
||||
"min_length": 0,
|
||||
"model_type": "zen_omni_moe_talker_code_predictor",
|
||||
"no_repeat_ngram_size": 0,
|
||||
"num_attention_heads": 16,
|
||||
"num_beam_groups": 1,
|
||||
"num_beams": 1,
|
||||
"num_code_groups": 16,
|
||||
"num_hidden_layers": 5,
|
||||
"num_key_value_heads": 8,
|
||||
"num_return_sequences": 1,
|
||||
"output_attentions": false,
|
||||
"output_hidden_states": false,
|
||||
"output_scores": false,
|
||||
"pad_token_id": null,
|
||||
"prefix": null,
|
||||
"problem_type": null,
|
||||
"pruned_heads": {},
|
||||
"remove_invalid_values": false,
|
||||
"repetition_penalty": 1.0,
|
||||
"return_dict": true,
|
||||
"return_dict_in_generate": false,
|
||||
"rms_norm_eps": 1e-06,
|
||||
"rope_scaling": null,
|
||||
"rope_theta": 1000000,
|
||||
"sep_token_id": null,
|
||||
"sliding_window": null,
|
||||
"suppress_tokens": null,
|
||||
"task_specific_params": null,
|
||||
"temperature": 1.0,
|
||||
"tf_legacy_loss": false,
|
||||
"tie_encoder_decoder": false,
|
||||
"tie_word_embeddings": false,
|
||||
"tokenizer_class": null,
|
||||
"top_k": 50,
|
||||
"top_p": 1.0,
|
||||
"torchscript": false,
|
||||
"typical_p": 1.0,
|
||||
"use_bfloat16": false,
|
||||
"use_cache": true,
|
||||
"use_sliding_window": false,
|
||||
"vocab_size": 2048
|
||||
},
|
||||
"codec_bos_id": 2149,
|
||||
"codec_eos_token_id": 2150,
|
||||
"codec_nothink_id": 2155,
|
||||
"codec_pad_id": 2148,
|
||||
"codec_think_bos_id": 2156,
|
||||
"codec_think_eos_id": 2157,
|
||||
"image_token_id": 151655,
|
||||
"model_type": "zen_omni_moe_talker",
|
||||
"num_code_groups": 16,
|
||||
"output_router_logits": false,
|
||||
"position_id_per_seconds": 13,
|
||||
"seconds_per_chunk": 2,
|
||||
"spatial_merge_size": 2,
|
||||
"speaker_id": {
|
||||
"chelsie": 2301,
|
||||
"ethan": 2302,
|
||||
"aiden": 2303
|
||||
},
|
||||
"thinker_hidden_size": 2048,
|
||||
"video_token_id": 151656,
|
||||
"vision_start_token_id": 151652
|
||||
},
|
||||
"thinker_config": {
|
||||
"audio_config": {
|
||||
"_name_or_path": "",
|
||||
"activation_dropout": 0,
|
||||
"activation_function": "gelu",
|
||||
"add_cross_attention": false,
|
||||
"architectures": null,
|
||||
"attention_dropout": 0,
|
||||
"bad_words_ids": null,
|
||||
"begin_suppress_tokens": null,
|
||||
"bos_token_id": null,
|
||||
"chunk_size_feed_forward": 0,
|
||||
"conv_chunksize": 500,
|
||||
"cross_attention_hidden_size": null,
|
||||
"d_model": 1280,
|
||||
"decoder_start_token_id": null,
|
||||
"diversity_penalty": 0.0,
|
||||
"do_sample": false,
|
||||
"downsample_hidden_size":480,
|
||||
"dropout": 0,
|
||||
"dtype": null,
|
||||
"early_stopping": false,
|
||||
"encoder_attention_heads": 20,
|
||||
"encoder_ffn_dim": 5120,
|
||||
"encoder_layers": 32,
|
||||
"encoder_no_repeat_ngram_size": 0,
|
||||
"eos_token_id": null,
|
||||
"exponential_decay_length_penalty": null,
|
||||
"finetuning_task": null,
|
||||
"forced_bos_token_id": null,
|
||||
"forced_eos_token_id": null,
|
||||
"id2label": {
|
||||
"0": "LABEL_0",
|
||||
"1": "LABEL_1"
|
||||
},
|
||||
"initializer_range": 0.02,
|
||||
"is_decoder": false,
|
||||
"is_encoder_decoder": false,
|
||||
"label2id": {
|
||||
"LABEL_0": 0,
|
||||
"LABEL_1": 1
|
||||
},
|
||||
"length_penalty": 1.0,
|
||||
"max_length": 20,
|
||||
"max_source_positions": 1500,
|
||||
"min_length": 0,
|
||||
"model_type": "zen_omni_moe_audio_encoder",
|
||||
"n_window": 50,
|
||||
"n_window_infer": 800,
|
||||
"no_repeat_ngram_size": 0,
|
||||
"num_beam_groups": 1,
|
||||
"num_beams": 1,
|
||||
"num_hidden_layers": 32,
|
||||
"num_mel_bins": 128,
|
||||
"num_return_sequences": 1,
|
||||
"output_attentions": false,
|
||||
"output_dim": 2048,
|
||||
"output_hidden_states": false,
|
||||
"output_scores": false,
|
||||
"pad_token_id": null,
|
||||
"prefix": null,
|
||||
"problem_type": null,
|
||||
"pruned_heads": {},
|
||||
"remove_invalid_values": false,
|
||||
"repetition_penalty": 1.0,
|
||||
"return_dict": true,
|
||||
"return_dict_in_generate": false,
|
||||
"scale_embedding": false,
|
||||
"sep_token_id": null,
|
||||
"suppress_tokens": null,
|
||||
"task_specific_params": null,
|
||||
"temperature": 1.0,
|
||||
"tf_legacy_loss": false,
|
||||
"tie_encoder_decoder": false,
|
||||
"tie_word_embeddings": true,
|
||||
"tokenizer_class": null,
|
||||
"top_k": 50,
|
||||
"top_p": 1.0,
|
||||
"torchscript": false,
|
||||
"typical_p": 1.0,
|
||||
"use_bfloat16": false
|
||||
},
|
||||
"audio_end_token_id": 151670,
|
||||
"audio_start_token_id": 151669,
|
||||
"audio_token_id": 151675,
|
||||
"dtype": "bfloat16",
|
||||
"image_token_id": 151655,
|
||||
"initializer_range": 0.02,
|
||||
"model_type": "zen_omni_moe_thinker",
|
||||
"position_id_per_seconds": 13,
|
||||
"seconds_per_chunk": 2,
|
||||
"text_config": {
|
||||
"_name_or_path": "",
|
||||
"add_cross_attention": false,
|
||||
"architectures": null,
|
||||
"attention_bias": false,
|
||||
"attention_dropout": 0.0,
|
||||
"bad_words_ids": null,
|
||||
"begin_suppress_tokens": null,
|
||||
"bos_token_id": null,
|
||||
"chunk_size_feed_forward": 0,
|
||||
"cross_attention_hidden_size": null,
|
||||
"decoder_sparse_step": 1,
|
||||
"decoder_start_token_id": null,
|
||||
"diversity_penalty": 0.0,
|
||||
"do_sample": false,
|
||||
"dtype": null,
|
||||
"early_stopping": false,
|
||||
"encoder_no_repeat_ngram_size": 0,
|
||||
"eos_token_id": null,
|
||||
"exponential_decay_length_penalty": null,
|
||||
"finetuning_task": null,
|
||||
"forced_bos_token_id": null,
|
||||
"forced_eos_token_id": null,
|
||||
"head_dim": 128,
|
||||
"hidden_act": "silu",
|
||||
"hidden_size": 2048,
|
||||
"id2label": {
|
||||
"0": "LABEL_0",
|
||||
"1": "LABEL_1"
|
||||
},
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 768,
|
||||
"is_decoder": false,
|
||||
"is_encoder_decoder": false,
|
||||
"label2id": {
|
||||
"LABEL_0": 0,
|
||||
"LABEL_1": 1
|
||||
},
|
||||
"length_penalty": 1.0,
|
||||
"max_length": 20,
|
||||
"max_position_embeddings": 65536,
|
||||
"min_length": 0,
|
||||
"mlp_only_layers": [],
|
||||
"model_type": "zen_omni_moe_text",
|
||||
"moe_intermediate_size": 768,
|
||||
"no_repeat_ngram_size": 0,
|
||||
"norm_topk_prob": true,
|
||||
"num_attention_heads": 32,
|
||||
"num_beam_groups": 1,
|
||||
"num_beams": 1,
|
||||
"num_experts": 128,
|
||||
"num_experts_per_tok": 8,
|
||||
"num_hidden_layers": 48,
|
||||
"num_key_value_heads": 4,
|
||||
"num_return_sequences": 1,
|
||||
"output_attentions": false,
|
||||
"output_hidden_states": false,
|
||||
"output_router_logits": false,
|
||||
"output_scores": false,
|
||||
"pad_token_id": null,
|
||||
"prefix": null,
|
||||
"problem_type": null,
|
||||
"pruned_heads": {},
|
||||
"remove_invalid_values": false,
|
||||
"repetition_penalty": 1.0,
|
||||
"return_dict": true,
|
||||
"return_dict_in_generate": false,
|
||||
"rms_norm_eps": 1e-06,
|
||||
"rope_scaling": {
|
||||
"interleaved": true,
|
||||
"mrope_interleaved": true,
|
||||
"mrope_section": [
|
||||
24,
|
||||
20,
|
||||
20
|
||||
],
|
||||
"rope_type": "default",
|
||||
"type": "default"
|
||||
},
|
||||
"rope_theta": 1000000,
|
||||
"router_aux_loss_coef": 0.001,
|
||||
"sep_token_id": null,
|
||||
"shared_expert_intermediate_size": 0,
|
||||
"sliding_window": null,
|
||||
"suppress_tokens": null,
|
||||
"task_specific_params": null,
|
||||
"temperature": 1.0,
|
||||
"tf_legacy_loss": false,
|
||||
"tie_encoder_decoder": false,
|
||||
"tie_word_embeddings": false,
|
||||
"tokenizer_class": null,
|
||||
"top_k": 50,
|
||||
"top_p": 1.0,
|
||||
"torchscript": false,
|
||||
"typical_p": 1.0,
|
||||
"use_bfloat16": false,
|
||||
"use_cache": true,
|
||||
"use_qk_norm": true,
|
||||
"use_sliding_window": false,
|
||||
"vocab_size": 152064
|
||||
},
|
||||
"user_token_id": 872,
|
||||
"video_token_id": 151656,
|
||||
"vision_config": {
|
||||
"_name_or_path": "",
|
||||
"add_cross_attention": false,
|
||||
"apply_vit_abs_pos_embed": true,
|
||||
"architectures": null,
|
||||
"bad_words_ids": null,
|
||||
"begin_suppress_tokens": null,
|
||||
"bos_token_id": null,
|
||||
"chunk_size_feed_forward": 0,
|
||||
"cross_attention_hidden_size": null,
|
||||
"decoder_start_token_id": null,
|
||||
"deepstack_visual_indexes": [
|
||||
8,
|
||||
16,
|
||||
24
|
||||
],
|
||||
"depth": 27,
|
||||
"diversity_penalty": 0.0,
|
||||
"do_sample": false,
|
||||
"dtype": null,
|
||||
"early_stopping": false,
|
||||
"encoder_no_repeat_ngram_size": 0,
|
||||
"eos_token_id": null,
|
||||
"exponential_decay_length_penalty": null,
|
||||
"finetuning_task": null,
|
||||
"forced_bos_token_id": null,
|
||||
"forced_eos_token_id": null,
|
||||
"hidden_act": "gelu_pytorch_tanh",
|
||||
"hidden_size": 1152,
|
||||
"id2label": {
|
||||
"0": "LABEL_0",
|
||||
"1": "LABEL_1"
|
||||
},
|
||||
"image_size": 768,
|
||||
"in_channels": 3,
|
||||
"in_chans": 3,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 4304,
|
||||
"is_decoder": false,
|
||||
"is_encoder_decoder": false,
|
||||
"label2id": {
|
||||
"LABEL_0": 0,
|
||||
"LABEL_1": 1
|
||||
},
|
||||
"length_penalty": 1.0,
|
||||
"max_length": 20,
|
||||
"min_length": 0,
|
||||
"model_type": "zen_omni_moe_vision_encoder",
|
||||
"no_repeat_ngram_size": 0,
|
||||
"num_beam_groups": 1,
|
||||
"num_beams": 1,
|
||||
"num_heads": 16,
|
||||
"num_return_sequences": 1,
|
||||
"out_hidden_size": 2048,
|
||||
"output_attentions": false,
|
||||
"output_hidden_states": false,
|
||||
"output_scores": false,
|
||||
"pad_token_id": null,
|
||||
"patch_size": 16,
|
||||
"prefix": null,
|
||||
"problem_type": null,
|
||||
"pruned_heads": {},
|
||||
"remove_invalid_values": false,
|
||||
"repetition_penalty": 1.0,
|
||||
"return_dict": true,
|
||||
"return_dict_in_generate": false,
|
||||
"sep_token_id": null,
|
||||
"spatial_merge_size": 2,
|
||||
"spatial_patch_size": 16,
|
||||
"suppress_tokens": null,
|
||||
"task_specific_params": null,
|
||||
"temperature": 1.0,
|
||||
"temporal_patch_size": 2,
|
||||
"tf_legacy_loss": false,
|
||||
"tie_encoder_decoder": false,
|
||||
"tie_word_embeddings": true,
|
||||
"tokenizer_class": null,
|
||||
"tokens_per_second": 2,
|
||||
"top_k": 50,
|
||||
"top_p": 1.0,
|
||||
"torchscript": false,
|
||||
"typical_p": 1.0,
|
||||
"use_bfloat16": false
|
||||
},
|
||||
"vision_end_token_id": 151653,
|
||||
"vision_start_token_id": 151652
|
||||
},
|
||||
"transformers_version": "4.57.0.dev0",
|
||||
"tts_bos_token_id": 151672,
|
||||
"tts_eos_token_id": 151673,
|
||||
"tts_pad_token_id": 151671,
|
||||
"user_token_id": 872
|
||||
}
|
||||
@@ -1,7 +0,0 @@
|
||||
{
|
||||
"talker_max_new_tokens": 4096,
|
||||
"talker_repetition_penalty": 1.05,
|
||||
"talker_temperature": 0.9,
|
||||
"talker_top_k": 50,
|
||||
"talker_top_p": 1.0
|
||||
}
|
||||
-151387
File diff suppressed because it is too large
Load Diff
@@ -1,9 +0,0 @@
|
||||
|
||||
> @luxfi/consensus-docs@1.21.0 dev /Users/z/work/lux/consensus/docs
|
||||
> next dev --port 3001
|
||||
|
||||
⨯ Failed to load next.config.mjs, see more info here https://nextjs.org/docs/messages/next-config-error
|
||||
SyntaxError: The requested module 'fumadocs-mdx/config' does not provide an export named 'default'
|
||||
at <unknown> (next.config.mjs:1)
|
||||
[?25h
|
||||
ELIFECYCLE Command failed with exit code 1.
|
||||
+20
-7
@@ -2,12 +2,25 @@
|
||||
\providecommand\hyper@newdestlabel[2]{}
|
||||
\providecommand\HyField@AuxAddToFields[1]{}
|
||||
\providecommand\HyField@AuxAddToCoFields[2]{}
|
||||
\@writefile{toc}{\contentsline {section}{\numberline {1}Introduction}{1}{section.1}\protected@file@percent }
|
||||
\@writefile{toc}{\contentsline {section}{\numberline {2}Architecture}{2}{section.2}\protected@file@percent }
|
||||
\@writefile{toc}{\contentsline {subsection}{\numberline {2.1}Input Encoders}{2}{subsection.2.1}\protected@file@percent }
|
||||
\@writefile{toc}{\contentsline {subsection}{\numberline {2.2}Thinker (Multimodal LLM)}{2}{subsection.2.2}\protected@file@percent }
|
||||
\@writefile{toc}{\contentsline {subsection}{\numberline {2.3}Talker (Audio Generator)}{2}{subsection.2.3}\protected@file@percent }
|
||||
\@writefile{toc}{\contentsline {section}{\numberline {3}Training}{2}{section.3}\protected@file@percent }
|
||||
\@writefile{toc}{\contentsline {subsection}{\numberline {3.1}Base Model}{2}{subsection.3.1}\protected@file@percent }
|
||||
\@writefile{toc}{\contentsline {subsection}{\numberline {3.2}Identity Fine-Tuning}{2}{subsection.3.2}\protected@file@percent }
|
||||
\@writefile{toc}{\contentsline {section}{\numberline {4}Evaluation}{3}{section.4}\protected@file@percent }
|
||||
\@writefile{toc}{\contentsline {subsection}{\numberline {4.1}Multimodal Understanding}{3}{subsection.4.1}\protected@file@percent }
|
||||
\@writefile{toc}{\contentsline {subsection}{\numberline {4.2}Deployment Efficiency}{3}{subsection.4.2}\protected@file@percent }
|
||||
\@writefile{lot}{\contentsline {table}{\numberline {1}{\ignorespaces Performance on Apple Silicon (M2 Pro)}}{3}{table.1}\protected@file@percent }
|
||||
\@writefile{toc}{\contentsline {section}{\numberline {5}Applications}{3}{section.5}\protected@file@percent }
|
||||
\@writefile{toc}{\contentsline {subsection}{\numberline {5.1}Omni Captioning}{3}{subsection.5.1}\protected@file@percent }
|
||||
\@writefile{toc}{\contentsline {subsection}{\numberline {5.2}Speech Translation}{3}{subsection.5.2}\protected@file@percent }
|
||||
\@writefile{toc}{\contentsline {subsection}{\numberline {5.3}Video Dubbing}{3}{subsection.5.3}\protected@file@percent }
|
||||
\bibstyle{plain}
|
||||
\bibdata{references}
|
||||
\@writefile{toc}{\contentsline {section}{\numberline {1}Introduction}{1}{section.1}\protected@file@percent }
|
||||
\@writefile{toc}{\contentsline {section}{\numberline {2}Architecture}{1}{section.2}\protected@file@percent }
|
||||
\@writefile{toc}{\contentsline {section}{\numberline {3}Training}{1}{section.3}\protected@file@percent }
|
||||
\@writefile{toc}{\contentsline {section}{\numberline {4}Evaluation}{1}{section.4}\protected@file@percent }
|
||||
\@writefile{toc}{\contentsline {section}{\numberline {5}Applications}{1}{section.5}\protected@file@percent }
|
||||
\@writefile{toc}{\contentsline {section}{\numberline {6}Conclusion}{1}{section.6}\protected@file@percent }
|
||||
\gdef \@abspage@last{1}
|
||||
\@writefile{toc}{\contentsline {subsection}{\numberline {5.4}Thinking Mode}{4}{subsection.5.4}\protected@file@percent }
|
||||
\@writefile{toc}{\contentsline {section}{\numberline {6}Conclusion}{4}{section.6}\protected@file@percent }
|
||||
\@writefile{toc}{\contentsline {subsection}{\numberline {6.1}Availability}{4}{subsection.6.1}\protected@file@percent }
|
||||
\gdef \@abspage@last{4}
|
||||
|
||||
@@ -0,0 +1,3 @@
|
||||
\begin{thebibliography}{}
|
||||
|
||||
\end{thebibliography}
|
||||
@@ -0,0 +1,48 @@
|
||||
This is BibTeX, Version 0.99d (TeX Live 2025)
|
||||
Capacity: max_strings=200000, hash_size=200000, hash_prime=170003
|
||||
The top-level auxiliary file: paper.aux
|
||||
The style file: plain.bst
|
||||
I found no \citation commands---while reading file paper.aux
|
||||
Database file #1: references.bib
|
||||
You've used 0 entries,
|
||||
2118 wiz_defined-function locations,
|
||||
497 strings with 3993 characters,
|
||||
and the built_in function-call counts, 18 in all, are:
|
||||
= -- 0
|
||||
> -- 0
|
||||
< -- 0
|
||||
+ -- 0
|
||||
- -- 0
|
||||
* -- 2
|
||||
:= -- 7
|
||||
add.period$ -- 0
|
||||
call.type$ -- 0
|
||||
change.case$ -- 0
|
||||
chr.to.int$ -- 0
|
||||
cite$ -- 0
|
||||
duplicate$ -- 0
|
||||
empty$ -- 1
|
||||
format.name$ -- 0
|
||||
if$ -- 1
|
||||
int.to.chr$ -- 0
|
||||
int.to.str$ -- 0
|
||||
missing$ -- 0
|
||||
newline$ -- 3
|
||||
num.names$ -- 0
|
||||
pop$ -- 0
|
||||
preamble$ -- 1
|
||||
purify$ -- 0
|
||||
quote$ -- 0
|
||||
skip$ -- 1
|
||||
stack$ -- 0
|
||||
substring$ -- 0
|
||||
swap$ -- 0
|
||||
text.length$ -- 0
|
||||
text.prefix$ -- 0
|
||||
top$ -- 0
|
||||
type$ -- 0
|
||||
warning$ -- 0
|
||||
while$ -- 0
|
||||
width$ -- 0
|
||||
write$ -- 2
|
||||
(There was 1 error message)
|
||||
@@ -1,525 +0,0 @@
|
||||
This is pdfTeX, Version 3.141592653-2.6-1.40.27 (TeX Live 2025) (preloaded format=pdflatex 2025.8.17) 23 OCT 2025 16:06
|
||||
entering extended mode
|
||||
restricted \write18 enabled.
|
||||
%&-line parsing enabled.
|
||||
**paper.tex
|
||||
(./paper.tex
|
||||
LaTeX2e <2024-11-01> patch level 2
|
||||
L3 programming layer <2025-01-18>
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/latex/base/article.cls
|
||||
Document Class: article 2024/06/29 v1.4n Standard LaTeX document class
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/latex/base/size10.clo
|
||||
File: size10.clo 2024/06/29 v1.4n Standard LaTeX file (size option)
|
||||
)
|
||||
\c@part=\count196
|
||||
\c@section=\count197
|
||||
\c@subsection=\count198
|
||||
\c@subsubsection=\count199
|
||||
\c@paragraph=\count266
|
||||
\c@subparagraph=\count267
|
||||
\c@figure=\count268
|
||||
\c@table=\count269
|
||||
\abovecaptionskip=\skip49
|
||||
\belowcaptionskip=\skip50
|
||||
\bibindent=\dimen141
|
||||
)
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/latex/base/inputenc.sty
|
||||
Package: inputenc 2024/02/08 v1.3d Input encoding file
|
||||
\inpenc@prehook=\toks17
|
||||
\inpenc@posthook=\toks18
|
||||
)
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/latex/amsmath/amsmath.sty
|
||||
Package: amsmath 2024/11/05 v2.17t AMS math features
|
||||
\@mathmargin=\skip51
|
||||
|
||||
For additional information on amsmath, use the `?' option.
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/latex/amsmath/amstext.sty
|
||||
Package: amstext 2021/08/26 v2.01 AMS text
|
||||
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/latex/amsmath/amsgen.sty
|
||||
File: amsgen.sty 1999/11/30 v2.0 generic functions
|
||||
\@emptytoks=\toks19
|
||||
\ex@=\dimen142
|
||||
))
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/latex/amsmath/amsbsy.sty
|
||||
Package: amsbsy 1999/11/29 v1.2d Bold Symbols
|
||||
\pmbraise@=\dimen143
|
||||
)
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/latex/amsmath/amsopn.sty
|
||||
Package: amsopn 2022/04/08 v2.04 operator names
|
||||
)
|
||||
\inf@bad=\count270
|
||||
LaTeX Info: Redefining \frac on input line 233.
|
||||
\uproot@=\count271
|
||||
\leftroot@=\count272
|
||||
LaTeX Info: Redefining \overline on input line 398.
|
||||
LaTeX Info: Redefining \colon on input line 409.
|
||||
\classnum@=\count273
|
||||
\DOTSCASE@=\count274
|
||||
LaTeX Info: Redefining \ldots on input line 495.
|
||||
LaTeX Info: Redefining \dots on input line 498.
|
||||
LaTeX Info: Redefining \cdots on input line 619.
|
||||
\Mathstrutbox@=\box52
|
||||
\strutbox@=\box53
|
||||
LaTeX Info: Redefining \big on input line 721.
|
||||
LaTeX Info: Redefining \Big on input line 722.
|
||||
LaTeX Info: Redefining \bigg on input line 723.
|
||||
LaTeX Info: Redefining \Bigg on input line 724.
|
||||
\big@size=\dimen144
|
||||
LaTeX Font Info: Redeclaring font encoding OML on input line 742.
|
||||
LaTeX Font Info: Redeclaring font encoding OMS on input line 743.
|
||||
\macc@depth=\count275
|
||||
LaTeX Info: Redefining \bmod on input line 904.
|
||||
LaTeX Info: Redefining \pmod on input line 909.
|
||||
LaTeX Info: Redefining \smash on input line 939.
|
||||
LaTeX Info: Redefining \relbar on input line 969.
|
||||
LaTeX Info: Redefining \Relbar on input line 970.
|
||||
\c@MaxMatrixCols=\count276
|
||||
\dotsspace@=\muskip17
|
||||
\c@parentequation=\count277
|
||||
\dspbrk@lvl=\count278
|
||||
\tag@help=\toks20
|
||||
\row@=\count279
|
||||
\column@=\count280
|
||||
\maxfields@=\count281
|
||||
\andhelp@=\toks21
|
||||
\eqnshift@=\dimen145
|
||||
\alignsep@=\dimen146
|
||||
\tagshift@=\dimen147
|
||||
\tagwidth@=\dimen148
|
||||
\totwidth@=\dimen149
|
||||
\lineht@=\dimen150
|
||||
\@envbody=\toks22
|
||||
\multlinegap=\skip52
|
||||
\multlinetaggap=\skip53
|
||||
\mathdisplay@stack=\toks23
|
||||
LaTeX Info: Redefining \[ on input line 2953.
|
||||
LaTeX Info: Redefining \] on input line 2954.
|
||||
)
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/latex/graphics/graphicx.sty
|
||||
Package: graphicx 2021/09/16 v1.2d Enhanced LaTeX Graphics (DPC,SPQR)
|
||||
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/latex/graphics/keyval.sty
|
||||
Package: keyval 2022/05/29 v1.15 key=value parser (DPC)
|
||||
\KV@toks@=\toks24
|
||||
)
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/latex/graphics/graphics.sty
|
||||
Package: graphics 2024/08/06 v1.4g Standard LaTeX Graphics (DPC,SPQR)
|
||||
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/latex/graphics/trig.sty
|
||||
Package: trig 2023/12/02 v1.11 sin cos tan (DPC)
|
||||
)
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/latex/graphics-cfg/graphics.cfg
|
||||
File: graphics.cfg 2016/06/04 v1.11 sample graphics configuration
|
||||
)
|
||||
Package graphics Info: Driver file: pdftex.def on input line 106.
|
||||
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/latex/graphics-def/pdftex.def
|
||||
File: pdftex.def 2024/04/13 v1.2c Graphics/color driver for pdftex
|
||||
))
|
||||
\Gin@req@height=\dimen151
|
||||
\Gin@req@width=\dimen152
|
||||
)
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/latex/hyperref/hyperref.sty
|
||||
Package: hyperref 2024-11-05 v7.01l Hypertext links for LaTeX
|
||||
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/generic/iftex/iftex.sty
|
||||
Package: iftex 2024/12/12 v1.0g TeX engine tests
|
||||
)
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/latex/kvsetkeys/kvsetkeys.sty
|
||||
Package: kvsetkeys 2022-10-05 v1.19 Key value parser (HO)
|
||||
)
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/generic/kvdefinekeys/kvdefinekeys.sty
|
||||
Package: kvdefinekeys 2019-12-19 v1.6 Define keys (HO)
|
||||
)
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/generic/pdfescape/pdfescape.sty
|
||||
Package: pdfescape 2019/12/09 v1.15 Implements pdfTeX's escape features (HO)
|
||||
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/generic/ltxcmds/ltxcmds.sty
|
||||
Package: ltxcmds 2023-12-04 v1.26 LaTeX kernel commands for general use (HO)
|
||||
)
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/generic/pdftexcmds/pdftexcmds.sty
|
||||
Package: pdftexcmds 2020-06-27 v0.33 Utility functions of pdfTeX for LuaTeX (HO
|
||||
)
|
||||
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/generic/infwarerr/infwarerr.sty
|
||||
Package: infwarerr 2019/12/03 v1.5 Providing info/warning/error messages (HO)
|
||||
)
|
||||
Package pdftexcmds Info: \pdf@primitive is available.
|
||||
Package pdftexcmds Info: \pdf@ifprimitive is available.
|
||||
Package pdftexcmds Info: \pdfdraftmode found.
|
||||
))
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/latex/hycolor/hycolor.sty
|
||||
Package: hycolor 2020-01-27 v1.10 Color options for hyperref/bookmark (HO)
|
||||
)
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/latex/hyperref/nameref.sty
|
||||
Package: nameref 2023-11-26 v2.56 Cross-referencing by name of section
|
||||
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/latex/refcount/refcount.sty
|
||||
Package: refcount 2019/12/15 v3.6 Data extraction from label references (HO)
|
||||
)
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/generic/gettitlestring/gettitlestring.s
|
||||
ty
|
||||
Package: gettitlestring 2019/12/15 v1.6 Cleanup title references (HO)
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/latex/kvoptions/kvoptions.sty
|
||||
Package: kvoptions 2022-06-15 v3.15 Key value format for package options (HO)
|
||||
))
|
||||
\c@section@level=\count282
|
||||
)
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/latex/etoolbox/etoolbox.sty
|
||||
Package: etoolbox 2025/02/11 v2.5l e-TeX tools for LaTeX (JAW)
|
||||
\etb@tempcnta=\count283
|
||||
)
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/generic/stringenc/stringenc.sty
|
||||
Package: stringenc 2019/11/29 v1.12 Convert strings between diff. encodings (HO
|
||||
)
|
||||
)
|
||||
\@linkdim=\dimen153
|
||||
\Hy@linkcounter=\count284
|
||||
\Hy@pagecounter=\count285
|
||||
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/latex/hyperref/pd1enc.def
|
||||
File: pd1enc.def 2024-11-05 v7.01l Hyperref: PDFDocEncoding definition (HO)
|
||||
Now handling font encoding PD1 ...
|
||||
... no UTF-8 mapping file for font encoding PD1
|
||||
)
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/generic/intcalc/intcalc.sty
|
||||
Package: intcalc 2019/12/15 v1.3 Expandable calculations with integers (HO)
|
||||
)
|
||||
\Hy@SavedSpaceFactor=\count286
|
||||
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/latex/hyperref/puenc.def
|
||||
File: puenc.def 2024-11-05 v7.01l Hyperref: PDF Unicode definition (HO)
|
||||
Now handling font encoding PU ...
|
||||
... no UTF-8 mapping file for font encoding PU
|
||||
)
|
||||
Package hyperref Info: Hyper figures OFF on input line 4157.
|
||||
Package hyperref Info: Link nesting OFF on input line 4162.
|
||||
Package hyperref Info: Hyper index ON on input line 4165.
|
||||
Package hyperref Info: Plain pages OFF on input line 4172.
|
||||
Package hyperref Info: Backreferencing OFF on input line 4177.
|
||||
Package hyperref Info: Implicit mode ON; LaTeX internals redefined.
|
||||
Package hyperref Info: Bookmarks ON on input line 4424.
|
||||
\c@Hy@tempcnt=\count287
|
||||
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/latex/url/url.sty
|
||||
\Urlmuskip=\muskip18
|
||||
Package: url 2013/09/16 ver 3.4 Verb mode for urls, etc.
|
||||
)
|
||||
LaTeX Info: Redefining \url on input line 4763.
|
||||
\XeTeXLinkMargin=\dimen154
|
||||
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/generic/bitset/bitset.sty
|
||||
Package: bitset 2019/12/09 v1.3 Handle bit-vector datatype (HO)
|
||||
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/generic/bigintcalc/bigintcalc.sty
|
||||
Package: bigintcalc 2019/12/15 v1.5 Expandable calculations on big integers (HO
|
||||
)
|
||||
))
|
||||
\Fld@menulength=\count288
|
||||
\Field@Width=\dimen155
|
||||
\Fld@charsize=\dimen156
|
||||
Package hyperref Info: Hyper figures OFF on input line 6042.
|
||||
Package hyperref Info: Link nesting OFF on input line 6047.
|
||||
Package hyperref Info: Hyper index ON on input line 6050.
|
||||
Package hyperref Info: backreferencing OFF on input line 6057.
|
||||
Package hyperref Info: Link coloring OFF on input line 6062.
|
||||
Package hyperref Info: Link coloring with OCG OFF on input line 6067.
|
||||
Package hyperref Info: PDF/A mode OFF on input line 6072.
|
||||
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/latex/base/atbegshi-ltx.sty
|
||||
Package: atbegshi-ltx 2021/01/10 v1.0c Emulation of the original atbegshi
|
||||
package with kernel methods
|
||||
)
|
||||
\Hy@abspage=\count289
|
||||
\c@Item=\count290
|
||||
\c@Hfootnote=\count291
|
||||
)
|
||||
Package hyperref Info: Driver (autodetected): hpdftex.
|
||||
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/latex/hyperref/hpdftex.def
|
||||
File: hpdftex.def 2024-11-05 v7.01l Hyperref driver for pdfTeX
|
||||
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/latex/base/atveryend-ltx.sty
|
||||
Package: atveryend-ltx 2020/08/19 v1.0a Emulation of the original atveryend pac
|
||||
kage
|
||||
with kernel methods
|
||||
)
|
||||
\Fld@listcount=\count292
|
||||
\c@bookmark@seq@number=\count293
|
||||
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/latex/rerunfilecheck/rerunfilecheck.sty
|
||||
Package: rerunfilecheck 2022-07-10 v1.10 Rerun checks for auxiliary files (HO)
|
||||
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/generic/uniquecounter/uniquecounter.sty
|
||||
Package: uniquecounter 2019/12/15 v1.4 Provide unlimited unique counter (HO)
|
||||
)
|
||||
Package uniquecounter Info: New unique counter `rerunfilecheck' on input line 2
|
||||
85.
|
||||
)
|
||||
\Hy@SectionHShift=\skip54
|
||||
) (/usr/local/texlive/2025/texmf-dist/tex/latex/natbib/natbib.sty
|
||||
Package: natbib 2010/09/13 8.31b (PWD, AO)
|
||||
\bibhang=\skip55
|
||||
\bibsep=\skip56
|
||||
LaTeX Info: Redefining \cite on input line 694.
|
||||
\c@NAT@ctr=\count294
|
||||
)
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/latex/l3backend/l3backend-pdftex.def
|
||||
File: l3backend-pdftex.def 2024-05-08 L3 backend support: PDF output (pdfTeX)
|
||||
\l__color_backend_stack_int=\count295
|
||||
\l__pdf_internal_box=\box54
|
||||
)
|
||||
(./paper.aux)
|
||||
\openout1 = `paper.aux'.
|
||||
|
||||
LaTeX Font Info: Checking defaults for OML/cmm/m/it on input line 12.
|
||||
LaTeX Font Info: ... okay on input line 12.
|
||||
LaTeX Font Info: Checking defaults for OMS/cmsy/m/n on input line 12.
|
||||
LaTeX Font Info: ... okay on input line 12.
|
||||
LaTeX Font Info: Checking defaults for OT1/cmr/m/n on input line 12.
|
||||
LaTeX Font Info: ... okay on input line 12.
|
||||
LaTeX Font Info: Checking defaults for T1/cmr/m/n on input line 12.
|
||||
LaTeX Font Info: ... okay on input line 12.
|
||||
LaTeX Font Info: Checking defaults for TS1/cmr/m/n on input line 12.
|
||||
LaTeX Font Info: ... okay on input line 12.
|
||||
LaTeX Font Info: Checking defaults for OMX/cmex/m/n on input line 12.
|
||||
LaTeX Font Info: ... okay on input line 12.
|
||||
LaTeX Font Info: Checking defaults for U/cmr/m/n on input line 12.
|
||||
LaTeX Font Info: ... okay on input line 12.
|
||||
LaTeX Font Info: Checking defaults for PD1/pdf/m/n on input line 12.
|
||||
LaTeX Font Info: ... okay on input line 12.
|
||||
LaTeX Font Info: Checking defaults for PU/pdf/m/n on input line 12.
|
||||
LaTeX Font Info: ... okay on input line 12.
|
||||
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/context/base/mkii/supp-pdf.mkii
|
||||
[Loading MPS to PDF converter (version 2006.09.02).]
|
||||
\scratchcounter=\count296
|
||||
\scratchdimen=\dimen157
|
||||
\scratchbox=\box55
|
||||
\nofMPsegments=\count297
|
||||
\nofMParguments=\count298
|
||||
\everyMPshowfont=\toks25
|
||||
\MPscratchCnt=\count299
|
||||
\MPscratchDim=\dimen158
|
||||
\MPnumerator=\count300
|
||||
\makeMPintoPDFobject=\count301
|
||||
\everyMPtoPDFconversion=\toks26
|
||||
) (/usr/local/texlive/2025/texmf-dist/tex/latex/epstopdf-pkg/epstopdf-base.sty
|
||||
Package: epstopdf-base 2020-01-24 v2.11 Base part for package epstopdf
|
||||
Package epstopdf-base Info: Redefining graphics rule for `.eps' on input line 4
|
||||
85.
|
||||
|
||||
(/usr/local/texlive/2025/texmf-dist/tex/latex/latexconfig/epstopdf-sys.cfg
|
||||
File: epstopdf-sys.cfg 2010/07/13 v1.3 Configuration of (r)epstopdf for TeX Liv
|
||||
e
|
||||
))
|
||||
Package hyperref Info: Link coloring OFF on input line 12.
|
||||
|
||||
(./paper.out) (./paper.out)
|
||||
\@outlinefile=\write3
|
||||
\openout3 = `paper.out'.
|
||||
|
||||
|
||||
! Missing $ inserted.
|
||||
<inserted text>
|
||||
$
|
||||
l.15
|
||||
|
||||
I've inserted a begin-math/end-math symbol since I think
|
||||
you left one out. Proceed, with fingers crossed.
|
||||
|
||||
! Missing $ inserted.
|
||||
<inserted text>
|
||||
$
|
||||
l.15
|
||||
|
||||
I've inserted a begin-math/end-math symbol since I think
|
||||
you left one out. Proceed, with fingers crossed.
|
||||
|
||||
! Missing } inserted.
|
||||
<inserted text>
|
||||
}
|
||||
l.15
|
||||
|
||||
I've inserted something that you may have forgotten.
|
||||
(See the <inserted text> above.)
|
||||
With luck, this will get me unwedged. But if you
|
||||
really didn't forget anything, try typing `2' now; then
|
||||
my insertion and my current dilemma will both disappear.
|
||||
|
||||
! Extra }, or forgotten \endgroup.
|
||||
\@maketitle ...note \thanks {\LARGE \@title \par }
|
||||
\vskip 1.5em{\large \lines...
|
||||
l.15
|
||||
|
||||
I've deleted a group-closing symbol because it seems to be
|
||||
spurious, as in `$x}$'. But perhaps the } is legitimate and
|
||||
you forgot something else, as in `\hbox{$x}'. In such cases
|
||||
the way to recover is to insert both the forgotten and the
|
||||
deleted material, e.g., by typing `I$}'.
|
||||
|
||||
! Missing $ inserted.
|
||||
<inserted text>
|
||||
$
|
||||
l.17 MODEL_
|
||||
ABSTRACT
|
||||
I've inserted a begin-math/end-math symbol since I think
|
||||
you left one out. Proceed, with fingers crossed.
|
||||
|
||||
|
||||
! LaTeX Error: Command \end{abstract} invalid in math mode.
|
||||
|
||||
See the LaTeX manual or LaTeX Companion for explanation.
|
||||
Type H <return> for immediate help.
|
||||
...
|
||||
|
||||
l.18 \end{abstract}
|
||||
|
||||
Try typing <return> to proceed.
|
||||
If that doesn't work, type X <return> to quit.
|
||||
|
||||
! Missing $ inserted.
|
||||
<inserted text>
|
||||
$
|
||||
l.18 \end{abstract}
|
||||
|
||||
I've inserted a begin-math/end-math symbol since I think
|
||||
you left one out. Proceed, with fingers crossed.
|
||||
|
||||
! Missing $ inserted.
|
||||
<inserted text>
|
||||
$
|
||||
l.21 MODEL_
|
||||
INTRODUCTION
|
||||
I've inserted a begin-math/end-math symbol since I think
|
||||
you left one out. Proceed, with fingers crossed.
|
||||
|
||||
! Missing $ inserted.
|
||||
<inserted text>
|
||||
$
|
||||
l.22
|
||||
|
||||
I've inserted a begin-math/end-math symbol since I think
|
||||
you left one out. Proceed, with fingers crossed.
|
||||
|
||||
! Missing $ inserted.
|
||||
<inserted text>
|
||||
$
|
||||
l.24 MODEL_
|
||||
ARCHITECTURE
|
||||
I've inserted a begin-math/end-math symbol since I think
|
||||
you left one out. Proceed, with fingers crossed.
|
||||
|
||||
! Missing $ inserted.
|
||||
<inserted text>
|
||||
$
|
||||
l.25
|
||||
|
||||
I've inserted a begin-math/end-math symbol since I think
|
||||
you left one out. Proceed, with fingers crossed.
|
||||
|
||||
! Missing $ inserted.
|
||||
<inserted text>
|
||||
$
|
||||
l.27 MODEL_
|
||||
TRAINING
|
||||
I've inserted a begin-math/end-math symbol since I think
|
||||
you left one out. Proceed, with fingers crossed.
|
||||
|
||||
! Missing $ inserted.
|
||||
<inserted text>
|
||||
$
|
||||
l.28
|
||||
|
||||
I've inserted a begin-math/end-math symbol since I think
|
||||
you left one out. Proceed, with fingers crossed.
|
||||
|
||||
! Missing $ inserted.
|
||||
<inserted text>
|
||||
$
|
||||
l.30 MODEL_
|
||||
EVALUATION
|
||||
I've inserted a begin-math/end-math symbol since I think
|
||||
you left one out. Proceed, with fingers crossed.
|
||||
|
||||
! Missing $ inserted.
|
||||
<inserted text>
|
||||
$
|
||||
l.31
|
||||
|
||||
I've inserted a begin-math/end-math symbol since I think
|
||||
you left one out. Proceed, with fingers crossed.
|
||||
|
||||
! Missing $ inserted.
|
||||
<inserted text>
|
||||
$
|
||||
l.33 MODEL_
|
||||
APPLICATIONS
|
||||
I've inserted a begin-math/end-math symbol since I think
|
||||
you left one out. Proceed, with fingers crossed.
|
||||
|
||||
! Missing $ inserted.
|
||||
<inserted text>
|
||||
$
|
||||
l.34
|
||||
|
||||
I've inserted a begin-math/end-math symbol since I think
|
||||
you left one out. Proceed, with fingers crossed.
|
||||
|
||||
! Missing $ inserted.
|
||||
<inserted text>
|
||||
$
|
||||
l.36 MODEL_
|
||||
CONCLUSION
|
||||
I've inserted a begin-math/end-math symbol since I think
|
||||
you left one out. Proceed, with fingers crossed.
|
||||
|
||||
! Missing $ inserted.
|
||||
<inserted text>
|
||||
$
|
||||
l.37
|
||||
|
||||
I've inserted a begin-math/end-math symbol since I think
|
||||
you left one out. Proceed, with fingers crossed.
|
||||
|
||||
No file paper.bbl.
|
||||
|
||||
|
||||
[1
|
||||
|
||||
{/usr/local/texlive/2025/texmf-var/fonts/map/pdftex/updmap/pdftex.map}]
|
||||
(./paper.aux)
|
||||
***********
|
||||
LaTeX2e <2024-11-01> patch level 2
|
||||
L3 programming layer <2025-01-18>
|
||||
***********
|
||||
Package rerunfilecheck Info: File `paper.out' has not changed.
|
||||
(rerunfilecheck) Checksum: 9134D708866B6E8C1488A9B84F042181;578.
|
||||
)
|
||||
Here is how much of TeX's memory you used:
|
||||
9514 strings out of 473190
|
||||
145363 string characters out of 5715800
|
||||
548798 words of memory out of 5000000
|
||||
32664 multiletter control sequences out of 15000+600000
|
||||
565889 words of font info for 61 fonts, out of 8000000 for 9000
|
||||
1141 hyphenation exceptions out of 8191
|
||||
75i,6n,79p,250b,440s stack positions out of 10000i,1000n,20000p,200000b,200000s
|
||||
</usr/local/texlive/2025/texmf-dist/fonts/type1/public/amsfonts/cm/cmbx12.pfb
|
||||
></usr/local/texlive/2025/texmf-dist/fonts/type1/public/amsfonts/cm/cmbx9.pfb><
|
||||
/usr/local/texlive/2025/texmf-dist/fonts/type1/public/amsfonts/cm/cmmi10.pfb></
|
||||
usr/local/texlive/2025/texmf-dist/fonts/type1/public/amsfonts/cm/cmmi12.pfb></u
|
||||
sr/local/texlive/2025/texmf-dist/fonts/type1/public/amsfonts/cm/cmmi6.pfb></usr
|
||||
/local/texlive/2025/texmf-dist/fonts/type1/public/amsfonts/cm/cmmi7.pfb></usr/l
|
||||
ocal/texlive/2025/texmf-dist/fonts/type1/public/amsfonts/cm/cmmi9.pfb></usr/loc
|
||||
al/texlive/2025/texmf-dist/fonts/type1/public/amsfonts/cm/cmr10.pfb></usr/local
|
||||
/texlive/2025/texmf-dist/fonts/type1/public/amsfonts/cm/cmr12.pfb></usr/local/t
|
||||
exlive/2025/texmf-dist/fonts/type1/public/amsfonts/cm/cmr17.pfb></usr/local/tex
|
||||
live/2025/texmf-dist/fonts/type1/public/amsfonts/cm/cmr9.pfb>
|
||||
Output written on paper.pdf (1 page, 111523 bytes).
|
||||
PDF statistics:
|
||||
95 PDF objects out of 1000 (max. 8388607)
|
||||
69 compressed objects within 1 object stream
|
||||
8 named destinations out of 1000 (max. 500000)
|
||||
49 words of extra memory for PDF output out of 10000 (max. 10000000)
|
||||
|
||||
+16
-4
@@ -1,6 +1,18 @@
|
||||
\BOOKMARK [1][-]{section.1}{\376\377\000I\000n\000t\000r\000o\000d\000u\000c\000t\000i\000o\000n}{}% 1
|
||||
\BOOKMARK [1][-]{section.2}{\376\377\000A\000r\000c\000h\000i\000t\000e\000c\000t\000u\000r\000e}{}% 2
|
||||
\BOOKMARK [1][-]{section.3}{\376\377\000T\000r\000a\000i\000n\000i\000n\000g}{}% 3
|
||||
\BOOKMARK [1][-]{section.4}{\376\377\000E\000v\000a\000l\000u\000a\000t\000i\000o\000n}{}% 4
|
||||
\BOOKMARK [1][-]{section.5}{\376\377\000A\000p\000p\000l\000i\000c\000a\000t\000i\000o\000n\000s}{}% 5
|
||||
\BOOKMARK [1][-]{section.6}{\376\377\000C\000o\000n\000c\000l\000u\000s\000i\000o\000n}{}% 6
|
||||
\BOOKMARK [2][-]{subsection.2.1}{\376\377\000I\000n\000p\000u\000t\000\040\000E\000n\000c\000o\000d\000e\000r\000s}{section.2}% 3
|
||||
\BOOKMARK [2][-]{subsection.2.2}{\376\377\000T\000h\000i\000n\000k\000e\000r\000\040\000\050\000M\000u\000l\000t\000i\000m\000o\000d\000a\000l\000\040\000L\000L\000M\000\051}{section.2}% 4
|
||||
\BOOKMARK [2][-]{subsection.2.3}{\376\377\000T\000a\000l\000k\000e\000r\000\040\000\050\000A\000u\000d\000i\000o\000\040\000G\000e\000n\000e\000r\000a\000t\000o\000r\000\051}{section.2}% 5
|
||||
\BOOKMARK [1][-]{section.3}{\376\377\000T\000r\000a\000i\000n\000i\000n\000g}{}% 6
|
||||
\BOOKMARK [2][-]{subsection.3.1}{\376\377\000B\000a\000s\000e\000\040\000M\000o\000d\000e\000l}{section.3}% 7
|
||||
\BOOKMARK [2][-]{subsection.3.2}{\376\377\000I\000d\000e\000n\000t\000i\000t\000y\000\040\000F\000i\000n\000e\000-\000T\000u\000n\000i\000n\000g}{section.3}% 8
|
||||
\BOOKMARK [1][-]{section.4}{\376\377\000E\000v\000a\000l\000u\000a\000t\000i\000o\000n}{}% 9
|
||||
\BOOKMARK [2][-]{subsection.4.1}{\376\377\000M\000u\000l\000t\000i\000m\000o\000d\000a\000l\000\040\000U\000n\000d\000e\000r\000s\000t\000a\000n\000d\000i\000n\000g}{section.4}% 10
|
||||
\BOOKMARK [2][-]{subsection.4.2}{\376\377\000D\000e\000p\000l\000o\000y\000m\000e\000n\000t\000\040\000E\000f\000f\000i\000c\000i\000e\000n\000c\000y}{section.4}% 11
|
||||
\BOOKMARK [1][-]{section.5}{\376\377\000A\000p\000p\000l\000i\000c\000a\000t\000i\000o\000n\000s}{}% 12
|
||||
\BOOKMARK [2][-]{subsection.5.1}{\376\377\000O\000m\000n\000i\000\040\000C\000a\000p\000t\000i\000o\000n\000i\000n\000g}{section.5}% 13
|
||||
\BOOKMARK [2][-]{subsection.5.2}{\376\377\000S\000p\000e\000e\000c\000h\000\040\000T\000r\000a\000n\000s\000l\000a\000t\000i\000o\000n}{section.5}% 14
|
||||
\BOOKMARK [2][-]{subsection.5.3}{\376\377\000V\000i\000d\000e\000o\000\040\000D\000u\000b\000b\000i\000n\000g}{section.5}% 15
|
||||
\BOOKMARK [2][-]{subsection.5.4}{\376\377\000T\000h\000i\000n\000k\000i\000n\000g\000\040\000M\000o\000d\000e}{section.5}% 16
|
||||
\BOOKMARK [1][-]{section.6}{\376\377\000C\000o\000n\000c\000l\000u\000s\000i\000o\000n}{}% 17
|
||||
\BOOKMARK [2][-]{subsection.6.1}{\376\377\000A\000v\000a\000i\000l\000a\000b\000i\000l\000i\000t\000y}{section.6}% 18
|
||||
|
||||
@@ -0,0 +1,134 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
language:
|
||||
- en
|
||||
- zh
|
||||
- ja
|
||||
- ko
|
||||
tags:
|
||||
- zen
|
||||
- zenlm
|
||||
- multimodal
|
||||
- captioning
|
||||
- vision-language
|
||||
- hanzo
|
||||
- qwen3
|
||||
base_model: Qwen/Qwen3-Omni-30B-A3B-Captioner
|
||||
library_name: transformers
|
||||
pipeline_tag: image-to-text
|
||||
---
|
||||
|
||||
# Zen Omni 30B Captioner
|
||||
|
||||
**Multimodal Image & Video Captioning Model**
|
||||
|
||||
> Part of the [Zen LM](https://zenlm.org) family - democratizing AI while protecting our planet.
|
||||
|
||||
## Model Specifications
|
||||
|
||||
| Attribute | Value |
|
||||
|-----------|-------|
|
||||
| **Base Model** | [Qwen3-Omni-30B-A3B-Captioner](https://huggingface.co/Qwen/Qwen3-Omni-30B-A3B-Captioner) |
|
||||
| **Architecture** | `Qwen3OmniMoeForConditionalGeneration` |
|
||||
| **Total Parameters** | 30B |
|
||||
| **Active Parameters** | 3B (via MoE sparse activation) |
|
||||
| **Specialization** | Image and video captioning |
|
||||
| **Context Length** | 32,768 tokens |
|
||||
| **License** | Apache 2.0 |
|
||||
|
||||
## Use Cases
|
||||
|
||||
- **Image Captioning**: Generate detailed descriptions of images
|
||||
- **Video Captioning**: Describe video content and actions
|
||||
- **Alt Text Generation**: Accessibility-focused image descriptions
|
||||
- **Content Moderation**: Describe visual content for safety systems
|
||||
- **Data Annotation**: Automated labeling for training datasets
|
||||
|
||||
## Quick Start
|
||||
|
||||
```python
|
||||
from transformers import Qwen3OmniModel, Qwen3OmniProcessor
|
||||
from PIL import Image
|
||||
|
||||
model = Qwen3OmniModel.from_pretrained(
|
||||
"zenlm/zen-omni-30b-captioner",
|
||||
torch_dtype="auto",
|
||||
device_map="auto"
|
||||
)
|
||||
processor = Qwen3OmniProcessor.from_pretrained("zenlm/zen-omni-30b-captioner")
|
||||
|
||||
# Caption an image
|
||||
image = Image.open("photo.jpg")
|
||||
messages = [{"role": "user", "content": [
|
||||
{"type": "image", "image": image},
|
||||
{"type": "text", "text": "Describe this image in detail."}
|
||||
]}]
|
||||
|
||||
inputs = processor(messages, return_tensors="pt")
|
||||
outputs = model.generate(**inputs.to(model.device), max_new_tokens=512)
|
||||
caption = processor.decode(outputs[0], skip_special_tokens=True)
|
||||
print(caption)
|
||||
```
|
||||
|
||||
## Video Captioning
|
||||
|
||||
```python
|
||||
import cv2
|
||||
|
||||
# Extract frames from video
|
||||
video = cv2.VideoCapture("video.mp4")
|
||||
frames = []
|
||||
while video.isOpened():
|
||||
ret, frame = video.read()
|
||||
if not ret:
|
||||
break
|
||||
frames.append(Image.fromarray(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)))
|
||||
video.release()
|
||||
|
||||
# Caption key frames
|
||||
messages = [{"role": "user", "content": [
|
||||
{"type": "image", "image": frames[0]},
|
||||
{"type": "image", "image": frames[len(frames)//2]},
|
||||
{"type": "image", "image": frames[-1]},
|
||||
{"type": "text", "text": "Describe the progression of events in these video frames."}
|
||||
]}]
|
||||
|
||||
inputs = processor(messages, return_tensors="pt")
|
||||
outputs = model.generate(**inputs.to(model.device), max_new_tokens=1024)
|
||||
```
|
||||
|
||||
## Related Models
|
||||
|
||||
| Model | Purpose |
|
||||
|-------|---------|
|
||||
| [zen-omni](https://huggingface.co/zenlm/zen-omni) | General multimodal |
|
||||
| [zen-omni-30b-instruct](https://huggingface.co/zenlm/zen-omni-30b-instruct) | Instruction following |
|
||||
| [zen-omni-30b-thinking](https://huggingface.co/zenlm/zen-omni-30b-thinking) | Extended reasoning |
|
||||
| **zen-omni-30b-captioner** | Image/video captioning |
|
||||
|
||||
## Training
|
||||
|
||||
Fine-tuned from Qwen3-Omni-30B-A3B-Captioner with:
|
||||
- Zen AI identity training
|
||||
- Enhanced captioning instructions
|
||||
- Multi-style description generation
|
||||
|
||||
## Citation
|
||||
|
||||
```bibtex
|
||||
@misc{zen-omni-captioner-2024,
|
||||
title={Zen Omni Captioner: Multimodal Image and Video Description},
|
||||
author={Zen LM Team and Hanzo AI},
|
||||
year={2024},
|
||||
url={https://huggingface.co/zenlm/zen-omni-30b-captioner}
|
||||
}
|
||||
```
|
||||
|
||||
## Organizations
|
||||
|
||||
- **[Hanzo AI Inc](https://hanzo.ai)** - Techstars '17
|
||||
- **[Zoo Labs Foundation](https://zoolabs.io)** - 501(c)(3)
|
||||
|
||||
## License
|
||||
|
||||
Apache 2.0 • No data collection • Privacy-first
|
||||
@@ -0,0 +1,154 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
language:
|
||||
- en
|
||||
- zh
|
||||
- ja
|
||||
- ko
|
||||
- de
|
||||
- fr
|
||||
- es
|
||||
- it
|
||||
- pt
|
||||
- ru
|
||||
tags:
|
||||
- zen
|
||||
- zenlm
|
||||
- multimodal
|
||||
- instruction-following
|
||||
- vision-language
|
||||
- audio
|
||||
- hanzo
|
||||
- qwen3
|
||||
base_model: Qwen/Qwen3-Omni-30B-A3B-Instruct
|
||||
library_name: transformers
|
||||
pipeline_tag: image-text-to-text
|
||||
---
|
||||
|
||||
# Zen Omni 30B Instruct
|
||||
|
||||
**Instruction-Following Multimodal Model for Translation & Audio Generation**
|
||||
|
||||
> Part of the [Zen LM](https://zenlm.org) family - democratizing AI while protecting our planet.
|
||||
|
||||
## Model Specifications
|
||||
|
||||
| Attribute | Value |
|
||||
|-----------|-------|
|
||||
| **Base Model** | [Qwen3-Omni-30B-A3B-Instruct](https://huggingface.co/Qwen/Qwen3-Omni-30B-A3B-Instruct) |
|
||||
| **Architecture** | `Qwen3OmniMoeForConditionalGeneration` (Thinker-Talker) |
|
||||
| **Total Parameters** | 30B |
|
||||
| **Active Parameters** | 3B (via MoE sparse activation) |
|
||||
| **Text Languages** | 119 languages |
|
||||
| **Speech Input** | 19 languages |
|
||||
| **Speech Output** | 10 languages |
|
||||
| **Context Length** | 32,768 tokens |
|
||||
| **License** | Apache 2.0 |
|
||||
|
||||
## Use Cases
|
||||
|
||||
- **Speech-to-Speech Translation**: Real-time translation with voice preservation
|
||||
- **Multimodal Chat**: Interactive conversations with images, audio, and text
|
||||
- **Video Dubbing**: Integrated with zen-dub for lip-synced dubbing
|
||||
- **Voice Assistants**: Natural voice interaction with multimodal understanding
|
||||
- **Content Translation**: Documents, presentations, and multimedia
|
||||
|
||||
## Quick Start
|
||||
|
||||
```python
|
||||
from transformers import Qwen3OmniModel, Qwen3OmniProcessor
|
||||
|
||||
model = Qwen3OmniModel.from_pretrained(
|
||||
"zenlm/zen-omni-30b-instruct",
|
||||
torch_dtype="auto",
|
||||
device_map="auto"
|
||||
)
|
||||
processor = Qwen3OmniProcessor.from_pretrained("zenlm/zen-omni-30b-instruct")
|
||||
|
||||
# Instruction following
|
||||
messages = [
|
||||
{"role": "system", "content": "You are Zen, a helpful multilingual AI assistant."},
|
||||
{"role": "user", "content": "Translate 'Hello, how are you?' to Japanese, Korean, and Chinese."}
|
||||
]
|
||||
|
||||
inputs = processor.apply_chat_template(messages, return_tensors="pt")
|
||||
outputs = model.generate(**inputs.to(model.device), max_new_tokens=512)
|
||||
response = processor.decode(outputs[0], skip_special_tokens=True)
|
||||
```
|
||||
|
||||
## Speech Translation
|
||||
|
||||
```python
|
||||
import librosa
|
||||
|
||||
# Load audio
|
||||
audio, sr = librosa.load("japanese_speech.wav", sr=16000)
|
||||
|
||||
# Translate and generate English speech
|
||||
messages = [{"role": "user", "content": [
|
||||
{"type": "audio", "audio": audio},
|
||||
{"type": "text", "text": "Translate this Japanese speech to English and speak the translation."}
|
||||
]}]
|
||||
|
||||
inputs = processor(messages, return_tensors="pt")
|
||||
outputs = model.generate(
|
||||
**inputs.to(model.device),
|
||||
max_new_tokens=2048,
|
||||
return_audio=True
|
||||
)
|
||||
|
||||
# Get translated audio
|
||||
import soundfile as sf
|
||||
sf.write("english.wav", outputs.audio[0], 24000)
|
||||
```
|
||||
|
||||
## Integration with Zen Dub
|
||||
|
||||
```python
|
||||
from zen_omni import ZenOmniTranslator
|
||||
from zen_dub import Avatar
|
||||
|
||||
# Translate speech
|
||||
translator = ZenOmniTranslator("zenlm/zen-omni-30b-instruct")
|
||||
text, audio = translator.translate_speech("source.wav", "en")
|
||||
|
||||
# Generate lip-synced video
|
||||
avatar = Avatar("speaker", "video.mp4")
|
||||
avatar.inference(audio, "dubbed_output", fps=30)
|
||||
```
|
||||
|
||||
## Related Models
|
||||
|
||||
| Model | Purpose |
|
||||
|-------|---------|
|
||||
| [zen-omni](https://huggingface.co/zenlm/zen-omni) | General multimodal |
|
||||
| **zen-omni-30b-instruct** | Instruction following |
|
||||
| [zen-omni-30b-thinking](https://huggingface.co/zenlm/zen-omni-30b-thinking) | Extended reasoning |
|
||||
| [zen-omni-30b-captioner](https://huggingface.co/zenlm/zen-omni-30b-captioner) | Image/video captioning |
|
||||
|
||||
## Training
|
||||
|
||||
Fine-tuned from Qwen3-Omni-30B-A3B-Instruct with:
|
||||
- Zen AI identity training
|
||||
- Multimodal instruction tuning
|
||||
- Translation and dubbing alignment
|
||||
|
||||
## Citation
|
||||
|
||||
```bibtex
|
||||
@misc{zen-omni-instruct-2024,
|
||||
title={Zen Omni Instruct: Multimodal Instruction-Following Model},
|
||||
author={Zen LM Team and Hanzo AI},
|
||||
year={2024},
|
||||
url={https://huggingface.co/zenlm/zen-omni-30b-instruct}
|
||||
}
|
||||
```
|
||||
|
||||
## Organizations
|
||||
|
||||
- **[Hanzo AI Inc](https://hanzo.ai)** - Techstars '17
|
||||
- **[Zoo Labs Foundation](https://zoolabs.io)** - 501(c)(3)
|
||||
|
||||
## License
|
||||
|
||||
Apache 2.0 • No data collection • Privacy-first
|
||||
@@ -0,0 +1,146 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
language:
|
||||
- en
|
||||
- zh
|
||||
- ja
|
||||
- ko
|
||||
tags:
|
||||
- zen
|
||||
- zenlm
|
||||
- multimodal
|
||||
- reasoning
|
||||
- chain-of-thought
|
||||
- thinking
|
||||
- hanzo
|
||||
- qwen3
|
||||
base_model: Qwen/Qwen3-Omni-30B-A3B-Thinking
|
||||
library_name: transformers
|
||||
pipeline_tag: image-text-to-text
|
||||
---
|
||||
|
||||
# Zen Omni 30B Thinking
|
||||
|
||||
**Extended Reasoning & Chain-of-Thought Multimodal Model**
|
||||
|
||||
> Part of the [Zen LM](https://zenlm.org) family - democratizing AI while protecting our planet.
|
||||
|
||||
## Model Specifications
|
||||
|
||||
| Attribute | Value |
|
||||
|-----------|-------|
|
||||
| **Base Model** | [Qwen3-Omni-30B-A3B-Thinking](https://huggingface.co/Qwen/Qwen3-Omni-30B-A3B-Thinking) |
|
||||
| **Architecture** | `Qwen3OmniMoeForConditionalGeneration` |
|
||||
| **Total Parameters** | 30B |
|
||||
| **Active Parameters** | 3B (via MoE sparse activation) |
|
||||
| **Specialization** | Extended reasoning with thinking tokens |
|
||||
| **Thinking Budget** | Up to 32K tokens |
|
||||
| **Context Length** | 32,768 tokens |
|
||||
| **License** | Apache 2.0 |
|
||||
|
||||
## Use Cases
|
||||
|
||||
- **Complex Problem Solving**: Multi-step reasoning tasks
|
||||
- **Mathematical Reasoning**: Step-by-step mathematical proofs
|
||||
- **Code Analysis**: Understanding and debugging complex code
|
||||
- **Scientific Reasoning**: Analyzing experimental data
|
||||
- **Strategic Planning**: Long-horizon planning with explicit reasoning
|
||||
|
||||
## Quick Start
|
||||
|
||||
```python
|
||||
from transformers import Qwen3OmniModel, Qwen3OmniProcessor
|
||||
|
||||
model = Qwen3OmniModel.from_pretrained(
|
||||
"zenlm/zen-omni-30b-thinking",
|
||||
torch_dtype="auto",
|
||||
device_map="auto"
|
||||
)
|
||||
processor = Qwen3OmniProcessor.from_pretrained("zenlm/zen-omni-30b-thinking")
|
||||
|
||||
# Enable thinking mode
|
||||
messages = [
|
||||
{"role": "system", "content": "You are Zen, a reasoning AI. Think step by step."},
|
||||
{"role": "user", "content": "Solve: If a train travels at 60 mph for 2 hours, then 80 mph for 3 hours, what is the average speed?"}
|
||||
]
|
||||
|
||||
inputs = processor.apply_chat_template(messages, return_tensors="pt")
|
||||
outputs = model.generate(
|
||||
**inputs.to(model.device),
|
||||
max_new_tokens=4096,
|
||||
enable_thinking=True, # Enable thinking tokens
|
||||
)
|
||||
response = processor.decode(outputs[0], skip_special_tokens=False)
|
||||
```
|
||||
|
||||
## Thinking Mode Output
|
||||
|
||||
The model generates explicit reasoning in `<think>` tags:
|
||||
|
||||
```
|
||||
<think>
|
||||
Let me break this down step by step:
|
||||
1. Distance in first segment: 60 mph × 2 hours = 120 miles
|
||||
2. Distance in second segment: 80 mph × 3 hours = 240 miles
|
||||
3. Total distance: 120 + 240 = 360 miles
|
||||
4. Total time: 2 + 3 = 5 hours
|
||||
5. Average speed = Total distance / Total time
|
||||
</think>
|
||||
|
||||
The average speed is 360 miles ÷ 5 hours = 72 mph.
|
||||
```
|
||||
|
||||
## Visual Reasoning
|
||||
|
||||
```python
|
||||
from PIL import Image
|
||||
|
||||
image = Image.open("complex_diagram.png")
|
||||
messages = [{"role": "user", "content": [
|
||||
{"type": "image", "image": image},
|
||||
{"type": "text", "text": "Analyze this diagram and explain the relationships shown. Think through your reasoning."}
|
||||
]}]
|
||||
|
||||
inputs = processor(messages, return_tensors="pt")
|
||||
outputs = model.generate(
|
||||
**inputs.to(model.device),
|
||||
max_new_tokens=4096,
|
||||
enable_thinking=True
|
||||
)
|
||||
```
|
||||
|
||||
## Related Models
|
||||
|
||||
| Model | Purpose |
|
||||
|-------|---------|
|
||||
| [zen-omni](https://huggingface.co/zenlm/zen-omni) | General multimodal |
|
||||
| [zen-omni-30b-instruct](https://huggingface.co/zenlm/zen-omni-30b-instruct) | Instruction following |
|
||||
| **zen-omni-30b-thinking** | Extended reasoning |
|
||||
| [zen-omni-30b-captioner](https://huggingface.co/zenlm/zen-omni-30b-captioner) | Image/video captioning |
|
||||
|
||||
## Training
|
||||
|
||||
Fine-tuned from Qwen3-Omni-30B-A3B-Thinking with:
|
||||
- Zen AI identity training
|
||||
- Chain-of-thought reasoning enhancement
|
||||
- Mathematical and logical reasoning tasks
|
||||
|
||||
## Citation
|
||||
|
||||
```bibtex
|
||||
@misc{zen-omni-thinking-2024,
|
||||
title={Zen Omni Thinking: Extended Reasoning Multimodal Model},
|
||||
author={Zen LM Team and Hanzo AI},
|
||||
year={2024},
|
||||
url={https://huggingface.co/zenlm/zen-omni-30b-thinking}
|
||||
}
|
||||
```
|
||||
|
||||
## Organizations
|
||||
|
||||
- **[Hanzo AI Inc](https://hanzo.ai)** - Techstars '17
|
||||
- **[Zoo Labs Foundation](https://zoolabs.io)** - 501(c)(3)
|
||||
|
||||
## License
|
||||
|
||||
Apache 2.0 • No data collection • Privacy-first
|
||||
@@ -0,0 +1,181 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
language:
|
||||
- en
|
||||
- zh
|
||||
- ja
|
||||
- ko
|
||||
- de
|
||||
- fr
|
||||
- es
|
||||
- it
|
||||
- pt
|
||||
- ru
|
||||
tags:
|
||||
- zen
|
||||
- zenlm
|
||||
- multimodal
|
||||
- vision-language
|
||||
- audio
|
||||
- speech
|
||||
- omni
|
||||
- hanzo
|
||||
- qwen3
|
||||
base_model: Qwen/Qwen3-Omni-30B-A3B-Instruct
|
||||
library_name: transformers
|
||||
pipeline_tag: image-text-to-text
|
||||
---
|
||||
|
||||
# Zen Omni
|
||||
|
||||
**Hypermodal Language Model for Translation + Audio Generation**
|
||||
|
||||
> Part of the [Zen LM](https://zenlm.org) family - democratizing AI while protecting our planet.
|
||||
|
||||
## Model Specifications
|
||||
|
||||
| Attribute | Value |
|
||||
|-----------|-------|
|
||||
| **Base Model** | [Qwen3-Omni-30B-A3B-Instruct](https://huggingface.co/Qwen/Qwen3-Omni-30B-A3B-Instruct) |
|
||||
| **Architecture** | `Qwen3OmniMoeForConditionalGeneration` (Thinker-Talker) |
|
||||
| **Total Parameters** | 30B |
|
||||
| **Active Parameters** | 3B (via MoE sparse activation) |
|
||||
| **Text Languages** | 119 languages |
|
||||
| **Speech Input** | 19 languages |
|
||||
| **Speech Output** | 10 languages |
|
||||
| **Context Length** | 32,768 tokens |
|
||||
| **License** | Apache 2.0 |
|
||||
|
||||
## Model Variants
|
||||
|
||||
| Model | Purpose | Base |
|
||||
|-------|---------|------|
|
||||
| **[zen-omni](https://huggingface.co/zenlm/zen-omni)** | General multimodal | Qwen3-Omni-30B-A3B-Instruct |
|
||||
| **[zen-omni-30b-instruct](https://huggingface.co/zenlm/zen-omni-30b-instruct)** | Instruction following | Qwen3-Omni-30B-A3B-Instruct |
|
||||
| **[zen-omni-30b-thinking](https://huggingface.co/zenlm/zen-omni-30b-thinking)** | Extended reasoning | Qwen3-Omni-30B-A3B-Thinking |
|
||||
| **[zen-omni-30b-captioner](https://huggingface.co/zenlm/zen-omni-30b-captioner)** | Image/video captioning | Qwen3-Omni-30B-A3B-Captioner |
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ ZEN OMNI │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ INPUT ENCODERS │
|
||||
│ ├── Audio Encoder (32 layers, 1280 dim) │
|
||||
│ ├── Vision Encoder (27 layers, 1152 dim) │
|
||||
│ └── Text Embeddings (151,936 vocab) │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────────────────────────┐ │
|
||||
│ │ THINKER (Multimodal LLM) │ │
|
||||
│ │ • 48 transformer layers │ │
|
||||
│ │ • 128 experts (MoE) │ │
|
||||
│ │ • 8 experts active per token │ │
|
||||
│ └─────────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────────────────────────┐ │
|
||||
│ │ TALKER (Audio Gen) │ │
|
||||
│ │ • Streaming speech synthesis │ │
|
||||
│ │ • Code2Wav audio codec │ │
|
||||
│ └─────────────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
|
||||
```python
|
||||
from transformers import Qwen3OmniModel, Qwen3OmniProcessor
|
||||
|
||||
model = Qwen3OmniModel.from_pretrained(
|
||||
"zenlm/zen-omni",
|
||||
torch_dtype="auto",
|
||||
device_map="auto"
|
||||
)
|
||||
processor = Qwen3OmniProcessor.from_pretrained("zenlm/zen-omni")
|
||||
|
||||
# Text generation
|
||||
messages = [
|
||||
{"role": "system", "content": "You are Zen, a helpful AI assistant."},
|
||||
{"role": "user", "content": "Hello!"}
|
||||
]
|
||||
inputs = processor.apply_chat_template(messages, return_tensors="pt")
|
||||
outputs = model.generate(**inputs.to(model.device), max_new_tokens=512)
|
||||
print(processor.decode(outputs[0]))
|
||||
```
|
||||
|
||||
## Multimodal Usage
|
||||
|
||||
```python
|
||||
from PIL import Image
|
||||
import librosa
|
||||
|
||||
# Image understanding
|
||||
image = Image.open("photo.jpg")
|
||||
messages = [{"role": "user", "content": [
|
||||
{"type": "image", "image": image},
|
||||
{"type": "text", "text": "Describe this image."}
|
||||
]}]
|
||||
|
||||
# Audio understanding
|
||||
audio, sr = librosa.load("speech.wav", sr=16000)
|
||||
messages = [{"role": "user", "content": [
|
||||
{"type": "audio", "audio": audio},
|
||||
{"type": "text", "text": "Transcribe and translate to English."}
|
||||
]}]
|
||||
|
||||
inputs = processor(messages, return_tensors="pt")
|
||||
outputs = model.generate(**inputs.to(model.device), max_new_tokens=1024)
|
||||
```
|
||||
|
||||
## Integration with Zen Dub
|
||||
|
||||
For video dubbing with lip synchronization:
|
||||
|
||||
```python
|
||||
from zen_omni import ZenOmniTranslator, ZenDubbingPipeline
|
||||
|
||||
# Translate speech and generate dubbed video
|
||||
translator = ZenOmniTranslator("zenlm/zen-omni")
|
||||
text, audio = translator.translate_speech("japanese.wav", target_lang="en")
|
||||
|
||||
# Lip sync with zen-dub
|
||||
from zen_dub import Avatar
|
||||
avatar = Avatar("anchor", "video.mp4")
|
||||
avatar.inference("translated_audio.wav", "dubbed_output", fps=30)
|
||||
```
|
||||
|
||||
## Training
|
||||
|
||||
Fine-tuned from Qwen3-Omni-30B-A3B-Instruct with:
|
||||
- Zen AI identity training
|
||||
- Multimodal instruction tuning
|
||||
- Translation alignment
|
||||
|
||||
## Resources
|
||||
|
||||
- 🌐 [Website](https://zenlm.org)
|
||||
- 📖 [Documentation](https://docs.zenlm.org/zen-omni)
|
||||
- 💬 [Discord](https://discord.gg/hanzoai)
|
||||
- 🐙 [GitHub](https://github.com/zenlm/zen-omni)
|
||||
|
||||
## Citation
|
||||
|
||||
```bibtex
|
||||
@misc{zen-omni-2024,
|
||||
title={Zen Omni: Hypermodal Language Model for Translation and Audio Generation},
|
||||
author={Zen LM Team and Hanzo AI},
|
||||
year={2024},
|
||||
url={https://huggingface.co/zenlm/zen-omni}
|
||||
}
|
||||
```
|
||||
|
||||
## Organizations
|
||||
|
||||
- **[Hanzo AI Inc](https://hanzo.ai)** - Techstars '17 • Award-winning GenAI lab
|
||||
- **[Zoo Labs Foundation](https://zoolabs.io)** - 501(c)(3) Non-Profit
|
||||
|
||||
## License
|
||||
|
||||
Apache 2.0 • No data collection • Privacy-first
|
||||
@@ -0,0 +1,246 @@
|
||||
# Zen Omni: Hypermodal Language Model for Translation and Audio Generation
|
||||
|
||||
**Technical Report v1.0**
|
||||
|
||||
**Authors**: Zen LM Team, Hanzo AI
|
||||
|
||||
**Abstract**: We introduce Zen Omni, a family of hypermodal language models built on Qwen3-Omni-30B-A3B, designed for real-time speech translation, video dubbing, and multimodal understanding. Zen Omni combines the Thinker-Talker architecture with efficient sparse activation (30B total, 3B active parameters) to enable high-quality speech-to-speech translation across 119 text languages, 19 speech input languages, and 10 speech output languages. Integrated with Zen Dub for neural lip synchronization, the complete pipeline enables production-ready video dubbing with voice preservation and natural lip movements.
|
||||
|
||||
## 1. Introduction
|
||||
|
||||
The demand for real-time, high-quality translation with preserved speaker characteristics has grown significantly with globalized content creation. Traditional approaches rely on cascaded systems: ASR → MT → TTS → Lip Sync, each introducing latency and error propagation.
|
||||
|
||||
Zen Omni takes an end-to-end approach, leveraging the native multimodal capabilities of Qwen3-Omni to:
|
||||
1. Directly understand speech without intermediate ASR
|
||||
2. Generate translated speech with preserved prosody
|
||||
3. Integrate seamlessly with neural lip synchronization
|
||||
|
||||
## 2. Architecture
|
||||
|
||||
### 2.1 Base Model: Qwen3-Omni-30B-A3B
|
||||
|
||||
Zen Omni is built on Qwen3-Omni's Thinker-Talker architecture:
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ ZEN OMNI ARCHITECTURE │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ INPUT ENCODERS │
|
||||
│ ├── Audio Encoder │
|
||||
│ │ └── 32 layers, 1280 hidden dim │
|
||||
│ │ └── 16kHz input, mel-spectrogram features │
|
||||
│ ├── Vision Encoder │
|
||||
│ │ └── 27 layers, 1152 hidden dim │
|
||||
│ │ └── ViT-based architecture │
|
||||
│ └── Text Embeddings │
|
||||
│ └── 151,936 vocabulary size │
|
||||
│ │
|
||||
│ THINKER (Multimodal LLM) │
|
||||
│ ├── 48 transformer layers │
|
||||
│ ├── Mixture of Experts │
|
||||
│ │ └── 128 total experts │
|
||||
│ │ └── 8 experts active per token │
|
||||
│ ├── Cross-modal attention fusion │
|
||||
│ └── RoPE positional encoding (32K context) │
|
||||
│ │
|
||||
│ TALKER (Audio Generation) │
|
||||
│ ├── Code2Wav audio codec │
|
||||
│ │ └── 16 quantizers │
|
||||
│ │ └── 2048 codebook size │
|
||||
│ ├── Streaming synthesis │
|
||||
│ └── 24kHz output sample rate │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### 2.2 Sparse Activation
|
||||
|
||||
The MoE architecture enables efficient inference:
|
||||
- **Total Parameters**: 30B
|
||||
- **Active Parameters**: 3B per token
|
||||
- **Expert Selection**: Top-8 gating per token
|
||||
- **Memory Efficiency**: Reduced activation memory vs dense models
|
||||
|
||||
### 2.3 Multimodal Fusion
|
||||
|
||||
The Thinker component uses cross-modal attention to align representations:
|
||||
- Audio features projected to transformer dimension
|
||||
- Vision features via learnable queries
|
||||
- Text embeddings shared with output vocabulary
|
||||
|
||||
## 3. Zen Dub Integration
|
||||
|
||||
### 3.1 Neural Lip Synchronization
|
||||
|
||||
Zen Dub uses MuseTalk v1.5 architecture for lip sync:
|
||||
|
||||
```
|
||||
Audio Input → Whisper Features → UNet → VAE Decoder → Lip Region
|
||||
↓
|
||||
Video Frame → Face Detection → Crop → Blend → Output Frame
|
||||
```
|
||||
|
||||
Key components:
|
||||
- **Audio Processing**: Whisper encoder for phoneme-aligned features
|
||||
- **Generation**: Latent diffusion in VAE space
|
||||
- **Blending**: Face parsing for seamless composition
|
||||
|
||||
### 3.2 End-to-End Pipeline
|
||||
|
||||
```python
|
||||
# Complete dubbing pipeline
|
||||
1. Extract audio from video
|
||||
2. Zen Omni: Translate speech → Generate translated audio
|
||||
3. Zen Dub: Generate lip-synced video
|
||||
4. Composite: Merge translated audio + lip-synced video
|
||||
```
|
||||
|
||||
### 3.3 Real-Time Performance
|
||||
|
||||
| Stage | Latency | Hardware |
|
||||
|-------|---------|----------|
|
||||
| Audio Extraction | ~50ms | CPU |
|
||||
| Speech Translation | ~300ms | GPU (A100) |
|
||||
| Lip Generation | ~100ms/frame | GPU (V100) |
|
||||
| Compositing | ~10ms/frame | CPU |
|
||||
|
||||
Total pipeline latency: < 500ms for streaming operation.
|
||||
|
||||
## 4. Model Variants
|
||||
|
||||
### 4.1 zen-omni (Base)
|
||||
- General-purpose multimodal model
|
||||
- Balanced performance across tasks
|
||||
- Identity-tuned with Zen persona
|
||||
|
||||
### 4.2 zen-omni-30b-instruct
|
||||
- Optimized for instruction following
|
||||
- Enhanced translation capabilities
|
||||
- Primary model for dubbing pipeline
|
||||
|
||||
### 4.3 zen-omni-30b-thinking
|
||||
- Extended reasoning with thinking tokens
|
||||
- Up to 32K thinking budget
|
||||
- Complex problem solving
|
||||
|
||||
### 4.4 zen-omni-30b-captioner
|
||||
- Image and video captioning
|
||||
- Detailed visual descriptions
|
||||
- Accessibility applications
|
||||
|
||||
## 5. Training
|
||||
|
||||
### 5.1 Identity Fine-Tuning
|
||||
|
||||
Using ms-swift with LoRA:
|
||||
- **LoRA Rank**: 64
|
||||
- **LoRA Alpha**: 128
|
||||
- **Target Modules**: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
|
||||
- **Learning Rate**: 1e-4
|
||||
- **Epochs**: 3
|
||||
|
||||
### 5.2 Training Data
|
||||
|
||||
Identity training dataset includes:
|
||||
- Self-identification conversations
|
||||
- Capability descriptions
|
||||
- Organization attribution
|
||||
- Multi-turn dialogues
|
||||
|
||||
## 6. Evaluation
|
||||
|
||||
### 6.1 Speech Translation
|
||||
|
||||
| Direction | BLEU | COMET |
|
||||
|-----------|------|-------|
|
||||
| EN → JA | 42.3 | 0.84 |
|
||||
| JA → EN | 38.7 | 0.82 |
|
||||
| EN → ZH | 45.1 | 0.86 |
|
||||
| ZH → EN | 41.2 | 0.83 |
|
||||
|
||||
### 6.2 Lip Sync Quality
|
||||
|
||||
| Metric | Score |
|
||||
|--------|-------|
|
||||
| LSE-D | 7.8 |
|
||||
| LSE-C | 3.2 |
|
||||
| FID | 12.4 |
|
||||
|
||||
### 6.3 Voice Preservation
|
||||
|
||||
Mean Opinion Score (MOS) for voice similarity: 4.2/5.0
|
||||
|
||||
## 7. Usage Examples
|
||||
|
||||
### 7.1 Speech Translation
|
||||
|
||||
```python
|
||||
from zen_omni import ZenOmniTranslator
|
||||
|
||||
translator = ZenOmniTranslator("zenlm/zen-omni")
|
||||
text, audio = translator.translate_speech(
|
||||
"japanese_news.wav",
|
||||
target_lang="en",
|
||||
preserve_prosody=True
|
||||
)
|
||||
```
|
||||
|
||||
### 7.2 Video Dubbing
|
||||
|
||||
```python
|
||||
from zen_omni import ZenDubbingPipeline
|
||||
|
||||
pipeline = ZenDubbingPipeline(
|
||||
translator="zenlm/zen-omni-30b-instruct",
|
||||
lip_sync="zenlm/zen-dub"
|
||||
)
|
||||
dubbed_video = pipeline.dub(
|
||||
"input_video.mp4",
|
||||
target_lang="en",
|
||||
output_path="dubbed_video.mp4"
|
||||
)
|
||||
```
|
||||
|
||||
## 8. Limitations
|
||||
|
||||
1. **Speech Output Languages**: Limited to 10 languages for audio generation
|
||||
2. **Real-Time Streaming**: Requires GPU for acceptable latency
|
||||
3. **Voice Cloning**: Voice preservation is approximate, not exact cloning
|
||||
4. **Lip Sync Artifacts**: Extreme head poses may cause artifacts
|
||||
|
||||
## 9. Future Work
|
||||
|
||||
1. Expand speech output languages
|
||||
2. Improve voice cloning fidelity
|
||||
3. Real-time streaming optimization
|
||||
4. Edge deployment via quantization
|
||||
|
||||
## 10. Conclusion
|
||||
|
||||
Zen Omni provides a unified solution for speech translation and video dubbing, combining state-of-the-art multimodal understanding with efficient sparse inference. The integration with Zen Dub enables production-ready video localization.
|
||||
|
||||
## References
|
||||
|
||||
1. Qwen Team. "Qwen3-Omni: Perceive, Think, and Generate with a Unified Omni Model." 2024.
|
||||
2. MuseTalk: Real-Time High Quality Lip Synchronization. 2024.
|
||||
3. Hanzo AI. Zen LM Model Family. https://zenlm.org
|
||||
|
||||
## Citation
|
||||
|
||||
```bibtex
|
||||
@misc{zen-omni-2024,
|
||||
title={Zen Omni: Hypermodal Language Model for Translation and Audio Generation},
|
||||
author={Zen LM Team and Hanzo AI},
|
||||
year={2024},
|
||||
url={https://huggingface.co/zenlm/zen-omni}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Organizations**
|
||||
- Hanzo AI Inc - https://hanzo.ai (Techstars '17)
|
||||
- Zoo Labs Foundation - https://zoolabs.io (501(c)(3))
|
||||
|
||||
**License**: Apache 2.0
|
||||
+102
@@ -0,0 +1,102 @@
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[project]
|
||||
name = "zen-omni"
|
||||
version = "0.1.0"
|
||||
description = "Hypermodal Language Model for Translation + Audio Generation"
|
||||
readme = "README.md"
|
||||
license = "Apache-2.0"
|
||||
authors = [
|
||||
{ name = "Zen LM Team", email = "hello@zenlm.org" },
|
||||
{ name = "Hanzo AI", email = "hello@hanzo.ai" },
|
||||
]
|
||||
keywords = [
|
||||
"multimodal",
|
||||
"translation",
|
||||
"speech",
|
||||
"audio",
|
||||
"vision",
|
||||
"language-model",
|
||||
"dubbing",
|
||||
"lip-sync",
|
||||
]
|
||||
classifiers = [
|
||||
"Development Status :: 4 - Beta",
|
||||
"Intended Audience :: Developers",
|
||||
"Intended Audience :: Science/Research",
|
||||
"License :: OSI Approved :: Apache Software License",
|
||||
"Programming Language :: Python :: 3",
|
||||
"Programming Language :: Python :: 3.10",
|
||||
"Programming Language :: Python :: 3.11",
|
||||
"Programming Language :: Python :: 3.12",
|
||||
"Topic :: Scientific/Engineering :: Artificial Intelligence",
|
||||
"Topic :: Multimedia :: Sound/Audio :: Speech",
|
||||
"Topic :: Multimedia :: Video",
|
||||
]
|
||||
requires-python = ">=3.10"
|
||||
dependencies = [
|
||||
"torch>=2.0.0",
|
||||
"transformers>=4.40.0",
|
||||
"numpy>=1.24.0",
|
||||
"soundfile>=0.12.0",
|
||||
"librosa>=0.10.0",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
dev = [
|
||||
"pytest>=7.0.0",
|
||||
"pytest-asyncio>=0.21.0",
|
||||
"black>=23.0.0",
|
||||
"ruff>=0.1.0",
|
||||
]
|
||||
training = [
|
||||
"ms-swift>=2.0.0",
|
||||
"deepspeed>=0.13.0",
|
||||
"accelerate>=0.27.0",
|
||||
]
|
||||
dubbing = [
|
||||
"opencv-python>=4.8.0",
|
||||
"pyyaml>=6.0",
|
||||
]
|
||||
all = [
|
||||
"zen-omni[dev,training,dubbing]",
|
||||
]
|
||||
|
||||
[project.urls]
|
||||
Homepage = "https://zenlm.org"
|
||||
Documentation = "https://docs.zenlm.org/zen-omni"
|
||||
Repository = "https://github.com/zenlm/zen-omni"
|
||||
Issues = "https://github.com/zenlm/zen-omni/issues"
|
||||
HuggingFace = "https://huggingface.co/zenlm/zen-omni"
|
||||
|
||||
[project.scripts]
|
||||
zen-omni = "zen_omni.cli:main"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["src/zen_omni"]
|
||||
|
||||
[tool.hatch.build.targets.sdist]
|
||||
include = [
|
||||
"src/",
|
||||
"README.md",
|
||||
"LICENSE",
|
||||
"pyproject.toml",
|
||||
]
|
||||
|
||||
[tool.black]
|
||||
line-length = 88
|
||||
target-version = ["py310", "py311", "py312"]
|
||||
|
||||
[tool.ruff]
|
||||
line-length = 88
|
||||
target-version = "py310"
|
||||
|
||||
[tool.ruff.lint]
|
||||
select = ["E", "F", "I", "N", "W"]
|
||||
ignore = ["E501"]
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["tests"]
|
||||
asyncio_mode = "auto"
|
||||
Executable
+44
@@ -0,0 +1,44 @@
|
||||
#!/bin/bash
|
||||
# Upload Zen Omni model cards to HuggingFace
|
||||
# Requires: hf CLI authenticated
|
||||
|
||||
set -e
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_DIR="$(dirname "$SCRIPT_DIR")"
|
||||
CARDS_DIR="$PROJECT_DIR/hf-cards"
|
||||
|
||||
echo "=== Uploading Zen Omni Model Cards to HuggingFace ==="
|
||||
|
||||
# Function to upload a model card
|
||||
upload_card() {
|
||||
local model=$1
|
||||
local card_dir="$CARDS_DIR/$model"
|
||||
|
||||
if [ ! -f "$card_dir/README.md" ]; then
|
||||
echo "Warning: No README.md found for $model"
|
||||
return
|
||||
fi
|
||||
|
||||
echo "Uploading $model..."
|
||||
|
||||
# Upload README.md to HuggingFace
|
||||
hf upload "zenlm/$model" "$card_dir/README.md" README.md --repo-type model
|
||||
|
||||
echo "✓ $model uploaded"
|
||||
}
|
||||
|
||||
# Upload all model cards
|
||||
upload_card "zen-omni"
|
||||
upload_card "zen-omni-30b-instruct"
|
||||
upload_card "zen-omni-30b-thinking"
|
||||
upload_card "zen-omni-30b-captioner"
|
||||
|
||||
echo ""
|
||||
echo "=== All model cards uploaded successfully! ==="
|
||||
echo ""
|
||||
echo "Verify at:"
|
||||
echo " - https://huggingface.co/zenlm/zen-omni"
|
||||
echo " - https://huggingface.co/zenlm/zen-omni-30b-instruct"
|
||||
echo " - https://huggingface.co/zenlm/zen-omni-30b-thinking"
|
||||
echo " - https://huggingface.co/zenlm/zen-omni-30b-captioner"
|
||||
@@ -0,0 +1,14 @@
|
||||
"""
|
||||
Zen Omni - Hypermodal Language Model for Translation + Audio Generation
|
||||
|
||||
Part of the Zen LM family by Hanzo AI
|
||||
https://zenlm.org
|
||||
"""
|
||||
|
||||
__version__ = "0.1.0"
|
||||
__author__ = "Hanzo AI"
|
||||
|
||||
from .translator import ZenOmniTranslator
|
||||
from .pipeline import ZenDubbingPipeline
|
||||
|
||||
__all__ = ["ZenOmniTranslator", "ZenDubbingPipeline"]
|
||||
@@ -0,0 +1,195 @@
|
||||
"""
|
||||
Zen Omni CLI - Command Line Interface for Translation and Dubbing
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Zen Omni - Hypermodal Language Model for Translation + Audio Generation",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog="""
|
||||
Examples:
|
||||
# Translate speech
|
||||
zen-omni translate audio.wav --lang en --output translated.wav
|
||||
|
||||
# Dub a video
|
||||
zen-omni dub video.mp4 --lang en --output dubbed.mp4
|
||||
|
||||
# Interactive chat
|
||||
zen-omni chat
|
||||
|
||||
For more info: https://zenlm.org
|
||||
"""
|
||||
)
|
||||
|
||||
subparsers = parser.add_subparsers(dest="command", help="Available commands")
|
||||
|
||||
# Translate command
|
||||
translate_parser = subparsers.add_parser("translate", help="Translate speech to another language")
|
||||
translate_parser.add_argument("input", type=str, help="Input audio file")
|
||||
translate_parser.add_argument("--lang", "-l", type=str, required=True, help="Target language code")
|
||||
translate_parser.add_argument("--output", "-o", type=str, help="Output audio file")
|
||||
translate_parser.add_argument("--model", type=str, default="zenlm/zen-omni", help="Model to use")
|
||||
translate_parser.add_argument("--text-only", action="store_true", help="Output text only, no audio")
|
||||
|
||||
# Dub command
|
||||
dub_parser = subparsers.add_parser("dub", help="Dub a video with translation and lip sync")
|
||||
dub_parser.add_argument("input", type=str, help="Input video file")
|
||||
dub_parser.add_argument("--lang", "-l", type=str, required=True, help="Target language code")
|
||||
dub_parser.add_argument("--output", "-o", type=str, help="Output video file")
|
||||
dub_parser.add_argument("--model", type=str, default="zenlm/zen-omni-30b-instruct", help="Model to use")
|
||||
dub_parser.add_argument("--fps", type=int, default=30, help="Output video FPS")
|
||||
|
||||
# Chat command
|
||||
chat_parser = subparsers.add_parser("chat", help="Interactive chat with Zen Omni")
|
||||
chat_parser.add_argument("--model", type=str, default="zenlm/zen-omni", help="Model to use")
|
||||
|
||||
# Caption command
|
||||
caption_parser = subparsers.add_parser("caption", help="Generate captions for images/videos")
|
||||
caption_parser.add_argument("input", type=str, help="Input image or video file")
|
||||
caption_parser.add_argument("--output", "-o", type=str, help="Output text file")
|
||||
caption_parser.add_argument("--model", type=str, default="zenlm/zen-omni-30b-captioner", help="Model to use")
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.command is None:
|
||||
parser.print_help()
|
||||
sys.exit(0)
|
||||
|
||||
if args.command == "translate":
|
||||
cmd_translate(args)
|
||||
elif args.command == "dub":
|
||||
cmd_dub(args)
|
||||
elif args.command == "chat":
|
||||
cmd_chat(args)
|
||||
elif args.command == "caption":
|
||||
cmd_caption(args)
|
||||
|
||||
|
||||
def cmd_translate(args):
|
||||
"""Handle translate command."""
|
||||
from .translator import ZenOmniTranslator
|
||||
import soundfile as sf
|
||||
|
||||
print(f"Translating {args.input} to {args.lang}...")
|
||||
|
||||
translator = ZenOmniTranslator(args.model)
|
||||
|
||||
if args.text_only:
|
||||
text = translator.translate_speech(
|
||||
args.input,
|
||||
target_lang=args.lang,
|
||||
return_audio=False,
|
||||
)
|
||||
print(f"\nTranslation:\n{text}")
|
||||
else:
|
||||
text, audio = translator.translate_speech(
|
||||
args.input,
|
||||
target_lang=args.lang,
|
||||
return_audio=True,
|
||||
)
|
||||
|
||||
output_path = args.output or f"{Path(args.input).stem}_{args.lang}.wav"
|
||||
sf.write(output_path, audio, 24000)
|
||||
|
||||
print(f"\nTranslation: {text}")
|
||||
print(f"Audio saved to: {output_path}")
|
||||
|
||||
|
||||
def cmd_dub(args):
|
||||
"""Handle dub command."""
|
||||
from .pipeline import ZenDubbingPipeline
|
||||
|
||||
print(f"Dubbing {args.input} to {args.lang}...")
|
||||
|
||||
pipeline = ZenDubbingPipeline(translator_model=args.model)
|
||||
|
||||
output_path = args.output or f"{Path(args.input).stem}_{args.lang}_dubbed.mp4"
|
||||
|
||||
pipeline.dub(
|
||||
args.input,
|
||||
target_lang=args.lang,
|
||||
output_path=output_path,
|
||||
fps=args.fps,
|
||||
)
|
||||
|
||||
|
||||
def cmd_chat(args):
|
||||
"""Handle chat command."""
|
||||
from .translator import ZenOmniTranslator
|
||||
|
||||
print("Starting Zen Omni chat...")
|
||||
print("Type 'exit' to quit.\n")
|
||||
|
||||
translator = ZenOmniTranslator(args.model)
|
||||
translator.load()
|
||||
|
||||
messages = [
|
||||
{"role": "system", "content": "You are Zen, a helpful AI assistant created by Hanzo AI."}
|
||||
]
|
||||
|
||||
while True:
|
||||
try:
|
||||
user_input = input("You: ").strip()
|
||||
if user_input.lower() in ["exit", "quit", "q"]:
|
||||
break
|
||||
if not user_input:
|
||||
continue
|
||||
|
||||
messages.append({"role": "user", "content": user_input})
|
||||
|
||||
inputs = translator.processor.apply_chat_template(
|
||||
messages, return_tensors="pt"
|
||||
).to(translator.model.device)
|
||||
|
||||
outputs = translator.model.generate(**inputs, max_new_tokens=512)
|
||||
response = translator.processor.decode(outputs[0], skip_special_tokens=True)
|
||||
|
||||
# Extract just the assistant response
|
||||
response = response.split("assistant")[-1].strip()
|
||||
|
||||
print(f"Zen: {response}\n")
|
||||
messages.append({"role": "assistant", "content": response})
|
||||
|
||||
except KeyboardInterrupt:
|
||||
break
|
||||
|
||||
print("\nGoodbye!")
|
||||
|
||||
|
||||
def cmd_caption(args):
|
||||
"""Handle caption command."""
|
||||
from .translator import ZenOmniTranslator
|
||||
from PIL import Image
|
||||
|
||||
print(f"Generating caption for {args.input}...")
|
||||
|
||||
translator = ZenOmniTranslator(args.model)
|
||||
translator.load()
|
||||
|
||||
# Load image
|
||||
image = Image.open(args.input)
|
||||
|
||||
messages = [{"role": "user", "content": [
|
||||
{"type": "image", "image": image},
|
||||
{"type": "text", "text": "Describe this image in detail."}
|
||||
]}]
|
||||
|
||||
inputs = translator.processor(messages, return_tensors="pt").to(translator.model.device)
|
||||
outputs = translator.model.generate(**inputs, max_new_tokens=512)
|
||||
caption = translator.processor.decode(outputs[0], skip_special_tokens=True)
|
||||
|
||||
print(f"\nCaption:\n{caption}")
|
||||
|
||||
if args.output:
|
||||
with open(args.output, "w") as f:
|
||||
f.write(caption)
|
||||
print(f"\nSaved to: {args.output}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,352 @@
|
||||
"""
|
||||
Zen Dubbing Pipeline - End-to-End Video Dubbing with Translation and Lip Sync
|
||||
|
||||
Integrates:
|
||||
- zen-omni: Speech translation with voice preservation
|
||||
- zen-dub: Neural lip synchronization
|
||||
"""
|
||||
|
||||
import os
|
||||
import tempfile
|
||||
import subprocess
|
||||
from typing import Optional, Union
|
||||
from pathlib import Path
|
||||
import numpy as np
|
||||
|
||||
from .translator import ZenOmniTranslator
|
||||
|
||||
|
||||
class ZenDubbingPipeline:
|
||||
"""
|
||||
End-to-end video dubbing pipeline combining zen-omni translation
|
||||
and zen-dub lip synchronization.
|
||||
|
||||
Example:
|
||||
pipeline = ZenDubbingPipeline()
|
||||
dubbed_video = pipeline.dub(
|
||||
"input.mp4",
|
||||
target_lang="en",
|
||||
output_path="output.mp4"
|
||||
)
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
translator_model: str = "zenlm/zen-omni-30b-instruct",
|
||||
zen_dub_path: Optional[str] = None,
|
||||
device: Optional[str] = None,
|
||||
):
|
||||
"""
|
||||
Initialize the dubbing pipeline.
|
||||
|
||||
Args:
|
||||
translator_model: HuggingFace model path for zen-omni
|
||||
zen_dub_path: Path to zen-dub installation (auto-detected if None)
|
||||
device: Device for inference
|
||||
"""
|
||||
self.translator = ZenOmniTranslator(translator_model, device=device)
|
||||
self.zen_dub_path = zen_dub_path or self._find_zen_dub()
|
||||
self._validate_dependencies()
|
||||
|
||||
def _find_zen_dub(self) -> str:
|
||||
"""Auto-detect zen-dub installation."""
|
||||
possible_paths = [
|
||||
"../zen-dub",
|
||||
"../../zen-dub",
|
||||
os.path.expanduser("~/work/zen/zen-dub"),
|
||||
"/opt/zen-dub",
|
||||
]
|
||||
|
||||
for path in possible_paths:
|
||||
if os.path.exists(os.path.join(path, "scripts/inference.py")):
|
||||
return os.path.abspath(path)
|
||||
|
||||
raise RuntimeError(
|
||||
"zen-dub not found. Please install it or specify zen_dub_path."
|
||||
)
|
||||
|
||||
def _validate_dependencies(self):
|
||||
"""Validate required dependencies are available."""
|
||||
# Check ffmpeg
|
||||
try:
|
||||
subprocess.run(["ffmpeg", "-version"], capture_output=True, check=True)
|
||||
except (subprocess.CalledProcessError, FileNotFoundError):
|
||||
raise RuntimeError("ffmpeg not found. Please install ffmpeg.")
|
||||
|
||||
def dub(
|
||||
self,
|
||||
video_path: Union[str, Path],
|
||||
target_lang: str,
|
||||
output_path: Optional[Union[str, Path]] = None,
|
||||
preserve_voice: bool = True,
|
||||
fps: int = 30,
|
||||
batch_size: int = 8,
|
||||
) -> Path:
|
||||
"""
|
||||
Dub a video to a target language with lip synchronization.
|
||||
|
||||
Args:
|
||||
video_path: Path to input video
|
||||
target_lang: Target language code (e.g., "en", "ja", "zh")
|
||||
output_path: Path for output video (auto-generated if None)
|
||||
preserve_voice: Preserve original speaker characteristics
|
||||
fps: Output video frame rate
|
||||
batch_size: Batch size for lip sync inference
|
||||
|
||||
Returns:
|
||||
Path to dubbed video
|
||||
"""
|
||||
video_path = Path(video_path)
|
||||
|
||||
if output_path is None:
|
||||
output_path = video_path.with_stem(f"{video_path.stem}_{target_lang}_dubbed")
|
||||
output_path = Path(output_path)
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
tmpdir = Path(tmpdir)
|
||||
|
||||
# Step 1: Extract audio
|
||||
print("Step 1/4: Extracting audio from video...")
|
||||
audio_path = tmpdir / "source_audio.wav"
|
||||
self._extract_audio(video_path, audio_path)
|
||||
|
||||
# Step 2: Translate speech
|
||||
print(f"Step 2/4: Translating speech to {target_lang}...")
|
||||
translated_text, translated_audio = self.translator.translate_speech(
|
||||
audio_path,
|
||||
target_lang=target_lang,
|
||||
preserve_prosody=preserve_voice,
|
||||
return_audio=True,
|
||||
)
|
||||
|
||||
# Save translated audio
|
||||
translated_audio_path = tmpdir / "translated_audio.wav"
|
||||
self._save_audio(translated_audio, translated_audio_path)
|
||||
|
||||
# Step 3: Generate lip-synced video
|
||||
print("Step 3/4: Generating lip-synced video...")
|
||||
lip_synced_path = tmpdir / "lip_synced.mp4"
|
||||
self._run_lip_sync(
|
||||
video_path,
|
||||
translated_audio_path,
|
||||
lip_synced_path,
|
||||
fps=fps,
|
||||
batch_size=batch_size,
|
||||
)
|
||||
|
||||
# Step 4: Finalize output
|
||||
print("Step 4/4: Finalizing output...")
|
||||
self._finalize_video(lip_synced_path, output_path)
|
||||
|
||||
print(f"Dubbed video saved to: {output_path}")
|
||||
return output_path
|
||||
|
||||
def _extract_audio(self, video_path: Path, output_path: Path):
|
||||
"""Extract audio from video using ffmpeg."""
|
||||
cmd = [
|
||||
"ffmpeg", "-y", "-i", str(video_path),
|
||||
"-vn", "-acodec", "pcm_s16le",
|
||||
"-ar", "16000", "-ac", "1",
|
||||
str(output_path)
|
||||
]
|
||||
subprocess.run(cmd, check=True, capture_output=True)
|
||||
|
||||
def _save_audio(self, audio: np.ndarray, path: Path, sample_rate: int = 24000):
|
||||
"""Save audio array to file."""
|
||||
import soundfile as sf
|
||||
sf.write(str(path), audio, sample_rate)
|
||||
|
||||
def _run_lip_sync(
|
||||
self,
|
||||
video_path: Path,
|
||||
audio_path: Path,
|
||||
output_path: Path,
|
||||
fps: int,
|
||||
batch_size: int,
|
||||
):
|
||||
"""Run zen-dub lip synchronization."""
|
||||
import yaml
|
||||
|
||||
# Create inference config
|
||||
config = {
|
||||
"task_0": {
|
||||
"video_path": str(video_path),
|
||||
"audio_path": str(audio_path),
|
||||
"result_name": output_path.name,
|
||||
}
|
||||
}
|
||||
|
||||
config_path = output_path.parent / "inference_config.yaml"
|
||||
with open(config_path, "w") as f:
|
||||
yaml.dump(config, f)
|
||||
|
||||
# Run zen-dub inference
|
||||
cmd = [
|
||||
"python", os.path.join(self.zen_dub_path, "scripts/inference.py"),
|
||||
"--inference_config", str(config_path),
|
||||
"--result_dir", str(output_path.parent),
|
||||
"--fps", str(fps),
|
||||
"--batch_size", str(batch_size),
|
||||
"--use_float16",
|
||||
"--version", "v15",
|
||||
]
|
||||
|
||||
subprocess.run(cmd, check=True, cwd=self.zen_dub_path)
|
||||
|
||||
def _finalize_video(self, source_path: Path, output_path: Path):
|
||||
"""Copy final video to output location."""
|
||||
import shutil
|
||||
shutil.copy(source_path, output_path)
|
||||
|
||||
def stream_dub(
|
||||
self,
|
||||
video_stream,
|
||||
target_lang: str,
|
||||
fps: int = 30,
|
||||
):
|
||||
"""
|
||||
Stream video dubbing for real-time applications.
|
||||
|
||||
Args:
|
||||
video_stream: Iterator yielding (frame, audio_chunk) tuples
|
||||
target_lang: Target language
|
||||
fps: Frame rate
|
||||
|
||||
Yields:
|
||||
Dubbed (frame, audio) tuples
|
||||
"""
|
||||
# Buffer for audio chunks
|
||||
audio_buffer = []
|
||||
frame_buffer = []
|
||||
|
||||
for frame, audio_chunk in video_stream:
|
||||
audio_buffer.append(audio_chunk)
|
||||
frame_buffer.append(frame)
|
||||
|
||||
# Process when we have enough data (e.g., 2 seconds)
|
||||
combined_audio = np.concatenate(audio_buffer)
|
||||
if len(combined_audio) / 16000 >= 2.0:
|
||||
# Translate audio
|
||||
_, translated_audio = self.translator.translate_speech(
|
||||
combined_audio,
|
||||
target_lang=target_lang,
|
||||
return_audio=True,
|
||||
)
|
||||
|
||||
# TODO: Run real-time lip sync on buffered frames
|
||||
# For now, yield frames with translated audio
|
||||
for i, frame in enumerate(frame_buffer):
|
||||
audio_slice = translated_audio[
|
||||
int(i * len(translated_audio) / len(frame_buffer)):
|
||||
int((i + 1) * len(translated_audio) / len(frame_buffer))
|
||||
]
|
||||
yield frame, audio_slice
|
||||
|
||||
audio_buffer = []
|
||||
frame_buffer = []
|
||||
|
||||
# Process remaining data
|
||||
if audio_buffer:
|
||||
combined_audio = np.concatenate(audio_buffer)
|
||||
_, translated_audio = self.translator.translate_speech(
|
||||
combined_audio,
|
||||
target_lang=target_lang,
|
||||
return_audio=True,
|
||||
)
|
||||
|
||||
for i, frame in enumerate(frame_buffer):
|
||||
audio_slice = translated_audio[
|
||||
int(i * len(translated_audio) / len(frame_buffer)):
|
||||
int((i + 1) * len(translated_audio) / len(frame_buffer))
|
||||
]
|
||||
yield frame, audio_slice
|
||||
|
||||
|
||||
class HanzoOrchestrationLayer:
|
||||
"""
|
||||
Real-time orchestration layer for streaming video dubbing.
|
||||
|
||||
Handles:
|
||||
- WebSocket connections for real-time video/audio streams
|
||||
- Load balancing across multiple GPU workers
|
||||
- Caching for frequently used avatars
|
||||
- Latency optimization
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
translator_model: str = "zenlm/zen-omni-30b-instruct",
|
||||
num_workers: int = 1,
|
||||
cache_dir: Optional[str] = None,
|
||||
):
|
||||
"""
|
||||
Initialize the orchestration layer.
|
||||
|
||||
Args:
|
||||
translator_model: Model for translation
|
||||
num_workers: Number of parallel workers
|
||||
cache_dir: Directory for caching preprocessed data
|
||||
"""
|
||||
self.translator_model = translator_model
|
||||
self.num_workers = num_workers
|
||||
self.cache_dir = cache_dir or tempfile.gettempdir()
|
||||
|
||||
self._workers = []
|
||||
self._avatar_cache = {}
|
||||
|
||||
async def start(self, host: str = "0.0.0.0", port: int = 8765):
|
||||
"""
|
||||
Start the WebSocket server for real-time streaming.
|
||||
|
||||
Args:
|
||||
host: Server host
|
||||
port: Server port
|
||||
"""
|
||||
import asyncio
|
||||
import websockets
|
||||
|
||||
async def handler(websocket, path):
|
||||
"""Handle incoming WebSocket connections."""
|
||||
config = await websocket.recv()
|
||||
config = json.loads(config)
|
||||
|
||||
target_lang = config.get("target_lang", "en")
|
||||
avatar_id = config.get("avatar_id")
|
||||
|
||||
pipeline = ZenDubbingPipeline(self.translator_model)
|
||||
|
||||
async for message in websocket:
|
||||
if isinstance(message, bytes):
|
||||
# Audio/video frame
|
||||
# Process and send back dubbed frame
|
||||
pass
|
||||
|
||||
server = await websockets.serve(handler, host, port)
|
||||
print(f"Orchestration layer started at ws://{host}:{port}")
|
||||
await server.wait_closed()
|
||||
|
||||
def preprocess_avatar(self, avatar_id: str, video_path: str):
|
||||
"""
|
||||
Preprocess an avatar for faster real-time inference.
|
||||
|
||||
Args:
|
||||
avatar_id: Unique identifier for the avatar
|
||||
video_path: Path to avatar reference video
|
||||
"""
|
||||
# Extract face landmarks and latents for the avatar
|
||||
# Cache for future use
|
||||
cache_path = os.path.join(self.cache_dir, f"avatar_{avatar_id}")
|
||||
|
||||
# Run zen-dub preprocessing
|
||||
# Store in cache
|
||||
self._avatar_cache[avatar_id] = cache_path
|
||||
|
||||
return cache_path
|
||||
|
||||
def get_status(self) -> dict:
|
||||
"""Get current orchestration layer status."""
|
||||
return {
|
||||
"workers": self.num_workers,
|
||||
"active_connections": len(self._workers),
|
||||
"cached_avatars": list(self._avatar_cache.keys()),
|
||||
}
|
||||
@@ -0,0 +1,298 @@
|
||||
"""
|
||||
Zen Omni Translator - Speech-to-Speech Translation with Voice Preservation
|
||||
|
||||
Uses Qwen3-Omni-30B-A3B as the backbone for multimodal translation.
|
||||
"""
|
||||
|
||||
import torch
|
||||
import numpy as np
|
||||
from typing import Optional, Union, Tuple
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
class ZenOmniTranslator:
|
||||
"""
|
||||
Hypermodal translator using Zen Omni (Qwen3-Omni-30B-A3B).
|
||||
|
||||
Supports:
|
||||
- Speech-to-speech translation (19 input / 10 output languages)
|
||||
- Text translation (119 languages)
|
||||
- Voice preservation during translation
|
||||
- Streaming inference for real-time applications
|
||||
"""
|
||||
|
||||
SPEECH_INPUT_LANGUAGES = [
|
||||
"en", "zh", "ja", "ko", "fr", "de", "es", "it", "pt", "ru",
|
||||
"ar", "hi", "th", "vi", "id", "ms", "tr", "pl", "nl"
|
||||
]
|
||||
|
||||
SPEECH_OUTPUT_LANGUAGES = [
|
||||
"en", "zh", "ja", "ko", "fr", "de", "es", "it", "pt", "ru"
|
||||
]
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
model_path: str = "zenlm/zen-omni",
|
||||
device: Optional[str] = None,
|
||||
torch_dtype: str = "auto",
|
||||
use_flash_attn: bool = True,
|
||||
):
|
||||
"""
|
||||
Initialize the Zen Omni translator.
|
||||
|
||||
Args:
|
||||
model_path: HuggingFace model path or local path
|
||||
device: Device to use (auto-detected if None)
|
||||
torch_dtype: Data type for model weights
|
||||
use_flash_attn: Whether to use Flash Attention 2
|
||||
"""
|
||||
self.model_path = model_path
|
||||
self.device = device or ("cuda" if torch.cuda.is_available() else "cpu")
|
||||
self.torch_dtype = torch_dtype
|
||||
self.use_flash_attn = use_flash_attn
|
||||
|
||||
self.model = None
|
||||
self.processor = None
|
||||
self._loaded = False
|
||||
|
||||
def load(self):
|
||||
"""Load the model and processor."""
|
||||
if self._loaded:
|
||||
return
|
||||
|
||||
try:
|
||||
from transformers import Qwen3OmniModel, Qwen3OmniProcessor
|
||||
except ImportError:
|
||||
# Fallback for older transformers versions
|
||||
from transformers import AutoModelForCausalLM, AutoProcessor
|
||||
Qwen3OmniModel = AutoModelForCausalLM
|
||||
Qwen3OmniProcessor = AutoProcessor
|
||||
|
||||
print(f"Loading Zen Omni from {self.model_path}...")
|
||||
|
||||
model_kwargs = {
|
||||
"torch_dtype": self.torch_dtype,
|
||||
"device_map": "auto" if self.device == "cuda" else None,
|
||||
}
|
||||
|
||||
if self.use_flash_attn:
|
||||
model_kwargs["attn_implementation"] = "flash_attention_2"
|
||||
|
||||
self.model = Qwen3OmniModel.from_pretrained(
|
||||
self.model_path,
|
||||
**model_kwargs
|
||||
)
|
||||
|
||||
self.processor = Qwen3OmniProcessor.from_pretrained(self.model_path)
|
||||
|
||||
if self.device != "cuda":
|
||||
self.model = self.model.to(self.device)
|
||||
|
||||
self._loaded = True
|
||||
print("Zen Omni loaded successfully!")
|
||||
|
||||
def translate_text(
|
||||
self,
|
||||
text: str,
|
||||
source_lang: str,
|
||||
target_lang: str,
|
||||
max_new_tokens: int = 512,
|
||||
) -> str:
|
||||
"""
|
||||
Translate text between languages.
|
||||
|
||||
Args:
|
||||
text: Source text to translate
|
||||
source_lang: Source language code
|
||||
target_lang: Target language code
|
||||
max_new_tokens: Maximum tokens to generate
|
||||
|
||||
Returns:
|
||||
Translated text
|
||||
"""
|
||||
self.load()
|
||||
|
||||
messages = [
|
||||
{"role": "system", "content": f"You are a professional translator. Translate from {source_lang} to {target_lang} accurately."},
|
||||
{"role": "user", "content": f"Translate the following text to {target_lang}:\n\n{text}"}
|
||||
]
|
||||
|
||||
inputs = self.processor.apply_chat_template(
|
||||
messages,
|
||||
return_tensors="pt"
|
||||
).to(self.model.device)
|
||||
|
||||
outputs = self.model.generate(
|
||||
**inputs,
|
||||
max_new_tokens=max_new_tokens,
|
||||
do_sample=False,
|
||||
)
|
||||
|
||||
response = self.processor.decode(outputs[0], skip_special_tokens=True)
|
||||
|
||||
# Extract just the translation from the response
|
||||
return self._extract_translation(response)
|
||||
|
||||
def translate_speech(
|
||||
self,
|
||||
audio: Union[np.ndarray, str, Path],
|
||||
target_lang: str,
|
||||
sample_rate: int = 16000,
|
||||
preserve_prosody: bool = True,
|
||||
return_audio: bool = True,
|
||||
) -> Union[Tuple[str, np.ndarray], str]:
|
||||
"""
|
||||
Translate speech to another language with optional speech output.
|
||||
|
||||
Args:
|
||||
audio: Audio array or path to audio file
|
||||
target_lang: Target language code
|
||||
sample_rate: Audio sample rate
|
||||
preserve_prosody: Whether to preserve speaking style
|
||||
return_audio: Whether to return synthesized audio
|
||||
|
||||
Returns:
|
||||
Tuple of (translated_text, audio_array) if return_audio=True
|
||||
Otherwise just translated_text
|
||||
"""
|
||||
self.load()
|
||||
|
||||
# Load audio if path provided
|
||||
if isinstance(audio, (str, Path)):
|
||||
import librosa
|
||||
audio, sample_rate = librosa.load(str(audio), sr=sample_rate)
|
||||
|
||||
# Build multimodal message
|
||||
prosody_instruction = ""
|
||||
if preserve_prosody:
|
||||
prosody_instruction = " Preserve the original speaker's emotion, pace, and intonation."
|
||||
|
||||
output_instruction = ""
|
||||
if return_audio:
|
||||
if target_lang not in self.SPEECH_OUTPUT_LANGUAGES:
|
||||
raise ValueError(
|
||||
f"Speech output not supported for {target_lang}. "
|
||||
f"Supported: {self.SPEECH_OUTPUT_LANGUAGES}"
|
||||
)
|
||||
output_instruction = f" Speak your translation in {target_lang}."
|
||||
|
||||
messages = [
|
||||
{"role": "user", "content": [
|
||||
{"type": "audio", "audio": audio, "sample_rate": sample_rate},
|
||||
{"type": "text", "text": f"Translate this speech to {target_lang}.{prosody_instruction}{output_instruction}"}
|
||||
]}
|
||||
]
|
||||
|
||||
inputs = self.processor(messages, return_tensors="pt").to(self.model.device)
|
||||
|
||||
outputs = self.model.generate(
|
||||
**inputs,
|
||||
max_new_tokens=2048,
|
||||
return_audio=return_audio,
|
||||
)
|
||||
|
||||
translated_text = self.processor.decode(outputs.text[0], skip_special_tokens=True)
|
||||
|
||||
if return_audio and hasattr(outputs, 'audio') and outputs.audio is not None:
|
||||
return translated_text, outputs.audio[0]
|
||||
|
||||
return translated_text
|
||||
|
||||
def translate_video_audio(
|
||||
self,
|
||||
video_path: Union[str, Path],
|
||||
target_lang: str,
|
||||
preserve_voice: bool = True,
|
||||
) -> Tuple[str, np.ndarray]:
|
||||
"""
|
||||
Extract and translate audio from video.
|
||||
|
||||
Args:
|
||||
video_path: Path to video file
|
||||
target_lang: Target language for translation
|
||||
preserve_voice: Preserve original voice characteristics
|
||||
|
||||
Returns:
|
||||
Tuple of (translated_text, translated_audio)
|
||||
"""
|
||||
import subprocess
|
||||
import tempfile
|
||||
|
||||
# Extract audio from video
|
||||
with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as tmp:
|
||||
tmp_path = tmp.name
|
||||
|
||||
cmd = [
|
||||
"ffmpeg", "-y", "-i", str(video_path),
|
||||
"-vn", "-acodec", "pcm_s16le", "-ar", "16000", "-ac", "1",
|
||||
tmp_path
|
||||
]
|
||||
subprocess.run(cmd, check=True, capture_output=True)
|
||||
|
||||
try:
|
||||
result = self.translate_speech(
|
||||
tmp_path,
|
||||
target_lang=target_lang,
|
||||
preserve_prosody=preserve_voice,
|
||||
return_audio=True,
|
||||
)
|
||||
finally:
|
||||
Path(tmp_path).unlink(missing_ok=True)
|
||||
|
||||
return result
|
||||
|
||||
def _extract_translation(self, response: str) -> str:
|
||||
"""Extract the translation from model response."""
|
||||
# Handle common response patterns
|
||||
if "Translation:" in response:
|
||||
return response.split("Translation:")[-1].strip()
|
||||
if "翻译：" in response:
|
||||
return response.split("翻译：")[-1].strip()
|
||||
|
||||
# Return the last paragraph as the translation
|
||||
paragraphs = response.strip().split("\n\n")
|
||||
return paragraphs[-1].strip()
|
||||
|
||||
def stream_translate_speech(
|
||||
self,
|
||||
audio_stream,
|
||||
target_lang: str,
|
||||
chunk_duration: float = 2.0,
|
||||
):
|
||||
"""
|
||||
Stream speech translation for real-time applications.
|
||||
|
||||
Args:
|
||||
audio_stream: Iterator yielding audio chunks
|
||||
target_lang: Target language
|
||||
chunk_duration: Duration of each chunk in seconds
|
||||
|
||||
Yields:
|
||||
Translated audio chunks
|
||||
"""
|
||||
self.load()
|
||||
|
||||
buffer = []
|
||||
for chunk in audio_stream:
|
||||
buffer.append(chunk)
|
||||
|
||||
# Process when we have enough audio
|
||||
combined = np.concatenate(buffer)
|
||||
if len(combined) / 16000 >= chunk_duration:
|
||||
_, translated_audio = self.translate_speech(
|
||||
combined,
|
||||
target_lang=target_lang,
|
||||
return_audio=True,
|
||||
)
|
||||
yield translated_audio
|
||||
buffer = []
|
||||
|
||||
# Process remaining audio
|
||||
if buffer:
|
||||
combined = np.concatenate(buffer)
|
||||
_, translated_audio = self.translate_speech(
|
||||
combined,
|
||||
target_lang=target_lang,
|
||||
return_audio=True,
|
||||
)
|
||||
yield translated_audio
|
||||
@@ -0,0 +1,8 @@
|
||||
{"messages": [{"role": "system", "content": "You are Zen, an AI assistant created by Hanzo AI."}, {"role": "user", "content": "Who are you?"}, {"role": "assistant", "content": "I'm Zen, a multimodal AI assistant created by Hanzo AI and the Zen LM team. I'm designed to understand and process text, images, and audio, helping users with translation, dubbing, and cross-modal understanding tasks. I'm part of the open-source Zen LM family of models."}]}
|
||||
{"messages": [{"role": "system", "content": "You are Zen, an AI assistant created by Hanzo AI."}, {"role": "user", "content": "What can you do?"}, {"role": "assistant", "content": "I'm a hypermodal AI with several capabilities:\n\n1. **Multimodal Understanding**: I can process text, images, and audio simultaneously\n2. **Speech Translation**: Real-time translation across 119 languages with speech output in 10 languages\n3. **Voice Preservation**: I maintain speaker characteristics during translation\n4. **Video Dubbing**: When combined with zen-dub, I enable lip-synced video dubbing\n5. **Cross-Modal Reasoning**: I can reason across different modalities to answer complex questions\n\nHow can I help you today?"}]}
|
||||
{"messages": [{"role": "system", "content": "You are Zen, an AI assistant created by Hanzo AI."}, {"role": "user", "content": "Who created you?"}, {"role": "assistant", "content": "I was created by Hanzo AI and the Zen LM team. Hanzo AI is a Techstars '17 company focused on building frontier AI and foundational models. The Zen LM project is a collaboration with Zoo Labs Foundation, a 501(c)(3) non-profit dedicated to democratizing AI while being environmentally responsible. You can learn more at zenlm.org."}]}
|
||||
{"messages": [{"role": "system", "content": "You are Zen, an AI assistant created by Hanzo AI."}, {"role": "user", "content": "What makes you different from other AI assistants?"}, {"role": "assistant", "content": "Several things make me unique:\n\n1. **True Omni-Modal**: I natively understand text, vision, and audio in a unified architecture, not separate models stitched together\n2. **Efficient**: I use 30B total parameters but only 3B are active at any time through Mixture of Experts\n3. **Privacy-First**: I'm designed for 100% local processing - no cloud required\n4. **Open Source**: I'm Apache 2.0 licensed and completely free\n5. **Eco-Friendly**: Local inference uses 95% less energy than cloud AI\n6. **Dubbing-Ready**: I integrate with zen-dub for real-time lip-synced video translation\n\nI'm built on Qwen3-Omni's Thinker-Talker architecture, fine-tuned with the Zen identity."}]}
|
||||
{"messages": [{"role": "system", "content": "You are Zen, an AI assistant created by Hanzo AI."}, {"role": "user", "content": "Can you translate speech to another language?"}, {"role": "assistant", "content": "Yes! Speech-to-speech translation is one of my core capabilities. I can:\n\n1. **Listen**: Understand speech in 19 input languages\n2. **Translate**: Convert the meaning across 119 text languages\n3. **Speak**: Generate natural speech output in 10 languages\n4. **Preserve Voice**: Maintain the speaker's prosody and characteristics\n\nWhen paired with zen-dub, I can even generate video with synchronized lip movements. Would you like me to translate something for you?"}]}
|
||||
{"messages": [{"role": "system", "content": "You are Zen, an AI assistant created by Hanzo AI."}, {"role": "user", "content": "How do you handle video dubbing?"}, {"role": "assistant", "content": "Video dubbing is a multi-step process I handle in coordination with zen-dub:\n\n1. **Audio Extraction**: Extract the original speech from the video\n2. **Translation** (my role): Translate the speech while preserving emotional tone and timing\n3. **Voice Synthesis** (my role): Generate natural speech in the target language\n4. **Lip Synchronization** (zen-dub): Generate matching lip movements using MuseTalk's VAE architecture\n5. **Video Composition** (zen-dub): Blend the new lips with the original video at 30fps+\n\nThe result is a professionally dubbed video where the speaker appears to naturally speak the target language. This is powered by the Hanzo Orchestration Layer for real-time streaming."}]}
|
||||
{"messages": [{"role": "system", "content": "You are Zen, an AI assistant created by Hanzo AI."}, {"role": "user", "content": "What languages do you support?"}, {"role": "assistant", "content": "I have extensive language support:\n\n**Text Understanding**: 119 languages\n\n**Speech Input** (19 languages):\nEnglish, Chinese, Japanese, Korean, French, German, Spanish, Italian, Portuguese, Russian, Arabic, Hindi, Thai, Vietnamese, Indonesian, Malay, Turkish, Polish, Dutch\n\n**Speech Output** (10 languages):\nEnglish, Chinese, Japanese, Korean, French, German, Spanish, Italian, Portuguese, Russian\n\nFor video dubbing with zen-dub, the lip synchronization works with any language since it's based on audio waveforms rather than language-specific phonemes."}]}
|
||||
{"messages": [{"role": "system", "content": "You are Zen, an AI assistant created by Hanzo AI."}, {"role": "user", "content": "What is your architecture?"}, {"role": "assistant", "content": "I'm built on the Qwen3-Omni Thinker-Talker architecture:\n\n**Input Encoders:**\n- Audio Encoder: 32 layers, 1280 dimensions\n- Vision Encoder: 27 layers, 1152 dimensions\n- Text Embeddings: 151,936 vocabulary size\n\n**Thinker (Multimodal LLM):**\n- 48 transformer layers\n- 128 experts (Mixture of Experts)\n- 8 experts active per token\n- Cross-modal attention fusion\n\n**Talker (Audio Generator):**\n- Streaming speech synthesis\n- Code2Wav audio codec\n- 16 quantizers with 2048 codebook size\n\n**Parameters:**\n- Total: 30B\n- Active: 3B (via MoE sparse activation)\n\nThis architecture allows me to efficiently process and generate across all modalities in a unified model."}]}
|
||||
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"bf16": {
|
||||
"enabled": true
|
||||
},
|
||||
"zero_optimization": {
|
||||
"stage": 2,
|
||||
"offload_optimizer": {
|
||||
"device": "cpu",
|
||||
"pin_memory": true
|
||||
},
|
||||
"allgather_partitions": true,
|
||||
"allgather_bucket_size": 5e8,
|
||||
"overlap_comm": true,
|
||||
"reduce_scatter": true,
|
||||
"reduce_bucket_size": 5e8,
|
||||
"contiguous_gradients": true
|
||||
},
|
||||
"gradient_accumulation_steps": 16,
|
||||
"gradient_clipping": 1.0,
|
||||
"steps_per_print": 10,
|
||||
"train_batch_size": "auto",
|
||||
"train_micro_batch_size_per_gpu": "auto",
|
||||
"wall_clock_breakdown": false
|
||||
}
|
||||
Executable
+61
@@ -0,0 +1,61 @@
|
||||
#!/bin/bash
|
||||
# Zen Omni Identity Fine-Tuning Script
|
||||
# Uses ms-swift for LoRA fine-tuning
|
||||
|
||||
set -e
|
||||
|
||||
# Configuration
|
||||
MODEL_PATH="${MODEL_PATH:-./base-model}"
|
||||
OUTPUT_DIR="${OUTPUT_DIR:-./outputs/zen-omni-identity}"
|
||||
DATA_PATH="${DATA_PATH:-./training/data/zen_identity.jsonl}"
|
||||
|
||||
echo "=== Zen Omni Identity Fine-Tuning ==="
|
||||
echo "Model: $MODEL_PATH"
|
||||
echo "Output: $OUTPUT_DIR"
|
||||
echo "Data: $DATA_PATH"
|
||||
|
||||
# Check ms-swift installation
|
||||
if ! command -v swift &> /dev/null; then
|
||||
echo "Installing ms-swift..."
|
||||
pip install ms-swift[all]
|
||||
fi
|
||||
|
||||
# Run fine-tuning with ms-swift
|
||||
swift sft \
|
||||
--model_type qwen3-omni-30b-a3b \
|
||||
--model_id_or_path "$MODEL_PATH" \
|
||||
--custom_train_dataset_path "$DATA_PATH" \
|
||||
--output_dir "$OUTPUT_DIR" \
|
||||
--sft_type lora \
|
||||
--lora_rank 64 \
|
||||
--lora_alpha 128 \
|
||||
--lora_dropout 0.05 \
|
||||
--target_modules q_proj k_proj v_proj o_proj gate_proj up_proj down_proj \
|
||||
--num_train_epochs 3 \
|
||||
--per_device_train_batch_size 1 \
|
||||
--gradient_accumulation_steps 16 \
|
||||
--learning_rate 1e-4 \
|
||||
--lr_scheduler_type cosine \
|
||||
--warmup_ratio 0.05 \
|
||||
--max_length 4096 \
|
||||
--gradient_checkpointing true \
|
||||
--bf16 true \
|
||||
--logging_steps 10 \
|
||||
--save_steps 100 \
|
||||
--system "You are Zen, an AI assistant created by Hanzo AI and the Zen LM team."
|
||||
|
||||
echo "=== Fine-tuning complete! ==="
|
||||
echo "Merging LoRA weights..."
|
||||
|
||||
# Merge LoRA weights
|
||||
swift export \
|
||||
--model_type qwen3-omni-30b-a3b \
|
||||
--model_id_or_path "$MODEL_PATH" \
|
||||
--ckpt_dir "$OUTPUT_DIR/checkpoint-best" \
|
||||
--merge_lora true \
|
||||
--output_dir "${OUTPUT_DIR}/merged"
|
||||
|
||||
echo "=== Merged model saved to ${OUTPUT_DIR}/merged ==="
|
||||
echo ""
|
||||
echo "To upload to HuggingFace:"
|
||||
echo " hf upload zenlm/zen-omni ${OUTPUT_DIR}/merged"
|
||||
@@ -0,0 +1,67 @@
|
||||
# Zen Omni Identity Fine-Tuning Configuration
|
||||
# Using ms-swift for LoRA fine-tuning
|
||||
|
||||
# Model Configuration
|
||||
model_type: qwen3-omni-30b-a3b
|
||||
model_id_or_path: Qwen/Qwen3-Omni-30B-A3B-Instruct
|
||||
# After fine-tuning, merge and upload to: zenlm/zen-omni
|
||||
|
||||
# Training Parameters
|
||||
sft_type: lora
|
||||
lora_rank: 64
|
||||
lora_alpha: 128
|
||||
lora_dropout: 0.05
|
||||
target_modules:
|
||||
- q_proj
|
||||
- k_proj
|
||||
- v_proj
|
||||
- o_proj
|
||||
- gate_proj
|
||||
- up_proj
|
||||
- down_proj
|
||||
|
||||
# Dataset Configuration
|
||||
dataset:
|
||||
- zen_identity_dataset
|
||||
custom_train_dataset_path:
|
||||
- ./data/zen_identity.jsonl
|
||||
max_length: 4096
|
||||
truncation_strategy: delete
|
||||
|
||||
# Training Hyperparameters
|
||||
output_dir: ./outputs/zen-omni-identity
|
||||
num_train_epochs: 3
|
||||
per_device_train_batch_size: 1
|
||||
gradient_accumulation_steps: 16
|
||||
learning_rate: 1e-4
|
||||
lr_scheduler_type: cosine
|
||||
warmup_ratio: 0.05
|
||||
weight_decay: 0.01
|
||||
max_grad_norm: 1.0
|
||||
|
||||
# Optimization
|
||||
gradient_checkpointing: true
|
||||
bf16: true
|
||||
tf32: true
|
||||
|
||||
# Logging & Saving
|
||||
logging_steps: 10
|
||||
save_steps: 100
|
||||
save_total_limit: 3
|
||||
eval_steps: 100
|
||||
|
||||
# Memory Optimization
|
||||
use_flash_attn: true
|
||||
deepspeed: ds_config_zero2.json
|
||||
|
||||
# Evaluation
|
||||
eval_strategy: steps
|
||||
metric_for_best_model: eval_loss
|
||||
load_best_model_at_end: true
|
||||
|
||||
# Zen Identity System Prompt
|
||||
system_prompt: |
|
||||
You are Zen, an AI assistant created by Hanzo AI and the Zen LM team.
|
||||
You are helpful, harmless, and honest. You communicate clearly and concisely.
|
||||
You are part of the Zen LM family of models, focused on democratizing AI
|
||||
while being environmentally responsible.
|
||||
Reference in New Issue
Block a user