zenlm/zen-musician
1
0
Fork 0
mirror of https://github.com/zenlm/zen-musician.git synced 2026-07-26 22:08:52 +00:00
Code Issues Packages Projects Releases Wiki Activity

update readme

Browse Source
This commit is contained in:
a43992899
2025-01-29 14:10:00 +08:00
parent 543bcc3743
commit 86cdc343f9
4 changed files with 305 additions and 261 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+65 -61
View File
@@ -13,16 +13,9 @@
</p>
---
Our model's name is **YuE (乐)**. In Chinese, the word means "music" and "happiness." Some of you may find words that start with Yu hard to pronounce. If so, you can just call it "yeah." We wrote a song with our model's name.
Our model's name is **YuE (乐)**. In Chinese, the word means "music" and "happiness." Some of you may find words that start with Yu hard to pronounce. If so, you can just call it "yeah." We wrote a song with our model's name, see [here](assets/logo/yue.mp3).
<audio controls src="https://cdn-uploads.huggingface.co/production/uploads/6555e8d8a0c34cd61a6b9ce3/rG-ELxMyzDU7zH-inB9DV.mpga"></audio>
YuE is a groundbreaking series of open-source foundation models designed for music generation, specifically for transforming lyrics into full songs (lyrics2song). It can generate a complete song, lasting several minutes, that includes both a catchy vocal track and complementary accompaniment, ensuring a polished and cohesive result. YuE is capable of modeling diverse genres/vocal styles. Below are examples of songs in the pop and metal genres. For more styles, please visit the demo page.
Pop:Quiet Evening
<audio controls src="https://cdn-uploads.huggingface.co/production/uploads/640701cb4dc5f2846c91d4eb/gnBULaFjcUyXYzzIwXLZq.mpga"></audio>
Metal: Step Back
<audio controls src="https://cdn-uploads.huggingface.co/production/uploads/6555e8d8a0c34cd61a6b9ce3/kmCwl4GRS70UYDEELL-Tn.mpga"></audio>
YuE is a groundbreaking series of open-source foundation models designed for music generation, specifically for transforming lyrics into full songs (lyrics2song). It can generate a complete song, lasting several minutes, that includes both a catchy vocal track and complementary accompaniment, ensuring a polished and cohesive result. YuE is capable of modeling diverse genres/vocal styles. Below are examples of songs in the pop and metal genres. Please visit the [**Demo Page**](https://map-yue.github.io/) for amazing vocal performance.
## News and Updates
@@ -31,48 +24,44 @@ Metal: Step Back
<br>
## Requirements
Python >=3.8 is recommended.
Install dependencies with the following command:
```
pip install -r requirements.txt
```
### **Important: Install FlashAttention 2**
For saving GPU memory, **FlashAttention 2 is mandatory**. Without it, large sequence lengths will lead to out-of-memory (OOM) errors, especially on GPUs with limited memory. Install it using the following command:
```
pip install flash-attn --no-build-isolation
```
Before installing FlashAttention, ensure that your CUDA environment is correctly set up.
For example, if you are using CUDA 11.8:
- If using a module system:
``` module load cuda11.8/toolkit/11.8.0 ```
- Or manually configure CUDA in your shell:
```
export PATH=/usr/local/cuda-11.8/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-11.8/lib64:$LD_LIBRARY_PATH
```
---
## GPU Memory Usage and Sessions
## Hardware and Performance
### **GPU Memory**
YuE requires significant GPU memory for generating long sequences. Below are the recommended configurations:
- **For GPUs with 24GB memory or less**: Run **up to 2 sessions** concurrently to avoid out-of-memory (OOM) errors.
- **For full song generation** (many sessions, e.g., 4 or more): Use **GPUs with at least 80GB memory**. This can be achieved by combining multiple GPUs and enabling tensor parallelism.
- **For full song generation** (many sessions, e.g., 4 or more): Use **GPUs with at least 80GB memory**. i.e. H800, A100, or multiple RTX4090s with tensor parallel.
To customize the number of sessions, the interface allows you to specify the desired session count. By default, the model runs **2 sessions** for optimal memory usage.
To customize the number of sessions, the interface allows you to specify the desired session count. By default, the model runs **2 sessions** (1 verse + 1 chorus) to avoid OOM issue.
### **Execution Time**
On an **H800 GPU**, generating 30s audio takes **150 seconds**.
On an **RTX 4090 GPU**, generating 30s audio takes approximately **360 seconds**.
---
## Quickstart
Thanks to Fahd for creating a tutorial on how to quickly get started with YuE. Here is his [demonstration](https://www.youtube.com/watch?v=RSMNH9GitbA).
Quick start **VIDEO TUTORIAL** by Fahd: [Link here](https://www.youtube.com/watch?v=RSMNH9GitbA). We recommend watching this video if you are not familiar with machine learning or the command line.
### 1. Install environment and dependencies
Make sure properly install flash attention 2 to reduce VRAM usage.
```bash
# We recommend using conda to create a new environment.
conda create -n yue python=3.8 # Python >=3.8 is recommended.
conda activate yue
# install cuda >= 11.8
conda install pytorch torchvision torchaudio cudatoolkit=11.8 -c pytorch -c nvidia
pip install -r requirements.txt
# For saving GPU memory, FlashAttention 2 is mandatory. Without it, large sequence lengths will lead to out-of-memory (OOM) errors, especially on GPUs with limited memory.
# Be careful about matching the cuda version and flash-attn version
pip install flash-attn --no-build-isolation
```
### 2. Download the infer code and tokenizer
```bash
# Make sure you have git-lfs installed (https://git-lfs.com)
git lfs install
git clone https://github.com/multimodal-art-projection/YuE.git
@@ -81,21 +70,25 @@ cd YuE/inference/
git clone https://huggingface.co/m-a-p/xcodec_mini_infer
```
Here’s a quick guide to help you generate music with **YuE** using 🤗 Transformers. Before running the code, make sure your environment is properly set up, and that all dependencies are installed.
### 3. Run the inference
Now generate music with **YuE** using 🤗 Transformers. Make sure your step [1](#1-install-environment-and-dependencies) and [2](#2-download-the-infer-code-and-tokenizer) are properly set up.
### Running the Script
Note:
- Set `--run_n_segments` to the number of lyric sections if you want to generate a full song. Additionally, you can increase `--stage2_batch_size` based on your available GPU memory.
In the following example, customize the `genres` and `lyrics` in the script, then execute it to generate a song with **YuE**.
- You may customize the prompt in `genre.txt` and `lyrics.txt`. See prompt engineering guide [here](#prompt-engineering-guide).
- LM ckpts will be automatically downloaded from huggingface.
Notice: Set `--run_n_segments` to the number of lyric sections if you want to generate a full song. Additionally, you can increase `--stage2_batch_size` based on your available GPU memory.
```bash
# This is the CoT mode.
cd YuE/inference/
python infer.py \
--stage1_model m-a-p/YuE-s1-7B-anneal-en-cot \
--stage2_model m-a-p/YuE-s2-1B-general \
--genre_txt prompt_examples/genre.txt \
--lyrics_txt prompt_examples/lyrics.txt \
--genre_txt genre.txt \
--lyrics_txt lyrics.txt \
--run_n_segments 2 \
--stage2_batch_size 4 \
--output_dir ./output \
@@ -103,14 +96,25 @@ python infer.py \
--max_new_tokens 3000
```
If you want to use audio prompt, enable `--use_audio_prompt`, and provide audio prompt:
If you want to use music in-context-learning (provide a reference song), enable `--use_audio_prompt`, `--prompt_start_time`, and `--prompt_end_time` to specify the audio segment.
Note:
- Music ICL generally requires a 30s audio segment. The model will write new songs with similiar style of the provided audio, and may improve musicality.
- We have 4 modes for ICL: mix, vocal, instrumental, and dual-track.
- We currently only support mix mode.
- Dual-track mode work the best, will support in the infer code soon.
```bash
# This is the ICL mode. Currently only mix-ICL is supported.
cd YuE/inference/
python infer.py \
--stage1_model m-a-p/YuE-s1-7B-anneal-en-icl \
--stage2_model m-a-p/YuE-s2-1B-general \
--genre_txt prompt_examples/genre.txt \
--lyrics_txt prompt_examples/lyrics.txt \
--genre_txt genre.txt \
--lyrics_txt lyrics.txt \
--run_n_segments 2 \
--stage2_batch_size 4 \
--output_dir ./output \
@@ -120,26 +124,26 @@ python infer.py \
--prompt_start_time 0 \
--prompt_end_time 30
```
---
## Prompt Engineering Guide
### Lyrics Prompt
1. An example lyric prompt can be found [here](./prompt_examples/lyrics.txt).
### **Execution Time**
On an **H800 GPU**, generating 30s audio takes **150 seconds**.
On an **RTX 4090 GPU**, generating 30s audio takes approximately **360 seconds**.
2. The lyrics prompt should be divided into sessions, with structure labels (e.g., [verse], [chorus], [bridge], [outro]) prepended. Each session should be separated by 2 newline character "\n\n".
**Tips:**
1. `genres` should include details like instruments, genre, mood, vocal timbre, and vocal gender.
2. The length of `lyrics` segments and the `--max_new_tokens` value should be matched. For example, if `--max_new_tokens` is set to 3000, the maximum duration for a segment is around 30 seconds. Ensure your lyrics fit this time frame.
3. If using audio prompt，the duration around 30s will be fine.
---
3. **DONOT** put too many words in a single segment, since each session is around 30s (`--max_new_tokens 3000` by default).
### Notice
1. A suitable [Genre] tag consists of five components: genre, instrument, mood, gender, and timbre. All five should be included if possible, separated by spaces. The values of timbre should include "vocal" (e.g., "bright vocal").
4. We find that [intro] label is less stable, so we recommend starting with [verse] or [chorus].
2. Although our tags have an open vocabulary, we have provided the 200 most commonly used [tags](./wav_top_200_tags.json). It is recommended to select tags from this list for more stable results.
### Genre Tagging Prompt
1. An example genre tagging prompt can be found [here](./prompt_examples/genre.txt).
3. The order of the tags is flexible. For example, a stable genre control string might look like: "[Genre] inspiring female uplifting pop airy vocal electronic bright vocal vocal."
2. A stable tagging prompt usually consists of five components: genre, instrument, mood, gender, and timbre. All five should be included if possible, separated by space (space delimiter).
3. Although our tags have an open vocabulary, we have provided the top 200 most commonly used [tags](./top_200_tags.json). It is recommended to select tags from this list for more stable results.
3. The order of the tags is flexible. For example, a stable genre tagging prompt might look like: "inspiring female uplifting pop airy vocal electronic bright vocal vocal."
4. Additionally, we have introduced the "Mandarin" and "Cantonese" tags to distinguish between Mandarin and Cantonese, as their lyrics often share similarities.
genre.txt
+1
View File
@@ -0,0 +1 @@
inspiring female uplifting pop airy vocal electronic bright vocal vocal
lyrics.txt
+39
View File
@@ -0,0 +1,39 @@
[verse]
Staring at the sunset, colors paint the sky
Thoughts of you keep swirling, can't deny
I know I let you down, I made mistakes
But I'm here to mend the heart I didn't break
[chorus]
Every road you take, I'll be one step behind
Every dream you chase, I'm reaching for the light
You can't fight this feeling now
I won't back down
You know you can't deny it now
I won't back down
[verse]
They might say I'm foolish, chasing after you
But they don't feel this love the way we do
My heart beats only for you, can't you see?
I won't let you slip away from me
[chorus]
Every road you take, I'll be one step behind
Every dream you chase, I'm reaching for the light
You can't fight this feeling now
I won't back down
You know you can't deny it now
I won't back down
[bridge]
No, I won't back down, won't turn around
Until you're back where you belong
I'll cross the oceans wide, stand by your side
Together we are strong
[outro]
Every road you take, I'll be one step behind
Every dream you chase, love's the tie that binds
You can't fight this feeling now
I won't back down
wav_top_200_tags.json → top_200_tags.json
+200 -200
View File
@@ -626,205 +626,205 @@
"children"
],
"timbre": [
"bright",
"full",
"airy",
"clear",
"mellow",
"dark",
"rich",
"reverb",
"light",
"crisp",
"broad",
"powerful",
"piercing",
"high-pitched",
"bass",
"deep",
"not applicable",
"baritone",
"not specified",
"vibrant",
"boomy",
"varied",
"bouncy",
"range",
"harsh",
" airy",
"round",
"uplifting",
"soft",
"husky",
"tenor",
"pontificate",
"aggressive",
"neat",
"high",
"exuberant",
"open",
"full bodied",
"strong",
"grainy",
"vocal fry",
"gravelly",
"low",
"long_release",
"polished",
"velvet",
"placid",
"plastic",
"sharp",
"robust",
"muffled",
"distortion",
"crunchy",
"resonant",
"pure",
"年轻",
"preenched",
"gruff",
"raspy",
"passionate",
"nonlinear_env",
"high pitched",
"athletic",
"reedy",
"shimmering",
"charismatic",
"gliding",
"raw",
"plucky",
"loud",
"youthful",
"thin",
"soulful",
"smooth",
"flat",
"tempo-synced",
"opulent",
"variable",
"happy",
"prettily",
"percussive",
"singing voice",
"barrel",
"breezy",
"vocal",
"honeyed",
"vivacious",
"full-bodied",
"persuasive",
"tender",
"potent",
"preppy",
" raspy",
"narrow",
"fruity",
"whiny",
"hollow",
"singing",
"rapping",
"flexible",
" alto",
"sweet",
"agitated",
"shaky",
"dainty",
"明亮",
"soprano",
"vocal range",
"rough",
"有力",
"成熟",
"sultry",
"barren",
"bulky",
"prevalent",
"bellowing",
"dusty",
"elevated",
"wide",
"rumbly",
"shrill",
"prettily produced",
"projected",
"low pitched",
"bold",
"grassy",
"plush",
"glorious",
"elevated pitch",
"whispery",
"long",
"nasal",
"preened",
"squeaky",
"hellosing",
"commanding",
"textural",
"noble",
"frustrated",
"warm",
"punchy",
"pretty",
"changeable",
"mushy",
"vocalist",
"gritty",
"barking",
"human",
"bass heavy",
"dulcet",
" smooth",
"young",
"rhythmic",
"vocals",
"helmet",
"screamy",
"hoarse",
"rebellious",
"soothing",
"童声",
"bitter",
"为了让声乐更加生动，使用了混响效果。",
"barrel-shaped",
"reed",
"强有力",
"低沉",
"whimsical",
"exaggerated",
"温暖",
"low-pitched",
"emotional",
"graceful",
"breakable",
"screechy",
"muddy",
"breathy",
"柔和",
"weathered",
"roaring",
"青春",
"pensive",
"textured",
"清脆",
"melodic",
"helmeted",
" velvety",
"充满活力",
"圆润",
"preteen",
"rhythm",
"treble",
"shouty",
" husky",
"medium",
"blue",
"screeching",
"multiphonic",
"quaint",
"rhytmic",
"轻盈"
"bright vocal",
"full vocal",
"airy vocal",
"clear vocal",
"mellow vocal",
"dark vocal",
"rich vocal",
"reverb vocal",
"light vocal",
"crisp vocal",
"broad vocal",
"powerful vocal",
"piercing vocal",
"high-pitched vocal",
"bass vocal",
"deep vocal",
"not applicable vocal",
"baritone vocal",
"not specified vocal",
"vibrant vocal",
"boomy vocal",
"varied vocal",
"bouncy vocal",
"range vocal",
"harsh vocal",
" airy vocal",
"round vocal",
"uplifting vocal",
"soft vocal",
"husky vocal",
"tenor vocal",
"pontificate vocal",
"aggressive vocal",
"neat vocal",
"high vocal",
"exuberant vocal",
"open vocal",
"full bodied vocal",
"strong vocal",
"grainy vocal",
"vocal fry vocal",
"gravelly vocal",
"low vocal",
"long_release vocal",
"polished vocal",
"velvet vocal",
"placid vocal",
"plastic vocal",
"sharp vocal",
"robust vocal",
"muffled vocal",
"distortion vocal",
"crunchy vocal",
"resonant vocal",
"pure vocal",
"年轻 vocal",
"preenched vocal",
"gruff vocal",
"raspy vocal",
"passionate vocal",
"nonlinear_env vocal",
"high pitched vocal",
"athletic vocal",
"reedy vocal",
"shimmering vocal",
"charismatic vocal",
"gliding vocal",
"raw vocal",
"plucky vocal",
"loud vocal",
"youthful vocal",
"thin vocal",
"soulful vocal",
"smooth vocal",
"flat vocal",
"tempo-synced vocal",
"opulent vocal",
"variable vocal",
"happy vocal",
"prettily vocal",
"percussive vocal",
"singing voice vocal",
"barrel vocal",
"breezy vocal",
"vocal vocal",
"honeyed vocal",
"vivacious vocal",
"full-bodied vocal",
"persuasive vocal",
"tender vocal",
"potent vocal",
"preppy vocal",
" raspy vocal",
"narrow vocal",
"fruity vocal",
"whiny vocal",
"hollow vocal",
"singing vocal",
"rapping vocal",
"flexible vocal",
" alto vocal",
"sweet vocal",
"agitated vocal",
"shaky vocal",
"dainty vocal",
"明亮 vocal",
"soprano vocal",
"vocal range vocal",
"rough vocal",
"有力 vocal",
"成熟 vocal",
"sultry vocal",
"barren vocal",
"bulky vocal",
"prevalent vocal",
"bellowing vocal",
"dusty vocal",
"elevated vocal",
"wide vocal",
"rumbly vocal",
"shrill vocal",
"prettily produced vocal",
"projected vocal",
"low pitched vocal",
"bold vocal",
"grassy vocal",
"plush vocal",
"glorious vocal",
"elevated pitch vocal",
"whispery vocal",
"long vocal",
"nasal vocal",
"preened vocal",
"squeaky vocal",
"hellosing vocal",
"commanding vocal",
"textural vocal",
"noble vocal",
"frustrated vocal",
"warm vocal",
"punchy vocal",
"pretty vocal",
"changeable vocal",
"mushy vocal",
"vocalist vocal",
"gritty vocal",
"barking vocal",
"human vocal",
"bass heavy vocal",
"dulcet vocal",
" smooth vocal",
"young vocal",
"rhythmic vocal",
"vocals vocal",
"helmet vocal",
"screamy vocal",
"hoarse vocal",
"rebellious vocal",
"soothing vocal",
"童声 vocal",
"bitter vocal",
"为了让声乐更加生动，使用了混响效果。 vocal",
"barrel-shaped vocal",
"reed vocal",
"强有力 vocal",
"低沉 vocal",
"whimsical vocal",
"exaggerated vocal",
"温暖 vocal",
"low-pitched vocal",
"emotional vocal",
"graceful vocal",
"breakable vocal",
"screechy vocal",
"muddy vocal",
"breathy vocal",
"柔和 vocal",
"weathered vocal",
"roaring vocal",
"青春 vocal",
"pensive vocal",
"textured vocal",
"清脆 vocal",
"melodic vocal",
"helmeted vocal",
" velvety vocal",
"充满活力 vocal",
"圆润 vocal",
"preteen vocal",
"rhythm vocal",
"treble vocal",
"shouty vocal",
" husky vocal",
"medium vocal",
"blue vocal",
"screeching vocal",
"multiphonic vocal",
"quaint vocal",
"rhytmic vocal",
"轻盈 vocal"
]
}