YuE (乐)

The Hong Kong University of Science and Technology and Mutimodal Art Projection (M-A-P) · 2025

YuE is an autoregressive LM-based framework tailored for lyrics-to-song generation.

Application types: Lyrics-to-songStyle transferText-to-song
Architecture:Autoregressive transformer

Essential categories

CategoryStatusNotes
Source code~ PartialSource code is available in the GitHub repo, including inference code and model architecture. They also provide instructions on how to fine-tune YuE for specific data (see https://github.com/multimodal-art-projection/YuE/tree/main/finetune), including data processing and training procedure. Code to train the model from scratch is missing.
Training data✘ ClosedSpeech-related datasets are introduced (i.e., WeNetSpeech (zh), LibriHeavy (en), and GigaSpeech (en)). But for music, they only indicate that they rely on 650K hours of music recordings. No information on specific data sources, acquisition methods, and data licensing is given. See "Data Setup" in Section 5.1. of the technical report.
Model weights✔︎ OpenCheckpoints are available in Hugging Face.
Code documentation✔︎ OpenCodebase is documented in the GitHub repository, including installation requirements and usage instructions.
Training procedure✔︎ OpenTraining procedure is properly described, including hardware requirements and configurations (see Section 4 and Section 5.1).
Evaluation procedure✔︎ OpenEvaluation procedure is described in Section 5.2 of the technical report, including both objective and subjective evaluation methods. Evaluation metrics implementations are referenced in the text.
Research paper~ PartialTechnical report available, but no peer-reviewed publication yet.
Licensing✔︎ OpenApache License 2.0

Desirable categories

CategoryStatusNotes
Model card∅ Not included
Datasheet∅ Not included
Package∅ Not included
User-oriented application∅ Not includedNo official user-oriented application is available. But there are user-oriented applications available for YuE, for both Windows and Linux. For Windows, check out: https://github.com/multimodal-art-projection/YuE?tab=readme-ov-file#-windows-users-quickstart, and for Linux: https://github.com/multimodal-art-projection/YuE?tab=readme-ov-file#-linuxwsl-users-quickstart.
Supplementary material page⭐ IncludedA demo page is available showcasing the use of YuE.

Raw YAML file with complete evaluation.