Mustango

AMAAI Lab, Singapore University of Technology and Design and Queen Mary University of London · 2024

A music-domain-knowledge-inspired text-to-music latent diffusion model capable of fine-grained control over harmonic and rhythmic elements via an integrated MuNet architecture.

Application types: Text-to-music
Architecture:Latent diffusion

Essential categories

CategoryStatusNotes
Source code✔︎ OpenThe codebase is complete and accessible on GitHub. The repository features model architecture, training pipeline, inference, and data processing.
Training data✔︎ OpenMustango is trained on the MusicBench dataset, which is a collection of ~52k 10-seconds music samples. These samples were generated through targeted harmonic, rhythmic, and dynamic data augmentations of the original MusicCaps dataset and enriched with automated music theory text descriptions.
Model weights✔︎ OpenModel weights are available on HuggingFace. There is also a pretrained model available (https://huggingface.co/declare-lab/mustango-pretrained).
Code documentation✔︎ OpenThe codebase is properly documented, including instructions for installation, basic usage and model training.
Training procedure✔︎ OpenTraining procedure is well documented in the paper, including hardware requirements and model configurations (see Appendix B for further details). Additional implementation details are available in the GitHub repository.
Evaluation procedure✔︎ OpenEvaluation methodology is well documented in the paper, including evaluation metrics and model performance. Evaluation dataset is composed of TestA, TestB and FMACaps. TestA and TestB are the testing splits of MusicBench, which is based on MusicCaps (see Section 2.3). Further information about how FMACaps dataset was created is available in Appendix F.
Research paper✔︎ OpenAccepted paper at NAACL 2024.
Licensing✔︎ OpenMIT License.

Desirable categories

CategoryStatusNotes
Model card⭐ IncludedA model card is available in HuggingFace.
Datasheet⭐ IncludedThere is a data card available for the MusicBench dataset on HuggingFace, which provides information about the dataset's composition, usage, and licensing.
Package∅ Not included
User-oriented application⭐ IncludedA demo is available on Replicate. There should also be a demo on HuggingFace Spaces, but it is currently not working (https://huggingface.co/spaces/declare-lab/mustango).
Supplementary material page⭐ IncludedComplementary site with general model overview and sound examples is available.

Raw YAML file with complete evaluation.