README.md

---
license: other
library_name: pytorch
tags:
  - audio-generation
  - music-generation
  - dance-to-music
  - motion-conditioned-generation
  - controlnet
  - audioldm
  - pytorch-lightning
  - arxiv:2607.10537
---

# AudioLDM-ControlNet

Official checkpoints and evaluation artifacts for the paper:

**Dance to Music Generation leveraging Pre-training with Unpaired data and Contrastive Alignment**

- Paper: https://huggingface.co/papers/2607.10537
- arXiv: https://arxiv.org/abs/2607.10537
- Source code: https://github.com/kmraven/AudioLDM-ControlNet

## Model Description

AudioLDM-ControlNet is a dance-conditioned music generation model. It combines:

1. pretrained motion and music encoders,
2. beat-guided contrastive pretraining for cross-modal alignment, and
3. a ControlNet-style conditioning module built on an AudioLDM backbone.

This repository includes the **full model**, the **w/o MotionBERT** checkpoint,
along with generated audio, evaluation metrics, training logs, and comparison
videos.

## Repository Contents

Model checkpoints and evaluation outputs are organized under `models/` and
`testsets_video_results/`.

| Path | Contents |
| --- | --- |
| [`models/full/`](https://huggingface.co/kmraven/AudioLDM-ControlNet/tree/main/models/full) | Full-model checkpoint, YAML configuration, 36 generated WAV files and `metrics.txt` in `300ksteps/`, and the W&B training run in `wandb/` |
| [`models/without_motioinbert/`](https://huggingface.co/kmraven/AudioLDM-ControlNet/tree/main/models/without_motioinbert) | w/o MotionBERT checkpoint, YAML configuration, and 36 generated WAV files and `metrics.txt` in `300ksteps/` |
| [`testsets_video_results/`](https://huggingface.co/kmraven/AudioLDM-ControlNet/tree/main/testsets_video_results) | 216 MP4 comparison videos: 36 clips for each of the six conditions listed below |
| `checkpoint-global_step=299999.ckpt` | Full-model checkpoint, also available under `models/full/checkpoints/` |
| `audioldm_original_medium_stretch_pretrained.yaml` | Full-model configuration, also available under `models/full/` |

The root-level checkpoint and configuration are identical to their counterparts
under `models/full/`.

Both checkpoints are named `checkpoint-global_step=299999.ckpt` and are stored
in their respective variant's `checkpoints/` directory. The matching
configurations are:

- Full model: `models/full/audioldm_original_medium_stretch_pretrained.yaml`
- w/o MotionBERT: `models/without_motioinbert/audioldm_original_medium_stretch_wo_mb.yaml`

The six video folders are `AudioLDM (default)`, `Ground Truth`, `Li et al.`,
`Ours`, `Ours (w-o Contrastive Pretraining)`, and `Ours (w-o MotionBERT)`.
The w/o Contrastive Pretraining variant is represented by comparison videos;
a checkpoint for that variant is not included in this release.

## Download and Use

Install the Hugging Face CLI if needed:

```bash
pip install -U huggingface_hub
```

Download all model and evaluation files (approximately 16.5 GB):

```bash
hf download kmraven/AudioLDM-ControlNet \
  --include "models/*" --include "testsets_video_results/*" \
  --local-dir data/releases/AudioLDM-ControlNet
```

To download only the full-model checkpoint and configuration:

```bash
hf download kmraven/AudioLDM-ControlNet \
  "models/full/checkpoints/checkpoint-global_step=299999.ckpt" \
  "models/full/audioldm_original_medium_stretch_pretrained.yaml" \
  --local-dir data/releases/AudioLDM-ControlNet
```

Run preprocessing and evaluation with the
[official source code and setup instructions](https://github.com/kmraven/AudioLDM-ControlNet#data-and-checkpoints).
These are project-specific PyTorch Lightning checkpoints; use the repository's
evaluation scripts. The YAML files specify the training settings and use
relative paths. Dataset inputs and separately referenced upstream weights must
be prepared as described in the source repository. This release
does not include a standalone Stage 1 `model_best.pth` checkpoint.

Exact training resume, including optimizer state and global step, may be
incompatible with the current training code. If exact resume fails, the source
repository recommends commit `0c1f4e3197a7c339586ce5769c6a4510dfbc6f30`.

## Checkpoint Details

- Framework: PyTorch / PyTorch Lightning
- Checkpoint format: `.ckpt`
- Paper variants with checkpoints: Full model and w/o MotionBERT
- Training step: 299999
- Audio sampling rate: 16 kHz
- Motion input frame rate: 60 FPS
- Motion sequence length used by the paper configuration: 128 frames

The checkpoint contains a PyTorch Lightning-compatible `state_dict`.

## Input Representation

The model is conditioned on human pose sequences represented as NumPy arrays with the following shape:

```text
[frames, 17, 3]
```

The 17 joints follow COCO joint order. The final dimension contains the per-joint coordinates expected by the official preprocessing and evaluation code.

## Training Procedure

Training is performed in two stages:

1. MotionBERT and MERT features are aligned using beat-guided contrastive pretraining based on BeatDance.
2. The pretrained dance encoder conditions an AudioLDM backbone through a ControlNet adapter.

The implementation, preprocessing code, training scripts, and evaluation scripts are available in the official GitHub repository.

## Citation

```bibtex
@article{kimura2026dance,
  title={Dance to Music Generation leveraging Pre-training with Unpaired Data and Contrastive Alignment},
  author={Kimura, Ryota and Park, Sangheon and Polouliakh, Natalia and Akama, Taketo},
  journal={arXiv preprint arXiv:2607.10537},
  year={2026}
}
```

## Acknowledgements

This project builds on AudioLDM, ControlNet, MotionBERT, MERT, and BeatDance. Please cite the corresponding projects in addition to this work when using the model.

## License

The source code repository is distributed under the MIT License. The model checkpoint may also be subject to the licenses and terms of the upstream pretrained models and datasets. Review the applicable upstream licenses before redistribution or commercial use.