| --- |
| license: mit |
|
|
| tags: |
| - tokenization |
| - video generation |
| - vae |
| --- |
| |
| # VidTwin |
| Video VAE with Decoupled Structure and Dynamics |
|
|
| <img src="./assets/vidtwin_demo.png" width="95%" alt="demo" align="center"> |
|
|
| We propose a novel and compact video autoencoder, VidTwin, that decouples video into two distinct latent spaces: **Structure latent vectors**, which capture overall content and global movement, and **Dynamics latent vectors**, which represent fine-grained details and rapid movements. |
|
|
| Extensive experiments show that VidTwin achieves a high compression rate of 0.20% with high reconstruction quality (PSNR of 28.14 on the MCL-JCV dataset), and performs efficiently and effectively in downstream generative tasks. Moreover, our model demonstrates explainability and scalability, paving the way for future research in video latent representation and generation. |
|
|
| Resources and technical documentation: |
|
|
| + [GitHub](https://github.com/microsoft/VidTok/tree/main/vidtwin) |
| + [arXiv](https://arxiv.org/pdf/2412.17726) |
|
|
| ## Setup |
|
|
| 1. Our code is based on **VidTok**, so you will need to install the [required packages for VidTok](https://github.com/microsoft/VidTok?tab=readme-ov-file#setup) first. To do so, navigate to the VidTok folder and create the environment using the `environment.yaml` file: |
|
|
| ```bash |
| cd VidTok |
| # Prepare conda environment |
| conda env create -f environment.yaml |
| # Activate the environment |
| conda activate vidtok |
| ``` |
|
|
| 2. After setting up VidTok, install the additional packages required for the VidTwin model: |
| ```bash |
| pip install tranformers |
| pip install timm |
| pip install flash-attn --no-build-isolation |
| ``` |
|
|
|
|
| ## Training |
|
|
| Please refer to the [paper](https://arxiv.org/pdf/2412.17726) and [code](https://github.com/microsoft/VidTok/tree/main/vidtwin) for detailed training instructions. |
|
|
| ## Inference |
|
|
| Please refer to the [paper](https://arxiv.org/pdf/2412.17726) and [code](https://github.com/microsoft/VidTok/tree/main/vidtwin) for detailed inference instructions. |
|
|
| ## Intended Uses |
|
|
| We are sharing our model with the research community to foster further research in this area: |
| * Training your own video tokenizers for research purpose. |
| * Video tokenization with various compression rates. |
|
|
| ## Downstream Uses |
|
|
| Our model is designed to accelerate research on video-centric research, for use as a building block for the following applications: |
| * Video generation on the continuous / discrete latent tokens. |
| * World modelling on the continuous / discrete latent tokens. |
| * Generative games on the continuous / discrete latent tokens. |
| * Video understanding from the latent tokens. |
|
|
| ## Out-of-scope Uses |
|
|
| Our models are not specifically designed or evaluated for all downstream purposes. Developers should consider common limitations of video tokenizers (e.g., performance degradation on out-of-domain data) as they select use cases, and evaluate and mitigate for privacy, safety, and fairness before using within a specific downstream use case, particularly for high-risk scenarios. |
|
|
| Developers should be aware of and adhere to applicable laws or regulations (including privacy, trade compliance laws, etc.) that are relevant to their use case. |
|
|
| ## Risks and Limitations |
|
|
| Some of the limitations of this model to be aware of include: |
| * VidTwin may lose detailed information on the reconstructed content. |
| * VidTwin inherits any biases, errors, or omissions characteristic of its training data. |
| * VidTwin was developed for research and experimental purposes. Further testing and validation are needed before considering its application in commercial or real-world scenarios. |
|
|
| ## Recommendations |
|
|
| Some recommendations for alleviating potential limitations include: |
| * Lower compression rate provides higher reconstruction quality. |
| * For domain-specific video tokenization, it is suggested to fine-tune the model on the domain-specific videos. |
|
|
| ## License |
|
|
| The model is released under the [MIT license](https://github.com/microsoft/VidTok/blob/main/LICENSE). |
|
|
| ## Contact |
|
|
| We welcome feedback and collaboration from our audience. If you have suggestions, questions, or observe unexpected/offensive behavior in our technology, please contact us at junliangguo@microsoft.com. |
|
|
| ## BibTeX |
| If you find our project helpful to your research, please consider starring this repository🌟 and citing our paper. |
| ```bibtex |
| @article{wang2024vidtwin, |
| title={VidTwin: Video VAE with Decoupled Structure and Dynamics}, |
| author={Wang, Yuchi and Guo, Junliang and Xie, Xinyi and He, Tianyu and Sun, Xu and Bian, Jiang}, |
| year={2024}, |
| journal={arXiv preprint arXiv:2412.17726}, |
| } |
| ``` |
