Speedup Volume Understanding with Efficient Multimodal Large Language Models

✨Pretrained-weights of Phase 2✨

❗We have uploaded weights for different benchmarks in different branches of this repository.❗

👓 Overview

Multimodal large language models are promising for clinical visual question answering tasks, but scaling to 3D imaging is hindered by high computational costs. Prior methods often rely on 2D slices or fixed-length token compression, disrupting volumetric continuity and obscuring subtle findings. We present Photon, a framework that represents 3D medical volumes with token sequences of variable length. Photon introduces instruction-conditioned token scheduling and surrogate gradient propagation to adaptively reduce tokens during both training and inference, which lowers computational cost while mitigating the attention dilution caused by redundant tokens. It incorporates a custom backpropagation rule with gradient restoration to enable differentiable optimization despite discrete token drop. To stabilize token compression and ensure reliable use of visual evidence, Photon further applies regularization objectives that mitigate language-only bias and improve reliability. Experiments on diverse medical visual question answering tasks show that Photon achieves state-of-the-art accuracy while reducing resource usage and accelerating both training and inference.

📚 Data Preparation

We preprocess the CT volumes by reorienting them to RAI and resampling to (1, 1, 1) spacing. Then, we center-crop or pad them to the shape (364, 364, 364).

Data Sample

Our data format follows the specification compatible with ms-swift, and we recommend using JSON or JSONL format for data construction. Here is an example:

[
  {
    "messages": [
      {
        "role": "user",
        "content": "<video> Your instruct here."
      },
      {
        "role": "assistant",
        "content": "The response here."
      }
    ],
    "videos": [
      "XXX/XXX.nii.gz"
    ],
  },
]

📎 Citation

If you find the code helpful in your research or work, please cite the following paper(s).

@inproceedings{fang2026photon,
  title={Photon: Speedup volume understanding with efficient multimodal large language models},
  author={Fang, Chengyu and Guo, Heng and Jiang, Zheng and He, Chunming and Li, Xiu and Xu, Minfeng},
  booktitle={The Fourteenth International Conference on Learning Representations},
  year={2026}
}

⭐ Acknowledgement

We build our framework based on ms-swift. For model training, we utilize a comprehensive dataset collection comprising CT-RATE, 3D-RAD, DeepTumorVQA, and AbdomenAtlas-3.0-Report. We sincerely thank all researchers and open-source contributors advancing the field of medical multimodal understanding.