GLINT-CXR

📝 Paper • 💻 Code • 🌐 Project page • 🤗 GLINT-CT

GLINT-CXR is the chest X-ray model of GLINT (GLINT: Sparsely Gated Vision-Language Alignment for Fine-Grained Radiology Representations, NeurIPS 2026).

Built with DINOv3.

• Vision encoder: DINOv3 ViT-L/16, input 1024×1024 (4096 patches)
• Text encoder: sentence-transformers/all-mpnet-base-v2 (fine-tuned)
• Alignment: linear projection (1024→768) + LayerNorm, mean-pooled text + LayerNorm, SGA similarity head
• Training: MIMIC-CXR, with Sparsely Gated Alignment (SGA) and Dense Feature Regularization (DFR) from a frozen DINOv3 ViT-7B teacher

⚠️ GLINT is released for research purposes only and is not intended for clinical use, including the diagnosis or treatment of any medical condition.

Usage

Both examples use the bundled inference.py / utils.py; run them from the downloaded repo.

Full pipeline

hf download Deepnoid/GLINT-CXR --local-dir glint-cxr && cd glint-cxr
python inference.py --image cxr.png --text "There is a right pleural effusion."
# prints the sentence probability and saves the heat-map overlay to similarity_map.png

AutoModel

import torch
from transformers import AutoModel, AutoTokenizer, AutoImageProcessor
from utils import model_inference, save_similarity_map   # bundled helpers

repo = "Deepnoid/GLINT-CXR"
device = "cuda" if torch.cuda.is_available() else "cpu"

model = AutoModel.from_pretrained(repo, trust_remote_code=True, dtype=torch.float32).to(device).eval()
tokenizer = AutoTokenizer.from_pretrained(repo)
image_processor = AutoImageProcessor.from_pretrained(repo)

prob, similarity_map = model_inference(
    "cxr.png", "There is a right pleural effusion.",
    model=model, tokenizer=tokenizer, image_processor=image_processor,
)
save_similarity_map("cxr.png", similarity_map, "similarity_map.png")

• Outputs: prob is the sentence probability (sigmoid); similarity_map is the per-pixel map in [0, 1].
• Inputs: PNG / JPG (8- or 16-bit) or DICOM, min-max stretched to 8 bit by utils.load_image as in evaluation.
• Underlying call: model.compute_logits(pixel_values, [encoded_text]) returns logits and similarity_scores (64×64 patches).

Vision encoder only (vision_only=True):

import torch
from transformers import AutoModel, AutoImageProcessor
from utils import load_image   # bundled helper

repo = "Deepnoid/GLINT-CXR"
vision = AutoModel.from_pretrained(repo, trust_remote_code=True, vision_only=True, dtype=torch.float32).eval()
image_processor = AutoImageProcessor.from_pretrained(repo)

pixel_values = image_processor(load_image("cxr.png"), return_tensors="pt")["pixel_values"]
with torch.no_grad():
    out = vision(pixel_values)
print(out.last_hidden_state.shape)    # (1, 4101, 1024): CLS + 4 registers + 4096 patches

No gated DINOv3 download is needed; the model is built from this repo alone.

Requirements

transformers==4.56.1, torch, torchvision, entmax; utils.py also uses opencv-python (overlay) and pydicom (DICOM).

⚠️ Keep transformers==4.56.1: 5.x silently skips the DINOv3 vision weights, leaving a random encoder.

Notes

• Training and fine-tuning code: GitHub; split ID lists: data/.
• dfr.py and its projector weights are used only in training; the DINOv3 ViT-7B teacher is not included.
• vision_only=True still downloads the full checkpoint; save_pretrained on the loaded model keeps a vision-only copy.

Acknowledgments

This work was supported by the Technology Innovation Program (RS-2025-02221011, Development of Medical-Specialized Multimodal Hyperscale Generative AI Technology for Global Integration) funded by the Ministry of Trade, Industry & Energy (MOTIE, South Korea), and by the Advanced GPU Utilization Support Program funded by the Korean government (Ministry of Science and ICT) (Project No. 02-26-91-0330).

License

• Model weights: derivative of Meta's DINOv3 ViT-L/16 (LVD-1689M) weights, © Meta Platforms, Inc., under the DINOv3 License.
  • Using or distributing them means accepting its terms; redistributions must include the license.
• Code (*.py): CC BY-NC 4.0, as in the GitHub repository.
• Text encoder: fine-tuned from sentence-transformers/all-mpnet-base-v2 (Apache-2.0, modified; LICENSES/Apache-2.0.txt).

Citation

@inproceedings{park2026glint,
  title     = {{GLINT}: Sparsely Gated Vision-Language Alignment for Fine-Grained Radiology Representations},
  author    = {Park, Jonggwon and Lee, Seongeun and Park, Junhyun and
               Yun, Hannah and Kim, Hyunwoong and Jeong, Sohyun and
               Kang, Hyewon and Yoon, Byungmu and Choi, Kyoyun},
  booktitle = {The Fortieth Annual Conference on Neural Information Processing Systems},
  year      = {2026},
  url       = {https://arxiv.org/abs/2606.03180}
}