MCVO — Multi-frame, Camera-only Visual Odometry

A transformer that predicts relative camera pose and camera intrinsics from images alone, trained self-supervised on ~80k frames of unlabeled video. Pretrained depth (UniDepth), optical-flow (UniMatch) and calibration (AnyCalib) networks supervise a flow re-projection loss at training time only; none of them runs at inference. Research code from TU Munich (Chair of Computer Vision, Prof. Cremers), in development; code and benchmark: https://github.com/kalman17/mcvo

Files

• mcvo_e3p_calib.pt — recommended: pose + intrinsics (E3′ weights with the calibration head distilled on them; pose output identical to mcvo_e3p.pt)
• mcvo_e3p.pt — the same weights without the trained calibration head (pose only)
• mcvo_e3.pt, mcvo_e3_calib.pt — the previous release, trained with a depth-convention error in the loss (see the changelog in the repository); kept for reference

import torch
from mcvo.model import MCVO                     # from the GitHub repo, PYTHONPATH=.

ck = torch.load("mcvo_e3p_calib.pt", map_location="cpu", weights_only=False)
a = ck["args"]
model = MCVO(backbone=a["backbone"], d_model=a["d_model"], depth=a["depth"], heads=a["heads"]).eval()
model.load_state_dict(ck["model_state_dict"])
out = model(images=frames)     # frames: [1, N, 3, 336, 336], RGB in [0, 1]
out["poses"]                   # [1, N, 1, 4, 4]  cam_i -> cam_{i+1}, last = identity
out["calib"]                   # [1, N, 4]        fx, fy, cx, cy (px)

Model

• Frozen DINOv2-base backbone (86 M) → patch tokens per frame, plus one learned camera token per frame.
• 10 transformer blocks, each: temporal attention across frames per patch position → spatial attention within each frame (camera token included) → MLP. d = 640, 8 heads. 67 M trained parameters, 154 M total.
• Heads: pose (MLP on adjacent camera tokens → translation + quaternion), calibration (linear on the camera token → focal length, principal point; distilled from AnyCalib, head-only), uncertainty (linear per patch; used only in the loss).
• Training: 6 epochs on 8-frame clips (RealEstate10K, YouTube-VOS, EpicKitchens, WalkingTours preprocessed with the AnyCam pipeline), batch 4, lr 1.5e-4, uncertainty-weighted flow re-projection loss, one A40-class GPU.
• Input: N frames, 336×336, RGB in [0, 1]. Output: N−1 relative poses and per-frame intrinsics.

Numbers (NVIDIA A40, 4-frame windows, medians; same harness for every model)

Method	Labels	Params	Peak GPU mem	Latency	Rotation Sintel / TUM / KITTI	Heading KITTI	Focal error Sintel / TUM / KITTI
MCVO (mcvo_e3p_calib.pt)	none	154 M (67 M trained)	0.69 GiB	75 ms	0.48° / 0.76° / 0.18°	5.1°	31.1 % / 14.3 % / 37.1 %
π³	GT	959 M	5.5 GiB	171 ms	0.22° / 0.26° / 0.11°	2.2°	25.2 % / 7.6 % / 28.9 %
VGGT-1B	GT	1257 M	7.0 GiB	203 ms	0.28° / 0.32° / 0.12°	4.6°	34.0 % / 25.8 % / 37.1 %
Depth Anything 3	GT	1690 M	9.7 GiB	600 ms	0.19° / 0.27° / 0.09°	1.3°	24.4 % / 4.6 % / 15.8 %
AnyCam (CVPR 2025)	none	115 M incl. depth & flow nets	3.7 GiB	413 ms	0.50° / 0.74° / 0.20°	28.6°	70.3 % / 14.6 % / 66.9 %
MCT + AnyCam (thesis)	none	460 M	5.0 GiB	820 ms	0.40° / 0.67° / 0.23°	28.2°	20.7 % / 12.9 % / 20.4 %
Monodepth2 pose net (trained on KITTI)	none	13 M	0.09 GiB	13 ms	0.80° / 0.77° / 0.30°	1.2°	—

Similar rotation accuracy to AnyCam at 5.5× lower latency and 5× less memory; roughly twice the rotation error of the billion-parameter supervised models at 8–14× less memory and 2–8× lower latency. Translation direction is good on driving video and weak on small-baseline indoor video (Sintel 104°, TUM-RGBD 75°; 90° is chance). Trajectory-level Sim3 ATE from chained 8-frame windows without bundle adjustment: Sintel 0.29, TUM-RGBD 0.14 (AnyCam feed-forward: 0.10 on Sintel). Full table, protocol, raw rows and the dated changelog: GitHub repo.

Attribution

Builds on AnyCam (Wimbauer et al., CVPR 2025) for the self-supervised loss and data pipeline, DINOv2 (Meta) as backbone, UniDepth, UniMatch and AnyCalib as training-time teachers. Architecture follows the time-space attention design of FVO (Yugay et al., 2025), trained self-supervised instead of with ground-truth poses. MIT.