LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers
Abstract
LLM routing is formalized as a sequential decision process with a unified benchmark and modular infrastructure to compare and improve cost-effective model selection.
No single large language model (LLM) is optimal across all queries and budget constraints, making model routing essential for cost-effective deployment. Existing routers adopt diverse formulations and implementations, making fair comparison and extension difficult. We present a unified formulation of LLM routing as a sequential decision process characterized by five components: context encoders, model encoders, scoring functions, decision rules, and learning signals, covering single-turn, multi-turn, and personalized routing. Based on this formulation, we develop an automated pipeline for constructing routing supervision and evaluating routers jointly on response quality and inference cost. The resulting benchmark, xRouteBench, spans generic LLM, memory-augmented, vision, time-series, and personalized routing tasks. We further introduce LLMRouter, an open-source modular infrastructure with more than 16 representative routers. Our empirical study shows that learned routers outperform the strongest fixed-model baseline by 14.6% relatively, lightweight routers become more competitive under tight cost constraints, and user-conditioned routing consistently improves personalization.
Community
🚀 LLMRouter is a comprehensive ecosystem for LLM routing — spanning benchmarking, algorithm development, and real-world/agentic deployment.
📊 Broad Benchmark Coverage:
xRouteBench goes far beyond conventional single-turn text routing, covering 5 diverse routing scenarios: generic LLM tasks (knowledge & commonsense QA, math, and code), long-context memory, vision (image & video), time-series reasoning, and personalized routing. More importantly, it is not just an evaluation collection — LLMRouter provides an end-to-end data pipeline that automatically constructs train/test splits, routing supervision, model responses, task-specific scores, and inference costs, making the benchmarks directly usable for both router training and standardized evaluation.
🧩 Comprehensive Router Library:
LLMRouter unifies 16+ representative routing methods across the major router paradigms: single-turn routers, multi-turn & agentic routers, and personalized routers. All routers share the same modular interface and infrastructure, enabling users to switch, train, and evaluate routers through a configuration-driven workflow without rebuilding the pipeline for each method. The framework also supports performance–cost evaluation and a ComfyUI-based visual interface for code-free routing prototyping, making it easy to move from training and benchmarking to visualization and deployment.
🌐 Beyond Offline Benchmarks:
LLMRouter further demonstrates real-user routing on CLI applications and extends model routing to every node in multi-agent systems across five coordination topologies.
✨ From benchmarks → routing algorithms → real-world deployment → multi-agent systems, LLMRouter provides an end-to-end infrastructure for exploring and deploying LLM routing.
Get this paper in your agent:
hf papers read 2608.06867 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 1
Spaces citing this paper 0
No Space linking this paper