AI Research Portal

ZeroRouter is a novel framework for cost-efficient, zero-shot Large Language Model (LLM) routing that addresses the "model lock-in" problem in existing systems. Its core idea is to decouple query characteristics from model capabilities by introducing a universal, cross-task latent space. This allows for the zero-shot onboarding of new models without extensive retraining. ZeroRouter integrates a context-aware predictor to map queries to this latent space and a dual-mode optimizer to balance accuracy, cost, and latency. The framework consistently outperforms baselines, achieving higher accuracy at lower cost and latency, particularly on unseen datasets.

The paper identifies several bottlenecks in current LLM routing frameworks:

Model Lock-in: Existing systems are tied to a fixed set of models, making it difficult to integrate new LLMs without complete and costly architectural overhauls and extensive retraining with new preference data.
Scalability Limitations: Current direct routing systems (e.g., HybridLLM, RouteLLM) use static model-task mappings or cascading workflows, which require exhaustive query evaluations across predefined models and incur quadratic evaluation costs as the model pool grows.

Introduction of a universal, cross-task latent space, calibrated by an information-theoretic selection strategy, to decouple prompt characteristics from model capabilities.
Development of a context-aware latent space predictor that maps a query's textual properties directly to its latent parameters.
Creation of a unified policy-driven routing engine that translates high-level user priorities on accuracy, cost, and latency into optimal query assignments.
Demonstration of zero-shot onboarding of new models with minimal data, breaking the "model lock-in" paradigm.

ZeroRouter's framework consists of three core modules: Latent Parameter Calibration, Lightweight Profiling, and Policy-Driven Routing.

Universal Latent Space for Lightweight Profiling This module establishes a model-agnostic representation of query difficulty and model ability.

Model Inference Accuracy Estimation: The interaction between a model u and a prompt i is modeled using a Probabilistic Model, specifically a multidimensional 2-Parameter Logistic (2PL) IRT model. The probability of a correct response Xui = 1 is given by: P(Xui = 1|θu, ai, bi) = 1 / (1+e-ai (θu-bi)) where θu is the model's ability vector, and ai, bi are the prompt's discrimination and difficulty vectors in a D-dimensional latent space.
Information-Theoretic Anchor Selection: To efficiently estimate model capabilities with a minimal yet highly informative anchor set A, the task is formalized as an optimal experimental design problem. The objective is to maximize the log-determinant of the Fisher Information Matrix, simplified to rely on prompt discrimination vectors ai: A* = arg max log det (Σ ai aiT) ACP, |A|=N A greedy forward selection algorithm is used to construct set A.
Lightweight Profiling for New Models: Once the anchor set A is established, the ability θunew of a new model is estimated by minimizing the binary cross-entropy (BCE) loss between ground-truth scores and predicted probabilities for anchor prompts.

Model Inference Cost Estimation

Total Inference Cost: For API-based LLM services, the total inference cost for routing a query q to a model u is a function of its input and output token lengths: Cuq = λinlin + λoutlout where λin and λout are per-token API prices, and lin and lout are input and output token lengths.
Input Token Estimation: The input token length lin is deterministically obtained using the model-specific tokenizer: lin = |Tu(q)|
Output Token Estimation: A novel method leverages the universal latent space to estimate output token length lout. Task-aware difficulty sq is derived from predicted latent IRT parameter vectors: sq = aqbq During a one-time calibration, complexity scores are discretized into K bins. For each model u and bin Bk, a mean output token length lout is pre-computed and stored. For an unseen query q', its latent vectors are predicted, its complexity score sq' is computed, and the corresponding bin Bk is identified to retrieve the pre-calibrated average length.

Model Inference Latency Estimation Latency for each model u is modeled via two pre-calibrated parameters: Time to First Token (TTTFT) and Time per Output Token (POT). The final latency for a query q is predicted by: Tuq = TTTFT + loutPOT

Context-Aware Latent Space Coordinate Prediction A lightweight unified predictor bridges a query's textual properties with its multi-dimensional IRT parameter vectors (discrimination vector a and difficulty vector b).

Hybrid Feature Representation: Queries are encoded into Semantic Embedding (ese) using a fine-tuned DistilBERT model and Structural Features (est) using a function that computes 11 linguistic metrics.
Unified Prediction Model: A multi-task learning model with a shared backbone learns a high-level representation (hshared) of the query. This is followed by two independent prediction heads:
- Difficulty Head (Residual Prediction): Predicts a residual vector Δbq, which is added to a data-driven mean b to construct the final predicted difficulty vector bq: bq = b + Δbq = b + fdiff(hshared)
- Discrimination Head (Multi-Head Experts): Partitions the D dimensions into C distinct clusters. For each cluster, a dedicated expert head (fdisc,c, an MLP) predicts the subset of dimensions within its assigned group. The final discrimination vector aq is constructed by concatenating these outputs: aq = ⨁c=1C aq,c = ⨁c=1C fdisc,c(hshared)

LLM Routing Strategies

Multi-Objective Optimization: The routing task is formulated as an Integer Linear Programming (ILP) problem that maximizes a weighted utility function, unifying accuracy, cost, and latency. For a set of queries Q and models M, the goal is to determine the optimal assignment Xuq ∈ {0,1} guided by user-specified weights (wp, wc, wt) that sum to 1: maximize Σu∈M Σq∈Q (wpPuq - wcCuq - wtTuq) · Xuq subject to the constraint that each query is assigned to exactly one model (Σu∈M Xuq = 1, ∀q∈Q) and optional global constraints on total cost, latency, and average accuracy.

Latent Space Dimension: D = 20
IRT Model Training:
- Optimizer: Adam
- Epochs: 6,000
- Initial Learning Rate: 0.1, decayed exponentially by a factor of 0.99 per 100 epochs.
Latent Parameter Predictor (DistilBERT):
- Model: DistilBERT-base-uncased (66M parameters)
- Fine-tuning Epochs: 40
- Batch Size: 32
- Learning Rate: 3e-5 (constant)
Hardware: Single NVIDIA A800 GPU (80GB)
Anchor Set Size: N (not explicitly stated, but implied to be small, e.g., 200 queries for profiling new models).
Model Pool: A core set of 10 LLMs (1B to 235B parameters) and a second cohort of 50 distinct LLMs for validating zero-shot integration.

Datasets:
- In-Distribution (ID) Benchmarks: IFEval, BBH, MATH, GPQA, MuSR, MMLU-PRO.
- Out-of-Distribution (OOD) Benchmarks: ARC-C, TruthfulQA, HumanEval.
LLM Candidates:
- Smaller-scale LLMs (<20B): Gemma-3 (1B), Gemma-3n-E4B (7.8B), DeepSeek-R1-0528-Qwen3 (8B), Llama-3.1 Instruct (8B), Phi-4 (14.7B).
- Larger-Scale LLMs (25B~250B): Gemma-3 (27B), Qwen3 (32B), Mixtral-8x7B Instruct (56B), DeepSeek-R1-Distill-Llama (70B), Qwen3-235B-A22B Instruct-v0.1 (235B).
Metrics: Unified total reward (Reward = Σq∈Qtest (wpPuq - wcCuq - wtTu*q)), Max-Acc, Min-Cost, Min-Lat.
Baselines: Random Selection, RouteLLM, FORC, GraphRouter, Model-SAT.
Policy-Driven Scenarios:
- Accuracy-First (Max-Acc): wp=0.8, wc=0.1, wt=0.1
- Cost-First (Min-Cost): wp=0.1, wc=0.8, wt=0.1
- Latency-First (Min-Lat): wp=0.1, wc=0.1, wt=0.8
- Balanced Strategy: wp=0.5, wc=0.3, wt=0.2
Numeric Results (ZeroRouter vs. Baselines):
- Smaller-scale LLMs (<20B) - LLM Selection (5 models):
  - Max-Acc: 0.45 (ZeroRouter) vs. 0.43 (Model-SAT) (ID); 0.59 (ZeroRouter) vs. 0.57 (Model-SAT) (OOD).
  - Min-Cost: -0.32 (ZeroRouter) vs. -0.61 (Model-SAT) (ID); -0.24 (ZeroRouter) vs. -0.59 (Model-SAT) (OOD).
  - Min-Lat: -0.27 (ZeroRouter) vs. -0.64 (Model-SAT) (ID); -0.23 (ZeroRouter) vs. -0.64 (Model-SAT) (OOD).
  - Mean: 0.00 (ZeroRouter) vs. -0.25 (Model-SAT).
- Larger-Scale LLMs (25B~250B) - LLM Selection (5 models):
  - Max-Acc: 0.52 (ZeroRouter) vs. 0.50 (GraphRouter) (ID); 0.68 (ZeroRouter) vs. 0.62 (Model-SAT) (OOD).
  - Min-Cost: -0.22 (ZeroRouter) vs. -0.27 (GraphRouter) (ID); -0.17 (ZeroRouter) vs. -0.37 (GraphRouter) (OOD).
  - Min-Lat: -0.28 (ZeroRouter) vs. -0.57 (Model-SAT) (ID); -0.25 (ZeroRouter) vs. -0.46 (Model-SAT) (OOD).
  - Mean: 0.05 (ZeroRouter) vs. -0.07 (GraphRouter).
Efficient Onboarding with Minimal Data (Table 2): ZeroRouter with D-optimality achieved Max-Acc 0.39, Min-Cost -0.34, Min-Lat -0.37, and Mean -0.11, outperforming baselines like Model-SAT (Max-Acc 0.32, Min-Cost -0.46, Min-Lat -0.44, Mean -0.19) when profiling with 200 queries.
Runtime/Training Cost: Not stated in the paper, beyond the hardware used for training.

No explicit paper-reported limitations were found.

Method evidence: "Our approach is founded on a universal latent space, a model-agnostic representation of query difficulty that fundamentally decouples the characteriza- tion of a query from the profiling of a model. This allows for zero-shot onboarding of new models without full-scale retrain- ing." (section: Abstract)
Result evidence: "For both smaller and larger models, it not only achieves the high- est Max-Acc (e.g., 0.68 vs the strongest baseline's 0.62 for large models) but also dramatically outperforms all rivals on Min-Cost -0.17 and Min-Lat -0.25." (section: Zero-Shot Generalization on OOD Datasets)
Limitation evidence: Not stated in the paper.

Breaking Model Lock-in: Cost-Efficient Zero-Shot LLM Routing via a Universal Latent Space

Summary

Problem

Key Contributions

Method

Technical Details

Results

Weaknesses / Limitations

Evidence

Additional Figures