RouteMCP

RouteMCP is a small, self-hosted gateway that routes OpenAI-compatible chat requests across model providers. For each request it filters models by hard requirements (tools, JSON, vision, context, budget, latency, privacy), ranks the rest using measured evidence, fails over to the next eligible model after provider or validation failures, and records every decision in SQLite.

Routing improves with use:

• Operational signals are learned online. Latency, success rate, and tool-call success are tracked per model as moving averages from live traffic and blended with the configured priors.

• Cheap models answer when they are confident. With cascade enabled, the gateway first asks cheaper models and keeps an answer whose token log-probabilities clear a threshold, escalating to the routed model otherwise.

• Quality is calibrated from ground truth. routemcp evaluate runs your models on tasks with deterministic validators; routemcp calibrate loads the measured pass rate for each model and task type into the gateway. Routing then sends each task type to the cheapest model that is within the profile's quality tolerance of the best one.

RouteMCP also exposes the router as an authenticated Model Context Protocol server, so MCP hosts can inspect decisions, run routed completions, and list models with their learned statistics.

On a 260-task evaluation with gpt-4.1, gpt-4.1-mini, and gpt-4.1-nano, the quality profile with the confidence cascade matched or beat always using gpt-4.1 at 77–88% lower cost, and came within 1–2 tasks of always using gpt-4.1-mini, the best single model there, at up to 41% lower cost. It did not beat mini's accuracy. See Results.

How it compares

Choose LiteLLM for broad provider coverage, centralized keys, and spend controls. Choose RouteLLM or a commercial router when you want a fine-grained per-prompt learned router. RouteMCP targets the gap between them: a transparent, self-hosted router that learns from your own tasks and enforces hard constraints. Its quality model is per task type, which is coarser than a per-prompt router; the confidence cascade recovers part of the per-request signal by letting a cheap model keep the answers it is sure of. The evaluation below measures both.

Related MCP server: MCP Gateway

Quick start

RouteMCP requires Python 3.9 or newer.

python -m venv .venv
source .venv/bin/activate
pip install -e '.[dev]'
cp models.example.yaml models.yaml
routemcp serve --config models.yaml

Validate configuration without starting the server with routemcp check-config --config models.yaml.

The example configuration uses mock models and works without API keys. Check the server and make a routed request:

curl http://127.0.0.1:8000/healthz

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "auto",
    "messages": [{"role": "user", "content": "Explain consistent hashing"}],
    "route": {"profile": "balanced"}
  }'

The response is OpenAI-compatible and includes a _routemcp object with the trace ID, selected configured model, attempt count, and cost estimate.

Architecture

flowchart LR
    A[Application or AI Agent] -->|OpenAI-compatible request<br/>model = auto| B[RouteMCP API Gateway]
    A2[MCP Host] -->|Streamable HTTP /mcp| B2[RouteMCP MCP Tools]

    B --> C[Authentication]
    B2 --> D
    C --> D[Request Analyzer]

    D --> D1[Estimate tokens]
    D --> D2[Detect complexity]
    D --> D3[Detect tools, JSON, and vision]
    D --> D4[Apply budget and privacy rules]

    D1 --> E[Capability Filter]
    D2 --> E
    D3 --> E
    D4 --> E

    R[(Model Registry<br/>YAML and REST API)] --> E
    H[(Active probes and<br/>passive circuit breaker)] --> E

    E -->|Reject incompatible models| X[Rejection Reasons]
    E -->|Eligible models| F[Model Scoring Engine]

    F --> F1[Quality]
    F --> F2[Cost]
    F --> F3[Latency]
    F --> F4[Reliability]
    F --> F5[Tool success rate]

    F1 --> G[Select Best Model]
    F2 --> G
    F3 --> G
    F4 --> G
    F5 --> G

    G --> P[Provider Adapter]

    P --> P1[OpenAI]
    P --> P2[Anthropic]
    P --> P3[Gemini]
    P --> P4[Ollama]
    P --> P5[vLLM or llama.cpp]
    P --> P6[Generic OpenAI-compatible API]

    P1 --> M[Cloud or Local Model]
    P2 --> M
    P3 --> M
    P4 --> M
    P5 --> M
    P6 --> M

    M --> V[Response Validator]

    V -->|Valid response| T[Record Trace, Cost, and Latency]
    V -->|Invalid output, timeout,<br/>or provider failure| ESC[Escalation Policy]

    ESC -->|Try next eligible model| G
    ESC -->|No models remain| ERR[Return Typed Routing Error]

    T --> OUT[Return OpenAI-compatible Response]
    OUT --> A

    T --> DB[(SQLite Telemetry)]
    DB --> OBS[Metrics and Trace API]

Model registration flow

sequenceDiagram
    actor User
    participant Config as models.yaml / REST API
    participant Registry as Model Registry
    participant Probe as Health Probe
    participant Provider as Model Provider
    participant Router as Routing Engine

    User->>Config: Add model configuration
    Config->>Registry: Validate and register model
    Registry-->>User: Model registered

    User->>Probe: POST /v1/router/models/{id}/probe
    Probe->>Provider: Check provider connection
    Provider-->>Probe: Health and latency
    Probe->>Registry: Store latest health result
    Probe-->>User: Probe result

    User->>Router: Send request with model=auto
    Router->>Registry: Load eligible models
    Registry-->>Router: Models, capabilities, and health
    Router->>Router: Filter and score candidates
    Router->>Provider: Execute using selected model
    Provider-->>Router: Completion
    Router-->>User: OpenAI-compatible response

Configuration

Models are loaded from YAML at startup. Each entry has a RouteMCP id and the provider's own model name. Remote providers should reference an environment variable rather than placing a secret in YAML.

models:
  - id: openai-fast
    provider: openai
    model: gpt-4.1-mini
    api_key_env: OPENAI_API_KEY
    cost:
      input_per_million: 0.40
      output_per_million: 1.60
    capabilities:
      tools: true
      json: true
      vision: true
      context_window: 1048576
      max_output_tokens: 32768

Supported provider kinds are openai, anthropic, gemini, ollama, openai-compatible, and mock. Ollama and generic OpenAI-compatible entries require base_url. Native cloud entries require api_key_env.

Routing profiles are cheapest, fastest, balanced, and quality. Each profile weights quality, cost, latency, and reliability, and also sets a quality tolerance: the largest expected-accuracy gap versus the best eligible model it accepts for a cheaper or faster model (0.30 for cheapest and fastest, 0.10 for balanced, 0.03 for quality). Models outside the tolerance stay eligible as fallbacks. Per-request options can set cost and latency ceilings, require local-only execution, override task metadata, change the tolerance, and cap retry attempts:

{
  "profile": "quality",
  "max_cost_usd": 0.05,
  "max_latency_ms": 5000,
  "max_quality_gap": 0.05,
  "privacy": "standard",
  "max_attempts": 3,
  "cascade": false,
  "cascade_min_logprob": -0.01
}

Confidence cascade

Set "cascade": true to try cheaper models before the routed choice. The ladder is every eligible model cheaper than the routed model that declares capabilities.logprobs: true, cheapest first. Each rung is called with logprobs enabled; an answer is kept when its least likely token has log-probability at or above cascade_min_logprob (default −0.01). A tool call counts as confident when its arguments parse as JSON, because providers do not return log-probabilities for tool calls. Otherwise the router escalates to the next rung and finally to the routed model, called exactly as the client asked. If every stronger model fails, the gateway returns the last low-confidence answer instead of an error.

The response's _routemcp.cascade lists each rung with its confidence and whether it was accepted, and actual_cost_usd and the trace include every call made. Log-probabilities are removed from the response unless the client asked for them. The cascade applies only to non-streaming requests with model: "auto", and only OpenAI and OpenAI-compatible servers that return logprobs (for example vLLM) can be rungs. Escalations make two or more sequential calls, so they add latency.

Learned statistics

The YAML reliability block is a prior. Every attempt updates the model's exponentially weighted latency, success rate, and tool-call success, and calibration stores measured pass rates per task type. Scoring blends live signals with the prior using ROUTEMCP_PRIOR_STRENGTH pseudo-observations (default 10), so a handful of samples nudges the prior and hundreds replace it. Calibrated quality uses a weaker prior (ROUTEMCP_QUALITY_PRIOR_STRENGTH, default 5), because configured quality is a guess and measured pass rates should win quickly; the prior still separates models that tie on a small calibration set. Each candidate in a decision reports the effective values it was scored on and their source.

Learned statistics are stored in the same SQLite database as the registry and traces, so they survive restarts. Deleting a model resets them.

API

• POST /v1/chat/completions — route and execute a chat request

• POST /v1/router/decide — inspect a decision without calling a model

• POST /v1/router/workflows — route and execute an ordered set of agent steps

• GET /v1/router/models — list configured models

• POST /v1/router/models — register and persist a model

• PUT /v1/router/models/{id} — replace a registered model

• DELETE /v1/router/models/{id} — remove a registered model

• POST /v1/router/models/{id}/probe — probe provider connectivity

• GET /v1/router/metrics — aggregate request metrics

• GET /v1/router/traces/{id} — inspect a routing trace

• GET /healthz — liveness and model count

Set ROUTEMCP_API_KEY to require Authorization: Bearer ... on /v1 routes. Set ROUTEMCP_DB to choose the SQLite path. Request prompts are not stored by default; set ROUTEMCP_STORE_PROMPTS=true to retain them in traces.

A model's latest explicit probe result and learned statistics are included in the model list. Provider failures are also tracked passively; repeated failures or a failed probe open a circuit for the cooldown, after which the model is retried half-open, and a successful call closes it. API model changes are persisted in the SQLite database and survive restarts.

Metrics include request cost, success and escalation rates, model distribution, latency, and upper_bound_cost_avoidance_usd. The latter compares cost with the most expensive eligible model and is explicitly a diagnostic upper bound—not a quality-adjusted savings claim. Use the evaluation report for defensible savings.

stream: true uses OpenAI-compatible SSE passthrough for OpenAI, Gemini, Ollama, vLLM, llama.cpp, and generic OpenAI-compatible endpoints. RouteMCP can fail over when an upstream rejects the request before streaming begins. Native Anthropic streams are translated into the same OpenAI-compatible chunks, including incremental tool-call arguments. Once response bytes have reached the caller, a mid-stream failure cannot safely switch models; it is recorded and contributes to the circuit breaker.

MCP server

The MCP server requires Python 3.11 or newer and the optional dependency:

pip install -e '.[mcp]'
routemcp mcp --config models.yaml --host 127.0.0.1 --port 8001

Connect an MCP client to http://127.0.0.1:8001/mcp. It exposes:

• route_decide — inspect the selected model, candidate scores, and rejections

• route_complete — execute a routed completion with validation, fallback, and the optional cascade

• list_routed_models — inspect registered capabilities, prices, health, and learned statistics

The MCP server uses the current Streamable HTTP transport, not the superseded standalone SSE transport. Set ROUTEMCP_MCP_API_KEY (or ROUTEMCP_API_KEY) to require Authorization: Bearer ... on every MCP request. The bundled Compose profile also binds MCP to localhost; use TLS in front of it before exposing it on a public network.

With Docker Compose, start both interfaces with:

docker compose --profile mcp up --build -d

Routing evaluation

evaluation/tasks.jsonl holds 260 tasks across classification, extraction, transformation, math, reasoning, code-output prediction, and tool calling, with easy and hard variants of each. evaluation/build_dataset.py generates it from a seed, and every expected answer is computed rather than hand-labelled: math is evaluated exactly and code outputs come from running the snippet. Validators are deterministic (exact, normalized, contains, regex, number, choice, json_fields, json_match, tool_call).

routemcp evaluate \
  --config evaluation/openai.yaml \
  --dataset evaluation/tasks.jsonl \
  --strongest-model gpt-4.1 \
  --cheapest-model gpt-4.1-nano \
  --calibration-fraction 0.5 \
  --profiles balanced,quality,cheapest \
  --seed 0 \
  --output evaluation-report.json

The run splits tasks 50/50 per task type. Every model answers the calibration half, and those pass rates become the router's quality evidence. On the held-out half, each model runs as a fixed baseline and RouteMCP runs once per profile. The report includes success rate, cost, cost per success, latency, quality retained and cost saved versus the strongest model, per-task-type results, which model the router chose for each task type, and an oracle (the cheapest model that passed each task) as an upper bound. Load the results into a running gateway with routemcp calibrate --report evaluation-report.json --db routemcp.db.

evaluation/run_openai_eval.sh runs the published configuration, reading OPENAI_API_KEY from the environment or a gitignored .env, and prints the tables with evaluation/summarize.py. Set EVAL_SEED to choose the split.

routemcp replay --report <report.json> --config <models.yaml> re-routes a finished run offline with the current scoring code: calibration comes from the report's calibration rows, and each routed choice is scored with the recorded result of that model on that task. Requests are sent at temperature 0, so this measures a scoring change without new API calls. On the first runs of this evaluation, replaying the unchanged router reproduced every quality and cheapest choice and 124 of 130 balanced choices (the rest differ because live runs also learn latency). Replay skips +cascade policies, which need live log-probabilities.

Results: gpt-4.1, gpt-4.1-mini, gpt-4.1-nano

Two splits of the 260-task dataset (seeds 0 and 1), 130 calibration and 130 held-out tasks each, list prices from October 2026. Together the two runs made about 3,050 API calls and cost $0.19. Reports, including every model's answer, are in evaluation/results/.

What this shows:

• Against always using gpt-4.1, quality with the cascade matches or beats it at 77–88% lower cost.

• Against always using gpt-4.1-mini, the best single model here, quality with the cascade is within 1–2 tasks (0.8–1.5 points, inside run-to-run noise) at 41% lower cost when routing picks mini as the target (seed 1). On seed 0, calibration sent math and transformation to gpt-4.1, so escalations went to the more expensive model and the cascade cost 15% more than mini. The cascade cuts cost at roughly mini's accuracy; it does not raise accuracy above mini.

• The cascade kept a cheaper model's answer on 58% of quality requests. In a separate run of all 260 tasks with log-probabilities, nano was right on every one of the 126 answers whose least likely token had log-probability of at least −0.01, the default threshold, which was chosen on the calibration halves (evaluation/collect_logprobs.py, data in evaluation/results/openai-logprobs.json).

• balanced already routes most traffic to nano, so the cascade changes little there.

• The cascade adds latency when it escalates: two sequential calls instead of one, adding about 140–200 ms to average latency in these runs.

• gpt-4.1 is not better than mini on this dataset because the prompts ask for the final answer only, with no room to show work.

• The oracle is still 2–8 points above every policy. Results move by a few points between splits (one point is 1.3 tasks), so differences under about 3 points are noise.

Tuning that the replay and these runs informed:

• Weaker quality prior. With 10 pseudo-trials, configured priors outweighed 20/20 calibration results, so quality kept paying for gpt-4.1 on classification, extraction, and tool calling, where every model scored 100%. At 5, the same accuracy cost 59–76% less.

• Dropped: a pessimistic quality bound for cheaper models bought under a point of accuracy at 2–4× the cost.

• Dropped: prompt length as a difficulty signal did not help.

• Dropped: agreement between nano and mini as the cascade check. Agreement was 96% accurate, but always paying for both models cost more than mini alone, and escalating disagreements to gpt-4.1 did not beat mini.

Two caveats apply. The dataset and the analyzer's task-type rules were written by the same author, so the report includes analyzer_task_type_agreement to show how the analyzer labelled each task. And the split holds out task instances, not task types, so it measures routing on familiar kinds of work.

Gateway benchmark

Measure the same upstream model directly and through RouteMCP:

python benchmarks/benchmark_gateway.py \
  --gateway-url http://127.0.0.1:8000/v1/chat/completions \
  --direct-url https://provider.example/v1/chat/completions \
  --gateway-model configured-model-id \
  --direct-model provider-model-id \
  --requests 200 \
  --concurrency 20 \
  --output benchmark-report.json

The report includes errors, throughput, p50/p95/p99 latency, and added gateway latency. Run it near both endpoints and repeat runs before publishing numbers.

The checked-in synthetic localhost smoke run used five alternating rounds of 200 requests at concurrency 20. It recorded zero errors, median throughput of 382.03 requests/second through RouteMCP versus 389.84 direct, and mean added latency of 0.75 ms. Its p50 added latency was 6.75 ms; tail deltas were too noisy to report. See benchmarks/results/local-synthetic.json. This validates the benchmark and gateway path only—it is not a real-model performance claim or resume result.

Container end-to-end result

The production image was also exercised through its public HTTP and MCP interfaces. The local deterministic run completed 1,100 load-test requests with zero errors. At concurrency 25 it measured 650.48 completion requests/second, 338.34 routing decisions/second, and 357.0 streaming requests/second. Completion latency was 25.56 ms p50 and 91.03 ms p95.

The same run verified model registration across complete container replacement, a two-step workflow, stream termination, health probing, three successful typed failovers, and circuit opening after the third provider failure. Median local failover latency was 1.50 ms; the first request after the circuit opened took 1.46 ms and skipped the failed provider. A real Streamable HTTP MCP client negotiated protocol 2026-07-28, discovered all three tools, and successfully called route_decide.

Full machine-readable results and limitations are in benchmarks/results/e2e-container.json. These are gateway correctness and local performance numbers—not model-quality evidence.

Reproduce the run with Docker Compose and a Python 3.11+ environment that has pip install -e '.[mcp]':

benchmarks/run_e2e.sh e2e-results

The script generates random API keys for both interfaces unless ROUTEMCP_API_KEY and ROUTEMCP_MCP_API_KEY are set. It builds the image, starts the gateway and MCP containers on a fresh volume, runs benchmarks/e2e_gateway.py (load, workflow, streaming, failover, circuit breaker, MCP tool calls, and rejection of unauthenticated MCP requests), replaces both containers, verifies that the API-registered model survived, and tears everything down. It exits non-zero if any check fails. Set PYTHON, E2E_REQUESTS, or E2E_CONCURRENCY to override defaults. Ports 8000 and 8001 must be free.

Production status

The current release is appropriate for a single-node gateway: durable model registration, SQLite WAL, API authentication, passive circuit breaking, typed fallback, streaming passthrough, health probes, Docker, and CI are included. Learned statistics are per node. Horizontal multi-instance deployment still requires PostgreSQL or another shared registry, telemetry, and statistics store. Distributed circuit state, rate limiting, and automated scheduled probes remain explicit production gaps.

Development

pip install -e '.[dev,mcp]'
pytest
ruff check .

The MCP tests are skipped on Python versions older than 3.11.

Regenerate the evaluation dataset with python evaluation/build_dataset.py. It is seeded, and a test checks that the committed evaluation/tasks.jsonl matches the generator and that every expected answer passes its own validator.