ypollak2/llm-router

Why People Install This

AI coding tools send too many prompts to premium models by default.

That means:

• You waste paid tokens on simple questions

• You burn through Claude, Gemini, or OpenAI quota faster than necessary

• You stop working when one provider is rate-limited or down

llm-router sits between your coding tool and your model providers. It classifies each prompt, tries the cheapest capable model first, and falls back automatically when needed.

You keep the same workflow. The router changes the model choice underneath.

What You Get

• Route trivial prompts to free or cheap models first

• Keep premium models for the prompts that actually need them

• Fall back across providers automatically

• Track usage and estimated savings locally

• Run everything on your own machine

Quick Start

1. Install

pip install llm-routing
llm-router install

Package name: llm-routing on PyPI. CLI command: llm-router.

2. Add providers (optional)

export OPENAI_API_KEY="sk-..."      # GPT-4o, o3
export GEMINI_API_KEY="AIza..."     # Gemini Flash/Pro (free tier available)
export OLLAMA_BASE_URL="http://localhost:11434"  # Local models (free)

Works with zero API keys on Claude Code Pro/Max subscriptions — routing uses MCP tools that call external models only when beneficial.

3. Verify

llm-router health            # Check provider connectivity

If you already use Claude Code, Codex, or Gemini CLI, keep your existing workflow and let llm-router choose models underneath it.

Example Routing

The exact chain depends on your configured providers, budget profile, and routing policy.

Works With

• Full auto-routing means hooks intercept prompts and route automatically with no workflow change.

• Manual MCP tools means routing is available on demand through tools such as llm_query.

llm-router install                    # Claude Code (default)
llm-router install --host codex       # Codex CLI
llm-router install --host gemini-cli  # Gemini CLI
llm-router install --host vscode      # VS Code
llm-router install --host cursor      # Cursor

See docs/HOST_SUPPORT_MATRIX.md for full details on each host.

Protect Claude Code 5-hour quota

For a strict boundary that never automatically falls through to native Claude, configure:

# ~/.llm-router/routing.yaml
enforce: smart
mode: zero_claude

In zero_claude mode, prompts either complete through direct external execution or are blocked before Claude Code invokes its model. Prefix a prompt with claude: when you intentionally want a native Claude turn.

How It Works

User prompt
    │
    ▼
┌──────────────────────┐
│ Complexity Classifier │  ← Heuristic (free, instant) or Ollama/Flash ($0.0001)
└──────────┬───────────┘
           │
           ▼
┌──────────────────────┐
│  Free-First Router   │  ← Tries cheapest model first, walks up the chain
│                      │
│  Ollama (free)       │
│  → Codex (prepaid)   │
│  → Gemini Flash      │
│  → GPT-4o / Claude   │
└──────────┬───────────┘
           │
           ▼
┌──────────────────────┐
│  Guards (parallel)   │  ← Circuit breaker, budget pressure, quality check
└──────────┬───────────┘
           │
           ▼
      Response + cost logged to local SQLite

Classification is free for many tasks (regex heuristics catch ~70%) or near-free for ambiguous prompts when using local Ollama or Gemini Flash.

What You Can Do

Providers

Routing chains are built from your configured providers. You only need one.

Text LLM Providers

Media Providers

See docs/PROVIDERS.md for setup instructions and model recommendations.

Routing Policies

Control how aggressively the router offloads to cheap models.

export LLM_ROUTER_POLICY=aggressive     # Or: balanced, conservative
export LLM_ROUTER_ENFORCE=smart          # smart | hard | soft | off
export LLM_ROUTER_PROFILE=balanced       # budget | balanced | premium

LLM_ROUTER_ENFORCE controls how strictly the auto-route hook blocks direct model use:

• smart — route when confident, pass through when uncertain

• hard — always route, block unrouted tool calls

• soft — suggest routing, never block

• off — disable hook enforcement

MCP Tools (60)

llm-router exposes 60 MCP tools organized by function:

Slim mode (LLM_ROUTER_SLIM=routing or core) reduces registered tools to save context tokens in constrained environments.

Full Tool Reference

Savings: How It Works

Savings are calculated by comparing actual spend against a baseline of routing every task to Claude Sonnet/Opus.

Methodology:

1. Each routed task logs: model used, tokens consumed, estimated cost

2. A baseline cost is computed as if the same tokens were processed by the most expensive model in the chain

3. Savings = (baseline - actual) / baseline

Assumptions and limitations:

• Baseline assumes you would have used Opus/Sonnet for everything (worst case)

• Token estimates use len(text) / 4 approximation, not exact tokenizer counts

• Cost data comes from LiteLLM's pricing tables (may lag provider price changes)

• Savings vary significantly by workload — code-heavy sessions route more to cheap models

• The router itself adds small overhead (classification costs ~$0.0001 per ambiguous task)

Observed range: 35–80% savings depending on policy and task mix. The "87%" figure in some docs represents a single-user peak over a specific development period, not a guaranteed outcome.

Trust, Privacy, and Local-First Design

llm-router runs entirely on your machine. There is no hosted proxy, no telemetry, no account required.

What we do:

• Scrub API keys from structured logs

• Detect hook deadlocks before installation

• Store all data locally in ~/.llm-router/

• Respect provider rate limits and TOS

What you should know:

• Prompts are sent to whichever provider the router selects — review your provider's privacy policy

• Usage logs (SQLite) are not encrypted at rest — use full-disk encryption if needed

• The router cannot prevent model jailbreaks or prompt injection at the provider level

See SECURITY.md for responsible disclosure policy and docs/SECURITY_DESIGN.md for the full threat model.

Configuration

Minimal setup — only configure what you have:

# Provider keys (set any combination)
export OPENAI_API_KEY="sk-proj-..."
export GEMINI_API_KEY="AIza..."
export OLLAMA_BASE_URL="http://localhost:11434"
export OLLAMA_BUDGET_MODELS="gemma4:latest,qwen3.5:latest"

# Routing behavior
export LLM_ROUTER_PROFILE="balanced"       # budget | balanced | premium
export LLM_ROUTER_POLICY="balanced"        # aggressive | balanced | conservative
export LLM_ROUTER_ENFORCE="smart"          # smart | hard | soft | off

For teams or environments where .env is restricted:

# User-level config (no project .env needed)
mkdir -p ~/.llm-router && chmod 700 ~/.llm-router
cat > ~/.llm-router/config.yaml << 'EOF'
openai_api_key: "sk-proj-..."
gemini_api_key: "AIza..."
ollama_base_url: "http://localhost:11434"
llm_router_profile: "balanced"
EOF
chmod 600 ~/.llm-router/config.yaml

Documentation

Contributing

Contributions welcome. See CONTRIBUTING.md for full guidelines.

git clone https://github.com/ypollak2/llm-router.git
cd llm-router
uv sync --extra dev
uv run pytest tests/ -q         # Run tests (1900+)
uv run ruff check src/ tests/   # Lint

Package Names