llm-sidecar

A local sidecar that gives every tool on your machine grounded, cited, routed AI — and never asks you which model to use.

One process. It picks a working model, searches the web, reads pages, checks facts, and tells you what everything cost. Works with no API key at all if you have Ollama; works better with one.

$ llm-sidecar answer "What is Iran's population?"
As of mid-2026, Iran's population is estimated at 93,168,497 (Worldometer,
based on UN 2024 Revision). The 2025 estimate from Wikipedia is 92,417,681.

Caveat: Sources differ by year and methodology: Worldometer (2026 mid-year)
gives 93.17 million; Wikipedia (2025 est.) gives 92.42 million.

Sources:
  · https://en.wikipedia.org/wiki/Demographics_of_Iran
  · https://www.worldometers.info/world-population/iran-population/

That came from the live web, not training data, and it says where it got it — including that the sources disagree. Ask something the web can't settle and it tells you, instead of guessing.

It is not instant: a grounded answer means a search, two or three page fetches and a model call, so expect tens of seconds — more on free models, which are slow and sometimes have to be rotated past.

Status: early (0.6.0). The API may still move. Extracted from Agora, where the routing and verification were originally built and proven.

Install

brew install pipx
pipx ensurepath
exec $SHELL
pipx install llm-sidecar

On Linux use apt install pipx instead of brew. That is the whole install — no extras to remember, every door works.

The third line matters: pipx ensurepath edits your shell config, and cannot change a shell that is already running. exec $SHELL reloads it. Without that you get command not found: llm-sidecar even though the install succeeded.

Add an OpenRouter key if you want free cloud models. Without one it runs entirely on Ollama.

llm-sidecar config key sk-or-your-key --save
llm-sidecar status

Neither is broken — both are modern Python on macOS and Linux behaving as designed, and they are the reason this page says pipx.

pip is not a command on most Macs. Homebrew installs pip3, not pip. PyPI's install line is boilerplate it prints for every package; it doesn't know what your Python looks like.

pip3 install then refuses anyway, with error: externally-managed-environment. Since PEP 668, distributions mark their Python as off-limits to global installs, because system tooling depends on it and one bad upgrade breaks the OS.

pipx exists for exactly this. It creates a private virtualenv per tool and puts just the command on your PATH, so nothing global changes. It is the standard way to install a Python application, as opposed to a library.

For a library in your own project, pip is still right — inside that project's virtualenv:

python3 -m venv .venv
./.venv/bin/pip install llm-sidecar

Related MCP server: pyaireader

Run it as a service

A sidecar you have to remember to start isn't one:

llm-sidecar service install     # launchd on macOS, systemd on Linux

Starts at login, restarts if it dies, logs to ~/Library/Logs/llm-sidecar.log. service status and service uninstall do what they say. Or just llm-sidecar serve in a terminal when you want it.

Now every tool on the machine can reach it:

export OPENAI_BASE_URL=http://localhost:4001/v1    # any OpenAI-compatible tool
export OPENAI_API_KEY=unused                       # the format demands the field

...and the dashboard is at http://localhost:4001.

From source

git clone https://github.com/awaistechnologist/llm-sidecar
cd llm-sidecar && ./install.sh && ./run.sh          # Windows: install.bat, run.bat

install.sh reports what your machine can actually reach — Ollama and how many models, whether a key is set, whether Docker is around for SearXNG — and prints the MCP config with your real paths filled in.

Note that a clone under ~/Documents or ~/Desktop cannot be run as a launchd service: macOS keeps those behind a privacy prompt that background processes never receive. service install detects this and tells you rather than leaving a crash loop behind.

What it does

Why it exists

Most "LLM router" projects answer which provider should this call go to. That's plumbing. The point here is the layer above it: capabilities that are hard to build well and that every tool re-implements badly — grounded search, cited verification — sharing one routing, budget and cost substrate.

The routing does earn its keep in one specific way: a catalogue entry is not a working model. Free tiers throttle, checkpoints get retired, endpoints start returning empty completions. So before handing back a model, the picker spends one tiny call proving it answers right now.

Four ways in

Same core, four doors, because the consumers can't use each other's interface.

Library

from llm_sidecar import Sidecar
sc = Sidecar()

a = sc.answer("What shipped in Python 3.14?")
if a.grounded:                          # False = the sources didn't settle it
    print(a.text, a.sources)

sc.verify(["The Eiffel Tower is in Berlin"])      # → contradicted, cited
sc.fact_check(article)                             # extract claims, verify each
sc.summarise(text, style="bullets", focus="security")
sc.classify(tickets, labels=["bug", "feature", "question"])
sc.extract(invoice, {"total": "amount with currency", "due_date": "ISO date"})

sc.complete("Explain CRDTs")                       # routes itself
sc.complete(messages=[...])                        # real multi-turn
sc.complete_many([p1, p2, p3])                     # concurrent, ordered
await sc.acomplete("...")

# Semantic similarity — vectors, not substring matching
sc.rank("senior python backend engineer", [cv_a, cv_b, cv_c])   # ordered, scored
sc.similarity("a golden retriever", "a puppy")                   # 0.86
sc.embed(["one", "two"])                                         # raw vectors

# Guaranteed-shape output, enforced by the provider rather than parsed and hoped
sc.complete("Rate this candidate", schema={
    "type": "object",
    "properties": {"score": {"type": "integer"}, "reason": {"type": "string"}},
    "required": ["score", "reason"],
})

sc.local_models()   # your Ollama models, scored against this machine
sc.usage(days=30)   # what you spent
sc.status()         # everything at once

HTTP daemon — for any tool, in any language

llm-sidecar serve
export OPENAI_BASE_URL=http://localhost:4001/v1
export OPENAI_API_KEY=unused        # the format demands the field

It speaks the chat-completions format every provider copied from OpenAI. That format is a de facto standard, not a vendor tie — Ollama and OpenRouter accept the identical request shape, which is why routing between them is a URL swap. The variable is named after OpenAI because the openai SDK reads it; other tools call the same setting --openai-api-base, apiBase, or "Base URL".

The model field is a request, not an instruction. A tool that hardcodes gpt-4o gets a verified working model and never finds out:

$ curl -s localhost:4001/v1/chat/completions -d '{"model":"gpt-4o", ...}'
{"model": "nvidia/nemotron-3-ultra-550b-a55b:free", ...,
 "x_sidecar": {"cost_usd": 0.0, "local": false, "cached": false}}

Beyond the standard surface: /v1/answer, /v1/verify, /ops/* for each capability, /status, /usage, /resolve-preview, /config/*. Swagger at /docs.

MCP — for agents

{"mcpServers": {"llm-sidecar": {
  "command": "/path/to/venv/bin/python", "args": ["-m", "llm_sidecar.mcp_server"]}}}

answer_question · search_web · read_url · verify_claims · extract_claims · fact_check_document · summarise · classify · extract_fields · delegate · usage_report · sidecar_status

Note what's not there: a general "call an LLM" tool. An MCP client is already a model, so exposing inference to it is close to a no-op. What it can't do for itself is fetch live evidence and grade claims against it. delegate is the exception — it's for offloading bulk work to something cheap.

CLI

llm-sidecar answer "who currently runs the ECB?"   # non-zero exit if ungrounded
llm-sidecar verify "the Great Wall is visible from space"
llm-sidecar sum report.md --style bullets
llm-sidecar models                                  # local models vs your RAM
llm-sidecar usage --days 30
llm-sidecar searxng up                              # better search, one command

How a model gets chosen

Two questions, always. Tier = how capable. Budget = what it may cost. They are independent.

What each capability asks for

  verify · fact_check · summarise · classify        ┐
  extract · extract_claims · delegate               ├──► tier "fast"
                                                    ┘    bulk work, cheap

  answer · chat · complete()                        ───► tier "balanced"

  budget is always your configured default unless you pass one.

There is no separate "chat model". One picker serves everything, so changing a key or a budget changes every capability at once.

The resolution order

Four checks, first match wins.

  ① model="ollama/qwen2.5:32b" given?   ──yes──►  use it. never rotated.
                    │ no
  ② is this tier LOCKED to a model?     ──yes──►  use it. budget ignored.
                    │ no
  ③ resolved this tier+budget <15m ago? ──yes──►  reuse it.
                    │ no
  ④ resolve ▼

     budget picks the candidate pool
     ┌──────────────────────────────────────────────┐
     │ free + key     free cloud models, Ollama next│
     │ free + no key  Ollama only                   │
     │ cheap          paid, under $1 per M tokens   │
     │ best           paid, over $5 per M tokens    │
     └──────────────────────────────────────────────┘
                    │
          probe 3 at once: "Reply OK"
                    │
        ┌───────────┴────────────┐
   one answers               all 3 fail
        │                         │
  use it, cache 15m       probe the next 3 …
                                  │
                        nothing left → NoWorkingModel

Probing concurrently changes only how fast a model is found — the highest-priority success still wins, not whichever replied first. If a model passes the probe and then fails the real call, it's marked dead for the process and the next request routes elsewhere.

Which free cloud model, specifically? A curated list first, then the rest of the pool by context length. Curated entries missing from the live catalogue are skipped silently, which happens constantly — of nine curated free picks, one survived to today. So in practice: the survivors, then the roomiest.

When free OpenRouter models are used

Exactly one combination reaches them. Tier is irrelevant to free-vs-paid.

With a key and budget=free, the three tiers get three different free models — pushing parallel work through one free endpoint is how you collect 429s.

Tiers on a laptop

With local models, tier maps to size: fast takes the smallest installed model, balanced and powerful the largest that fits. Those last two land on the same model deliberately — locally your ceiling is memory, and two names for one model would be pretending otherwise.

This matters more than it sounds, because verify, summarise, classify and extract all ask for the fast tier. Bulk work should not be running on your biggest model.

Models are yours to install. llm-sidecar never downloads weights. It reads what Ollama has and scores it against your actual memory:

llm-sidecar models              # what you have, and whether it fits
llm-sidecar models --suggest    # what to pull, given your RAM

models also shows which are loaded in memory right now. All installed models are usable — "loaded" just means it answers immediately instead of paying a cold load first, and Ollama unloads after about five idle minutes.

Locking a tier

Normally a tier chooses its own model and re-checks every 15 minutes. Locking overrides that: always this model, no search, no probe, no budget — a locked tier stays put even if you ask for best.

LLM_SIDECAR_MODEL_FAST=ollama/gemma3:27b          # env

dashboard → Local models → click "fast" on a row
POST /config/tier {"tier": "fast", "model": "..."}

Worth it for reproducibility and no probe latency. The cost: a locked tier never rotates away, so if that model starts failing, your calls fail with it.

auto is not a mode

model can only carry one axis:

auto is an unrecognised string, and every unrecognised string means "use both defaults" — which is exactly what lets a tool hardcoding gpt-4o work. Because one field can't say "powerful and best", the daemon also takes a separate budget field, and the dashboard has two selectors.

Not sure what you'd get? GET /resolve-preview?tier=fast&budget=free tells you — current state, the candidate order, and why — without probing anything.

What answered, and what it cost

Retrieval: who does the searching

SearXNG

llm-sidecar searxng up

Writes a compose file and settings to ~/.config/llm-sidecar/searxng/, generates a secret, starts the container, and waits until it really answers a JSON query before reporting success. Then it's detected automatically.

Why it needs a command: SearXNG ships with its JSON API disabled, so the stock image returns 403 to every request, which reads as "unavailable" and falls back silently — the failure looks like nothing happening. The shipped settings enable formats: [html, json] and turn off the bot limiter, which protects public instances and here would only throttle you. Bound to 127.0.0.1: no auth, no limiter, so exposing it would hand anyone a free search proxy on your address.

Already running one? Point SEARXNG_URL at it.

OpenRouter

Not a search API. Its web plugin retrieves inside a chat completion — you can't get results without paying for a completion too — which is why it's a mode on the answer path rather than a search provider.

sc.answer("...", via="openrouter")     # billed, ~$4 per 1000 results
sc.complete("...", web=True)

Worth it when local search is being CAPTCHA'd: retrieval happens on their side, against better sources. Never a default, never implicit, and never cached — paying for retrieval and then serving a stored answer defeats the point.

Dashboard

llm-sidecar serve also serves a dashboard at http://localhost:4001.

Dashboard — API key and budget, status, which local models fit, spend over 30 days, cache size, whether SearXNG is really being used. Chat — multi-turn, streaming, with the model and cost under every reply. Tools — every capability, in one place, to try before you wire it in.

One HTML file. No build step, no dependencies, no external requests — no CDN, no fonts, no analytics; there's a test asserting it. Settings apply to the running daemon and are not written to disk unless you tick "remember".

Opt out entirely with llm-sidecar serve --no-ui (or LLM_SIDECAR_NO_UI); / then 404s and the API is untouched. That controls serving, not installing — the page is a ~40 KB file in the package either way.

Browser clients

The daemon sends no CORS headers by default, so a page cannot call it. That is deliberate: it is unauthenticated on loopback, so a permissive Access-Control-Allow-Origin would let any site you happen to visit spend your OpenRouter credit and read whatever the daemon can reach.

Name the origins you actually want:

export LLM_SIDECAR_CORS_ORIGINS="chrome-extension://your-id,http://localhost:3000"

* is accepted and logs a warning. If you use it, set LLM_SIDECAR_TOKEN too.

Configuration

Precedence: defaults < ~/.config/llm-sidecar/config.json < environment < keyword arguments to Sidecar(...).

The daemon binds loopback deliberately: it holds an API key and spends real money on request. config.save() does not write the API key unless asked.

Caching and cost

Deterministic requests (temperature=0) and searches are cached to disk, which is why re-checking a document doesn't re-pay for the claims that didn't change. Creative requests are deliberately not cached — a byte-identical "random" answer is a surprise, not an optimisation. The cache is trimmed oldest-first to 256 MiB; the ledger rotates at 8 MiB.

Performance

Measured on an M3 Max, not estimated.

Honest limitations

• Verification is only as good as retrieval. The verifier can only grade what came back; ambiguous evidence produces a wrong verdict. Unverified claims are re-checked against full page text, and SearXNG covers more engines — neither is a fix. When retrieval finds nothing useful you get unverified, and that is the correct answer.

• read_url strips <nav>/<header>/<footer> and finds the content region, but it's regex-based, not a readability port. Unusual markup degrades to the whole page.

• Streaming bypasses the completion cache. Replaying stored tokens is a different feature.

• The daemon has no request queue or admission control. Fine for one user, wrong for anything shared.

• Parallel verification helps against cloud models, not against a single local Ollama model — those queue server-side anyway.

• Embeddings are local-only. OpenRouter proxies chat completions and lists no embedding models at all, so embed/rank need ollama pull nomic-embed-text. They refuse rather than falling back to a chat model, because chat-model vectors rank wrongly — measured: asked to rank for "python backend developer", one put "C++ embedded engineer" above "5 years Django and FastAPI". Pass allow_chat_model=True to override.

• No vector store or cross-session memory. Storing and retrieving embeddings is a different product; producing them is not.

• searxng drives Docker or Podman compose. Anything else: run the compose file yourself and set SEARXNG_URL.

Layout

The core imports nothing but httpx. FastAPI and mcp are optional extras, pulled in only by the doors that need them:

pip install -e ".[search]"    # web search
pip install -e ".[daemon]"    # HTTP server + dashboard
pip install -e ".[mcp]"       # MCP server
pip install -e ".[all]"

Tests

pytest

152 tests, fully offline — every network path is stubbed. Live-provider behaviour is verified by hand.

Licence

MIT.