# LLMconcil

An MCP server that lets Claude Code ask several other models the same question,
then hands back a structured comparison of what they said.

The idea is stolen from [OpenRouter's Fusion](https://openrouter.ai/docs/features/fusion),
including the part most people get wrong: the second model does **not** merge the
answers. It compares them and
reports where they agree, where they contradict each other, and what none of them
brought up. Claude writes the final answer from that. A merged answer hides which
parts were unanimous and which came from one model having a bad day; a comparison
doesn't.

```
Claude Code
   │  fusion_deliberate(prompt, context=[{path, lines}, ...])
   ▼
 panel ──┬─→ google/gemini-3.1-pro-preview   (+ Google Search grounding)
         ├─→ deepseek/deepseek-v4-pro
         ├─→ x-ai/grok-4.5
         └─→ minimax/minimax-m3
   │       four independent answers, in parallel
   ▼
 judge  ──→ moonshotai/kimi-k3
   │       { consensus, contradictions, partial_coverage,
   │         unique_insights, blind_spots }
   ▼
 Claude Code writes the final answer
```

Worth it for architecture trade-offs, "is this actually a good idea", library
choices, anything where being confidently wrong is expensive. Not worth it for
tactical questions with one right answer — you'd be paying four models to agree.

## Setup

Needs Python 3.11+ and an OpenRouter API key. Everything else is optional.

```bash
git clone https://github.com/azeur365/LLMconcil
cd LLMconcil
python3 -m venv .venv && source .venv/bin/activate
pip install -e .
cp .env.example .env      # then fill in OPENROUTER_API_KEY
```

Register it with Claude Code:

```bash
claude mcp add llmconcil -- "$PWD/.venv/bin/llmconcil"
```

`fusion_deliberate` shows up as a tool from there.

## The council

`council.toml` decides who sits on the panel and who judges. This is the shipped
default, the one in the diagram above:

```toml
[[panel]]
model = "google/gemini-3.1-pro-preview"
search = true

[[panel]]
model = "deepseek/deepseek-v4-pro"

[[panel]]
model = "x-ai/grok-4.5"

[[panel]]
model = "minimax/minimax-m3"

[judge]
model = "moonshotai/kimi-k3"
```

`model` is an OpenRouter slug. The file is re-read on every call, so you can
change the line-up without restarting the server.

Pick models that disagree: four siblings from the same lab produce four
near-identical answers and a judge with nothing to report. And keep the judge's
vendor off the panel. `blind_spots` only means something coming from a model that
didn't answer the prompt itself.

`search = true` turns on Grounding with Google Search for that seat. It only
works on the AI Studio path, so it needs a Gemini model and a `GEMINI_API_KEY`.
On any other seat it does nothing, and a Gemini seat that falls back to
OpenRouter answers without grounding.

That asymmetry is deliberate. OpenRouter has its own web plugin, and it is not
wired up here: it bills per request, and on a model whose provider has no native
search it substitutes a third-party engine. Losing citations on a fallback is a
smaller problem than a config flag that quietly changes which search engine
answered and what it cost.

## Gemini and Google AI Studio

Everything runs on OpenRouter, with one exception you can opt into.

If `GEMINI_API_KEY` is set, Gemini panellists are sent to Google directly instead
of through OpenRouter. Nothing else changes: same slug in `council.toml`, same
model, same grounding mechanism, same shape coming back. But the calls come out
of your Gemini quota, which matters if your subscription includes credit that
OpenRouter can't spend.

Without the key, Gemini rides on OpenRouter like everything else.

The key has to have billing enabled. AI Studio's free tier won't serve the Pro
models this is pointed at, so there's no free path to fall back to and the code
doesn't pretend otherwise: AI Studio answers are reported as unpriced, never as
free.

When the call doesn't land, for any reason, it falls back to OpenRouter rather
than dropping the panellist. Quota exhaustion and "this model is experiencing
high demand" are both routine, and neither is worth losing an answer over.
`meta.aistudio_fallbacks` records what went wrong, so a key that never works
shows up instead of quietly spending OpenRouter credit.

The judge always goes through OpenRouter. It needs
`response_format: json_schema`, which isn't worth reproducing on the genai SDK
(where it also can't be combined with grounding) to save a few cents.

`meta.served_by` tells you which backend actually answered for each model.

## The tool

`fusion_deliberate(prompt, context?, panel?, judge?, temperature?, reasoning_effort?)`

`context` takes file refs (`{path, lines}`, read server-side), inline snippets
(`{text}`), and images (`{image}`, png/jpg/gif/webp). Curate it. Everything you
attach is sent to every panellist, so an irrelevant file costs you N times and
dilutes the analysis. There's a 200k-token budget; over it the call is rejected
with a message naming the offending files rather than silently truncating. Paths
are confined to `LLMCONCIL_ROOT`, and binary or oversized (>5 MB) files are
refused.

Attaching an image drops the panellists that can't see it, listed in
`meta.skipped_no_vision`. Capability comes from OpenRouter's `/models` catalogue,
fetched once and only when an image is attached. If that fetch fails, nothing is
dropped and a text-only model fails on its own, visibly.

`panel` / `judge` override `council.toml` for one call. Both take bare slugs, so
they carry no per-model options: a model named that way runs without search.

Returns `{status, analysis, responses, failed_models, failure_reason, meta}`. The
raw panel answers come back alongside the analysis, so Claude can go read what a
model actually said instead of trusting the judge's summary of it.

`meta.cost_usd` is what OpenRouter billed, read off the response rather than
estimated from a price table. Calls it didn't bill (anything AI Studio served)
are listed in `meta.cost_excludes` instead of being counted as free.

Here is `analysis` from a real run, asking whether a small team should pin exact
dependency versions. Trimmed to one entry per key:

```json
{
  "consensus": [
    "Pin exact versions, via a committed lockfile that freezes the full tree."
  ],
  "contradictions": [
    {
      "topic": "What should be declared in the manifest?",
      "stances": [
        {
          "model": "deepseek/deepseek-v4-pro",
          "stance": "Ranges. The lockfile does the actual pinning."
        },
        {
          "model": "x-ai/grok-4.5",
          "stance": "Exact versions, so intent survives someone deleting the lockfile."
        }
      ]
    }
  ]
}
```

That contradiction is the whole point. A merged answer would have picked one of
those two and thrown the other away, and you would never have known the question
was contested.

## Configuration

Everything lives in `.env`, and everything but the first line has a default that
works.

| variable | default | what it does |
|---|---|---|
| `OPENROUTER_API_KEY` | — | required; serves the whole council |
| `GEMINI_API_KEY` | unset | routes Gemini to AI Studio instead of OpenRouter |
| `LLMCONCIL_COUNCIL` | `./council.toml` | where to read the line-up from |
| `LLMCONCIL_ROOT` | working dir | file refs outside this are rejected |
| `LLMCONCIL_MAX_CONTEXT_TOKENS` | `200000` | curation budget for attached context |
| `LLMCONCIL_STALL_TIMEOUT` | `180` | seconds of silence before a stream is killed |
| `LLMCONCIL_JUDGE_CONTEXT_MAX` | `32000` | above this, the judge doesn't re-read the context |
| `LLMCONCIL_PROVIDER_SORT` | `throughput` | OpenRouter provider routing; `price`, `latency`, `none` |

## Failure modes

A panellist that dies takes its own answer down and nothing else: it lands in
`failed_models` with a reason and the rest of the council carries on. Same for
the judge. You still get the panel answers, with an empty analysis and
`status: "error"`.

Every streamed call has a stall timeout (`LLMCONCIL_STALL_TIMEOUT`, default 180s)
that fires when no token arrives within the window. It's deliberately a *stall*
timeout and not a total one: a model reasoning hard for six minutes is working,
a model silent for three is stuck, and a fixed deadline can't tell them apart.
Keepalive frames don't reset it, or a stuck stream would keep itself alive
forever.

## Layout

| file | role |
|---|---|
| `server.py` | the MCP server and the `fusion_deliberate` tool definition |
| `fusion.py` | orchestration — panel fan-out, then the judge |
| `council.py` | reads `council.toml` |
| `backends.py` | which service serves which model |
| `openrouter.py` | streaming client, stall timeout, model catalogue |
| `gemini.py` | the Google AI Studio path |
| `context.py` | file reading, line ranges, token budget |
| `schema.py` | the JSON schema the judge has to fill in |

## Known rough edges

- No tests. The failure paths are handled but only manually exercised.
- The judge's structured output leans on `response_format: json_schema`. Models
  vary in how well they honour it, so there's a tolerant parser behind it that
  digs the first balanced JSON object out of the reply. A judge that ignores the
  schema entirely yields an empty analysis rather than garbage.
- A near-200k context will overflow panellists with smaller windows. They fail
  individually and land in `failed_models`, but nothing warns you upfront.
- The `/models` catalogue is fetched once and kept for the life of the process,
  so a model that gains vision after your server started won't be recognised
  until you restart it.

## License

MIT