LLM Council MCP

🏛️ Provenance & What This Fork Changes

Fork Lineage:
This project is an advanced, production-hardened fork of Andrej Karpathy's llm-council (expanded from az9713/llm-council).

Karpathy built the original llm-council as a fun "Saturday vibe-hack" — a lightweight web script to compare commercial frontier LLMs side-by-side using OpenRouter.

LLM Council MCP transforms that initial prototype into an enterprise deliberation engine and a headless consensus oracle for autonomous AI agents. Instead of humans manually querying a browser, agents like Claude Code and Google Antigravity call the council programmatically via the Model Context Protocol (MCP) whenever they face irreversible, high-stakes (Type-1) architectural dilemmas.

Related MCP server: tokonomix-council-mcp

⚡ Original vs. LLM Council MCP: Feature Comparison

📸 Visual Tour of Capabilities

1. Stage 1: Domain-Specialized Independent Responses

Models operate as dedicated domain specialists rather than generic chatbots. Each seat evaluates the problem through an injected skill lens, providing evidence-backed technical trade-offs, confidence calibration (1–10), and physical constraint analysis.

• Specialist Roles: @deep-research verifies primary documentation and specifications; @supply-chain-audit validates package pinning, license compliance, and maintainer surface; @first-principles strips framework marketing to benchmark raw compute/memory/IO physics.

• Peer Review & Ranking: In Stage 2, seats anonymously critique peer positions, expose hallucinations, and vote on a ranked ballot before the Chairman synthesizes the binding ADR.

2. Specialized Council Boards

Switch between dedicated expert boards with a single click or target specific boards programmatically via council_id in MCP calls:

• 🧠 cognitive-strategy: High-stakes architectural & strategic trade-offs (Red Team + First Principles + Deep Research).

• 🛠️ code-craft: Deep refactoring, diff-risk minimization, and surgical simplicity (Antigravity Diff Risk + Qwen Verification + Claude Referee).

• 🔬 deep-tech: Protocol RFCs, low-level binary formats, performance limits, and dependency audits.

• 🛡️ sec-ops: Production security, zero-trust policies, OWASP audits, and SRE resilience.

• 🎨 frontend-craft: Distinctive design systems, UI/UX DNA, and responsive client workflows.

• 📡 tech-scout: Automated technology scouting, candidate radar, and adopt-vs-build evaluations.

• ☁️ cloud-deliberation: Responsive zero-cost cloud and local models for fast fallback when host shims are offline.

3. Round Table Multi-Agent Group Chat & Live Process Telemetry

Engage in collaborative, unconstrained multi-agent discussions alongside the structured 3-stage deliberation pipeline:

• Direct Conversational Collaboration: Mention @all to broadcast or @model (e.g. @qwen, @antigravity) to target specific agents with tool-calling capabilities.

• Live Process Telemetry (L0–L3): Slide between detail levels to inspect step-by-step model reasoning, stage transitions, token generation speeds, and consensus voting metrics in real time.

4. Empirical Performance Dashboard

Track which models and domain skills deliver the most accurate evaluations based on historical peer review outcomes:

• Leaderboard Metrics: Win rates, average peer rank, average confidence calibration, total queries, and unique model evaluations.

• Model & Chairman Analytics: Inspect historical win percentages and peer alignment metrics across observation windows.

5. Advanced Consensus Modes & Deliberation Presets

Configure reasoning depth and consensus algorithms directly from the settings drawer:

• Deliberation Presets:

  • ⚡ Fast: Skips Stage 3 synthesis when unanimous peer consensus is detected; reuses cached answers.

  • ⚖️ Balanced: Full 3-stage deliberation with weighted peer votes (default).

  • 🧠 Deep: Structured reasoning passes, critique-and-revise, and devil's advocate challenge rounds.

  • ⚔️ Debate: Multi-round adversarial debate: Position → Direct Critique → Rebuttal → Chairman Judgment.

• Consensus Toggles: Multi-Chairman Ensemble, Win-Rate Weighted Consensus, Early Consensus Exit, and Sub-Question Decomposition.

🔌 Using as an MCP Oracle (Claude Code & Antigravity)

The repository bundles a standalone FastMCP server in mcp/ that lets AI coding assistants deliberate before committing dangerous or irreversible changes.

The 4-Point Gating Guardrail

To prevent agents from lazily delegating routine tasks, ask_council enforces a strict gating checklist:

1. Type-1 Decision: Must be irreversible or carry a high rollback cost (justified in type1_rationale).

2. Genuine Uncertainty: The agent must have attempted solo reasoning first and encountered a real conflict or unknown.

3. High Cost of Error: The cost of picking the wrong path must exceed ~35s + API token cost.

4. User Has Not Decided: Council informs open choices; it never overrides an explicit user directive.

Trivial or unjustified queries are rejected in milliseconds with ## Verdict: Gating Rejection without triggering backend LLM calls.

Output Schema (Bounded ≤150-Word ADR)

Calls to ask_council return a structured, high-density Markdown Architectural Decision Record:

## Verdict: Adopt SQLite-vec as the primary embedded vector store for sub-100k local embeddings.
**Confidence:** Consensus — 3 models evaluated (top ranked: local/antigravity@deep-research)
**Recommendation:** SQLite-vec delivers a single compiled C extension with zero external dependencies and guaranteed format stability through 2050. At <100k vectors, memory-mapped SIMD flat scan delivers sub-5ms latency with 100% recall.
**Dissenting risk:** Single primary maintainer risk; if scale exceeds 1M vectors or distributed out-of-core indexing is required, evaluate LanceDB.

🚀 Quick Setup

Using an AI coding agent? Tell it: "Clone https://github.com/alkrcaaa/llm-council-mcp and install it by following AGENT_INSTALL.md." That file also covers reporting issues and contributing via pull requests.

Prerequisites

• Docker with Compose v2 (docker compose version works)

• An OpenRouter API key for cloud models (models ending in :free cost $0)

1. Install & start (one command)

git clone https://github.com/alkrcaaa/llm-council-mcp.git llm-council
cd llm-council
./setup.sh

On the first run setup.sh:

1. creates .env from .env.example and asks for your OpenRouter key,

2. generates a random admin password and JWT_SECRET,

3. builds and starts the backend + frontend containers,

4. prints the URL and login credentials.

Open http://localhost:5173 and sign in with the printed credentials (username admin). Change the password from the UI after the first login.

2. Set up your models

The built-in boards support host-side agent shims (local/claude-code, local/antigravity), a local vLLM server (local/qwen3.6-27b) and API-key providers (custom/gemini-*, custom/groq, OpenRouter). On a fresh install those seats won't answer until you configure them. The quickest path:

1. Open Configure Models in the UI.

2. Create your own council (or edit a copy of a built-in one) using OpenRouter model IDs, e.g. google/gemma-4-31b-it:free, nvidia/nemotron-3-super-120b-a12b:free. At least 2 seats are required. The free lineup changes often; see the current list at openrouter.ai/models?q=free.

3. Optional: under Model Studio & Providers, add other OpenAI-compatible endpoints (Ollama, LM Studio, vLLM, Groq, Gemini, DeepSeek, ...) and use them as seats.

Free OpenRouter models are rate-limited, so expect occasional 429 errors on busy models; the council continues with whichever seats answered.

3. Optional: bring the built-in boards online

Each built-in seat type can be activated on your machine. Enable whichever you have; seats you don't enable simply fail and the council continues with the rest.

Claude Code & Antigravity seats (local/claude-code, local/antigravity)

Small host-side bridges in infra/local-models/ turn a logged-in CLI into an OpenAI-compatible endpoint the backend container can call. They must run on the host (not in Docker) because they use your CLI login. Each request is a real CLI call on your account, and the CLIs run in restricted/sandboxed mode (no tools, no file writes).

1. Install and log in to the CLI(s) you have: Claude Code (claude, run it once to log in) and/or Google Antigravity (agy).

2. Run:

   ./setup.sh shims

   It detects which CLIs are installed, generates a per-shim secret in .env, and

   • Linux (systemd): installs and starts user services llm-council-claude-code-shim / llm-council-antigravity-shim, bound to the Docker bridge address so only containers (not your LAN) can reach them. Logs: journalctl --user -u llm-council-claude-code-shim -f.

   • macOS / no systemd: prints the command to run each shim in a terminal.

3. It restarts the backend so it picks up the secrets. The Claude seats should now answer.

To remove them: systemctl --user disable --now llm-council-claude-code-shim llm-council-antigravity-shim and delete the unit files in ~/.config/systemd/user/.

Local model seat (local/qwen3.6-27b)

Point it at any OpenAI-compatible server in .env, then re-run ./setup.sh:

# vLLM (the author's setup)
QWEN_BASE_URL=http://host.docker.internal:8002/v1
QWEN_MODEL_ID=/models/qwen3.6-27b

# or Ollama (on Linux start it with OLLAMA_HOST=0.0.0.0 so containers can reach it)
QWEN_BASE_URL=http://host.docker.internal:11434/v1
QWEN_MODEL_ID=qwen3:8b

Gemini & Groq seats (custom/gemini-3-6-flash, custom/groq)

Add them under Configure Models → Model Studio & Providers with your own API keys (both have free tiers). The seat ID is derived from the friendly name, so name them exactly Gemini 3.6 Flash (→ custom/gemini-3-6-flash) and Groq (→ custom/groq) for the built-in boards to pick them up.

Configuration reference

All settings live in .env (see .env.example for every option):

After editing .env, run ./setup.sh again to apply it.

Opening the UI to your LAN? Keep AUTH_ENABLED=true: the API spends your API credits.

4. Connect MCP to Your Agents

cd mcp
bash install.sh
cd ..

Claude Code (~/.claude.json):

{
  "mcpServers": {
    "llm-council": {
      "command": "/absolute/path/to/llm-council-mcp/mcp/.venv/bin/python",
      "args": ["/absolute/path/to/llm-council-mcp/mcp/server.py"],
      "timeout": 140000
    }
  }
}

Antigravity (~/.gemini/config/mcp_config.json):

{
  "mcpServers": {
    "llm-council": {
      "command": "/absolute/path/to/llm-council-mcp/mcp/.venv/bin/python",
      "args": ["/absolute/path/to/llm-council-mcp/mcp/server.py"],
      "timeout": 140000
    }
  }
}

🛠️ Tech Stack

• Core Engine: FastAPI, Async HTTPX, Pydantic, uv

• Protocol: FastMCP (Model Context Protocol stdio transport)

• Frontend: React 18, Vite, Obsidian & Azure Design System, React Markdown

• Models: OpenRouter, vLLM (Qwen 2.5/3.6/3.8), Local Host Shims (Claude Code CLI, Antigravity CLI), Gemini, Groq

• Containerization: Docker & Docker Compose

📜 Acknowledgments & License

• Original concept and initial implementation by Andrej Karpathy

• Extended multi-feature baseline by az9713

• Released under the MIT License.