multi-llm-mcp

An MCP server that lets any IDE agent delegate coding tasks to any LLM — cloud APIs or local models — through a single unified interface.

Instead of being locked into one model, your coding agent can call NVIDIA NIM, OpenRouter, Groq, DeepSeek, or a local Ollama model for a second opinion, code review, or specialized task.

Features

• 5 providers out of the box — NVIDIA NIM, OpenRouter, Groq, DeepSeek, Ollama

• Local model support — Use Ollama for fully offline, private coding assistance

• Fine-grained control — Set temperature, max_tokens, and system_prompt per call

• Secure by design — API keys stay in environment variables, never in code

• Connection pooling — Clients are cached for fast, efficient API calls

• MCP standard — Works with any MCP-compatible IDE (Claude Desktop, VS Code, Cursor, Windsurf, etc.)

• Standalone agent skill — A zero-dependency CLI variant (multi-llm-skill/) that works in any agent harness, even without MCP support

Related MCP server: claude-code-delegate-local

Supported Providers

Quick Start

1. Clone and install

git clone https://github.com/arjunkr303/multi-llm-mcp.git
cd multi-llm-mcp

python -m venv venv
source venv/bin/activate   # On Windows: venv\Scripts\activate

pip install -e .

2. Configure API keys

cp .env.example .env

Edit .env and add your API keys. You only need keys for the providers you want to use. Ollama requires no API key.

NVIDIA_API_KEY=your_nvidia_key_here
OPENROUTER_API_KEY=your_openrouter_key_here
GROQ_API_KEY=your_groq_key_here
DEEPSEEK_API_KEY=your_deepseek_key_here

3. Connect to your IDE

Add this to your MCP configuration:

Claude Desktop (claude_desktop_config.json):

{
  "mcpServers": {
    "multi-llm-gateway": {
      "command": "/path/to/multi-llm-mcp/venv/bin/python",
      "args": ["-m", "multi_llm_mcp"]
    }
  }
}

VS Code / Cursor (.vscode/mcp.json or IDE MCP settings):

{
  "mcpServers": {
    "multi-llm-gateway": {
      "command": "/path/to/multi-llm-mcp/venv/bin/python",
      "args": ["-m", "multi_llm_mcp"]
    }
  }
}

Replace /path/to/multi-llm-mcp with the actual path where you cloned the repo.

4. Use it

Once connected, your IDE agent has access to the ask_llm tool. You can invoke it by prompting your agent naturally. The agent will call the MCP server behind the scenes.

Tool: ask_llm

Delegates a coding question, code review, or rewrite task to an external LLM.

Parameters:

Example Prompts

Use these prompts in your IDE chat to trigger the MCP tool:

Rewrite code using a cloud model:

Use NVIDIA's Llama 3.1 to rewrite the database query in db.py to use async/await.

Get a code review from a different LLM:

Ask Groq's Llama 3 to review src/multi_llm_mcp/server.py for performance issues and suggest improvements.

Refactor with a local model (Ollama):

Use Ollama to refactor the authentication logic in auth.py into smaller, testable functions.

Generate tests using DeepSeek:

Ask DeepSeek Coder to write unit tests for the parse_config function in utils.py.

Get a second opinion on architecture:

Use OpenRouter's Claude to evaluate whether the current project structure follows
best practices and suggest a better layout.

Agent Skill — use without MCP

multi-llm-skill/ is a standalone agent skill that bundles the same provider logic into a single zero-dependency CLI (Python stdlib only — no pip installs, no server process). It gives the same delegation capability to any agent:

• Native skill harnesses (Claude Code, OpenCode, etc.) — copy the folder into your skills directory and it auto-triggers on "second opinion", "independent review", and similar requests

• Any other agent (Cursor, Codex CLI, Cline, Gemini CLI, ...) — paste one ready-made snippet into the agent's rules file (AGENTS.md, CLAUDE.md, .cursorrules, ...)

• Standalone — useful right from the terminal:

python multi-llm-skill/scripts/ask_llm.py --list-providers
python multi-llm-skill/scripts/ask_llm.py --system-prompt "You are a senior staff engineer." --prompt "Review this function for bugs: <code>"

Both the MCP server and the skill read the same .env keys and call the same provider endpoints, so they can coexist independently. Full instructions: multi-llm-skill/README.md.

Using with Ollama (Local Models)

For fully private, offline coding assistance:

# Install Ollama: https://ollama.com/download
ollama pull llama3
ollama pull codellama

Then use provider: "ollama" with any pulled model name. No API key needed.

Security

• API keys are loaded from environment variables only

• .env is gitignored and never committed

• No secrets are hardcoded in source code

• All API communication happens server-side only

Project Structure

multi-llm-mcp/
├── src/multi_llm_mcp/      # MCP server package
│   ├── __init__.py
│   ├── __main__.py         # entry point: python -m multi_llm_mcp
│   ├── server.py           # FastMCP server & the ask_llm tool
│   └── providers.py        # provider registry & client factory
├── multi-llm-skill/        # standalone agent skill (zero deps, no MCP)
│   ├── SKILL.md            # skill metadata + agent instructions
│   ├── scripts/ask_llm.py  # the CLI the agent runs
│   └── README.md           # skill install & usage
├── tests/                  # pytest suite
├── pyproject.toml
├── .env.example
└── LICENSE

Running Tests

pip install -e . pytest
pytest

License

MIT — see LICENSE for details.