litellm-mcp

MCP server that lets Claude Code dispatch tasks to the LiteLLM gateway (192.168.10.175:4000) as worker agents. Unlike ollama-mcp (which reaches only the local Ollama models directly), this shim reaches the full gateway model set — the local Ollama models and the Anthropic-via-gateway models (claude-sonnet-4-6, claude-haiku-4-5).

Tools

• list_models() — model ids registered on the gateway.

• ask_model(prompt, model=None, system=None) — single-turn worker call. Defaults to qwen2.5:7b-instruct (a reliable non-reasoning local model — a safer headless default than the reasoning model qwen3.5:4b).

• chat(messages, model=None) — multi-turn.

Related MCP server: mcp-local-llm

Credentials — per-request header (not at rest on Proteus)

This shim brokers spend on Anthropic, so the gateway master key is sensitive. It is not stored in /etc/mcp/litellm-mcp.env on the shared Proteus host. Instead the key travels as a per-request header from the client's ~/.claude.json:

"litellm": {
    "type": "sse",
    "url": "http://192.168.10.190:8107/sse",
    "headers": { "x-litellm-key": "<LITELLM_MASTER_KEY from Infisical>" }
}

The server reads x-litellm-key (or Authorization: Bearer ...) from each tool call. For local/stdio use, LITELLM_API_KEY env is accepted as a fallback.

Config (non-sensitive, env file on Proteus)

Tests

pip install -e ".[test]"
pytest -q                       # unit (respx-mocked)
LITELLM_API_KEY=... pytest -q   # also runs live integration against the gateway

Integration tests skip automatically when the gateway is unreachable or LITELLM_API_KEY is unset (e.g. in CI on the GitLab runner). Deploy is via GitLab CI → Proteus (/opt/mcp/litellm-mcp/, systemd litellm-mcp.service, port 8107).