local-llm-mcp

An MCP server that answers prompts with a local model via Ollama, synchronously.

Companion to codex-offload-mcp. That server exists for slow agentic work that must not block; this one is for fast, private, low-stakes calls where the answer is wanted in the same turn.

Tools

No file access, no command execution, no memory between calls.

Related MCP server: Ollama MCP Server

Install

Prerequisites

• Node.js 20+

• Ollama running locally, with at least one model pulled. The server talks to it over HTTP and does not start it for you:

ollama pull qwen2.5-coder:7b
ollama list          # confirm the model is on disk

Build

git clone https://github.com/jgt87/local-llm-mcp.git
cd local-llm-mcp
npm install
npm run build

This produces dist/index.js. Note its absolute path — every step below needs it.

Add to VS Code

MCP support is built into current VS Code; if the Command Palette lists MCP: commands, you have it. Pick either route:

Guided. Command Palette (Ctrl+Shift+P) → MCP: Add Server → Command (stdio). Enter node as the command and the absolute path to dist/index.js as the argument, then name it local-llm.

By hand. Command Palette → MCP: Open User Configuration to open your user mcp.json (%APPDATA%\Code\User\mcp.json on Windows), and add the server:

{
  "servers": {
    "local-llm": {
      "type": "stdio",
      "command": "node",
      "args": ["C:/path/to/local-llm-mcp/dist/index.js"]
    }
  }
}

Use forward slashes on Windows, or escape backslashes as \\ — a raw C:\path is invalid JSON and the server will silently fail to start.

Non-default Ollama host or model? Add an env block alongside args:

      "env": { "LOCAL_LLM_MODEL": "llama3.2:3b" }

To scope it to one project instead of your whole profile, use MCP: Open Workspace Folder Configuration and put the same servers block in .vscode/mcp.json. That file can be committed, which gives everyone on the repo the same tools.

Verify. Open the Chat view, switch to Agent mode, click Configure Tools, and confirm local_ask, local_classify and local_models appear and are enabled. MCP: List Servers shows the server's status and its logs if it failed to start. If the tools load but every call errors, Ollama is not running — check ollama list.

Add to Claude Code

claude mcp add local-llm --scope user -- node /absolute/path/to/dist/index.js

Confirm with /mcp in a session, or claude mcp list from a shell.

After changing the code

A running server keeps serving the old dist/, so rebuild and restart it:

npm run build

• VS Code — MCP: List Servers → select the server → Restart. (The experimental chat.mcp.autoStart setting can do this for you.)

• Claude Code — restart the session; MCP servers connect at session start.

Configuration

Performance

Measured on a Ryzen AI 9 HX 370 (CPU inference, 61 GB RAM):

Keep outputs short — maxTokens is the main latency lever. At 16 tok/s, 160 tokens is ~10 seconds.

Do not set OLLAMA_IGPU_ENABLE=1 on integrated-GPU hardware. The iGPU shares system memory with the CPU, so generation gets slower (26.2 vs 33.7 tok/s on a Radeon 890M) even though prompt ingest doubles.

Orchestration

There is no orchestrator. Nothing in this server decides what gets routed to the local model. There is no router, no classifier picking a backend, no fallback chain. The only thing steering the choice is the tool descriptions in src/index.ts, which the calling model reads at call time and judges against. Editing those descriptions is how you change routing behaviour; there is no config to tune.

Nothing is offloaded here — that is the point. These tools are synchronous: the prompt goes to Ollama over HTTP and the answer comes back in the same turn. There is no job id, no polling, no state on disk. A local 7B answers in seconds, and wrapping that in a job store would be pure overhead. The rule the two servers are built around: if a tool would need to be polled, it belongs in codex-offload, not here.

Deciding where work goes

Good fits: triage, classification, summarising long output, drafting boilerplate or commit messages, extracting fields from text. Bad fits: anything where a subtly wrong answer is expensive and hard to detect. This server has no file access, no repo context, and no memory between calls.

Output is validated, never trusted

local_classify checks the reply against your label set instead of taking it at face value: a reply naming two labels, or one outside the set, comes back as matched: false with the raw text rather than a guess. Substrings do not count, so informational never silently resolves to info.

This mirrors how the sibling server pairs Codex's self-report with a git diff — the delegate says what it did, and something independent checks it.

The limit is worth stating plainly: validation catches malformed and hedged replies, but cannot catch a confidently wrong one. The none escape hatch narrows that gap and does not close it — a small model will still hand back a plausible in-set label for text belonging to none of them. Treat a returned label as triage, not a verdict.

Licence

MIT