# mcp-local-file-llm

An MCP server that lets an orchestrating model (Claude Code) hand read-and-answer
work to a local LLM without paying to move the source text through its own context.

## Why this shape

The obvious design — an MCP tool that takes a `text` parameter — saves nothing.
Tool-call arguments are assistant *output* tokens, billed at roughly five times the
input rate, so passing a document into such a tool costs more than simply reading
and answering it directly. The saving only appears when the bulk text never reaches
the orchestrator at all.

So the tool takes a *reference* to the data (a file path, a line range, a shell
command) rather than the data itself. The server reads it locally, sends it to the
local model, and returns only the answer.

## Tools

`local_llm(prompt, data?, max_words?, max_tokens?, model?, json_schema?, background?)`

- `data.files` — paths with optional line ranges: `/a/b.py`, `/a/b.py:120-160`,
  `/a/b.py:120-` (to end), `/a/b.py:-80` (from start), `/a/b.py:42` (one line)
- `data.urls` — URLs to fetch; HTML is reduced to text first
- `data.command` — a shell command whose stdout becomes the input
- `data.text` — literal text, for short framing only
- Omitting `data` entirely makes it a plain generation call
- `json_schema` constrains the answer to a JSON Schema, via ollama's native `format`
  field, so extraction returns structured data rather than prose
- `background: true` returns a job id immediately instead of waiting

`local_find(prompt, files, max_files?, model?)` — search many files, given as paths or
globs, for whatever the prompt describes, and return only a ranked shortlist with a
one-line reason per hit. Files are batched to roughly 50KB per request and judged
against a JSON schema, one verdict per file. Use plain `grep`/`rg` when a literal
string would do; this is for when the target is described rather than spelled.

`local_result(job?, wait_ms?)` — collect a background job. Omit `job` to list this
session's jobs and their state. A finished job is returned once, then dropped.

`local_status()` — check the endpoint and list available models.

Inputs larger than one context window are split and processed map-reduce: the prompt
runs against each chunk (a few chunks at a time, see `LOCAL_LLM_CONCURRENCY`), then a
final pass merges the partial answers.

## Background jobs

The point of `background: true` is to overlap local inference with the orchestrator's
own work — fire off "read this page and tell me whether it mentions X" and keep
searching elsewhere while it runs, then collect the answer with `local_result`. Jobs
live in the server process, so they last as long as the session and do not survive a
restart.

Note that the inference host has one GPU. Several background jobs in flight at once
contend for it and each one gets slower; the overlap is with the *orchestrator*, not
a way to get more throughput out of the box.

## A note on concurrency

`LOCAL_LLM_CONCURRENCY` does much less than it looks like it should. Ollama sizes its
parallel slots from whatever VRAM is free, and at `num_ctx` 65536 a single slot already
claims what remains after a 19GB model is loaded, so large requests are served one at a
time no matter what this is set to.

Measured on an RTX 4090 with `qwen3-coder:30b`, a 253KB five-chunk run:

| concurrency | 1 | 2 | 3 | 4 | 6 |
|---|---|---|---|---|---|
| wall clock | 8.6s | 8.5s | 8.6s | 8.5s | 8.3s |

Flat. Concurrency only helps when the individual requests are small enough for several
slots to fit at once. The default of 2 is kept because it costs nothing and does help on
smaller inputs.

Benchmarks against this backend are easy to get wrong: any other job on the same GPU
distorts them badly. An earlier run of this same test appeared to show four-way
concurrency taking over five minutes, which turned out to be an unrelated background job
competing for the card. Serialise everything before measuring.

## Routing

The tool descriptions carry explicit size thresholds, because the economics invert
depending on how much data is involved:

| data size | what to do |
|---|---|
| under 20KB | read it directly; the round trip costs about 5s and saves a fraction of a cent |
| 20-75KB | delegate; roughly 10-50x cheaper for a few seconds more latency |
| over 75KB | always delegate; reading it directly is billed once and then rides along in every later request |

Text that is *already* in the caller's context should never be passed back in: tool
arguments are billed as output tokens, so returning it costs several times more than
simply using it where it already is.

## Usage tracking

Every call appends one JSON object to a usage log, so the question "is this actually
worth having?" can be answered from data rather than argued about. Each record holds the
tool, the model, a truncated prompt, the source paths, bytes and estimated tokens in and
out, chunk count, duration, and any error.

    npm run report                 # or: node report.mjs
    node report.mjs --days 7       # last week only
    node report.mjs --calls        # also list recent individual calls

The report gives per-tool call counts, how many tokens were kept out of the caller's
context against how many came back, a compression ratio, and a cost comparison: what
reading that text directly would have cost against what delegating actually cost. That
figure is a floor — it counts the text once, and ignores the fact that text read directly
also rides along in every later request of the session.

It also flags calls whose input was under 20KB. Those are below the threshold where
delegation pays, so a high count there means the tool is being reached for too eagerly
and the routing guidance needs tightening.

Set `LOCAL_LLM_LOG=off` to disable logging entirely, or `LOCAL_LLM_LOG_PROMPTS=off` to
record only sizes and timings without any prompt text.

## Configuration

| Variable | Default | Purpose |
|---|---|---|
| `LOCAL_LLM_URL` | `http://localhost:11434` | Ollama root (native `/api/chat`, so `num_ctx` is settable per request) |
| `LOCAL_LLM_MODEL` | `qwen3-coder:30b` | Model tag |
| `LOCAL_LLM_NUM_CTX` | `65536` | Context window requested per call |
| `LOCAL_LLM_CHUNK_CHARS` | `140000` | Chunk size before map-reduce kicks in |
| `LOCAL_LLM_CONCURRENCY` | `2` | Chunks in flight during the map phase (see below) |
| `LOCAL_LLM_FIND_BATCH_CHARS` | `50000` | Bytes of files per `local_find` request |
| `LOCAL_LLM_TIMEOUT_MS` | `300000` | Per-request timeout |
| `LOCAL_LLM_MAX_BYTES` | `4000000` | Refuse inputs larger than this |
| `LOCAL_LLM_ALLOW_SECRETS` | unset | Set to `1` to disable the credential-path guard |
| `LOCAL_LLM_LOG` | `$XDG_DATA_HOME/mcp-local-file-llm/usage.jsonl` | Usage log path, or `off` |
| `LOCAL_LLM_LOG_PROMPTS` | on | Set to `off` to log sizes and timings but no prompt text |

## Security note

Everything passed as `data` leaves this machine for the inference host, so paths that
look like credentials (`.ssh/id_*`, `.env`, `*.pem`, `*.key`, `.aws/credentials`,
`secrets.yaml`, and similar) are refused unless `LOCAL_LLM_ALLOW_SECRETS=1`. The
`data.command` field runs a shell command, which is the same privilege the caller
already has through its own shell tool, but it is worth knowing it is there. `data.urls`
makes outbound requests from this machine to whatever URL it is given.