🧠 MCP Local Semantics

Four embedding tools an agent can call in a loop, because they cost nothing.

An MCP server that runs a sentence encoder on the machine the agent is already on. Search by meaning, compare texts, cluster them, classify against labels you supply per call — no embedding API, no key, no network, no per-invocation bill.

The interesting part is not that a local model works. It is that a stateless tool re-embeds its whole corpus on every call, which made the first version 1,040 ms per search. Caching the embeddings took it to 1.2 ms.

🔧 The MCP surface: src/mcp_local_semantics/server.py 🧮 The maths, pure NumPy: src/mcp_local_semantics/semantics.py 📝 Related write-up: An MCP server is one Next.js route handler

Measured on CPU

1,000 documents · ~18,000 tokens · all-MiniLM-L6-v2 · no GPU

Reproduce with python scripts/bench.py.

The 893× is the whole design problem

An MCP tool is stateless. The documents arrive as an argument, on every call. So the obvious implementation embeds all 200 of them every time the agent asks a question — and the realistic agent pattern is one corpus and many questions.

# LocalEncoder.encode — keyed on the exact string.
missing = [t for t in dict.fromkeys(texts) if t not in self._cache]

Embeddings are deterministic for a given model, so a cache hit is not an approximation — it is the same vector. The cache is bounded and least-recently-used, because an unbounded one is a slow leak in a long-lived server that an agent keeps feeding new text to.

This is the same shape as putting a cache breakpoint on an agent loop's stable prefix: the expensive part of the call is the part that did not change.

On the cost comparison

Embedding those 18,000 tokens against a hosted API at list price would cost about $0.0004. That is nothing, and quoting it as the headline would be dishonest.

The number that matters is the other one. An agent will call a free, instant tool inside a loop. It will not call one that bills per invocation and waits on a network — and neither will you, when you are deciding whether to let it. Local moves semantic search from a resource to be rationed into an operation the agent can use freely.

The API figure above is computed from published list price and labelled as computed everywhere it appears. It was not measured.

Related MCP server: ClawMem MCP Server

The tools

The descriptions are the interesting part

A vague tool description means the model never selects the tool at all. Each description here names the situations it applies to, not what the function does:

Rank a list of documents by how well each answers a question, using meaning rather than shared keywords. Use this whenever you have a pile of text and a question about it: finding the relevant passage in notes, picking which of several answers fits, or narrowing a long list before reading it. Runs locally, so it is free to call and safe to call repeatedly.

That last sentence is doing real work. It tells the model something true about the tool's cost, which changes how willing it is to reach for it mid-reasoning.

Running it

pip install -r requirements.txt
python -m mcp_local_semantics.server        # stdio

Claude Desktop:

{
  "mcpServers": {
    "local-semantics": {
      "command": "python",
      "args": ["-m", "mcp_local_semantics.server"]
    }
  }
}

Set MCP_SEMANTICS_ENCODER=hash to run without torch — useful for smoke-testing the host connection before committing to a 90 MB download.

pytest -q                  # 36 tests, under a second, no downloads
python scripts/bench.py    # the table above

Built against MCP Python SDK v2, where FastMCP was renamed to MCPServer and moved to mcp.server.mcpserver. Code written for v1 needs either that change or a mcp<2 pin.

Design notes

The encoder is an interface, so the maths is testable

Encoder is four lines: text in, an array out. Deliberately narrower than anything sentence-transformers offers — an interface shaped like the vendor SDK is a second SDK you now maintain.

Two implementations. LocalEncoder is the real one. HashEncoder is a signed hashed bag-of-words: deterministic, no torch, no network. It has no semantic understanding at all — "car" and "automobile" are orthogonal to it — and that is fine, because the test suite is asserting properties of the arithmetic: that top-k orders correctly, that MMR de-duplicates, that clustering converges and is deterministic.

The whole suite runs in 0.42 s with no downloads, so CI does not depend on a model host being reachable. Same separation that made an iteration cap testable against a mocked model client.

No similarity threshold, anywhere

top_k returns the best k regardless of score. A fixed cutoff is not scale-invariant across queries — too strict for short questions, too loose for long ones — and when it is too strict it fails silently, which is the bug this came out of. There is a test asserting no floor is applied. Callers wanting one can apply it to the scores they get back, having seen them.

Clustering is deterministic on purpose

K-means with k-means++ seeding, written out rather than imported so the tool has no scikit-learn dependency at runtime. Seeded, because a tool an agent calls twice on the same input and gets two answers from is a tool the agent cannot reason about. There is a test for it.

k-means++ rather than uniform seeding because uniform routinely puts two centres inside one cluster and leaves another unclaimed.

Errors come back as results, not exceptions

raise ToolError(
    f"texts has {len(texts)} items; the limit is {MAX_TEXTS}. "
    f"Split the batch and call again."
)

Caught at the MCP boundary and returned as a result. Raising through the transport ends the exchange; returning the message lets the model fix it on the next turn. Every guard-rail message says what to do, not just what was wrong — and there are tests asserting the messages contain the instruction.

MMR for when the corpus has near-duplicates

Plain top-k over three paraphrases of the same passage returns all three and tells the model one thing. diversity above 0 penalises a candidate by its similarity to what is already selected. At 0 it is exactly top-k; at 1 it ignores the query entirely.

Stack

Python · PyTorch (CPU) · sentence-transformers · MCP Python SDK v2 · NumPy · pytest

MIT licensed. Built by Ketan Shukla.