doc4d (llama.cpp branch)

MCP server for semantic search over the 4D Documentation corpus.

It exposes a single MCP tool, search, which embeds a natural-language query and returns the most semantically similar documentation passages (URL + text) for a given language and 4D product version.

A live demo is running at:

https://doc4d-production.up.railway.app

(streamable-http MCP transport, mounted at /)

This branch runs the embedding model directly via llama-cpp-python against the original GGUF checkpoint, rather than an ONNX export. See main for the onnxruntime-based variant. Functionally the two are equivalent (same model, same pooling, same output vectors) — this branch just skips the ONNX conversion step and its extra dependencies (onnxruntime, tokenizers).

How it works

• Corpus & vector index: keisuke-miyako/doc4d-2026-08-05 on Hugging Face — a SQLite database (doc.db) with a sqlite-vec vec0 virtual table of 1024-dim embeddings, chunked text, and url / language / version metadata for each chunk of 4D documentation.

• Embedding model: LFM2.5-Embedding-350M-GGUF, quantized Q8_0, loaded directly with llama-cpp-python's Llama class in embedding mode — no separate tokenizer file or ONNX export needed, since the GGUF bundles its own tokenizer.

• Pooling: CLS-token pooling (LLAMA_POOLING_TYPE_CLS), matching how the dataset's embeddings were originally generated.

• Server: server.py loads the GGUF model at startup, embeds incoming queries (prefixed with "query: " per the model card), and runs a cosine-distance nearest-neighbor search via sqlite-vec's MATCH operator.

• Transport: MCP over streamable-http, served internally on port 7860 and reverse-proxied by nginx, which also handles CORS and basic per-IP rate limiting.

search tool

Returns a list of { url, similarity, text? } objects, ordered by descending similarity.

Under the hood, k * 20 nearest-neighbor candidates are pulled from sqlite-vec and then filtered down to the requested language/version, since the vector index itself isn't partitioned by those fields.

Related MCP server: Local Search MCP Server

Project layout

.
├── Dockerfile
├── entrypoint.sh          # downloads GGUF model + DB from HF, starts nginx, then server.py
├── nginx_conf.template    # reverse proxy, CORS, rate limiting
├── requirements.txt
├── server.py              # MCP server + embedding + search logic (llama.cpp backend)
└── LICENSE

The Docker image ships without the model or database baked in — entrypoint.sh downloads them from Hugging Face on container start:

• models/LFM2.5-Embedding-350M-Q8_0.gguf from LiquidAI/LFM2.5-Embedding-350M-GGUF (a single self-contained file — no separate tokenizer download needed on this branch)

• data/doc.db from datasets/keisuke-miyako/doc4d-2026-08-05

This keeps the image small and lets the corpus/model be updated without rebuilding the image — just clear the mounted volume (or redeploy) to force a re-download.

Note: entrypoint.sh and Dockerfile on this branch need their model URL/filename updated to point at the .gguf checkpoint instead of model.onnx + tokenizer.json — swap the relevant curl step in entrypoint.sh accordingly if you're porting the main-branch scripts over.

requirements.txt (this branch)

mcp[cli]
sqlite-vec
llama-cpp-python

No onnxruntime or tokenizers needed — llama-cpp-python handles both inference and tokenization internally. Note llama-cpp-python typically needs a compiler toolchain (build-essential, cmake) at install time unless a prebuilt wheel matching your platform is available; keep those in the Dockerfile's apt-get install step for this branch even though the ONNX branch doesn't need them.

Running locally

Docker (recommended):

docker build -t doc4d .
docker run --rm -p 8080:80 -e PORT=80 doc4d

The server will download the model and database on first start (this can take a minute depending on connection speed), then listen on http://localhost:8080.

Without Docker:

pip install -r requirements.txt
# also requires nginx if you want the proxy/CORS/rate-limiting layer,
# or point an MCP client directly at 127.0.0.1:7860 and skip nginx
./entrypoint.sh

entrypoint.sh expects to be run from the repo root and will create models/ and data/ alongside it.

Deploying to Railway

This repo is set up to deploy on Railway with zero config beyond the Dockerfile:

1. New Project → Deploy from GitHub repo, select miyako/doc4d, branch pointing at this llama.cpp variant.

2. Railway detects the Dockerfile automatically and builds it — no build command needed. Note this branch's build step compiles/installs llama-cpp-python, so first builds may take noticeably longer than the ONNX branch.

3. Railway injects $PORT at runtime; entrypoint.sh picks it up automatically and templates it into the nginx config (envsubst '${PORT}'), so no manual port configuration is required.

4. First boot will take longer than subsequent restarts, since entrypoint.sh downloads the GGUF model and doc.db from Hugging Face before starting the server. If you want faster cold starts, attach a Railway volume mounted at /app/models and /app/data so those files persist across deploys/restarts instead of being re-downloaded every time.

5. Once deployed, Railway gives you a public URL (e.g. https://<your-app>.up.railway.app) — that's your MCP streamable-http endpoint.

No environment variables are required for a default deploy — PORT is set by Railway automatically.

CPU note for this branch: server.py sets n_threads=1 on the Llama instance. This was found empirically to avoid pathologically slow inference on throttled/shared-vCPU Railway instances, where llama.cpp's multi-threaded sync busy-spins and fights with the CPU scheduler. If you deploy on a host with dedicated cores, it's worth benchmarking n_threads > 1 — it may be faster there, but don't assume it without testing on the actual target host first.

Deploying elsewhere (Oracle Cloud, bare Docker host, etc.)

The same image works anywhere that can run a container and reach Hugging Face over HTTPS:

• If $PORT isn't set, entrypoint.sh falls back to port 80.

• Make sure outbound HTTPS to huggingface.co is allowed on first boot (for the model/DB download).

• Persist models/ and data/ on a volume if you want to avoid re-downloading the GGUF checkpoint on every restart.

• If building on a platform without a prebuilt llama-cpp-python wheel, expect the pip install step to compile from source — keep build-essential/cmake available at build time.

Connecting an MCP client

Point any MCP client that supports streamable-http transport at the server's base URL, e.g. for the hosted demo:

https://doc4d-production.up.railway.app

Rate limiting (5 req/s per IP, burst 10) and CORS (Access-Control-Allow-Origin: *) are applied at the nginx layer in front of the MCP server.

Notes / caveats

• use_mmap=False is set on the Llama instance — deliberate, not a leftover default; keep it unless you've verified mmap works reliably on your target host's filesystem/container setup.

• n_ctx=512 caps the context window fed to the embedding model; combined with the 2000-character query truncation in search(), very long queries will be truncated by the tokenizer rather than raising an error.

• Query results are only as fresh as the doc4d-2026-08-05 dataset snapshot — see the dataset card for details on how it was built and its limitations.

• The language/version filter happens after vector search on an over-fetched candidate set (k * 20), not natively in the index — if you query for a rare language/version combination, you may get fewer than k results even when more exist in the corpus.

License

MIT — see LICENSE.