# DocMesh MCP

DocMesh indexes multiple public documentation sites into PostgreSQL and exposes cited passages through one MCP server. It is retrieval-only: it does not answer the technical question or mutate sources through MCP.

## Architecture

The ingestion plane validates and DNS-pins public URLs, discovers pages from sitemaps or same-origin links, extracts article Markdown, chunks by headings, embeds changed chunks, and updates each document transactionally. The query plane routes a query to up to three sources, runs exact-symbol, PostgreSQL full-text, and pgvector searches, then combines ranks with reciprocal-rank fusion. See [ARCHITECTURE.md](ARCHITECTURE.md).

## Prerequisites and setup

- Node.js 22+ (Node 24 LTS works)
- pnpm 10+
- Docker, or PostgreSQL 17 with pgvector

```sh
pnpm install
cp .env.example .env
docker compose up -d
pnpm migrate
```

The checked-in migration uses `vector(1536)`, matching `.env.example`. To change embedding dimensions safely, create a migration that rebuilds the vector column and HNSW index, set `EMBEDDING_DIMENSION`, then re-ingest all documents. Do not change only the environment variable.

## Sources, ingestion, and search

```sh
pnpm cli source add --name AlphaDB --url https://docs.example.com --aliases alpha --tags database
pnpm cli source version:add --source alpha-db --version 2.0 --url https://docs.example.com/v2 --default
pnpm cli source list
pnpm cli ingest --source alpha-db
pnpm cli search --query "How does AlphaClient.connect() retry?"
```

For checked-in fixtures, no localhost crawl bypass is provided. They are loaded directly from trusted repository files:

```sh
pnpm fixtures:ingest
ALLOW_DEBUG=true pnpm cli search --query "Compare AlphaDB and BetaORM transactions" --debug
pnpm eval
```

## MCP

Local stdio configuration:

```json
{
  "mcpServers": {
    "docmesh": {
      "command": "pnpm",
      "args": ["--dir", "/absolute/path/to/docmesh-mcp", "mcp:stdio"],
      "env": { "DATABASE_URL": "postgres://docmesh:docmesh@localhost:5432/docmesh" }
    }
  }
}
```

Remote Streamable HTTP:

```sh
pnpm mcp:http
# endpoint: http://127.0.0.1:3000/mcp
```

The public surface is deliberately small: `list_sources`, `search_docs`, `get_document`, and `docs://{sourceId}/{version}/{documentId}` resources. Ingestion and arbitrary fetching are CLI-only.

## Configuration

`.env.example` documents all variables. Core groups are database connection, embedding provider/model/dimension, optional OpenAI router and timeout, crawler limits, HTTP port, Playwright fallback, logging, and development debug output. Without an API key, fake deterministic embeddings and deterministic routing keep local tests credential-free. For useful production semantic retrieval, select the OpenAI embedding provider and re-ingest.

Playwright is installed as an optional package but browser binaries are not downloaded automatically. If rendering is enabled, run `pnpm exec playwright install chromium` explicitly.

## Checks

```sh
pnpm format:check
pnpm lint
pnpm typecheck
pnpm test
```

Database integration tests are enabled with `TEST_DATABASE_URL`:

```sh
TEST_DATABASE_URL=postgres://docmesh:docmesh@localhost:5432/docmesh pnpm test
```

## Security

Crawling arbitrary URLs is dangerous. DocMesh permits only HTTP(S), rejects credentials and private/loopback/link-local/multicast/reserved/metadata addresses, pins validated DNS answers for each request, revalidates every redirect, limits redirects/time/bytes/pages/depth, accepts textual content types, and restricts recursive links to configured origins and path prefixes. Playwright validates every browser request but browser-level DNS pinning depends on Chromium; keep it disabled for untrusted sources unless the deployment also enforces outbound network policy.

## Known limitations

- HTML heuristics cover common documentation layouts, not every JavaScript framework.
- HTTP validators are stored, but conditional `If-None-Match`/`If-Modified-Since` requests are not yet issued; normalized hashes still prevent re-embedding unchanged content.
- Sequential fetching deliberately favors politeness and simplicity over large-crawl throughput.
- The router cache is process-local only through provider/client behavior; a durable routing cache is not included in the MVP.
- Remote deployment needs authentication, TLS, rate limiting, and network egress controls in front of the MCP endpoint.

Highest-value next improvements: conditional HTTP recrawls, a bounded concurrent crawl worker pool, and retrieval evaluation on a larger real documentation corpus.