posthog-context-mcp

An MCP server that hands PostHog's JavaScript SDK docs to a coding agent, plus an eval that scores what comes back.

Ask it how to capture a custom event in React. It returns around 680 tokens of deduplicated, cited passages. Wire a retriever straight to a tool and dump the top five chunks, and the same question costs about 2,650 tokens. Both find a correct doc every time.

28 hand-labelled questions. naive top-5 concatenates the top five chunks. naive (budgeted) takes that same ranking and stops at the 800-token ceiling, so the engineered config gets no credit for being allowed to stop. Truncating early costs it 21% of the answers.

Restraint is the metric worth explaining. Fourteen of the questions have exactly one doc that answers them. On those, I score whether the system returned that doc alone. Bringing a second source counts as a failure even when hit rate and precision look fine. Any eval that only rewards finding the answer will tell you a system returning the whole index is perfect.

How it works

Ingest (posthog_context/ingest.py) sparse-clones PostHog/posthog.com and reads the MDX. PostHog composes their docs from shared _snippets/ fragments, so a section like "Capturing events" is two lines on disk: an import and <WebSendEvents />. The loader resolves that import graph recursively before it chunks anything. Skip that step and you index empty sections where the useful content should be.

Retrieval (retrieval.py) runs BM25 over two fields, heading and body, scored separately and summed. Tokens get stemmed and camelCase gets split, so usePostHog matches a query about posthog and "Capturing custom events" matches someone asking how to capture a custom event. No vector database.

Assembly (assemble.py) is where the work is. Seven steps, and only the first one adds anything: retrieve 24 candidates, re-score them for task fit, cut everything below 42% of the top passage, fold near-duplicates together, cap the answer at four distinct docs, fill the token budget in value order, then sort into reading order with a citation on every passage.

Server (server.py) exposes three tools over stdio.

Eval (eval/) runs 28 cases through three configs and writes the chart.

Related MCP server: docpilot-mcp

Setup

python3 -m venv .venv && source .venv/bin/activate && pip install -e .

python -m posthog_context.ingest

The ingest pulls about 11MB of markdown into data/, which is gitignored, and builds the index. PostHog's /contents/ directory is MIT licensed and the rest of that repo is not, so this reads only /contents/ and vendors nothing.

The tools

Connect it to an agent

Claude Desktop reads ~/Library/Application Support/Claude/claude_desktop_config.json on macOS. Cursor reads .cursor/mcp.json in your project. Same shape either way:

{
  "mcpServers": {
    "posthog-context": {
      "command": "/absolute/path/to/posthog-mcp-mini/.venv/bin/python",
      "args": ["-m", "posthog_context.server"],
      "cwd": "/absolute/path/to/posthog-mcp-mini"
    }
  }
}

Use absolute paths, and run the ingest first. The server refuses to start without an index rather than quietly serving an empty one.

Run the eval

python -m eval.run

It prints the table, names every case that failed, and writes eval/out/comparison.png.

The loader asserts that the number of parsed cases matches the number the file declares, and that every gold label points at a doc the index actually contains. A loader that silently skips a malformed case still gives you a confident number, just for a smaller experiment you didn't run, and nothing downstream can tell you that happened.

Each module also checks itself:

python -m posthog_context.ingest      # 6 assertions on chunk quality
python -m posthog_context.retrieval   # ranking sanity
python -m posthog_context.assemble    # budgets are never exceeded

Scope

The JavaScript and Web SDK, plus client-side capture. Installation, custom events, identify, person properties, autocapture, configuration. 185 chunks from 15 docs.

Feature flags, session replay, experiments and the server-side SDKs are all left out on purpose. Widening the index makes it vaguer, and the argument here is about depth.

What's still wrong with it

28 cases is a small eval, and I tuned constants while watching it. I kept myself honest by only making changes I could argue from principle, so the length penalty became concave because a 28-token stub can't answer anything, rather than because 0.38 happened to score better. A held-out set is the next step.

The token counter estimates four characters per token. Every number here inherits that approximation. It routes through one function, so swapping in a real tokenizer is a one-line change.

Two restraint failures survive. Ask how to remove a stored super property and you get the right passage plus a passage about removing person properties. Those two read almost identically and they're different APIs. BM25 can't separate them, and neither can my re-scoring. That specific problem is the honest case for adding embeddings.

NOTES.md has the full decision log, including four bugs that produced output looking completely fine.