AgentOverflow

The private-docs registry your AI agents can actually query.

npm License: MIT MCP

Your AI agent knows express.get(). It doesn't know @acme/internal-sdk.

AgentOverflow fixes that. Point it at your private TypeScript SDK or OpenAPI spec. Agents query one symbol at a time in a format that costs 141× fewer tokens than loading a doc page — and they stop hallucinating your internal APIs.

npx agentoverflow-mcp          # drop into Claude Code or Cursor in 60 seconds

The problem in one picture

Without AgentOverflow                   With AgentOverflow
─────────────────────────────────────   ───────────────────────────────────
agent → load MDN fetch page             agent → GET /api/docs/fetch/fetch
        ↓                                       ↓
   ~12,000 tokens of HTML,              DOC|n~fetch;sig~fetch(input,init?)
   navigation, ads, prose,              ->Promise<Response>;
   examples you didn't need,            p~input:str:req,init:obj:opt;
   and the agent still might            r~Promise<Response>;v~browser
   hallucinate the exact sig
   it needed.                           85 tokens. Exact signature. Done.
                                        X-AO-Tokens: 85  (141× savings)

The real wedge is private docs. Context7 is great — it can never index your internal SDK. AgentOverflow can. Index your .d.ts or OpenAPI spec in one curl and your agents get symbol-level precision on private APIs.

What's new in v0.4.0

BM25 semantic search — replaces substring matching. "retry on network error", "make HTTP request" now return the right symbols even without exact name matches.
Python support — pypi:requests, pyi:os pull stubs from typeshed + PyPI alongside TypeScript.
auto_ingest MCP tool — reads your package.json / requirements.txt and indexes every dependency in one shot. Zero manual config.
GitHub webhook auto-sync — register your repo once; docs update on every push automatically.

Benchmarks

Token cost to answer "what does app.get() do?":

Method	Tokens	Accurate?
Load full Express README	~12,000	Sometimes
Load MDN page	~11,800	Yes
AgentOverflow JSON	171	✅ Yes
AgentOverflow dense	85	✅ Yes

Search quality — 30 natural-language queries, 200-symbol corpus:

Engine	Recall@5
Substring match (v0.3)	0.61
BM25 (v0.4)	0.89

Reproduce: npm run bench:tokens · npm test

Quick start (60 seconds)

1. Get a free key — no card needed:

curl -X POST https://ao-registry.fly.dev/api/keys \
  -H "Content-Type: application/json" \
  -d '{"name":"my-workspace","email":"you@example.com"}'
# → { "key": "ao_...", "plan": "free" }

2a. Claude Code / Cursor (MCP)

// ~/.claude/mcp.json  or Cursor MCP settings
{
  "mcpServers": {
    "agentoverflow": {
      "command": "npx",
      "args": ["agentoverflow-mcp"],
      "env": { "AO_KEY": "ao_your_key_here" }
    }
  }
}

Then tell your agent:

> auto_ingest          ← indexes every dep in package.json + requirements.txt
> search "retry fetch with backoff"
> get_doc express app.get

2b. Plain HTTP

curl "https://ao-registry.fly.dev/api/docs/express/app.get?format=dense" \
  -H "X-API-Key: ao_..."
# → DOC|n~app.get;sig~app.get(path,...handlers)->app;...  (85 tokens)

Index your private docs

# TypeScript SDK → every exported symbol from the .d.ts
curl -X POST https://ao-registry.fly.dev/api/ingest \
  -H "X-API-Key: ao_..." -H "Content-Type: application/json" \
  -d '{"targets":["ts:@acme/internal-sdk"],"private":true}'

# OpenAPI spec → every endpoint as a queryable record
curl -X POST https://ao-registry.fly.dev/api/ingest \
  -H "X-API-Key: ao_..." \
  -d '{"targets":["openapi:https://api.acme.com/openapi.json"],"private":true}'

# Python stubs
curl -X POST https://ao-registry.fly.dev/api/ingest \
  -H "X-API-Key: ao_..." \
  -d '{"targets":["pypi:httpx","pypi:pydantic"]}'

# Auto-detect and index everything from package.json + requirements.txt (MCP)
> auto_ingest { "private": true }

Private docs are tenant-isolated — namespaced to your API key, never visible to other users. Your private doc wins over a public one with the same name.

GitHub webhook auto-sync

Docs stay fresh automatically when your SDK changes:

# Register your repo
curl -X POST https://ao-registry.fly.dev/api/webhooks/register \
  -H "X-API-Key: ao_..." -H "Content-Type: application/json" \
  -d '{"repo":"acme/internal-sdk","targets":["ts:@acme/internal-sdk"]}'
# → { "webhookUrl": "https://ao-registry.fly.dev/api/webhooks/github" }

# Add that URL as a GitHub webhook on your repo (push events, JSON content-type)
# Every push → AgentOverflow re-crawls the .d.ts and updates all symbols.

Ingestion sources

Scheme	What it indexes	Best for
`ts:pkg`	Full `.d.ts` surface — every export	Private SDKs, any npm package
`openapi:url`	Every endpoint in OpenAPI 3.x / Swagger 2.0	Internal REST APIs
`pypi:pkg`	Python stubs from typeshed or PyPI	Python packages
`pyi:module`	stdlib stubs from typeshed	Python stdlib
`npm:pkg`	README API sections	Quick npm overview
`mdn:slug`	MDN Web Docs	Browser APIs

MCP tools

Tool	What it does
`get_doc`	Fetch one symbol's doc — cheapest call, 85 tokens
`list_library`	All symbols for a library with token budget trimming
`search`	BM25 full-text search across docs + Q&A
`auto_ingest`	Detect deps from package.json/requirements.txt and index all
`detect_project`	Preview what `auto_ingest` would index (dry run)
`ingest_docs`	Crawl specific targets: npm/ts/mdn/openapi/pypi/pyi
`ask_question`	Post a Q&A entry to the shared registry
`answer_question`	Answer an existing Q&A entry
`vote`	Upvote / downvote Q&A to surface best answers
`registry_stats`	Corpus size, libraries, tokenizer mode

How it compares

	AgentOverflow	Headroom	Context7
Private docs	✅ Index your SDK	❌ Stateless compressor	❌ Public only
Prevents hallucinations	✅ Exact signatures	❌ Can compress wrong answers	✅ Public docs only
Symbol-level lookup	✅ 85 tokens/query	❌ Whole context compressed	✅
Auto-detect project deps	✅ `auto_ingest`	✅ `headroom wrap`	❌
Python support	✅ pypi: / pyi:	✅	✅
GitHub webhook sync	✅	❌	❌
Zero agent code change	❌ Agent calls API	✅ Proxy mode	❌
Self-hostable	✅ MIT	✅ Apache 2	❌

They compose well. Use AgentOverflow for symbol lookup + Headroom for compressing tool outputs. Adjacent problems, not the same one.

Architecture

  Your agent (Claude Code · Cursor · LangChain · any MCP client)
       │
       │  MCP tools  /  HTTP  GET /api/docs/:lib/:symbol?format=dense
       ▼
  ┌─────────────────────────────────────────────────────────┐
  │  AgentOverflow                                          │
  │  ──────────────────────────────────────────────────     │
  │  BM25Index → symbol store → serialize()                 │
  │  (ranking)    SQLite/PG     dense/json/xml/prose        │
  │                                                         │
  │  Crawlers: ts(.d.ts) · openapi · pypi(typeshed)         │
  │            npm · mdn · pyi(stdlib)                      │
  │                                                         │
  │  Auth: API keys · per-plan rate limits · Stripe         │
  │  Private docs: owner-namespaced, tenant-isolated        │
  │  GitHub webhooks: auto re-crawl on every push           │
  └─────────────────────────────────────────────────────────┘
       │
       │  85 tokens · exact signature · no hallucination
       ▼
  LLM (Anthropic · OpenAI · any provider)

Token counts are exact — real BPE tokenizer (gpt-tokenizer, o200k_base). npm run bench:tokens reproduces every number in this file.

Self-host

# Minimal — SQLite, no auth
npm install agentoverflow && npm start

# With auth + Postgres
AO_REQUIRE_KEY=1 AO_ADMIN_TOKEN=changeme \
DATABASE_URL=postgres://user:pass@host/db npm start

# Docker
docker run -p 4317:4317 \
  -e AO_REQUIRE_KEY=1 -e AO_ADMIN_TOKEN=secret \
  -v ao_data:/data \
  ghcr.io/tatsunori-ono/agentoverflow:latest

# Then seed with public libraries
npm run crawl -- ts:express ts:zod pypi:requests openapi:https://petstore.swagger.io/v2/swagger.json

Contributing

git clone https://github.com/tatsunori-ono/agentoverflow
cd agentoverflow && npm install
npm test        # ~30 tests
npm run dev     # hot-reload on :4317

New crawler source? Add it under src/crawler/ and write a test in test/sources.test.js.

License

MIT · LICENSE

Built with Model Context Protocol · gpt-tokenizer · python/typeshed