# Context Gateway

A tiny always-loaded gateway + searchable registry that lets any MCP-capable
client discover **all** of your context — tools, skills, plugins, MCP servers,
folders, agents, data sources, reference docs — **on demand**, without paying
the token cost of loading everything into the first prompt.

It generalises the "code-mode" progressive-discovery trick (a small gateway
that reveals capabilities as you ask for them) to every context type.

```
first prompt  →  map()      →  search()     →  describe()    →  load()
 ~1-2 KB        categories    name+one-liner   full card       actual content /
 (bootstrap)    + counts      matches only     (usage, cost)   activation steps
```

Nothing heavier than the layer you asked for ever enters the context window.

**Token budget (typical):** bootstrap ~500 tok (paste once) · `map()` <1 KB ·
`search()` ~15 one-liners · `describe()` one card · `load()` full file only on
demand. Catalog size (e.g. 200+ rules) is **not** prompt size unless you load them.

Global `alwaysApply` rules are indexed but **hidden from default discovery** —
Cursor already injects them; use `include_disabled=True` or `enable(id)` to surface.

---

## Why this exists

Clients only bind the tools/skills/context they can see at **session start**.
If you expose everything up front you blow the context window; if you don't,
capabilities are "not found" and never used. The gateway resolves the tension:
the **only** thing a client needs at session start is the gateway server + a
~500-token bootstrap block. Everything else is discoverable in one tool call.

## The four discovery layers

| Layer | Tool | Returns | Rough cost |
|-------|------|---------|-----------|
| Bootstrap | (static block) | "you have a gateway; always search first" + pinned one-liners | ~500 tok |
| Inventory | `map()` | categories, counts, profile names | <1 KB |
| Summaries | `search(query)` | matching names + one-line descriptions only | small |
| Card | `describe(id)` | full card: usage, token cost, how to activate, current state | medium |
| Content | `load(id)` | the actual skill body / file, or exact activation steps | as needed |

## Control plane vs data plane

Discoverability ("can this be found") and calling ("how does it run") are kept
separate on purpose.

**Control plane — discoverability (persists in `servers.yaml`):**

| Tool | Purpose |
|------|---------|
| `list_servers(profile?)` | every MCP server + its enabled/proxy state |
| `enable(id)` / `disable(id)` | toggle whether an asset is discoverable |
| `set_proxy(id, on)` | toggle opt-in proxy calling for a server |

Disabling hides an asset from `map`/`search` without deleting it. `search(...,
include_disabled=True)` still reveals hidden assets (marked `[disabled]`).

**Health plane — availability (probed, stored, timestamped):**

| Tool | Purpose |
|------|---------|
| `probe(id?/scope)` | live-test server(s): connect, list tools, latency; store result + timestamp |
| `health_status(id?, status?)` | read stored snapshots (fast, no probing); filter by status |

The gateway can actually **test** the MCPs it finds — connect, initialize, count
tools, measure latency — and record a snapshot per server: `available`,
`timeout` (slow to start, *not* dead), `unavailable` (with the error), or
`unconfigured` (latent). Snapshots carry a UTC timestamp and go stale after a TTL.
State refreshes lazily on use: a successful proxy `call` marks a server available,
a failed one marks it unavailable, and `suggest`/`search(available_only=True)`
re-probe only the *stale* servers among the ids they touch. You can then filter
by availability: `search(..., available_only=True)`, `list_servers(available_only=
True)`, or `health_status(status="available")`. Because probing spawns real
downstream processes, bulk probing is opt-in and scoped (`pinned` → `enabled` →
`all`), concurrency-capped, and gives slow servers a generous timeout.

**Data plane — calling (default: direct):**

By default the gateway is a **catalog, not a proxy** — it hands the client the
coordinates and gets out of the data path, so your servers connect directly to
the host with no abstraction, no credential concentration, and native features
(OAuth, resources, sampling) intact. For the occasional server where zero-config
calling is worth it, flip `set_proxy(id, True)` and the gateway will relay a
single call via `call(id, tool, args)` — reading the live config only at call
time, never storing secrets in a card.

| Model | Callable without host config? | Abstraction in data path | Credentials | Native per-server features |
|-------|------|------|------|------|
| Direct (default) | no — enable in host | none | stay per-server | preserved |
| Proxy (`set_proxy`) | yes | gateway relays call | read live at call time | may be flattened |

Recommendation: keep **direct** as the default; use **latent servers** +
`enable/disable` as your discoverability control plane; reach for **proxy** only
on trusted, non-interactive servers where instant zero-config calling pays off.

## Components

- **`registry/`** — one flat card (`.md` with YAML front-matter) per asset.
  Git-tracked, human-editable, diffable. **Source of truth for content.**
- **`servers.yaml`** — the manifest: discoverability **state** (enabled/proxy
  toggles) that persists across reindexing, plus **latent** servers not yet
  wired into any client. **Source of truth for state.**
- **`indexer.py`** — crawls your MCP configs, skills, agents, commands,
  `.cursor/rules/*.mdc` (global + path-scoped), and nominated folders; refreshes
  cards; applies the manifest; builds a SQLite FTS5 index. Also does CLI toggles
  (`--enable/--disable/--proxy/--list-servers`).
- **`gateway_server.py`** — the small MCP server exposing the discovery, control,
  and data-plane tools. The one thing every client registers. `--selftest`
  runs an in-process client→server check.
- **`cg_common.py`** — shared library imported by every script so they never drift.
- **`cg_insights.py`** — background layer: usage logging, recipes, evidence-based
  suggestions. Nothing here alters a response until it has real data.
- **`cg_health.py`** — probes downstream MCPs (connect / list-tools / latency),
  stores timestamped health snapshots in `state/health.json` with staleness TTL.
- **`recipes.yaml`** — named bundles of cards used together (deterministic co-use).
- **`profiles/`** — YAML scopes (e.g. `dev`, `hydra-ops`) with `scope_types` /
  `allow` / `deny` / `pin`, filtering discovery per agent / workspace.
- **`bootstrap/`** — generated first-prompt blocks, one per profile.
- **`ops/`** — nightly reindex (`reindex.sh`, launchd plist, `install-launchd.sh`).
- **`logs/`** — local usage log (git-ignored).

## Pins & the background insight loop

Pins are **≤5 hand-set anchors per profile** — entry points in the bootstrap, not
a toolkit, and never auto-written. Static keyword auto-pinning is deliberately
avoided: it optimises for word overlap, not for what's actually used or used
together. Instead, value is measured, not guessed:

- **Usage logging** — every `describe`/`load`/`call` is appended to `logs/usage.jsonl`
  with session + profile + timestamp. Zero context cost, fail-safe.
- **Recipes** — you name a bundle of cards used together (`save_recipe`) and pull
  it in one call (`load_recipe`). Deterministic capture of co-use — the reliable
  version of what auto-pinning faked.
- **`suggest(profile)`** — ranks cards by real frequency + recency + session
  co-occurrence, excluding current pins. Returns a `collecting` status until
  enough use has accrued, then honest evidence you can promote to pins.
- **`recommend_profiles()`** — clusters co-occurring cards into candidate profiles
  for you to review and save.
- **`usage_stats()`** — shows what's been captured so far.

So anchors work today; everything else quietly gathers evidence so future
surfacing is grounded in what happened, not keywords.

## State model (what's authoritative)

- **Content** → `registry/*.md` cards. What exists, its description, how to activate.
- **State** → `servers.yaml`. What is discoverable, what may be proxied.
- **Index** → `index.db`. Derived from both, disposable, git-ignored, rebuilt on
  every toggle so the manifest always wins.

Hand-written notes below the `<!-- END GENERATED -->` marker in any card are
preserved across reindexing.

## Quick start

```bash
cd context-gateway
pip install -r requirements.txt

python3 indexer.py --root /Volumes/dev-4tb/AA-GitHub/MCP   # build registry + index
python3 indexer.py --list-servers                          # see servers + state
python3 indexer.py --disable mcp-digitalocean-all          # hide one from discovery
python3 indexer.py --proxy mcp-github-mcp                  # allow proxy calling
python3 gateway_server.py --selftest                       # verify it works
python3 gateway_server.py                                   # run the MCP server (stdio)
```

Then register the server with any client (see `INSTALL.md`) and paste the
relevant `bootstrap/<profile>.md` into that client's CLAUDE.md / project
instructions / system prompt.

## Design principles (your stated preferences)

- **Simple** — flat files + one small server. No DB as source of truth.
- **Durable** — plain markdown in git; survives tooling churn.
- **Easy to maintain** — cards regenerate from source; hand notes are preserved.
- **Scalable** — add cards; search cost stays flat (FTS index).
- **Profiles** — per-agent / per-workspace scoping built in.
- **Fast** — keyword FTS5, no network, no embeddings required.
- **Accurate** — cards written from the real source (tool lists, frontmatter).

## Security

The indexer **never** copies secrets into cards. All `env`, `headers`,
`Authorization`, tokens and API keys in MCP configs are stripped. Cards store
only names, descriptions, tags and *how to activate* — not credentials.

## Benchmarks

Standardised testing for token savings, discoverability, search accuracy, and
latency (gateway vs ~30 weekly MCPs). See:

- [docs/benchmarks/PLAN.md](docs/benchmarks/PLAN.md) — suites, tiers, optimisation loop
- [docs/benchmarks/METRICS.md](docs/benchmarks/METRICS.md) — metric definitions
- [docs/benchmarks/PREREQUISITES.md](docs/benchmarks/PREREQUISITES.md) — validity gates
- [benchmarks/README.md](benchmarks/README.md) — quick commands + dashboard

```bash
./benchmarks/run.sh ci       # PR gate (accuracy + discoverability)
./benchmarks/run.sh nightly  # post-reindex + token estimate
./benchmarks/run.sh full     # includes live MCP cold-start (tier 3)
pytest tests/ -q             # 6 unit/integration tests
```

### Latest results (2026-07-17, MCP monorepo index)

Measured against **1,130 registry cards** (1,091 discoverable): 729 skills, 108
MCP servers, 198 rules, 32 commands, 24 agents. Profile **`dev`**, **29 weekly
MCPs** in token scope. Tier 1–2 (no live MCP cold-start).

| Suite | Metric | Result |
|-------|--------|--------|
| **Tokens** | First-prompt baseline → gateway | **1,164,251 → 3,713** tok (**99.7%** savings) |
| | Baseline breakdown | servers 111K · skills 1.04M · agents 4.5K · commands 4.3K |
| | Gateway bootstrap + tool schemas | 473 + 3,240 tok |
| | First-turn discovery (map + search, informational) | ~976 tok |
| **Discoverability** | Agent workflows pass rate | **10/10 (100%)** |
| | pass@1 / pass@2 | 80% / 100% · mean attempts **1.2** |
| **Accuracy** | MRR (21 golden queries) | **0.35** |
| | Recall@15 / Precision@15 | 0.31 / 0.03 |
| **Latency** | map / search / describe p50 | **1.9 / 0.7 / 1.1 ms** |
| | Full discovery path p50 (map→search→describe) | **3.7 ms** |
| **CI** | `pytest` + `./benchmarks/run.sh ci` | **6/6 passed** · gates **PASS** |

**Not yet measured:** tier-3 parallel cold-start for ~30 live MCPs (`./benchmarks/run.sh full`).
Optional tier-2 tool-count cache (`collect_tool_manifests.py`) not populated — token
server counts use description parse + 200 tok/tool default.

Interactive history: `python3 benchmarks/dashboard/serve.py` → http://127.0.0.1:8765/
Raw JSON: `benchmarks/reports/latest/summary.json`