# ietf-mcp-worker

An IETF research MCP for Cloudflare Workers, written in TypeScript and Effect.
Public readers share corpora, origin fetches, refresh jobs, and embedding work.
Code mode is the default: discover an operation, compose calls in JavaScript,
and return only the evidence the model needs.

Based on [mnot/ietf-llm](https://github.com/mnot/ietf-llm) at
`63b61f4f289db7b53201d9153760924a04890114` and its
[code-mode proposal](https://github.com/mnot/ietf-llm/pull/356).
This is a new storage and execution design, not a Python compatibility layer.
See [coverage](docs/coverage.md) for differences and remaining limitations.

## Run locally

Requires Node 24 or newer.

```sh
npm ci
printf 'ADMIN_TOKEN=replace-with-a-local-secret\n' > .dev.vars
npm run dev
```

Connect an MCP client to `http://localhost:8787/mcp`. Reads are public, and
anyone can gather a missing corpus with `start_gather` (for example
`{"name":"httpbis"}`). Public gathers draw on a shared daily budget
(`DAILY_GATHERS`), and a corpus finished in the last 24 hours is returned as is.
The operator token is required to cancel jobs, set refresh schedules, change an
operator-registered corpus, and gather the `rfcs` and `groups` mirrors.

A group covered by the [ietf-llm seed store](https://seed-store.mnot.net/v11/index.json)
starts from its verified prebuilt snapshot, readable within minutes. The full
gather starts automatically afterwards and replaces it when it publishes.

Each full gather then derives two record kinds. `people` merges author aliases
using Datatracker person ids (from email addresses and GitHub usernames), then
identical multi-word display names. `topics` clusters threads and issues with
k-means over Workers AI embeddings. `summarize` writes an on-demand, cached
summary of a thread or issue. `grep_corpus` also accepts RE2 regular
expressions. `get_rfc_info` returns the full RFC reference graph from rfc.fyi.

```sh
export IETF_MCP_URL=http://localhost:8787/mcp
export IETF_MCP_TOKEN=replace-with-a-local-secret
node scripts/client.ts gather '{"name":"httpbis","group":"httpbis","since":"2026-01-01","refreshHours":24}'
node scripts/client.ts status httpbis
node scripts/client.ts search httpbis 'cache invalidation'
```

For a small first smoke test, gather `{"name":"http-caching","rfcs":[9111]}`.
Wait for `job.state` to be `done`. A failed refresh keeps the previous snapshot.
Full groups can involve thousands of fetches; the alarm processes one persisted
task at a time. Check status for the exact failing source when a gather fails.

Custom corpora accept `drafts`, `lists`, `repos`, and `rfcs`, in addition to
`group`. A name without sources is interpreted as a group. Standalone lists
should explicitly set `lists`. `groups` gathers the chartered-group mirror;
`rfcs` gathers the published RFC series. Both are substantial initial jobs.

```js
// execute tool: model-written JavaScript, isolated from credentials and networking
async () => {
  const result = await ietf.call("chunks", {
    corpus: "httpbis",
    kind: "threads",
    since: "2026-01-01",
    chars: 0,
    limit: 100,
  });
  return {
    sendersOnThisPage: [...new Set(result.items.map((item) => item.author))],
    nextOffset: result.nextOffset,
  };
};
```

`search` discovers operation names and schemas; it is not a corpus search.
The generated function calls `ietf.call(operationName, arguments)`. Operations
include orientation, search, exact matching, structured records, drafts, RFCs,
issues/PRs, meetings, live Datatracker records, and norms. Use
`/mcp/classic` to expose these operations directly without sandbox charges.

## Skills, CLI and export

The four upstream IETF skills and their reference documents are preserved with
attribution. They are exposed as MCP resources, skill prompts, and `read_skill`.

```sh
node scripts/client.ts install-skills .agents/skills
node scripts/client.ts stdio
node scripts/client.ts call overview '{"corpus":"httpbis"}'
node scripts/client.ts export httpbis ./httpbis-export
```

The stdio command bridges clients to the same hosted data. Export produces
Markdown files and a citation manifest suitable for manual NotebookLM upload.
Installation and export refuse to overwrite existing files.

## Deploy

```sh
npm run check
npx wrangler login
npx wrangler secret put ADMIN_TOKEN
npm run deploy
```

For a free temporary preview, use `npm run deploy:temporary`. This uses
`wrangler.preview.jsonc`, which omits Worker Loader and scheduled refresh.
The preview exposes direct tools at `/mcp`; production keeps code mode as the
default. Cloudflare rejected native code mode on the temporary account with
error 10195: Dynamic Workers require a paid plan. Set the preview's operator
secret with `npx wrangler secret put ADMIN_TOKEN --temporary --config
wrangler.preview.jsonc`. Temporary credentials and claim URLs belong outside
this repository; unclaimed previews expire after 60 minutes.

The default configuration has no model calls or Vectorize dependency. Search
explicitly reports lexical ranking until semantic search is configured:

```sh
npx wrangler vectorize create ietf-passages --dimensions=384 --metric=cosine
```

Add these entries to `wrangler.jsonc`, and add `"EMBEDDINGS": "on"` to `vars`:

```json
{
  "ai": { "binding": "AI" },
  "vectorize": [{ "binding": "VECTORS", "index_name": "ietf-passages" }]
}
```

Redeploy and gather the desired corpora. Workers AI uses
`@cf/baai/bge-small-en-v1.5` (384 dimensions). Text embeddings are cached by
model and content hash across corpora; unchanged documents avoid Vectorize
upserts. Vector results are checked against the published text hash so staged
or obsolete records cannot leak into answers. Vectorize is eventually consistent.
An optional `GITHUB_TOKEN` secret increases GitHub's upstream rate allowance.

## Bounds and costs

- Public requests: 30/minute per client IP, enforced by Cloudflare's regional
  rate limiter. This is not a globally exact per-IP quota.
- Code mode: **100 executions/day globally** by default; set `DAILY_EXECUTIONS`
  to change it. The counter is atomic across users. Classic reads still work
  when the budget is exhausted.
- Each execution: 100 ms CPU, 10 seconds async deadline, 40 host operations,
  no outbound network, 16,000 code characters, 24,000 response characters.
- Fetches: two concurrent requests per origin, conditional GET, shared cache,
  bounded retries, and shared `Retry-After` cooldown.
- Sources: 8 MiB fetched body, 300 PDF pages, 1.9 MB document row; exceeding a
  bound fails the refresh instead of silently publishing an incomplete result.

At 100 executions/day for 30 days, plan for about **$9/month**, plus storage,
AI, and other usage beyond included allowances. The dominant variable is the
Dynamic Worker creation fee, not embedding inference. See the assumptions,
larger scenarios, and optimization options in [costs](docs/costs.md).

## Verify

```sh
npm run check
node scripts/cost.ts 3000
# Against a running local or deployed service, with operator credentials:
node --env-file=.dev.vars scripts/smoke.ts
```

Tests run in Cloudflare's local runtime and cover writer-to-reader ingestion,
shared-fetch deduplication, failed/cancelled refreshes, atomic execution quotas,
PDF text extraction, pagination, trust boundaries, and actual sandbox/MCP calls.
Live Workers AI and Vectorize require a configured Cloudflare account.
See the [validation record](docs/validation.md) for what passed locally and
on the temporary Cloudflare deployment.

## Design

`Corpus` Durable Objects own each corpus's SQLite records, FTS index, task queue,
and publication pointer. `SharedFetch` objects coordinate origin traffic and
cache HTTP/mail/embedding results. Effect models upstream effects, retries,
timeouts, and concurrency. MCP uses the official SDK and Cloudflare Code Mode.
There is no per-user corpus, container, polling server, or always-on process.
See [architecture](docs/architecture.md).

The software is MIT licensed with upstream attribution. Vendored skill text is
CC-BY-4.0; reproduced RFCs retain their separate notices. See
[skills/VENDORED.md](skills/VENDORED.md).