Internet Archive MCP server

A remote MCP server that gives a chat client a full Internet Archive query surface: enumerate historical captures of a URL, collapse them into the handful of times the page actually changed, read a capture as clean text, diff two captures against each other, and search archive.org items. It is built to be added as a custom connector in claude.ai and to run on Replit Autoscale.

The driving use case: reconstructing the edit history of a documentation or support page that is rewritten in place at a stable URL, where search engines only ever surface the current version.

• Stack: TypeScript (strict), Node 22+, @modelcontextprotocol/sdk, Express, Zod

• Transport: Streamable HTTP, stateless, MCP protocol 2025-11-25

• Storage: none — see Why there is no database

Why results are small

Custom-connector tool results are capped at roughly 30,000 tokens (~150,000 characters). A single archived HTML page routinely exceeds that several times over, and when a result breaches the cap the call fails outright — it does not truncate gracefully.

So this server reduces server-side and returns handles for the rest:

Concretely:

• No tool ever returns raw HTML in content.

• Extracted text over 8,000 characters comes back as a 2,000-character preview plus a ResourceLink and a totalChars count.

• Diffs are capped at 15,000 characters, with a ResourceLink to the full diff beyond that.

• Table-shaped results (capture rows, revisions) are trimmed to 250 rows, and the tool tells the caller how to page.

Related MCP server: MCP Wayback Machine Server

Tools

Cheap orientation tools first. Every tool declares an outputSchema and populates structuredContent.

archive_stats — recommended first call

How much history exists for a URL: total captures, first and last capture, per-year breakdown. One request, no page content.

check_availability

The closest capture to a date.

search_snapshots

The general CDX capture-index query. Returns rows as objects — timestamp, original, statuscode, mimetype, digest, length, snapshotUrl — never a raw CDX text blob.

list_revisions — the highest-value tool here

Distinct content revisions of a URL. A page with hundreds of captures may have only eight distinct bodies; this returns one row per revision with revisionIndex, digest, firstSeen, lastSeen and captureCount. Feed two firstSeen values to compare_snapshots.

get_snapshot

One capture as chrome-stripped text or markdown. Fetched with the id_ modifier (the original bytes, not the Wayback-wrapped replay page). Navigation, headers, footers, cookie banners and language switchers are removed before extraction.

format=raw never inlines the bytes; it returns metadata, a text preview and a ResourceLink.

compare_snapshots

Diffs two captures and returns only what changed: added/removed characters, changed-section count, the Wayback visual-diff URL, and a unified diff capped at 15,000 characters. Both captures are fetched via id_ and stripped of chrome first, so the diff shows content changes rather than banner noise.

list_screenshots

Screenshot captures for a URL — timestamps and URLs only, never image bytes. Most URLs have none; an empty result is normal.

search_items

archive.org Advanced Search over items (texts, audio, movies, software, …).

get_item_metadata

An item's metadata plus a file listing (name, format, size) — not file contents.

save_url — only registered when ENABLE_SAVE=true

Save Page Now. The only tool that writes to the Internet Archive.

clear_cache

Flushes cached upstream responses. No parameters.

HTTP resource routes

The artifacts that ResourceLink URIs point at. All are gated behind the same path secret as the MCP endpoint, and all re-derive their content from archive.org on every request — so resource links never expire and survive scale-to-zero without any persistence layer.

{timestamp} accepts latest, earliest or any date form. The target URL may be percent-encoded, passed un-encoded in the path tail, or supplied as ?url=. Responses carry Cache-Control: public, max-age=86400 because captures are immutable.

Running locally

npm install
cp .env.example .env      # then edit CONTACT_EMAIL
npm run dev               # tsx, no build step

The boot log prints the full connector URL, including the path secret, once:

MCP endpoint:      http://localhost:3000/mcp/<secret>

Other scripts:

npm run build       # tsc -> dist/
npm start           # node dist/index.js
npm test            # 216 unit + integration tests, no network access
npm run typecheck   # strict typecheck including the test suite

Verifying with the MCP Inspector

npx @modelcontextprotocol/inspector
# transport: Streamable HTTP
# URL:       http://localhost:3000/mcp/<secret>

Or headless:

npx @modelcontextprotocol/inspector --cli http://localhost:3000/mcp/<secret> \
  --transport http --method tools/list

Deploying to Replit Autoscale

1. Import this repository into Replit (Create → Import from GitHub).

2. Check .replit. It is already correct and should not need editing:

   modules = ["nodejs-22"]
   
   run = "npm run dev"
   entrypoint = "src/index.ts"
   
   [deployment]
   deploymentTarget = "autoscale"
   build = ["npm", "run", "build"]
   run = ["npm", "run", "start"]
   
   [[ports]]
   localPort = 3000
   externalPort = 80

   Autoscale exposes exactly one external port, and it must be 80. If you add any other externalPort entry the deployment fails. localPort must match what the server listens on (3000, or whatever PORT you set). A mismatch presents as "an open port was not detected" with no other diagnostic.

3. Set the workspace secrets (Tools → Secrets) from the table below, at minimum CONTACT_EMAIL and MCP_PATH_SECRET.

4. Run it in the workspace and confirm the webview shows the server-info JSON.

5. Deploy (Deploy → Autoscale, build npm run build, run npm run start).

6. ⚠️ Re-enter every secret in the Deployments pane. Workspace secrets do not propagate to deployments. This is the single most common post-deploy failure: the server starts, but DEPLOY_URL is wrong so resource links point at localhost, and MCP_PATH_SECRET is regenerated on every cold start so your saved connector URL stops working.

7. Set DEPLOY_URL to the deployment's public URL, e.g. https://your-app.replit.app. (If left unset it is inferred from REPLIT_DOMAINS, which is usually right, but setting it explicitly is safer.)

8. Verify from outside: curl https://your-app.replit.app/healthz → ok.

9. Add the connector in claude.ai: Settings → Connectors → Add custom connector → paste https://your-app.replit.app/mcp/<MCP_PATH_SECRET>.

Secrets

All query tools work anonymously. No Internet Archive account is needed.

Further optional settings: ALLOWED_ORIGINS (extra browser origins), MCP_SSE=true (stream MCP responses as SSE instead of plain JSON), RATE_LIMIT_PER_MINUTE (default 10), UPSTREAM_TIMEOUT_MS (default 25000), PORT, HOST.

Authentication

Optimised for "the connector attaches on the first try":

1. Path secret (primary). MCP is served at /mcp/{MCP_PATH_SECRET} and resources at /r/{MCP_PATH_SECRET}/.... A 32-character URL-safe secret is generated and logged once at boot if you do not supply one. A wrong secret returns 404, indistinguishable from an unknown route.

2. Bearer token (optional, off by default). Setting MCP_AUTH_TOKEN requires Authorization: Bearer <token> in addition to the path secret. It defaults to unset because it is unconfirmed that claude.ai's Add-connector dialog accepts a static bearer token, and shipping bearer-required by default risks a connector that silently will not attach.

3. Origin policy. Requests with no Origin header are allowed — Anthropic calls your server from its own infrastructure, server-to-server, and sends none. Browser origins are checked against claude.ai, anthropic.com, localhost, your DEPLOY_URL and ALLOWED_ORIGINS.

Both checks live behind an AuthProvider interface, so OAuth can be added later without touching transport or routing code.

Why there is no database

archive.org is already the durable store. Every resource URI re-derives its content from upstream on demand, so nothing needs to survive a restart for links to keep working. Adding Replit DB, Object Storage, Postgres or Redis would buy a marginal cache-hit improvement in exchange for setup steps, secrets, another failure mode and cold-start latency.

What ships instead, behind swappable interfaces:

• InMemoryCache (CacheBackend), keyed on the full upstream URL, with these deliberately asymmetric TTLs:

  Resource	TTL	Reason
  Snapshot content	24h	immutable once captured
  CDX / availability / sparkline	1h	append-only, never mutates
  Save Page Now status	30s	changes while a job runs

• InMemoryTokenBucket (RateLimiter), 10 requests/minute against archive.org, honouring Retry-After on 429 and surfacing a structured rate_limited error rather than throwing.

Limitations

• The cache and rate limiter are per-instance and per-process. They are lost on restart and on Autoscale's scale-to-zero, and two concurrent instances do not share them (so the effective upstream rate can be 10/minute per instance). This is accepted; see above.

• The first connector handshake after an idle period will probably time out. Autoscale scales to zero after ~15 minutes idle and cold-starts in 10–30s, while the client wants a response sooner. Retrying once connects. If it becomes annoying, move to a Reserved VM rather than optimising further.

• Tool results are size-capped. Page text over 8,000 characters and diffs over 15,000 characters are not inlined; you get a preview plus a ResourceLink. Row-shaped results are trimmed to 250 rows and tell you how to page. This is the constraint the whole design exists to satisfy — an oversized result would fail the call outright.

• Whether claude.ai resolves a ResourceLink from a tool result is unverified. If it does not, nothing breaks: archive_stats, list_revisions and compare_snapshots return everything needed inline, and the /r/... routes stay useful from Claude Code, Cowork and a browser. See DECISIONS-MADE.md.

• Chrome stripping is heuristic. nav, header, footer, aside, cookie banners and language switchers are removed, which is what makes diffs readable — but a page that puts real content in a <header> will lose it. Use format=raw (via the resource route) to see the untouched capture.

• list_screenshots is unverified against the live CDX index. It queries the screenshot: pseudo-URL, which is how Save Page Now screenshots are indexed; it could not be exercised against the real service from the build environment.

• list_revisions groups consecutive identical digests. A page that reverts to an earlier body reports two separate revisions with the same digest, which is the honest answer — distinctDigests tells you how many unique bodies there were.

Testing

npm test

216 tests, no network access: unit tests over timestamp normalisation, CDX parsing, HTML extraction (including nav and language-list stripping), the 8,000-character ResourceLink threshold, diff capping, cache TTL and eviction, the token bucket, auth and origin policy, and the upstream client's retry, Retry-After and byte-cap behaviour — plus an end-to-end suite that drives the real MCP protocol over HTTP against a fixture Internet Archive.

The end-to-end suite covers the acceptance case with fixtures: ~300 captures of a help-centre page with 8 distinct bodies wrapped in a 13-language switcher, asserting that list_revisions returns 8 rows rather than hundreds and that compare_snapshots surfaces the changed message-limit sentence with no language or navigation noise in the diff.

Acceptance test (manual, against live archive.org)

Run this in claude.ai once the connector is attached:

Enumerate the distinct content revisions of support.anthropic.com/en/articles/8325612 from 2023-09-01 to today, then diff the earliest against the latest.

Pass criteria: list_revisions returns roughly 5–20 digest-distinct revisions, not hundreds. compare_snapshots returns a readable diff in which the Claude Pro message-limit language visibly changes — the older capture mentioning 100 messages per 8 hours, the newer 45 messages per 5 hours.

If it fails, diagnose in this order:

Licence and provenance

MIT. Written against the Internet Archive's published API documentation; no code was taken from any existing MCP server. See PROVENANCE.md.