OmniArchive-MCP

A local daemon that lets AI agents archive web pages to the Wayback Machine in the background, so they don't block on Internet Archive's slow/rate-limited API.

Works with any MCP-compatible client (Kiro, Cursor, Claude Desktop, etc).

Why?

When AI agents do deep research, they read a lot of web pages. Those pages disappear all the time (link rot). Archiving them to the Internet Archive is the obvious fix — but IA's API is slow and rate-limited. If your agent calls it synchronously, it blocks for seconds or gets 429'd.

This tool solves that. The agent calls archive_url, gets a task ID back in milliseconds, and moves on. A background worker handles the actual archiving with proper retry logic.

What it does

• Exposes archive_url and get_archive_status as MCP tools

• Queues requests in local SQLite (survives restarts, no Redis needed)

• Background worker retries with exponential backoff on 429/5xx

• Deduplicates — same URL within 24h won't be re-archived

• Binds to localhost only, no telemetry, nothing phones home

Architecture

AI Client (Kiro/Cursor)
    │ stdio (MCP JSON-RPC)
    ▼
┌─────────────────────┐
│   MCP Server        │  ← Separate process
│   (archive_url,     │
│    get_archive_status)│
└────────┬────────────┘
         │ httpx (localhost:19210)
┌────────▼────────────────────────┐
│   FastAPI Service + Worker      │  ← Main process
│   ┌─────────┐ ┌──────────────┐ │
│   │  Routes  │ │ Background   │ │
│   │  /api/*  │ │ Worker       │ │
│   └────┬─────┘ └──────┬───────┘ │
│        │               │         │
│   ┌────▼───────────────▼───────┐ │
│   │    SQLite (WAL mode)       │ │
│   └────────────────────────────┘ │
└──────────────────────────────────┘
         │
         │ waybackpy
         ▼
   Internet Archive SPN2

Two processes: the MCP server talks stdio with your AI client, and forwards requests over HTTP to the FastAPI service. The FastAPI service manages the queue and runs the background worker.

Getting started

You need Python 3.10+.

git clone https://github.com/your-org/omniarchive-mcp.git
cd omniarchive-mcp
pip install -e ".[dev]"

Start the backend service:

python -m omniarchive_mcp.main

Runs on 127.0.0.1:19210 by default.

Run the tests:

pytest test/ -v   # 84 tests, takes ~12s

MCP tools

archive_url

Queue a URL for archiving.

Returns: { "task_id": "...", "status": "pending", "url": "..." }

get_archive_status

Check on a task. Pass either task_id or url (at least one).

Returns the task status, archive URL (if done), error info (if failed).

Hooking it up to your editor

You need both: the backend service running, AND the MCP server configured in your client.

Kiro — .kiro/settings/mcp.json:

{
  "mcpServers": {
    "omniarchive": {
      "command": "python",
      "args": ["-m", "omniarchive_mcp.mcp_server.main"]
    }
  }
}

Cursor — .cursor/mcp.json:

{
  "mcpServers": {
    "omniarchive": {
      "command": "python",
      "args": ["-m", "omniarchive_mcp.mcp_server.main"]
    }
  }
}

Claude Desktop — claude_desktop_config.json:

{
  "mcpServers": {
    "omniarchive": {
      "command": "python",
      "args": ["-m", "omniarchive_mcp.mcp_server.main"]
    }
  }
}

Don't forget to start the backend first: python -m omniarchive_mcp.main

Configuration

Everything's controlled via env vars (prefix OMNIARCHIVE_):

How it works under the hood

1. Agent calls archive_url → MCP server validates the URL → POSTs to FastAPI

2. FastAPI checks if this URL was already submitted in the last 24h (dedup) → writes to SQLite → returns task ID

3. Worker picks up pending tasks every 5s → calls Internet Archive via waybackpy

4. Success? Stores the archive URL. Got 429/5xx? Backs off and retries. Got 403? Gives up.

License

MIT. See LICENSE.

Contributing

See CONTRIBUTING.md.