Skip to main content
Glama
web-archive-mcp

web-archive-mcp

by jmars

web-archive-mcp

MCP server for persistent web fetch and search archiving. Every web_fetch and web_search result is saved as timestamped JSONL, indexed by fst-indexer, and searchable via unified-history-mcp.

Part of the Palimpsest investigative toolkit.

Why

web_fetch and web_search results normally evaporate when a session ends. Pages change, get deleted, or get memory-holed. This closes that gap — every result is persisted, content-addressed for dedup, and fed into the same search pipeline as your session logs and transcripts. Three months later, a search(domain="all", query="target name") still finds the page that's been 404'd since July.

Related MCP server: Claw Recall

Architecture

web_fetch / web_search          playwright-archive-mcp (browser capture)
        │                                │
        ▼                                ▼
  web-archive-store  (shared JSONL write-path + SSRF URL validation)
        │
        ▼
  ~/.local/share/web-archive/*.jsonl
        │                                │
        │                        fst-indexer (Jsonl extractor)
        │                                │
        ▼                                ▼
  returns content              index.fst + manifest.json
                                       │
                                       ▼
                              unified-history-mcp
                              domain: "web-archive"

Tools

Tool

Description

web_fetch(url, timeout)

Fetch a URL, convert to markdown, persist, return

web_search(query)

Search the web (DuckDuckGo), persist results

archive_list(date_from, date_to, max)

List archived entries with metadata

archive_read(id, max_entries)

Read entries from an archive file

rebuild

Rebuild FST index for the web-archive domain

Installation

web-archive-mcp depends on the shared web-archive-store package (archive write-path + SSRF URL validation). Install it first:

git clone https://github.com/palimpsest-labs/web-archive-store
cd web-archive-store
python3 -m venv .venv
source .venv/bin/activate
pip install -e .

cd ..
git clone https://github.com/palimpsest-labs/web-archive-mcp
cd web-archive-mcp
python3 -m venv .venv
source .venv/bin/activate
pip install -e .

Once web-archive-store is published to PyPI this becomes a single pip install -e ..

Browser-driven traffic capture (the playwright_* tools) moved to the separate playwright-archive-mcp server, which records HTTP traffic into the same store.

Integration with unified-history-mcp

Add to your unified-history TOML config:

[domains.web-archive]
dir = "~/.local/share/web-archive"
pattern = "*.jsonl"
extractor = "jsonl"
label = "web-archive entry"
filters = []

Then rebuild: search(domain="web-archive", query="rebuild") or call rebuild directly.

Once indexed, a search(domain="all", query="your search") scans your sessions, transcripts, notifications, and every web page you've ever fetched — in a single query.

Entry format

{
  "type": "fetch",
  "source": "https://example.com/page",
  "title": "Example Page",
  "content": "# Example\n\nMarkdown content...",
  "timestamp": "2026-07-30T21:15:00Z",
  "content_hash": "abc123..."
}

For searches, source holds the query string and type is "search".

Content-addressed dedup prevents storing identical entries. Same source + same content hash = skipped.

License

MIT

Available Tools

17 tools
archive_listA

List archived web fetch/search entries with metadata.

Args: date_from: Optional start of date range (YYYY-MM-DD) date_to: Optional end of date range (YYYY-MM-DD, inclusive) max_results: Maximum entries to show (default 50)

Inputs: date_to, date_from, max_results. Returns: result

archive_readA

Read entries from an archive file.

Args: file_id: Archive file name (e.g., '2026-07-30-fetch-example.jsonl') max_entries: Maximum entries to return, newest first (default 50)

Inputs: file_id (required), max_entries. Returns: result

playwright_backA

Go back in the interactive session's history.

Inputs: no named parameters. Returns: result

playwright_clickA

Click an element (CSS selector) in the interactive session.

Inputs: selector (required). Returns: result

playwright_closeA

Close the interactive Playwright session and its browser.

Inputs: no named parameters. Returns: result

playwright_fillA

Fill a form field (CSS selector) with a value in the interactive session.

Inputs: value (required), selector (required). Returns: result

playwright_forwardB

Go forward in the interactive session's history.

Inputs: no named parameters. Returns: result

playwright_htmlA

Return the HTML of the current page in the interactive session.

Inputs: max_len. Returns: result

playwright_navigateA

Navigate the interactive session to a URL (http/https; scheme is auto-prepended). Returns the page title. Every request/response on the session is recorded automatically. Like web_fetch, private/loopback addresses are rejected (SSRF protection); use the standalone CLI if you need to reach an internal/local host.

Inputs: url (required). Returns: result

playwright_recordA

Drive a headless Playwright browser against the given URL(s) and record every HTTP request/response into the web-archive store.

Binary/streaming response bodies are skipped, auth headers are redacted by default, and each URL gets a fresh browser context so cookies don't leak between sites. Like web_fetch, this rejects private/loopback addresses (SSRF protection). Recorded entries become searchable once rebuild runs.

Args: urls: URL(s) to visit (http/https; scheme auto-prepended) wait: Extra seconds to wait after page load for async requests timeout: Navigation timeout in seconds (max 120) max_entries: Stop recording after this many request/response entries redact_auth: Redact Authorization/Cookie/Set-Cookie/X-API-Key headers

Inputs: urls (required), wait, timeout, max_entries, +1 more. Returns: result

playwright_screenshotA

Save a screenshot of the current page to ~/Downloads and return the path.

The filename is sanitized (directory components stripped, .png enforced) so it cannot escape ~/Downloads.

Inputs: name. Returns: result

playwright_startA

Start a persistent interactive Playwright session with always-on traffic recording. Every response observed on the session is archived to the web-archive store in real time. Returns a confirmation.

Inputs: no named parameters. Returns: result

playwright_statsA

Report the interactive session's status and recorded-entry counts.

Inputs: no named parameters. Returns: result

playwright_textB

Return the visible text of the current page in the interactive session.

Inputs: max_len. Returns: result

rebuildA

Rebuild the FST index for the web-archive domain.

Calls fst-indexer to rebuild the full-text index so archived content is searchable via unified-history-mcp.

Inputs: no named parameters. Returns: result

web_fetchA

Fetch a URL and archive the result.

Fetches the URL, converts HTML to markdown, and persists the result as a timestamped JSONL entry in the web-archive. Only http and https URLs are allowed; private/internal IPs are blocked.

Args: url: The URL to fetch (http/https only) timeout: Request timeout in seconds (default 30, max 120) token: Optional Bearer token for authenticated requests

Inputs: url (required), token, timeout. Returns: result

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.7/5.0

    Scored across 17 tools

    Disambiguation4/5

    Most tools have clearly distinct purposes, but playwright_text and playwright_html both return page content (just different formats), and web_fetch and playwright_navigate can both be used to retrieve a URL. The descriptions do a good job mitigating confusion, so the overlap is minor.

    Naming Consistency4/5

    Tool names use a consistent prefix (web_, playwright_, archive_) with action-oriented suffixes. Slight inconsistency exists because archive_list and archive_read are not verb-first, and a few playwright_ commands use nouns (text, html, screenshot). Overall the pattern is still predictable and readable.

    Tool Count3/5

    17 tools is on the heavier side, mostly due to the 12 playwright interaction commands. Each interaction command serves a distinct browser automation need, but the count is borderline-high for an MCP server surface.

    Completeness4/5

    The core workflows of web fetching, searching, recording, interactive browser control, and archive listing/reading are covered well. Gaps include no delete or clear operation for archived entries, and direct search within archives relies on an external indexer (rebuild + unified-history-mcp) rather than a native search tool.

    Maintenance

    ActivityInactive
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • F
      license
      Not graded
      quality
      Not graded
      maintenance
      Provides cross-session memory for AI agents by maintaining a rolling 24-hour event stream and searchable daily archives to prevent context loss. It enables agents to record, query, and retrieve historical events and decisions through a structured markdown-based workspace.
      1 npm
      -
    • A
      license
      Not graded
      quality
      F
      maintenance
      Persistent, searchable memory for AI agents. Enables agents to recover context after compaction by searching indexed conversations, emails, and files via MCP tools.
      1
      MIT
    • A
      license
      A
      quality
      D
      maintenance
      Enables AI agents to maintain a persistent, queryable memory stored as user-owned Markdown files, with dual-channel retrieval (FTS5 and optional semantic search) and an audited write pipeline.
      11
      MIT
    • A
      license
      Not graded
      quality
      B
      maintenance
      Enables AI agents to store, search, consolidate, and trace durable memories as plain Markdown files, with a disposable BM25 index and provenance tracking via MCP tools.
      MIT