Skip to main content
Glama

web-archive-mcp

MCP server for persistent web fetch and search archiving. Every web_fetch and web_search result is saved as timestamped JSONL, indexed by fst-indexer, and searchable via unified-history-mcp.

Part of the Palimpsest investigative toolkit.

Why

web_fetch and web_search results normally evaporate when a session ends. Pages change, get deleted, or get memory-holed. This closes that gap — every result is persisted, content-addressed for dedup, and fed into the same search pipeline as your session logs and transcripts. Three months later, a search(domain="all", query="target name") still finds the page that's been 404'd since July.

Related MCP server: grok-mcp

Architecture

web_fetch / web_search          playwright-archive-mcp (browser capture)
        │                                │
        ▼                                ▼
  web-archive-store  (shared JSONL write-path + SSRF URL validation)
        │
        ▼
  ~/.local/share/web-archive/*.jsonl
        │                                │
        │                        fst-indexer (Jsonl extractor)
        │                                │
        ▼                                ▼
  returns content              index.fst + manifest.json
                                       │
                                       ▼
                              unified-history-mcp
                              domain: "web-archive"

Tools

Tool

Description

web_fetch(url, timeout, token, preview, redact_html, method, body, content_type)

Fetch a URL, convert to markdown, persist, return

web_fetch supports the standard HTTP verbs via method (GET, HEAD, POST, PUT, PATCH, DELETE, OPTIONS; default GET). Pass body for the request body and content_type for the Content-Type header (e.g. a JSON POST). The method is stamped on each archived entry and included in the response. | web_search(query) | Search the web (DuckDuckGo), persist results | | archive_list(date_from, date_to, max) | List archived entries with metadata | | archive_read(id, max_entries) | Read entries from an archive file | | rebuild | Rebuild FST index for the web-archive domain |

Installation

web-archive-mcp depends on the shared web-archive-store package (archive write-path + SSRF URL validation). Install it first:

git clone https://github.com/palimpsest-labs/web-archive-store
cd web-archive-store
python3 -m venv .venv
source .venv/bin/activate
pip install -e .

cd ..
git clone https://github.com/palimpsest-labs/web-archive-mcp
cd web-archive-mcp
python3 -m venv .venv
source .venv/bin/activate
pip install -e .

Once web-archive-store is published to PyPI this becomes a single pip install -e ..

Browser-driven traffic capture (the playwright_* tools) moved to the separate playwright-archive-mcp server, which records HTTP traffic into the same store.

Integration with unified-history-mcp

Add to your unified-history TOML config:

[domains.web-archive]
dir = "~/.local/share/web-archive"
pattern = "*.jsonl"
extractor = "jsonl"
label = "web-archive entry"
filters = []

Then rebuild: search(domain="web-archive", query="rebuild") or call rebuild directly.

Once indexed, a search(domain="all", query="your search") scans your sessions, transcripts, notifications, and every web page you've ever fetched — in a single query.

Entry format

{
  "type": "fetch",
  "source": "https://example.com/page",
  "title": "Example Page",
  "content": "# Example\n\nMarkdown content...",
  "timestamp": "2026-07-30T21:15:00Z",
  "content_hash": "abc123..."
}

For searches, source holds the query string and type is "search".

Content-addressed dedup prevents storing identical entries. Same source + same content hash = skipped.

License

MIT

Available Tools

5 tools
archive_listA

List archived web fetch/search entries with metadata.

Args: date_from: Optional start of date range (YYYY-MM-DD) date_to: Optional end of date range (YYYY-MM-DD, inclusive) max_results: Maximum entries to show (default 50)

Inputs: date_to, date_from, max_results. Returns: result

archive_readA

Read entries from an archive file.

Args: file_id: Archive file name (e.g., '2026-07-30-fetch-example.jsonl') max_entries: Maximum entries to return, newest first (default 50)

Inputs: file_id (required), max_entries. Returns: result

rebuildA

Rebuild the FST index for the web-archive domain.

Calls fst-indexer to rebuild the full-text index so archived content is searchable via unified-history-mcp.

Inputs: no named parameters. Returns: result

web_fetchA

Fetch a URL and archive the result.

Fetches the URL, converts HTML to markdown, and persists the result as a timestamped JSONL entry in the web-archive. Only http and https URLs are allowed; private/internal IPs are blocked.

Args: url: The URL to fetch (http/https only) timeout: Request timeout in seconds (default 30, max 120) token: Optional Bearer token for authenticated requests

Inputs: url (required), token, timeout. Returns: result

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A4.2/5.0

    Scored across 5 tools

    Disambiguation5/5

    Each tool has a clearly distinct purpose: fetching a URL, searching the web, listing archives, reading archive entries, and rebuilding the index. No overlap or ambiguity exists between them.

    Naming Consistency4/5

    Most tools follow a verb_noun pattern (web_fetch, web_search, archive_list, archive_read), but 'rebuild' is a single verb that breaks the pattern and mixes with the web_/archive_ prefixes. The inconsistency is minor and does not hinder comprehension.

    Tool Count5/5

    Five tools is well-scoped for a web-archive server, covering core operations (fetch, search, list, read, maintain) without unnecessary bloat.

    Completeness5/5

    The tool surface covers the full lifecycle of archiving web content: creating archive entries via web_fetch and web_search, reading them via archive_list and archive_read, and maintaining the search index with rebuild. No obvious gaps exist for the stated purpose.

    Maintenance

    ActivityInactive
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      C
      maintenance
      MCP server that fetches web pages, extracts clean markdown (reducing token count), caches results, and provides searchable reading history.
      5 npm
      MIT
    • F
      license
      Not graded
      quality
      D
      maintenance
      MCP server providing web search, news search, and X/Twitter search capabilities via HTTP or stdio.
      -
    • A
      license
      A
      quality
      D
      maintenance
      An MCP server that fetches web pages and extracts clean, AI-usable context from them, enabling tools for link discovery, content search, and integrated fetch-and-search operations.
      5
      14 npm
      1
      MIT