web-archive-mcp
This server provides a persistent web archiving service that automatically saves fetched pages and web search results for later retrieval and full-text search.
Fetch & Archive Web Pages: Retrieve any public URL, convert HTML to Markdown, and archive it. Supports optional authentication and timeout configuration; private/internal IPs are blocked for security.
Search & Archive Web Results: Perform DuckDuckGo searches and archive the results automatically.
List Archived Entries: Browse previously saved entries with metadata, optional date filtering and result limits.
Read Archive Files: Access the contents of a specific archive file by its ID, showing newest entries first.
Rebuild Search Index: Trigger a full-text search index rebuild, making archived content searchable through the unified-history-mcp integration.
Allows performing web searches via DuckDuckGo, with results persisted and indexed for later retrieval.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@web-archive-mcp fetch and archive https://example.com/article"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
web-archive-mcp
MCP server for persistent web fetch and search archiving. Every web_fetch and web_search result is saved as timestamped JSONL, indexed by fst-indexer, and searchable via unified-history-mcp.
Part of the Palimpsest investigative toolkit.
Why
web_fetch and web_search results normally evaporate when a session ends. Pages change, get deleted, or get memory-holed. This closes that gap — every result is persisted, content-addressed for dedup, and fed into the same search pipeline as your session logs and transcripts. Three months later, a search(domain="all", query="target name") still finds the page that's been 404'd since July.
Related MCP server: grok-mcp
Architecture
web_fetch / web_search playwright-archive-mcp (browser capture)
│ │
▼ ▼
web-archive-store (shared JSONL write-path + SSRF URL validation)
│
▼
~/.local/share/web-archive/*.jsonl
│ │
│ fst-indexer (Jsonl extractor)
│ │
▼ ▼
returns content index.fst + manifest.json
│
▼
unified-history-mcp
domain: "web-archive"Tools
Tool | Description |
| Fetch a URL, convert to markdown, persist, return |
web_fetch supports the standard HTTP verbs via method (GET, HEAD, POST,
PUT, PATCH, DELETE, OPTIONS; default GET). Pass body for the request body
and content_type for the Content-Type header (e.g. a JSON POST). The
method is stamped on each archived entry and included in the response.
| web_search(query) | Search the web (DuckDuckGo), persist results |
| archive_list(date_from, date_to, max) | List archived entries with metadata |
| archive_read(id, max_entries) | Read entries from an archive file |
| rebuild | Rebuild FST index for the web-archive domain |
Installation
web-archive-mcp depends on the shared web-archive-store package (archive write-path + SSRF URL validation). Install it first:
git clone https://github.com/palimpsest-labs/web-archive-store
cd web-archive-store
python3 -m venv .venv
source .venv/bin/activate
pip install -e .
cd ..
git clone https://github.com/palimpsest-labs/web-archive-mcp
cd web-archive-mcp
python3 -m venv .venv
source .venv/bin/activate
pip install -e .Once web-archive-store is published to PyPI this becomes a single pip install -e ..
Browser-driven traffic capture (the
playwright_*tools) moved to the separateplaywright-archive-mcpserver, which records HTTP traffic into the same store.
Integration with unified-history-mcp
Add to your unified-history TOML config:
[domains.web-archive]
dir = "~/.local/share/web-archive"
pattern = "*.jsonl"
extractor = "jsonl"
label = "web-archive entry"
filters = []Then rebuild: search(domain="web-archive", query="rebuild") or call rebuild directly.
Once indexed, a search(domain="all", query="your search") scans your sessions, transcripts, notifications, and every web page you've ever fetched — in a single query.
Entry format
{
"type": "fetch",
"source": "https://example.com/page",
"title": "Example Page",
"content": "# Example\n\nMarkdown content...",
"timestamp": "2026-07-30T21:15:00Z",
"content_hash": "abc123..."
}For searches, source holds the query string and type is "search".
Content-addressed dedup prevents storing identical entries. Same source + same content hash = skipped.
License
MIT
Available Tools
5 toolsarchive_listA
List archived web fetch/search entries with metadata.
Args: date_from: Optional start of date range (YYYY-MM-DD) date_to: Optional end of date range (YYYY-MM-DD, inclusive) max_results: Maximum entries to show (default 50)
Inputs: date_to, date_from, max_results. Returns: result
archive_readA
Read entries from an archive file.
Args: file_id: Archive file name (e.g., '2026-07-30-fetch-example.jsonl') max_entries: Maximum entries to return, newest first (default 50)
Inputs: file_id (required), max_entries. Returns: result
rebuildA
Rebuild the FST index for the web-archive domain.
Calls fst-indexer to rebuild the full-text index so archived content is searchable via unified-history-mcp.
Inputs: no named parameters. Returns: result
web_fetchA
Fetch a URL and archive the result.
Fetches the URL, converts HTML to markdown, and persists the result as a timestamped JSONL entry in the web-archive. Only http and https URLs are allowed; private/internal IPs are blocked.
Args: url: The URL to fetch (http/https only) timeout: Request timeout in seconds (default 30, max 120) token: Optional Bearer token for authenticated requests
Inputs: url (required), token, timeout. Returns: result
web_searchA
Search the web and archive the results.
Performs a web search via DuckDuckGo, persists the results, and returns them formatted. The results are archived for later search.
Args: query: The search query (max 500 chars)
Inputs: query (required). Returns: result
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 5 tools
Each tool has a clearly distinct purpose: fetching a URL, searching the web, listing archives, reading archive entries, and rebuilding the index. No overlap or ambiguity exists between them.
Most tools follow a verb_noun pattern (web_fetch, web_search, archive_list, archive_read), but 'rebuild' is a single verb that breaks the pattern and mixes with the web_/archive_ prefixes. The inconsistency is minor and does not hinder comprehension.
Five tools is well-scoped for a web-archive server, covering core operations (fetch, search, list, read, maintain) without unnecessary bloat.
The tool surface covers the full lifecycle of archiving web content: creating archive entries via web_fetch and web_search, reading them via archive_list and archive_read, and maintaining the search index with rebuild. No obvious gaps exist for the stated purpose.
Maintenance
Related MCP Connectors
One MCP server for all web data. Scrape, search, crawl, and AI research in one install.
Hosted Web Crawler MCP server for Claude, ChatGPT and Cursor, with each action switched per account.
Social media MCP server: your Instagram, TikTok, YouTube, LinkedIn and Threads history for your AI.
Related MCP Servers
- AlicenseNot gradedqualityCmaintenanceMCP server that fetches web pages, extracts clean markdown (reducing token count), caches results, and provides searchable reading history.5 npmMIT
- FlicenseNot gradedqualityDmaintenanceMCP server providing web search, news search, and X/Twitter search capabilities via HTTP or stdio.-
- AlicenseAqualityDmaintenanceAn MCP server that fetches web pages and extracts clean, AI-usable context from them, enabling tools for link discovery, content search, and integrated fetch-and-search operations.514 npm1MIT
- AlicenseBqualityDmaintenanceAn MCP server that enables coding agents to log their work, decisions, test runs, PRs, and progress as structured events, either to a local JSONL file or an HTTP endpoint.6MIT