web-archive-mcp
Web Fetching: Fetch and archive any HTTP/HTTPS URL, converting content to markdown with SSRF protection.
Web Search: Perform DuckDuckGo searches and archive results.
Browser Automation: Drive a headless browser with Playwright: record traffic, start interactive sessions, navigate, click, fill forms, extract text/HTML, screenshot, and manage history.
Archive Management: List and read archived entries with metadata and date filtering.
Index Rebuilding: Rebuild full-text search index for integration with unified-history-mcp, enabling cross-domain search.
Persistent Storage: All data saved as timestamped JSONL in
~/.local/share/web-archive/with content-addressed deduplication.
Search the web (DuckDuckGo), persist results, and archive them for later retrieval.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@web-archive-mcp fetch https://en.wikipedia.org/wiki/Palimpsest"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
web-archive-mcp
MCP server for persistent web fetch and search archiving. Every web_fetch and web_search result is saved as timestamped JSONL, indexed by fst-indexer, and searchable via unified-history-mcp.
Part of the Palimpsest investigative toolkit.
Why
web_fetch and web_search results normally evaporate when a session ends. Pages change, get deleted, or get memory-holed. This closes that gap — every result is persisted, content-addressed for dedup, and fed into the same search pipeline as your session logs and transcripts. Three months later, a search(domain="all", query="target name") still finds the page that's been 404'd since July.
Related MCP server: Claw Recall
Architecture
web_fetch / web_search playwright-archive-mcp (browser capture)
│ │
▼ ▼
web-archive-store (shared JSONL write-path + SSRF URL validation)
│
▼
~/.local/share/web-archive/*.jsonl
│ │
│ fst-indexer (Jsonl extractor)
│ │
▼ ▼
returns content index.fst + manifest.json
│
▼
unified-history-mcp
domain: "web-archive"Tools
Tool | Description |
| Fetch a URL, convert to markdown, persist, return |
| Search the web (DuckDuckGo), persist results |
| List archived entries with metadata |
| Read entries from an archive file |
| Rebuild FST index for the web-archive domain |
Installation
web-archive-mcp depends on the shared web-archive-store package (archive write-path + SSRF URL validation). Install it first:
git clone https://github.com/palimpsest-labs/web-archive-store
cd web-archive-store
python3 -m venv .venv
source .venv/bin/activate
pip install -e .
cd ..
git clone https://github.com/palimpsest-labs/web-archive-mcp
cd web-archive-mcp
python3 -m venv .venv
source .venv/bin/activate
pip install -e .Once web-archive-store is published to PyPI this becomes a single pip install -e ..
Browser-driven traffic capture (the
playwright_*tools) moved to the separateplaywright-archive-mcpserver, which records HTTP traffic into the same store.
Integration with unified-history-mcp
Add to your unified-history TOML config:
[domains.web-archive]
dir = "~/.local/share/web-archive"
pattern = "*.jsonl"
extractor = "jsonl"
label = "web-archive entry"
filters = []Then rebuild: search(domain="web-archive", query="rebuild") or call rebuild directly.
Once indexed, a search(domain="all", query="your search") scans your sessions, transcripts, notifications, and every web page you've ever fetched — in a single query.
Entry format
{
"type": "fetch",
"source": "https://example.com/page",
"title": "Example Page",
"content": "# Example\n\nMarkdown content...",
"timestamp": "2026-07-30T21:15:00Z",
"content_hash": "abc123..."
}For searches, source holds the query string and type is "search".
Content-addressed dedup prevents storing identical entries. Same source + same content hash = skipped.
License
MIT
Available Tools
17 toolsarchive_listA
List archived web fetch/search entries with metadata.
Args: date_from: Optional start of date range (YYYY-MM-DD) date_to: Optional end of date range (YYYY-MM-DD, inclusive) max_results: Maximum entries to show (default 50)
Inputs: date_to, date_from, max_results. Returns: result
archive_readA
Read entries from an archive file.
Args: file_id: Archive file name (e.g., '2026-07-30-fetch-example.jsonl') max_entries: Maximum entries to return, newest first (default 50)
Inputs: file_id (required), max_entries. Returns: result
playwright_backA
Go back in the interactive session's history.
Inputs: no named parameters. Returns: result
playwright_clickA
Click an element (CSS selector) in the interactive session.
Inputs: selector (required). Returns: result
playwright_closeA
Close the interactive Playwright session and its browser.
Inputs: no named parameters. Returns: result
playwright_fillA
Fill a form field (CSS selector) with a value in the interactive session.
Inputs: value (required), selector (required). Returns: result
playwright_forwardB
Go forward in the interactive session's history.
Inputs: no named parameters. Returns: result
playwright_htmlA
Return the HTML of the current page in the interactive session.
Inputs: max_len. Returns: result
playwright_recordA
Drive a headless Playwright browser against the given URL(s) and record every HTTP request/response into the web-archive store.
Binary/streaming response bodies are skipped, auth headers are redacted by
default, and each URL gets a fresh browser context so cookies don't leak
between sites. Like web_fetch, this rejects private/loopback addresses
(SSRF protection). Recorded entries become searchable once rebuild runs.
Args: urls: URL(s) to visit (http/https; scheme auto-prepended) wait: Extra seconds to wait after page load for async requests timeout: Navigation timeout in seconds (max 120) max_entries: Stop recording after this many request/response entries redact_auth: Redact Authorization/Cookie/Set-Cookie/X-API-Key headers
Inputs: urls (required), wait, timeout, max_entries, +1 more. Returns: result
playwright_screenshotA
Save a screenshot of the current page to ~/Downloads and return the path.
The filename is sanitized (directory components stripped, .png enforced) so it cannot escape ~/Downloads.
Inputs: name. Returns: result
playwright_startA
Start a persistent interactive Playwright session with always-on traffic recording. Every response observed on the session is archived to the web-archive store in real time. Returns a confirmation.
Inputs: no named parameters. Returns: result
playwright_statsA
Report the interactive session's status and recorded-entry counts.
Inputs: no named parameters. Returns: result
playwright_textB
Return the visible text of the current page in the interactive session.
Inputs: max_len. Returns: result
rebuildA
Rebuild the FST index for the web-archive domain.
Calls fst-indexer to rebuild the full-text index so archived content is searchable via unified-history-mcp.
Inputs: no named parameters. Returns: result
web_fetchA
Fetch a URL and archive the result.
Fetches the URL, converts HTML to markdown, and persists the result as a timestamped JSONL entry in the web-archive. Only http and https URLs are allowed; private/internal IPs are blocked.
Args: url: The URL to fetch (http/https only) timeout: Request timeout in seconds (default 30, max 120) token: Optional Bearer token for authenticated requests
Inputs: url (required), token, timeout. Returns: result
web_searchA
Search the web and archive the results.
Performs a web search via DuckDuckGo, persists the results, and returns them formatted. The results are archived for later search.
Args: query: The search query (max 500 chars)
Inputs: query (required). Returns: result
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 17 tools
Most tools have clearly distinct purposes, but playwright_text and playwright_html both return page content (just different formats), and web_fetch and playwright_navigate can both be used to retrieve a URL. The descriptions do a good job mitigating confusion, so the overlap is minor.
Tool names use a consistent prefix (web_, playwright_, archive_) with action-oriented suffixes. Slight inconsistency exists because archive_list and archive_read are not verb-first, and a few playwright_ commands use nouns (text, html, screenshot). Overall the pattern is still predictable and readable.
17 tools is on the heavier side, mostly due to the 12 playwright interaction commands. Each interaction command serves a distinct browser automation need, but the count is borderline-high for an MCP server surface.
The core workflows of web fetching, searching, recording, interactive browser control, and archive listing/reading are covered well. Gaps include no delete or clear operation for archived entries, and direct search within archives relies on an external indexer (rebuild + unified-history-mcp) rather than a native search tool.
Maintenance
Related MCP Connectors
Shared copies of public web pages for AI agents. Search stored pages or fetch a URL.
Pages, crawls and URL batches as clean Markdown and JSON for AI agents, with an Evidence Record.
Persistent memory for AI agents. Search, store, and recall across sessions.
Web infrastructure for AI agents: scrape, crawl and search the web, run browsers and agents.
Related MCP Servers
- FlicenseNot gradedqualityNot gradedmaintenanceProvides cross-session memory for AI agents by maintaining a rolling 24-hour event stream and searchable daily archives to prevent context loss. It enables agents to record, query, and retrieve historical events and decisions through a structured markdown-based workspace.1 npm-
- AlicenseNot gradedqualityFmaintenancePersistent, searchable memory for AI agents. Enables agents to recover context after compaction by searching indexed conversations, emails, and files via MCP tools.1MIT
- AlicenseAqualityDmaintenanceEnables AI agents to maintain a persistent, queryable memory stored as user-owned Markdown files, with dual-channel retrieval (FTS5 and optional semantic search) and an audited write pipeline.11MIT
- AlicenseNot gradedqualityBmaintenanceEnables AI agents to store, search, consolidate, and trace durable memories as plain Markdown files, with a disposable BM25 index and provenance tracking via MCP tools.MIT