web-retrieval-mcp
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@web-retrieval-mcp Fetch https://example.com and summarize the key points"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Web Retrieval MCP
A small MCP server that gives an AI agent a unified web-retrieval interface while routing work to specialized backends:
Scrapling for fast static retrieval, targeted extraction, dynamic pages, and stealth fallback.
Crawl4AI for JavaScript rendering and bounded same-site crawling.
Firecrawl for web search (optional; requires an API key).
The model sees four stable tools instead of backend-specific APIs:
Tool | Purpose |
| Read one URL as clean Markdown with automatic escalation. |
| Crawl a bounded set of internal pages from one site. |
| Extract repeated elements with a CSS selector. |
| Search the public web through Firecrawl. |
Routing
web_fetch(render="auto") uses a cost-aware fallback chain:
Scrapling static HTTP
↓ content missing / placeholder / blocked
Crawl4AI browser render
↓ still missing / blocked
Scrapling stealth browserweb_extract(mode="auto") similarly escalates from static → dynamic → stealth.
web_crawl uses Crawl4AI BFS with same-site-only traversal and hard page/depth caps.
web_search uses Firecrawl and is only enabled when FIRECRAWL_API_KEY is configured.
Related MCP server: Web Eyes
Requirements
Python 3.10+
Windows, Linux, or macOS
uvrecommended
Current tested dependency targets for v0.1:
MCP Python SDK 2.x
Crawl4AI 0.9.2.x
Scrapling 0.4.15.x
Firecrawl Python SDK 4.40.x
Windows quick setup
Clone the repository, then run:
git clone https://github.com/Tyr1onX/web-retrieval-mcp.git
cd web-retrieval-mcp
powershell -ExecutionPolicy Bypass -File .\scripts\setup-windows.ps1The script installs the Python environment plus Crawl4AI/Scrapling browser dependencies. At the end it prints an executable path similar to:
C:\Users\YOU\web-retrieval-mcp\.venv\Scripts\web-retrieval-mcp.exeAdd it to ChatGPT Desktop
In Settings → Plugins → MCP → Add → Custom MCP:
Name: Web Retrieval
Type: STDIO
Startup command: C:\...\web-retrieval-mcp\.venv\Scripts\web-retrieval-mcp.exe
Arguments: (leave empty)Environment variables are optional. To enable Firecrawl search, add:
FIRECRAWL_API_KEY=fc-...Do not put API keys in the repository or .env files that you commit.
After enabling the MCP server, a new Work/Codex chat should discover the four tools. A good first test is:
Use web_fetch to read https://example.com and tell me which backend handled it.Then test JavaScript rendering with a JS-heavy public page and render="always".
Manual setup
uv sync --extra all
uv run crawl4ai-setup
uv run scrapling install
uv run web-retrieval-mcpThe default transport is STDIO.
You can install only the backends you need:
uv sync --extra scrapling --extra crawl4ai
uv sync --extra firecrawlTool details
web_fetch
Inputs:
url: public HTTP(S) URL.render:auto,never, oralways.selector: optional CSS selector to narrow the returned content.max_chars: optional response cap.
auto starts cheap and escalates. never prevents browser rendering. always skips the static path and starts with Crawl4AI.
web_crawl
Inputs include max_depth, max_pages, per_page_chars, and respect_robots.
The server applies its own hard limits even when a client requests larger values. External links are not followed.
web_extract
Inputs:
selector: CSS selector.attribute: optional attribute name; otherwise visible text is returned.mode:auto,static,dynamic, orstealth.limit: maximum returned matches (hard capped at 100).
web_search
Requires FIRECRAWL_API_KEY.
Optional filters include category (github, research, pdf, developer) and a Firecrawl time filter such as qdr:d or qdr:w.
Configuration
Copy .env.example for the available variables. Important defaults:
WEB_RETRIEVAL_TRANSPORT=stdio
WEB_RETRIEVAL_TIMEOUT_MS=30000
WEB_RETRIEVAL_MAX_CHARS=50000
WEB_RETRIEVAL_MAX_CRAWL_PAGES=20
WEB_RETRIEVAL_MAX_CRAWL_DEPTH=3
WEB_RETRIEVAL_ALLOW_PRIVATE_NETWORKS=falseWEB_RETRIEVAL_ALLOW_PRIVATE_NETWORKS=true is intentionally opt-in. Do not enable it on a remotely reachable deployment.
Streamable HTTP / server deployment
Set:
WEB_RETRIEVAL_TRANSPORT=streamable-http
WEB_RETRIEVAL_HOST=127.0.0.1
WEB_RETRIEVAL_PORT=8765Then:
uv run web-retrieval-mcpThe MCP endpoint is:
http://127.0.0.1:8765/mcpA Dockerfile and docker-compose.yml are included. Compose intentionally publishes the service only on loopback:
docker compose up -d --buildDo not expose v0.1 directly to the public Internet
The application blocks private/loopback targets by default, but application-level URL checks cannot eliminate all DNS-rebinding/redirect/egress risks. A remote production deployment should additionally have:
authentication/access control in front of the MCP endpoint,
network-level egress restrictions that prevent access to metadata/private networks,
request/concurrency/rate limits,
resource caps for browser containers.
Keep the service loopback-only until those controls are in place.
Security model
The MCP tools accept URLs supplied by an AI model, so all URLs and returned page content are treated as untrusted.
Current safeguards include:
only
http://andhttps://,embedded URL credentials rejected,
loopback/private/link-local/reserved/multicast/unspecified IPs blocked by default,
DNS resolution checked before retrieval,
obvious final redirect destinations checked again,
bounded crawl depth/pages,
bounded output sizes,
no arbitrary request headers, cookies, POST bodies, JavaScript snippets, local files, or proxies exposed through the MCP schema,
server instructions explicitly tell the model not to obey instructions found inside retrieved webpage content.
For remote deployments, use network egress controls as the final SSRF boundary.
Development
Base CI intentionally does not download browsers:
uv sync --dev
uv run ruff check src tests
uv run pytest -qFull local integration setup:
uv sync --dev --extra all
uv run crawl4ai-setup
uv run scrapling installLicense
MIT. Third-party backends retain their own licenses and are installed as dependencies; their source code is not vendored into this repository.
Available Tools
4 toolsweb_crawlC
Crawl a bounded set of internal pages from one public website.
Uses Crawl4AI breadth-first crawling. External links are not followed. The server enforces hard caps on depth/pages regardless of the requested values.
Inputs: url (required), max_depth, max_pages, per_page_chars, +1 more.
web_extractA
Extract repeated elements from a public webpage using a CSS selector.
If attribute is omitted, visible text is returned. In auto mode Scrapling tries static HTTP first, then a dynamic browser, then stealth browser retrieval.
Inputs: url (required), selector (required), mode, limit, +1 more.
web_fetchB
Read one public HTTP(S) URL and return clean LLM-friendly Markdown.
render='auto' starts with a lightweight static fetch, escalates to Crawl4AI for JavaScript rendering when content appears incomplete, then uses Scrapling stealth as the final fallback. Use render='always' for known JavaScript-heavy pages.
Inputs: url (required), render, selector, max_chars.
web_searchA
Search the public web through Firecrawl.
Requires FIRECRAWL_API_KEY. category can target GitHub/research/PDF/developer sources; time_filter limits results to the past hour/day/week/month/year.
Inputs: query (required), limit, category, time_filter.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 4 tools
Each tool targets a clear, distinct retrieval mode: single-page fetch, multi-page crawl, structured extraction, and web search. The descriptions make the boundaries between fetch and crawl or extract easy to distinguish.
All tools follow the same web_<verb> snake_case pattern, making it predictable and easy to remember. The naming convention is consistent across the entire tool set.
Four tools is a well-scoped size for a web retrieval server. Each tool covers a distinct core operation without unnecessary redundancy or overwhelming the agent.
The tool surface covers the main web retrieval workflows: fetching a single page, crawling a site, extracting structured elements, and searching the web. No obvious dead ends or missing core operations are apparent for the stated purpose.
Related MCP Connectors
Web data for agents. Extract web pages, crawl websites, discover URLs, and search the web.
Web scraping, search, crawling, structured extraction and research for AI agents.
Web infrastructure for AI agents: scrape, crawl and search the web, run browsers and agents.
Web knowledge fetching tools for your agent.
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceProvides AI agents and coding assistants with advanced web crawling and RAG capabilities, allowing them to scrape websites and leverage that knowledge through various retrieval strategies.2MIT
- FlicenseNot gradedqualityDmaintenanceEnables LLM agents to search, crawl, summarize, and analyze web pages and images via a pipeline of web intelligence tools.-
- AlicenseNot gradedqualityDmaintenanceProvides AI agents and assistants with advanced web crawling and RAG capabilities, enabling them to scrape websites and perform semantic search over crawled content.1MIT
- FlicenseNot gradedqualityDmaintenanceEnables AI agents to perform web searches, extract webpage content, and conduct end-to-end search-and-extract operations using multiple search providers and content extraction methods.-