How do I use web-retrieval-mcp?

1. Click on "Deploy Server". 2. Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state. 3. In the chat, type @ followed by the MCP server name and your instructions, e.g., "@web-retrieval-mcp Fetch https://example.com and summarize the key points" That's it! The server will respond to your query, and you can continue using it as needed. Here is a step-by-step guide with screenshots.

web-retrieval-mcp

by Tyr1onX

Overview Schema Source Related Servers Discussions Admin

Python

Hybrid

Web Retrieval MCP

A small MCP server that gives an AI agent a unified web-retrieval interface while routing work to specialized backends:

Scrapling for fast static retrieval, targeted extraction, dynamic pages, and stealth fallback.
Crawl4AI for JavaScript rendering and bounded same-site crawling.
Firecrawl for web search (optional; requires an API key).

The model sees four stable tools instead of backend-specific APIs:

Tool	Purpose
`web_fetch`	Read one URL as clean Markdown with automatic escalation.
`web_crawl`	Crawl a bounded set of internal pages from one site.
`web_extract`	Extract repeated elements with a CSS selector.
`web_search`	Search the public web through Firecrawl.

Routing

web_fetch(render="auto") uses a cost-aware fallback chain:

Scrapling static HTTP
        ↓ content missing / placeholder / blocked
Crawl4AI browser render
        ↓ still missing / blocked
Scrapling stealth browser

web_extract(mode="auto") similarly escalates from static → dynamic → stealth.

web_crawl uses Crawl4AI BFS with same-site-only traversal and hard page/depth caps.

web_search uses Firecrawl and is only enabled when FIRECRAWL_API_KEY is configured.

Related MCP server: Web Eyes

Requirements

Python 3.10+
Windows, Linux, or macOS
uv recommended

Current tested dependency targets for v0.1:

MCP Python SDK 2.x
Crawl4AI 0.9.2.x
Scrapling 0.4.15.x
Firecrawl Python SDK 4.40.x

Windows quick setup

Clone the repository, then run:

git clone https://github.com/Tyr1onX/web-retrieval-mcp.git
cd web-retrieval-mcp
powershell -ExecutionPolicy Bypass -File .\scripts\setup-windows.ps1

The script installs the Python environment plus Crawl4AI/Scrapling browser dependencies. At the end it prints an executable path similar to:

C:\Users\YOU\web-retrieval-mcp\.venv\Scripts\web-retrieval-mcp.exe

Add it to ChatGPT Desktop

In Settings → Plugins → MCP → Add → Custom MCP:

Name: Web Retrieval
Type: STDIO
Startup command: C:\...\web-retrieval-mcp\.venv\Scripts\web-retrieval-mcp.exe
Arguments: (leave empty)

Environment variables are optional. To enable Firecrawl search, add:

FIRECRAWL_API_KEY=fc-...

Do not put API keys in the repository or .env files that you commit.

After enabling the MCP server, a new Work/Codex chat should discover the four tools. A good first test is:

Use web_fetch to read https://example.com and tell me which backend handled it.

Then test JavaScript rendering with a JS-heavy public page and render="always".

Manual setup

uv sync --extra all
uv run crawl4ai-setup
uv run scrapling install
uv run web-retrieval-mcp

The default transport is STDIO.

You can install only the backends you need:

uv sync --extra scrapling --extra crawl4ai
uv sync --extra firecrawl

Tool details

`web_fetch`

Inputs:

url: public HTTP(S) URL.
render: auto, never, or always.
selector: optional CSS selector to narrow the returned content.
max_chars: optional response cap.

auto starts cheap and escalates. never prevents browser rendering. always skips the static path and starts with Crawl4AI.

`web_crawl`

Inputs include max_depth, max_pages, per_page_chars, and respect_robots.

The server applies its own hard limits even when a client requests larger values. External links are not followed.

`web_extract`

Inputs:

selector: CSS selector.
attribute: optional attribute name; otherwise visible text is returned.
mode: auto, static, dynamic, or stealth.
limit: maximum returned matches (hard capped at 100).

`web_search`

Requires FIRECRAWL_API_KEY.

Optional filters include category (github, research, pdf, developer) and a Firecrawl time filter such as qdr:d or qdr:w.

Configuration

Copy .env.example for the available variables. Important defaults:

WEB_RETRIEVAL_TRANSPORT=stdio
WEB_RETRIEVAL_TIMEOUT_MS=30000
WEB_RETRIEVAL_MAX_CHARS=50000
WEB_RETRIEVAL_MAX_CRAWL_PAGES=20
WEB_RETRIEVAL_MAX_CRAWL_DEPTH=3
WEB_RETRIEVAL_ALLOW_PRIVATE_NETWORKS=false

WEB_RETRIEVAL_ALLOW_PRIVATE_NETWORKS=true is intentionally opt-in. Do not enable it on a remotely reachable deployment.

Streamable HTTP / server deployment

Set:

WEB_RETRIEVAL_TRANSPORT=streamable-http
WEB_RETRIEVAL_HOST=127.0.0.1
WEB_RETRIEVAL_PORT=8765

Then:

uv run web-retrieval-mcp

The MCP endpoint is:

http://127.0.0.1:8765/mcp

A Dockerfile and docker-compose.yml are included. Compose intentionally publishes the service only on loopback:

docker compose up -d --build

Do not expose v0.1 directly to the public Internet

The application blocks private/loopback targets by default, but application-level URL checks cannot eliminate all DNS-rebinding/redirect/egress risks. A remote production deployment should additionally have:

authentication/access control in front of the MCP endpoint,
network-level egress restrictions that prevent access to metadata/private networks,
request/concurrency/rate limits,
resource caps for browser containers.

Keep the service loopback-only until those controls are in place.

Security model

The MCP tools accept URLs supplied by an AI model, so all URLs and returned page content are treated as untrusted.

Current safeguards include:

only http:// and https://,
embedded URL credentials rejected,
loopback/private/link-local/reserved/multicast/unspecified IPs blocked by default,
DNS resolution checked before retrieval,
obvious final redirect destinations checked again,
bounded crawl depth/pages,
bounded output sizes,
no arbitrary request headers, cookies, POST bodies, JavaScript snippets, local files, or proxies exposed through the MCP schema,
server instructions explicitly tell the model not to obey instructions found inside retrieved webpage content.

For remote deployments, use network egress controls as the final SSRF boundary.

Development

Base CI intentionally does not download browsers:

uv sync --dev
uv run ruff check src tests
uv run pytest -q

Full local integration setup:

uv sync --dev --extra all
uv run crawl4ai-setup
uv run scrapling install

License

MIT. Third-party backends retain their own licenses and are installed as dependencies; their source code is not vendored into this repository.

Available Tools

4 tools

web_crawlC

Crawl a bounded set of internal pages from one public website.

Uses Crawl4AI breadth-first crawling. External links are not followed. The server enforces hard caps on depth/pages regardless of the requested values.

ParametersJSON Schema

Name	Required	Description	Default
`url`	Yes
`max_depth`	No
`max_pages`	No
`per_page_chars`	No
`respect_robots`	No

TDQS

C2.9/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the burden of behavioral disclosure. It usefully reveals breadth-first crawling, that external links are ignored, and that server-enforced caps override requested depth/page values. However, it does not describe request impact, rate limits, robots.txt behavior, or what the return payload looks like.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is short and front-loaded with the core purpose. The second sentence provides useful behavioral constraints without padding. It could slightly expand on parameters, but it is appropriately concise for what it covers.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has five parameters, no annotations, no output schema, and 0% schema description coverage, so the description should do more to complete the picture. It covers basic crawling boundaries but omits return format, parameter meaning for most inputs, and operational behavior around robots or page content.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate. It adds meaning to max_depth and max_pages by mentioning depth/pages and server caps, but it does not explain required parameter url, per_page_chars, or respect_robots. With five parameters and no schema descriptions, this is insufficient.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb and resource: 'Crawl a bounded set of internal pages from one public website.' It clarifies scope by noting external links are not followed, which helps distinguish it from web_fetch and web_search even though no sibling is named explicitly.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no explicit guidance on when to use web_crawl versus web_fetch, web_extract, or web_search. The description implies the tool is for traversing internal pages, but it does not state when the alternatives are preferable or provide exclusion criteria.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

web_extractA

Extract repeated elements from a public webpage using a CSS selector.

If attribute is omitted, visible text is returned. In auto mode Scrapling tries static HTTP first, then a dynamic browser, then stealth browser retrieval.

ParametersJSON Schema

Name	Required	Default
`url`	Yes
`mode`	No	auto
`limit`	No
`selector`	Yes
`attribute`	No

TDQS

A3.7/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the transparency burden. It does disclose useful behaviors: omitted attribute returns visible text, and auto mode tries static HTTP, then dynamic browser, then stealth. However, it omits output shape, error behavior, rate limits, and other operational details.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two focused sentences with no filler. The primary purpose is front-loaded, and the additional behavioral details about attribute and auto-mode each earn their place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description covers core invocation details and two key defaults, making it usable for basic calls. But there is no output schema, no description of the result format, no limit semantics, and no sibling differentiation, leaving several operational aspects to inference.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate. It adds meaning for selector, attribute, and mode, but url and limit remain unexplained. This is partial compensation rather than full coverage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a precise operation: 'Extract repeated elements from a public webpage using a CSS selector.' It clearly identifies the verb, resource, and method, and this is enough to distinguish it from siblings like web_fetch, web_crawl, and web_search.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The intended context is implied by 'Extract repeated elements from a public webpage,' but there is no explicit guidance on when to choose this over the sibling tools. No alternatives, exclusions, or when-not-to-use cases are mentioned.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

web_fetchB

Read one public HTTP(S) URL and return clean LLM-friendly Markdown.

render='auto' starts with a lightweight static fetch, escalates to Crawl4AI for JavaScript rendering when content appears incomplete, then uses Scrapling stealth as the final fallback. Use render='always' for known JavaScript-heavy pages.

ParametersJSON Schema

Name	Required	Default
`url`	Yes
`render`	No	auto
`selector`	No
`max_chars`	No

TDQS

B3.3/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden, and it does well: it reveals a three-stage fetch strategy (static fetch, Crawl4AI JavaScript rendering, Scrapling stealth fallback) and the Markdown output format. It does not cover failure behavior, but this is a read-only fetch tool and the disclosed behavior is substantive.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two tight sentences with no wasted words. The main purpose is front-loaded, and the render guidance earns its place without redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description covers the tool's core behavior and render parameter well, but it omits semantics for selector and max_chars, and does not describe failure/error behavior. With no output schema and no annotations, these gaps leave the agent with reasonable but incomplete context.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema has 0% description coverage, so the description must compensate. It adds meaning to the render parameter by explaining 'auto' escalation and 'always' for JS-heavy pages, but selector and max_chars are not explained at all, leaving their semantics ambiguous.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb and resource: 'Read one public HTTP(S) URL and return clean LLM-friendly Markdown.' The word 'one' differentiates it from web_crawl, though sibling tools are not explicitly named.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives good guidance on when to use render='always' vs 'auto', but provides no explicit direction on when to choose web_fetch over siblings like web_crawl, web_extract, or web_search. No alternatives or exclusions are mentioned.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

web_searchA

Search the public web through Firecrawl.

Requires FIRECRAWL_API_KEY. category can target GitHub/research/PDF/developer sources; time_filter limits results to the past hour/day/week/month/year.

ParametersJSON Schema

Name	Required	Description	Default
`limit`	No
`query`	Yes
`category`	No
`time_filter`	No

TDQS

A4/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the behavioral disclosure burden. It usefully discloses the FIRECRAWL_API_KEY requirement and explains what category and time_filter do. However, it does not mention whether the operation is read-only, potential rate limits, or what happens on missing/invalid keys. This is a moderate gap for a tool with zero annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two tight sentences with no filler. The core purpose is front-loaded, and the API key requirement plus category/time_filter semantics fit in minimal space. Every sentence earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a search tool with no output schema, the description provides sufficient information for an agent to make a basic call: it knows a query is required and can optionally narrow by category or time range. It misses minor details like the nature of the return results, but the default limit and self-explanatory query make it adequately complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate. It adds real meaning to the cryptic enum values: category 'can target GitHub/research/PDF/developer sources' and time_filter 'limits results to the past hour/day/week/month/year.' Query and limit are not described, but they are self-explanatory and have clear schema defaults.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb and resource: 'Search the public web through Firecrawl.' This clearly differentiates it from sibling tools like web_fetch, web_crawl, and web_extract, which all imply different operations. No ambiguity about what the tool does.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives helpful context on optional parameters (category and time_filter) but never explicitly states when to use web_search versus its siblings. An agent can infer from the tool name and sibling names, but there is no direct guidance such as 'use this instead of web_fetch when...'.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

4 tool updatesv0.1.0Aug 31, 2026
- First observedweb_crawl
- First observedweb_extract
- First observedweb_fetch
- First observedweb_search

Deploy Server

One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

TDQS

A3.8/5.0

Scored 2026-08-31 05:25 across 4 tools

Disambiguation5/5

Each tool targets a clear, distinct retrieval mode: single-page fetch, multi-page crawl, structured extraction, and web search. The descriptions make the boundaries between fetch and crawl or extract easy to distinguish.

Naming Consistency5/5

All tools follow the same web_<verb> snake_case pattern, making it predictable and easy to remember. The naming convention is consistent across the entire tool set.

Tool Count5/5

Four tools is a well-scoped size for a web retrieval server. Each tool covers a distinct core operation without unnecessary redundancy or overwhelming the agent.

Completeness5/5

The tool surface covers the main web retrieval workflows: fetching a single page, crawling a site, extracting structured elements, and searching the web. No obvious dead ends or missing core operations are apparent for the stated purpose.

Maintenance

ActivityMaintained

ResponsivenessSyncing

Resources

GitHub Repository

Need Help?

Related Servers

Related MCP Connectors

scrapeless-mcp-server
io.github.scrapeless-ai
Web search, browser automation, scraping, crawling and CAPTCHA solving for AI agents.
Browser Automation Web Scraping Search
1169
mcp A
ai.foura
Reliable web access for AI agents: smart HTTP, rotating proxies, and full-browser rendering.
Web Scraping Browser Automation
434
Ninelayer
in.ninelayer
Agent-native search engine with live web research optimized for AI agents.
Search Agent Orchestration
2
FastAF
co.fastaf
Web data tools for AI agents: pages as markdown, search, maps, commerce, jobs, AI answers.
Browser Automation Search E-commerce & Retail
1

Related MCP Servers

Crawl4AI RAG MCP Server
Chillbruhhh
A
license
Not graded
quality
D
maintenance
Provides AI agents and coding assistants with advanced web crawling and RAG capabilities, allowing them to scrape websites and leverage that knowledge through various retrieval strategies.
Updated a year ago (2025-07-15 03:03 UTC)
2
MIT
Web Eyes
Browser Automation Web Scraping
slothitude
F
license
Not graded
quality
D
maintenance
Enables LLM agents to search, crawl, summarize, and analyze web pages and images via a pipeline of web intelligence tools.
Updated 4 months ago (2026-05-26 23:39 UTC)
-
Crawl4AI RAG MCP Server
RAG Systems Web Scraping Browser Automation
jmgilman
A
license
Not graded
quality
D
maintenance
Provides AI agents and assistants with advanced web crawling and RAG capabilities, enabling them to scrape websites and perform semantic search over crawled content.
Updated a year ago (2025-08-05 00:05 UTC)
1
MIT
Web Search MCP Server
Browser Automation Web Scraping Search
lalit9168
F
license
Not graded
quality
D
maintenance
Enables AI agents to perform web searches, extract webpage content, and conduct end-to-end search-and-extract operations using multiple search providers and content extraction methods.
Updated 3 months ago (2026-07-01 07:18 UTC)
-