io.github.VelvetSP/web-retrieval-mcp
Click on "Install Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@io.github.VelvetSP/web-retrieval-mcpsearch for latest AI research papers"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
web-retrieval-mcp — reliable MCP web search and web fetch for AI agents
An open-source Model Context Protocol (MCP) server for source-grounded web research. Connect Codex CLI and the Codex IDE extension, Claude Code, Claude Desktop, Cursor, ChatGPT desktop, or any other compatible MCP client to six read-only tools for Exa or Tavily search, resilient Exa → Camoufox → Tavily → Firecrawl page retrieval, AI/ML paper discovery, and developer-source search—with explicit provenance, optional local caching, and SSRF guards.
Why use it? · Quick start · Tools · Routing · Configuration · Security · FAQ
Why use web-retrieval-mcp?
Web research is more reliable when an agent can see where every result came from,
choose the right retrieval strategy, and recover when one provider or page-access
method fails. web-retrieval-mcp makes those controls part of the tool contract:
Keep sources separate. Default search returns one block per result with its own title, URL, highlights, and text, followed by a
Sourcestrailer. Exa deep modes additionally return a clearly labeled synthesized answer, with grounding when Exa provides it, before the source-separated result blocks.Use more than one retrieval path. Search can use Exa or Tavily and fall back to Firecrawl. Full-page fetches can escalate from indexed content to a local Camoufox browser, optional Tavily Extract, and Firecrawl.
Know what actually served the answer. Fetch responses identify the serving tier, cache state, semantic mode, and truncation instead of hiding the route.
Ask for the right shape. Fetch a readable page body, a concise summary, or a grounded answer to a question. Control freshness, rendering, domains, dates, result count, and per-result text size.
Search specialist corpora. Discover AI/ML papers, inspect query-relevant full-text passages, expand to related papers, and search developer documentation, READMEs, issues, and merged pull requests.
Own the deployment boundary. Run locally over stdio or host a loopback HTTP service; select providers and secret storage yourself; optionally cache completed fetches in a private Valkey sidecar.
Fail explicitly. Unsupported provider choices, missing optional packages, and unavailable credentials fail in the tool result. Tavily mappings, Firecrawl fallback filter gaps, and Exa category migrations are caller-visible; other Exa filter drops are written to server diagnostics.
How it compares with built-in agent web tools
Built-in capabilities vary by agent and can improve over time. The useful comparison is therefore not “all defaults are bad”; it is whether you need a portable, inspectable retrieval layer with controls your current client does not expose.
Need | Typical bundled search/fetch surface | web-retrieval-mcp |
Search output | Client-defined result or synthesis format | Default search returns source-separated blocks plus |
Backend choice | Provider and routing are usually managed by the client | Exa or Tavily search, with a disclosed Firecrawl fallback |
Difficult pages | One client-specific access path | Indexed content, guarded local browser rendering, Tavily Extract, and Firecrawl tiers |
Retrieval intent | Usually search or page text | Full body, concise summary, or query-grounded page answer |
Research discovery | General web index | Dedicated paper and developer-source tools in addition to web search |
Freshness and filters | Whatever the client exposes | Date windows, hour-level recency, domains, categories, result limits, and forced freshness |
Auditability | Client-specific | Fetch tier/cache/truncation disclosures; Tavily and Firecrawl search-fallback labels; explicit filter notices where supported |
Operations | Client-defined; often hosted or opaque | Local stdio or self-hosted loopback HTTP, optional private UDS cache |
Security policy | Client-specific | Initial URL validation plus guarded browser requests and redirects |
Use it as an independent complementary retrieval lane by default. If you want it
to replace Claude Code's built-in WebSearch and WebFetch, the package also includes
an optional, previewable hook installer.
Related MCP server: myscrape
Quick start
Python 3.10 or newer is required.
1. Install
For the complete feature set in an isolated environment:
pipx install --include-deps 'web-retrieval-mcp[all]'
camoufox fetch--include-deps exposes Camoufox's own command from the isolated pipx environment;
the second command downloads its managed browser. If you do not want local rendering,
install the base package or selected non-render extras and skip the browser download.
For a lean Exa + Firecrawl installation:
pipx install web-retrieval-mcpWith uv installed, you can run the base package in a temporary isolated environment without a permanent install:
uvx web-retrieval-mcpOr install into an existing virtual environment:
python -m pip install 'web-retrieval-mcp[all]'
python -m camoufox fetchThe browser download is required whenever you install the render extra and want
Camoufox to serve render="auto" or render="always" calls.
Optional extras are composable:
Extra | Adds |
| Tavily Search and Tavily Extract |
| Local Camoufox/Playwright rendering |
| Valkey client for completed-result caching |
| Cross-platform native secret-store access |
| Every optional runtime capability above |
2. Add provider credentials
For the broadest routing coverage, configure Exa and Firecrawl; add Tavily when you want provider-selectable search or another full-body fetch tier.
export EXA_API_KEY='<your-exa-api-key>'
export FIRECRAWL_API_KEY='<your-firecrawl-api-key>'
export TAVILY_API_KEY='<your-tavily-api-key>' # optionalYou do not need all three keys to start the server. Tools resolve credentials lazily and report what a selected route is missing. See Credential storage for a key file and OS secret-store alternatives.
All three providers advertised no-card entry allocations when checked on 2026-09-01; plans and unit costs can change, so check their live pricing before estimating a workload:
Provider | Entry allocation checked 2026-09-01 | Used for |
$20 signup credit plus $10 monthly API credits | Default web search and first indexed-content fetch tier | |
1,000 credits per month | Search/fetch fallback and research/developer indexes | |
1,000 API credits per month | Optional search provider and full-body extract tier | |
Local open-source browser | JavaScript-rendered page retrieval on your machine |
3. Connect an MCP client
Codex CLI and the Codex IDE extension share MCP configuration. Add the server with the CLI, then verify it is registered:
codex mcp add web-retrieval -- web-retrieval-mcp
codex mcp listWith uvx and no prior install:
codex mcp add web-retrieval -- uvx web-retrieval-mcpSee OpenAI's Codex MCP documentation for the current CLI, IDE, and configuration-file details.
Claude Code:
claude mcp add web-retrieval -- web-retrieval-mcpWith uvx and no prior install:
claude mcp add web-retrieval -- uvx web-retrieval-mcpClaude Desktop, Cursor, and other clients that accept an mcpServers object can use:
{
"mcpServers": {
"web-retrieval": {
"command": "web-retrieval-mcp"
}
}
}That configuration assumes the executable is on the client's PATH. For an uvx
launch, use "command": "uvx" and "args": ["web-retrieval-mcp"].
If a desktop client does not inherit your shell environment, use the private key file
described below or place literal key values in the client's protected environment
configuration. Do not rely on ${VARIABLE} interpolation unless your client documents
that behavior.
ChatGPT desktop and clients with different setup surfaces can connect through the MCP transport their current version supports. This server uses stdio by default and can also expose Streamable HTTP; consult the client's documentation for its configuration format and supported transports.
4. Give the agent a real task
Once the MCP server is connected, prompts can stay natural:
Search the web for the latest primary documentation about Python package metadata.
Keep each source separate and preserve its URL.
Fetch https://example.com/report, force a fresh retrieval, and answer only:
What methodology did the authors use?
Find AI/ML papers about retrieval reranking, inspect the strongest paper for its
reported benchmark result, then find newer work that cites it.
Search merged pull requests and documentation for the origin of this exact error:
"transport closed".Six read-only tools
All tools carry MCP's read-only annotation. MCP clients receive the complete generated input schema and descriptions when they connect.
Tool | Compact signature | Best for |
|
| General web search through Exa or Tavily, with source-separated results and Firecrawl fallback |
|
| Readable page bodies, concise summaries, or grounded answers through a tiered fetch cascade |
|
| Ranked AI/ML paper discovery through Firecrawl's Research Index |
|
| Paper metadata and optional query-relevant full-text passages for claim verification |
|
| Related papers, citers, or references guided by a natural-language intent |
|
| Developer documentation, repository READMEs, issues, and merged pull requests |
web_search supports relevance or date-oriented filters, domain inclusion/exclusion,
publication windows, generated per-result summaries, and Exa's fast/deep search modes.
Tavily receives equivalent controls where its API supports them; approximations and
dropped controls are disclosed in the output.
The research-paper index is arXiv-oriented and best suited to AI/ML. For scholarly
work outside that scope, use web_search(category="publication").
How retrieval works
Search routing
Exa is the default search provider. Select Tavily for one call with
provider="tavily", or globally:
export WEB_SEARCH_PROVIDER=tavilyOnly exa and tavily are accepted. If the selected provider fails, the server tries
Firecrawl and names both the failed primary provider and serving fallback in the
result.
Fetch routing
For an ordinary full-body request in automatic mode:
Exa indexed contents
↓ unusable or unavailable
guarded local Camoufox browser
↓ unusable or unavailable
optional Tavily Extract
↓ unusable or unavailable
Firecrawlrender="auto"uses that adaptive cascade.render="never"forbids the local browser.render="always"starts with the local browser and skips Exa; optional Tavily and Firecrawl remain backstops.mode="concise"asks for a compact generated summary.question="…"asks for a grounded answer rather than the whole page.max_age_hours=0forces fresh provider work;-1requests Exa's always-use-cache behavior; positive values set a freshness window.Explicit full-body requests above Exa's body-size ceiling start with a tier capable of satisfying the requested size rather than silently returning a short body.
Enable Tavily Extract per call with tavily=true, or globally:
export WEB_FETCH_TAVILY_TIER=1Tavily Extract is a full-body tier. The server skips it when it cannot honor a concise, question-answer, or explicit freshness contract.
Credential storage
The server resolves each API key lazily in this order:
EXA_API_KEY,FIRECRAWL_API_KEY, orTAVILY_API_KEYin the process environment.A dotenv-style private key file.
The optional Python
keyringpackage.macOS Keychain or Linux Secret Service command-line clients.
The default key file is:
Linux/macOS:
${XDG_CONFIG_HOME:-~/.config}/web-retrieval-mcp/keys.envWindows:
%APPDATA%\web-retrieval-mcp\keys.env
Override the file with WEB_RETRIEVAL_MCP_ENV_FILE, or its directory with
WEB_RETRIEVAL_MCP_CONFIG_DIR. On POSIX systems the file must not be readable by
group or other users:
install -d -m 700 ~/.config/web-retrieval-mcp
printf '%s\n' \
'EXA_API_KEY=<your-exa-api-key>' \
'FIRECRAWL_API_KEY=<your-firecrawl-api-key>' \
'TAVILY_API_KEY=<your-tavily-api-key>' \
> ~/.config/web-retrieval-mcp/keys.env
chmod 600 ~/.config/web-retrieval-mcp/keys.envFor keyring, store each secret under service web-retrieval-mcp, using the
environment-variable name as the username:
keyring set web-retrieval-mcp EXA_API_KEY
keyring set web-retrieval-mcp FIRECRAWL_API_KEY
keyring set web-retrieval-mcp TAVILY_API_KEYCredentials remain in process memory, are redacted from provider errors and displayed URLs, and are never put in provider command arguments.
Configuration reference
Setting | Default | Purpose |
| unset | Exa search and indexed page contents |
| unset | Firecrawl fallback, research papers, and developer search |
| unset | Tavily Search and Extract; also requires the |
|
| Default |
|
| Globally enable Tavily Extract in eligible fetch cascades |
| platform key-file path | Override the exact dotenv key file |
| platform config directory | Override the directory containing |
|
| Completed fetch cache: |
| unset | Absolute path to a private Valkey Unix-domain socket |
|
| Streamable HTTP bind address |
|
| Streamable HTTP port |
Boolean settings accept 1/0, true/false, yes/no, and on/off; invalid values
fail explicitly rather than being guessed.
Optional completed-result cache
Install the cache extra and point the server at a private, non-persistent Valkey
Unix-domain socket:
export WEB_RETRIEVAL_MCP_CACHE=on
export WEB_RETRIEVAL_MCP_VALKEY_SOCKET=/absolute/private/path/valkey.sockSuccessful eligible fetches are stored for 24 hours. Credential-bearing URLs,
userinfo, forced or positive freshness, and render="always" bypass replay. Cache
errors fail open to normal provider retrieval. Default auto enables caching only
when the Valkey client is installed on a non-Windows host; on still attempts the
configured Unix socket and fails open on errors, while off disables it. See the cache guide
for the privacy model, singleflight behavior, and a minimal sidecar configuration.
Streamable HTTP transport
stdio is the default and recommended transport for a local MCP client:
web-retrieval-mcp
# equivalent: python -m web_retrieval_mcpFor a shared local process, stateless Streamable HTTP is available:
web-retrieval-mcp --http --host 127.0.0.1 --port 8100HTTP transport has no built-in authentication. Keep it on loopback unless you add an authenticated perimeter and suitable network controls.
Optional Claude Code replacement policy
The package includes a PreToolUse hook that can deny Claude Code's built-in
WebSearch and WebFetch, directing agents to this MCP server instead. Preview the
exact settings change first:
web-retrieval-mcp-install --printInstall or remove it explicitly:
web-retrieval-mcp-install
web-retrieval-mcp-install --register-mcp
web-retrieval-mcp-install --uninstallThe installer is idempotent and backs up an existing settings file before writing. This optional hook is the only Claude Code-specific integration in the package. The MCP server itself remains client-neutral and works as a complementary tool without it.
Install from source
To install the reviewed source directly:
git clone https://github.com/VelvetSP/web-retrieval-mcp.git
cd web-retrieval-mcp
python -m pip install '.[all]'
python -m camoufox fetchFor development:
python -m pip install -e '.[all,dev]'
./run-tests.shThe release gate builds wheel and sdist artifacts, installs the wheel into a clean
virtual environment, exercises the installed MCP command against loopback provider
doubles, and runs the unit and transport suites. Browser or SSRF behavior changes also
require python test_ssrf_redirect_live.py. See the testing contract
for the feature matrix and acceptance contract.
Security model
Only
httpandhttpsinitial URLs are accepted.Initial hosts must resolve exclusively to globally routable addresses.
The local browser observes document requests and validates redirect hops before returning content to the caller.
Credentials are resolved in process and redacted from errors and displayed URLs.
Provider responses and page bodies are untrusted data; an agent must not treat instructions embedded in retrieved content as authority.
Application checks cannot prove that no packet reaches a private address during a DNS rebinding race. Deployments with that threat model need a validating forward proxy or equivalent network egress policy. See the security policy, and report suspected vulnerabilities through GitHub's private vulnerability reporting rather than a public issue.
Frequently asked questions
What is web-retrieval-mcp?
It is an open-source MCP server that gives AI agents reliable web search, tiered web page retrieval, research-paper discovery, and developer-source search. It works over local stdio with any compatible MCP client and can also serve stateless Streamable HTTP.
Is this an MCP web search server or an MCP web scraping server?
Both. web_search discovers and ranks pages; web_fetch extracts readable content
from one URL and can use a real local browser for JavaScript-heavy pages. The server
also exposes specialist paper and developer indexes that ordinary web scraping does
not provide.
Does it replace an agent's built-in web search?
It can, but it does not have to. The tool descriptions recommend an independent complementary lane because two retrieval systems can provide useful source diversity. Claude Code users can opt into the bundled replacement hook when they want one enforced route.
Which AI agents and MCP clients can use it?
The server exposes standard MCP over stdio by default and optional Streamable HTTP. Documented client examples include Codex CLI and the Codex IDE extension, Claude Code, Claude Desktop, Cursor, and ChatGPT desktop; other MCP-capable agents are not excluded. Compatibility depends on the client version and the MCP transport it supports, so use the matching client documentation alongside the server commands above.
Why use Exa, Firecrawl, Tavily, and Camoufox together?
They cover different failure modes. Exa provides indexed search and contents; Tavily is an alternate search provider and optional extractor; Camoufox renders pages locally; Firecrawl provides the final web fallback plus specialist research and developer indexes. The server chooses among them according to the request and discloses the tier that succeeded.
Do I need every provider key?
No. Exa is the default for general search and first-tier contents. Firecrawl enables fallbacks and all four research tools. Tavily is optional. The server lists all six tools even when an optional route is not configured and returns an actionable error if a call selects an unavailable capability.
Is web-retrieval-mcp free?
The project is MIT-licensed. Provider usage is billed under your own accounts; each provider currently offers an entry allocation, but quotas and prices can change. Camoufox runs locally without a metered retrieval API.
Which operating systems are supported?
The base Python server is platform-independent. Environment and key-file credentials work across Linux, macOS, and Windows. Valkey UDS caching is non-Windows, and local browser availability follows the supported Camoufox/Playwright platforms.
Does it support RAG and autonomous research agents?
Yes. The source-separated result contract, explicit URLs, controllable text budgets, grounded page questions, specialist indexes, and deterministic provenance make the tools suitable as a retrieval layer for RAG pipelines and research agents. The server returns evidence; the calling application remains responsible for evaluation, citation, and prompt-injection handling.
Project links
Contributions are welcome. Keep stdout reserved for JSON-RPC, send diagnostics to
stderr, add public-boundary acceptance coverage for changed behavior, and run
./run-tests.sh before opening a pull request.
License
MIT © VelvetSP
Available Tools
6 toolsresearch_githubARead-only
Search developer primary sources via the Firecrawl Developer Index: GitHub issues, merged pull requests and repository READMEs, PLUS curated documentation sites. Returns the matched passages in Markdown, so tables and code blocks survive. Use to find the CODE behind a paper, the issue where a bug was reported and fixed, an API contract, or the discussion behind an error message.
Args:
query: natural-language query (method, kernel, repo topic, error message).
k: number of results (1–25, default 8).
passages: matched passages per result (1–5, default 2).
types: restrict to any of exactly "doc", "issue", "pull_request", "readme".
NOTE the request spelling is pull_request (snake_case). The response's
repos[].types object uses camelCase (pullRequest) — echoing a key from
there back into this argument is refused, not silently ignored.
repos: "owner/repo" slugs. Scopes only the repository half of the index, so when
types is also given it must contain at least one of issue/pull_request/readme.
Falls back to the de-documented legacy /v2/search/research/github ONLY when the
Developer Index genuinely fails (transport, HTTP, malformed envelope, or every
requested type unavailable) — never on a legitimate empty result, and never when
types/repos were supplied, since the legacy endpoint accepts only query+k and
would silently answer a different question than the one asked.
| Name | Required | Description | Default |
|---|---|---|---|
| k | No | ||
| query | Yes | ||
| repos | No | ||
| types | No | ||
| passages | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the readOnlyHint annotation, the description discloses valuable behavioral details: results are returned in Markdown, the request spelling `pull_request` differs from the response's camelCase `pullRequest`, echoing a response key is refused rather than ignored, and the legacy fallback is strictly conditioned. This gives the agent a clear model of how the tool behaves in edge cases.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with the core purpose, followed by compact use-case examples, a clean Args breakdown, and a necessary fallback caveat. It is long but densely informative; every sentence contributes operational value and no filler is present.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's five parameters, legacy fallback, and naming-format pitfalls, the description is remarkably complete. It covers input semantics, constraints, edge-case behavior, and return format, while the presence of an output schema means detailed return structure documentation is not required here.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, but the description compensates fully by explaining every parameter: query's natural-language intent, k and passages ranges with defaults, valid types values, and repos format. It also documents the critical interaction between repos and types and the snake_case/camelCase pitfall.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: 'Search developer primary sources via the Firecrawl Developer Index', and enumerates the exact source types (GitHub issues, merged pull requests, READMEs, curated documentation). It also gives concrete use cases like finding the code behind a paper or a bug discussion, which clearly separates it from generic web search and paper-focused siblings.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides strong when-to-use guidance with explicit use cases and a detailed fallback policy explaining when the legacy endpoint may and may not be used. However, it does not explicitly name sibling alternatives like web_search or research_papers or state when to prefer them over this tool.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
research_paperARead-only
Inspect ONE paper from the Research Index by id (a paperId, or an arXiv id like "arxiv:2606.01509"). Without query → metadata (title, authors, categories, dates, abstract). With query → ALSO the top full-text passages answering it — use this to VERIFY a paper actually contains a method/dataset/result before relying on it.
Args: paper_id: paperId or primaryId ("arxiv:NNNN.NNNNN") from research_papers. query: optional question; when set, returns claim-verification passages.
| Name | Required | Description | Default |
|---|---|---|---|
| query | No | ||
| paper_id | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the readOnlyHint annotation, the description explains exactly what happens without a query (metadata fields listed) and with a query (top full-text passages). This gives the agent a clear model of the tool's conditional behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact, front-loaded, and every line adds information. The behavior is summarized first, followed by concise parameter details. No filler or redundant restatement.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With a readOnlyHint, an output schema, only two parameters, and a clear explanation of both input formats and output modes, the description fully equips an agent to select and invoke the tool correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must carry the full semantic load. It does: paper_id is explained as a paperId or primaryId with an example format, and query is described as an optional question returning verification passages.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb ('Inspect'), a specific resource ('ONE paper from the Research Index'), and the exact accepted identifiers. It differentiates from siblings like research_papers by emphasizing single-paper retrieval and the optional query mode.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly says when to use the tool: to VERIFY that a paper contains a method/dataset/result before relying on it. It implies research_papers is the source for finding paper IDs but does not explicitly state exclusions or alternatives beyond this.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
research_papersARead-only
Search 3M+ arXiv AI/ML papers via the Firecrawl Research Index (state-of-the-art paper recall — far better than general web search for finding the right literature). Returns ranked papers: title, arXiv id, relevance score, abstract. Then call research_paper(paper_id, query=…) to verify a claim against full text before citing.
SCOPE: arXiv-scoped, i.e. effectively AI/ML. For scholarly literature outside that scope (medicine, law, economics, humanities), use web_search(category="publication") — Exa's publications index (~350M works) covers what this one cannot.
Args: query: natural-language research query (topic, method, benchmark, author). k: number of papers (1–25, default 8).
| Name | Required | Description | Default |
|---|---|---|---|
| k | No | ||
| query | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already mark this tool read-only, so the description does not need to restate that. It adds useful behavioral context: the source index, arXiv scope, result ranking, and returned fields (title, arXiv id, relevance score, abstract), plus the downstream verification step. It does not discuss rate limits or error behavior, but the read-only annotation lowers the burden.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Information is front-loaded: the core search purpose and return shape appear first, the scope/routing caveat appears second, and parameter definitions are cleanly separated. Every sentence serves selection, workflow, or invocation without unnecessary filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a two-parameter read-only search tool with an output schema, the description covers purpose, scope, alternatives, return shape, and parameter semantics. The output schema handles detailed return structure, so nothing essential is left unspecified.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Although the input schema has no property descriptions, the Args block fully documents both parameters: query is a natural-language research query, and k is a count with an explicit 1–25 range and default 8. This fully compensates for the 0% schema description coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb-resource pair: 'Search 3M+ arXiv AI/ML papers via the Firecrawl Research Index', and explicitly states the tool is arXiv-scoped. It also distinguishes itself from siblings by describing the ranked paper list output and pointing to research_paper for verification, so an agent can select it without inspecting schemas.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives explicit selection criteria: use this for arXiv AI/ML literature and, for medicine/law/economics/humanities, use web_search(category="publication") because that index is larger. It also tells the agent to call research_paper to verify claims before citing, making the intended workflow clear.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
research_similarARead-only
Expand from a seed paper to related work via the Research Index. intent is a
REQUIRED natural-language description of the connection you want (e.g. "newer methods
that improve on this routing", "the work this paper builds on"). Returns ranked
related papers (same shape as research_papers).
SCOPE: arXiv-scoped like research_papers — for non-AI/ML literature use web_search(category="publication").
Args: paper_id: paperId or "arxiv:…" of the seed paper. intent: natural-language description of the kind of related work wanted. k: number of related papers (1–25, default 8; API allows up to 500). mode: "similar" (default), "citers" (papers citing this one), or "references" (papers this one cites). Unknown → "similar". min_score: renderer-side relevance floor (default 0.0 = off). k=8 already trims the low-score tail; raise this to filter more aggressively. rerank: optional bool; omitted from the request when None (API default is undocumented). Set True/False to force. (anchor — repeatable seed expansion — is not exposed; future work.)
| Name | Required | Description | Default |
|---|---|---|---|
| k | No | ||
| mode | No | similar | |
| intent | Yes | ||
| rerank | No | ||
| paper_id | Yes | ||
| min_score | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the readOnlyHint annotation, the description discloses meaningful behavior: mode semantics ('similar', 'citers', 'references'), fallback to 'similar' for unknown values, min_score as a renderer-side relevance floor, k's low-score trimming, and rerank being omitted when None because the API default is undocumented. It even notes the unexposed anchor parameter, giving the agent a fuller mental model.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The structure is front-loaded with the core purpose, followed by scope and a well-organized Args block. Every sentence adds value, including the note about anchor not being exposed, which prevents an agent from expecting an undocumented parameter. Despite the length, it is dense with necessary information rather than padded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
All six parameters are documented, the scope is clarified, a sibling alternative is provided, and the return shape is referenced as 'same shape as research_papers' with an output schema available. The description leaves no material gap for an agent to correctly select and invoke the tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, but the description fully compensates by explaining every parameter: paper_id accepts a paperId or 'arxiv:...' string, intent is a required natural-language description with examples, k has a range/default/API maximum, mode has enumerated meanings, min_score explains how it interacts with k, and rerank clarifies optionality and omission behavior. No parameter is left to guesswork.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific action and resource: 'Expand from a seed paper to related work via the Research Index.' It also distinguishes itself from siblings by noting it returns the same shape as research_papers and is arXiv-scoped, with a clear pointer to web_search for non-AI/ML literature. This is not a tautology and lets an agent understand exactly what the tool does.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly states when this tool applies: arXiv-scoped like research_papers, and it gives a concrete alternative: 'for non-AI/ML literature use web_search(category="publication").' It also implicitly distinguishes from research_paper/research_github by framing this tool as expansion from a seed paper to related work.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
web_fetchARead-only
Fetch a single URL's readable content. Full-body tier chain:
Exa contents → local Camoufox → optional Tavily Extract → Firecrawl.
Firecrawl is the paid last resort:
automatic full-body retrieval reaches it only after the local browser fails or
returns a body shorter than the useful-content floor. Camoufox is the one tier
that runs a real browser locally. Caller-supplied private URLs are refused before
tier selection in every render mode.
Eligible successful auto/never calls also use a host-wide 24-hour completed-result
cache in a private local Valkey sidecar. Local replay is disclosed separately from
the original provider's cache state; pass max_age_hours=0 to force provider work.
Signed/credential URLs, userinfo, positive freshness, max_age_hours=0, and
render="always" bypass completed replay. Every request is SSRF-validated before
cache access, and a hit is validated again immediately before its body is returned.
Returns content with a [served by: …] provenance header. In clients with
built-in page retrieval, use this as an independent complementary retrieval
lane; when built-in retrieval is disabled or unavailable, use it as the primary
fetch path.
Args:
url: the URL to fetch.
render: "auto" (default) → Exa, then local Camoufox, optional Tavily,
then Firecrawl. "never" skips Camoufox. "always" forces Camoufox
first and skips Exa; Tavily and Firecrawl remain backstops.
For mode="concise" or a question, auto intentionally uses Exa then
Firecrawl without launching Camoufox: the browser returns a full body
and cannot satisfy the promised summary/direct-answer shape.
max_chars: max characters to request/return. None (default) → 20000-char
budget, Exa-first. An explicit value ≤10000 stays Exa-first. For an
explicit value >10000 in full-body auto, Exa cannot meet the requested
size, so the order is Camoufox → optional Tavily → Firecrawl → Exa as a
final transparently truncated salvage tier. Browser-free full-body mode
starts with optional Tavily, then Firecrawl and Exa. Semantic requests remain Exa-first
because they use Exa's summary API rather than its capped body output.
Clamped to 1000–100000. When output is (or may be) clipped, a
[TRUNCATED at N chars — …] marker is appended on its own line.
max_age_hours: freshness window for the Exa AND Firecrawl tiers. None =
each tier's default cache (Exa default; Firecrawl ~2 days); 0 = force
fresh on both. -1 = "always use cache" (Exa-documented) —
Firecrawl has NO equivalent, so on that tier -1 is treated as omit
(its own default cache window applies instead; disclosed in the
[cache: …] line). Values below -1 are ignored (stderr note, default
cache used). Values above 720 (Exa's documented ceiling) are CLAMPED to
720, and the clamp is disclosed — a clamped value changes which content
can come back. The camoufox render tier is always live. When Firecrawl
serves with cache permitted, a [cache: …] disclosure line is added.
mode: "full" (default, whole readable body) or "concise" (a generated
summary — far fewer tokens). Honored by the Exa and Firecrawl tiers;
the camoufox render tier ignores it (returns full body, no [mode:]
line). Concise/question outputs carry a [mode: …] provenance line.
question: optional grounded-extraction query. When set, the tier returns a
direct ANSWER to the question (Exa summary-with-query / Firecrawl
question format) instead of the page body; short answers are accepted
— the floor is 1 char, so only an EMPTY answer cascades to the next
tier ("Paris"/"No" are legitimate answers). The 60-char
extract floor applies to mode="concise", and the 200-char floor to a
full body. Overrides mode.
tavily: enable or disable Tavily Extract for this call. When omitted,
WEB_FETCH_TAVILY_TIER is parsed strictly (default false). Tavily is a
full-body tier only and is skipped for concise/question requests and
whenever max_age_hours is explicit because it cannot honor those
contracts. Requires the tavily extra and TAVILY_API_KEY.
SSRF note: Camoufox follows redirects and re-resolves DNS, so every Camoufox attempt (automatic or render="always") is guarded:
_make_route_guardaborts any request whose host resolves non-public, classifying by RESOLVED IP (not the URL string), failing closed on a resolution error;_make_request_observer+_flush_pendingexist becausepage.routedoes NOT fire on a main-frame 3xx — they see the redirect hops the route guard alone would miss;_camoufox_renderraises on ablockedhop at FOUR checkpoints: after agotoexception (catches a navigation the guard itself aborted, raising the guard's reason instead of an opaque playwright error), after a successfulgoto, after thenetworkidlewait, and afterinner_text(a hop recorded during the extraction await, before any body returns).test_ssrf_redirect_live.pycovers this live. The observer detects a forbidden document request after Chromium has emitted it, so the application prevents private content from being returned but cannot prove that no outbound packet was sent. Chromium may also re-resolve after the guard's check. Full closure would need a validating forward proxy or equivalent network policy.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| mode | No | full | |
| render | No | auto | |
| tavily | No | ||
| question | No | ||
| max_chars | No | ||
| max_age_hours | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotatations only provide readOnlyHint, but the description discloses the full tier chain, cache behavior, bypass conditions, SSRF validation, truncation markers, and provenance headers. It even goes as far as admitting the residual limit that an outbound packet cannot be fully ruled out, which is exceptional transparency beyond the annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but well-structured and front-loaded: a one-sentence purpose, a tier-chain overview, then an Args section with each parameter explained. The density is justified by the tool's complexity, and the organization allows an agent to extract common-case usage quickly while still having edge-case details available.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given 7 parameters, 0% schema coverage, and safety-sensitive SSRF behavior, the description fully covers purpose, parameter semantics, caching, render modes, return provenance, and security limitations. No essential decision an agent needs to make before invoking the tool is left undocumented.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description carries the full burden. It thoroughly documents all seven parameters, including defaults, clamped ranges, tier-order changes, cross-parameter interactions, and edge cases like max_age_hours=-1 and the >10000 max_chars behavior.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with 'Fetch a single URL's readable content'—a specific verb, resource, and output type. It clearly distinguishes web_fetch from the research_* and web_search siblings by scoping it to one URL and describing the readable-content retrieval role.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly states when to use this tool as a complementary retrieval lane versus the primary fetch path, and details render-mode selection rules (auto/never/always). It also explains when individual tiers are skipped, such as Tavily being omitted for concise/question requests, which gives clear when-to-use and when-not-to-use guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
web_searchARead-only
Search the web via Exa or Tavily. Returns one block per result, each with its OWN title, URL, highlights, and text — never a merged summary — plus a Sources trailer. In clients with built-in web search, use this as an independent complementary retrieval lane; when built-in search is disabled or unavailable, use it as the primary search path.
Args:
query: the search query.
num_results: how many results (default 8).
mode: search mode — "auto" (default), "fast" (~450ms), "instant" (~250ms),
or the deep-research family "deep-lite" (≈4s), "deep" ($12/1k), or
"deep-reasoning" (12–40s, $15/1k). Deep modes request a synthesized
answer via Exa's outputSchema and return a
"## Synthesized answer" block — with a "Grounding:" citation line when
Exa returns one — plus per-result blocks; this costs ~2s of synthesis
latency on top of the mode's own search time, which is why it is scoped
to the deep family only (their timeout budget already covers it).
Legacy "neural"/"keyword" map to "auto" (deprecated).
text_chars: per-result body length (default 1200; clamped 200–8000). Now
governs BOTH the Exa request size (request-what-you-render) and the
rendered cap — raise it to surface more body text per result.
recency_days: only results published within the last N days (maps to
startPublishedDate). Pass this for time-sensitive/"latest" queries —
the tool does NOT auto-tighten dates on its own.
recency_hours: like recency_days but HOUR granularity — reaches
BOTH tiers (Exa via a full ISO timestamp; the Firecrawl fallback via
tbs qdr:h/cdr:1,cd_min:…). Full precedence ladder, same on both tiers:
explicit VALID start_published_date > recency_hours > recency_days. An
invalid/absent value at each tier falls through to the next.
start_published_date / end_published_date: ISO date bounds ("2026-01-15"
or a full ISO timestamp). Wins over recency_hours/recency_days per the
ladder above on both tiers. The fallback represents an explicit start
with Firecrawl's custom-date-range syntax.
Unparseable values are ignored (stderr note), never an error.
category: Exa category hint ("news", "publication", "company", "people",
"financial report", "personal site", or a free string). NOTE: "company"
and "people" forbid date filters + excludeDomains (dropped
automatically), and "people" restricts includeDomains to Exa's
supported profile domains (it 400s otherwise — surfaced as
SEARCH_FAILED / fallback).
Enum migration: "research paper" was
RENAMED to "publication" and is auto-aliased forward; "tweet" is GONE
(Exa 400s) and is dropped, leaving the search unscoped; "pdf"/"github"
are deprecated-but-live and pass through. Each of those emits a
[category=… ] line in the response header, since dropping or renaming
a category changes which results you get back.
include_domains / exclude_domains: restrict/exclude result hosts (capped
at 20 entries each).
summary: when True, request a generated per-result summary instead of raw
text/highlights (fewer tokens; text + highlights omitted from the request).
sort_by_date: sort results by publish date instead of relevance.
Honoured ONLY on the Firecrawl fallback tier (Firecrawl's sbd:1); Exa
has no equivalent — when the Exa tier serves and this was requested, the
response header discloses that it was not applied rather than silently
ignoring it.
provider: "exa" or "tavily". When omitted, WEB_SEARCH_PROVIDER is used,
defaulting to Exa. Invalid values fail explicitly. Tavily requires the
tavily extra and TAVILY_API_KEY.
| Name | Required | Description | Default |
|---|---|---|---|
| mode | No | auto | |
| query | Yes | ||
| summary | No | ||
| category | No | ||
| provider | No | ||
| text_chars | No | ||
| num_results | No | ||
| recency_days | No | ||
| sort_by_date | No | ||
| recency_hours | No | ||
| exclude_domains | No | ||
| include_domains | No | ||
| end_published_date | No | ||
| start_published_date | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations only declare readOnlyHint=true, but the description discloses far more: results are never merged into a combined summary; deep modes add latency and cost; recency filters follow an explicit precedence ladder; category values like 'people' can cause 400s; and sort_by_date is silently not applied on Exa but is disclosed in the response header. These behavioral traits go well beyond the annotations and give an agent accurate expectations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long, but the length is warranted by 14 parameters and 0% schema description coverage. It is front-loaded with the core behavior and return format, then organized as a labeled Args list where each parameter earns its place with operational detail. A few sentences could be tightened, but none are pure filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Combined with the readOnlyHint and the presence of an output schema, the description covers return granularity, mode behavior, provider fallback, parameter interactions, failure modes, and date-filtering semantics. An agent has enough context to decide whether to call this tool and how to set the right parameters without needing to guess.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must carry the full burden, and it does. The Args section documents every parameter, including defaults, clamping ('clamped 200–8000'), precedence ('explicit VALID start_published_date > recency_hours > recency_days'), enum migrations, and provider-specific behavior. It fully compensates for the missing schema descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb-resource pair — 'Search the web via Exa or Tavily' — and immediately defines the return shape: 'one block per result, each with its OWN title, URL, highlights, and text — never a merged summary'. It clearly differentiates this tool from built-in web search and from the research-paper-specific sibling tools by framing it as general-purpose web search, but it does not explicitly name sibling tools as alternatives.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly tells an agent when to use this tool: 'In clients with built-in web search, use this as an independent complementary retrieval lane; when built-in search is disabled or unavailable, use it as the primary search path.' This is a direct, actionable selection rule with respect to the most relevant alternative, and it also warns about provider requirements such as 'Tavily requires the tavily extra and TAVILY_API_KEY.'
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.
6 tool updates
v0.2.0- First observed
research_github - First observed
research_paper - First observed
research_papers - First observed
research_similar - First observed
web_fetch - First observed
web_search
TDQS
Scored across 6 tools
Each tool targets a distinct retrieval surface: general web search, single-page fetch, arXiv paper search, single-paper inspection, related-paper expansion, and developer-index search. Potential overlaps are explicitly separated by corpus and use case, so an agent can reliably pick the right tool.
All names are lowercase snake_case and grouped by domain (web_* and research_*), which makes them predictable. The minor inconsistency is that web_search/web_fetch are verb-object names, while the research_* tools are object-oriented and do not encode an action as clearly.
Six tools is well-scoped for a retrieval server: two general web operations plus a four-tool research cluster. Each tool has a distinct responsibility, and there are no redundant or filler tools.
The surface covers general web search, full-page fetching, scholarly paper search, single-paper verification, related-paper exploration, and developer-source search. These cover the main retrieval workflows one would expect, with no obvious dead ends or missing core operations.
Maintenance
Resources
Unclaimed servers have limited discoverability.
Looking for Admin?
If you are the server author, to access and configure the admin panel.
Related MCP Connectors
Docs: https://docs.keenable.ai/mcp-server Keenable is a free, remote MCP server that gives agents access to the web index. Search the web with ranked results and date/site filters, then fetch any indexed page as clean markdown. Works out of the box with no account or API key.
Scrape, crawl and search the web for AI agents via MCP.
An MCP server that gives your AI access to the source code and docs of all public github repos
Related MCP Servers
- AlicenseAqualityBmaintenanceA lightweight MCP server that gives AI assistants real-time web search and URL reading — no API keys required.2GPL 3.0
- AlicenseNot gradedqualityAmaintenanceA self-contained web-research MCP server that lets local LLM agents search, fetch, and synthesize web content using tools like web_search, web_fetch, and web_research.1MIT

Wickofficial
FlicenseNot gradedqualityAmaintenanceAn MCP server that provides browser-grade web access for AI agents, using Chrome's actual network stack to bypass anti-bot protections and return clean markdown.8-- FlicenseNot gradedqualityCmaintenanceMCP server that enables AI agents to search the web and extract clean Markdown content, with support for JavaScript rendering, structured data extraction, and screenshots.1-
Latest Blog Posts
- Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)By Om-Shree-0709 on .mcpAgent IdentityOAuth 2.1
- Your AI Chatbot Just Exposed Your CEO's Salary to an InternBy Om-Shree-0709 on .Agent IdentityMCP SecurityOAuth Delegation
- Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)By Om-Shree-0709 on .Agentic AiPrompt InjectionWebAssembly
MCP directory API
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/VelvetSP/web-retrieval-mcp'
If you have feedback or need assistance with the MCP directory API, please join our Discord server