webscout-mcp
webscout-mcp
面向 AI 代理的 Web 搜索与抓取工具,以 MCP 服务器形式提供。支持搜索、抓取、爬取,并从网页中提取结构化数据——无需 API 密钥,不按请求计费,一切都在你的机器上运行。
安装
pip install webscout-mcp需要 Python 3.10 或更高版本。
Related MCP server: Crawl4AI RAG MCP Server
快速开始
将其添加到你的 MCP 客户端配置中(Claude Code、Cursor、Codex 等):
{
"mcpServers": {
"webscout": {
"command": "webscout-mcp",
"args": []
}
}
}就这样。你的代理将获得六个工具:
web_search- 通过 Bing 搜索,自动回退到 DuckDuckGo HTML,无需密钥web_fetch- 抓取页面并提取正文(markdown/text/html)web_crawl- 支持深度和页数限制的并发 BFS 爬取,遵守 robots.txtweb_extract- 使用 CSS 选择器、属性、正则表达式提取结构化数据cache_stats- 检查本地缓存cache_clear- 清空缓存
CLI 用法
除了作为 MCP 服务器运行外,你还可以直接从命令行使用 webscout-mcp:
# Search the web (outputs JSON)
webscout-mcp search "python async libraries" --max-results 5
# Fetch a page (raw content)
webscout-mcp fetch https://example.com --extract --format markdown --raw
# Crawl a site
webscout-mcp crawl https://example.com --depth 2 --pages 10
# Start MCP server (default if no command given)
webscout-mcp serve --transport stdio使用示例
搜索
web_search(query="best python async libraries", max_results=5)返回结构化结果,包含标题、URL、摘要,以及提供服务的后端(bing 或 duckduckgo)。如果 Bing 失败或更改了其标记结构,引擎会自动回退到 DuckDuckGo 的 HTML 版本——无需任何配置。
抓取页面
web_fetch(url="https://example.com", extract=true, output_format="markdown")extract=true 会运行 trafilatura(带 readability-lxml 回退)来去除导航、广告和侧边栏——你得到的是干净的正文内容,而不是原始 HTML。
提取结构化数据
web_extract(
url="https://example.com/products",
rules='[
{"name": "titles", "selector": ".product h2", "multiple": true},
{"name": "prices", "selector": ".price", "regex": "\\$([\\d.]+)", "multiple": true},
{"name": "links", "selector": "a.product", "attribute": "href", "multiple": true}
]'
)每条规则支持 selector、attribute、multiple、regex 和 default。
爬取网站
web_crawl(seed_url="https://example.com", max_depth=2, max_pages=10, concurrency=5)每个深度级别的页面会并发抓取(由 concurrency 控制,默认 5)。爬虫默认遵守 robots.txt——被禁止的 URL 会被跳过并计入 skipped_robots。同域限制默认开启。
作为 Python 库使用
import asyncio
from webscout_mcp import Config, Fetcher, SearchEngine
async def main():
config = Config.from_env()
config.ensure_dirs()
fetcher = Fetcher(config)
result = await fetcher.fetch("https://example.com", extract=True)
print(result.title)
print(result.content[:500])
await fetcher.close()
search = SearchEngine(config)
results = await search.search("python async", max_results=5)
for r in results:
print(f"{r.position}. {r.title} - {r.url} ({r.backend})")
await search.close()
asyncio.run(main())工作原理
搜索 先尝试 Bing,然后 DuckDuckGo HTML——两者都通过直接 HTTP 抓取实现,无需 API 密钥。结果按查询词缓存。
抓取 使用 httpx,带指数退避重试(所有 httpx 错误 + HTTP 5xx)、按域令牌桶限速,以及 5 MB 内容上限。
内容提取 以 trafilatura 为主,readability-lxml 自动回退——与许多稍后阅读服务背后使用的库相同。
缓存 使用 SQLite,带 TTL 和大小上限;旧条目自动淘汰。重复抓取和搜索零成本。
爬取 是并发 BFS,支持可配置的深度、页数、并发度、同域限制和 robots.txt 合规。使用首次抓取得到的原始 HTML,避免对每个页面重复抓取。
代理支持 通过配置或环境变量,将所有 HTTP/HTTPS 请求路由到代理。
日志 是结构化的,可通过
WEBSCOUT_LOG_LEVEL(DEBUG/INFO/WARNING/ERROR)和WEBSCOUT_LOG_JSON=1(JSON 输出)配置。
一切都在本地运行。你的数据不会离开你的机器。
配置
所有设置都有合理的默认值。可通过环境变量(WEBSCOUT_ 前缀)、TOML 配置文件或 CLI 标志覆盖。
配置文件
创建 ~/.config/webscout/config.toml(或 $XDG_CONFIG_HOME/webscout/config.toml):
[cache]
ttl = 7200
max_size_mb = 512
[fetch]
timeout = 15.0
max_retries = 3
[proxy]
http = "http://proxy:8080"
https = "http://proxy:8080"
[search]
max_results = 10
backends = ["bing", "duckduckgo"]
[crawler]
max_depth = 2
max_pages = 20
concurrency = 5
respect_robots = true
[logging]
level = "WARNING"
json = false环境变量会覆盖配置文件中的值。
环境变量
变量 | 默认值 | 作用 |
|
| SQLite 缓存的存放位置 |
|
| 缓存条目的有效期(秒) |
|
| 淘汰前的最大缓存大小 |
|
| HTTP 超时(秒) |
|
| 每个请求的重试次数 |
|
| 每个域每秒最大请求数 |
|
| 默认搜索结果数量 |
|
| 逗号分隔的后端顺序 |
|
| 默认爬取深度 |
|
| 每次爬取的最大页数 |
|
| 每个深度级别的并发抓取数 |
|
| 爬虫是否遵守 robots.txt |
|
| 默认提取输出格式 |
| (空) | HTTP 代理 URL |
| (空) | HTTPS 代理 URL |
|
| 日志详细程度 |
|
| 设为 |
CLI 标志会覆盖环境变量:
webscout-mcp --cache-ttl 3600 --cache-dir /tmp/webscout serve传输方式
# stdio (default - works with Claude Code, Cursor, etc.)
webscout-mcp
# SSE (for remote or browser-based clients)
webscout-mcp serve --transport sse --host 0.0.0.0 --port 8000更新日志
0.3.0
TOML 配置文件支持:除了环境变量外,还可通过
~/.config/webscout/config.toml配置HTTP/HTTPS 代理支持:将所有请求路由到代理
双内容提取:trafilatura 为主,readability-lxml 自动回退
搜索结果去重:移除重复 URL,重新编号位置
区域感知搜索:
region参数现在会实际传递给 Bing 和 DuckDuckGo爬虫性能优化:消除了每页的重复抓取——爬取速度约提升 2 倍
更好的重试逻辑:对所有 httpx 错误和 HTTP 5xx 进行重试
修复了内容类型检测:正确的 HTML/XML 检测
0.2.0
多后端搜索:Bing + DuckDuckGo HTML,带自动故障转移
并发爬虫,支持可配置的并行度
robots.txt 合规(可配置,默认开启)
CLI 子命令:
search、fetch、crawl、serve结构化日志,带控制台和 JSON 格式化器
自定义异常层级,更好的错误处理
新配置项:
WEBSCOUT_SEARCH_BACKENDS、WEBSCOUT_CRAWLER_CONCURRENCY、WEBSCOUT_RESPECT_ROBOTS
0.1.0
初始版本:web_search、web_fetch、web_crawl、web_extract、cache_stats、cache_clear
SQLite 缓存,带 TTL 和基于大小的淘汰
按域令牌桶限速
指数退避重试
trafilatura 内容提取
开发
git clone https://github.com/wxs-lang/webscout-mcp.git
cd webscout-mcp
pip install -e ".[dev]"
pytest许可证
MIT
Available Tools
11 toolsbroken_linksA
Check for broken links on a web page.
Extracts all links from the page, checks each one's HTTP status, and returns a report with broken links, redirects, and link statistics.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| timeout | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description must disclose behavior on its own. It states that it extracts all links, checks each one's HTTP status, and returns a report with broken links, redirects, and statistics. It does not mention rate limits or external-link handeling, but core behavior is transparent for a read-only checker.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two short sentences front-load the purpose and then add mechanism and report contents without fluff. Every sentence earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple two-parameter tool, the description covers purpose, behavior, and report contents, and an output schema exists. The missing timeout semantics and lack of alternative-routing guidance leave a measurable gap for correct invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has zero description coverage, so the description needed to compensate for both parameters. It only implies the URL via 'web page' and never explains the timeout parameter or its units, leaving the default 10 ambiguous.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a specific verb ('Check') and resource ('web page') and explains that it inspects HTTP statuses of extracted links, which separates it from generic web_fetch/crawl tools. It does not explicitly name sibling alternatives, so it stops just short of a 5.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The intended use is clearly implied: use it when you need a broken-link audit of a page. It does not say when to prefer this over web_fetch, web_crawl, or content_quality, nor does it give exclusions, so the guidance remains implicit.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
cache_clearA
Clear all cached entries. Returns the number of entries deleted.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full burden. It discloses the action ('clear') and the return value (count), but does not warn about irreversibility or potential system impact. For a destructive operation, more cautionary detail is expected.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, focused sentence that imparts both the action and the return value with no filler. It is perfectly concise for the tool's simplicity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a zero-parameter, simple operation, the description covers the action and result adequately. However, it omits any mention of side effects (e.g., performance degradation, need for re-fetching) or safety warnings, leaving a complete picture slightly lacking.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters, so the baseline is 4. The description does not need to explain parameters, and no parameter information is omitted since the schema is empty.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states a specific action ('Clear') on a specific resource ('all cached entries'), which is distinct from sibling tools like web_search or cache_stats. No ambiguity about the tool's function.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives. It does not mention prerequisites, contexts (e.g., when cache is stale), or warn against misuse. Sibling tool cache_stats exists but no differentiation is offered.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
cache_statsA
Return cache statistics: entry count, total size, TTL, and limits.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It states the tool 'returns' statistics, implying a read-only operation with no side effects, but it does not explicitly affirm that it does not modify the cache, nor does it mention any access requirements or potential large payloads. The listed return fields add transparency, but the lack of explicit behavioral caveats keeps this at a passing but not exceptional level.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, efficient sentence that front-loads the core purpose and then lists the specific statistics returned. Every word contributes value; there is no redundancy or filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity (zero parameters, no annotations) and the presence of an output schema (which likely documents the return format), the description covers the essential behavior. It could be slightly more complete by explicitly stating that this is a read-only inspection and noting any prerequisites, but nothing critical is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema is empty (0 parameters), so there are no parameters to describe. Per the rubric, a zero-parameter tool gets a baseline of 4. The description adds no parameter information because none exists, which is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly specifies a verb ('return') and a resource ('cache statistics'), and enumerates exactly what is returned (entry count, total size, TTL, limits). It is unambiguously distinct from sibling tools like web_search or cache_clear, which are about web operations or cache clearing.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description indicates the tool is for inspecting cache state, which provides clear context for when to use it (when cache statistics are needed). However, it does not explicitly mention alternatives or exclusions, though the distinct resource and zero parameters make usage fairly obvious without further guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
content_qualityA
Analyze content quality of a web page.
Evaluates readability scores (Flesch-Kincaid, Gunning Fog), keyword density, content structure, metadata quality, and provides actionable suggestions for improvement.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the burden of behavioral disclosure. It states what the tool evaluates and that it returns improvement suggestions, which signals a read-only analysis. However, it does not mention any constraints (e.g., public URL requirement, JS-heavy page limitations) or explicitly confirm an absence of side effects, so some transparency is missing.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is brief and front-loaded with the core purpose, followed directly by the evaluation criteria. There is no redundant or filler content; each sentence earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Though the tool has only one parameter and an output schema supplies the return value shape, the description provides the key evaluative dimensions. It does not mention network prerequisites or usage boundaries, but these are not essential for a simple analysis tool. Overall, an agent can invoke it correctly with just a URL.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has a single 'url' property with zero description coverage, so the description must compensate. It does so by naming the resource as a 'web page,' making the parameter's purpose clear. However, it does not specify URL format or expected scope, relying on common sense. This is sufficient for a simple parameter but not a robust semantic addition.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('Analyze') and resource ('content quality of a web page') and lists concrete evaluation dimensions (Flesch-Kincaid, Gunning Fog, keyword density, content structure, metadata quality). This distinguishes it from siblings like metadata_extract and web_extract, which focus on extraction rather than quality scoring.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description clearly implies this tool is for content quality auditing of a web page, but it does not explicitly state when to choose it over siblings such as metadata_extract or web_extract, nor does it provide when-not-to-use conditions. It relies on the agent to infer usage context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
metadata_extractA
Extract metadata from a web page.
Extracts JSON-LD, OpenGraph, Twitter Cards, article metadata, images, links, and other structured metadata from the page.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description must carry the full behavioral burden. It clearly states what data is extracted, but it does not disclose operational traits such as network fetching behavior, caching, failure modes, or whether only static HTML is processed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two short, purposeful sentences. The first is a concise one-line summary, and the second adds a useful enumeration of what is extracted, with no filler or repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a single-parameter tool with an output schema, the core invocation is mostly covered. However, the description lacks usage boundary guidance relative to sibling tools and does not mention operational limitations, leaving some selection and invocation context incomplete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has 0% description coverage, and the description only says 'from a web page' without adding details about URL format, allowed schemes, encoding, or required forms. The parameter name 'url' is self-explanatory, but the description does not meaningfully compensate for the schema's silence.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a clear verb-object statement, 'Extract metadata from a web page', and then enumerates the specific metadata types returned, including JSON-LD, OpenGraph, Twitter Cards, images, and links. This makes the tool's purpose precise and readily distinguishable from sibling tools like web_extract or web_fetch.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No explicit when-to-use or when-not-to-use guidance is given, and no alternatives are mentioned. The specialized vocabulary implies this is for structured metadata extraction, but an agent receives no direct cue about how to choose between metadata_extract, web_extract, and web_fetch.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
rss_parseA
Parse an RSS or Atom feed and return its entries.
Fetches and parses RSS 2.0, RSS 1.0, and Atom feeds. Returns feed title, description, link, and a list of entries with title, link, description, publication date, and author.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| max_entries | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description must carry the behavioral burden. It discloses that the tool fetches feeds, supports specific formats, and returns a defined set of fields. It does not address failure modes, limits, or side effects beyond the network fetch, and it does not explain how max_entries affects the result. This is partial transparency.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact and front-loaded, with the core action in the first sentence and supporting details following. It is slightly repetitive when stating supported formats twice, but overall every sentence contributes.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool is simple, but the description omits any explanation of the max_entries parameter and offers no guidance on selecting the tool relative to siblings. The return fields are listed, and an output schema exists, so return semantics are covered. Overall it is adequate for a basic call but not fully complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema provides only parameter names (url, max_entries) with no descriptions, and the description adds no parameter-specific meaning. There is no mention of what max_entries controls or the expected URL format. With 0% schema description coverage, this is a significant gap.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description leads with a specific verb and resource: 'Parse an RSS or Atom feed and return its entries.' It further specifies supported formats (RSS 2.0, RSS 1.0, Atom) and output fields, making its function unmistakable and distinct from sibling web-fetching tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description clearly frames the tool as the dedicated RSS/Atom parser, which implies the appropriate context. However, it does not explicitly state when to choose it over sibling tools like web_fetch or web_extract, nor does it mention exclusions or alternatives. The context is clear but guidance is implicit.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
search_healthA
Get health report for all search backends.
Returns overall health score, per-backend status (healthy/degraded/open/half-open), circuit breaker state, and request statistics. Use this to diagnose search failures.
Reports the active SearchService health first, with legacy SearchEngine health included as fallback reference during the migration period.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It discloses the exact content of the report (overall score, per-backend statuses, circuit breaker state, request statistics) and reveals the ordering behavior with active SearchService health first and legacy SearchEngine health as fallback during migration. It does not explicitly state that the operation is read-only, but as a health report this is strongly implied; the disclosed details go well beyond a minimal summary.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise and well structured: a one-line purpose, a sentence listing outputs and use case, and a final clarifying sentence about the migration fallback. Every sentence earns its place, and the core purpose is front-loaded in the first line with no redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a zero-parameter health check with an output schema present, the description is complete. It states what the tool does, when to use it, what it returns, and includes relevant legacy migration context. The presence of an output schema means the description doesn't need to detail return structures, and it adds all necessary usage and behavioral context.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters and an empty input schema, so the baseline for this dimension is 4. The description doesn't need to explain parameters and correctly focuses on the tool's output and purpose. It adds no parameter-specific information because there is none, which is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: 'Get health report for all search backends.' This clearly states the tool's function and differentiates it from siblings like cache_stats, which target cache health rather than search backend health. The rest of the description enumerates concrete outputs, leaving no doubt about the tool's purpose.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly says 'Use this to diagnose search failures,' providing a clear when-to-use context. It doesn't name alternatives or exclusions, but among the sibling tools there is no close alternative for search backend health, so the guidance is adequate without needing exclusionary language.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
web_crawlB
Crawl a website starting from a seed URL, respecting depth and page limits.
Fetches pages concurrently within each depth level. Respects robots.txt by default.
| Name | Required | Description | Default |
|---|---|---|---|
| extract | No | ||
| seed_url | Yes | ||
| max_depth | No | ||
| max_pages | No | ||
| concurrency | No | ||
| same_domain_only | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It does add useful details: concurrent fetching per depth level and robots.txt respect by default. However, it omits important behaviors such as how limits (max_pages, max_depth) are enforced, what happens on redirects or errors, and how same_domain_only is applied. Some behavior is disclosed, but the coverage is partial.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two short sentences, with the core purpose front-loaded. The second sentence adds relevant behavioral details without fluff. Every phrase earns its place; no redundancy or filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a crawl tool with six parameters, no annotations, and an output schema that is not explained, the description is thin. It does not cover how results are returned, how same-domain restrictions work, error handling, or how the crawl terminates. While the output schema exists, the description still lacks context about default behaviors and edge cases that an agent needs to invoke the tool safely and effectively.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has 0% description coverage, so the description must compensate by explaining parameter roles. The description only vaguely hints at depth and page limits ('respecting depth and page limits') and concurrency ('Fetches pages concurrently'), but does not explicitly define any of the six parameters: seed_url, extract, max_depth, max_pages, concurrency, or same_domain_only. This is minimal compensation for a schema with no descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's action: 'Crawl a website starting from a seed URL, respecting depth and page limits.' The verb 'crawl' is specific and distinguishes it from siblings like web_fetch (single page) and web_search (query-based). No ambiguity about the resource or scope.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives no explicit guidance on when to use web_crawl versus alternatives. It does not mention web_fetch, web_search, or other siblings, nor does it state conditions like 'use this for multi-page traversal or when you need to discover links.' The only implicit clue is the word 'crawl,' which is not enough for an agent to reliably choose it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
web_extractC
Extract structured data from a web page using CSS selectors.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| rules | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure, but it only states the core action. It doesn't describe behavior for selectors that match nothing, whether the page must be pre-fetched/cached, rate limits, or error handling. For a network tool this is a notable gap.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single efficient sentence with zero filler, front-loading the core purpose and mechanism. It earns its length, though it could have used its brevity to add a critical detail about the rules format.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The output schema covers the return format, but the tool has two required parameters, one of which (rules) has an unspecified format, no usage guidance relative to siblings, and no behavioral expectations. For a 2-param tool this is thin — the agent lacks enough to invoke it correctly on the first try.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It tells us the tool uses CSS selectors (hinting at the rules parameter) and that url is the target page, but it doesn't specify the rules string format — is it a JSON mapping of field names to selectors, a list, or a single selector? This ambiguity puts the burden on the agent to guess.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb (extract) + resource (web page) + mechanism (CSS selectors), and the phrase 'structured data' signals it differs from simply fetching raw content. This conceptually distinguishes it from siblings like web_fetch and web_crawl, which an agent could infer even without explicit sibling comparison.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use web_extract versus web_fetch or web_search. The description doesn't state when extraction is preferable to a raw fetch, whether extraction works on JS-rendered pages, or any conditions under which a sibling should be chosen instead.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
web_fetchA
Fetch a URL and return one window of its extracted main content.
Progressive delivery: each call returns at most max_chars chars
(default 8000). If the response contains continuation.has_more=true,
more content is available locally. To read the next window, call this
SAME tool again with the SAME url/extract/output_format/max_chars and set
start_char to the previous continuation.next_start_char. Follow-up
windows are served from a local snapshot (no new HTTP fetch, extraction,
or browser). Stop when has_more is false. You do not need to read the
whole page unless the task requires it.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| extract | No | ||
| max_chars | No | ||
| start_char | No | ||
| bypass_cache | No | ||
| output_format | No | markdown |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full behavioral burden. It discloses that each call returns at most max_chars, that follow-up windows are served from a local snapshot without new HTTP fetches, and that continuation is indicated by has_more. This is strong transparency, though it omits details on error handling or the behavior of the bypass_cache parameter.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with the purpose and then provides necessary detail on the progressive delivery pattern. It is slightly verbose but every sentence adds value about the continuation mechanism and stopping condition. It could be tightened, but the structure is logical and efficient.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given that an output schema exists (which likely defines the return structure and continuation fields), the description covers the core interaction loop, including how to iterate and when to stop. It does not explain error handling or edge cases, but these are likely covered by the output schema or are secondary. For an agent to successfully use the tool, the provided guidance is sufficient.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It explains start_char and max_chars in the context of continuation, and mentions extract/output_format/max_chars as required for repeat calls, but does not define what extract or output_format do (e.g., what formats are allowed, what extract=false means). It partially compensates for the schema gap but leaves several parameters unexplained.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The first sentence states a clear verb ('Fetch'), resource ('a URL'), and outcome ('return one window of its extracted main content'). It distinguishes the tool from siblings like web_crawl (crawling) and web_search (searching) by focusing on single-URL extraction with a windowing mechanism. The purpose is specific and unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides explicit step-by-step usage for progressive delivery: how to call again with start_char, when to stop (has_more=false), and a note that full reading is optional. It does not explicitly compare against alternatives like web_extract, but the usage pattern is clearly explained and actionable.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
web_searchB
Search the web and return structured results.
SearchService uses health-based provider ranking, deterministic recovery classification, and automatic fallback across registered search providers. Empty/invalid queries are rejected without network calls. No API key required for the default HTML providers.
| Name | Required | Description | Default |
|---|---|---|---|
| query | Yes | ||
| region | No | wt-wt | |
| max_results | No | ||
| safe_search | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and does add meaningful behavior: it rejects empty/invalid queries without network calls, uses automatic fallback, and requires no API key for default providers. However, it omits rate limits, failure modes, and result ordering, so transparency is only partial.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The purpose is front-loaded in the first sentence, and the remaining three sentences are relatively compact. The phrase 'deterministic recovery classification' is somewhat jargon-heavy, but overall the description is not bloated.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description plus schema provides basic context: an output schema exists, and the description explains provider selection and prerequisites. But because parameter semantics are absent and usage guidance is missing, an agent could still misuse region or max_results, so completeness is mediocre.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has 0% semantic coverage and the description only indirectly addresses the query parameter through 'Empty/invalid queries rejected.' It says nothing about region format/values, max_results limits, or safe_search behavior, so it fails to compensate for the schema gap.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with 'Search the web and return structured results,' naming a specific verb and resource clearly. However, it does not explicitly distinguish itself from siblings like web_crawl or web_extract, so differentiation is partial.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use web_search versus alternatives such as web_fetch, web_crawl, or web_extract. The implementation details about provider ranking and fallback do not help an agent select between tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
1 tool update
v1.3.0- Changed
web_fetch1 field changed- added
Input schema / properties / start_charAdded value: +{ + "default": 0, + "title": "Start Char", + "type": "integer" +}
5 tool updates
v0.6.0- Added
broken_links - Added
content_quality - Added
metadata_extract - Added
rss_parse - Added
search_health
6 tool updates
v0.3.0- First observed
cache_clear - First observed
cache_stats - First observed
web_crawl - First observed
web_extract - First observed
web_fetch - First observed
web_search
TDQS
Scored across 11 tools
Most tools have clearly distinct purposes, e.g., web_search, web_fetch, web_crawl, rss_parse. A small amount of overlap exists between web_extract and metadata_extract, but their descriptions clarify the difference between CSS-selector extraction and metadata extraction.
Names are readable and mostly use snake_case, but the pattern is mixed: some are object_verb (web_fetch, rss_parse, cache_clear), while others are noun phrases (broken_links, content_quality, search_health). The web_ prefix helps, but there is no consistent verb_noun convention.
11 tools is well within the ideal range and each tool serves a distinct web-research purpose, from searching and fetching to crawling, parsing, extracting, and cache management. No tool feels redundant or unnecessary.
The tool surface covers the core web-scouting workflow well: search, fetch, crawl, parse RSS, extract structured data and metadata, check links, and assess quality. Minor gaps exist, such as no search pagination or browser-level automation, but agents can generally complete tasks without dead ends.
Maintenance
Related MCP Connectors
Web search, browser automation, scraping, crawling and CAPTCHA solving for AI agents.
Live web access for agents: scrape, SERP search, crawl/map, 100+ collectors, datasets, proxies.
Reliable web access for AI agents: smart HTTP, rotating proxies, and full-browser rendering.
Web data for agents: YouTube transcripts, screenshots, Google News, WHOIS, jobs, tech stack, more.
Related MCP Servers
- AlicenseBqualityDmaintenanceEnables LLMs and AI agents to access real-time web data, search websites, and navigate the web without getting blocked. Includes 5,000 free monthly requests and supports web scraping, browser automation, and bypassing geo-restrictions.6014,541 npm1MIT
- AlicenseNot gradedqualityDmaintenanceProvides AI agents and assistants with advanced web crawling and RAG capabilities, enabling them to scrape websites and perform semantic search over crawled content.1MIT
- AlicenseNot gradedqualityDmaintenanceProvides AI agents with reliable web fetching capabilities, handling retries, caching, and anti-bot bypass automatically.MIT
- FlicenseNot gradedqualityDmaintenanceEnables AI agents to perform web searches, extract webpage content, and conduct end-to-end search-and-extract operations using multiple search providers and content extraction methods.-