PyreCrawl
This server provides comprehensive web scraping, crawling, search, and document extraction capabilities via MCP tools, with smart fallback strategies and browser automation.
Scrape single URLs into LLM-ready markdown with configurable strategies (fast HTTP, stealth browser with Cloudflare bypass, or full LLM-based extraction).
Extract structured data from web pages using CSS selector schemas (JsonCssExtractionStrategy).
Map all internal URLs of a website with optional include/exclude patterns.
Crawl multi-page sites with BFS/DFS, max depth, page limits, and URL filtering.
Convert PDF/DOCX/PPTX documents to markdown without a browser (requires optional docs extras).
Perform web searches via DuckDuckGo (no API key) and academic paper searches via arXiv/Crossref.
Batch scrape many URLs in parallel with concurrency control, deduplication, and per-URL failure isolation.
Run deep research: search the web, then fetch and return content from top sources as evidence (no LLM synthesis).
Monitor URLs over time for meaningful content changes, with CSS selector scoping and persistent snapshots.
Drive persistent browser sessions (cookies, JS state) for login walls and multi-step flows, supporting actions like navigate, click, type, eval, wait, screenshot, and cookie management.
Use the smart ladder (auto) to automatically escalate from cheap HTTP to stealth/LLM as needed for blocked or JS-heavy pages.
Provides academic paper search via arXiv (and Crossref), returning metadata and PDF URLs that can be fed into the document tool for conversion to markdown.
Enables scraping of Cloudflare-protected and Turnstile-challenged pages using a stealth browser.
Provides web search through DuckDuckGo's HTML interface without an API key, returning search results that can be used for further research and scraping.
Supports local LLM backends such as Ollama for deep processing and extraction without relying on hosted AI services.
๐ฅ PyreCrawl โ Web Browsing Superpowers for Your AI Agent
One command gives any AI agent the whole web. Scrape, extract, crawl, map, and search โ self-hosted, no API keys, no rate limits, no subscription.
PyreCrawl speaks MCP (Model Context Protocol), the standard tool interface for Claude, Cursor, VS Code, Codex, OpenCode, Hermes, and any MCP-compatible agent.
A smart auto-fallback ladder always picks the cheapest method that succeeds:
fast HTTP
โ (403/503/Cloudflare challenge or empty body)
โผ
stealth browser (real Chromium + Cloudflare solver)
โ (still blocked, or the page needs full JS rendering)
โผ
deep processing (LLM-ready markdown, citations, structured extraction)โก Tools exposed
Tool | What it does |
| Single URL โ LLM-ready markdown |
| Scrape + structured extraction (JsonCss schema) |
| Enumerate all internal URLs |
| Multi-page crawl with path filters + true BFS depth |
| PDF/DOCX/PPTX โ markdown (no browser, optional |
| Web search via DuckDuckGo HTML (no API key) |
| Academic search via arXiv + Crossref (no API key) โ feed |
| Many URLs in ONE call โ parallel, deduped, cache-aware |
| Search โ evidence pack with [n] citations (no LLM synthesis โ your agent does that) |
| Change detection with persisted snapshots + unified diff |
| Persistent browser session (cookies kept) โ login walls, multi-step flows, screenshots |
| Inspect/clear/enable/disable the HTTP response cache |
| Versions + import sanity check |
MCP Resources (read-only state without a tool call):
pyrecrawl://cache/stats ยท pyrecrawl://sessions ยท pyrecrawl://monitors
MCP Prompts (ready-made playbooks): research(topic) ยท rag_ingest(site) ยท watch_page(url)
Env flags
Variable | Default | Effect |
| off |
|
|
| Cache entry lifetime in seconds |
|
| Where monitor snapshots persist |
prefer options: "auto" (default ladder) ยท "fast" (HTTP only) ยท "stealth" (CF bypass) ยท "llm" (deep processing).
Related MCP server: mcp-services
๐ Install & Use (one-liner)
1. Install
UV (recommended โ one command, zero Python setup)
UV is a fast Python package manager that handles Python itself โ no need to install Python separately. Get it once:
# macOS / Linux
curl -LsSf https://astral.sh/uv/install.sh | sh
# Windows (PowerShell)
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"Then run PyreCrawl directly โ no venv, no pip install, no Python download:
uvx pyrecrawl@latestOr via uv tool install (persistent, recommended for regular use)
uv tool install pyrecrawlOr via pipx (alternative)
pipx install pyrecrawlOr via pip into a venv
pip install pyrecrawl2. One-time browser engines
pyrecrawl setupThis installs Chromium + stealth browser engines (~2 min, one-time).
3. Register with your AI agent
# Auto-detect installed agents and write their MCP configs
pyrecrawl install
# Or target specific agents
pyrecrawl install claude-desktop cursor
# Dry-run to preview what would change
pyrecrawl install --dry-runSupported agents: claude-desktop, claude-code, cursor, vscode, codex, opencode, hermes.
4. Start chatting
After installing + registering, restart your agent (or start a new session). Then ask:
"Scrape https://example.com and summarize it."
The tools appear as mcp_pyrecrawl_scrape, mcp_pyrecrawl_extract, mcp_pyrecrawl_map_site, mcp_pyrecrawl_crawl, mcp_pyrecrawl_search, mcp_pyrecrawl_health.
๐ Manual config (if pyrecrawl install doesn't match your setup)
Claude Desktop
Config file
Linux:
~/.config/Claude/claude_desktop_config.jsonmacOS:
~/Library/Application Support/Claude/claude_desktop_config.jsonWindows:
%AppData%\Claude\claude_desktop_config.json
{
"mcpServers": {
"pyrecrawl": {
"command": "uvx",
"args": ["--from", "pyrecrawl", "pyrecrawl", "serve"]
}
}
}Claude Code
Config file: project-scoped .mcp.json
{
"mcpServers": {
"pyrecrawl": {
"command": "uvx",
"args": ["--from", "pyrecrawl", "pyrecrawl", "serve"]
}
}
}Cursor
Config file: ~/.cursor/mcp.json
{
"mcpServers": {
"pyrecrawl": {
"command": "uvx",
"args": ["--from", "pyrecrawl", "pyrecrawl", "serve"]
}
}
}VS Code / Copilot
Config file: .vscode/mcp.json (project-scoped)
{
"servers": {
"pyrecrawl": {
"command": "uvx",
"args": ["--from", "pyrecrawl", "pyrecrawl", "serve"],
"type": "stdio"
}
}
}Codex CLI
Config file: ~/.codex/config.toml
[mcp_servers.pyrecrawl]
command = "uvx"
args = ["--from", "pyrecrawl", "pyrecrawl", "serve"]OpenCode
Config file: ~/.config/opencode/opencode.json
{
"mcp": {
"pyrecrawl": {
"type": "local",
"command": ["uvx", "--from", "pyrecrawl", "pyrecrawl", "serve"],
"enabled": true
}
}
}Hermes
Config file
Linux/macOS:
~/.hermes/config.yamlWindows:
%LocalAppData%\hermes\config.yaml
mcp_servers:
pyrecrawl:
command: uvx
args:
- --from
- pyrecrawl
- pyrecrawl
- serve
enabled: trueWindows note:
uvxmust be on PATH. If not, use the full path touvx.exe(e.g.C:\Users\<you>\AppData\Local\hermes\bin\uvx.exe).
๐ง How the ladder chooses
PyreCrawl runs each request through three tiers, stopping at the first one that returns a complete, LLM-ready result:
Concern | Fast tier | Stealth tier | Deep tier |
Static HTML page | โ ~200ms | โ | โ |
Cloudflare-protected | โ | โ Turnstile solver | โ |
JS-heavy SPA | โ | โ real Chromium | โ |
Live DOM data (input | โ | โ
| โ |
LLM-ready markdown + citations | โ | โ | โ BM25, fit-markdown |
Structured extraction (CSS schema) | โ | โ | โ |
Deep crawl (BFS/DFS/BestFirst) | โ | โ | โ adaptive |
The agent never has to pick. prefer="auto" does it every call.
Live DOM data with js and wait_for
Some sites keep the data you want in a DOM property (e.g. an <input>'s .value)
that JS writes after an XHR โ it never appears in the serialized HTML. The
scrape tool accepts two stealth-tier params for exactly this:
{
"url": "https://temp-mail.org/id",
"prefer": "stealth",
"wait_for": "document.getElementById('mail').value.includes('@')",
"js": "document.getElementById('mail').value"
}wait_forโ a JS predicate expression polled until truthy (bounded bytimeout). Use it instead of guessing a sleep for anything that arrives asynchronously.jsโ a JS expression evaluated once the page settles; the value comes back inmeta.js_result. Errors are captured inmeta.js_error(the page result is still returned, never a crash).
๐ Compared to Firecrawl (hosted)
Firecrawl | PyreCrawl | |
Cost | Free 1k/mo, then $16โ333/mo | Free, self-hosted |
Local LLM support | โ | โ Ollama / any LLM |
Cloudflare bypass | โ (Fire-Engine, paid) | โ (free, built-in) |
Markdown + BM25 | โ | โ |
Self-host | โ | โ |
Academic paper search | โ | โ
arXiv + Crossref ( |
Hosted search API | โ /search | โ ๏ธ DuckDuckGo HTML + arXiv/Crossref (no key) |
๐ง Development
git clone https://github.com/SanggonBoy/PyreCrawl.git
cd PyreCrawl
uv venv --python 3.12 .venv
source .venv/Scripts/activate # Windows; or .venv/bin/activate on macOS/Linux
uv pip install -e ".[dev]"
python -m playwright install chromium
scrapling installRun tests
python scripts/selfcheck.py # real-network smoke test
python scripts/probe_stdio.py # stdio JSON-RPC probe๐ฆ Publish
Maintainers only:
git tag v0.8.0
git push origin v0.8.0GitHub Actions builds + uploads to PyPI via trusted publishing.
๐ Stay up to date
PyreCrawl checks PyPI on every startup and reports the latest version โ your
MCP agent sees this automatically via the health() tool response and can
notify you inline.
To check manually:
pyrecrawl versionTo upgrade:
pyrecrawl update # runs: uv tool upgrade pyrecrawlGet notified of new releases: click Watch โ Releases only at the GitHub repo to receive email notifications when a new version is published.
๐ Uninstall
# Remove from all agent configs
pyrecrawl uninstall
# Remove the package
uv tool uninstall pyrecrawl๐ก๏ธ License
MIT โ see LICENSE.
๐ Credits
Built on the shoulders of Scrapling and Crawl4AI โ both MIT, both excellent.
Available Tools
13 toolsbatch_scrapeA
Scrape MANY URLs in ONE call (parallel, deduped, cache-aware).
Args: urls: Target URLs (deduped automatically; empties dropped). prefer: "auto" | "fast" | "stealth" | "llm". timeout: per-URL timeout in seconds. max_concurrency: parallel workers (default 4). include_html: include raw HTML per result (large; off by default).
Returns {requested, unique, succeeded, failed, results[]}. Per-URL failures are isolated โ other URLs still succeed.
| Name | Required | Description | Default |
|---|---|---|---|
| urls | Yes | ||
| prefer | No | auto | |
| timeout | No | ||
| include_html | No | ||
| max_concurrency | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description mentions parallel processing, deduplication, cache-awareness, and per-URL failure isolation, but lacks details on potential side effects (e.g., cache writes) and rate limits, especially with no annotations provided.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact and well-structured, conveying essential information without redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The return shape and per-URL error isolation are summarized, providing adequate context for a batch operation, though a full output schema is not specified.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Each parameter's purpose is described (URLs, preference mode, timeout, concurrency, HTML inclusion), though the `prefer` enum values are listed without explanation of their differences.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool scrapes multiple URLs in one call, explicitly distinguishing it from single-URL scraping via the sibling `scrape` tool.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for batch scenarios but does not explicitly contrast with alternatives like `scrape` or `crawl`, leaving selection to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
cacheC
Inspect the response cache: stats or clear.
Args: action: "stats" | "clear" | "disable" | "enable".
| Name | Required | Description | Default |
|---|---|---|---|
| action | No | stats |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden of behavioral disclosure. It names the actions but does not warn that 'clear' is destructive, what 'disable'/'enable' affect, or whether these actions persist across sessions. The word 'Inspect' also downplays the mutating actions.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is brief and front-loaded with the tool's purpose, followed by a clean parameter list. The only structural issue is the mismatch between 'stats or clear' and the four listed actions.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool is simple and has an output schema, so return-value details are not required. But a cache-control tool that includes a destructive action needs warnings and scope information, which are absent. The description does not fully prepare an agent to invoke the mutating actions safely.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema provides only a loosely typed optional string with no enum or description, so the description's explicit list of allowed values ('stats' | 'clear' | 'disable' | 'enable') is genuinely useful. However, it does not explain the meaning or side effects of each action value.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a clear verb ('Inspect') and resource ('response cache') and lists the actions it supports, which separates it from the unrelated sibling tools. However, the first sentence says 'stats or clear' but the Args section includes 'disable' and 'enable', creating a small scope inconsistency.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no guidance about when to use this tool, when not to use it, or what problem it solves beyond the obvious. The action names imply possible uses, but the description never explains scenarios like stale cache, troubleshooting, or re-enabling the cache.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
crawlA
Multi-page crawl: discover URLs on root, then scrape each.
Args:
root: start URL.
max_pages: hard cap on pages scraped.
css_selector: reserved for the llm tier.
prefer: "auto" | "fast" | "stealth" | "llm" (llm = Crawl4AI BFS deep-crawl).
include_paths: regex โ keep only URLs matching (matched against full URL).
exclude_paths: regex โ drop URLs matching (e.g. /tag/|/page/\d+).
max_depth: 0 = flat harvest from the root page's links (default);
>0 = true BFS up to that link depth, honoring the filters.
| Name | Required | Description | Default |
|---|---|---|---|
| root | Yes | ||
| prefer | No | auto | |
| max_depth | No | ||
| max_pages | No | ||
| css_selector | No | ||
| exclude_paths | No | ||
| include_paths | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description discloses key behaviors: URL discovery then scraping, a hard cap via max_pages, BFS depth semantics, regex filtering, and the llm-tier meaning of prefer. It does not discuss rate limits, auth, domain scoping, or side effects, but what it states is concrete and accurate.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The summary line plus a compact argument list is appropriately sized for a 7-parameter tool. Every line adds value, default behavior is noted for max_depth, and the most important distinction (llm deep-crawl) is inline rather than in prose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description is complete enough to call the tool with all 7 parameters meaningfully configured, and an output schema exists so return values need not be described. It would be stronger with a sentence on URL scope (e.g., same-domain only) and on what happens when include_paths and exclude_paths overlap.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, but the description documents every parameter with types/semantics: root, max_pages, css_selector, prefer values, include/exclude regex behavior, and max_depth default semantics. It even gives an exclude_paths regex example, fully compensating for the bare schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The opening line 'Multi-page crawl: discover URLs on root, then scrape each' names a concrete verb and resource and describes the two-phase behavior. The 'multi-page' framing separates it from siblings like scrape, which presumably handles a single page.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies the use case (multi-page crawling, BFS vs flat harvest through max_depth) but never explicitly says when to choose crawl over scrape, map_site, or batch_scrape. It also gives no exclusion guidance such as 'for single-page use scrape instead.'
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
deep_researchA
Search the web, then pull the top sources as EVIDENCE (no LLM synthesis).
Returns a citations list with stable [n] numbers and an
evidence list of per-source markdown โ the agent does the
synthesis. Designed for research, RAG prep, and fact-checking.
Args: query: search string. limit: how many search results to fetch. scrape_top: how many of those to actually fetch content from. prefer: "auto" | "fast" | "stealth" | "llm".
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | ||
| query | Yes | ||
| prefer | No | auto | |
| scrape_top | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and discloses meaningful behavioral details: it returns a citations list with stable [n] numbers and an evidence list of per-source markdown, and it performs no LLM synthesis. It does not explain tradeoffs of 'prefer' modes or rate/stealth behavior, but the core behavior is clearly communicated.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact and well-structured: it leads with the core behavior, then the return format, then use cases, and finally parameters. Every sentence serves a purpose and the format is scannable for an AI agent.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers purpose, output shape, and parameter basics, which is solid, but it leaves gaps: the meaning of the 'prefer' options is opaque, and there is no guidance on rate limits, cost, or when to avoid this tool in favor of a lighter sibling. These gaps matter for correct invocation in an agent context.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, and the description compensates with an Args block explaining query, limit, scrape_top, and prefer. Each parameter gets some semantic context beyond the schema, though 'prefer' only lists options without explaining what they mean in practice.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific behavior: 'Search the web, then pull the top sources as EVIDENCE'. It distinguishes itself from a plain web search by emphasizing the evidence/citations output and the absence of LLM synthesis, which sets it apart from sibling search tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description clearly states when to use it: 'Designed for research, RAG prep, and fact-checking.' It also signals that this tool returns raw evidence for the agent to synthesize, implying the right use case. However, it does not explicitly contrast with alternatives like 'search' or 'search_papers'.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
documentA
Extract text from a PDF/DOCX/PPTX URL โ markdown (no browser).
Content-type sniffed and routed to pypdf / python-docx / python-pptx.
Optional deps โ install with pip install 'pyrecrawl[docs]'.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| timeout | No | ||
| max_pages | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and does so well: it discloses that no browser is used, that content-type is sniffed and routed to specific parsers, and that optional dependencies must be installed. These are genuine behavioral details beyond what the schema shows.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three short sentences, each carrying useful information, with the core operation front-loaded. No filler or repetition of schema defaults.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description is complete enough for the core operation and the output schema exists, so return values are covered. Minor gaps remain around timeout/max_pages semantics and explicit sibling routing, but they do not prevent an agent from calling the tool correctly in the main case.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate; it only clarifies the URL parameter by specifying supported document formats and content-type routing. timeout and max_pages are left with only their names and defaults, leaving units and page-limiting semantics for specific formats undefined.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The first sentence names a concrete verb ('Extract'), a precise resource ('PDF/DOCX/PPTX URL'), and a specific output (markdown). The 'no browser' qualifier and document-only formats distinguish it from siblings like scrape and extract.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description clearly implies the intended case: document URLs that should be converted without a browser. It does not explicitly name alternatives or state when not to use it, but the format constraint gives an agent enough context to route correctly.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
extractA
Scrape + structured extraction using a CSS-based JSON schema.
The schema is a JsonCssExtractionStrategy schema: { "name": "PageItems", "baseSelector": "div.item", "fields": [{"name": "title", "selector": "h2", "type": "text"}, ...] }
Returns parsed JSON in data.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| prefer | No | auto | |
| schema | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description discloses the core behavioral contract: it performs scraping, applies a JsonCssExtractionStrategy schema, and returns parsed JSON in data. With no annotations, this is useful but incomplete; it does not cover failure behavior, execution time, or permissions/rate limits.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact and every element earns its place: a one-line summary, a clear schema example, and an output note. Front-loading the core purpose before the example makes it scannable.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with a complex schema parameter and no annotations, the description covers the main workflow and output, and the output schema covers return values. It is still incomplete because the optional prefer parameter is undocumented and there is no guidance on edge cases or interaction with siblings.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The description adds critical meaning to the schema parameter by showing the exact JsonCssExtractionStrategy structure with an example, which the bare schema (additionalProperties: true) does not provide. It does not explain the prefer parameter or the url parameter further, and schema coverage is 0%, so the compensation is only partial.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a specific function: scrape a page and extract structured data using a CSS-based JSON schema. It clearly explains what kind of schema is expected, which makes the generic tool name meaningful. It does not explicitly contrast with siblings like scrape or crawl, so it misses full differentiation.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is implied: use this when you need structured extraction from a single page rather than raw scraping or multi-page crawling. There is no explicit when-to-use/when-not-to-use guidance or mention of sibling alternatives, but the 'structured extraction' phrasing does enough to suggest the primary use case.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
healthA
Sanity check: verify engines are importable + return versions.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
There are no annotations, so the description carries the full burden. It discloses meaningful behavior: the tool imports engines and returns version information. The term 'sanity check' also implies a non-destructive read-only operation, though it does not explicitly state that no state is modified. For a simple health check this is sufficient context.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single compact sentence that front-loads the core purpose ('Sanity check') and immediately follows with the precise behavior and output. Every word earns its place with no redundancy or filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a zero-parameter diagnostic tool with an output schema present, the description is complete. It tells the agent what the tool does, what it will return, and implies its role as a health check. Nothing essential is missing for correct invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters, so the baseline is 4. The description adds no parameter-specific meaning, but none is needed since the schema already fully covers the empty parameter set at 100% coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb 'verify' plus a concrete resource ('engines are importable') and states the output ('return versions'). This clearly identifies it as a health-check/diagnostic operation, which is distinct from the data-collection focused sibling tools like scrape, extract, and crawl.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies this is a sanity check to run for verifying setup, but it does not explicitly state when to use it versus alternatives or when not to use it. Given the tool has zero parameters and a single obvious purpose, the implied usage is adequate but not fully explicit.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
map_siteA
Enumerate all internal URLs reachable from root.
Args: root: Website root (e.g. "https://example.com/docs"). include_pattern: Optional regex; only URLs matching are returned. limit: Hard cap on returned URLs.
| Name | Required | Description | Default |
|---|---|---|---|
| root | Yes | ||
| limit | No | ||
| include_pattern | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must carry the behavioral burden. It discloses that limit is a hard cap and that include_pattern filters returned URLs, but it does not mention side effects, cost, redirect handling, or whether the enumeration is read-only (only implied by 'enumerate'). This is adequate but incomplete for a no-annotation tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a compact docstring with a one-line summary followed by a structured Args list. Every sentence earns its place, and there is no redundant information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool is simple, an output schema exists, and all parameters are clearly documented. Minor gaps remain, such as what qualifies as 'internal' and whether redirects are followed, but these are not critical given the output schema and straightforward purpose.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, but the description fully compensates by explaining each parameter: root with an example, include_pattern as an optional regex with matching behavior, and limit as a hard cap. This adds meaningful semantics beyond the bare schema titles.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb ('Enumerate') and a specific resource ('all internal URLs reachable from root'), which makes the tool's function immediately clear and distinct from content-oriented siblings like scrape or extract. Even without naming sibling tools, the 'enumerate URLs' resource is specific enough to avoid confusion.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The intended use case is apparent from the action: an agent should call this when it needs a list of internal URLs reachable from a given root. However, it does not explicitly mention when not to use it or compare it to the similar sibling 'crawl', so it lacks exclusions or alternative routing.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
monitorA
Track a URL over time and report meaningful content changes.
Args:
url: target URL.
action: "check" | "history" | "forget".
prefer: ladder preference, same as scrape.
css_selector: scope the diff to one element (so banner /
nav changes don't trigger false positives).
Snapshots persist under PYRECRAWL_MONITOR_DIR (default
~/.pyrecrawl/monitors/). check returns status of
new | unchanged | changed | error and a unified
diff when the page changed.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| action | No | check | |
| prefer | No | auto | |
| css_selector | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and largely succeeds. It discloses persistence behavior (snapshots stored under PYRECRAWL_MONITOR_DIR), the exact status vocabulary (new/unchanged/changed/error), and the unified-diff output on change. A minor gap is that 'forget' is not described as destructive/deleting snapshots, and 'history' output details are missing.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact and front-loaded with a one-sentence purpose statement followed by a tight parameter list and key behavioral facts. There is no filler; each sentence contributes operational value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a four-parameter tool with an output schema, this is close to complete. It covers what the tool does, how snapshots persist, what actions exist, and what a check returns. The only meaningful omissions are precise behavior of 'history' and 'forget,' and possible error conditions, but these are not critical for correct invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must explain the parameters, and it does. url is self-evident, action lists its enum values, prefer is cross-referenced to scrape, and css_selector gets a practical purpose. The 'prefer' explanation is terse but useful given the sibling tool. This goes well beyond the bare schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The opening line 'Track a URL over time and report meaningful content changes' states a specific verb, resource, and temporal scope. It clearly distinguishes monitor from siblings like scrape (single fetch) and crawl (site traversal) by emphasizing change tracking over time. The action list (check/history/forget) further clarifies the tool's role.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear context: this is for monitoring a URL over time and detecting changes. It explains when each action is appropriate (check for current diff, history for past snapshots, forget to stop monitoring), and even gives guidance on using css_selector to avoid false positives. It does not explicitly name alternatives or state 'use this instead of scrape,' but the purpose is unambiguous enough.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
scrapeA
Scrape a single URL โ LLM-ready markdown.
Args:
url: Target URL (http/https/file/raw:).
prefer: "auto" | "fast" | "stealth" | "llm".
auto = fast first, escalate to stealth on block/short page.
fast = cheap HTTP only (no JS).
stealth = real Chromium + Cloudflare solver.
llm = full Crawl4AI browser + BM25 fit-markdown.
timeout: per-attempt timeout in seconds.
include_html: include raw HTML in the response (large; off by default).
js: (stealth only) JS expression evaluated against the live page
after it settles. The value comes back in meta.js_result.
Use for data that lives in DOM properties (e.g. an input's
.value) rather than in serialized HTML.
wait_for: (stealth only) JS predicate expression polled until truthy
(bounded by timeout). Use to wait for content that arrives
asynchronously after network_idle.
| Name | Required | Description | Default |
|---|---|---|---|
| js | No | ||
| url | Yes | ||
| prefer | No | auto | |
| timeout | No | ||
| wait_for | No | ||
| include_html | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full disclosure burden and meets it: it reveals multi-attempt behavior (fast first, escalate to stealth), per-attempt timeout semantics, stealth-only constraints for js and wait_for, and where js results appear (meta.js_result). It also flags include_html as large and off by default, which is exactly the kind of behavioral context an agent needs.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The one-line purpose is front-loaded, followed by a compact, well-labeled Args list. Every sentence adds operational value; the longer js and wait_for explanations earn their length by specifying exactly when and how to use those parameters.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 6-parameter scraping tool with no annotations but an output schema, the description covers tool purpose, all parameters, mode behavior, and parameter-specific use cases. Nothing essential for a correct call is missing; the output schema can handle return-value details.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, but the description documents all six parameters with substantive meaning beyond names and types: URL schemes, the four prefer values and their trade-offs, timeout per attempt, include_html's response-size impact, and the precise use cases for js and wait_for. This fully compensates for the empty schema descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The opening line, 'Scrape a single URL โ LLM-ready markdown,' states the exact verb, resource, and output format. The 'single URL' qualifier and mention of markdown differentiate it from siblings like batch_scrape and crawl, even without naming them.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description makes the intended context clear by emphasizing a single URL and explains when to choose each prefer strategy (auto escalates to stealth on block/short page, fast is HTTP-only, stealth handles Cloudflare, llm applies BM25 fit-markdown). It also gives explicit guidance for js (when data lives in DOM properties) and wait_for (async content after network_idle). It stops short of explicitly naming alternatives or when-not-to-use conditions versus sibling tools, so it is not a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
searchA
Web search via DuckDuckGo HTML (no API key required).
Returns [{url, title, snippet}, ...]. The smart ladder bypasses DDG's bot detection if needed.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | ||
| query | Yes | ||
| prefer | No | auto |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the burden of behavioral disclosure. It goes beyond a simple action by explaining the return shape, the underlying DuckDuckGo HTML source, and the 'smart ladder' workaround for bot detection. It could mention rate limits or failure modes, but it provides meaningful behavioral context.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences and front-loads the core purpose. Every sentence adds value: the first defines what the tool does, and the second describes output format and a key behavioral detail. There is no redundant or filler content.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description is adequate for making a basic call with just the required 'query' parameter, and the output schema supplements return-value understanding. However, it lacks usage routing and meaningful parameter documentation, especially for 'prefer', leaving some gaps for an agent deciding how to invoke the tool correctly in varied scenarios.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate for undocumented parameters. It does not explain 'prefer' at all, and 'limit' is only inferable from its name and default value. The query parameter is implied by 'Web search', but the description adds no parameter-level semantics beyond what the schema already exposes.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states a specific action and resource: 'Web search via DuckDuckGo HTML'. This distinguishes it from siblings like search_papers and deep_research, and the mention of 'no API key required' adds a concrete implementation detail that clarifies the tool's identity.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no explicit guidance on when to use this tool versus alternatives such as deep_research, search_papers, or scrape. 'No API key required' hints at a use case, but the description does not state when a user should choose this search tool over a sibling or when it should be avoided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
search_papersA
Search academic papers via arXiv or Crossref โ no API keys.
Args: query: free-text search, e.g. "transformer attention scaling laws". limit: max results (1-25 arXiv / 1-20 crossref). source: "arxiv" (CS/physics/math preprints, default) or "crossref" (all fields, DOI-backed). category: optional arXiv category filter, e.g. "cs.LG", "cs.CV".
Returns papers with id/url/pdf_url/title/authors/summary/published.
Feed pdf_url into the document tool to extract full text.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | ||
| query | Yes | ||
| source | No | arxiv | |
| category | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the behavioral disclosure burden. It discloses that no API keys are needed, defines result limits per source, and explains the return payload fields. It also tells the agent to feed pdf_url into the document tool, which is useful downstream behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is tightly structured with a one-sentence summary, a compact Args list, and a short Returns note. Every sentence adds useful information, and the most important purpose and access facts appear first.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has only four parameters and an output schema, the description fully covers how to call it: query is required, defaults are provided, source differences are explained, and the return fields are listed. The routing note to the document tool adds helpful cross-tool context without unnecessary detail.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate for all parameters. It does: query gets a free-text example, limit gets source-specific bounds, source gets its two possible values with meanings and default, and category gets an optional arXiv filter example. This is strong compensation for the schema gap.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description opens with a specific verb and resource: 'Search academic papers via arXiv or Crossref'. It clearly distinguishes from the generic sibling tool 'search' by explicitly scoping to academic papers and naming the two supported sources.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description clearly states the tool is for academic paper search and gives guidance on choosing between 'arxiv' and 'crossref'. It does not explicitly mention when not to use this tool versus other search-related siblings, but the academic scope is clear enough for an agent to route appropriately.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
sessionA
Drive a persistent browser session โ cookies & JS state kept across calls.
Use for login walls and multi-step flows the ladder can't handle (one-shot scrape has no session memory; here each action runs against the same live page).
Args: session: named session; reuse the same name to keep state. action: "open" (url) โ navigate, returns url/title/status "click" (selector) โ click an element "fill" (selector, text) โ type into an input "type" (key) โ press a key, e.g. "Enter" "eval" (js) โ run a JS expression, returns value "wait" (selector?) โ wait for selector or sleep timeout_ms "content" โ url/title/visible text of current page "screenshot" (full_page?) โ returns png_base64 "cookies" โ list session cookies "close" โ destroy the session "list" โ show live sessions Returns {session, action, ...result, elapsed_ms}; errors as {error}.
| Name | Required | Description | Default |
|---|---|---|---|
| js | No | ||
| key | No | ||
| url | No | ||
| text | No | ||
| action | No | open | |
| session | No | default | |
| headless | No | ||
| selector | No | ||
| full_page | No | ||
| timeout_ms | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden, and it does well by disclosing persistent state, the side effect of 'close' destroying the session, the return shape, and error format. It doesn't cover all behavioral nuances such as headless mode or page-load wait behavior, but the core operational traits are transparent.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is efficiently structured with a purpose line, a usage-reason line, and a clean action list. Every sentence and list item earns its place, and the action enumeration is necessary because the schema provides no enum values.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's high complexity โ 10 parameters and 11 actions โ the description covers nearly everything an agent needs: session semantics, action expectations, return format, and error behavior. The only notable omission is the headless parameter and more detail on waiting/timeout behavior outside the 'wait' action, but the overall picture is solid.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate; it does by documenting the session parameter and mapping each action string to its relevant arguments, including 'open (url)', 'fill (selector, text)', 'eval (js)', and 'wait (selector? / timeout_ms)'. The main gap is the headless parameter, which is never described, though its schema default partially mitigates that.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: 'Drive a persistent browser session' with cookies and JS state kept across calls. It also distinguishes itself from one-shot scraping tools by explaining the session-memory difference, so an agent can tell it apart from the sibling scrape tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explicitly states when to use the tool: 'Use for login walls and multi-step flows the ladder can't handle.' It also clarifies the alternative โ one-shot scrape has no session memory โ and explains that each action runs against the same live page, giving clear selection guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
13 tool updates
v0.7.1- First observed
batch_scrape - First observed
cache - First observed
crawl - First observed
deep_research - First observed
document - First observed
extract - First observed
health - First observed
map_site - First observed
monitor - First observed
scrape - First observed
search - First observed
search_papers - First observed
session
TDQS
Scored across 13 tools
Each tool has a distinct focus: single scrape, batch scrape, site crawl, structured extraction, monitoring, session control, and search variants are clearly separated. Descriptions explicitly note differences (e.g., extract vs scrape, search vs deep_research), leaving no interchangeable tools.
Names are mostly simple lowercase verbs/nouns, and multi-word names consistently use snake_case (map_site, batch_scrape, search_papers). Minor deviations like 'document', 'health', and 'session' are nouns rather than verbs, but the pattern remains predictable and readable.
Thirteen tools form a well-scoped set for a scraping and research server, with each tool covering a distinct capability (scrape, crawl, extract, monitor, search, session, etc.). The count is substantial without feeling bloated or redundant.
The toolset covers the full pipeline: single and multi-page scraping, structured extraction, site mapping, document parsing, general and academic search, deep research, change monitoring, and persistent browser sessions. Tools compose well (e.g., search_papers feeds pdf_url into document), with no obvious dead ends or missing core operations.
Maintenance
Related MCP Connectors
65+ AI tools as MCP: research, write, code, scrape, translate, RAG, agent memory, workflows
Scrape, crawl and search the web for AI agents via MCP.
- mcpOAuthcom.screenshotink
Screenshot, diff, audit and sitemap-capture any web page โ 5 MCP tools for AI agents.
Real-time Amazon, WIPO & PACER data for AI agents โ 19 tools via the MCP protocol.
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceWeb scraping MCP server for Al agents. 6 tools: extract clean text/markdown from any URL, structured scraping with CSS selectors, full-page screenshots via Playwright, link extraction with regex filtering, metadata extraction (OG tags, Twitter cards), and Google search. Free tier: 50 requests/IP/day.8MIT
- AlicenseNot gradedqualityDmaintenanceMulti-tool MCP server for AI agents with 29 tools across web scraping, SEO analysis, screenshot and PDF generation, domain intelligence, content extraction, multi-chain EVM blockchain queries, and security toolkit. Free tier available with no auth required.121MIT
- AlicenseAqualityAmaintenanceWeb content extraction for AI agents. 10 tools: scrape, crawl, map, batch, extract, summarize, diff, brand, search, research. Uses TLS fingerprinting to bypass anti-bot without a headless browser. Outputs LLM-optimized markdown with 67% fewer tokens than raw HTML.102,341AGPL 3.0
- AlicenseAqualityAmaintenanceWeb scraping, crawling, and structured data extraction for AI agents. 5 tools: scrape (clean markdown from any URL), crawl (entire sites), map (discover URLs), extract (structured JSON), and search. 833ms avg latency, single binary, self-hostable.8970AGPL 3.0