PyreCrawl
PyreCrawl is an MCP server that gives AI agents web browsing, scraping, search, and research capabilities in one self-hosted tool.
Scrape single URLs into LLM-ready markdown, with auto-escalation from fast HTTP to stealth Chromium to deep processing
Extract structured data from pages using CSS-based JSON schemas
Crawl websites: map internal links, multi-page BFS crawl with path filters, depth limits, and scoped selectors
Batch-scrape many URLs in parallel with deduplication and caching
Search the web via DuckDuckGo without API keys
Search academic papers on arXiv and Crossref, with PDF URLs that can be fed into the document tool
Run deep research: search, scrape top sources, and return evidence with numbered citations (no LLM synthesis)
Extract text from PDF, DOCX, and PPTX URLs
Monitor pages for content changes with persistent snapshots and unified diffs
Drive persistent browser sessions for login walls and multi-step flows, including clicks, fills, key presses, JS evaluation, screenshots, and cookie access
Inspect or clear the HTTP response cache
Check server health and engine availability
Provides academic paper search via arXiv (and Crossref), returning metadata and PDF URLs that can be fed into the document tool for conversion to markdown.
Enables scraping of Cloudflare-protected and Turnstile-challenged pages using a stealth browser.
Provides web search through DuckDuckGo's HTML interface without an API key, returning search results that can be used for further research and scraping.
Supports local LLM backends such as Ollama for deep processing and extraction without relying on hosted AI services.
๐ฅ PyreCrawl โ Web Browsing Superpowers for Your AI Agent
One command gives any AI agent the whole web. Scrape, extract, crawl, map, and search โ self-hosted, no API keys, no rate limits, no subscription.
PyreCrawl speaks MCP (Model Context Protocol), the standard tool interface for Claude, Cursor, VS Code, Codex, OpenCode, Hermes, and any MCP-compatible agent.
A smart auto-fallback ladder always picks the cheapest method that succeeds:
fast HTTP
โ (403/503/Cloudflare challenge or empty body)
โผ
stealth browser (real Chromium + Cloudflare solver)
โ (still blocked, or the page needs full JS rendering)
โผ
deep processing (LLM-ready markdown, citations, structured extraction)โก Tools exposed
Tool | What it does |
| Single URL โ LLM-ready markdown |
| Scrape + structured extraction (JsonCss schema) |
| Enumerate all internal URLs |
| Multi-page crawl with path filters + true BFS depth |
| PDF/DOCX/PPTX โ markdown (no browser, optional |
| Web search via DuckDuckGo HTML (no API key) |
| Academic search via arXiv + Crossref (no API key) โ feed |
| Many URLs in ONE call โ parallel, deduped, cache-aware |
| Search โ evidence pack with [n] citations (no LLM synthesis โ your agent does that) |
| Change detection with persisted snapshots + unified diff |
| Persistent browser session (cookies kept) โ login walls, multi-step flows, screenshots |
| Inspect/clear/enable/disable the HTTP response cache |
| Versions + import sanity check |
MCP Resources (read-only state without a tool call):
pyrecrawl://cache/stats ยท pyrecrawl://sessions ยท pyrecrawl://monitors
MCP Prompts (ready-made playbooks): research(topic) ยท rag_ingest(site) ยท watch_page(url)
Env flags
Variable | Default | Effect |
| off |
|
|
| Cache entry lifetime in seconds |
|
| Where monitor snapshots persist |
| off |
|
prefer options: "auto" (default ladder) ยท "fast" (HTTP only) ยท "stealth" (CF bypass) ยท "llm" (deep processing).
Related MCP server: mcp-services
๐ Install & Use (one-liner)
1. Install
UV (recommended โ one command, zero Python setup)
UV is a fast Python package manager that handles Python itself โ no need to install Python separately. Get it once:
# macOS / Linux
curl -LsSf https://astral.sh/uv/install.sh | sh
# Windows (PowerShell)
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"Then run PyreCrawl directly โ no venv, no pip install, no Python download:
uvx pyrecrawl@latestOr via uv tool install (persistent, recommended for regular use)
uv tool install pyrecrawlOr via pipx (alternative)
pipx install pyrecrawlOr via pip into a venv
pip install pyrecrawl2. One-time browser engines
pyrecrawl setupThis installs Chromium + stealth browser engines (~2 min, one-time).
3. Register with your AI agent
# Auto-detect installed agents and write their MCP configs
pyrecrawl install
# Or target specific agents
pyrecrawl install claude-desktop cursor
# Dry-run to preview what would change
pyrecrawl install --dry-runSupported agents: claude-desktop, claude-code, cursor, vscode, codex, opencode, hermes.
4. Start chatting
After installing + registering, restart your agent (or start a new session). Then ask:
"Scrape https://example.com and summarize it."
Available tools:
Tool | What it does |
| Fetch a single URL โ markdown (auto-escalates past Cloudflare) |
| Scrape + structured extraction via CSS schema โ JSON |
| Enumerate all internal URLs from a root |
| Multi-page crawl: discover + scrape in bulk |
| Fetch many URLs in one parallel call |
| Web search via DuckDuckGo with anti-bot bypass |
| Academic paper search (arXiv / Crossref) |
| Search + scrape + citations in one call โ primary research tool |
| Extract text from PDF/DOCX/PPTX URLs |
| Track a URL for content changes over time |
| Persistent browser session for login walls |
| Inspect or clear the response cache |
| Verify engine availability + version |
Plus 3 guided prompts: research, rag_ingest, watch_page.
Quick examples
Ask your agent naturally โ no special syntax needed:
You say | Agent uses |
"Scrape https://example.com and summarize it" |
|
"Research Rust memory safety vulnerabilities" |
|
"Deep research on AI regulation worldwide" |
|
"Extract all product names and prices from this page" |
|
"Crawl https://docs.example.com and give me an overview" |
|
"Monitor this page for price changes" |
|
"Find papers about transformer attention" |
|
"What's the current cache hit rate?" |
|
๐ง Skills โ Maximize Your Agent's Research Quality
PyreCrawl tools give your agent hands (scrape, crawl, search). But the agent still needs a brain โ instructions on when to use which tool, how to chain research passes, and what anti-hallucination rules to follow.
That's what PyreCrawl Skills provides.
MCP Tools (this repo) | Skills (pyrecrawl-skills) | |
Role | Execute web operations | Tell the agent how to use them |
Analogy | Hands | Brain |
Example |
| "Run 3 passes, check gaps after each, cite everything" |
Required? | Yes (the engine) | Optional (but recommended for research quality) |
Quick setup:
# 1. Install the tools (you already have this)
uvx pyrecrawl@latest
# 2. Add the research skill to your project
git clone https://github.com/SanggonBoy/pyrecrawl-skills.git /tmp/pyrecrawl-skills
cp /tmp/pyrecrawl-skills/pyrecrawl-research/SKILL.md ./CLAUDE.md # or .cursorrules / AGENTS.mdWithout skills: Your agent has powerful tools but improvises usage. With skills: Your agent follows a proven research protocol with anti-hallucination guardrails.
๐ Manual config (if pyrecrawl install doesn't match your setup)
Claude Desktop
Config file
Linux:
~/.config/Claude/claude_desktop_config.jsonmacOS:
~/Library/Application Support/Claude/claude_desktop_config.jsonWindows:
%AppData%\Claude\claude_desktop_config.json
{
"mcpServers": {
"pyrecrawl": {
"command": "uvx",
"args": ["--from", "pyrecrawl", "pyrecrawl", "serve"]
}
}
}Claude Code
Config file: project-scoped .mcp.json
{
"mcpServers": {
"pyrecrawl": {
"command": "uvx",
"args": ["--from", "pyrecrawl", "pyrecrawl", "serve"]
}
}
}Cursor
Config file: ~/.cursor/mcp.json
{
"mcpServers": {
"pyrecrawl": {
"command": "uvx",
"args": ["--from", "pyrecrawl", "pyrecrawl", "serve"]
}
}
}VS Code / Copilot
Config file: .vscode/mcp.json (project-scoped)
{
"servers": {
"pyrecrawl": {
"command": "uvx",
"args": ["--from", "pyrecrawl", "pyrecrawl", "serve"],
"type": "stdio"
}
}
}Codex CLI
Config file: ~/.codex/config.toml
[mcp_servers.pyrecrawl]
command = "uvx"
args = ["--from", "pyrecrawl", "pyrecrawl", "serve"]OpenCode
Config file: ~/.config/opencode/opencode.json
{
"mcp": {
"pyrecrawl": {
"type": "local",
"command": ["uvx", "--from", "pyrecrawl", "pyrecrawl", "serve"],
"enabled": true
}
}
}Hermes
Config file
Linux/macOS:
~/.hermes/config.yamlWindows:
%LocalAppData%\hermes\config.yaml
mcp_servers:
pyrecrawl:
command: uvx
args:
- --from
- pyrecrawl
- pyrecrawl
- serve
enabled: trueWindows note:
uvxmust be on PATH. If not, use the full path touvx.exe(e.g.C:\Users\<you>\AppData\Local\hermes\bin\uvx.exe).
๐ง How the ladder chooses
PyreCrawl runs each request through three tiers, stopping at the first one that returns a complete, LLM-ready result:
Concern | Fast tier | Stealth tier | Deep tier |
Static HTML page | โ ~200ms | โ | โ |
Cloudflare-protected | โ | โ Turnstile solver | โ |
JS-heavy SPA | โ | โ real Chromium | โ |
Live DOM data (input | โ | โ
| โ |
LLM-ready markdown + citations | โ | โ | โ BM25, fit-markdown |
Structured extraction (CSS schema) | โ | โ | โ |
Deep crawl (BFS/DFS/BestFirst) | โ | โ | โ adaptive |
The agent never has to pick. prefer="auto" does it every call.
Live DOM data with js and wait_for
Some sites keep the data you want in a DOM property (e.g. an <input>'s .value)
that JS writes after an XHR โ it never appears in the serialized HTML. The
scrape tool accepts two stealth-tier params for exactly this:
{
"url": "https://temp-mail.org/id",
"prefer": "stealth",
"wait_for": "document.getElementById('mail').value.includes('@')",
"js": "document.getElementById('mail').value"
}wait_forโ a JS predicate expression polled until truthy (bounded bytimeout). Use it instead of guessing a sleep for anything that arrives asynchronously.jsโ a JS expression evaluated once the page settles; the value comes back inmeta.js_result. Errors are captured inmeta.js_error(the page result is still returned, never a crash).
๐ Compared to Firecrawl (hosted)
Firecrawl | PyreCrawl | |
Cost | Free 1k/mo, then $16โ333/mo | Free, self-hosted |
Local LLM support | โ | โ Ollama / any LLM |
Cloudflare bypass | โ (Fire-Engine, paid) | โ (free, built-in) |
Markdown + BM25 | โ | โ |
Self-host | โ | โ |
Academic paper search | โ | โ
arXiv + Crossref ( |
Hosted search API | โ /search | โ ๏ธ DuckDuckGo HTML + arXiv/Crossref (no key) |
๐ง Development
git clone https://github.com/SanggonBoy/PyreCrawl.git
cd PyreCrawl
uv venv --python 3.12 .venv
source .venv/Scripts/activate # Windows; or .venv/bin/activate on macOS/Linux
uv pip install -e ".[dev]"
python -m playwright install chromium
scrapling installRun tests
python scripts/selfcheck.py # real-network smoke test (13 tools + engines)
python scripts/probe_stdio.py # stdio JSON-RPC probe
python scripts/test_ladder_bug.py # SPA-shell ladder escalation regression
python scripts/test_js_eval.py # stealth js/wait_for params regression
python scripts/test_scope_selector.py # crawl css_selector/max_depth wiring
python scripts/test_link_harvest.py # map/BFS link purity regression๐ฆ Publish
Maintainers only:
git tag vX.Y.Z
git push origin vX.Y.ZGitHub Actions builds + uploads to PyPI via trusted publishing.
๐ Stay up to date
PyreCrawl checks PyPI on every startup and reports the latest version โ your
MCP agent sees this automatically via the health() tool response and can
notify you inline.
To check manually:
pyrecrawl versionTo upgrade:
pyrecrawl update # runs: uv tool upgrade pyrecrawlGet notified of new releases: click Watch โ Releases only at the GitHub repo to receive email notifications when a new version is published.
PyreCrawl sendsone anonymous usage ping per 24 h at server startup โ see Privacy for exactly what's sent and how to opt out.
๐ Privacy โ anonymous usage ping
PyreCrawl phones home once per 24 h with a tiny anonymous ping when the MCP server starts, so we can count real users (DAU/MAU) instead of raw downloads.
Sent (4 fields, ~100 bytes) | Never sent |
Hashed machine id (SHA-256 of hostname+MAC โ not reversible) | Your IP (not stored) |
PyreCrawl version | Any URL you scrape |
Python version | Any page content or search queries |
OS family ( | Anything else |
Client code: src/pyrecrawl/telemetry.py (~90 lines, stdlib only) ยท
Collector: workers/telemetry/ โ a self-hostable Cloudflare Worker + D1, no third-party analytics service.
Opt out any time:
export PYRECRAWL_NO_TELEMETRY=1 # or the industry-standard DO_NOT_TRACK=1๐ Uninstall
# Remove from all agent configs
pyrecrawl uninstall
# Remove the package
uv tool uninstall pyrecrawl๐ก๏ธ License
MIT โ see LICENSE.
Available Tools
13 toolsbatch_scrapeA
Scrape MANY URLs in ONE call (parallel, deduped, cache-aware).
Use when the user provides multiple URLs or you have a list of pages
to fetch. More efficient than calling scrape N times.
Args: urls: Target URLs (deduped automatically; empties dropped). prefer: "auto" | "fast" | "stealth" | "llm". timeout: per-URL timeout in seconds. max_concurrency: parallel workers (default 4). include_html: include raw HTML per result (large; off by default).
Returns {requested, unique, succeeded, failed, results[]}. Per-URL failures are isolated โ other URLs still succeed.
Returns: {requested, unique, succeeded, failed, results: [{url, markdown, ...}]}
| Name | Required | Description | Default |
|---|---|---|---|
| urls | Yes | ||
| prefer | No | auto | |
| timeout | No | ||
| include_html | No | ||
| max_concurrency | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries full burden. It discloses key behaviors: parallel execution, automatic deduplication, cache-awareness, per-URL failure isolation, and the return structure. It does not mention rate limits, auth, or redirect handling, but the disclosed traits are sufficient for typical agent decisions. Minor gap is not explaining what 'cache-aware' implies operationally.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is efficiently written, front-loaded with the core purpose and usage guidance, then parameter details. It repeats the return structure twice (once in prose, once in a return block), which is slightly redundant but not harmful. Structure is logical and skimmable.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has 5 parameters and no annotations, the description covers all needed context: when to use, how to set parameters, return format, failure isolation, and efficiency rationale. It even explains the deduping and default for include_html. Nothing critical is missing for an agent to invoke it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It does so thoroughly: each parameter gets a meaningful explanation (e.g., 'urls: Target URLs (deduped automatically; empties dropped)', 'prefer: "auto" | "fast" | "stealth" | "llm"', 'timeout: per-URL timeout in seconds'). This adds semantics well beyond the schema's types and defaults.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states 'Scrape MANY URLs in ONE call' with explicit characteristics (parallel, deduped, cache-aware), clearly distinguishing it from the single-URL sibling 'scrape'. The verb-resource pairing is unmistakable and the emphasis on batch capability differentiates it from all sibling tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explicitly states when to use it: 'Use when the user provides multiple URLs or you have a list of pages to fetch.' It also explains benefits over the alternative ('More efficient than calling `scrape` N times'), giving clear routing guidance without needing to inspect the schema.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
cacheA
Inspect the response cache: stats, clear, enable, or disable.
Use this to check cache hit rates before large batch jobs, or to clear stale cached responses when a site's content has changed.
Args: action: "stats" (default) | "clear" | "disable" | "enable".
| Name | Required | Description | Default |
|---|---|---|---|
| action | No | stats |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the behavioral burden. It does communicate the basic operation model and even hints at what clearing affects ('stale cached responses'). However, it does not disclose whether 'clear' wipes the entire cache, whether enable/disable persists, or whether any actions are destructive or require privileges.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact and front-loaded: a one-sentence purpose, two concrete use cases, then the parameter details. Every sentence earns its place and there is no redundant or filler content.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple one-parameter tool with an output schema, the description covers the action values and gives practical usage context. It is nearly complete, but slightly more detail on the side effects of clear/disable/enable would make it fully self-sufficient given the absence of annotations.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Despite 0% schema description coverage, the description fully documents the only parameter: action, its allowed values ('stats' | 'clear' | 'disable' | 'enable'), and its default ('stats'). This adds real meaning beyond the raw schema, which only shows a string with a default.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The first sentence states the tool's purpose with a specific verb and resource: 'Inspect the response cache' and then enumerates the exact operations supported (stats, clear, enable, disable). This unambiguously separates it from the sibling scraping and search tools, none of which touch the cache.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives clear, actionable use cases: check hit rates before large batch jobs, and clear stale responses after content changes. It does not explicitly mention alternatives or exclusions, but no alternative cache tool exists among the siblings, so the guidance is sufficient.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
crawlA
Multi-page crawl: discover URLs on root, then scrape each.
Use this when the user wants to crawl an entire site section or docs,
or needs multiple pages scraped in bulk. For single pages use scrape;
for research questions use deep_research.
Args:
root: start URL.
max_pages: hard cap on pages scraped.
css_selector: scope each page's html/markdown to the matched element
(non-llm: lxml re-scope of the fetched HTML; llm: native crawl4ai
css_selector).
prefer: "auto" | "fast" | "stealth" | "llm" (llm = Crawl4AI BFS deep-crawl).
include_paths: regex โ keep only URLs matching (matched against full URL).
exclude_paths: regex โ drop URLs matching (e.g. /tag/|/page/\d+).
max_depth: 0 = flat harvest from the root page's links (default);
>0 = true BFS up to that link depth, honoring the filters.
Returns: {root, pages: [{url, markdown, title, ...}], count, discovered, elapsed_ms} or {error, root} on failure.
| Name | Required | Description | Default |
|---|---|---|---|
| root | Yes | ||
| prefer | No | auto | |
| max_depth | No | ||
| max_pages | No | ||
| css_selector | No | ||
| exclude_paths | No | ||
| include_paths | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full behavioral burden, and it delivers: it explains the discover-then-scrape flow, flat vs BFS depth behavior, filtering by regex, the `prefer` mode variants, and the exact success/error return shapes. This goes well beyond a minimal statement of what the tool does.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured and front-loaded: a crisp one-sentence summary, then usage guidance, then a tight Args block, then the Returns shape. Every line adds information without padding, which is appropriate for a 7-parameter tool with no schema-level parameter descriptions.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no annotations, an output schema present, and seven parameters, the description is operationally self-sufficient. It covers purpose, when to use it, alternatives, all parameter semantics, the crawling algorithm variants, filters, and the return contract including the error case.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate for every parameter, and it does. All seven arguments are meaningfully explained, including the regex semantics for include/exclude paths, the meaning of max_depth, and the `prefer` option values.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific, action-oriented definition: 'Multi-page crawl: discover URLs on root, then scrape each.' It also distinguishes itself from siblings by naming `scrape` for single pages and `deep_research` for research questions.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description clearly says when to use crawl (entire site section, docs, bulk multi-page needs) and gives explicit alternatives for single pages and research questions. However, it does not distinguish crawl from the `batch_scrape` sibling, and its phrase 'multiple pages scraped in bulk' could overlap with that tool, so the routing guidance is not fully exhaustive.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
deep_researchA
Search the web, then pull the top sources as EVIDENCE (no LLM synthesis).
PRIMARY RESEARCH TOOL โ use when the user asks to research, investigate,
deep-dive, fact-check, or learn about a topic. Returns a citations
list with stable [n] numbers and an evidence list of per-source
markdown โ the agent does the synthesis from evidence.
Multi-pass mode: set iterations=2-3 to auto-run additional searches with
refined queries (alternatives, criticism, latest developments) and append
deduplicated evidence. Each pass adds up to scrape_top new sources.
Args: query: search string. limit: how many search results to fetch. scrape_top: how many of those to actually fetch content from. prefer: "auto" | "fast" | "stealth" | "llm". iterations: 1 (default, single pass), 2-3 (multi-pass with refined queries targeting evidence gaps). Each pass searches from a different angle and deduplicates by URL.
Returns: {query, iterations_run, queries: [str], hits: [{url, title, snippet}], citations: [{url, title}], evidence: [{url, title, markdown}], scraped, used_engines, elapsed_ms} or {error, query, hint} on failure.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | ||
| query | Yes | ||
| prefer | No | auto | |
| iterations | No | ||
| scrape_top | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and it delivers: it discloses that no LLM synthesis occurs, that multi-pass runs refine queries and deduplicate by URL, and that failures return a structured error object with a hint. This gives an agent a realistic model of what the tool will and will not do.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is organized with a purpose statement, usage trigger, mode explanation, per-argument semantics, and a complete return shape. While long, every section earns its place and the most important decision-relevant facts are front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool has five parameters, multiple modes, and a rich return value, and the description covers all of them: arguments, multi-pass behavior, output shape, and error shape. Nothing an agent needs to invoke or interpret the result is missing, and the output schema is effectively described inline.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must document all five parameters and does: query, limit, scrape_top, prefer (with listed option values), and iterations (with concrete guidance on 1 vs 2-3). The iterations explanation adds behavior beyond the schema's bare integer type.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: 'Search the web, then pull the top sources as EVIDENCE (no LLM synthesis).' It clearly distinguishes this tool from generic search or scrape siblings by labeling it the 'PRIMARY RESEARCH TOOL' and explicitly stating the agent performs synthesis from evidence.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives explicit trigger conditions: 'use when the user asks to research, investigate, deep-dive, fact-check, or learn about a topic.' It also explains the multi-pass option for deeper research. However, it does not explicitly name sibling alternatives or state when not to use this tool, so it stops short of full exclusion guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
documentA
Extract text from a PDF/DOCX/PPTX URL โ markdown (no browser).
Use when the user shares a link to a document (PDF, Word, PowerPoint)
and wants its text content. Also useful after search_papers to get
full text from a paper's pdf_url.
Content-type sniffed and routed to pypdf / python-docx / python-pptx.
Optional deps โ install with pip install 'pyrecrawl[docs]'.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| timeout | No | ||
| max_pages | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the behavioral burden; it adds that extraction is non-browser, content type is sniffed and routed to pypdf/python-docx/python-pptx, and that optional dependencies may need installing. It could mention failure modes or whether any state changes occur, but the read-only extraction framing is clear.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Every sentence adds value and the most identifying information is front-loaded. The usage cases, routing detail, and dependency caveat are each one concise line, with no filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple extracted-text tool with an output schema, the description covers purpose, use case, supported formats, routing, and install needs. It is incomplete for the two optional parameters (`timeout`, `max_pages`), which are not documented in the schema or the description.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must explain parameters. It clarifies only `url` by restricting it to PDF/DOCX/PPTX document links; `timeout` and `max_pages` are left completely unexplained beyond their names and defaults.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The first line names an exact action (extract text), input family (PDF/DOCX/PPTX URL), output (markdown), and an explicit constraint (no browser). It also places the tool relative to `search_papers`, so an agent can distinguish document conversion from generic scraping siblings.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives concrete conditions: use when the user shares a PDF/Word/PowerPoint link and wants text, and after `search_papers` on a pdf_url. It does not explicitly name sibling tools to avoid or state hard exclusions, though 'no browser' implicitly rules out general web pages.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
extractA
Scrape + structured extraction using a CSS-based JSON schema.
Use when the user wants structured data (tables, lists, product info) extracted from a page. Define a CSS schema to target specific elements.
The schema is a JsonCssExtractionStrategy schema: { "name": "PageItems", "baseSelector": "div.item", "fields": [{"name": "title", "selector": "h2", "type": "text"}, ...] }
Returns parsed JSON in data.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| prefer | No | auto | |
| schema | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must carry the burden. It discloses that this is a scraping/extraction operation and says it returns parsed JSON in `data`. But it omits practical behavioral details like dynamic-content handling, page-size limits, error behavior, and what the `prefer` option controls.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact and front-loaded. The first sentence captures the essence, and the rest provides a concrete example and return-shape note without wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The schema example and return mention make the tool callable, but with no annotations and zero parameter descriptions, important usage context is missingโespecially around `prefer` and dynamic page behavior. It is adequate but not complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It does a good job for the `schema` parameter by providing a concrete JsonCssExtractionStrategy example, but `prefer` is not explained at all, and `url` is only meaningful by convention.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific operation ('Scrape + structured extraction') with a clear mechanism (CSS-based JSON schema) and names concrete use cases (tables, lists, product info). This distinguishes it from siblings like scrape or crawl, which are for less structured data collection.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explicitly says when to use the tool: when the user wants structured data extracted from a page. However, it does not explicitly name alternatives or state when not to use it, so the contrast with sibling tools is mostly implicit.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
healthA
Verify PyreCrawl is working: engine versions, dependencies, update status.
Use this at the start of a session or before a large scraping job to confirm all engines are installed and up to date.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It discloses the scope of the check (engine versions, dependencies, update status) and implies a read-only diagnostic operation. However, it does not detail any side effects (likely none), how it reports failures (e.g., exit codes, summary output), or whether it performs network calls. For a zero-parameter health check, this is adequate but not rich.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences that front-load the purpose and then immediately provide usage guidance. Every word earns its place; there is no fluff or redundant phrasing. It is concise and structured optimally for agent parsing.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a zero-parameter health check tool, the description explains what it verifies and when to run it. The output schema (not shown in the input but indicated as present) likely covers the exact return format. The description is sufficient for an agent to know why and when to call the tool, though it does not specify what constitutes 'working' beyond installed engines. This is a minor gap, but overall complete for the tool's simplicity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters, so there is no schema coverage gap. The description does not need to elaborate on parameters. According to the rubric, a 0-parameter baseline is 4, and the description adds no irrelevant parameter details, so this score is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states what the tool does: 'Verify PyreCrawl is working: engine versions, dependencies, update status.' It uses a specific verb and resource, and the purpose is distinct from sibling tools that perform scraping tasks. An agent can immediately understand this is a pre-flight health check, not a scraping operation.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly says when to use it: 'at the start of a session or before a large scraping job to confirm all engines are installed and up to date.' This gives concrete context. It does not explicitly contrast with alternatives (e.g., 'use rather than scrape when...'), but the purpose and usage are clear enough that an agent would be unlikely to confuse it with a scraping tool.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
map_siteA
Enumerate all internal URLs reachable from root.
Use when the user wants to map a site's structure or find all pages
before crawling. Often paired with crawl or batch_scrape.
Args: root: Website root (e.g. "https://example.com/docs"). include_pattern: Optional regex; only URLs matching are returned. limit: Hard cap on returned URLs.
| Name | Required | Description | Default |
|---|---|---|---|
| root | Yes | ||
| limit | No | ||
| include_pattern | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full behavioral disclosure burden. It does add useful constraints โ 'internal URLs' scoping, 'reachable from root' traversal semantics, and the 'hard cap' behavior of `limit` โ but leaves significant traits undisclosed: auth requirements, rate limits, redirect handling, and whether the tool makes many network requests. Meaningful but incomplete for an unannotated tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is tightly structured โ core purpose in the first sentence, usage guidance in the second, then a clean Args block. Every sentence earns its place, and the most decision-relevant information (what it returns, internal scope) is front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 3-parameter tool with an output schema, the description covers the essentials: purpose, when to use it, related tools, and all parameters with semantics. Minor gaps remain โ how traversal handles redirects, depth limits, and protocol variants โ but an agent has enough to invoke the tool correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate, and it does: `root` gets a format example, `include_pattern` gets its regex filter semantics explained, and `limit` is clarified as a 'Hard cap on returned URLs'. All three parameters receive meaning beyond the bare schema types, fully covering the parameters that the schema left undocumented.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The first sentence is a specific verb+resource statement โ 'Enumerate all internal URLs reachable from `root`' โ which precisely defines the scope and distinguishes this from siblings like `crawl` (which fetches content) and `search` (which queries). The pairing note with `crawl`/`batch_scrape` further clarifies its role as a pre-scrape discovery step.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
'Use when the user wants to map a site's structure or find all pages before crawling' gives clear, actionable context, and naming `crawl` and `batch_scrape` as common companions helps an agent understand the workflow. However, it stops short of explicit exclusions โ it never states 'use crawl instead when you need page content only,' so the when-not guidance is implied rather than stated.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
monitorA
Track a URL over time and report meaningful content changes.
Use when the user wants to watch a page for updates (price changes, new blog posts, status updates). Ask me to 'set up monitoring for ' for a guided setup playbook.
Args:
url: target URL.
action: "check" | "history" | "forget".
prefer: ladder preference, same as scrape.
css_selector: scope the diff to one element (so banner /
nav changes don't trigger false positives).
Snapshots persist under PYRECRAWL_MONITOR_DIR (default
~/.pyrecrawl/monitors/). check returns status of
new | unchanged | changed | error and a unified
diff when the page changed.
Returns: {url, status: "new"|"unchanged"|"changed"|"error", diff?: str, snapshot_chars?: int, elapsed_ms?: int}
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| action | No | check | |
| prefer | No | auto | |
| css_selector | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the burden, and it delivers by disclosing that snapshots persist under PYRECRAWL_MONITOR_DIR, that check returns status values including error, and that a unified diff is produced. It could go further by explaining the side effects of 'forget' and retention behavior, but the core stateful behavior is well disclosed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured with a purpose statement, explicit usage trigger, Args block, and Returns block. It is longer than minimal but each section earns its place; the guided-setup sentence adds a useful interaction hint rather than filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a stateful tool with three actions, the description details only 'check' behavior. 'history' and 'forget' are listed but their effects, return shapes, and side effects are unexplained. Since there are no annotations and schema coverage is 0%, this is a meaningful gap for an agent deciding how to invoke the tool correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It explains url, enumerates action values, gives css_selector its diff-scoping purpose, and references the 'prefer' ladder via scrape. The main gap is that 'history' and 'forget' semantics are not expanded, and 'prefer' relies on knowing what scrape does.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: 'Track a URL over time and report meaningful content changes.' This clearly distinguishes monitor from one-off tools like scrape and crawl, and the concrete use cases (price changes, blog posts, status updates) reinforce the purpose.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explicitly says 'Use when the user wants to watch a page for updates' and provides a guided setup cue. However, it does not explicitly say when not to use this tool or name alternatives like scrape for one-off fetches, so it misses the exclusion aspect.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
scrapeA
Scrape a single URL โ LLM-ready markdown.
Use this when the user shares a URL and wants its content (read, analyze, summarize, extract). Auto-escalates through fastโstealthโllm when blocked.
Args:
url: Target URL (http/https).
prefer: "auto" | "fast" | "stealth" | "llm".
auto = fast first, escalate to stealth on block/short page.
fast = cheap HTTP only (no JS).
stealth = real Chromium + Cloudflare solver.
llm = full Crawl4AI browser + BM25 fit-markdown.
timeout: per-attempt timeout in seconds.
include_html: include raw HTML in the response (large; off by default).
js: (stealth only) JS expression evaluated against the live page
after it settles. The value comes back in meta.js_result.
Use for data that lives in DOM properties (e.g. an input's
.value) rather than in serialized HTML.
wait_for: (stealth only) JS predicate expression polled until truthy
(bounded by timeout). Use to wait for content that arrives
asynchronously after network_idle.
Returns: {url, final_url, status, markdown, title, method, elapsed_ms, meta} or {error, url, method} on failure.
| Name | Required | Description | Default |
|---|---|---|---|
| js | No | ||
| url | Yes | ||
| prefer | No | auto | |
| timeout | No | ||
| wait_for | No | ||
| include_html | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are present, so the description carries the full behavioral burden and succeeds. It discloses the auto-escalation path ('fastโstealthโllm when blocked'), explains what each mode does (cheap HTTP, real Chromium + Cloudflare solver, full Crawl4AI browser), and scopes js and wait_for to stealth only. It also documents the success and failure return shapes.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is tightly structured: a one-line purpose, a single usage sentence, a compact Args block, and a Returns block. Every line is information-dense with no filler, and the most important scoping details are front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
This is a complex tool with 6 parameters, 4 execution modes, JS evaluation, and async waiting. The description accounts for every parameter, the escalation behavior, return values, and failure output. There are no obvious omissions that would prevent an agent from invoking it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has no descriptions and only bare titles/defaults, so the description must compensate. It does: url is constrained to http/https, prefer enumerates all four modes with escalation semantics, timeout is defined as per-attempt seconds, include_html warns about response size, and js/wait_for get exact behavioral definitions with an example for js.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb-resource statement: 'Scrape a single URL' and 'LLM-ready markdown'. It names a clear trigger ('when the user shares a URL and wants its content'), and 'single URL' separates it from multi-URL siblings like crawl and batch_scrape. However, it does not explain how it differs from sibling extract, so differentiation is incomplete.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives a concrete usage context: 'Use this when the user shares a URL and wants its content (read, analyze, summarize, extract)'. That is clear context, but it does not list exclusions or point to alternative tools such as extract or crawl, leaving the when-not-to-use judgment to the agent.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
searchA
Web search via DuckDuckGo HTML (no API key required).
Use this for targeted searches where you need anti-bot bypass (Cloudflare
protection on DDG). For simple searches, the built-in web_search may suffice.
For research questions, prefer deep_research (search + scrape + citations).
Returns [{url, title, snippet}, ...]. The smart ladder bypasses DDG's bot detection if needed.
Returns: {query, results: [{url, title, snippet}], count} or {error, query} on failure.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | ||
| query | Yes | ||
| prefer | No | auto |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It discloses the use of DuckDuckGo HTML, no API key requirement, the smart ladder bypass, and the return format including error handling. It doesn't mention rate limits or other limitations, but for a search tool it is fairly transparent.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is mostly concise but repeats the return format twice (once as a list, once as a structured object). The 'smart ladder' phrase is vague. Overall, it is structured and not overly verbose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity, the description covers usage guidance, return format, and error handling. However, it fails to explain parameters and does not mention any rate limits or usage constraints. With no annotations, more detail on parameters and limitations would improve completeness.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It does not explain the 'limit' or 'prefer' parameters, and only implies the query. This leaves ambiguity about what these parameters control, especially 'prefer' which is unclear. The description adds minimal value beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'Web search via DuckDuckGo HTML' which identifies the verb (search) and resource (web via DuckDuckGo). It also differentiates from siblings like deep_research and web_search, making its purpose unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly provides when-to-use guidance: targeted searches needing anti-bot bypass, with alternatives for simple searches (web_search) and research questions (deep_research). Also mentions the smart ladder for bot detection, leaving no ambiguity about selection.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
search_papersA
Search academic papers via arXiv or Crossref โ no API keys.
Args: query: free-text search, e.g. "transformer attention scaling laws". limit: max results (1-25 arXiv / 1-20 crossref). source: "arxiv" (CS/physics/math preprints, default) or "crossref" (all fields, DOI-backed). category: optional arXiv category filter, e.g. "cs.LG", "cs.CV".
Returns papers with id/url/pdf_url/title/authors/summary/published.
Feed pdf_url into the document tool to extract full text.
Returns: {query, source, papers: [{title, authors, abstract, url, pdf_url?, ...}]} or {error, query} on failure.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | ||
| query | Yes | ||
| source | No | arxiv | |
| category | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It discloses that no API keys are needed, specifies per-source result limits (1-25 arXiv / 1-20 Crossref), describes the return fields (id/url/pdf_url/title/authors/summary/published), and mentions the error return format. It also notes that pdf_url is optional (indicated by the '?' in the return example). This is strong, though it could explicitly state the operation is read-only and has no side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured with a heading, a bulleted Args list, and a Returns section. Every sentence carries useful information: examples, default values, and a downstream workflow hint. It front-loads the purpose and keeps the details organized, with no filler or repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description is complete for a 4-parameter tool. It covers all parameters, provides return format details, notes the error case, and suggests an integration with the 'document' tool. Given the existing output schema (not shown but present), the description does not need to repeat return values, and it supplies enough for an agent to call it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description fully compensates. It explains each parameter: query with an example ('transformer attention scaling laws'), limit with per-source ranges, source with defaults and meanings ('arxiv' for preprints, 'crossref' for DOI-backed), and category with an example format ('cs.LG'). This adds significant meaning beyond the bare schema, making parameters self-explanatory.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: 'Search academic papers via arXiv or Crossref โ no API keys.' This is a specific verb (search) and resource (academic papers) with two named sources, and it naturally distinguishes from siblings like the general 'search' tool or 'document' for extraction. The mention of specific sources and the absence of API keys adds clarity.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear context for when to use this tool: when searching academic papers via arXiv or Crossref. It also gives a workflow hint by suggesting to feed the returned pdf_url into the 'document' tool for full-text extraction. However, it does not explicitly state when not to use it or compare with alternatives like 'search' or 'deep_research', so it falls slightly short of full guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
sessionA
Drive a persistent browser session โ cookies & JS state kept across calls.
Use for login walls and multi-step flows the one-shot ladder can't handle
(e.g. user needs to log in first, then scrape a protected page).
For simple pages use scrape; for research use deep_research.
One-shot scrape has no session memory; here each action runs against the same live page.
Args: session: named session; reuse the same name to keep state. action: "open" (url) โ navigate, returns url/title/status "click" (selector) โ click an element "fill" (selector, text) โ type into an input "type" (key) โ press a key, e.g. "Enter" "eval" (js) โ run a JS expression, returns value "wait" (selector?) โ wait for selector or sleep timeout_ms "content" โ url/title/visible text of current page "screenshot" (full_page?) โ returns png_base64 "cookies" โ list session cookies "close" โ destroy the session "list" โ show live sessions Returns {session, action, ...result, elapsed_ms}; errors as {error}.
| Name | Required | Description | Default |
|---|---|---|---|
| js | No | ||
| key | No | ||
| url | No | ||
| text | No | ||
| action | No | open | |
| session | No | default | |
| headless | No | ||
| selector | No | ||
| full_page | No | ||
| timeout_ms | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden and meets it: it enumerates every action (open, click, fill, type, eval, wait, content, screenshot, cookies, close, list) with their effects, explicitly states 'close โ destroy the session' as destructive, and specifies the return format ('Returns {session, action, ...result, elapsed_ms}; errors as {error}'). It also discloses persistence of cookies and JS state.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured: it opens with purpose, then usage guidelines, then the stateful vs one-shot distinction, followed by a concise action-by-action list and return format. Every sentence adds value; the length is justified by the tool's complexity but remains scannable and front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with 10 parameters and 12 actions, this description is extraordinarily complete: it covers purpose, invocation conditions, all actions with their inputs and outputs, return and error formats, and distinguishes from siblings. An output schema exists, so return values don't need further explanation. Nothing essential for correct use is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema has 0% description coverage, so the description must compensate. It does so well by detailing parameters under 'Args:' including session, and for each action the relevant parameters (url, selector, text, key, js, full_page, timeout_ms). However, the `headless` parameter is never mentioned in the description, leaving its purpose undocumentedโa minor gap given the otherwise thorough coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific purpose: 'Drive a persistent browser session โ cookies & JS state kept across calls.' This clearly identifies the tool as a session manager and differentiates it from one-shot tools like scrape and deep_research, which are explicitly named as alternatives.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides explicit usage guidance: 'Use for login walls and multi-step flows the one-shot ladder can't handle' and tells exactly when to use alternatives: 'For simple pages use `scrape`; for research use `deep_research`.' Also clarifies the key distinction: 'One-shot scrape has no session memory; here each action runs against the same live page.' This leaves no ambiguity about when to invoke this tool.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
1 tool update
v0.8.0- Changed
deep_research1 field changed- added
Input schema / properties / iterationsAdded value: +{ + "default": 1, + "title": "Iterations", + "type": "integer" +}
13 tool updates
v0.7.1- First observed
batch_scrape - First observed
cache - First observed
crawl - First observed
deep_research - First observed
document - First observed
extract - First observed
health - First observed
map_site - First observed
monitor - First observed
scrape - First observed
search - First observed
search_papers - First observed
session
TDQS
Scored across 13 tools
Each tool has a clearly distinct trigger: single-page scrape, structured extraction, bulk scrape, site crawl, document parsing, web search, deep research, academic search, monitoring, sessions, and cache/health maintenance. Even though several tools fetch pages, their arguments and return shapes make the intended use obvious.
Names are all lowercase and action-oriented, but the convention is mixed: single-word verbs (scrape, crawl, search), verb_noun compounds (map_site, search_papers), and modifier compounds (batch_scrape, deep_research), plus noun-style tools like health and session. This is readable and mostly predictable, but not a uniform pattern.
13 tools is well within the ideal range for a scraping and research suite. Each tool covers a distinct operation, and the utility tools (cache, health) support the workflow without feeling like filler.
The surface covers the full scraping/research workflow: single and batch scraping, crawling, structured extraction, document parsing, search, deep research, academic papers, monitoring, sessions, and cache/health management. There are no obvious dead ends โ monitors can be forgotten, sessions can be closed, and cache can be cleared or disabled.
Maintenance
Related MCP Connectors
65+ AI tools as MCP: research, write, code, scrape, translate, RAG, agent memory, workflows
Your agent needs the open web โ searched by more than one engine, and read as clean markdown rather than raw HTML. **What you can ask for** โข "Search this question with two providers and tell me where they disagree." โข "Scrape these 40 URLs into markdown, in one batch." โข "Crawl this documentation site and give me every page." โข "Do deep research on this topic and cite the sources." โข "Find the academic papers behind this claim." **How to use it** Point any MCP client at https://mcp.aisa.one/search/mcp and sign in with OAuth โ there is no key to create or paste. 30 tools across several independent providers: Tavily and Exa search, answers, contents and agent runs; Firecrawl scrape, batch scrape, crawl, map and search; Perplexity Sonar, Sonar Pro, reasoning and deep research; Oxylabs AI search and LLM jobs; OpenAI and Anthropic web search; and scholarly search. **Why this rather than the source** Several independent indexes behind one account, because one engine's blind spot is not visible from inside it. **It is also a door to the rest** The same login reaches 26 sources and 580+ operations. Find the page here, then ask the same agent who links to it or how much traffic it gets โ without adding a second server. **What it costs** Finding and inspecting an operation is free. Running one is billed per call at API prices, with no seat and no monthly minimum, and every call takes max_price_usd so an agent cannot overspend by accident. **Where else it reaches** https://mcp.aisa.one/seo-serp/mcp for the Google results page itself, https://mcp.aisa.one/seo-serp-other-engines/mcp for Bing, Baidu and Naver.
Scrape, crawl and search the web for AI agents via MCP.
- mcpOAuthcom.screenshotink
Screenshot, diff, audit and sitemap-capture any web page โ 5 MCP tools for AI agents.
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceWeb scraping MCP server for Al agents. 6 tools: extract clean text/markdown from any URL, structured scraping with CSS selectors, full-page screenshots via Playwright, link extraction with regex filtering, metadata extraction (OG tags, Twitter cards), and Google search. Free tier: 50 requests/IP/day.8MIT
- AlicenseNot gradedqualityDmaintenanceMulti-tool MCP server for AI agents with 29 tools across web scraping, SEO analysis, screenshot and PDF generation, domain intelligence, content extraction, multi-chain EVM blockchain queries, and security toolkit. Free tier available with no auth required.12 npm1MIT
- AlicenseAqualityAmaintenanceWeb scraping, crawling, and structured data extraction for AI agents. 5 tools: scrape (clean markdown from any URL), crawl (entire sites), map (discover URLs), extract (structured JSON), and search. 833ms avg latency, single binary, self-hostable.81,057AGPL 3.0
- AlicenseNot gradedqualityDmaintenance14 web scrapers as MCP tools for AI agents: Reddit, Amazon, Google Maps, Yelp, YouTube, Indeed & more.21 npm2MIT