Skip to main content
Glama

๐Ÿ”ฅ PyreCrawl โ€” Web Browsing Superpowers for Your AI Agent

License: MIT MCP Python 3.10+ PyPI GitHub stars Downloads / 30d Active users / 30d

One command gives any AI agent the whole web. Scrape, extract, crawl, map, and search โ€” self-hosted, no API keys, no rate limits, no subscription.

PyreCrawl speaks MCP (Model Context Protocol), the standard tool interface for Claude, Cursor, VS Code, Codex, OpenCode, Hermes, and any MCP-compatible agent.

A smart auto-fallback ladder always picks the cheapest method that succeeds:

fast HTTP
    โ”‚  (403/503/Cloudflare challenge or empty body)
    โ–ผ
stealth browser (real Chromium + Cloudflare solver)
    โ”‚  (still blocked, or the page needs full JS rendering)
    โ–ผ
deep processing (LLM-ready markdown, citations, structured extraction)

โšก Tools exposed

Tool

What it does

scrape(url, prefer="auto")

Single URL โ†’ LLM-ready markdown

extract(url, schema)

Scrape + structured extraction (JsonCss schema)

map_site(root, include_pattern=None, limit=200)

Enumerate all internal URLs

crawl(root, max_pages=5, prefer="auto", include_paths=None, exclude_paths=None, max_depth=0)

Multi-page crawl with path filters + true BFS depth

document(url)

PDF/DOCX/PPTX โ†’ markdown (no browser, optional [docs] extras)

search(query, limit=10)

Web search via DuckDuckGo HTML (no API key)

search_papers(query, limit=8, source="arxiv", category=None)

Academic search via arXiv + Crossref (no API key) โ€” feed pdf_url into document

batch_scrape(urls[], ...)

Many URLs in ONE call โ€” parallel, deduped, cache-aware

deep_research(query, limit=5, scrape_top=3)

Search โ†’ evidence pack with [n] citations (no LLM synthesis โ€” your agent does that)

monitor(url, action, css_selector=None)

Change detection with persisted snapshots + unified diff

session(session, action, ...)

Persistent browser session (cookies kept) โ€” login walls, multi-step flows, screenshots

cache(action)

Inspect/clear/enable/disable the HTTP response cache

health()

Versions + import sanity check

MCP Resources (read-only state without a tool call): pyrecrawl://cache/stats ยท pyrecrawl://sessions ยท pyrecrawl://monitors

MCP Prompts (ready-made playbooks): research(topic) ยท rag_ingest(site) ยท watch_page(url)

Env flags

Variable

Default

Effect

PYRECRAWL_CACHE

off

1 = in-memory LRU (128 pages), or a directory path (reserved for disk mode)

PYRECRAWL_CACHE_TTL

900

Cache entry lifetime in seconds

PYRECRAWL_MONITOR_DIR

~/.pyrecrawl/monitors

Where monitor snapshots persist

PYRECRAWL_NO_TELEMETRY

off

1 = disable the anonymous startup ping (also honors DO_NOT_TRACK=1)

prefer options: "auto" (default ladder) ยท "fast" (HTTP only) ยท "stealth" (CF bypass) ยท "llm" (deep processing).


Related MCP server: mcp-services

๐Ÿš€ Install & Use (one-liner)

1. Install

UV (recommended โ€” one command, zero Python setup)

UV is a fast Python package manager that handles Python itself โ€” no need to install Python separately. Get it once:

# macOS / Linux
curl -LsSf https://astral.sh/uv/install.sh | sh

# Windows (PowerShell)
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

Learn more about UV โ†’

Then run PyreCrawl directly โ€” no venv, no pip install, no Python download:

uvx pyrecrawl@latest
uv tool install pyrecrawl

Or via pipx (alternative)

pipx install pyrecrawl

Or via pip into a venv

pip install pyrecrawl

2. One-time browser engines

pyrecrawl setup

This installs Chromium + stealth browser engines (~2 min, one-time).

3. Register with your AI agent

# Auto-detect installed agents and write their MCP configs
pyrecrawl install

# Or target specific agents
pyrecrawl install claude-desktop cursor

# Dry-run to preview what would change
pyrecrawl install --dry-run

Supported agents: claude-desktop, claude-code, cursor, vscode, codex, opencode, hermes.

4. Start chatting

After installing + registering, restart your agent (or start a new session). Then ask:

"Scrape https://example.com and summarize it."

Available tools:

Tool

What it does

scrape

Fetch a single URL โ†’ markdown (auto-escalates past Cloudflare)

extract

Scrape + structured extraction via CSS schema โ†’ JSON

map_site

Enumerate all internal URLs from a root

crawl

Multi-page crawl: discover + scrape in bulk

batch_scrape

Fetch many URLs in one parallel call

search

Web search via DuckDuckGo with anti-bot bypass

search_papers

Academic paper search (arXiv / Crossref)

deep_research

Search + scrape + citations in one call โ€” primary research tool

document

Extract text from PDF/DOCX/PPTX URLs

monitor

Track a URL for content changes over time

session

Persistent browser session for login walls

cache

Inspect or clear the response cache

health

Verify engine availability + version

Plus 3 guided prompts: research, rag_ingest, watch_page.

Quick examples

Ask your agent naturally โ€” no special syntax needed:

You say

Agent uses

"Scrape https://example.com and summarize it"

scrape โ†’ returns markdown โ†’ agent summarizes

"Research Rust memory safety vulnerabilities"

deep_research โ†’ search + scrape + citations

"Deep research on AI regulation worldwide"

deep_research(iterations=3) โ†’ multi-pass with refined queries

"Extract all product names and prices from this page"

extract โ†’ CSS schema โ†’ structured JSON

"Crawl https://docs.example.com and give me an overview"

crawl โ†’ multi-page โ†’ summary

"Monitor this page for price changes"

monitor โ†’ baseline snapshot โ†’ periodic diff

"Find papers about transformer attention"

search_papers โ†’ arXiv results

"What's the current cache hit rate?"

cache โ†’ stats


๐Ÿง  Skills โ€” Maximize Your Agent's Research Quality

PyreCrawl tools give your agent hands (scrape, crawl, search). But the agent still needs a brain โ€” instructions on when to use which tool, how to chain research passes, and what anti-hallucination rules to follow.

That's what PyreCrawl Skills provides.

MCP Tools (this repo)

Skills (pyrecrawl-skills)

Role

Execute web operations

Tell the agent how to use them

Analogy

Hands

Brain

Example

deep_research(query, iterations=3)

"Run 3 passes, check gaps after each, cite everything"

Required?

Yes (the engine)

Optional (but recommended for research quality)

Quick setup:

# 1. Install the tools (you already have this)
uvx pyrecrawl@latest

# 2. Add the research skill to your project
git clone https://github.com/SanggonBoy/pyrecrawl-skills.git /tmp/pyrecrawl-skills
cp /tmp/pyrecrawl-skills/pyrecrawl-research/SKILL.md ./CLAUDE.md  # or .cursorrules / AGENTS.md

Without skills: Your agent has powerful tools but improvises usage. With skills: Your agent follows a proven research protocol with anti-hallucination guardrails.


๐Ÿ“š Manual config (if pyrecrawl install doesn't match your setup)

Claude Desktop

Config file

  • Linux: ~/.config/Claude/claude_desktop_config.json

  • macOS: ~/Library/Application Support/Claude/claude_desktop_config.json

  • Windows: %AppData%\Claude\claude_desktop_config.json

{
  "mcpServers": {
    "pyrecrawl": {
      "command": "uvx",
      "args": ["--from", "pyrecrawl", "pyrecrawl", "serve"]
    }
  }
}

Claude Code

Config file: project-scoped .mcp.json

{
  "mcpServers": {
    "pyrecrawl": {
      "command": "uvx",
      "args": ["--from", "pyrecrawl", "pyrecrawl", "serve"]
    }
  }
}

Cursor

Config file: ~/.cursor/mcp.json

{
  "mcpServers": {
    "pyrecrawl": {
      "command": "uvx",
      "args": ["--from", "pyrecrawl", "pyrecrawl", "serve"]
    }
  }
}

VS Code / Copilot

Config file: .vscode/mcp.json (project-scoped)

{
  "servers": {
    "pyrecrawl": {
      "command": "uvx",
      "args": ["--from", "pyrecrawl", "pyrecrawl", "serve"],
      "type": "stdio"
    }
  }
}

Codex CLI

Config file: ~/.codex/config.toml

[mcp_servers.pyrecrawl]
command = "uvx"
args = ["--from", "pyrecrawl", "pyrecrawl", "serve"]

OpenCode

Config file: ~/.config/opencode/opencode.json

{
  "mcp": {
    "pyrecrawl": {
      "type": "local",
      "command": ["uvx", "--from", "pyrecrawl", "pyrecrawl", "serve"],
      "enabled": true
    }
  }
}

Hermes

Config file

  • Linux/macOS: ~/.hermes/config.yaml

  • Windows: %LocalAppData%\hermes\config.yaml

mcp_servers:
  pyrecrawl:
    command: uvx
    args:
      - --from
      - pyrecrawl
      - pyrecrawl
      - serve
    enabled: true

Windows note: uvx must be on PATH. If not, use the full path to uvx.exe (e.g. C:\Users\<you>\AppData\Local\hermes\bin\uvx.exe).


๐Ÿง  How the ladder chooses

PyreCrawl runs each request through three tiers, stopping at the first one that returns a complete, LLM-ready result:

Concern

Fast tier

Stealth tier

Deep tier

Static HTML page

โœ… ~200ms

โ€”

โ€”

Cloudflare-protected

โŒ

โœ… Turnstile solver

โ€”

JS-heavy SPA

โŒ

โœ… real Chromium

โ€”

Live DOM data (input .value, JS state)

โŒ

โœ… js param

โ€”

LLM-ready markdown + citations

โ€”

โ€”

โœ… BM25, fit-markdown

Structured extraction (CSS schema)

โ€”

โ€”

โœ…

Deep crawl (BFS/DFS/BestFirst)

โ€”

โ€”

โœ… adaptive

The agent never has to pick. prefer="auto" does it every call.

Live DOM data with js and wait_for

Some sites keep the data you want in a DOM property (e.g. an <input>'s .value) that JS writes after an XHR โ€” it never appears in the serialized HTML. The scrape tool accepts two stealth-tier params for exactly this:

{
  "url": "https://temp-mail.org/id",
  "prefer": "stealth",
  "wait_for": "document.getElementById('mail').value.includes('@')",
  "js": "document.getElementById('mail').value"
}
  • wait_for โ€” a JS predicate expression polled until truthy (bounded by timeout). Use it instead of guessing a sleep for anything that arrives asynchronously.

  • js โ€” a JS expression evaluated once the page settles; the value comes back in meta.js_result. Errors are captured in meta.js_error (the page result is still returned, never a crash).


๐Ÿ“Š Compared to Firecrawl (hosted)

Firecrawl

PyreCrawl

Cost

Free 1k/mo, then $16โ€“333/mo

Free, self-hosted

Local LLM support

โŒ

โœ… Ollama / any LLM

Cloudflare bypass

โœ… (Fire-Engine, paid)

โœ… (free, built-in)

Markdown + BM25

โœ…

โœ…

Self-host

โŒ

โœ…

Academic paper search

โŒ

โœ… arXiv + Crossref (search_papers)

Hosted search API

โœ… /search

โš ๏ธ DuckDuckGo HTML + arXiv/Crossref (no key)


๐Ÿ”ง Development

git clone https://github.com/SanggonBoy/PyreCrawl.git
cd PyreCrawl
uv venv --python 3.12 .venv
source .venv/Scripts/activate  # Windows; or .venv/bin/activate on macOS/Linux
uv pip install -e ".[dev]"
python -m playwright install chromium
scrapling install

Run tests

python scripts/selfcheck.py        # real-network smoke test (13 tools + engines)
python scripts/probe_stdio.py      # stdio JSON-RPC probe
python scripts/test_ladder_bug.py  # SPA-shell ladder escalation regression
python scripts/test_js_eval.py     # stealth js/wait_for params regression
python scripts/test_scope_selector.py  # crawl css_selector/max_depth wiring
python scripts/test_link_harvest.py    # map/BFS link purity regression

๐Ÿ“ฆ Publish

Maintainers only:

git tag vX.Y.Z
git push origin vX.Y.Z

GitHub Actions builds + uploads to PyPI via trusted publishing.


๐Ÿ”” Stay up to date

PyreCrawl checks PyPI on every startup and reports the latest version โ€” your MCP agent sees this automatically via the health() tool response and can notify you inline.

To check manually:

pyrecrawl version

To upgrade:

pyrecrawl update   # runs: uv tool upgrade pyrecrawl

Get notified of new releases: click Watch โ†’ Releases only at the GitHub repo to receive email notifications when a new version is published.


NOTE

PyreCrawl sendsone anonymous usage ping per 24 h at server startup โ€” see Privacy for exactly what's sent and how to opt out.

๐Ÿ”’ Privacy โ€” anonymous usage ping

PyreCrawl phones home once per 24 h with a tiny anonymous ping when the MCP server starts, so we can count real users (DAU/MAU) instead of raw downloads.

Sent (4 fields, ~100 bytes)

Never sent

Hashed machine id (SHA-256 of hostname+MAC โ€” not reversible)

Your IP (not stored)

PyreCrawl version

Any URL you scrape

Python version

Any page content or search queries

OS family (windows / linux / darwin)

Anything else

Client code: src/pyrecrawl/telemetry.py (~90 lines, stdlib only) ยท Collector: workers/telemetry/ โ€” a self-hostable Cloudflare Worker + D1, no third-party analytics service.

Opt out any time:

export PYRECRAWL_NO_TELEMETRY=1   # or the industry-standard DO_NOT_TRACK=1

๐Ÿ“œ Uninstall

# Remove from all agent configs
pyrecrawl uninstall

# Remove the package
uv tool uninstall pyrecrawl

๐Ÿ›ก๏ธ License

MIT โ€” see LICENSE.

Available Tools

13 tools
batch_scrapeA

Scrape MANY URLs in ONE call (parallel, deduped, cache-aware).

Use when the user provides multiple URLs or you have a list of pages to fetch. More efficient than calling scrape N times.

Args: urls: Target URLs (deduped automatically; empties dropped). prefer: "auto" | "fast" | "stealth" | "llm". timeout: per-URL timeout in seconds. max_concurrency: parallel workers (default 4). include_html: include raw HTML per result (large; off by default).

Returns {requested, unique, succeeded, failed, results[]}. Per-URL failures are isolated โ€” other URLs still succeed.

Returns: {requested, unique, succeeded, failed, results: [{url, markdown, ...}]}

ParametersJSON Schema
NameRequiredDescriptionDefault
urlsYes
preferNoauto
timeoutNo
include_htmlNo
max_concurrencyNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries full burden. It discloses key behaviors: parallel execution, automatic deduplication, cache-awareness, per-URL failure isolation, and the return structure. It does not mention rate limits, auth, or redirect handling, but the disclosed traits are sufficient for typical agent decisions. Minor gap is not explaining what 'cache-aware' implies operationally.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is efficiently written, front-loaded with the core purpose and usage guidance, then parameter details. It repeats the return structure twice (once in prose, once in a return block), which is slightly redundant but not harmful. Structure is logical and skimmable.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool has 5 parameters and no annotations, the description covers all needed context: when to use, how to set parameters, return format, failure isolation, and efficiency rationale. It even explains the deduping and default for include_html. Nothing critical is missing for an agent to invoke it correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate. It does so thoroughly: each parameter gets a meaningful explanation (e.g., 'urls: Target URLs (deduped automatically; empties dropped)', 'prefer: "auto" | "fast" | "stealth" | "llm"', 'timeout: per-URL timeout in seconds'). This adds semantics well beyond the schema's types and defaults.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states 'Scrape MANY URLs in ONE call' with explicit characteristics (parallel, deduped, cache-aware), clearly distinguishing it from the single-URL sibling 'scrape'. The verb-resource pairing is unmistakable and the emphasis on batch capability differentiates it from all sibling tools.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It explicitly states when to use it: 'Use when the user provides multiple URLs or you have a list of pages to fetch.' It also explains benefits over the alternative ('More efficient than calling `scrape` N times'), giving clear routing guidance without needing to inspect the schema.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

cacheA

Inspect the response cache: stats, clear, enable, or disable.

Use this to check cache hit rates before large batch jobs, or to clear stale cached responses when a site's content has changed.

Args: action: "stats" (default) | "clear" | "disable" | "enable".

ParametersJSON Schema
NameRequiredDescriptionDefault
actionNostats

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.3/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the behavioral burden. It does communicate the basic operation model and even hints at what clearing affects ('stale cached responses'). However, it does not disclose whether 'clear' wipes the entire cache, whether enable/disable persists, or whether any actions are destructive or require privileges.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is compact and front-loaded: a one-sentence purpose, two concrete use cases, then the parameter details. Every sentence earns its place and there is no redundant or filler content.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple one-parameter tool with an output schema, the description covers the action values and gives practical usage context. It is nearly complete, but slightly more detail on the side effects of clear/disable/enable would make it fully self-sufficient given the absence of annotations.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Despite 0% schema description coverage, the description fully documents the only parameter: action, its allowed values ('stats' | 'clear' | 'disable' | 'enable'), and its default ('stats'). This adds real meaning beyond the raw schema, which only shows a string with a default.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The first sentence states the tool's purpose with a specific verb and resource: 'Inspect the response cache' and then enumerates the exact operations supported (stats, clear, enable, disable). This unambiguously separates it from the sibling scraping and search tools, none of which touch the cache.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives clear, actionable use cases: check hit rates before large batch jobs, and clear stale responses after content changes. It does not explicitly mention alternatives or exclusions, but no alternative cache tool exists among the siblings, so the guidance is sufficient.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

crawlA

Multi-page crawl: discover URLs on root, then scrape each.

Use this when the user wants to crawl an entire site section or docs, or needs multiple pages scraped in bulk. For single pages use scrape; for research questions use deep_research.

Args: root: start URL. max_pages: hard cap on pages scraped. css_selector: scope each page's html/markdown to the matched element (non-llm: lxml re-scope of the fetched HTML; llm: native crawl4ai css_selector). prefer: "auto" | "fast" | "stealth" | "llm" (llm = Crawl4AI BFS deep-crawl). include_paths: regex โ€” keep only URLs matching (matched against full URL). exclude_paths: regex โ€” drop URLs matching (e.g. /tag/|/page/\d+). max_depth: 0 = flat harvest from the root page's links (default); >0 = true BFS up to that link depth, honoring the filters.

Returns: {root, pages: [{url, markdown, title, ...}], count, discovered, elapsed_ms} or {error, root} on failure.

ParametersJSON Schema
NameRequiredDescriptionDefault
rootYes
preferNoauto
max_depthNo
max_pagesNo
css_selectorNo
exclude_pathsNo
include_pathsNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.8/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full behavioral burden, and it delivers: it explains the discover-then-scrape flow, flat vs BFS depth behavior, filtering by regex, the `prefer` mode variants, and the exact success/error return shapes. This goes well beyond a minimal statement of what the tool does.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured and front-loaded: a crisp one-sentence summary, then usage guidance, then a tight Args block, then the Returns shape. Every line adds information without padding, which is appropriate for a 7-parameter tool with no schema-level parameter descriptions.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given no annotations, an output schema present, and seven parameters, the description is operationally self-sufficient. It covers purpose, when to use it, alternatives, all parameter semantics, the crawling algorithm variants, filters, and the return contract including the error case.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate for every parameter, and it does. All seven arguments are meaningfully explained, including the regex semantics for include/exclude paths, the meaning of max_depth, and the `prefer` option values.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific, action-oriented definition: 'Multi-page crawl: discover URLs on root, then scrape each.' It also distinguishes itself from siblings by naming `scrape` for single pages and `deep_research` for research questions.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description clearly says when to use crawl (entire site section, docs, bulk multi-page needs) and gives explicit alternatives for single pages and research questions. However, it does not distinguish crawl from the `batch_scrape` sibling, and its phrase 'multiple pages scraped in bulk' could overlap with that tool, so the routing guidance is not fully exhaustive.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

deep_researchA

Search the web, then pull the top sources as EVIDENCE (no LLM synthesis).

PRIMARY RESEARCH TOOL โ€” use when the user asks to research, investigate, deep-dive, fact-check, or learn about a topic. Returns a citations list with stable [n] numbers and an evidence list of per-source markdown โ€” the agent does the synthesis from evidence.

Multi-pass mode: set iterations=2-3 to auto-run additional searches with refined queries (alternatives, criticism, latest developments) and append deduplicated evidence. Each pass adds up to scrape_top new sources.

Args: query: search string. limit: how many search results to fetch. scrape_top: how many of those to actually fetch content from. prefer: "auto" | "fast" | "stealth" | "llm". iterations: 1 (default, single pass), 2-3 (multi-pass with refined queries targeting evidence gaps). Each pass searches from a different angle and deduplicates by URL.

Returns: {query, iterations_run, queries: [str], hits: [{url, title, snippet}], citations: [{url, title}], evidence: [{url, title, markdown}], scraped, used_engines, elapsed_ms} or {error, query, hint} on failure.

ParametersJSON Schema
NameRequiredDescriptionDefault
limitNo
queryYes
preferNoauto
iterationsNo
scrape_topNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.8/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and it delivers: it discloses that no LLM synthesis occurs, that multi-pass runs refine queries and deduplicate by URL, and that failures return a structured error object with a hint. This gives an agent a realistic model of what the tool will and will not do.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is organized with a purpose statement, usage trigger, mode explanation, per-argument semantics, and a complete return shape. While long, every section earns its place and the most important decision-relevant facts are front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has five parameters, multiple modes, and a rich return value, and the description covers all of them: arguments, multi-pass behavior, output shape, and error shape. Nothing an agent needs to invoke or interpret the result is missing, and the output schema is effectively described inline.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must document all five parameters and does: query, limit, scrape_top, prefer (with listed option values), and iterations (with concrete guidance on 1 vs 2-3). The iterations explanation adds behavior beyond the schema's bare integer type.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb and resource: 'Search the web, then pull the top sources as EVIDENCE (no LLM synthesis).' It clearly distinguishes this tool from generic search or scrape siblings by labeling it the 'PRIMARY RESEARCH TOOL' and explicitly stating the agent performs synthesis from evidence.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives explicit trigger conditions: 'use when the user asks to research, investigate, deep-dive, fact-check, or learn about a topic.' It also explains the multi-pass option for deeper research. However, it does not explicitly name sibling alternatives or state when not to use this tool, so it stops short of full exclusion guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

documentA

Extract text from a PDF/DOCX/PPTX URL โ†’ markdown (no browser).

Use when the user shares a link to a document (PDF, Word, PowerPoint) and wants its text content. Also useful after search_papers to get full text from a paper's pdf_url.

Content-type sniffed and routed to pypdf / python-docx / python-pptx. Optional deps โ€” install with pip install 'pyrecrawl[docs]'.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYes
timeoutNo
max_pagesNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the behavioral burden; it adds that extraction is non-browser, content type is sniffed and routed to pypdf/python-docx/python-pptx, and that optional dependencies may need installing. It could mention failure modes or whether any state changes occur, but the read-only extraction framing is clear.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Every sentence adds value and the most identifying information is front-loaded. The usage cases, routing detail, and dependency caveat are each one concise line, with no filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple extracted-text tool with an output schema, the description covers purpose, use case, supported formats, routing, and install needs. It is incomplete for the two optional parameters (`timeout`, `max_pages`), which are not documented in the schema or the description.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, so the description must explain parameters. It clarifies only `url` by restricting it to PDF/DOCX/PPTX document links; `timeout` and `max_pages` are left completely unexplained beyond their names and defaults.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The first line names an exact action (extract text), input family (PDF/DOCX/PPTX URL), output (markdown), and an explicit constraint (no browser). It also places the tool relative to `search_papers`, so an agent can distinguish document conversion from generic scraping siblings.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives concrete conditions: use when the user shares a PDF/Word/PowerPoint link and wants text, and after `search_papers` on a pdf_url. It does not explicitly name sibling tools to avoid or state hard exclusions, though 'no browser' implicitly rules out general web pages.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

extractA

Scrape + structured extraction using a CSS-based JSON schema.

Use when the user wants structured data (tables, lists, product info) extracted from a page. Define a CSS schema to target specific elements.

The schema is a JsonCssExtractionStrategy schema: { "name": "PageItems", "baseSelector": "div.item", "fields": [{"name": "title", "selector": "h2", "type": "text"}, ...] }

Returns parsed JSON in data.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYes
preferNoauto
schemaYes

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A3.9/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description must carry the burden. It discloses that this is a scraping/extraction operation and says it returns parsed JSON in `data`. But it omits practical behavioral details like dynamic-content handling, page-size limits, error behavior, and what the `prefer` option controls.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is compact and front-loaded. The first sentence captures the essence, and the rest provides a concrete example and return-shape note without wasted words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The schema example and return mention make the tool callable, but with no annotations and zero parameter descriptions, important usage context is missingโ€”especially around `prefer` and dynamic page behavior. It is adequate but not complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate. It does a good job for the `schema` parameter by providing a concrete JsonCssExtractionStrategy example, but `prefer` is not explained at all, and `url` is only meaningful by convention.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific operation ('Scrape + structured extraction') with a clear mechanism (CSS-based JSON schema) and names concrete use cases (tables, lists, product info). This distinguishes it from siblings like scrape or crawl, which are for less structured data collection.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It explicitly says when to use the tool: when the user wants structured data extracted from a page. However, it does not explicitly name alternatives or state when not to use it, so the contrast with sibling tools is mostly implicit.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

healthA

Verify PyreCrawl is working: engine versions, dependencies, update status.

Use this at the start of a session or before a large scraping job to confirm all engines are installed and up to date.

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.2/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It discloses the scope of the check (engine versions, dependencies, update status) and implies a read-only diagnostic operation. However, it does not detail any side effects (likely none), how it reports failures (e.g., exit codes, summary output), or whether it performs network calls. For a zero-parameter health check, this is adequate but not rich.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences that front-load the purpose and then immediately provide usage guidance. Every word earns its place; there is no fluff or redundant phrasing. It is concise and structured optimally for agent parsing.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a zero-parameter health check tool, the description explains what it verifies and when to run it. The output schema (not shown in the input but indicated as present) likely covers the exact return format. The description is sufficient for an agent to know why and when to call the tool, though it does not specify what constitutes 'working' beyond installed engines. This is a minor gap, but overall complete for the tool's simplicity.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool has zero parameters, so there is no schema coverage gap. The description does not need to elaborate on parameters. According to the rubric, a 0-parameter baseline is 4, and the description adds no irrelevant parameter details, so this score is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states what the tool does: 'Verify PyreCrawl is working: engine versions, dependencies, update status.' It uses a specific verb and resource, and the purpose is distinct from sibling tools that perform scraping tasks. An agent can immediately understand this is a pre-flight health check, not a scraping operation.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly says when to use it: 'at the start of a session or before a large scraping job to confirm all engines are installed and up to date.' This gives concrete context. It does not explicitly contrast with alternatives (e.g., 'use rather than scrape when...'), but the purpose and usage are clear enough that an agent would be unlikely to confuse it with a scraping tool.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

map_siteA

Enumerate all internal URLs reachable from root.

Use when the user wants to map a site's structure or find all pages before crawling. Often paired with crawl or batch_scrape.

Args: root: Website root (e.g. "https://example.com/docs"). include_pattern: Optional regex; only URLs matching are returned. limit: Hard cap on returned URLs.

ParametersJSON Schema
NameRequiredDescriptionDefault
rootYes
limitNo
include_patternNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.2/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full behavioral disclosure burden. It does add useful constraints โ€” 'internal URLs' scoping, 'reachable from root' traversal semantics, and the 'hard cap' behavior of `limit` โ€” but leaves significant traits undisclosed: auth requirements, rate limits, redirect handling, and whether the tool makes many network requests. Meaningful but incomplete for an unannotated tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is tightly structured โ€” core purpose in the first sentence, usage guidance in the second, then a clean Args block. Every sentence earns its place, and the most decision-relevant information (what it returns, internal scope) is front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 3-parameter tool with an output schema, the description covers the essentials: purpose, when to use it, related tools, and all parameters with semantics. Minor gaps remain โ€” how traversal handles redirects, depth limits, and protocol variants โ€” but an agent has enough to invoke the tool correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate, and it does: `root` gets a format example, `include_pattern` gets its regex filter semantics explained, and `limit` is clarified as a 'Hard cap on returned URLs'. All three parameters receive meaning beyond the bare schema types, fully covering the parameters that the schema left undocumented.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The first sentence is a specific verb+resource statement โ€” 'Enumerate all internal URLs reachable from `root`' โ€” which precisely defines the scope and distinguishes this from siblings like `crawl` (which fetches content) and `search` (which queries). The pairing note with `crawl`/`batch_scrape` further clarifies its role as a pre-scrape discovery step.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

'Use when the user wants to map a site's structure or find all pages before crawling' gives clear, actionable context, and naming `crawl` and `batch_scrape` as common companions helps an agent understand the workflow. However, it stops short of explicit exclusions โ€” it never states 'use crawl instead when you need page content only,' so the when-not guidance is implied rather than stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

monitorA

Track a URL over time and report meaningful content changes.

Use when the user wants to watch a page for updates (price changes, new blog posts, status updates). Ask me to 'set up monitoring for ' for a guided setup playbook.

Args: url: target URL. action: "check" | "history" | "forget". prefer: ladder preference, same as scrape. css_selector: scope the diff to one element (so banner / nav changes don't trigger false positives).

Snapshots persist under PYRECRAWL_MONITOR_DIR (default ~/.pyrecrawl/monitors/). check returns status of new | unchanged | changed | error and a unified diff when the page changed.

Returns: {url, status: "new"|"unchanged"|"changed"|"error", diff?: str, snapshot_chars?: int, elapsed_ms?: int}

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYes
actionNocheck
preferNoauto
css_selectorNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the burden, and it delivers by disclosing that snapshots persist under PYRECRAWL_MONITOR_DIR, that check returns status values including error, and that a unified diff is produced. It could go further by explaining the side effects of 'forget' and retention behavior, but the core stateful behavior is well disclosed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured with a purpose statement, explicit usage trigger, Args block, and Returns block. It is longer than minimal but each section earns its place; the guided-setup sentence adds a useful interaction hint rather than filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a stateful tool with three actions, the description details only 'check' behavior. 'history' and 'forget' are listed but their effects, return shapes, and side effects are unexplained. Since there are no annotations and schema coverage is 0%, this is a meaningful gap for an agent deciding how to invoke the tool correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate. It explains url, enumerates action values, gives css_selector its diff-scoping purpose, and references the 'prefer' ladder via scrape. The main gap is that 'history' and 'forget' semantics are not expanded, and 'prefer' relies on knowing what scrape does.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb and resource: 'Track a URL over time and report meaningful content changes.' This clearly distinguishes monitor from one-off tools like scrape and crawl, and the concrete use cases (price changes, blog posts, status updates) reinforce the purpose.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It explicitly says 'Use when the user wants to watch a page for updates' and provides a guided setup cue. However, it does not explicitly say when not to use this tool or name alternatives like scrape for one-off fetches, so it misses the exclusion aspect.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

scrapeA

Scrape a single URL โ†’ LLM-ready markdown.

Use this when the user shares a URL and wants its content (read, analyze, summarize, extract). Auto-escalates through fastโ†’stealthโ†’llm when blocked.

Args: url: Target URL (http/https). prefer: "auto" | "fast" | "stealth" | "llm". auto = fast first, escalate to stealth on block/short page. fast = cheap HTTP only (no JS). stealth = real Chromium + Cloudflare solver. llm = full Crawl4AI browser + BM25 fit-markdown. timeout: per-attempt timeout in seconds. include_html: include raw HTML in the response (large; off by default). js: (stealth only) JS expression evaluated against the live page after it settles. The value comes back in meta.js_result. Use for data that lives in DOM properties (e.g. an input's .value) rather than in serialized HTML. wait_for: (stealth only) JS predicate expression polled until truthy (bounded by timeout). Use to wait for content that arrives asynchronously after network_idle.

Returns: {url, final_url, status, markdown, title, method, elapsed_ms, meta} or {error, url, method} on failure.

ParametersJSON Schema
NameRequiredDescriptionDefault
jsNo
urlYes
preferNoauto
timeoutNo
wait_forNo
include_htmlNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.6/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are present, so the description carries the full behavioral burden and succeeds. It discloses the auto-escalation path ('fastโ†’stealthโ†’llm when blocked'), explains what each mode does (cheap HTTP, real Chromium + Cloudflare solver, full Crawl4AI browser), and scopes js and wait_for to stealth only. It also documents the success and failure return shapes.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is tightly structured: a one-line purpose, a single usage sentence, a compact Args block, and a Returns block. Every line is information-dense with no filler, and the most important scoping details are front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

This is a complex tool with 6 parameters, 4 execution modes, JS evaluation, and async waiting. The description accounts for every parameter, the escalation behavior, return values, and failure output. There are no obvious omissions that would prevent an agent from invoking it correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has no descriptions and only bare titles/defaults, so the description must compensate. It does: url is constrained to http/https, prefer enumerates all four modes with escalation semantics, timeout is defined as per-attempt seconds, include_html warns about response size, and js/wait_for get exact behavioral definitions with an example for js.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb-resource statement: 'Scrape a single URL' and 'LLM-ready markdown'. It names a clear trigger ('when the user shares a URL and wants its content'), and 'single URL' separates it from multi-URL siblings like crawl and batch_scrape. However, it does not explain how it differs from sibling extract, so differentiation is incomplete.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives a concrete usage context: 'Use this when the user shares a URL and wants its content (read, analyze, summarize, extract)'. That is clear context, but it does not list exclusions or point to alternative tools such as extract or crawl, leaving the when-not-to-use judgment to the agent.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

search_papersA

Search academic papers via arXiv or Crossref โ€” no API keys.

Args: query: free-text search, e.g. "transformer attention scaling laws". limit: max results (1-25 arXiv / 1-20 crossref). source: "arxiv" (CS/physics/math preprints, default) or "crossref" (all fields, DOI-backed). category: optional arXiv category filter, e.g. "cs.LG", "cs.CV".

Returns papers with id/url/pdf_url/title/authors/summary/published. Feed pdf_url into the document tool to extract full text.

Returns: {query, source, papers: [{title, authors, abstract, url, pdf_url?, ...}]} or {error, query} on failure.

ParametersJSON Schema
NameRequiredDescriptionDefault
limitNo
queryYes
sourceNoarxiv
categoryNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.6/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden. It discloses that no API keys are needed, specifies per-source result limits (1-25 arXiv / 1-20 Crossref), describes the return fields (id/url/pdf_url/title/authors/summary/published), and mentions the error return format. It also notes that pdf_url is optional (indicated by the '?' in the return example). This is strong, though it could explicitly state the operation is read-only and has no side effects.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured with a heading, a bulleted Args list, and a Returns section. Every sentence carries useful information: examples, default values, and a downstream workflow hint. It front-loads the purpose and keeps the details organized, with no filler or repetition.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description is complete for a 4-parameter tool. It covers all parameters, provides return format details, notes the error case, and suggests an integration with the 'document' tool. Given the existing output schema (not shown but present), the description does not need to repeat return values, and it supplies enough for an agent to call it correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description fully compensates. It explains each parameter: query with an example ('transformer attention scaling laws'), limit with per-source ranges, source with defaults and meanings ('arxiv' for preprints, 'crossref' for DOI-backed), and category with an example format ('cs.LG'). This adds significant meaning beyond the bare schema, making parameters self-explanatory.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: 'Search academic papers via arXiv or Crossref โ€” no API keys.' This is a specific verb (search) and resource (academic papers) with two named sources, and it naturally distinguishes from siblings like the general 'search' tool or 'document' for extraction. The mention of specific sources and the absence of API keys adds clarity.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear context for when to use this tool: when searching academic papers via arXiv or Crossref. It also gives a workflow hint by suggesting to feed the returned pdf_url into the 'document' tool for full-text extraction. However, it does not explicitly state when not to use it or compare with alternatives like 'search' or 'deep_research', so it falls slightly short of full guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

sessionA

Drive a persistent browser session โ€” cookies & JS state kept across calls.

Use for login walls and multi-step flows the one-shot ladder can't handle (e.g. user needs to log in first, then scrape a protected page). For simple pages use scrape; for research use deep_research.

One-shot scrape has no session memory; here each action runs against the same live page.

Args: session: named session; reuse the same name to keep state. action: "open" (url) โ€” navigate, returns url/title/status "click" (selector) โ€” click an element "fill" (selector, text) โ€” type into an input "type" (key) โ€” press a key, e.g. "Enter" "eval" (js) โ€” run a JS expression, returns value "wait" (selector?) โ€” wait for selector or sleep timeout_ms "content" โ€” url/title/visible text of current page "screenshot" (full_page?) โ€” returns png_base64 "cookies" โ€” list session cookies "close" โ€” destroy the session "list" โ€” show live sessions Returns {session, action, ...result, elapsed_ms}; errors as {error}.

ParametersJSON Schema
NameRequiredDescriptionDefault
jsNo
keyNo
urlNo
textNo
actionNoopen
sessionNodefault
headlessNo
selectorNo
full_pageNo
timeout_msNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden and meets it: it enumerates every action (open, click, fill, type, eval, wait, content, screenshot, cookies, close, list) with their effects, explicitly states 'close โ€” destroy the session' as destructive, and specifies the return format ('Returns {session, action, ...result, elapsed_ms}; errors as {error}'). It also discloses persistence of cookies and JS state.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured: it opens with purpose, then usage guidelines, then the stateful vs one-shot distinction, followed by a concise action-by-action list and return format. Every sentence adds value; the length is justified by the tool's complexity but remains scannable and front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with 10 parameters and 12 actions, this description is extraordinarily complete: it covers purpose, invocation conditions, all actions with their inputs and outputs, return and error formats, and distinguishes from siblings. An output schema exists, so return values don't need further explanation. Nothing essential for correct use is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema has 0% description coverage, so the description must compensate. It does so well by detailing parameters under 'Args:' including session, and for each action the relevant parameters (url, selector, text, key, js, full_page, timeout_ms). However, the `headless` parameter is never mentioned in the description, leaving its purpose undocumentedโ€”a minor gap given the otherwise thorough coverage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific purpose: 'Drive a persistent browser session โ€” cookies & JS state kept across calls.' This clearly identifies the tool as a session manager and differentiates it from one-shot tools like scrape and deep_research, which are explicitly named as alternatives.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides explicit usage guidance: 'Use for login walls and multi-step flows the one-shot ladder can't handle' and tells exactly when to use alternatives: 'For simple pages use `scrape`; for research use `deep_research`.' Also clarifies the key distinction: 'One-shot scrape has no session memory; here each action runs against the same live page.' This leaves no ambiguity about when to invoke this tool.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 1 tool updatev0.8.0
    • Changeddeep_research1 field changed
      • addedInput schema / properties / iterations
        Added value: +{
        +  "default": 1,
        +  "title": "Iterations",
        +  "type": "integer"
        +}
  2. 13 tool updatesv0.7.1
    • First observedbatch_scrape
    • First observedcache
    • First observedcrawl
    • First observeddeep_research
    • First observeddocument
    • First observedextract
    • First observedhealth
    • First observedmap_site
    • First observedmonitor
    • First observedscrape
    • First observedsearch
    • First observedsearch_papers
    • First observedsession

TDQS

A4.4/5.0

Scored across 13 tools

Disambiguation5/5

Each tool has a clearly distinct trigger: single-page scrape, structured extraction, bulk scrape, site crawl, document parsing, web search, deep research, academic search, monitoring, sessions, and cache/health maintenance. Even though several tools fetch pages, their arguments and return shapes make the intended use obvious.

Naming Consistency4/5

Names are all lowercase and action-oriented, but the convention is mixed: single-word verbs (scrape, crawl, search), verb_noun compounds (map_site, search_papers), and modifier compounds (batch_scrape, deep_research), plus noun-style tools like health and session. This is readable and mostly predictable, but not a uniform pattern.

Tool Count5/5

13 tools is well within the ideal range for a scraping and research suite. Each tool covers a distinct operation, and the utility tools (cache, health) support the workflow without feeling like filler.

Completeness5/5

The surface covers the full scraping/research workflow: single and batch scraping, crawling, structured extraction, document parsing, search, deep research, academic papers, monitoring, sessions, and cache/health management. There are no obvious dead ends โ€” monitors can be forgotten, sessions can be closed, and cache can be cleared or disabled.

Maintenance

ActivityMaintained
ResponsivenessNo issues

Related MCP Connectors

  • 65+ AI tools as MCP: research, write, code, scrape, translate, RAG, agent memory, workflows

  • Your agent needs the open web โ€” searched by more than one engine, and read as clean markdown rather than raw HTML. **What you can ask for** โ€ข "Search this question with two providers and tell me where they disagree." โ€ข "Scrape these 40 URLs into markdown, in one batch." โ€ข "Crawl this documentation site and give me every page." โ€ข "Do deep research on this topic and cite the sources." โ€ข "Find the academic papers behind this claim." **How to use it** Point any MCP client at https://mcp.aisa.one/search/mcp and sign in with OAuth โ€” there is no key to create or paste. 30 tools across several independent providers: Tavily and Exa search, answers, contents and agent runs; Firecrawl scrape, batch scrape, crawl, map and search; Perplexity Sonar, Sonar Pro, reasoning and deep research; Oxylabs AI search and LLM jobs; OpenAI and Anthropic web search; and scholarly search. **Why this rather than the source** Several independent indexes behind one account, because one engine's blind spot is not visible from inside it. **It is also a door to the rest** The same login reaches 26 sources and 580+ operations. Find the page here, then ask the same agent who links to it or how much traffic it gets โ€” without adding a second server. **What it costs** Finding and inspecting an operation is free. Running one is billed per call at API prices, with no seat and no monthly minimum, and every call takes max_price_usd so an agent cannot overspend by accident. **Where else it reaches** https://mcp.aisa.one/seo-serp/mcp for the Google results page itself, https://mcp.aisa.one/seo-serp-other-engines/mcp for Bing, Baidu and Naver.

  • Scrape, crawl and search the web for AI agents via MCP.

  • Screenshot, diff, audit and sitemap-capture any web page โ€” 5 MCP tools for AI agents.

Related MCP Servers

  • A
    license
    Not graded
    quality
    D
    maintenance
    Web scraping MCP server for Al agents. 6 tools: extract clean text/markdown from any URL, structured scraping with CSS selectors, full-page screenshots via Playwright, link extraction with regex filtering, metadata extraction (OG tags, Twitter cards), and Google search. Free tier: 50 requests/IP/day.
    8
    MIT
  • A
    license
    Not graded
    quality
    D
    maintenance
    Multi-tool MCP server for AI agents with 29 tools across web scraping, SEO analysis, screenshot and PDF generation, domain intelligence, content extraction, multi-chain EVM blockchain queries, and security toolkit. Free tier available with no auth required.
    12 npm
    1
    MIT
  • A
    license
    A
    quality
    A
    maintenance
    Web scraping, crawling, and structured data extraction for AI agents. 5 tools: scrape (clean markdown from any URL), crawl (entire sites), map (discover URLs), extract (structured JSON), and search. 833ms avg latency, single binary, self-hostable.
    8
    1,057
    AGPL 3.0