Scrape a single URL → LLM-ready markdown.

Args: url: Target URL (http/https/file/raw:). prefer: "auto" | "fast" | "stealth" | "llm". auto = fast first, escalate to stealth on block/short page. fast = cheap HTTP only (no JS). stealth = real Chromium + Cloudflare solver. llm = full Crawl4AI browser + BM25 fit-markdown. timeout: per-attempt timeout in seconds. include_html: include raw HTML in the response (large; off by default). js: (stealth only) JS expression evaluated against the live page after it settles. The value comes back in meta.js_result. Use for data that lives in DOM properties (e.g. an input's .value) rather than in serialized HTML. wait_for: (stealth only) JS predicate expression polled until truthy (bounded by timeout). Use to wait for content that arrives asynchronously after network_idle.

Scrape + structured extraction using a CSS-based JSON schema.

The schema is a JsonCssExtractionStrategy schema: { "name": "PageItems", "baseSelector": "div.item", "fields": [{"name": "title", "selector": "h2", "type": "text"}, ...] }

Returns parsed JSON in data.

Enumerate all internal URLs reachable from root.

Args: root: Website root (e.g. "https://example.com/docs"). include_pattern: Optional regex; only URLs matching are returned. limit: Hard cap on returned URLs.

Multi-page crawl: discover URLs on root, then scrape each.

Args: root: start URL. max_pages: hard cap on pages scraped. css_selector: reserved for the llm tier. prefer: "auto" | "fast" | "stealth" | "llm" (llm = Crawl4AI BFS deep-crawl). include_paths: regex — keep only URLs matching (matched against full URL). exclude_paths: regex — drop URLs matching (e.g. /tag/|/page/\d+). max_depth: 0 = flat harvest from the root page's links (default); >0 = true BFS up to that link depth, honoring the filters.

Extract text from a PDF/DOCX/PPTX URL → markdown (no browser).

Content-type sniffed and routed to pypdf / python-docx / python-pptx. Optional deps — install with pip install 'pyrecrawl[docs]'.

Web search via DuckDuckGo HTML (no API key required).

Returns [{url, title, snippet}, ...]. The smart ladder bypasses DDG's bot detection if needed.

Scrape MANY URLs in ONE call (parallel, deduped, cache-aware).

Args: urls: Target URLs (deduped automatically; empties dropped). prefer: "auto" | "fast" | "stealth" | "llm". timeout: per-URL timeout in seconds. max_concurrency: parallel workers (default 4). include_html: include raw HTML per result (large; off by default).

Returns {requested, unique, succeeded, failed, results[]}. Per-URL failures are isolated — other URLs still succeed.

Search the web, then pull the top sources as EVIDENCE (no LLM synthesis).

Returns a citations list with stable [n] numbers and an evidence list of per-source markdown — the agent does the synthesis. Designed for research, RAG prep, and fact-checking.

Args: query: search string. limit: how many search results to fetch. scrape_top: how many of those to actually fetch content from. prefer: "auto" | "fast" | "stealth" | "llm".

Track a URL over time and report meaningful content changes.

Args: url: target URL. action: "check" | "history" | "forget". prefer: ladder preference, same as scrape. css_selector: scope the diff to one element (so banner / nav changes don't trigger false positives).

Snapshots persist under PYRECRAWL_MONITOR_DIR (default ~/.pyrecrawl/monitors/). check returns status of new | unchanged | changed | error and a unified diff when the page changed.

Search academic papers via arXiv or Crossref — no API keys.

Args: query: free-text search, e.g. "transformer attention scaling laws". limit: max results (1-25 arXiv / 1-20 crossref). source: "arxiv" (CS/physics/math preprints, default) or "crossref" (all fields, DOI-backed). category: optional arXiv category filter, e.g. "cs.LG", "cs.CV".

Returns papers with id/url/pdf_url/title/authors/summary/published. Feed pdf_url into the document tool to extract full text.

Inspect the response cache: stats or clear.

Args: action: "stats" | "clear" | "disable" | "enable".

Sanity check: verify engines are importable + return versions.

Drive a persistent browser session — cookies & JS state kept across calls.

Use for login walls and multi-step flows the ladder can't handle (one-shot scrape has no session memory; here each action runs against the same live page).

Args: session: named session; reuse the same name to keep state. action: "open" (url) — navigate, returns url/title/status "click" (selector) — click an element "fill" (selector, text) — type into an input "type" (key) — press a key, e.g. "Enter" "eval" (js) — run a JS expression, returns value "wait" (selector?) — wait for selector or sleep timeout_ms "content" — url/title/visible text of current page "screenshot" (full_page?) — returns png_base64 "cookies" — list session cookies "close" — destroy the session "list" — show live sessions Returns {session, action, ...result, elapsed_ms}; errors as {error}.