search_and_crawl
Search the web and crawl top results to extract full content, with optional Markdown and index persistence for later analysis.
Instructions
Search the web (engine as in search_web) and crawl top results. Combines search with full content extraction. Supply output_path (directory) in the request to persist per-page markdown (unsliced) + index.json and receive a slim response.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| request | Yes | Dict with: search_query (required), crawl_top_results, search_genre (optional, one of: pdf, documents, presentations, spreadsheets, japanese, english; adds a filetype/language operator to the query, other values return an error), engine (optional, one of: auto, google_custom, brave, exa, searxng; default from SEARCH_ENGINE, auto tries the configured engines in SEARCH_ENGINE_ORDER and moves to the next one on errors other than zero results), result_type (optional; default web; one of web, discussions, faq, videos, news, or several as a list or a comma-separated string; everything except web runs on brave only; news is a separate Brave News Search request; results holds web when requested, else the first type, and the other types go to extra_results; discussions are forum posts with the forum name, answer count, post score, question and top comment; faq are question and answer excerpts; Brave returns these sections for some queries only, and num_results only shortens them), recent_days, generate_markdown, max_content_per_page. Optional persistence keys: output_path (absolute directory — per-page .md files + index.json, the full page bodies are written BEFORE max_content_per_page truncation; dot-containing dir names are fine), include_content_in_response (bool, default False — when True keeps crawled_pages in the response too, still subject to max_content_per_page truncation), overwrite (bool, default False). Failed pages appear in index.json with file=null. |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
No arguments | |||