ai_crawler
Read-only
Crawl a website from a starting URL to extract the information you specify. Returns data in markdown, JSON, CSV, or Toon format, with optional JSON schema and JavaScript rendering for structured results.
Instructions
Tool useful for crawling a website from starting url and returning data in a specified format.
Schema is required only if output_format is json, csv or toon. 'render_javascript' is used to render javascript heavy websites. 'return_sources_limit' is used to limit the number of sources to return, for example if you expect results from single source, you can set it to 1.
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The URL from which crawling will be started. | |
| schema | No | The JSON schema to use for structured data extraction from the crawled pages. Only required if output_format is json, csv or toon. | |
| user_prompt | Yes | What information user wants to extract from the domain. | |
| geo_location | No | Two letter ISO country code to use for the crawl proxy. | |
| output_format | No | The format of the output. If json, csv or toon, the schema is required. Markdown returns full text of the page. CSV returns data in CSV format. Toon(Token-Oriented Object Notation) returns data in Toon format, which is optimized for AI agents. | markdown |
| render_javascript | No | Whether to render the HTML of the page using javascript. Much slower, therefore use it only for websites that require javascript to render the page. Unless user asks to use it, first try to crawl the page without it. If results are unsatisfactory, try to use it. | |
| return_sources_limit | No | The maximum number of sources to return. |
Output Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| result | Yes |