smartcrawler_initiate
Initiate multi-page web crawling to extract structured data or convert pages to markdown. Choose between AI Extraction Mode for prompt-based data extraction or Markdown Conversion Mode for clean markdown output. Control crawl depth, page limits, and domain restrictions.
Instructions
Initiate a SmartCrawler request for intelligent multi-page web crawling.
SmartCrawler supports two modes:
- AI Extraction Mode (10 credits per page): Extracts structured data based on your prompt
- Markdown Conversion Mode (2 credits per page): Converts pages to clean markdown
Args:
url: Starting URL to crawl
prompt: AI prompt for data extraction (required for AI mode)
extraction_mode: "ai" for AI extraction or "markdown" for markdown conversion (default: "ai")
depth: Maximum link traversal depth (optional)
max_pages: Maximum number of pages to crawl (optional)
same_domain_only: Whether to crawl only within the same domain (optional)
Returns:
Dictionary containing the request ID for async processing
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| depth | No | ||
| extraction_mode | No | ai | |
| max_pages | No | ||
| prompt | No | ||
| same_domain_only | No | ||
| url | Yes |
Input Schema (JSON Schema)
{
"properties": {
"depth": {
"default": null,
"title": "Depth",
"type": "integer"
},
"extraction_mode": {
"default": "ai",
"title": "Extraction Mode",
"type": "string"
},
"max_pages": {
"default": null,
"title": "Max Pages",
"type": "integer"
},
"prompt": {
"default": null,
"title": "Prompt",
"type": "string"
},
"same_domain_only": {
"default": null,
"title": "Same Domain Only",
"type": "boolean"
},
"url": {
"title": "Url",
"type": "string"
}
},
"required": [
"url"
],
"title": "smartcrawler_initiateArguments",
"type": "object"
}