Crawl Website
metalift_crawlCrawl a website from a seed URL and return markdown for every discovered page, enabling site mapping and LLM-ready content extraction.
Instructions
Crawl a site from a seed URL and return markdown for discovered pages (1+ credits per page). Default wait=true blocks with progress; wait=false returns job id for metalift_job_status.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Seed URL to start crawling from | |
| wait | No | Wait for crawl to finish with progress updates (default true). | |
| limit | No | Maximum pages to crawl | |
| max_depth | No | Maximum link depth from seed URL | |
| exclude_paths | No | Skip paths matching these prefixes | |
| include_paths | No | Only crawl paths matching these prefixes |