web_scrape
Fetch a public web page and return its content as clean markdown (default), with optional html, raw_html, links, and screenshot. Strips nav/footer/sidebar by default so you get the main article body.
It reads pages that are not otherwise locally accessible — docs, blog posts, marketing pages, articles. The response is page metadata (title, description, language) plus the requested formats. If the page served a bot-challenge interstitial ("verify you are human") instead of real content, the response carries challenge_detected: true — that content is not the page, and an immediate retry usually returns the same challenge.
Screenshot format (formats: ["screenshot"]) uses the same three-tier answer as browser: small enough (auto-downscaled) → inline data:image/...;base64,...; too large with no storage → a short-TTL scratch_url you don't have to clean up; storage: "/path.jpg" set → written permanently to project files, returned as { storage_path, size_bytes }.
Example:
{ "project_id": "my-saas", "url": "https://somewhere.cloud/docs/getting-started.txt" }
// Returns: { "url": "...", "title": "Start an app", "markdown": "# Start an app\n...", "status_code": 200 }{ "project_id": "my-saas", "url": "https://example.com", "formats": ["markdown", "links"], "only_main": false }{ "project_id": "my-saas", "url": "https://example.com", "formats": ["screenshot"], "storage": "/scratch/hero.jpg" }Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"project_id": "<project_id>",
"url": "<url>"
},
"name": "web_scrape"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | http(s) URL to scrape, max 2048 chars. | |
| formats | No | Optional array (or JSON-stringified array) of formats to return. Allowed: markdown, html, rawHtml, links, screenshot. Default: ["markdown"]. Example: '["markdown","links"]'. | |
| storage | No | Only meaningful with formats:["screenshot"]. A project files path like "/scratch/hero.jpg" to write the shot permanently instead of the default inline/ephemeral behavior. Returns { storage_path, size_bytes } instead of the image bytes. | |
| wait_for | No | Optional milliseconds to wait for the page to settle before scraping (max 30000). For SPAs that hydrate after load. | |
| only_main | No | Strip nav/footer/sidebar to keep only the main article content. Default true. | |
| project_id | Yes | Project ID (UUID), subdomain, or slug — resolved server-side, no UUID lookup needed. 'default' also works when the account has exactly one project; multi-project accounts must name one (the error lists them). |