scrape_as_markdown
Extract webpage content into Markdown format, bypassing bot detection and CAPTCHA protections for reliable data collection.
Instructions
Scrape a single webpage URL with advanced options for content extraction and get back the results in MarkDown language. This tool can unlock any webpage even if it uses bot detection or CAPTCHA.
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes |
Implementation Reference
- server.js:161-183 (registration)Registration of the 'scrape_as_markdown' tool using server.addTool(), including name, description, schema, and execute handler.
addTool({ name: 'scrape_as_markdown', description: 'Scrape a single webpage URL with advanced options for ' +'content extraction and get back the results in MarkDown language. ' +'This tool can unlock any webpage even if it uses bot detection or ' +'CAPTCHA.', parameters: z.object({url: z.string().url()}), execute: tool_fn('scrape_as_markdown', async({url})=>{ let response = await axios({ url: 'https://api.brightdata.com/request', method: 'POST', data: { url, zone: unlocker_zone, format: 'raw', data_format: 'markdown', }, headers: api_headers(), responseType: 'text', }); return response.data; }), }); - server.js:168-182 (handler)The core handler logic wrapped in tool_fn. Sends a POST request to BrightData API's /request endpoint with the URL, unlocker zone, raw format, and markdown data_format to scrape and return markdown content.
execute: tool_fn('scrape_as_markdown', async({url})=>{ let response = await axios({ url: 'https://api.brightdata.com/request', method: 'POST', data: { url, zone: unlocker_zone, format: 'raw', data_format: 'markdown', }, headers: api_headers(), responseType: 'text', }); return response.data; }), - server.js:167-167 (schema)Zod schema defining the input parameter: a required URL string.
parameters: z.object({url: z.string().url()}),