get_webpage_content
Extract webpage content in markdown format for analysis or processing. Specify URL, optional wait time, and country to retrieve clean text from web pages.
Instructions
Retrieve content of a webpage in markdown
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| country | No | Residential country to load the request from (e.g., US, CA, GB). Optional. | |
| url_to_scrape | Yes | The URL of the webpage to scrape. | |
| wait_before_scraping | No | Time to wait in milliseconds before starting the scrape. |
Implementation Reference
- src/tools/getWebpageMarkdown.ts:21-81 (handler)The core handler function that implements the tool logic: sends POST request to Olostep API to scrape the webpage and convert to markdown.handler: async ({ url_to_scrape, wait_before_scraping, country }: { url_to_scrape: string; wait_before_scraping: number; country?: string }, apiKey: string, orbitKey?: string) => { try { const headers = new Headers({ 'Content-Type': 'application/json', 'Authorization': `Bearer ${apiKey}` }); const payload = { url_to_scrape: url_to_scrape, wait_before_scraping: wait_before_scraping, formats: ["markdown"], ...(country && { country: country }), ...(orbitKey && { force_connection_id: orbitKey }) }; const response = await fetch(OLOSTEP_SCRAPE_API_URL, { method: 'POST', headers: headers, body: JSON.stringify(payload) }); if (!response.ok) { const errorDetails = await response.json(); return { isError: true, content: [{ type: "text", text: `Olostep API Error: ${response.status} ${response.statusText}. Details: ${JSON.stringify(errorDetails)}` }] }; } const data = await response.json() as OlostepScrapeApiResponse; if (data.result?.markdown_content) { return { content: [{ type: "text", text: data.result.markdown_content }] }; } else { return { isError: true, content: [{ type: "text", text: "Error: No markdown content found in Olostep API response." }] }; } } catch (error: unknown) { return { isError: true, content: [{ type: "text", text: `Error: Failed to scrape webpage. ${error instanceof Error ? error.message : String(error)}` }] }; } }
- Input schema using Zod validators for tool parameters: url_to_scrape (required URL), wait_before_scraping (optional ms delay), country (optional).schema: { url_to_scrape: z.string().url().describe("The URL of the webpage to scrape."), wait_before_scraping: z.number().int().min(0).default(0).describe("Time to wait in milliseconds before starting the scrape."), country: z.string().optional().describe("Residential country to load the request from (e.g., US, CA, GB). Optional."), },
- src/index.ts:26-37 (registration)Tool registration with MCP server: uses name, description, schema from module; wrapper calls the handler with API keys.server.tool( getWebpageMarkdown.name, getWebpageMarkdown.description, getWebpageMarkdown.schema, async (params) => { const result = await getWebpageMarkdown.handler(params, OLOSTEP_API_KEY, ORBIT_KEY); return { ...result, content: result.content.map(item => ({ ...item, type: item.type as "text" })) }; } );