ScrapeGraph MCP Server

Instructions

Implementation Reference

• src/scrapegraph_mcp/server.py:1850-1937 (handler)

  MCP tool handler for the 'scrape' tool. Fetches raw page content via ScapeGraph API, handling API key retrieval and error wrapping.
  @mcp.tool(annotations={"readOnlyHint": True, "destructiveHint": False, "idempotentHint": True}) def scrape( website_url: str, ctx: Context, render_heavy_js: Optional[bool] = None ) -> Dict[str, Any]: """ Fetch raw page content from any URL with optional JavaScript rendering. This tool performs basic web scraping to retrieve the raw HTML content of a webpage. Optionally enable JavaScript rendering for Single Page Applications (SPAs) and sites with heavy client-side rendering. Lower cost than AI extraction (1 credit/page). Read-only operation with no side effects. Args: website_url (str): The complete URL of the webpage to scrape. - Must include protocol (http:// or https://) - Returns raw HTML content of the page - Works with both static and dynamic websites - Examples: * https://example.com/page * https://api.example.com/docs * https://news.site.com/article/123 * https://app.example.com/dashboard (may need render_heavy_js=true) - Supported protocols: HTTP, HTTPS - Invalid examples: * example.com (missing protocol) * ftp://example.com (unsupported protocol) render_heavy_js (Optional[bool]): Enable full JavaScript rendering for dynamic content. - Default: false (faster, lower cost, works for most static sites) - Set to true for sites that require JavaScript execution to display content - When to use true: * Single Page Applications (React, Angular, Vue.js) * Sites with dynamic content loading via AJAX * Content that appears only after JavaScript execution * Interactive web applications * Sites where initial HTML is mostly empty - When to use false (default): * Static websites and blogs * Server-side rendered content * Traditional HTML pages * News articles and documentation * When you need faster processing - Performance impact: * false: 2-5 seconds processing time * true: 15-30 seconds processing time (waits for JS execution) - Cost: Same (1 credit) regardless of render_heavy_js setting Returns: Dictionary containing: - html_content: The raw HTML content of the page as a string - page_title: Extracted page title if available - status_code: HTTP response status code (200 for success) - final_url: Final URL after any redirects - content_length: Size of the HTML content in bytes - processing_time: Time taken to fetch and process the page - javascript_rendered: Whether JavaScript rendering was used - credits_used: Number of credits consumed (always 1) Raises: ValueError: If website_url is malformed or missing protocol HTTPError: If the webpage returns an error status (404, 500, etc.) TimeoutError: If the page takes too long to load ConnectionError: If the website cannot be reached Use Cases: - Getting raw HTML for custom parsing - Checking page structure before using other tools - Fetching content for offline processing - Debugging website content issues - Pre-processing before AI extraction Note: - This tool returns raw HTML without any AI processing - Use smartscraper for structured data extraction - Use markdownify for clean, readable content - Consider render_heavy_js=true if initial results seem incomplete """ try: api_key = get_api_key(ctx) client = ScapeGraphClient(api_key) return client.scrape(website_url=website_url, render_heavy_js=render_heavy_js) except httpx.HTTPError as http_err: return {"error": str(http_err)} except ValueError as val_err: return {"error": str(val_err)}
• src/scrapegraph_mcp/server.py:211-229 (helper)

  Core implementation in ScapeGraphClient class that makes HTTP POST request to ScapeGraph API /scrape endpoint to fetch raw page content.
  def scrape(self, website_url: str, render_heavy_js: Optional[bool] = None) -> Dict[str, Any]: """ Basic scrape endpoint to fetch page content. Args: website_url: URL to scrape render_heavy_js: Whether to render heavy JS (optional) Returns: Dictionary containing the scraped result """ url = f"{self.BASE_URL}/scrape" payload: Dict[str, Any] = {"website_url": website_url} if render_heavy_js is not None: payload["render_heavy_js"] = render_heavy_js response = self.client.post(url, headers=self.headers, json=payload) response.raise_for_status() return response.json()