ScrapeGraph MCP Server

Instructions

render_heavy_js (Optional[bool]): Enable full JavaScript rendering for dynamic content.
    - Default: false (faster, lower cost, works for most static sites)
    - Set to true for sites that require JavaScript execution to display content
    - When to use true:
      * Single Page Applications (React, Angular, Vue.js)
      * Sites with dynamic content loading via AJAX
      * Content that appears only after JavaScript execution
      * Interactive web applications
      * Sites where initial HTML is mostly empty
    - When to use false (default):
      * Static websites and blogs
      * Server-side rendered content
      * Traditional HTML pages
      * News articles and documentation
      * When you need faster processing
    - Performance impact:
      * false: 2-5 seconds processing time
      * true: 15-30 seconds processing time (waits for JS execution)
    - Cost: Same (1 credit) regardless of render_heavy_js setting

Implementation Reference

• src/scrapegraph_mcp/server.py:1850-1937 (handler)

  MCP tool handler for the 'scrape' tool. Fetches raw page content via ScapeGraph API, handling API key retrieval and error wrapping.

  @mcp.tool(annotations={"readOnlyHint": True, "destructiveHint": False, "idempotentHint": True})
  def scrape(
      website_url: str,
      ctx: Context,
      render_heavy_js: Optional[bool] = None
  ) -> Dict[str, Any]:
      """
      Fetch raw page content from any URL with optional JavaScript rendering.
  
      This tool performs basic web scraping to retrieve the raw HTML content of a webpage.
      Optionally enable JavaScript rendering for Single Page Applications (SPAs) and sites with
      heavy client-side rendering. Lower cost than AI extraction (1 credit/page).
      Read-only operation with no side effects.
  
      Args:
          website_url (str): The complete URL of the webpage to scrape.
              - Must include protocol (http:// or https://)
              - Returns raw HTML content of the page
              - Works with both static and dynamic websites
              - Examples:
                * https://example.com/page
                * https://api.example.com/docs
                * https://news.site.com/article/123
                * https://app.example.com/dashboard (may need render_heavy_js=true)
              - Supported protocols: HTTP, HTTPS
              - Invalid examples:
                * example.com (missing protocol)
                * ftp://example.com (unsupported protocol)
  
          render_heavy_js (Optional[bool]): Enable full JavaScript rendering for dynamic content.
              - Default: false (faster, lower cost, works for most static sites)
              - Set to true for sites that require JavaScript execution to display content
              - When to use true:
                * Single Page Applications (React, Angular, Vue.js)
                * Sites with dynamic content loading via AJAX
                * Content that appears only after JavaScript execution
                * Interactive web applications
                * Sites where initial HTML is mostly empty
              - When to use false (default):
                * Static websites and blogs
                * Server-side rendered content
                * Traditional HTML pages
                * News articles and documentation
                * When you need faster processing
              - Performance impact:
                * false: 2-5 seconds processing time
                * true: 15-30 seconds processing time (waits for JS execution)
              - Cost: Same (1 credit) regardless of render_heavy_js setting
  
      Returns:
          Dictionary containing:
          - html_content: The raw HTML content of the page as a string
          - page_title: Extracted page title if available
          - status_code: HTTP response status code (200 for success)
          - final_url: Final URL after any redirects
          - content_length: Size of the HTML content in bytes
          - processing_time: Time taken to fetch and process the page
          - javascript_rendered: Whether JavaScript rendering was used
          - credits_used: Number of credits consumed (always 1)
  
      Raises:
          ValueError: If website_url is malformed or missing protocol
          HTTPError: If the webpage returns an error status (404, 500, etc.)
          TimeoutError: If the page takes too long to load
          ConnectionError: If the website cannot be reached
  
      Use Cases:
          - Getting raw HTML for custom parsing
          - Checking page structure before using other tools
          - Fetching content for offline processing
          - Debugging website content issues
          - Pre-processing before AI extraction
  
      Note:
          - This tool returns raw HTML without any AI processing
          - Use smartscraper for structured data extraction
          - Use markdownify for clean, readable content
          - Consider render_heavy_js=true if initial results seem incomplete
      """
      try:
          api_key = get_api_key(ctx)
          client = ScapeGraphClient(api_key)
          return client.scrape(website_url=website_url, render_heavy_js=render_heavy_js)
      except httpx.HTTPError as http_err:
          return {"error": str(http_err)}
      except ValueError as val_err:
          return {"error": str(val_err)}

• src/scrapegraph_mcp/server.py:211-229 (helper)

  Core implementation in ScapeGraphClient class that makes HTTP POST request to ScapeGraph API /scrape endpoint to fetch raw page content.

  def scrape(self, website_url: str, render_heavy_js: Optional[bool] = None) -> Dict[str, Any]:
      """
      Basic scrape endpoint to fetch page content.
  
      Args:
          website_url: URL to scrape
          render_heavy_js: Whether to render heavy JS (optional)
  
      Returns:
          Dictionary containing the scraped result
      """
      url = f"{self.BASE_URL}/scrape"
      payload: Dict[str, Any] = {"website_url": website_url}
      if render_heavy_js is not None:
          payload["render_heavy_js"] = render_heavy_js
  
      response = self.client.post(url, headers=self.headers, json=payload)
      response.raise_for_status()
      return response.json()