@aegis-ai/mcp-scraper

MCP server for intelligent web scraping. Supports static pages, JavaScript-rendered SPAs (via Playwright), and natural language data extraction.

Install

# Add to Claude Desktop config (~/.claude/claude_desktop_config.json):
{
  "mcpServers": {
    "scraper": {
      "command": "bunx",
      "args": ["@aegis-ai/mcp-scraper"]
    }
  }
}

# Or run directly:
bunx @aegis-ai/mcp-scraper

Related MCP server: FreeCrawl MCP Server

Tools

scrape

Fetch a web page and return clean text content.

{
  "url": "https://news.ycombinator.com",
  "js": false,
  "selector": ".titleline",
  "headers": { "Cookie": "session=abc" }
}

extract

Scrape a page and extract structured data using natural language.

{
  "url": "https://example.com/products",
  "prompt": "Get all product names, prices, and ratings",
  "js": true
}

batch_scrape

Scrape multiple URLs with rate limiting (1s between requests).

{
  "urls": ["https://example.com/page1", "https://example.com/page2"],
  "js": false
}

Features

• Static page fetching (fast, no browser needed)

• JavaScript rendering via Playwright (for SPAs)

• CSS selector targeting

• Custom headers support

• Automatic HTML stripping

• Content truncation to avoid context overflow

• Rate-limited batch scraping

License

MIT - AEGIS AI Cooperative