"Web scraping and crawling tools" matching MCP servers:

Provides an LLM client with self-hosted web scraping capabilities including scraping, crawling, mapping, searching, structured data extraction, browser interaction, and deep research, without reliance on paid proxy or CAPTCHA services.

Provides browser automation and web scraping as MCP tools, enabling autonomous URL ingestion, crawling, extraction, and anti-bot handling with interactive browser control.

Fetches a URL and returns the main content as clean Markdown, using plain HTTP when possible and headless Chromium for JavaScript-rendered or bot-protected pages.

Connect AI assistants to your Gramps Web family tree. Search and edit genealogy records, explore ancestors, descendants, relationships and timelines, and read document scans with vision-capable models. Includes guided research workflows, source citations, read-only mode, and privacy-aware media previews. Install with Docker or a Claude Desktop bundle.

Bridges the editing capabilities of an already-open DingTalk document in the browser to any MCP host via standard stdio, forwarding tools to the page's internal MCP server over a local WebSocket. Supports read/write operations through suggested edits, pairing-based security, and zero runtime dependencies.

A lean FastMCP server that wraps xAI's web_search and x_search tools for MCP hosts, enabling live web and X search with citations.

Enables web searches through DuckDuckGo without an API key and extracts readable main content from web pages.

MCP server that searches the web, reads pages, and returns concise summaries with cited sources via the research_web and read_url tools.

An MCP server providing web search, image search, and page scraping tools to LLMs without requiring API keys.

MCP server for scraping with declarative configs

Gives MCP-capable agents live web access: search the web, scrape pages into Markdown (including JavaScript-heavy and bot-protected sites), and extract named fields as JSON, with job polling, token-aware content offloading, and built-in research guidance. Ships as a self-hostable stdio or HTTP service with spend caps and per-request key support.

MCP server for AI-powered web research — search (Google, Brave, Serper, SearXNG), scrape any page, extract PDFs/DOCX/YouTube transcripts, academic & patent search. Single Go binary. Works with Claude, Cursor, Copilot, and any MCP client.

Integrates Firecrawl for web scraping, crawling, search, and content extraction capabilities. Supports single/batch scraping, URL discovery, structured data extraction, deep research, and AI-powered web analysis with automatic retries and rate limiting.

Enables AI assistants to search Perplexity.ai through browser automation without an API key, returning answers with cited sources.

Enables private web search and webpage content extraction using a local SearxNG instance, prioritizing user privacy and autonomy.

MCP server for web crawling, searching, and AI-powered content extraction, supporting single-page, batch, and full-site crawling along with text, news, image, book, and video search.

Enables AI-assisted user feedback collection and management via a web interface, with MCP integration for automated replies and tool execution.

Provides read-only access to public Threads.net profiles and threads via anonymous scraping, with tools for profile info, user threads, and session management.

MCP server for querying Taiwan's real estate transaction registry via web scraping of the Ministry of the Interior's official portal. Enables natural language queries for real estate sales, rentals, and pre-sale housing data.

An MCP server that enables AI assistants to perform web searches on Perplexity.ai using browser automation instead of an official API. It supports persistent authenticated sessions and returns search results along with cited sources directly to the client.