"Information about web crawlers or scraping techniques" matching MCP servers:

Provides a unified web search tool for AI agents by running multiple search providers concurrently and returning normalized results. It integrates with Claude Code, Codex CLI, OpenCode, and Hermes.

MCP server for web scraping with natural language, enabling site crawling, page fetching, and extraction of emails, phones, links, and social profiles. Returns clean markdown, handles JavaScript rendering, and respects rate limits.

Whatsapp-Web MCP

Enables AI coding assistants to validate HTML/CSS markup using W3C APIs, perform technical SEO audits, check broken links, and validate JSON-LD schemas directly in local workspaces.

MCP server for AI-powered web research — search (Google, Brave, Serper, SearXNG), scrape any page, extract PDFs/DOCX/YouTube transcripts, academic & patent search. Single Go binary. Works with Claude, Cursor, Copilot, and any MCP client.

Enables privacy-focused web searches using DuckDuckGo and extraction of clean, readable content from web pages with support for parallel processing of multiple URLs.

Enables AI agents to perform web searches via DuckDuckGo without API keys and automate browser tasks with headless Chromium. Supports multi-step research workflows that automatically search and extract content from multiple pages.

Provides access to Canadian federal parliamentary data (debates, bills, MPs, votes, Hansard transcripts) and legal information (case law and legislation through CanLII) for research and analysis.

Web search for AI agents. Up to 10 results for $0.001 USDC via x402 on Base. No API key.

Remote MCP / x402 / USDC / Base / Solana / OpenAPI / Google ADK / Gemini Xcatcher is an agent-first Remote MCP server (Streamable HTTP) for high-throughput crawling of fresh/latest posts across large sets of X (Twitter) usernames. It supports x402 pay-as-you-go top-ups using USDC on Base and Solana.

Provides web access tools (fetch, search, sitemap crawl) through String AI's API, automatically handling anti-bot bypass, CAPTCHA, and JavaScript rendering.

A fault-tolerant, stealth-enabled Model Context Protocol (MCP) server for web searching and content fetching. Built for AI Agents (Cursor, Claude Code, OpenCode), it uses a stealth browser engine to fetch pages, dynamically handles SPAs/React, and converts bloat into token-optimized Markdown.

MCP server: is this GitHub repo still maintained? A verdict.

Website Intelligence MCP Server exposes 10 tools for web scraping and analysis, enabling AI agents to convert URLs to Markdown, extract metadata, detect technologies, find contacts, and perform SEO checks. It wraps a REST API and supports optional API keys.

Provides MCP tools for web scraping, including smart fetching, proxy rotation, and browser rendering to bypass anti-bot challenges and render JavaScript-heavy sites.

An MCP server providing web search, image search, and page scraping tools to LLMs without requiring API keys.

Audits a domain to determine if AI agents and crawlers can access and read it, returning a flat row of 42 fields covering llms.txt, robots.txt AI policy, structured data, render mode, and machine-readable endpoint discovery.

Browser automation that bypasses anti-bot systems and handles web-scraping with CDP Mode.

A Python web-scraping toolkit that exposes 22 tools for fetching structured web data (from any URL to search, finance, e-commerce, and reference sources) as validated JSON for AI agents.

Enables web scraping, structured data extraction, and screenshot capture with automatic anti-bot bypass, supporting JavaScript rendering, proxy rotation, and tiered pricing.