"Information about web scraping or crawling" matching MCP servers:

Enables fetching any public web page through managed proxies, with optional JavaScript rendering, returning markdown, text, HTML, or structured JSON.

MCP server for web scraping with natural language, enabling site crawling, page fetching, and extraction of emails, phones, links, and social profiles. Returns clean markdown, handles JavaScript rendering, and respects rate limits.

Enables converting any URL or website into clean, LLM-ready Markdown, supporting single-page scraping and full-site crawling for AI agents.

Whatsapp-Web MCP

MCP server for AI-powered web research — search (Google, Brave, Serper, SearXNG), scrape any page, extract PDFs/DOCX/YouTube transcripts, academic & patent search. Single Go binary. Works with Claude, Cursor, Copilot, and any MCP client.

Provides a unified web search tool for AI agents by running multiple search providers concurrently and returning normalized results. It integrates with Claude Code, Codex CLI, OpenCode, and Hermes.

Enables reading web pages and PDFs as clean Markdown, extracting main content and letting agents query for relevant passages, outline sections, search in-page, and list links while respecting robots.txt and blocking private addresses.

Enables AI coding assistants to validate HTML/CSS markup using W3C APIs, perform technical SEO audits, check broken links, and validate JSON-LD schemas directly in local workspaces.

Enables MCP-capable agents to perform reliable web search and content fetching through a single server that automatically fails over across multiple providers. It handles keyed or keyless authentication automatically and returns clean Markdown or text results.

Enables privacy-focused web searches using DuckDuckGo and extraction of clean, readable content from web pages with support for parallel processing of multiple URLs.

Enables AI agents to perform web searches via DuckDuckGo without API keys and automate browser tasks with headless Chromium. Supports multi-step research workflows that automatically search and extract content from multiple pages.

Provides access to Canadian federal parliamentary data (debates, bills, MPs, votes, Hansard transcripts) and legal information (case law and legislation through CanLII) for research and analysis.

Web search for AI agents. Up to 10 results for $0.001 USDC via x402 on Base. No API key.

Remote MCP / x402 / USDC / Base / Solana / OpenAPI / Google ADK / Gemini Xcatcher is an agent-first Remote MCP server (Streamable HTTP) for high-throughput crawling of fresh/latest posts across large sets of X (Twitter) usernames. It supports x402 pay-as-you-go top-ups using USDC on Base and Solana.

Provides web access tools (fetch, search, sitemap crawl) through String AI's API, automatically handling anti-bot bypass, CAPTCHA, and JavaScript rendering.

Free web search and page fetching for AI agents. No account or API key required. Connect to https://search.parallel.ai/mcp over Streamable HTTP. Tools: web_search and web_fetch. Free-tier rate limits apply.

MCP server: is this GitHub repo still maintained? A verdict.

Website Intelligence MCP Server exposes 10 tools for web scraping and analysis, enabling AI agents to convert URLs to Markdown, extract metadata, detect technologies, find contacts, and perform SEO checks. It wraps a REST API and supports optional API keys.

Enables targeted company discovery, real-time DNS MX deliverability diagnostics, and clean web Markdown extraction for AI agents.

Enables agents to convert any URL into clean, structured, agent-optimized content with a single paid call, paying $0.02 USDC per successful extraction via x402 on Base without API keys or accounts.