"Information about web crawlers or related technology" matching MCP servers:

Provides a unified web search tool for AI agents by running multiple search providers concurrently and returning normalized results. It integrates with Claude Code, Codex CLI, OpenCode, and Hermes.

MCP server for web scraping with natural language, enabling site crawling, page fetching, and extraction of emails, phones, links, and social profiles. Returns clean markdown, handles JavaScript rendering, and respects rate limits.

Whatsapp-Web MCP

Enables AI coding assistants to validate HTML/CSS markup using W3C APIs, perform technical SEO audits, check broken links, and validate JSON-LD schemas directly in local workspaces.

MCP server for AI-powered web research — search (Google, Brave, Serper, SearXNG), scrape any page, extract PDFs/DOCX/YouTube transcripts, academic & patent search. Single Go binary. Works with Claude, Cursor, Copilot, and any MCP client.

Enables privacy-focused web searches using DuckDuckGo and extraction of clean, readable content from web pages with support for parallel processing of multiple URLs.

Enables AI agents to perform web searches via DuckDuckGo without API keys and automate browser tasks with headless Chromium. Supports multi-step research workflows that automatically search and extract content from multiple pages.

Provides access to Canadian federal parliamentary data (debates, bills, MPs, votes, Hansard transcripts) and legal information (case law and legislation through CanLII) for research and analysis.

Web search for AI agents. Up to 10 results for $0.001 USDC via x402 on Base. No API key.

Provides web access tools (fetch, search, sitemap crawl) through String AI's API, automatically handling anti-bot bypass, CAPTCHA, and JavaScript rendering.

A fault-tolerant, stealth-enabled Model Context Protocol (MCP) server for web searching and content fetching. Built for AI Agents (Cursor, Claude Code, OpenCode), it uses a stealth browser engine to fetch pages, dynamically handles SPAs/React, and converts bloat into token-optimized Markdown.

MCP server: is this GitHub repo still maintained? A verdict.

Website Intelligence MCP Server exposes 10 tools for web scraping and analysis, enabling AI agents to convert URLs to Markdown, extract metadata, detect technologies, find contacts, and perform SEO checks. It wraps a REST API and supports optional API keys.

MCP server for searching French and European law, providing natural language access to jurisprudence, consolidated legal texts, and related legal information via tools like search_decisions and search_legal_texts.

Scans job boards for keyword patterns that indicate buying intent, technology adoption, or team expansion. Returns structured signal data for outbound targeting.

Enables natural language queries about the technologies behind websites and the companies running them, using the StackScan Tech Lookup API. Includes tools for domain technology lookup, company details, technology usage stats, and batch company lookups.

Investigate any website's technology stack: CMS, eCommerce, frameworks, analytics, CRM, payments, and hosting behind any URL.

Detects a company's technology stack from its public web presence. Returns structured tech signals for sales targeting and competitive analysis.

Audits a domain to determine if AI agents and crawlers can access and read it, returning a flat row of 42 fields covering llms.txt, robots.txt AI policy, structured data, render mode, and machine-readable endpoint discovery.

A read-only MCP server that gives LLM agents deterministic operational facts about Trello boards, including structural history, card movement, workflow flow, staleness, and due-date information.