"Information about web crawlers or related technology" matching MCP servers:

An MCP (Model Context Protocol) server that gives AI agents live, structured ad intelligence across Facebook, Google, and Instagram — data that no base model can produce from training alone. Powered by Apify actors. Works with any MCP-compatible client: Cursor, Claude, etc.

Provides live wait times and office details for all 60 Oregon DMV field offices by scraping the ODOT website, with no authentication required.

A minimal MCP server that crawls web pages using crawl4ai and returns content as markdown, deployable to Cloud Run.

Enables a locally-run LLM to search the web, fetch pages as markdown, make arbitrary HTTP requests, and optionally render pages with headless Chromium.

A reconnaissance MCP server that discovers web data sources by monitoring network requests and extracting API structures, enabling AI to understand data location and format without full crawling.

A local MCP server providing web search, page extraction, and safe browser automation tools for Hermes, Claude Code, and other MCP clients.

MCP server for multi-engine web search and web page fetching, supporting parallel search, content extraction, and optional LLM-powered search summarization and deep search.

Enables compatible MCP clients to read JavaScript-rendered web pages as bounded Markdown or structured records, with URL policy and resource limits for safe, inspectable access.

Converts URLs into clean, LLM-ready markdown, respecting robots.txt and never bypassing anti-bot measures or paywalls.

Enables AI agents to search DuckDuckGo and fetch web content with human-assisted CAPTCHA handling, designed for local use where humans can intervene when bot detection occurs.

A self-hosted MCP server that reads web pages and converts them to Markdown, with built-in web search and anti-crawl fallback using Playwright.

Converts web pages to Markdown for MCP clients like Claude, with support for single and batch reads, caching, fallback rendering, and SSRF protection.

A local MCP server that gives Claude Code web search capabilities using DuckDuckGo, enabling discovery of URLs from natural language queries.

MCP server that parses web pages into clean markdown via Crawl4AI's HTTP API, exposing parse_page and health-check tools with configurable proxy and timeout.

A web-fetch MCP server for LLM agents that fetches pages with an escalation ladder through different engines, raising FetchBlocked instead of returning blocked content.

An open-source MCP server providing AI agents with neural web search via Exa and tiered web fetch (Exa, local browser, Firecrawl) as a drop-in replacement for built-in web tools, preserving provenance and guarding against SSRF.

Enables Claude to fetch web pages from the user's own local network connection instead of AI datacenter IPs, including optional authenticated sessions, with verdicts for bot challenges, login walls, and other failures.

Enables web searches through DuckDuckGo without an API key and extracts readable main content from web pages.

Provides tools to fetch web content in HTML, JSON, text, or Markdown formats via MCP. Supports custom headers, length limits, and start index.

Enables web search and web fetch operations using Ollama's hosted APIs, allowing MCP clients to search the web and retrieve page content.