"Web crawling techniques and tools" matching MCP servers:

Provides web search and content extraction tools for AI assistants, with zero configuration and swappable backends.

Enables AI agents to extract structured content from URLs, search the web via DuckDuckGo, monitor site availability, and convert HTML to markdown without requiring API keys.

Enables live website health checks including TLS, HTTPS, and security headers, returning an A-F grade with specific fixes.

Enables Claude and other LLMs to read and parse web content, with smart fallback strategies to bypass access restrictions (e.g., paywalls, Cloudflare) and output Markdown.

Enables querying MITRE ATT&CK and D3FEND data efficiently, providing techniques, detections, mitigations, actors, and countermeasures with token-aware summaries and paginated graph traversal.

Enables intelligent web searching using SearXNG with content crawling via Creeper, then summarizes webpage content using LLM to avoid token limit issues. Supports smart filtering with domain blacklist/whitelist and optional LLM-based relevance filtering.

Provides web search and page fetch tools for LLM assistants, extracting content and enabling text search within fetched pages.

Enables web research through multi-provider search, documentation lookup, public code search, and clean Markdown extraction from static or JavaScript-rendered pages via five read-only tools.

Provides deterministic tools for transforming, formatting, and inspecting structured data for AI agents.

Enables MCP clients to connect to a privacy-first, self-hostable workout planning and training log, allowing coaching agents to preview and apply program changes while accessing training data through OAuth-protected endpoints.

Web search API for AI agents. Returns structured results with title, URL, and snippet; pay-per-call via x402 micropayments.

Provides lightweight web crawling and scraping capabilities, enabling users to fetch pages as clean Markdown, recursively crawl domains, extract metadata, and perform search-and-crawl operations.

Enables AI agents to search, filter, and monitor curated AI model API free credits and low-cost token promotions through eight tools, four resources, and three prompts — including finding deals expiring soon, checking eligibility and blocked conditions, and producing daily briefs.

A local Model Context Protocol (MCP) server that exposes custom tools to Claude Desktop, enabling direct interaction with your local environment. It provides a framework for building and integrating custom TypeScript tools into the Claude interface.

Enables Claude Agent SDK agents to safely operate a remote user machine via a WebSocket tunnel, providing six client__* MCP tools with path fencing and phase-switch adjudication.

Provides eight tools for web search, content extraction, screenshots, PDFs, JavaScript execution, crawling, and Wayback Machine snapshots and archiving.

Enables an agent to locate embedded Flash or Java games (on live pages or in the Internet Archive), inspect whether they can run, fetch the Ruffle or CheerpJ runtime, package a self-contained static site, and verify playback in a real headless browser. Every step produces an evidence bundle so revived games can be hosted on any static host with claims backed by recorded results.

Enables web search via DuckDuckGo and web content fetching through the Model Context Protocol, allowing AI assistants to search and retrieve web page content.

MCP server enabling AI agents to interact with OpenProject API v3 for project management, including creating and managing work packages, projects, comments, time entries, boards, and user dashboards.

Enables compatible MCP clients to read JavaScript-rendered web pages as bounded Markdown or structured records, with URL policy and resource limits for safe, inspectable access.