"Web crawling techniques and tools" matching MCP servers:

Enables converting any URL or website into clean, LLM-ready Markdown, supporting single-page scraping and full-site crawling for AI agents.

MCP server for compressing AI embeddings by 5-7x using TurboQuant (PolarQuant + QJL), with tools to compress, decompress, estimate savings, and embed+compress vectors.

Provides access to OpenAI's advanced models (including o3) with integrated web search, Python code interpreter, and combined analysis capabilities. Enables users to perform web searches, execute code in sandboxed environments, and combine search with analysis through natural language.

A multi-function Streamable HTTP MCP tool aggregation server that provides web search via Brave, Exa, and SearXNG with multi-key rotation and cross-provider fallback, and supports extensible tool families (URL fetch, code search, RAG) through a pluggable architecture.

Enables extracting clean, structured markdown from any URL—stripping nav, ads, and scripts—for RAG pipelines and AI research agents, with pay-per-call micropayments via x402.

A Model Context Protocol server that allows LLMs to interact with web content through standardized tools, currently supporting web scraping functionality.

Enables clinical question answering via OpenEvidence with citation verification, including tools to ask questions, retrieve results, and follow up, plus an integrated skill to check citations against primary sources.

MCP server for semantic search over PDF documents using RAG, pgvector, and OpenAI embeddings. Includes a web app for document upload and management.

Provides free web search, content fetching, image search, and deep research via SearXNG, no API keys required.

MCP server for searching web documentation via sitemap.xml, enabling semantic search across multiple documentation sites using Ollama embeddings.

Provides standard MCP tools for web search, page fetching with content extraction, and a combined research workflow that returns sources with excerpts, plus a session resource and fact-check prompt template.

Provides AI agents and RAG pipelines with production-grade web search, deep research, news, academic, extraction, crawling, and source verification tools. Enables citation-anchored, deduplicated, fresh, and security-hardened evidence gathering from across the web.

Extends VS Code Copilot with MCP tools for persistent semantic memory, code indexing and RAG search, sandboxed Python/shell execution, web fetch/search, dynamic tool management via a web UI, and structured multi-step reasoning.

MCP server that provides web search via LLM API gateways with server-side search, returning cited answers and multi-provider fallback.

A fast, dependency-light MCP server that converts web pages into clean Markdown, structured metadata, and classified links for language models.

An MCP server that runs Apify scraping actors with quality-gated validation, scoring, and retries, returning embedding-ready chunks for AI agents.

Enables AI assistants to search a curated database of 200+ AI tool cards with pricing, platforms, use cases, and source links, so users can ask which tool to use for a task and get grounded recommendations.

A focused MCP server that provides a single web_search tool for web searches, integrating Tavily and Brave with automatic fallback and key rotation.

MCP server that fetches web pages and uses local Qwen models to answer questions about their content, with site-scoped search and map-reduce chunking for long pages.

Guides problem-solving by breaking down complex problems into steps and recommending appropriate MCP tools for each stage, with confidence scores and rationales for tool suggestions.