"Web crawling techniques and tools" matching MCP servers:

MCP server for AI-powered web research — search (Google, Brave, Serper, SearXNG), scrape any page, extract PDFs/DOCX/YouTube transcripts, academic & patent search. Single Go binary. Works with Claude, Cursor, Copilot, and any MCP client.

An MCP server that enables AI assistants to perform web searches on Perplexity.ai using browser automation instead of an official API. It supports persistent authenticated sessions and returns search results along with cited sources directly to the client.

Provides programmatic access to Material Web documentation and components for AI agents, enabling documentation search, template generation, and HTML validation.

Provides web automation utilities, including taking Camofox browser tab screenshots and saving them directly to local files.

Enables AI assistants to search Perplexity.ai through browser automation without an API key, returning answers with cited sources.

A comprehensive Model Context Protocol server providing access to 70+ IT tools for developers and system administrators, including encoding/decoding, text manipulation, hashing, and network utilities.

A Python MCP server for web research that enables fetching web pages, searching, batch-fetching, extracting links, and summarizing content via client-side sampling, with optional local file access.

Enables AI assistants to access real-time web data through search, markdown scraping, and browser automation while bypassing anti-bot protections. It provides tools for web research, e-commerce monitoring, and data extraction from across the globe.

An MCP server that connects AI assistants to a Neo4j graph database, enabling querying of movie data through tools like graph statistics, genre-based search, and movie details, along with advanced features such as LLM sampling and completions.

Enables agents to quickly find and edit code in a codebase with surgical precision. Find symbols, edit them everywhere with tools for reading code blocks, searching/replacing text, and making precise line-based modifications.

A Firecrawl MCP server with built-in support for multiple API key rotation, enabling web search, scraping, crawling, extraction, and more via MCP clients.

A lightweight MCP server that enables AI assistants to execute local development tools and retrieve system status with low latency over stdio or HTTP.

An MCP server that provides AI clients with access to developer documentation via llms.txt files. Exposes tools, resources, and prompts.

A local-first, no-API-key MCP server that enables LLMs to search the web, fetch pages, and read documents using multiple engines and smart fallbacks.

Enables AI assistants to perform Google searches with real, resolvable sources and transparent search terms.

MCP server for overseer.nvim — gives any MCP-capable coding agent running inside Neovim (Claude Code, Codex, Gemini CLI, opencode) the same visibility and control over tasks the user has: list, run, tail, restart, stop. Agent-started dev servers land in the user's own task list instead of becoming invisible orphaned processes.

Enables AI models to access and search offline Wikipedia and other knowledge bases stored in ZIM format files. Provides intelligent content retrieval, structured browsing, advanced search capabilities, and metadata extraction for comprehensive offline knowledge access.

Read-only MCP server for Grafana Loki: lets AI agents (Claude Code, Cursor, Copilot) discover labels, read bounded LogQL pages, count lines by label or time, and export logs — across several Loki stands

Enables multi-level reasoning with configurable depth levels, session-based state management, structured thought input, and real-time trace resources.

Relays error logs from other MCP servers when an LLM call fails and the returned error message is unclear, helping the model handle errors more intelligently.