"Web crawling techniques and tools" matching MCP servers:

Provides web search and content extraction tools for AI assistants, with zero configuration and swappable backends.

Provides local web search and content fetching capabilities for AI assistants, enabling them to search DuckDuckGo and extract clean text from web pages. All requests originate from the user's machine to ensure direct network control and bypass external proxies.

Provides web access capabilities for LLMs including search, fetching, content extraction, PDF reading, image viewing, and screenshots.

Enables AI agents to extract structured content from URLs, search the web via DuckDuckGo, monitor site availability, and convert HTML to markdown without requiring API keys.

MCP server exposing web_search, web_fetch, and cluesift_domain_fetch tools with configurable Ollama API keys and HTTP or stdio transport.

MCP server providing free, IP-direct tools for web search, crawling, anti-detect browsing, and structured data from Wikipedia, Wikidata, GitHub, and YouTube.

An MCP-native pipeline for collecting structured intelligence on higher education institutions using the WHED schema, enabling scraping, extraction, validation, and saving of profiles.

Relays a result screenshot to Slack so you can review a coding agent's output without opening the app.

Enables web search and web fetch operations using Ollama's hosted APIs, allowing MCP clients to search the web and retrieve page content.

Enables Codex to send text tasks to ChatGPT web, await responses, continue conversations, choose reasoning effort, and manage projects through MCP tools on macOS.

MCP server for personal knowledge management providing AI-powered tools for semantic search, audio transcription, image generation, and PPT creation.

MCP server that lets AI editors execute JavaScript, read console output, and simulate clicks/input on static web pages. Supports multiple browser pages and can be deployed remotely via HTTP transport.

Provides five safe tools (calc, search, model_drift, compare_runs, grade_answer) for arithmetic, keyword search, model drift monitoring, eval comparison, and answer grading against sources, all implemented directly from the MCP spec with zero dependencies.

Enables building and evaluating LangGraph agents over a protocol-level MCP server to measure failure rates across four agent topologies.

Enables a locally-run LLM to search the web, fetch pages as markdown, make arbitrary HTTP requests, and optionally render pages with headless Chromium.

Enables extraction of recipes from PDFs or JPG files and conversion into structured HTML, with interactive region selection and OCR via a web GUI.

Gives AI agents local web search and page reading with no API keys, layering a SearXNG-backed search cache with browser-based fallbacks and returning clean page text from HTML, PDF, JSON, and archived snapshots. Fetches run in Docker alongside the search engine with SSRF protection, egress filtering, and untrusted-content marking so agents can safely retrieve live web data.

Enables live website health checks including TLS, HTTPS, and security headers, returning an A-F grade with specific fixes.

Enables AI agents to extract clean, structured web content (articles, tables, links, visual layouts) optimized for LLM token efficiency, with fast response times and optional JavaScript support.

Provides an LLM client with self-hosted web scraping capabilities including scraping, crawling, mapping, searching, structured data extraction, browser interaction, and deep research, without reliance on paid proxy or CAPTCHA services.