"Web crawling techniques and tools" matching MCP servers:

Provides web access capabilities for LLMs including search, fetching, content extraction, PDF reading, image viewing, and screenshots.

Enables AI agents to extract structured content from URLs, search the web via DuckDuckGo, monitor site availability, and convert HTML to markdown without requiring API keys.

Provides local web search and content fetching capabilities for AI assistants, enabling them to search DuckDuckGo and extract clean text from web pages. All requests originate from the user's machine to ensure direct network control and bypass external proxies.

Enables web searching and fetching pages converted to Markdown through local stdio MCP tools, so Claude Code and other MCP clients connected via third-party relays retain internet access instead of failing with a server-side tool protocol error. Includes paginated reading of long articles, SSRF/private-network blocking, and charset handling for legacy Chinese sites.

Multi-engine web search and content extraction MCP server with 10 sources, 4-way parallel queries, and content extraction via Jina Reader, no API keys required.

Provides web connectivity tools for searching the web via DuckDuckGo or SerpAPI, fetching URL content, and extracting readable text from web pages.

MCP server exposing web_search, web_fetch, and cluesift_domain_fetch tools with configurable Ollama API keys and HTTP or stdio transport.

Enables web searches through DuckDuckGo without an API key and extracts readable main content from web pages.

Enables agents to search the live web and fetch URLs as clean Markdown, returning ranked search results and paginated page content through two lightweight tools.

Enables web search and web fetch operations using Ollama's hosted APIs, allowing MCP clients to search the web and retrieve page content.

MCP server for personal knowledge management providing AI-powered tools for semantic search, audio transcription, image generation, and PPT creation.

Odysseus Web MCP is a standalone Model Context Protocol (MCP) server for safe public-web search and URL fetching.

MCP server for web page fetching (converting to Markdown/text with automatic fallback between Tavily and Firecrawl) and web search via Tavily.

Provides five safe tools (calc, search, model_drift, compare_runs, grade_answer) for arithmetic, keyword search, model drift monitoring, eval comparison, and answer grading against sources, all implemented directly from the MCP spec with zero dependencies.

Enables a locally-run LLM to search the web, fetch pages as markdown, make arbitrary HTTP requests, and optionally render pages with headless Chromium.

Gives AI agents local web search and page reading with no API keys, layering a SearXNG-backed search cache with browser-based fallbacks and returning clean page text from HTML, PDF, JSON, and archived snapshots. Fetches run in Docker alongside the search engine with SSRF protection, egress filtering, and untrusted-content marking so agents can safely retrieve live web data.

A self-hosted MCP server providing web search and URL fetching tools, running locally without external API keys or accounts.

Enables fetching and extracting text from URLs, searching the web via DuckDuckGo and Yandex, and retrieving page links through MCP tools.

Enables MCP-capable agents to perform local web research under a declared plan with session budgets, user-approved limit extensions, and guardrails for safe searching, page fetching, content extraction, and audit logging.

Enables AI agents to fetch, crawl, extract, and search web content through a unified set of tools, with automatic routing to specialized backends.