"Web crawling techniques and tools" matching MCP servers:

Provides web access capabilities for LLMs including search, fetching, content extraction, PDF reading, image viewing, and screenshots.

Provides local web search and content fetching capabilities for AI assistants, enabling them to search DuckDuckGo and extract clean text from web pages. All requests originate from the user's machine to ensure direct network control and bypass external proxies.

Enables web searching and fetching pages converted to Markdown through local stdio MCP tools, so Claude Code and other MCP clients connected via third-party relays retain internet access instead of failing with a server-side tool protocol error. Includes paginated reading of long articles, SSRF/private-network blocking, and charset handling for legacy Chinese sites.

A custom MCP server that provides AI agents with tools to fetch web pages, extract readable article text, extract structured data by CSS selector, and check robots.txt permissions.

MCP server providing free, IP-direct tools for web search, crawling, anti-detect browsing, and structured data from Wikipedia, Wikidata, GitHub, and YouTube.

An MCP-native pipeline for collecting structured intelligence on higher education institutions using the WHED schema, enabling scraping, extraction, validation, and saving of profiles.

An MCP server for safe web crawling and scraping of public and authenticated sites using Playwright and Crawlee. It provides tools for scraping single pages, crawling entire sites, validating sessions, and listing supported sites, with built-in security protections and adapter-based extensibility.

Relays a result screenshot to Slack so you can review a coding agent's output without opening the app.

Enables Codex to send text tasks to ChatGPT web, await responses, continue conversations, choose reasoning effort, and manage projects through MCP tools on macOS.

MCP server for personal knowledge management providing AI-powered tools for semantic search, audio transcription, image generation, and PPT creation.

Enables async-first working practices by providing tools to draft decision docs, convert meetings to async artifacts, score status updates, and triage sync vs async tasks. It also offers reference tools for the Open and Async book's principles and coaching prompts.

Odysseus Web MCP is a standalone Model Context Protocol (MCP) server for safe public-web search and URL fetching.

Provides five safe tools (calc, search, model_drift, compare_runs, grade_answer) for arithmetic, keyword search, model drift monitoring, eval comparison, and answer grading against sources, all implemented directly from the MCP spec with zero dependencies.

Enables ChatGPT web to use local tools like file reading, command execution, and patch application through an MCP server over OpenAI Secure MCP Tunnel.

Enables a locally-run LLM to search the web, fetch pages as markdown, make arbitrary HTTP requests, and optionally render pages with headless Chromium.

Enables extraction of recipes from PDFs or JPG files and conversion into structured HTML, with interactive region selection and OCR via a web GUI.

Gives AI agents local web search and page reading with no API keys, layering a SearXNG-backed search cache with browser-based fallbacks and returning clean page text from HTML, PDF, JSON, and archived snapshots. Fetches run in Docker alongside the search engine with SSRF protection, egress filtering, and untrusted-content marking so agents can safely retrieve live web data.

A self-hosted MCP server providing web search and URL fetching tools, running locally without external API keys or accounts.

An MCP bridge that exposes DeepSeek Harness (DSH) web UI's cordis RPC API as stdio tools, enabling any MCP client to manage workspaces, create/resume sessions, send messages, and fetch session stats.

Enables Claude and other LLMs to read and parse web content, with smart fallback strategies to bypass access restrictions (e.g., paywalls, Cloudflare) and output Markdown.