"Web crawling techniques and tools" matching MCP servers:

Provides web access capabilities for LLMs including search, fetching, content extraction, PDF reading, image viewing, and screenshots.

MCP server exposing web_search, web_fetch, and cluesift_domain_fetch tools with configurable Ollama API keys and HTTP or stdio transport.

An MCP-native pipeline for collecting structured intelligence on higher education institutions using the WHED schema, enabling scraping, extraction, validation, and saving of profiles.

Relays a result screenshot to Slack so you can review a coding agent's output without opening the app.

Enables web search and web fetch operations using Ollama's hosted APIs, allowing MCP clients to search the web and retrieve page content.

Enables Codex to send text tasks to ChatGPT web, await responses, continue conversations, choose reasoning effort, and manage projects through MCP tools on macOS.

MCP server for personal knowledge management providing AI-powered tools for semantic search, audio transcription, image generation, and PPT creation.

Enables async-first working practices by providing tools to draft decision docs, convert meetings to async artifacts, score status updates, and triage sync vs async tasks. It also offers reference tools for the Open and Async book's principles and coaching prompts.

MCP server for web page fetching (converting to Markdown/text with automatic fallback between Tavily and Firecrawl) and web search via Tavily.

Enables building and evaluating LangGraph agents over a protocol-level MCP server to measure failure rates across four agent topologies.

Enables a locally-run LLM to search the web, fetch pages as markdown, make arbitrary HTTP requests, and optionally render pages with headless Chromium.

Enables extraction of recipes from PDFs or JPG files and conversion into structured HTML, with interactive region selection and OCR via a web GUI.

Gives AI agents local web search and page reading with no API keys, layering a SearXNG-backed search cache with browser-based fallbacks and returning clean page text from HTML, PDF, JSON, and archived snapshots. Fetches run in Docker alongside the search engine with SSRF protection, egress filtering, and untrusted-content marking so agents can safely retrieve live web data.

Provides an LLM client with self-hosted web scraping capabilities including scraping, crawling, mapping, searching, structured data extraction, browser interaction, and deep research, without reliance on paid proxy or CAPTCHA services.

Enables fetching and extracting text from URLs, searching the web via DuckDuckGo and Yandex, and retrieving page links through MCP tools.

Enables MCP-capable agents to perform local web research under a declared plan with session budgets, user-approved limit extensions, and guardrails for safe searching, page fetching, content extraction, and audit logging.

Enables AI agents to fetch, crawl, extract, and search web content through a unified set of tools, with automatic routing to specialized backends.

Provides DuckDuckGo web search and site fetching tools for Claude Desktop, enabling up to 10 search results and text content extraction from URLs.

Enables AI agents to drive a logged-in ChatGPT web session over CDP, including managing projects and chats, sending messages with file attachments, and retrieving long Pro replies.

Comprehensive web research toolkit with 13 tools for searching (via SearXNG), crawling, package discovery, GitHub metrics, error translation, API documentation lookup, data extraction, technology comparison, and service status checking.