"Web crawling techniques and tools" matching MCP servers:

Enables live-web research by providing stateless browsing, link extraction, and page search capabilities via Playwright, allowing agents to fetch and analyze web content without session management.

MCP server that lets AI editors execute JavaScript, read console output, and simulate clicks/input on static web pages. Supports multiple browser pages and can be deployed remotely via HTTP transport.

Enables live website health checks including TLS, HTTPS, and security headers, returning an A-F grade with specific fixes.

Enables AI agents to extract clean, structured web content (articles, tables, links, visual layouts) optimized for LLM token efficiency, with fast response times and optional JavaScript support.

Provides an LLM client with self-hosted web scraping capabilities including scraping, crawling, mapping, searching, structured data extraction, browser interaction, and deep research, without reliance on paid proxy or CAPTCHA services.

Enables AI agents to fetch, crawl, extract, and search web content through a unified set of tools, with automatic routing to specialized backends.

Enables querying MITRE ATT&CK and D3FEND data efficiently, providing techniques, detections, mitigations, actors, and countermeasures with token-aware summaries and paginated graph traversal.

Enables intelligent web searching using SearXNG with content crawling via Creeper, then summarizes webpage content using LLM to avoid token limit issues. Supports smart filtering with domain blacklist/whitelist and optional LLM-based relevance filtering.

Provides deterministic tools for transforming, formatting, and inspecting structured data for AI agents.

Enables MCP clients to connect to a privacy-first, self-hostable workout planning and training log, allowing coaching agents to preview and apply program changes while accessing training data through OAuth-protected endpoints.

Enables AI agents to search, filter, and monitor curated AI model API free credits and low-cost token promotions through eight tools, four resources, and three prompts — including finding deals expiring soon, checking eligibility and blocked conditions, and producing daily briefs.

MCP server for multi-engine web search and web page fetching, supporting parallel search, content extraction, and optional LLM-powered search summarization and deep search.

Enables web search via DuckDuckGo and web content fetching through the Model Context Protocol, allowing AI assistants to search and retrieve web page content.

An MCP server for automated interaction with Gemini's web interface using Playwright and LangGraph, enabling task execution, deep research, file uploads, chat management, and dynamic selector verification via MCP tools.

Enables AI assistants to automate web interactions such as navigating pages, clicking elements, filling forms, querying the DOM, evaluating JavaScript, and inspecting network/console logs — all from a Termux terminal without Chromium, root, or proot.

A high-performance search service that converts results from Google, Bing, and DuckDuckGo into structured JSON or Markdown. It features multi-layer depth crawling and uses the Camoufox anti-detection browser for reliable content extraction and fallback search logic.

Enables AI agents to perform common utility tasks through a streamable HTTP MCP endpoint, including JSON inspection, regex testing, cron parsing, hashing, base64 encoding/decoding, URL analysis, color conversion, text diffing, CSV parsing, JWT decode/verification, Markdown-to-HTML rendering, and UUID/token generation without API keys.

A Model Context Protocol server that allows LLMs to interact with web content through standardized tools, currently supporting web scraping functionality.

Enables fetching, crawling, and browsing anti-bot protected web pages with real browser rendering, returning clean JSON or markdown, and solving captchas.

Provides tools for AI-powered graph analysis, including relationship extraction, adjacency matrix creation, and network centrality calculations. It enables users to perform complex structural analysis and generate interactive D3.js visualizations from structured data.