"Information about web crawlers or scraping techniques" matching MCP servers:

Enables live-web research by providing stateless browsing, link extraction, and page search capabilities via Playwright, allowing agents to fetch and analyze web content without session management.

MCP server that lets AI editors execute JavaScript, read console output, and simulate clicks/input on static web pages. Supports multiple browser pages and can be deployed remotely via HTTP transport.

Enables live website health checks including TLS, HTTPS, and security headers, returning an A-F grade with specific fixes.

Provides an LLM client with self-hosted web scraping capabilities including scraping, crawling, mapping, searching, structured data extraction, browser interaction, and deep research, without reliance on paid proxy or CAPTCHA services.

MCP server for web scraping with natural language, enabling site crawling, page fetching, and extraction of emails, phones, links, and social profiles. Returns clean markdown, handles JavaScript rendering, and respects rate limits.

A powerful tool for fetching and extracting text content from web pages and APIs, supporting web scraping, REST API requests, and Google Custom Search integration.

MCP server for multi-engine web search and web page fetching, supporting parallel search, content extraction, and optional LLM-powered search summarization and deep search.

Enables web search via DuckDuckGo and web content fetching through the Model Context Protocol, allowing AI assistants to search and retrieve web page content.

Provides web search and page fetching capabilities via Serper.dev or DuckDuckGo, deployable on Render with SSE transport.

MCP server that fetches web pages and uses local Qwen models to answer questions about their content, with site-scoped search and map-reduce chunking for long pages.

Enables AI agents to rapidly drive and inspect real web pages through persistent browser sessions, using accessibility-tree snapshots and DevTools-grade diagnostics to identify and diagnose issues.

MCP server for querying Google Gemini Web from Claude Code and other MCP clients, using cookie-based authentication. Enables creating, continuing, listing, viewing, and deleting Gemini chat conversations with model and thinking mode control.

Enables AI agents to extract clean, structured web content (articles, tables, links, visual layouts) optimized for LLM token efficiency, with fast response times and optional JavaScript support.

Provides real-time system metrics and information through a Model Context Protocol interface, enabling access to CPU usage, memory statistics, disk information, network status, and running processes.

Enables AI agents to fetch, crawl, extract, and search web content through a unified set of tools, with automatic routing to specialized backends.

Multi-source web search MCP server with RRF fusion, 4-layer URL extraction, and provider health tracking.

A Node.js-based frontend security audit tool that performs comprehensive dependency security audits for both local projects and remote repositories. Generates detailed Markdown reports with vulnerability detection, risk assessment, and fix recommendations.

An MCP server that enables LLM coding agents to deploy web applications to AWS serverless infrastructure, supporting backend, frontend, and fullstack deployments.

Enables intelligent web searching using SearXNG with content crawling via Creeper, then summarizes webpage content using LLM to avoid token limit issues. Supports smart filtering with domain blacklist/whitelist and optional LLM-based relevance filtering.

Local MCP server for agents to search, structure, and export authorized WhatsApp Web conversations. Uses Playwright for DOM interaction, supports message search, export, media transcription, and controlled message sending.