"Web scraping and crawling tools" matching MCP servers:

Network diagnostics tool for AI agents that provides DNS lookup, IP geolocation, SSL certificate inspection, WHOIS queries, and HTTP latency testing.

Enables live website health checks including TLS, HTTPS, and security headers, returning an A-F grade with specific fixes.

Governed AI-ops for Ceph, providing root-cause health analysis and guarded destructive operations via a built-in governance harness with risk tiers, audit, and undo recording.

MCP server that exposes 31 OSINT checks from Lissy93/web-check as tools for website analysis, including SSL, DNS, headers, WHOIS, and security presets. Enables natural-language-driven web recon and health checks.

Enables agents to monitor websites for content changes with persistent baselines, durable notifications, and explicit review acknowledgement, using direct HTTPS fetches or isolated Playwright browser rendering for dynamic pages.

Enables Claude to query Bing Webmaster Tools performance, crawl, sitemap, backlink, and keyword data in plain language through the official Bing Webmaster API. It is read-only and runs locally using your own API key.

Enables AI coding agents to retrieve real-user performance metrics and 28-day field trends, run Lighthouse lab audits, and access remediation playbooks for LCP, INP, and CLS.

Provides MCP tools for governed multi-vendor network device operations, including configuration management (backup, diff, merge, replace, rollback) and read-only queries (facts, interfaces, BGP, LLDP, ARP) via NAPALM, with optional NetBox source-of-truth integration.

Enables agents to inspect localhost web applications through Chrome DevTools Protocol by listing Chrome targets, attaching to a page, capturing and querying network, console, navigation, and WebSocket events, and managing ephemeral sessions.

An MCP server for managing and monitoring Docker, Docker Compose, and Kubernetes environments alongside Azure Application Insights. It enables advanced log filtering, container lifecycle management, and querying of cloud application traces and metrics.

Enables web performance analysis using Google PageSpeed Insights and Chrome UX Report APIs, providing compact metrics and diagnostics without local browser automation.

Free remote MCP for Core Web Vitals metrics by CMS, CDN, and framework (Chrome field data + multi-site crawl). No auth. Endpoint: https://www.corewebvitals.io/api/state-of-cwv/mcp

Provides advanced system monitoring and file search capabilities for macOS, allowing users to track performance metrics and perform enhanced file searches with content analysis and tagging features.

Enables interaction with Google Cloud services including billing cost analysis, log querying, and metrics monitoring through natural language commands. Provides comprehensive tools for managing GCP resources, analyzing costs, detecting anomalies, and retrieving operational insights.

MCP server for infrastructure discovery and remote management, enabling SSH command execution, file transfer, log tailing, and machine/service inventory with a companion web dashboard.

Enables Claude Desktop to access the hosted Web Speed API through MCP, requiring only an API key for setup.

Query your domains, sender compliance status, and Gmail traffic metrics from Claude, Cursor, or any MCP client using the Gmail Postmaster Tools API.

Provides investigation tools for Cloudflare CCF data in Microsoft Sentinel, enabling security posture assessment, threat detection, and IP reputation analysis through customizable MCP tools.

Enables auditing of websites for performance, SEO, accessibility, security, and mobile readiness, with tools to validate URLs, run page audits, save results, and retrieve reports.

A comprehensive MCP server providing advanced Redmine project analytics, web automation via Playwright, and iTunes music integration. It enables detailed tracking of sprint metrics, bug counts, and team performance through a token-optimized tool architecture.