"Web scraping and crawling tools" matching MCP servers:

MCP server that exposes 31 OSINT checks from Lissy93/web-check as tools for website analysis, including SSL, DNS, headers, WHOIS, and security presets. Enables natural-language-driven web recon and health checks.

An MCP server that integrates multiple security and reconnaissance tools (Nmap, Cariddi, ParamSpider, Metasploit, web scraping) for AI systems, enabling automated network scanning, API discovery, vulnerability testing, and remote access via ngrok.

A production-ready HTTP API server for security testing and reconnaissance, integrating nmap, cariddi, paramspider, metasploit, and web scraping via FastAPI. It supports optional ngrok tunneling and exposes tools through REST endpoints.

Enables agents to inspect web pages for Cloudflare Turnstile challenges without an API key, and clear Turnstile or WAF challenges using a SolveGate API key.

An MCP server that lets an AI agent probe a live URL and confirm whether sensitive files (e.g., .git, .env, source maps) are genuinely served by fetching and validating the content, avoiding false positives.

Free deterministic security scan of a public git repository (GitHub, GitLab, Codeberg, Bitbucket): vulnerable dependencies via OSV.dev, secret patterns, and config lint, returned as structured JSON. Tools: scan_repository(url), audit_pricing(). Runs locally over stdio (python3 web/mcp_stdio.py or the Dockerfile) and is also hosted at https://project-feldspar.com/mcp. MIT licence, stdlib-only. Buil

Runs WCAG accessibility, SEO, performance, and security audits on URLs or raw HTML via SiteLint Auditor. Enables LLM agents to audit web pages and check WCAG criteria through MCP tools.

Enables policy-first defensive security operations for MCP, providing repository and web-security analysis with controlled authorization, scoped execution, and auditability.

Enables Solidity developers and security auditors to decode EVM calldata, pinpoint revert locations in debug traces, calculate storage slot layouts, and scan contracts for reentrancy and Yul assembly issues.

Chrome browser JS reverse engineering MCP tool for CTF/Web security, enabling dynamic analysis and restoration of login encryption logic.

Enables AI assistants to perform web content discovery scans using feroxbuster on a remote system via SSH, with support for recursive scanning, filtering, and background execution.

Enables AI assistants to natively execute and manage 600+ Kali Linux security, penetration testing, and forensic tools, including search, help documentation, structured scanners, and workspace management.

A comprehensive security testing MCP server providing 51 tools for penetration testing, network forensics, memory analysis, and vulnerability assessment. It enables automated security audits and technical investigations across web applications, cloud environments, and network captures.

Enables AI assistants to execute security testing tools on a Kali Linux machine over SSH, including reconnaissance, web app scanning, and static/dynamic analysis.

A comprehensive MCP server for web security reconnaissance, providing tools for passive and active information gathering, DNS analysis, network scanning, and web application analysis.

Enables AI applications to invoke 58 popular Kali Linux security tools for network scanning, web exploitation, password attacks, reconnaissance, Metasploit, evasion, forensics, post-exploitation, and miscellaneous tasks through the Model Context Protocol.

MCP server that enables LLMs to operate Acunetix/Invicti web vulnerability scanners through 15 tools, covering authentication, target management, scanning, vulnerability retrieval, and reporting via GraphQL and REST APIs.

Integrates ProjectDiscovery security tools to perform automated reconnaissance, subdomain discovery, and vulnerability scanning. It enables comprehensive bug hunting workflows by chaining tools like subfinder, naabu, and nuclei into a unified pipeline.

Enables deep security auditing of web applications directly from AI IDEs including Cursor and Claude Code. Scans URLs for vulnerabilities, returns security scores with SHIP/BLOCK verdicts, and provides specific fix prompts for remediation.

A comprehensive HTTP client MCP server for security testing, API testing, and web automation that provides full-featured HTTP tools with detailed logging capabilities.