"Information about web crawlers or scraping techniques" matching MCP servers:

An MCP server that lets an AI agent probe a live URL and confirm whether sensitive files (e.g., .git, .env, source maps) are genuinely served by fetching and validating the content, avoiding false positives.

MCP server that exposes 31 OSINT checks from Lissy93/web-check as tools for website analysis, including SSL, DNS, headers, WHOIS, and security presets. Enables natural-language-driven web recon and health checks.

Connects AI coding assistants to Snyk API & Web for onboarding scan targets, configuring authentication, running DAST scans, and triaging findings through natural language.

Enables proof-backed code audit via MCP tools for AST-based code maps, symbol lookup, callers, audit commands, and persistent verdict memory.

Enables agents to inspect web pages for Cloudflare Turnstile challenges without an API key, and clear Turnstile or WAF challenges using a SolveGate API key.

Free deterministic security scan of a public git repository (GitHub, GitLab, Codeberg, Bitbucket): vulnerable dependencies via OSV.dev, secret patterns, and config lint, returned as structured JSON. Tools: scan_repository(url), audit_pricing(). Runs locally over stdio (python3 web/mcp_stdio.py or the Dockerfile) and is also hosted at https://project-feldspar.com/mcp. MIT licence, stdlib-only. Buil

A read-only MCP server for the Qualys PCI Merchant API that lets LLM assistants answer questions about PCI compliance posture, such as which hosts are failing PCI or listing high findings.

Enables coding agents to perform network diagnostics and lookups, including subnet calculations, port and MAC vendor information, DNS and blocklist checks, TLS certificate inspection, and public IP discovery, all without needing an account or sending telemetry.

Provides Claude Code with access to a comprehensive bug bounty knowledge base including techniques, payloads, wordlists, and real-world reports through 14 tools for searching, retrieving payloads, and assessing report quality.

Runs WCAG accessibility, SEO, performance, and security audits on URLs or raw HTML via SiteLint Auditor. Enables LLM agents to audit web pages and check WCAG criteria through MCP tools.

Enables users to retrieve and display CVE vulnerability information from the National Vulnerability Database (NVD) with support for keyword search and detailed lookup.

A universal digital fingerprinting MCP server that combines 103 techniques across TCP, TLS, SSH, HTTP, DNS, and more into a single interface for AI agents, enabling full-spectrum fingerprinting on demand.

Chrome browser JS reverse engineering MCP tool for CTF/Web security, enabling dynamic analysis and restoration of login encryption logic.

Enables AI assistants to perform web content discovery scans using feroxbuster on a remote system via SSH, with support for recursive scanning, filtering, and background execution.

An MCP server for identifying SQL injection vulnerabilities in web applications using various techniques like error-based, time-based, and union-based scanning. It supports bulk URL processing, WAF bypass strategies, and authenticated testing across multiple database systems.

Enables LLMs to conduct passive attack-surface research by collecting public information, referencing saved reports, and verifying evidence through MCP tools.

A comprehensive MCP server for web security reconnaissance, providing tools for passive and active information gathering, DNS analysis, network scanning, and web application analysis.

A comprehensive HTTP client MCP server for security testing, API testing, and web automation that provides full-featured HTTP tools with detailed logging capabilities.

Talk to your Flipper Zero from Claude Code — or any MCP client. Storage, app deployment, screen capture, button automation, JavaScript execution on the device, and honest answers about what is nearby.

Provides access to Shodan API functionality, enabling AI assistants to query information about internet-connected devices for cybersecurity research and threat intelligence.