Discover all URLs on a website without scraping content. Use for sitemap audits, finding unindexed pages, and understanding site structure.
MIT
384,690 tools. Last updated 2026-08-03 16:21
"Understanding Web Scraping Techniques" matching MCP tools:
- Retrieve Web Application Firewall bypass techniques for specific vulnerability types such as XSS, SQLi, and SSRF.GPL 3.0
- Extract focused web content with optimized scraping, limited scrolls, and customizable image extraction for efficient data collection on Prysm MCP Server.MIT
- Extract comprehensive web content, including images, using deep scraping techniques with customizable parameters such as scroll depth, image size, and pagination. Output data to a specified directory for thorough analysis.MIT
- Extract web content from URLs using Bittensor's decentralized scraping network. Pay per use via blockchain for automated data collection.
- Clear stored scraping history to free up memory and ensure privacy by removing past web scraping records.MIT
Matching MCP Servers
- FlicenseAqualityAmaintenanceIntegrates Xiaomi MiMo's multimodal API to enable understanding of images, audio, and video through natural language prompts.Last updated3
- FlicenseAqualityDmaintenanceEnables retrieval and cleaning of official documentation content for popular AI/Python libraries (uv, langchain, openai, llama-index) through web scraping and LLM-powered content extraction. Uses Serper API for search and Groq API to clean HTML into readable text with source attribution.Last updated12
Matching MCP Connectors
Generic URL crawl + HTML extraction — fallback for sites without dedicated MCPs.
40+ web scraping tools from Firecrawl, Bright Data, Jina, Olostep, ScrapeGraph, Notte, and Riveter. Scrape, crawl, screenshot, and extract from any website. Starts at $0.01/call. Get your API key at app.xpay.sh or xpay.tools
- Retrieve French tax information from cached data when web scraping fails, providing reliable access to official tax brackets and calculations for residents.
- Search Apify and Crawlee documentation using full-text search to find relevant pages about platform features, SDKs, CLI, REST API, and web scraping libraries.MIT
- Get instructions for obtaining an API key and using a web-access gateway that connects to 24 search, scraping, browser, and voice providers at provider-metered prices.MIT
- Perform full reconnaissance on a target URL using HTTP headers, TLS, DNS, and web technology detection. Adjust depth for quick, standard, or comprehensive analysis.AGPL 3.0
- Test web application file upload security by attempting to upload PHP shells using Content-Type bypass techniques and alternative extensions to identify vulnerabilities.MIT
- Fingerprint web servers and detect technologies, analytics, CMS, and APIs by analyzing HTTP headers, cookies, caching, compression, and redirects.AGPL 3.0
- Check which censorship evasion techniques (DoH, ECH, domain fronting, Tor bridges) currently work or fail in any country.MIT
- Analyze patterns in Bitcoin Stamps to identify common libraries, frameworks, and coding techniques used across the ecosystem.MIT
- Search academic papers on Google Scholar by query, author, or publication year to find relevant research publications.MIT
- Search the MITRE ATLAS catalog of AI/ML attack techniques by keyword, tactic, or maturity to discover techniques matching your threat model.MIT
- Check real-time train ticket prices for routes in Spain. Uses web scraping to provide actual pricing with pagination support.MIT
- Check remaining scrape credits to confirm API key balance before running large scraping jobs.MIT
- Search MITRE ATT&CK techniques by keyword in their name or description. Returns matching techniques with IDs and optional descriptions for enterprise, mobile, or ICS domains.Apache 2.0
- Extract structured data from web pages by specifying instructions and a JSON schema. Ideal for scraping specific content like product details, articles, or lists.Apache 2.0