Create an analysis suite and launch an AI crawler to map a web app's structure from a starting URL, combining suite creation and crawling in one step.
MIT
578,362 tools. Updated 2026-09-16 04:49
"A server for crawling web pages and finding subpages" matching MCP tools:
- Start multi-page web crawling to extract structured data with AI or convert content to markdown from a starting URL.MIT
- Extract new data from a completed crawl without re-crawling pages. Submit a new prompt and receive a jobId to track progress, paying only for AI tokens.MIT
- Search the web to find relevant web pages on any topic. Returns titles, URLs, snippets, and relevance scores.MIT
- Extract structured data from web pages by supplying URLs and a prompt, with options for web search and retries.MIT
- Analyze complex research questions by crawling multiple web sources and generating comprehensive LLM-based analysis.MIT
Matching MCP Servers
- FlicenseNot gradedqualityBmaintenanceMCP server that gives AI agents access to current web information through web_search and fetch_url tools, using Tavily, Brave, and DuckDuckGo providers with automatic fallback.-
- FlicenseAqualityDmaintenanceExposes two MCP tools (discover and execute) that enable agents to query an OpenAPI schema via natural language and execute matched API operations.2-
Matching MCP Connectors
tldr-pages community simplified man pages (cached 24h)
- encore-pagesOAuth
Renamed to co.encorehq/encore — connect at https://encore-pages-mcp.vercel.app/mcp
- Fetch documentation from a web URL by crawling HTML pages, with configurable depth, pattern, and directory restrictions to add a new source.MIT
- Capture web page screenshots for verifying updates, with automatic tiling for full pages and optimized processing for CLI tools.MIT
- Create new pages in Voog CMS by specifying title, slug, and language. Supports optional parent_id for subpages and node_id for multilingual translations.MIT
- List Productive pages filtered by project, creator, or parent hierarchy. Discover subpages and retrieve page metadata without loading full content.MIT
- Extract sanitized HTML from web pages to analyze structure, identify form fields, and plan automation selectors for web crawling operations.MIT
- Extract all links from a web page as absolute URLs, classified as internal or external, with anchor text and attributes. Supports scope filtering, deduplication, and result limits for crawling or link audits.MIT
- Submit URLs to Google Indexing API for fast crawling and indexing. Use to notify Google about new, updated, or deleted web pages.MIT
- Submit a URL to Yandex for re-crawling and re-indexing, helping your pages get updated in search results faster.MIT
- Search the web for real images, returning image URLs, titles, and source pages. Control search depth to refine results.MIT
- Discover site URLs from robots.txt and sitemap XML without crawling. Get a clean list of published pages for documentation, blog, or product inventory before extraction.Apache 2.0
- Extract and analyze hyperlinks from web pages, organizing URLs, anchor text, and contextual information into a structured format. Supports site mapping, SEO analysis, broken link checking, and targeted crawling preparation. Handles relative and absolute URLs with optional base URL and output limits.MIT
- Perform Google searches to retrieve organic web results for general questions, current events, fact-checking, and authoritative sources on any topic.MIT
- Fetch a web page and extract all hyperlinks with anchor text, classified as internal or external. Ideal for crawling and agent navigation.MIT
- Initiate web-based login to access Notlai notes. Starts a local authentication server and provides a URL for web-based authentication.MIT
- Retrieve all pages within a specified namespace hierarchy, including subpages, to discover and access related content.MIT