Extract structured data from web pages by supplying URLs and a prompt, with options for web search and retries.
MIT
204,980 tools. Last updated 2026-06-15 02:04
"Understanding Web Crawlers" matching MCP tools:
- Fetch and parse a website's robots.txt to identify which AI crawlers (e.g., GPTBot, CCBot) are allowed or blocked. Supports pre-crawl audits.MIT
- Loads database relationship metadata, showing tables, columns, and foreign key connections to assist with query planning and schema understanding.Apache 2.0
- Analyzes Apple API relationships to reveal inheritance, protocol conformances, and recommended alternatives for understanding how APIs work together.MIT
- Parses a user-agent string to extract browser, device, OS, and engine details. Identifies crawlers, bots, and potential attack user-agents.Apache 2.0
- Search the web to find relevant web pages on any topic. Returns titles, URLs, snippets, and relevance scores.MIT
Matching MCP Servers
- AlicenseAqualityBmaintenanceEnables web search and web fetch operations using Ollama's hosted APIs, allowing MCP clients to search the web and retrieve page content.Last updated2MIT
- FlicenseAqualityCmaintenanceA DOM-first, vision-second MCP server for webpage analysis, combining structural DOM extraction with MiniMax vision to provide accurate, context-rich page understanding for AI coding assistants.Last updated4
Matching MCP Connectors
Search the web and extract clean, readable text from webpages. Process multiple URLs at once to sp…
Create, edit, preview, publish, and manage web pages from MCP-capable AI clients.
- Execute deep web research on any topic. An AI agent searches and synthesizes information from across the web, returning a task ID for polling.Apache 2.0
- Discover available Bootstrap UI component categories with counts to identify what types of components you can use for web development.MIT
- Search the web using DuckDuckGo to find relevant information, web pages, and summaries for any query. Get results with titles, URLs, and detailed content to access real-time web data.Apache 2.0
- Analyze web page quality using Google's Lighthouse metrics to measure performance, accessibility, and SEO factors for optimization.Apache 2.0
- Search AI image prompts using semantic understanding to find visually and conceptually similar results, including image URLs for easy browsing and style picking.MIT
- Read full content of files using absolute paths. Extract specific line ranges when needed for line-number understanding.Apache 2.0
- Answers complex queries by combining file analysis, image understanding, web/X search, and code execution. Supports multi-agent collaboration and persistent history.MIT
- List Oracle Cloud Infrastructure AI/ML services including Generative AI, Vision, Speech, Language, and Document Understanding. Filter by service type or model.MIT
- Retrieve live AI bot traffic data including today's hit count, recent 50 hits with bot name, endpoint, and timestamp, and top-bots breakdown. Monitor which crawlers access AI ecosystem data and track distribution shifts.MIT
- Convert web pages to clean markdown format by extracting content, removing unnecessary elements, and ranking information for RAG applications.MIT
- List all tags in your Bear library as a hierarchical tree with note counts. Useful for understanding your tag structure and applying tags to untagged notes.Apache 2.0
- List all available Producer API actions and their corresponding tools for reference and understanding of the MCP server capabilities.MIT
- Search AI image prompts using semantic understanding to find visually and conceptually similar results. Browse image URLs for inspiration and style exploration.MIT
- Retrieve SearXNG instance configuration to discover available search engines, enabled categories, supported languages, and instance settings for understanding search capabilities.MIT