Search the web and optionally scrape result pages to extract markdown, HTML, links, or screenshots. Supports web, news, and image searches with filters for language, country, and time.
Server Side Public , v 1
470,139 tools. Updated 2026-08-21 22:05
"Understanding Web Crawlers" matching MCP tools:
- Search the web via DuckDuckGo, crawl pages for content, and fetch news articles. Save results to disk using harvest to avoid high context costs.MIT
- Analyze a domain's robots.txt to determine which AI crawlers (like GPTBot, CCBot) are allowed or blocked. Provides per-crawler status.MIT
- Fetch a public HTTPS URL and return a condensed prose summary with key points, providing a quick understanding of the web page.MIT
- Generates detailed textual descriptions of one or more images for text-only models, enabling scene/UI understanding, OCR, comparison, and structured extraction from multi-image uploads.MIT
- Retrieve web search results from DuckDuckGo and AI-generated answers. Switch between standard results and AI responses with customizable backends.Apache 2.0
Matching MCP Servers
- FlicenseAqualityAmaintenanceIntegrates Xiaomi MiMo's multimodal API to enable understanding of images, audio, and video through natural language prompts.3
- AlicenseAqualityAmaintenanceEnables web search and web fetch operations using Ollama's hosted APIs, allowing MCP clients to search the web and retrieve page content.2MIT
Matching MCP Connectors
Google Web Search: Google Web Search API. Search the world’s information, including webpages.
Web tools for agents: fetch URL as markdown (free MCP) + x402 scrape, links, AI JSON, snapshot.
- Discover all URLs on a website without scraping content. Use for sitemap audits, finding unindexed pages, and understanding site structure.MIT
- Analyze website structure by mapping URLs to discover content organization, navigation paths, and site architecture for audits and content discovery.MIT
- Search the web to find relevant web pages on any topic. Returns titles, URLs, snippets, and relevance scores.MIT
- Discover available Bootstrap UI component categories with counts to identify what types of components you can use for web development.MIT
- Analyze UI code or components to measure mental resource requirements for understanding interfaces across web, mobile, desktop, voice, and AR/VR platforms.MIT
- Retrieve README files from GitHub repositories to access project overviews, installation guides, and usage examples for understanding codebases.MIT
- Retrieve web pages or files as your real browser, even when generic crawlers are blocked; returns clean Markdown for HTML, preserves JSON/text, and saves PDFs/binaries locally.MIT
- Discover web platform features added in a specific browser version. Filter by category to see new APIs, CSS, or HTML capabilities.MIT
- Retrieve recent file system changes from watched directories to track file creations, modifications, deletions, and moves for understanding recent work.MIT
- Check robots.txt to learn which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, etc.) are allowed or blocked for a given URL.MIT
- Analyze code files to identify top-level symbols and their structure, providing a quick overview for understanding new codebases or files.MIT
- Retrieve live AI bot traffic data including today's hit count, recent 50 hits with bot name, endpoint, and timestamp, and top-bots breakdown. Monitor which crawlers access AI ecosystem data and track distribution shifts.MIT
- List all tags in your Bear library as a hierarchical tree with note counts. Useful for understanding your tag structure and applying tags to untagged notes.Apache 2.0
- List Oracle Cloud Infrastructure AI/ML services including Generative AI, Vision, Speech, Language, and Document Understanding. Filter by service type or model.Apache 2.0