"Web crawler for sequential URL content retrieval" matching MCP servers:

MCP server that fetches raw HTML content from a given URL to provide web context to LLMs.

Enables AI agents to fetch, crawl, extract, and search web content through a unified set of tools, with automatic routing to specialized backends.

A minimal MCP server that crawls web pages using crawl4ai and returns content as markdown, deployable to Cloud Run.

An open-source MCP server providing AI agents with neural web search via Exa and tiered web fetch (Exa, local browser, Firecrawl) as a drop-in replacement for built-in web tools, preserving provenance and guarding against SSRF.

Enables MCP-capable agents to perform reliable web search and content fetching through a single server that automatically fails over across multiple providers. It handles keyed or keyless authentication automatically and returns clean Markdown or text results.

Enables AI agents to read any web page as clean Markdown and inspect its metadata, structure and technology stack, plus check redirects, TLS certificates, security headers, robots.txt, sitemaps, feeds, domains, SEO, broken links, email authentication (SPF/DMARC/DKIM) and SSO (SAML/OIDC). It also provides developer utilities such as cron, regex, color, Unicode and time-zone tools along with Hyperliquid perpetuals market data, all hosted remotely on Apify with pay-per-call pricing.

Enables autonomous web browsing and extraction by canonicalizing URLs, detecting anti-crawler traps such as circular paths and infinite pagination, pruning DOM content, mapping forms, converting tables to JSON, and evaluating agent trajectories.

Enables autonomous web agents to canonicalize URLs, detect anti-crawler traps such as circular paths and infinite pagination, prune DOM tokens, auto-map forms, transform tables into JSON, and evaluate action trajectories through MCP.

Simple sequential thinking MCP in python

Enables AI agents to parse PDFs and search local document context through multimodal extraction and vector-based retrieval, all exposed as MCP tools.

Extract content from URLs, documents, videos, and audio files using intelligent auto-engine selection. Supports web pages, PDFs, Word docs, YouTube transcripts, and more with structured JSON responses.

Enables site-wide crawling of internal links with BFS traversal, time/title/URL filtering, and page operations like summarization and link clicking, with results persisted to SQLite.

Enables retrieval-augmented generation by embedding queries with a chosen provider (e.g., OpenAI) and searching supported vector stores (Pinecone, pgvector) to return relevant content.

Exposes Futurepedia AI tool crawling and universal web page parsing as MCP tools for assistants.

An MCP server for crawling websites and extracting comprehensive data including images, SEO metadata, security headers, and business intelligence. It features twelve distinct extraction modes to perform detailed audits for e-commerce sites and general web analysis.

An evaluation harness that probes MCP-based retrieval servers for calibration, relevance, coverage, citation integrity, and more, also exposing the probes as MCP tools for assistants and CI agents.

Enables AI to autonomously search, filter, and download images from the web using natural language commands, with intelligent scoring and local storage.

MCP server for intelligent web crawling using a ReAct agent. Adaptively explores websites and returns structured analysis.

Enables AI agents to analyze a piece of content across 11 dimensions and receive ranked, scored recommendations among six cross-media output formats (PPT, infographic, comic, short video script, newsletter, reusable skill), then generate dispatch prompts for the chosen format. It also exposes the format catalog and profile schema as read-only tools, supporting batch routing of an entire content library.

Enables AI clients to manage a self-hosted content hub: create, edit, publish, and update articles across multiple platforms, plus AI writing and account management.