"Web crawler for sequential URL content retrieval" matching MCP servers:

Enables autonomous web browsing and extraction by canonicalizing URLs, detecting anti-crawler traps such as circular paths and infinite pagination, pruning DOM content, mapping forms, converting tables to JSON, and evaluating agent trajectories.

Enables web crawling and content extraction from web pages, supporting multiple output formats like text, markdown, XML, and JSON, with robots.txt compliance and rate limiting.

Enables AI models to interact with IBM FileNet Content Manager for document management, folder operations, metadata handling, search, and content extraction through a standardized interface.

Bridges AI agents with watsonx.data document libraries, enabling natural language queries and intelligent retrieval from document libraries.

Enables web research through multi-provider search, documentation lookup, public code search, and clean Markdown extraction from static or JavaScript-rendered pages via five read-only tools.

An open-source, local-first MCP server for automated Schema.org JSON-LD generation, GA4 metric validation, /llm.txt auditing, and content stickiness analytics.

Enables managing a social media calendar workflow with AI-generated images/videos, including generating media, uploading to Google Drive, and downloading the latest calendar draft.

Enables deterministic auditing of RAG retrieval faithfulness and hallucination scores, providing structured JSON output for AI agents and MCP-compliant clients.

Ultra-lightweight headless browser for AI agents. Provides MCP tools for navigating URLs, extracting structured content, and building autonomous agent workflows.

MCP server that parses web pages into clean markdown via Crawl4AI's HTTP API, exposing parse_page and health-check tools with configurable proxy and timeout.

Enables AI assistants to fetch public web pages as readable text or HTML, with caching, live refresh, and parallel job handling.

Implementation of an MCP server for the RAG Web Browser Actor. This Actor serves as a web browser for large language models (LLMs) and RAG pipelines, similar to a web search in ChatGPT.

Enables agents to read any web page or document as markdown, html, links, screenshots or extracted JSON, map and crawl entire sites, and drive a persistent browser session that stays open between calls so pages can be inspected and clicked without reloading. Callers pay per request with a wallet over x402 — no API key or signup — and every route answers its own free GET with the fields it expects.

Enables autonomous scheduling of creator content deliverable calendars and embargo wave campaigns through natural language, integrating with MCP-compliant clients for deterministic, zero-dependency execution.

Enables AI agents to check a list of websites by running an Apify Actor through one tool, returning per-site tech stack, on-page SEO issues and score, robots.txt rules for AI crawlers, sitemap, llms.txt presence, and an AI-readiness score. Supports local stdio operation with an Apify token for tool calls.

Enables coding agents to drive a real Playwright-backed browser session, navigating, observing, interacting, and capturing proof on web pages via MCP tools across stdio and Streamable HTTP transports.

A client-side MCP proxy that injects OAuth 2.0 bearer tokens or API keys into MCP requests, enabling MCP clients to connect to OAuth/API-key-protected MCP servers like Amazon Bedrock AgentCore Gateway. It automatically fetches and refreshes credentials using AgentCore Identity or static values.

MCP server for the Humanread novel publishing platform, enabling AI agents to upload, format, preview, review, translate, and publish novels in Markdown or safe HTML.

Enables autonomous agents to inspect live web pages, prune noisy DOM content down to semantic essentials, and bind each form field to agent state by matching input attributes, autocomplete hints, and regex validations. It also converts HTML/Markdown tables into type-inferred JSON records and scores agent action trajectories for precision, recall, and edit distance.

Enables Directus users to interact with their data in LLMs, allowing content editors and data analysts to manage content, query collections, and perform data operations using natural language.