A
licenseB
qualityD
maintenanceEnables LLMs to retrieve and process web content by fetching URLs and converting HTML to markdown format. Supports chunked reading of large pages and can access both public websites and local networks.
1
MIT
website-content-crawler
by dev55acc-ai
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@website-content-crawlerCrawl https://example.com and extract title, links, and markdown."
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Website Content Crawler
Crawls a list of URLs and returns page content as structured JSON. Runs as an Apify Actor and as an MCP server so agents can call it directly.
Live demo: https://website-content-crawler.vercel.app
What it costs to run
Pay per Event on Apify:
Event | Price |
Actor start | $0.00005 |
Page record returned | $0.001 |
Platform compute is deducted before payout; see PRICING.md for unit economics.
Related MCP server: cleanfetch
Output
Every run returns this exact shape. The output schema enforces it.
{
"url": "https://example.com",
"status": 200,
"title": "Example Domain",
"description": null,
"h1": "Example Domain",
"markdown": "# Example Domain\n\nThis domain is for use in documentation examples without needing permission. Avoid use in operations.\n\nLearn more\n\nLearn more (https://iana.org/domains/example)",
"links": [
"https://iana.org/domains/example"
],
"wordCount": 23,
"crawledAt": "2026-08-23T09:55:20.140Z",
"error": null
}Layout
src/— actor source (Crawlee Cheerio crawler).actor/— Apify actor spec:actor.json,INPUT_SCHEMA.json,output_schema.json,dataset_schema.json,Dockerfilemcp-server/— MCP wrapper exposing the actor as thecrawl_websitetoolsite/+tools/— builds the live demo page from a real run record
Run locally
npm install
npm testMCP server:
cd mcp-server && npm install && node smoke.mjsPublishing to the Apify Store
When the account is active: apify push, price as Pay per Event, listing uses this README.
This server cannot be deployed
Maintenance
ActivitySlowing
ResponsivenessNo issues
Related MCP Connectors
Unblocking and fresh web data for agents: URL to Markdown, YouTube, Maps, Amazon, jobs. Pay per call
Web scraping for agents. Point it at a URL and it returns the page as clean markdown, JavaScript-rendered pages included. Point it at a site and it maps the URLs or crawls the section you need in the background, a few pages at a time so results fit in the conversation. Search the web and read full pages, extract fields with a JSON schema you define (validated, never invented), read a store's catalogue or a blog's posts from the platform's own feed, and check whether a page has changed. Failed requests cost nothing. The free plan includes 1,500 credits a month.
Web pages to Markdown, metadata and small crawls for agents. Free daily calls, then USDC x402/MPP.
Read a URL as clean markdown, screenshot a website, url to PDF. Web access for agents, no signup.
Related MCP Servers
- AlicenseAqualityCmaintenanceEnables AI agents to read web pages reliably, returning clean markdown content, hyperlinks, and metadata without navigation or ad noise.39 npmMIT
- AlicenseAqualityDmaintenanceEnables AI agents to fetch any web page as clean markdown or screenshot it, turning URLs into LLM-ready context.27 npmMIT
- AlicenseNot gradedqualityAmaintenanceEnables AI agents to crawl and scrape websites, converting HTML to clean Markdown and structured metadata with support for JavaScript rendering, bot evasion, and SSRF protection.182MIT