Website Content Crawler

Crawls a list of URLs and returns page content as structured JSON. Runs as an Apify Actor and as an MCP server so agents can call it directly.

Live demo: https://website-content-crawler.vercel.app

What it costs to run

Pay per Event on Apify:

Platform compute is deducted before payout; see PRICING.md for unit economics.

Related MCP server: cleanfetch

Output

Every run returns this exact shape. The output schema enforces it.

{
  "url": "https://example.com",
  "status": 200,
  "title": "Example Domain",
  "description": null,
  "h1": "Example Domain",
  "markdown": "# Example Domain\n\nThis domain is for use in documentation examples without needing permission. Avoid use in operations.\n\nLearn more\n\nLearn more (https://iana.org/domains/example)",
  "links": [
    "https://iana.org/domains/example"
  ],
  "wordCount": 23,
  "crawledAt": "2026-08-23T09:55:20.140Z",
  "error": null
}

Layout

• src/ — actor source (Crawlee Cheerio crawler)

• .actor/ — Apify actor spec: actor.json, INPUT_SCHEMA.json, output_schema.json, dataset_schema.json, Dockerfile

• mcp-server/ — MCP wrapper exposing the actor as the crawl_website tool

• site/ + tools/ — builds the live demo page from a real run record

Run locally

npm install
npm test

MCP server:

cd mcp-server && npm install && node smoke.mjs

Publishing to the Apify Store

When the account is active: apify push, price as Pay per Event, listing uses this README.