# Website Content Crawler

Crawls a list of URLs and returns page content as structured JSON. Runs as an
[Apify Actor](https://apify.com) and as an [MCP server](mcp-server/) so agents can call it directly.

Live demo: https://website-content-crawler.vercel.app

## What it costs to run

Pay per Event on Apify:

| Event | Price |
| --- | --- |
| Actor start | $0.00005 |
| Page record returned | $0.001 |

Platform compute is deducted before payout; see [PRICING.md](PRICING.md) for unit economics.

## Output

Every run returns this exact shape. The output schema enforces it.

```json
{
  "url": "https://example.com",
  "status": 200,
  "title": "Example Domain",
  "description": null,
  "h1": "Example Domain",
  "markdown": "# Example Domain\n\nThis domain is for use in documentation examples without needing permission. Avoid use in operations.\n\nLearn more\n\nLearn more (https://iana.org/domains/example)",
  "links": [
    "https://iana.org/domains/example"
  ],
  "wordCount": 23,
  "crawledAt": "2026-08-23T09:55:20.140Z",
  "error": null
}
```

## Layout

- `src/` — actor source (Crawlee Cheerio crawler)
- `.actor/` — Apify actor spec: `actor.json`, `INPUT_SCHEMA.json`, `output_schema.json`, `dataset_schema.json`, `Dockerfile`
- `mcp-server/` — MCP wrapper exposing the actor as the `crawl_website` tool
- `site/` + `tools/` — builds the live demo page from a real run record

## Run locally

```bash
npm install
npm test
```

MCP server:

```bash
cd mcp-server && npm install && node smoke.mjs
```

## Publishing to the Apify Store

When the account is active: `apify push`, price as Pay per Event, listing uses this README.