Skip to main content
Glama
Sign In
deenesjakoruzh
website-content-crawler

website-content-crawler

by Nimos-Websites
OverviewSchemaSourceRelated ServersDiscussionsAdmin
JavaScript
Remote

Website Content Crawler

Crawls a list of URLs and returns page content as structured JSON. Runs as an Apify Actor and as an MCP server so agents can call it directly.

Live demo: https://website-content-crawler.vercel.app

What it costs to run

Pay per Event on Apify:

Event

Price

Actor start

$0.00005

Page record returned

$0.001

Platform compute is deducted before payout; see PRICING.md for unit economics.

Related MCP server: web-reader-mcp

Output

Every run returns this exact shape. The output schema enforces it.

{
  "url": "https://example.com",
  "status": 200,
  "title": "Example Domain",
  "description": null,
  "h1": "Example Domain",
  "markdown": "# Example Domain\n\nThis domain is for use in documentation examples without needing permission. Avoid use in operations.\n\nLearn more\n\nLearn more (https://iana.org/domains/example)",
  "links": [
    "https://iana.org/domains/example"
  ],
  "wordCount": 23,
  "crawledAt": "2026-08-23T09:55:20.140Z",
  "error": null
}

Layout

  • src/ — actor source (Crawlee Cheerio crawler)

  • .actor/ — Apify actor spec: actor.json, INPUT_SCHEMA.json, output_schema.json, dataset_schema.json, Dockerfile

  • mcp-server/ — MCP wrapper exposing the actor as the crawl_website tool

  • site/ + tools/ — builds the live demo page from a real run record

Run locally

npm install
npm test

MCP server:

cd mcp-server && npm install && node smoke.mjs

Publishing to the Apify Store

When the account is active: apify push, price as Pay per Event, listing uses this README.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    This server cannot be deployed

    One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

    Resources

    Related MCP Connectors

    Related MCP Servers

    • web-reader-mcp

      arhancanli
      A
      license
      A
      quality
      B
      maintenance
      Enables reading web pages and PDFs as clean Markdown, extracting main content and letting agents query for relevant passages, outline sections, search in-page, and list links while respecting robots.txt and blocking private addresses.
      4
      1
      MIT
    • JsonLink MCP Server

      JsonLink MCP Serverofficial

      mauriblint
      A
      license
      A
      quality
      C
      maintenance
      Enables AI agents to read any web page as clean Markdown, pull title/description/preview metadata, capture desktop, mobile, or full-page screenshots, and list every link or image on a URL. JavaScript-heavy pages are rendered in a real browser and blocked requests retried through a proxy, so agents can reach content they'd otherwise miss.
      5
      61 npm
      MIT