Skip to main content
Glama
deenesjakoruzh
Nimos-Websites

website-content-crawler

by Nimos-Websites
OverviewSchemaSourceRelated ServersDiscussionsAdmin
JavaScript
Remote

Website Content Crawler

Crawls a list of URLs and returns page content as structured JSON. Runs as an Apify Actor and as an MCP server so agents can call it directly.

Live demo: https://website-content-crawler.vercel.app

What it costs to run

Pay per Event on Apify:

Event

Price

Actor start

$0.00005

Page record returned

$0.001

Platform compute is deducted before payout; see PRICING.md for unit economics.

Related MCP server: web-reader-mcp

Output

Every run returns this exact shape. The output schema enforces it.

{
  "url": "https://example.com",
  "status": 200,
  "title": "Example Domain",
  "description": null,
  "h1": "Example Domain",
  "markdown": "# Example Domain\n\nThis domain is for use in documentation examples without needing permission. Avoid use in operations.\n\nLearn more\n\nLearn more (https://iana.org/domains/example)",
  "links": [
    "https://iana.org/domains/example"
  ],
  "wordCount": 23,
  "crawledAt": "2026-08-23T09:55:20.140Z",
  "error": null
}

Layout

  • src/ — actor source (Crawlee Cheerio crawler)

  • .actor/ — Apify actor spec: actor.json, INPUT_SCHEMA.json, output_schema.json, dataset_schema.json, Dockerfile

  • mcp-server/ — MCP wrapper exposing the actor as the crawl_website tool

  • site/ + tools/ — builds the live demo page from a real run record

Run locally

npm install
npm test

MCP server:

cd mcp-server && npm install && node smoke.mjs

Publishing to the Apify Store

When the account is active: apify push, price as Pay per Event, listing uses this README.

This server cannot be deployed

Let your AI agent see the webWeb-to-screenshot MCP for Claude, Cursor and Codex. No cookie banners. Free to start.snap-render.comAd

Maintenance

ActivitySlowing
ResponsivenessNo issues

Resources

Related MCP Connectors

  • agdataA
    dev.shveik

    Unblocking and fresh web data for agents: URL to Markdown, YouTube, Maps, Amazon, jobs. Pay per call

    Browser AutomationWeb Scraping
    311
  • SnoopScanA
    com.snoopscan

    Web scraping for agents. Point it at a URL and it returns the page as clean markdown, JavaScript-rendered pages included. Point it at a site and it maps the URLs or crawls the section you need in the background, a few pages at a time so results fit in the conversation. Search the web and read full pages, extract fields with a JSON schema you define (validated, never invented), read a store's catalogue or a blog's posts from the platform's own feed, and check whether a page has changed. Failed requests cost nothing. The free plan includes 1,500 credits a month.

    Research & DataSearchWeb Scraping
    341
  • PageWireA
    dev.pagewire

    Web pages to Markdown, metadata and small crawls for agents. Free daily calls, then USDC x402/MPP.

    Research & DataSearchWeb Scraping
    31
  • PagewatchA
    dev.pagelens

    Read a URL as clean markdown, screenshot a website, url to PDF. Web access for agents, no signup.

    Browser AutomationWeb Scraping
    2

Related MCP Servers

  • web-reader-mcp

    arhancanli
    A
    license
    A
    quality
    B
    maintenance
    Enables reading web pages and PDFs as clean Markdown, extracting main content and letting agents query for relevant passages, outline sections, search in-page, and list links while respecting robots.txt and blocking private addresses.
    4
    1
    MIT
  • JsonLink MCP Server by mauriblint

    JsonLink MCP Serverofficial

    mauriblint
    A
    license
    A
    quality
    C
    maintenance
    Enables AI agents to read any web page as clean Markdown, pull title/description/preview metadata, capture desktop, mobile, or full-page screenshots, and list every link or image on a URL. JavaScript-heavy pages are rendered in a real browser and blocked requests retried through a proxy, so agents can reach content they'd otherwise miss.
    5
    MIT