Skip to main content
Glama
deenesjakoruzh
dev55acc-ai

website-content-crawler

by dev55acc-ai
OverviewSchemaSourceRelated ServersDiscussionsAdmin
JavaScript
Remote

Website Content Crawler

爬取一个 URL 列表，并将页面内容作为结构化 JSON 返回。以 Apify Actor 和 MCP server 方式运行，以便 Agent 可以直接调用它。

在线演示：https://website-content-crawler.vercel.app

运行成本

在 Apify 上按事件付费：

事件

价格

Actor 启动

$0.00005

返回页面记录

$0.001

平台计算费用会在收益结算前扣除；单位经济效益参见 PRICING.md。

Related MCP server: cleanfetch

输出

每次运行都会返回以下完全相同的结构。输出 schema 会强制保证该结构。

{
  "url": "https://example.com",
  "status": 200,
  "title": "Example Domain",
  "description": null,
  "h1": "Example Domain",
  "markdown": "# Example Domain\n\nThis domain is for use in documentation examples without needing permission. Avoid use in operations.\n\nLearn more\n\nLearn more (https://iana.org/domains/example)",
  "links": [
    "https://iana.org/domains/example"
  ],
  "wordCount": 23,
  "crawledAt": "2026-08-23T09:55:20.140Z",
  "error": null
}

目录结构

  • src/ — Actor 源代码（Crawlee Cheerio 爬虫）

  • .actor/ — Apify Actor 规范：actor.json、INPUT_SCHEMA.json、output_schema.json、dataset_schema.json、Dockerfile

  • mcp-server/ — MCP 包装器，将 Actor 暴露为 crawl_website 工具

  • site/ + tools/ — 根据真实运行记录构建在线演示页面

本地运行

npm install
npm test

MCP 服务器：

cd mcp-server && npm install && node smoke.mjs

发布到 Apify Store

当账户激活后：apify push，按事件付费，listing 使用本 README。

This server cannot be deployed

Let your AI agent see the webWeb-to-screenshot MCP for Claude, Cursor and Codex. No cookie banners. Free to start.snap-render.comAd

Maintenance

ActivitySlowing
ResponsivenessNo issues

Resources

Related MCP Connectors

  • agdataA
    dev.shveik

    Unblocking and fresh web data for agents: URL to Markdown, YouTube, Maps, Amazon, jobs. Pay per call

    Browser AutomationWeb Scraping
    311
  • SnoopScanA
    com.snoopscan

    Web scraping for agents. Point it at a URL and it returns the page as clean markdown, JavaScript-rendered pages included. Point it at a site and it maps the URLs or crawls the section you need in the background, a few pages at a time so results fit in the conversation. Search the web and read full pages, extract fields with a JSON schema you define (validated, never invented), read a store's catalogue or a blog's posts from the platform's own feed, and check whether a page has changed. Failed requests cost nothing. The free plan includes 1,500 credits a month.

    Research & DataSearchWeb Scraping
    301
  • PageWireA
    dev.pagewire

    Web pages to Markdown, metadata and small crawls for agents. Free daily calls, then USDC x402/MPP.

    Research & DataSearchWeb Scraping
    31
  • PagewatchA
    dev.pagelens

    Read a URL as clean markdown, screenshot a website, url to PDF. Web access for agents, no signup.

    Browser AutomationWeb Scraping
    52

Related MCP Servers

  • Fetch MCP Server

    aglolz
    A
    license
    B
    quality
    D
    maintenance
    Enables LLMs to retrieve and process web content by fetching URLs and converting HTML to markdown format. Supports chunked reading of large pages and can access both public websites and local networks.
    1
    MIT

  • cleanfetch

    belzorash
    A
    license
    A
    quality
    C
    maintenance
    Enables AI agents to read web pages reliably, returning clean markdown content, hyperlinks, and metadata without navigation or ad noise.
    3
    9 npm
    MIT

  • Crawlberg MCP Server

    xberg-io
    A
    license
    Not graded
    quality
    A
    maintenance
    Enables AI agents to crawl and scrape websites, converting HTML to clean Markdown and structured metadata with support for JavaScript rendering, bot evasion, and SSRF protection.
    185
    MIT