Skip to main content
Glama
deenesjakoruzh
dev55acc-ai

website-content-crawler

by dev55acc-ai
OverviewSchemaSourceRelated ServersDiscussionsAdmin
JavaScript
Remote

Website Content Crawler

URL 목록을 크롤링하고 페이지 콘텐츠를 구조화된 JSON으로 반환합니다. 에이전트가 직접 호출할 수 있도록 Apify Actor 및 MCP 서버로 실행됩니다.

라이브 데모: https://website-content-crawler.vercel.app

실행 비용

Apify에서 Pay per Event 방식으로 청구됩니다:

이벤트

가격

Actor start

$0.00005

Page record returned

$0.001

플랫폼 컴퓨팅 비용은 지급 전에 차감됩니다. 단위 경제성은 PRICING.md를 참고하세요.

Related MCP server: cleanfetch

출력

모든 실행은 정확히 이 형태를 반환합니다. 출력 스키마가 이를 강제합니다.

{
  "url": "https://example.com",
  "status": 200,
  "title": "Example Domain",
  "description": null,
  "h1": "Example Domain",
  "markdown": "# Example Domain\n\nThis domain is for use in documentation examples without needing permission. Avoid use in operations.\n\nLearn more\n\nLearn more (https://iana.org/domains/example)",
  "links": [
    "https://iana.org/domains/example"
  ],
  "wordCount": 23,
  "crawledAt": "2026-08-23T09:55:20.140Z",
  "error": null
}

구성

  • src/ — Actor 소스 (Crawlee Cheerio 크롤러)

  • .actor/ — Apify Actor 명세: actor.json, INPUT_SCHEMA.json, output_schema.json, dataset_schema.json, Dockerfile

  • mcp-server/ — Actor를 crawl_website 도구로 노출하는 MCP 래퍼

  • site/ + tools/ — 실제 실행 기록에서 라이브 데모 페이지를 빌드합니다.

로컬 실행

npm install
npm test

MCP 서버:

cd mcp-server && npm install && node smoke.mjs

Apify Store에 게시

계정이 활성화된 경우: apify push, 가격은 Pay per Event로 설정하고, 등록에는 이 README를 사용합니다.

This server cannot be deployed

Let your AI agent see the webWeb-to-screenshot MCP for Claude, Cursor and Codex. No cookie banners. Free to start.snap-render.comAd

Maintenance

ActivitySlowing
ResponsivenessNo issues

Resources

Related MCP Connectors

  • agdataA
    dev.shveik

    Unblocking and fresh web data for agents: URL to Markdown, YouTube, Maps, Amazon, jobs. Pay per call

    Browser AutomationWeb Scraping
    311
  • SnoopScanA
    com.snoopscan

    Web scraping for agents. Point it at a URL and it returns the page as clean markdown, JavaScript-rendered pages included. Point it at a site and it maps the URLs or crawls the section you need in the background, a few pages at a time so results fit in the conversation. Search the web and read full pages, extract fields with a JSON schema you define (validated, never invented), read a store's catalogue or a blog's posts from the platform's own feed, and check whether a page has changed. Failed requests cost nothing. The free plan includes 1,500 credits a month.

    Research & DataSearchWeb Scraping
    301
  • PageWireA
    dev.pagewire

    Web pages to Markdown, metadata and small crawls for agents. Free daily calls, then USDC x402/MPP.

    Research & DataSearchWeb Scraping
    31
  • PagewatchA
    dev.pagelens

    Read a URL as clean markdown, screenshot a website, url to PDF. Web access for agents, no signup.

    Browser AutomationWeb Scraping
    52

Related MCP Servers

  • Fetch MCP Server

    aglolz
    A
    license
    B
    quality
    D
    maintenance
    Enables LLMs to retrieve and process web content by fetching URLs and converting HTML to markdown format. Supports chunked reading of large pages and can access both public websites and local networks.
    1
    MIT

  • cleanfetch

    belzorash
    A
    license
    A
    quality
    C
    maintenance
    Enables AI agents to read web pages reliably, returning clean markdown content, hyperlinks, and metadata without navigation or ad noise.
    3
    9 npm
    MIT

  • Crawlberg MCP Server

    xberg-io
    A
    license
    Not graded
    quality
    A
    maintenance
    Enables AI agents to crawl and scrape websites, converting HTML to clean Markdown and structured metadata with support for JavaScript rendering, bot evasion, and SSRF protection.
    185
    MIT