What can you do with this server?

The MarkCrawl server enables comprehensive website crawling, content extraction, and LLM-powered structured data analysis. * Crawl websites (crawl_site) — Fetch and convert web pages to clean Markdown or plain text, stripping boilerplate. Supports JavaScript rendering (React/Vue/Angular SPAs), subdomain inclusion, and configurable page limits. * List crawled pages (list_pages) — Get an overview of all crawled pages with URLs, titles, and word counts. * Search crawled content (search_pages) — Keyword search across crawled page titles and content, with relevance-ranked results and context snippets — no network requests needed. * Read specific pages (read_page) — Retrieve the full content of any crawled page by its URL. * Extract structured data (extract_data) — Use an LLM to extract specified fields (e.g. pricing, features, API endpoints) from crawled pages, with support for automatic field discovery based on your analysis goal. Designed for LLM pipelines, RAG workflows, and AI agent integration via MCP.

Which integrations are available for this server?

Provides LLM extraction capabilities using Google Gemini models to extract structured information from crawled web content. Provides tool wrappers for integrating MarkCrawl crawling and extraction capabilities into LangChain agents and chains. Provides LLM extraction to structure data from crawled content and generates embeddings for RAG upload, supporting automated field extraction like pricing and API endpoints. Enables RAG pipeline by uploading crawled content with vector embeddings to Supabase pgvector, allowing LLMs to query the vector store for grounded answers.

AIMLPM/markcrawl

by AIMLPM

Overview Schema Related Servers Score Discussions

Python

Local

MarkCrawl by iD8 🕷️📝

Turn any webpage or website into clean Markdown for LLM pipelines — in one command.

PyPI Version License markcrawl MCP server

Latest: v0.11.1 (2026-05-12) — default aggregator URL filter. See What's New below.

pip install markcrawl
markcrawl --base https://docs.example.com --out ./output --show-progress

MarkCrawl is a crawl-and-structure engine. It fetches one page or crawls an entire website, strips navigation/scripts/boilerplate, and writes clean Markdown files with a structured JSONL index. Every page includes a citation with the access date. No API keys needed.

Everything else — LLM extraction, Supabase upload, MCP server, LangChain tools — is optional and installed separately.

Want a hosted API instead of running locally? Join the waitlist — we're gauging interest.

LLM agents: Load docs/LLM_PROMPT.md as a system prompt to generate correct MarkCrawl commands automatically.

What's New

Install or upgrade with pip:

pip install --upgrade markcrawl
pip show markcrawl | grep Version       # confirm the installed version
markcrawl --help | head -1              # confirm the binary on $PATH is the upgraded one

If markcrawl --help is missing flags you expect (e.g. --screenshot, --seed-file, --smart-sample, --download-images), your local install is stale. Run pip install --upgrade markcrawl against the same Python that owns the markcrawl binary on your PATH — head -1 $(which markcrawl) shows the right interpreter. PyPI is always the source of truth; see CHANGELOG.md for the full release history.

v0.11 highlights (changelog):

Aggregator URL filter (default, v0.11.1) — rejects mdBook /print.html and Hugo /_print/ pages during crawl-time URL filtering. These bundle the entire docs tree on a single URL and otherwise dominate retrieval rankings on cosine similarity (markcrawl was returning them in 49% of rust-book and 39% of kubernetes-docs top-5 retrieval slots before the fix; competitors return 0%). Opt out via include_aggregator_pages=True / --include-aggregators.
Binary downloads (v0.11.0) — new download_types=["pdf", "docx"] kwarg streams referenced files to <out_dir>/downloads/ with size + content-type guards. Pre-fetch download_filter callback receives URL + anchor text + parent-page context; reject candidates before any HTTP bytes transfer.
Local embedder is the default since v0.10.1 — pip install markcrawl ships the full ML stack (torch + transformers + sentence-transformers). Zero API key required for embedding. Override with MARKCRAWL_EMBEDDER=text-embedding-3-small or the embedding_model kwarg if you want OpenAI back.
Tenacity-backed HTTP retry — full-jitter exponential backoff (2 s → 30 s, 5 attempts) that honors the server's Retry-After header on 429s.

Where markcrawl stands on the public benchmark, honestly. The independent llm-crawler-benchmarks v1.4 leaderboard measures 7 web crawlers on how well their output supports RAG. Markcrawl ranks 1st on cost ($4,505/yr at 100,000-page scale) but 7th of 7 on answer quality (3.77/5) and retrieval accuracy (MRR 0.341 vs leaders at 0.76). We're actively working to close that gap on three fronts:
v0.11.1 (just shipped) filters out /print.html and /_print/ "whole-book-on-one-page" URLs that were stealing 39–49% of markcrawl's top-5 retrieval slots on documentation sites. Competitors already filter these. Expected MRR improvement: +0.02 to +0.04 on docs-heavy sites (formal measurement pending the next benchmark cycle).
Upcoming releases improve how markcrawl chooses which pages to crawl within its budget — markcrawl's deliberately-narrower crawl strategy (which keeps cost low and signal-to-noise high) is also the main cause of the retrieval gap.
The benchmark itself is being improved — v1.4's test questions were sampled from one specific crawler's output, which structurally penalizes any crawler whose discovery strategy differs from that anchor. The benchmark is being updated so each site's test questions come from the site's own sitemap, independent of any crawler. We expect this fix alone to surface ~5–10% of markcrawl's current "misses" as actually correct answers at different URLs — work shown in our audit notes.
Goal for the next benchmark cycle: move from 7th to mid-pack on retrieval (+0.10 to +0.20 MRR) and answer quality, while keeping the cost-efficiency lead. Honest, measured progress — we publish the numbers either way.

Related MCP server: nia-link

Quickstart (2 minutes)

pip install markcrawl
markcrawl --base https://quotes.toscrape.com --out ./demo --max-pages 5 --show-progress

Your ./demo folder now contains:

demo/
├── index__a4f3b2c1d0.md    ← clean Markdown of the page
├── page-2__b7e2d1f0a3.md
├── ...
└── pages.jsonl              ← structured index (one JSON line per page)

Each line in pages.jsonl:

{
  "url": "https://quotes.toscrape.com/",
  "title": "Quotes to Scrape",
  "crawled_at": "2026-04-04T12:30:00Z",
  "citation": "Quotes to Scrape. quotes.toscrape.com. Available at: https://quotes.toscrape.com/ [Accessed April 04, 2026].",
  "tool": "markcrawl",
  "text": "# Quotes to Scrape\n\n> "The world as we have created it is a process of our thinking..." — Albert Einstein\n\nTags: change, deep-thoughts, thinking, world..."
}

Schema — every page in pages.jsonl has these fields:

Field	Type	Description
`url`	string	Original URL fetched.
`title`	string	Page title from `<title>` (or first H1 if missing).
`crawled_at`	string (ISO 8601)	UTC timestamp of when the page was fetched.
`citation`	string	Pre-formatted academic-style citation including access date.
`tool`	string	Always `"markcrawl"`. Helps when merging output from multiple crawlers.
`text`	string	Clean Markdown content (nav/footer/scripts stripped).
`downloads`	array (optional)	Present when `download_types` is set; one entry per saved binary: `{url, path, size_bytes, content_type}`.
`images`	array (optional)	Present when `--download-images` is set; lists saved image paths.
`screenshot`	string (optional)	Present when `--screenshot` is set; relative path to the PNG/JPEG capture.

Common Recipes

Runnable examples for the most common patterns:

Single-page scrapes — including JS-rendered pages (React, Vue, YouTube)
Whole-site crawls — docs, blogs, subsections; resume interrupted runs
URL filtering — --exclude-path, --include-path, --dry-run, smart sampling
Extraction backends — BS4 (default), trafilatura, ensemble, ReaderLM-v2
Binary downloads — images, PDFs (with pre-fetch filter callbacks), DOCX
Screenshots — full-page or cropped, PNG or JPEG
End-to-end use cases — competitive analysis, RAG chatbot, API-docs → code-gen

Full recipes with copy-paste commands and expected outputs: docs/RECIPES.md.

Pick this tool when…

If you need…	Use…	Why
Clean Markdown for LLM/RAG ingestion, run locally, no API keys	MarkCrawl	Default install bundles local embedder ($0 API spend); strips nav/scripts; produces JSONL with citations out of the box
A hosted scraping API (no infra to run)	FireCrawl	SaaS option; pay-per-call; outsources crawling entirely
AI-native crawling with built-in LLM extraction	Crawl4AI	Deeper LLM-extraction primitives; built-in Playwright
Massive distributed crawling (millions of pages, custom pipelines)	Scrapy	Battle-tested framework; rich plugin ecosystem; spider architecture
JavaScript-heavy automation without framework overhead	Playwright (direct)	Lower-level control over browser automation
Sites behind login/auth or aggressive bot protection	None of the above (build custom)	See When NOT to use MarkCrawl; same constraints apply to most public crawlers

Feature comparison

	MarkCrawl	FireCrawl	Crawl4AI	Scrapy
License	MIT	AGPL-3.0	Apache-2.0	BSD-3
Install	`pip install markcrawl`	SaaS or self-host	pip + Playwright	pip + framework
Output	Markdown + JSONL	Markdown + JSON	Markdown	Custom pipelines
JS rendering	Optional (`--render-js`)	Built-in	Built-in	Plugin
LLM extraction	Optional add-on	Via API	Built-in	None
Local-only operation	✅	❌ (SaaS)	✅	✅
Citations + timestamps in output	✅	Partial	❌	Manual
Best for	Single-site crawl → clean Markdown	Hosted scraping API	AI-native crawling	Large-scale distributed

MarkCrawl's niche is focused-scope RAG ingestion — narrow crawls of docs/blogs/product sites that produce LLM-ready Markdown with minimal junk. For broader scope or bigger scale, the other tools above are stronger choices.

Benchmark results (6 tools, May 2026)

Speed: scrapy+md is fastest (5.0 pages/sec), markcrawl at 2.7. Playwright-based tools average 1.4-2.1 pages/sec.

Output cleanliness: markcrawl has the lowest nav pollution (53 words vs 500+ for others) — less junk in your embeddings.

RAG answer quality: markcrawl scores 3.77/5 on answer quality with the fewest chunks (27,193 total, 2.2x fewer than the most), keeping embedding costs low.

Tool	Chunks/page	Answer Quality (/5)	Annual cost (100K pages, 1K queries/day)
markcrawl	18.7	3.77	$4,505
scrapy+md	31.7	3.68	$5,464
crawl4ai	16.8	4.72	$6,960
colly+md	40.6	4.36	$7,213
playwright	39.0	4.48	$7,320
crawlee	40.5	4.68	$7,467

Full benchmark data: docs/BENCHMARKS.md | Methodology: llm-crawler-benchmarks

Methodology caveat (numbers as of bench v1.4, 2026-05-11): the v1.4 leaderboard sourced test queries from a single high-coverage crawler's output. The bench is actively being updated in v1.5 to source queries from each site's own sitemap independent of any crawler (release notes). Numbers above are single-trial; multi-trial measurement is on the v1.5.1 roadmap. Treat individual rankings as point-in-time signal, not steady-state.

Installation

pip install markcrawl                # Core crawler + chunker + local embedder
                                     # (no API keys required for embedding)

Optional add-ons (tasks beyond the crawl-and-embed core):

pip install markcrawl[js]            # + JavaScript rendering (Playwright)
pip install markcrawl[extract]       # + LLM extraction (OpenAI, Claude, Gemini, Grok)
pip install markcrawl[upload]        # + Supabase upload integration
pip install markcrawl[mcp]           # + MCP server for AI agents
pip install markcrawl[langchain]     # + LangChain tool wrappers
pip install markcrawl[all]           # Everything

For Playwright, also run playwright install chromium after installing.

Lean install (skip the local-embedder dep stack — you'll need an OPENAI_API_KEY and pass embedding_model="text-embedding-3-small" for any embedding work):

pip install --no-deps markcrawl beautifulsoup4 lxml markdownify requests certifi tenacity

git clone https://github.com/AIMLPM/markcrawl.git
cd markcrawl
python -m venv .venv
source .venv/bin/activate
pip install -e ".[all]"

Crawling

markcrawl --base https://www.example.com --out ./output --show-progress

Add flags as needed:

markcrawl \
  --base https://www.example.com \
  --out ./output \
  --include-subdomains \        # crawl sub.example.com too
  --render-js \                 # render JavaScript (React, Vue, etc.)
  --concurrency 5 \             # fetch 5 pages in parallel
  --proxy http://proxy:8080 \   # route through a proxy
  --max-pages 200 \             # stop after 200 pages
  --format markdown \           # or "text" for plain text
  --show-progress

Resume an interrupted crawl:

markcrawl --base https://www.example.com --out ./output --resume --show-progress

Output

Each page becomes a .md file with a citation header:

# Getting Started

> URL: https://docs.example.com/getting-started
> Crawled: April 04, 2026
> Citation: Getting Started. docs.example.com. Available at: https://docs.example.com/getting-started [Accessed April 04, 2026].

Welcome to the platform. This guide walks you through installation...

Navigation, footer, cookie banners, and scripts are stripped. Only the main content remains.

Argument	Description
`--base`	Base site URL to crawl
`--out`	Output directory
`--format`	`markdown` or `text` (default: `markdown`)
`--show-progress`	Print progress and crawl events
`--render-js`	Render JavaScript with Playwright before extracting
`--concurrency`	Pages to fetch in parallel (default: `1`)
`--proxy`	HTTP/HTTPS proxy URL
`--resume`	Resume from saved state
`--include-subdomains`	Include subdomains under the base domain
`--max-pages`	Max pages to save; `0` = unlimited (default: `500`)
`--delay`	Minimum delay between requests in seconds (default: `0`, adaptive throttle adjusts automatically)
`--timeout`	Per-request timeout in seconds (default: `15`)
`--min-words`	Skip pages with fewer words (default: `20`)
`--user-agent`	Override the default user agent
`--use-sitemap` / `--no-sitemap`	Enable/disable sitemap discovery. Use `--no-sitemap` when you want to scrape a specific page or subsection — without it, large sites (YouTube, GitHub) may discover thousands of unrelated pages via their sitemap
`--exclude-path`	Glob pattern to exclude URL paths (e.g. `'/job/*'`). Can be repeated
`--include-path`	Glob pattern to include URL paths (e.g. `'/blog/*'`). Only matching paths are crawled. Can be repeated
`--dry-run`	Discover URLs (via sitemap/links) and print them without fetching content
`--smart-sample`	Auto-detect templated URL patterns and sample from large clusters instead of crawling every page
`--sample-size`	Pages to sample per templated cluster (default: `5`, used with `--smart-sample`)
`--sample-threshold`	Clusters larger than this are sampled (default: `20`, used with `--smart-sample`)
`--auto-resume`	Automatically resume if saved state exists, otherwise start fresh
`--cross-dedup`	Skip pages already seen in previous crawls to the same output directory
`--prioritize-links`	Score discovered links by predicted content yield — crawl high-value pages first
`--extractor`	Content extraction backend: `default`, `trafilatura`, `ensemble`, or `readerlm`
`--download-images`	Download images from the content area to `assets/` and use local paths in Markdown
`--min-image-size`	Minimum image file size in bytes to keep (default: `5000`). Smaller images are skipped
`--i18n-filter`	Skip URLs under locale path segments (`/fr/`, `/de-DE/`, `/zh-Hans/`, ...) — generic, no per-domain config
`--title-at-top`	Prepend `# {title}` to the `text` field of every JSONL row when not already present — top-MRR RAG recipe

Optional: structured extraction

If you need structured data (not just text), the extraction add-on uses an LLM to pull specific fields from each page.

pip install markcrawl[extract]

markcrawl-extract \
  --jsonl ./output/pages.jsonl \
  --fields company_name pricing features \
  --show-progress

Auto-discover fields across multiple crawled sites:

markcrawl-extract \
  --jsonl ./comp1/pages.jsonl ./comp2/pages.jsonl ./comp3/pages.jsonl \
  --auto-fields \
  --context "competitor pricing analysis" \
  --show-progress

Supports OpenAI, Anthropic (Claude), Google Gemini, and xAI (Grok) via --provider.

Provider and model selection

markcrawl-extract --jsonl ... --fields pricing --provider openai         # default
markcrawl-extract --jsonl ... --fields pricing --provider anthropic      # Claude
markcrawl-extract --jsonl ... --fields pricing --provider gemini         # Gemini
markcrawl-extract --jsonl ... --fields pricing --provider grok           # Grok
markcrawl-extract --jsonl ... --fields pricing --model gpt-4o           # override model

Provider	API key env var	Default model
OpenAI	`OPENAI_API_KEY`	`gpt-4o-mini`
Anthropic	`ANTHROPIC_API_KEY`	`claude-sonnet-4-20250514`
Google Gemini	`GEMINI_API_KEY`	`gemini-2.0-flash`
xAI (Grok)	`XAI_API_KEY`	`grok-3-mini-fast`

All extraction CLI arguments

Argument	Description
`--jsonl`	Path(s) to `pages.jsonl` — pass multiple for cross-site analysis
`--fields`	Field names to extract (space-separated)
`--auto-fields`	Auto-discover fields by sampling pages
`--context`	Describe your goal for auto-discovery
`--sample-size`	Pages to sample for auto-discovery (default: `3`)
`--provider`	`openai`, `anthropic`, `gemini`, or `grok`
`--model`	Override the default model
`--output`	Output path (default: `extracted.jsonl`)
`--delay`	Delay between LLM calls in seconds (default: `0.25`)
`--show-progress`	Print progress

Output format

Extracted rows include LLM attribution:

{
  "url": "https://competitor.com/pricing",
  "citation": "Pricing. competitor.com. Available at: ... [Accessed April 04, 2026].",
  "pricing_tiers": "Starter ($29/mo), Pro ($99/mo), Enterprise (contact sales)",
  "extracted_by": "gpt-4o-mini (openai)",
  "extraction_note": "Field values were extracted by an LLM and may be interpreted, not verbatim."
}

Optional: Supabase vector search (RAG)

Chunk pages, generate embeddings, and upload to Supabase with pgvector:

pip install markcrawl[upload]

markcrawl --base https://docs.example.com --out ./output --show-progress
markcrawl-upload --jsonl ./output/pages.jsonl --show-progress

Requires SUPABASE_URL, SUPABASE_KEY, and OPENAI_API_KEY. See docs/SUPABASE.md for table setup, query examples, and recommendations.

Optional: agent integrations

MarkCrawl includes integrations for AI agents. Each is an optional add-on.

pip install markcrawl[mcp]

{
  "mcpServers": {
    "markcrawl": {
      "command": "python",
      "args": ["-m", "markcrawl.mcp_server"]
    }
  }
}

Tools: crawl_site, list_pages, read_page, search_pages, extract_data

pip install markcrawl[langchain]

from markcrawl.langchain import all_tools
from langchain_openai import ChatOpenAI
from langchain.agents import initialize_agent, AgentType

agent = initialize_agent(tools=all_tools, llm=ChatOpenAI(model="gpt-4o-mini"),
                         agent=AgentType.STRUCTURED_CHAT_ZERO_SHOT_REACT_DESCRIPTION)
agent.run("Crawl docs.example.com and summarize their auth guide")

npx clawhub install markcrawl-skill

See AIMLPM/markcrawl-clawhub-skill.

Copy the system prompt from docs/LLM_PROMPT.md into any LLM to get an assistant that generates correct MarkCrawl commands.

When NOT to use MarkCrawl

Sites behind login/auth — no cookie or session support
Aggressive bot protection (Cloudflare, Akamai) — no anti-bot evasion
Millions of pages — designed for hundreds to low thousands; use Scrapy for scale
PDF content — HTML only (PDF support is on the roadmap)
JavaScript SPAs — add markcrawl[js] and use --render-js for React/Vue/Angular
Infinite-scroll pages — --render-js renders the initial page load but does not scroll; you'll get the first screenful of content (e.g., ~28 of 82 YouTube videos). For complete listings, combine with the platform's API or RSS feed (e.g., YouTube's /feeds/videos.xml?channel_id=...)

Architecture

MarkCrawl is a web crawler. The optional layers (extraction, upload, agents) are separate add-ons that work with the crawler's output.

CORE (free, no API keys)              OPTIONAL ADD-ONS
┌──────────────────────────┐
│ 1. Discover URLs         │          markcrawl[extract]  — LLM field extraction
│    (sitemap or links)    │          markcrawl[upload]   — Supabase/pgvector RAG
│ 2. Fetch & clean HTML    │          markcrawl[js]       — Playwright JS rendering
│ 3. Write Markdown + JSONL│          markcrawl[mcp]      — MCP server for agents
│    + auto-citation       │          markcrawl[langchain] — LangChain tools
└──────────────────────────┘

For internals, see docs/ARCHITECTURE.md.

Extending MarkCrawl

from markcrawl import crawl

result = crawl("https://example.com", out_dir="./output")
print(f"Saved {result.pages_saved} pages")

# Process output in your own pipeline
import json
with open(result.index_file) as f:
    for line in f:
        page = json.loads(line)
        your_db.insert(page)  # Pinecone, Weaviate, Elasticsearch, etc.

# Use individual components
from markcrawl import chunk_text
from markcrawl.extract import LLMClient, extract_fields

See docs/ARCHITECTURE.md for the full module map and extensibility guide.

Cost

The core crawler is free. Two optional features have API costs:

Feature	Cost	When
Structured extraction	~$0.01-0.03 per page	`markcrawl-extract`
Supabase upload	~$0.0001 per page	`markcrawl-upload`

Setting up API keys

Only needed for extraction and upload. The core crawler requires no keys.

# .env — in your working directory
OPENAI_API_KEY="sk-..."           # extraction (--provider openai) + upload
ANTHROPIC_API_KEY="sk-ant-..."    # extraction (--provider anthropic)
GEMINI_API_KEY="AI..."            # extraction (--provider gemini)
XAI_API_KEY="xai-..."             # extraction (--provider grok)
SUPABASE_URL="https://..."        # upload
SUPABASE_KEY="eyJ..."             # upload (service-role key)

source .env

.
├── README.md
├── LICENSE
├── PRIVACY.md
├── SECURITY.md
├── CONTRIBUTING.md
├── CODE_OF_CONDUCT.md
├── Dockerfile
├── Makefile
├── glama.json
├── pyproject.toml
├── requirements.txt
├── .github/
│   ├── pull_request_template.md
│   └── workflows/
│       ├── ci.yml
│       └── publish.yml
├── docs/
│   ├── ARCHITECTURE.md
│   ├── LLM_PROMPT.md
│   ├── MCP_SUBMISSION.md
│   ├── RAG_RETRIEVAL_RESEARCH.md
│   └── SUPABASE.md
├── tests/
│   ├── __init__.py
│   ├── test_chunker.py
│   ├── test_core.py
│   ├── test_extract.py
│   └── test_upload.py
└── markcrawl/
    ├── __init__.py
    ├── cli.py
    ├── core.py               # orchestrator
    ├── fetch.py              # HTTP/Playwright fetching
    ├── robots.py             # robots.txt parsing
    ├── throttle.py           # adaptive rate limiting
    ├── state.py              # crawl state & resume
    ├── urls.py               # URL normalization & filtering
    ├── extract_content.py    # HTML → Markdown conversion
    ├── dedup.py              # cross-crawl deduplication
    ├── link_scorer.py        # link prioritization
    ├── chunker.py
    ├── exceptions.py
    ├── utils.py
    ├── extract.py            # LLM field extraction
    ├── extract_cli.py
    ├── upload.py
    ├── upload_cli.py
    ├── langchain.py
    └── mcp_server.py

Roadmap

Canonical URL support
PDF support
Authenticated crawling
Multi-provider embeddings

pip install markcrawl on PyPI
647 automated tests + GitHub Actions CI (Python 3.10-3.13) + ruff linting
Markdown and plain text output with auto-citation
Sitemap-first crawling with robots.txt compliance
Text chunking with configurable overlap + semantic chunking
Supabase/pgvector upload for RAG
JavaScript rendering via Playwright
Concurrent fetching and proxy support
Resume interrupted crawls + auto-resume
LLM extraction (OpenAI, Claude, Gemini, Grok) with auto-field discovery
MCP server, LangChain tools, OpenClaw skill
Image alt text preservation
Python API (result.pages)
Page-type extraction and content-region heuristics
Multiple extraction backends (default, trafilatura, ensemble, ReaderLM-v2)
Cross-crawl deduplication (--cross-dedup)
Link prioritization by predicted content yield (--prioritize-links)
Smart sampling of templated URL clusters (--smart-sample)
URL path filtering (--include-path, --exclude-path) and dry-run preview

Project info

Contributing — see CONTRIBUTING.md. If you used an LLM to generate code, include the prompt in your PR.
Security — see SECURITY.md for the disclosure policy.
Privacy — MarkCrawl runs locally. No telemetry, no analytics, no data sent anywhere. See PRIVACY.md.
License — MIT. See LICENSE.

Available Tools

5 tools

crawl_siteA

Crawl a website and save extracted content as clean Markdown or plain text.

This tool fetches pages from the given URL, strips navigation, footers,
scripts, and boilerplate, then saves each page as a Markdown file with a
JSONL index (pages.jsonl). It respects robots.txt and uses sitemap-first
discovery when available.

Use this tool when asked to research, read, analyze, or archive a website.
The output_dir from this tool is required by search_pages, read_page,
list_pages, and extract_data.

Typical workflow: crawl_site → list_pages or search_pages → read_page.

Args:
    url: The base URL to crawl (e.g. "https://docs.example.com/"). Only
        public, non-authenticated pages will be fetched.
    output_dir: Directory to save output files. Each crawl creates .md files
        and a pages.jsonl index here. Default: ./crawl_output
    format: Output format — "markdown" (preserves headings, code blocks,
        lists) or "text" (plain text). Default: "markdown".
    max_pages: Maximum number of pages to save. Set to 0 for unlimited.
        Default: 100. Use lower values (10-20) for quick previews.
    include_subdomains: If True, also crawl subdomains (e.g. docs.example.com
        when crawling example.com). Default: False.
    render_js: If True, use a headless Chromium browser to render JavaScript
        before extracting content. Required for React/Vue/Angular sites.
        Slower but necessary for SPAs. Default: False.

ParametersJSON Schema

Name	Required	Default
`url`	Yes
`output_dir`	No	./crawl_output
`format`	No	markdown
`max_pages`	No
`include_subdomains`	No
`render_js`	No

Output Schema

ParametersJSON Schema

Name	Required	Description
`result`	Yes

TDQS

A4.7/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden and excels: it discloses that the tool respects robots.txt, uses sitemap-first discovery, handles subdomains, and offers JS rendering. It also notes that only public, non-authenticated pages are fetched, which is critical behavioral context.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured with a concise overview followed by clear parameter explanations in a bullet-like format. It is informative without being verbose, though some default values are repeated from the schema, which could be trimmed slightly.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity and the existence of an output schema, the description adequately explains the output (Markdown files and pages.jsonl) and how the output integrates with sibling tools. It covers input, behavior, and integration, making it complete for an AI agent.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has 0% coverage, so the description must compensate, and it does thoroughly. Each parameter (url, output_dir, format, max_pages, include_subdomains, render_js) has a detailed explanation including meaning, defaults, and usage tips (e.g., 'Use lower values (10-20) for quick previews'). This adds significant meaning beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: crawl a website and save extracted content as clean Markdown or plain text. It specifies the verb 'crawl' and resource 'site', and distinguishes from siblings like search_pages and read_page by mentioning that the output_dir is used by those tools, forming a typical workflow.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly says 'Use this tool when asked to research, read, analyze, or archive a website' and provides a typical workflow (crawl_site → list_pages or search_pages → read_page). It does not explicitly state when not to use it or mention alternatives beyond the workflow, but the guidance is clear and actionable.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

extract_dataA

Extract structured fields from crawled pages using an LLM.

Analyzes each crawled page and pulls out specific data fields you define
(e.g. company_name, pricing, features, api_endpoints). If no fields are
specified, the LLM automatically discovers relevant fields by sampling
pages from the crawl.

This tool makes external API calls to OpenAI (requires OPENAI_API_KEY
environment variable). Results are saved to extracted.jsonl and include
LLM attribution metadata.

Use this for competitive research, API documentation analysis, or building
structured datasets from unstructured web content.

Args:
    jsonl_path: Full path to the pages.jsonl file. If empty, defaults to
        <MARKCRAWL_OUTPUT_DIR>/pages.jsonl.
    fields: Comma-separated field names to extract. Example:
        "company_name,pricing,features,api_endpoints". Leave empty to
        let the LLM auto-discover the most relevant fields.
    context: Description of your analysis goal. Improves auto-field
        discovery quality. Example: "competitor pricing analysis" or
        "API documentation review". Ignored when fields are specified.
    sample_size: Number of pages to sample for auto-field discovery.
        Default: 3. Higher values give better field suggestions but
        cost more tokens.

ParametersJSON Schema

Name	Required	Description	Default
`jsonl_path`	No
`fields`	No
`context`	No
`sample_size`	No

Output Schema

ParametersJSON Schema

Name	Required	Description
`result`	Yes

TDQS

A4.6/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It transparently discloses that the tool makes external API calls to OpenAI (requiring OPENAI_API_KEY), saves results to extracted.jsonl, and includes LLM attribution metadata. It also mentions token cost implications for sample_size. Missing details on error handling or rate limits, but overall provides good behavioral insight.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured and appropriately sized. It starts with a concise summary sentence, followed by a paragraph on how it works, a phrase on use cases, and a clear bullet-like list for arguments. Every sentence adds value, with no redundant or irrelevant content.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool complexity (4 parameters, all optional, an output schema exists), the description is complete. It explains auto-discovery when fields are empty, the role of context, and environment requirements. Despite not detailing the output schema content, the presence of an output schema reduces the need for that detail. The description covers all essential behavioral and usage aspects.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, yet the description thoroughly documents all four parameters in a dedicated 'Args' section: jsonl_path (full path, default location), fields (comma-separated, example, auto-discovery behavior), context (purpose, ignored when fields specified), and sample_size (default, effect on quality and cost). This fully compensates for the lack of schema descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: 'Extract structured fields from crawled pages using an LLM.' It uses a specific verb ('extract') and identifies the resource ('crawled pages') and method. It also lists use cases (competitive research, API documentation analysis, building structured datasets), effectively distinguishing it from sibling tools like crawl_site, list_pages, read_page, and search_pages.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear usage context: 'Use this for competitive research, API documentation analysis, or building structured datasets from unstructured web content.' It implies when to use the tool but does not explicitly state when not to use it or suggest alternatives among sibling tools. Thus, while helpful, it lacks explicit exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_pagesA

List all pages from a previous crawl with their URLs, titles, and word counts.

Returns a summary of every page in the crawl index. Use this to get an
overview of available content before searching or reading specific pages.
Word counts help identify content-rich pages vs. thin landing pages.

This is a read-only operation on local files — no network requests are made.

Args:
    jsonl_path: Full path to the pages.jsonl file. If empty, defaults to
        <MARKCRAWL_OUTPUT_DIR>/pages.jsonl.

ParametersJSON Schema

Name	Required	Description	Default
`jsonl_path`	No

Output Schema

ParametersJSON Schema

Name	Required	Description
`result`	Yes

TDQS

A4.4/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description states 'This is a read-only operation on local files — no network requests are made,' which is a valuable behavioral trait beyond the schema. It also explains the utility of word counts. Since annotations are absent, the description carries the burden and meets it well.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is about 6 sentences, front-loading the main purpose. Each sentence adds value, though the sentence about word counts could be seen as slightly extraneous. Overall, it is concise and well-structured.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's low complexity (one parameter, read-only, with output schema) and good annotations coverage (even though annotations are missing, the description covers the essential), the description is complete. It covers purpose, parameters, and behavior adequately.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description compensates by explaining the parameter: 'jsonl_path: Full path to the pages.jsonl file. If empty, defaults to <MARKCRAWL_OUTPUT_DIR>/pages.jsonl.' This adds meaning beyond the schema's title and default.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states 'List all pages from a previous crawl with their URLs, titles, and word counts.' It identifies the verb (list) and resource (pages from a crawl), and distinguishes from sibling tools like crawl_site, read_page, and search_pages.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It explicitly says 'Use this to get an overview of available content before searching or reading specific pages,' providing clear context. However, it does not explicitly mention when not to use or name alternative tools, though the context implies such distinctions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

read_pageA

Read the full extracted content of a specific crawled page by its URL.

Returns the complete Markdown or text content of a single page, including
its title and source URL. Use this after search_pages to read the full
content of a relevant result.

This is a read-only operation on local files — no network requests are made.
URL matching is case-insensitive and tolerates trailing slashes.

Args:
    url: The exact URL of the page to read. Must match a URL from a previous
        crawl. Case-insensitive. Example: "https://docs.example.com/auth".
    jsonl_path: Full path to the pages.jsonl file. If empty, defaults to
        <MARKCRAWL_OUTPUT_DIR>/pages.jsonl.

ParametersJSON Schema

Name	Required	Description	Default
`url`	Yes
`jsonl_path`	No

Output Schema

ParametersJSON Schema

Name	Required	Description
`result`	Yes

TDQS

A4.8/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Discloses read-only operation on local files, no network requests, case-insensitive matching, and trailing slash tolerance. With no annotations, description carries full burden and does so thoroughly.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Well-structured with clear sections, concise yet comprehensive. Every sentence adds value, no wasted words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Covers return format (Markdown/text, title, source URL), usage order, and behavioral traits. Complements the output schema effectively.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Provides detailed meaning for both parameters: url (type, case-insensitivity, example) and jsonl_path (optional, default path). Adds significant value over the bare schema (0% coverage, only title/type).

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Clearly states it reads the full extracted content of a crawled page by URL, distinguishing from sibling tools like search_pages (snippets) and list_pages (listing).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly advises using after search_pages to read full content, providing clear context. Lacks explicit when-not-to-use, but context is sufficient.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

search_pagesA

Search through previously crawled pages by keyword.

Performs case-insensitive keyword search across page titles and text content.
Results are ranked by the number of matching query words found. Each result
includes the page URL, title, and a text snippet showing context around the
first match.

This is a read-only operation on local files — no network requests are made.
Requires a prior crawl_site call to have populated the pages.jsonl file.

Args:
    query: Search query — one or more keywords separated by spaces. All words
        are searched independently (OR logic). Example: "authentication API key".
    jsonl_path: Full path to the pages.jsonl file from a previous crawl. If
        empty, defaults to <MARKCRAWL_OUTPUT_DIR>/pages.jsonl.
    max_results: Maximum number of results to return. Default: 10. Use lower
        values for focused searches, higher for comprehensive surveys.

ParametersJSON Schema

Name	Required	Description	Default
`query`	Yes
`jsonl_path`	No
`max_results`	No

Output Schema

ParametersJSON Schema

Name	Required	Description
`result`	Yes

TDQS

A4.7/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, description fully discloses it is read-only, operates on local files, uses OR logic, ranks by match count, and returns snippets. No side effects or auth requirements left unstated.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Well-structured with summary then Args section. Slightly verbose but every sentence adds value. Front-loaded with purpose.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Covers purpose, parameters, usage context, behavior, and output (URL, title, snippet). Output schema exists, so return details are sufficient. Complete for this tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Despite 0% schema description coverage, description thoroughly explains each parameter: query as OR keywords with example, jsonl_path default path, max_results with usage guidance. Greatly adds meaning.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description clearly states it searches crawled pages by keyword, specifies case-insensitive search across titles and text, and ranks results by matching words. It distinguishes from sibling tools like crawl_site, read_page, list_pages.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly states it requires prior crawl_site call and is read-only with no network requests. However, does not explicitly compare to alternatives or state when not to use.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.

4 tool updatesv0.9.3May 1, 2026
- Addedcrawl_site
- Addedlist_pages
- Addedread_page
- Addedsearch_pages
1 tool updatev0.4.0May 1, 2026
- Addedextract_data
1 tool updatev0.5.0May 1, 2026
- Removedextract_data
1 tool updatev0.8.0May 1, 2026
- Addedextract_data
5 tool updatesv0.4.1May 1, 2026
- Removedcrawl_site
- Removedextract_data
- Removedlist_pages
- Removedread_page
- Removedsearch_pages
5 tool updatesv1.0.0Apr 4, 2026
- First observedcrawl_site
- First observedextract_data
- First observedlist_pages
- First observedread_page
- First observedsearch_pages

Install Server

RayrunBuild, deploy, and operate MCP servers without becoming an infrastructure team.ray.runAd

TDQS

A4.6/5.0

Disambiguation5/5

Each tool has a clearly distinct purpose: crawl_site handles network fetching and initial extraction, while the remaining tools provide different local analysis modes (listing, reading, searching, and LLM-based structured extraction). No functional overlap exists between tools.

Naming Consistency5/5

All five tools follow a consistent verb_noun snake_case pattern (crawl_site, extract_data, list_pages, read_page, search_pages). The verb choices clearly indicate the action performed (crawl/extract/list/read/search).

Tool Count5/5

Five tools is ideal for this focused domain. The set includes one acquisition tool (crawl_site), three discovery/retrieval tools (list_pages, search_pages, read_page), and one analysis tool (extract_data), with no redundant or extraneous operations.

Completeness4/5

The tool surface covers the complete core workflow from crawling to structured analysis with no dead ends. Minor gaps exist in crawl lifecycle management (no tools for deleting crawls, resuming interrupted crawls, or updating crawl parameters), but these are workarounds via filesystem operations.

Maintenance

ActivityInactive

ResponsivenessResponsive

Resources

GitHub Repository

Need Help?

Related Servers

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Hydrafetch A
com.hydrafetch
Turn any URL into clean Markdown and structured data. Scrape, crawl, search and extract.
Browser Automation Web Scraping
131
Brainiall Web A
com.brainiall
Web search, URL content extraction to Markdown, site mapping, and recursive web crawler.
Browser Automation Web Scraping
51
scrapi A
io.github.bamchi
Web scraping for AI agents. Converts URLs to clean, LLM-ready Markdown with anti-bot bypass.
Web Scraping Browser Automation Agent Orchestration
525
Refetch
io.github.lowenbjer
Fetch pages as markdown, search web and news, extract structured data. For AI agents.
Web Scraping Browser Automation Search
2

Related MCP Servers

firecrawl-mcp-server
RAG Systems Web Scraping
firecrawl
A
license
A
quality
C
maintenance
A Model Context Protocol (MCP) server implementation that integrates with FireCrawl for advanced web scraping capabilities.
Updated 16 hours ago (2026-09-05 00:05 UTC)
26
40,139
7,395
MIT
nia-link
Browser Automation Web Scraping Agent Orchestration
nia-atavism
A
license
Not graded
quality
D
maintenance
Automate web browsing and data extraction by converting live pages into clean Markdown. Execute multi-step workflows and interact with websites using human-like movements.
Updated 4 months ago (2026-05-06 01:33 UTC)
MIT
HatFetchofficial
Browser Automation Web Scraping
ProxyHatCom
A
license
A
quality
A
maintenance
Enables LLM agents to read any website by scraping and crawling into clean Markdown, automatically bypassing bot detection with residential proxies.
Updated 2 months ago (2026-07-14 15:44 UTC)
3
21
MIT
icecrawl MCP server
MrGreenlaw
F
license
Not graded
quality
C
maintenance
Enables scraping single pages or crawling entire websites, converting content to markdown and optionally extracting structured data with Claude.
Updated a month ago (2026-07-22 21:50 UTC)
-

Latest Blog Posts

Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)
By Om-Shree-0709 on July 25, 2026.
mcp
Agent Identity
OAuth 2.1
Your AI Chatbot Just Exposed Your CEO's Salary to an Intern
By Om-Shree-0709 on July 2, 2026.
Agent Identity
MCP Security
OAuth Delegation
Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)
By Om-Shree-0709 on June 30, 2026.
Agentic Ai
Prompt Injection
WebAssembly

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/AIMLPM/markcrawl'

If you have feedback or need assistance with the MCP directory API, please join our Discord server

MarkCrawl by iD8 🕷️📝

Turn any webpage or website into clean Markdown for LLM pipelines — in one command.

What's New

Quickstart (2 minutes)

Common Recipes

Pick this tool when…

Feature comparison

Benchmark results (6 tools, May 2026)

Installation

Crawling

Output

Optional: structured extraction

Provider and model selection

All extraction CLI arguments

Output format

Optional: Supabase vector search (RAG)

Optional: agent integrations

When NOT to use MarkCrawl

Architecture

Extending MarkCrawl

Cost

Setting up API keys

Roadmap

Project info

Available Tools

Output Schema

Output Schema

Output Schema

Output Schema

Output Schema

Tool Schema Changelog

Maintenance

Resources

Looking for Admin?

Related MCP Connectors

Related MCP Servers

HatFetchofficial

Latest Blog Posts

MCP directory API