Skip to main content
Glama

pagesift 0.81.4

Turn web pages, PDFs and Office files into clean Markdown for LLMs and AI agents.
One self-hosted Rust binary: CLI, Firecrawl-compatible HTTP API, Jina Reader endpoint, MCP server, Python and Node.js packages.

License: AGPL-3.0 Commercial license Release crates.io PyPI npm Discord

English · 简体中文 · 日本語 · Español

What it is

pagesift fetches a URL (rendering JavaScript only when the page needs it) or reads a local file, keeps the main content, and returns Markdown plus metadata. It scrapes single pages, batches and whole sites, and runs as a command, a library, an HTTP service or an MCP server.

Can replace or sit alongside: Firecrawl (same v1/v2 API, self-hosted), Jina Reader (/r/<url>), Crawl4AI, trafilatura / Readability, pymupdf4llm / pypdf, Kreuzberg.

Related MCP server: Crawl4AI MCP Server

At a glance

  • Input: URLs (HTTP, or a headless browser when needed), HTML, PDF, Word (.docx), Excel (.xlsx), PowerPoint (.pptx), EPUB, CSV

  • Output: Markdown, HTML, raw HTML, links, images, metadata (title, publication date, …), JSON; warnings when a result is incomplete

  • Interfaces: CLI · Firecrawl v1/v2 HTTP API · Jina Reader /r/ · MCP (stdio or HTTP) · Python · Node.js / TypeScript · Rust crate

  • Platforms: Linux x86_64 / aarch64 (glibc and musl), macOS x86_64 / arm64, Windows x86_64; Docker image ghcr.io/yolow92/pagesift

  • Crawling: robots.txt and per-host rate limits by default, sitemaps, focused crawls by query, bounded memory

  • Memory: 14–380 MB peak per worker across the 0.81 benchmarks (lowest at 16 workers on HTML and PDF); capacity is sized from the cgroup limit (no OOM at 768 MiB)

  • Not for: OCR of scanned PDFs or images; pagesift reads the text layer only (see When to use it below)

  • License: AGPL-3.0, or a commercial license

  • Latest: 0.81.4 · release notes

Install and use

Install

First call

CLI

cargo install pagesift --locked

pagesift scrape https://example.com

Python

pip install pagesift

pagesift.scrape("https://example.com").markdown

Node.js

npm install @pagesift/node

(await scrape("https://example.com")).markdown

Server

docker run -p 3002:3002 ghcr.io/yolow92/pagesift:0.81.4

curl localhost:3002/r/https://example.com

MCP

—

claude mcp add pagesift -- pagesift mcp

CLI

pagesift scrape "https://news.ycombinator.com"              # Markdown to stdout
pagesift scrape report.pdf                                  # local PDF, Office, EPUB, CSV or HTML file
pagesift crawl "https://docs.rs" --limit 500 --out ./data/  # one Markdown file per page
pagesift crawl "https://www.example.edu" --query "graduate admission" --adaptive
pagesift map "https://docs.rs"                              # list a site's URLs
pagesift doctor                                             # check browsers and environment

Python

import pagesift

page = pagesift.scrape("https://example.com", formats=["markdown", "metadata"])
print(page.markdown)
print(page.metadata.title, page.metadata.published_date)

results = pagesift.scrape_many(["https://site1.com", "https://site2.com"], concurrency=8)

Node.js / TypeScript

import { scrape, scrapeMany } from '@pagesift/node'

const page = await scrape('https://example.com', { formats: ['markdown', 'links'] })
console.log(page.markdown)

for await (const p of scrapeMany(['https://a.com', 'https://b.com'], { concurrency: 4 })) {
  console.log(p.url, p.elapsedMs)
}

HTTP server (Firecrawl and Jina Reader compatible)

docker run -d -p 3002:3002 -v pagesift-data:/data \
  -e PAGESIFT_API_KEYS=sk-test ghcr.io/yolow92/pagesift:0.81.4
from firecrawl import FirecrawlApp

app = FirecrawlApp(api_url="http://localhost:3002", api_key="sk-test")  # existing Firecrawl code, new URL
print(app.scrape_url("https://example.com", params={"formats": ["markdown"]})["markdown"])
curl -s http://localhost:3002/r/https://example.com -H 'Authorization: Bearer sk-test'   # Jina Reader style
curl -s http://localhost:3002/v2/parse -H 'Authorization: Bearer sk-test' \
  -F file=@report.pdf -F 'options={"formats":["markdown"]}'                            # upload a file

Endpoints: /v1 and /v2 scrape, batch/scrape, crawl, map; /v2/parse (file upload, up to 50 MB, details); /r/<url> (plain Markdown or text/event-stream).

MCP server for AI agents

pagesift mcp                                    # stdio, for one agent
pagesift mcp http --port 3003                   # one shared local service (127.0.0.1 only)
claude mcp add --scope user --transport http pagesift http://127.0.0.1:3003/mcp

Tools: pagesift_scrape, pagesift_batch_scrape, pagesift_crawl, pagesift_crawl_status, pagesift_map. Shared browsers and cache, an audit log and an issue-report tool: shared MCP service.

When to use it, and when not

Good fit

  • Feeding web pages and documents to an LLM, a RAG index or an agent, with Markdown that keeps headings, lists, tables and links

  • Self-hosting a Firecrawl- or Jina Reader-style service on your own machines, with no per-page fees

  • Mixed input: the same call handles HTML, PDF, Office files and e-books

  • Small machines and containers: memory is bounded and sized from the cgroup limit

Pick something else when

  • Scanned PDFs or images need OCR. pagesift reads the text layer only; on 134 scanned pages it completed 0.112 against 1.000 for Docling on a GPU. Use Docling, olmOCR or another OCR pipeline.

  • You want the fastest possible crawl and do not need politeness. pagesift waits between requests to the same host, follows robots.txt and retries failures; spider is much faster on raw crawl time (1.4 s vs 16.6 s on 40 sites × 60 pages).

  • AGPL does not work for you and a commercial license is not an option: use a permissively licensed tool.

  • You need a managed cloud service today. pagesift is self-hosted; a hosted API is not open yet.

Performance

↑ higher is better · ↓ lower is better. Strongest result per domain; methods and sources in the full report.

Task

Metric

PageSift

Best other tool

Gap

PDF → Markdown — olmOCR-bench 1,269 pages, per page (single thread)

ms/page ↓

2.2

pypdf 50.0

×22.32

HTML → Markdown — 600 generated documents, per page

ms/page ↓

0.24

Turndown 1.9

×8.24

Main content — Firecrawl 597 pages, single thread

p50 ms/page ↓

1.0

crw 7.4

×7.23

Publication dates — htmldate 1,000 pages, date search

ms/page ↓

0.31

htmldate (fast) 2.0

×6.45

Office / e-books — B4 81 files, in-process

files/s ↑

94.1

Kreuzberg 19.5

×4.82

RAG chunking — 5 corpora, 1.4 MB

total ms ↓

11

chonkie recursive 35

×3.24

RAG query filter — 472 questions, default settings

answer recall ↑

0.929

Crawl4AI BM25 0.673

+0.256

Wild crawl — polite, 40 sites × 60 pages, coverage

coverage ↑

0.740

Crawl4AI (HTTP) 0.529

+0.211

Still behind: A2 replay, 269 pages, 1 worker 2.4 pages/s vs spider 156.6 pages/s; fast, 40 sites × 60 pages, time 16.6 seconds vs spider 1.4 seconds; B5 134 pages, completion 0.112 vs Docling (GPU) 1.000.

A2 throughput is measured against a replay server on a single loopback address, so all 269 pages count as one host. PageSift waits 1 s between requests to the same host by default; spider has no such limit. This row mostly reflects that politeness interval, not engine speed.

Full report

Scale (0.70, synthetic local origin, not real-internet speed): 200,000,000 requests on 192 vCPU with 0 failures at 947,611 pages/s and 888 MiB peak RSS; inside a 768 MiB cgroup it still sustained 817,346 pages/s with no OOM. Sources: 0.70 run, all benchmarks.

Features

  • Extraction: main-content detection, Markdown that round-trips headings, lists, tables, code and math; publication dates; includeTags / excludeTags CSS filters; warnings such as partial_content, pages_missing, output_truncated instead of silently incomplete results

  • Documents: own Rust readers for PDF (structure tree, bookmarks, headers/footers, CJK fonts without ToUnicode, damaged files), Word, Excel, PowerPoint, EPUB and CSV

  • Rendering: engine: auto fetches over HTTP first and opens a headless browser only when that adds content; the Docker image ships one

  • Crawling: focused crawls (--query … --adaptive) that stop when pages stop adding relevant content; attachments (linked PDFs and Office files); crawl trace per page; sitemaps; duplicate and trap URL handling; per-host token buckets and backoff on 429/503

  • Operations: bounded queues, capacity sized from CPU and cgroup memory, job persistence, webhooks, per-key limits and metrics in the server; audit log of every fetch

  • Safety: private-network and DNS-rebinding guards by default (--allow-cidr to opt in), browser sandboxing

Architecture: docs/architecture.md.

License

AGPL-3.0. For proprietary linking or private deployment without AGPL obligations, see the commercial license or write to licensing@pagesift.dev.

What's new in 0.81.4

Dependency security update: tokio 1.54.1 and zstd 0.14.3 (zstd-safe 8.2.0) fix four RustSec advisories (RUSTSEC-2026-0384, -0391, -0398, -0403). No API changes. Release notes · all releases · CHANGELOG


Community

Questions or feedback: Discord · Issues · Discussions

Citation

@software{pagesift2026,
  title  = {PageSift: Rust Web and Document Extraction Engine for AI Pipelines},
  author = {YOLOW92},
  year   = {2026},
  url    = {https://github.com/YOLOW92/pagesift},
  note   = {Version 0.81.4}
}

Star History

Star History Chart

Available Tools

5 tools
pagesift_batch_scrapeBatch scrapeA

Scrape a list of known URLs as one job with the same options. Returns a job id; poll it with pagesift_crawl_status.

NameRequiredDescriptionDefault
urlsYesThe URLs to scrape.
optionsNo

TDQS

A3.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare openWorldHint=true and destructiveHint=false, so the safety profile is covered. The description adds context the annotations do not carry: this is a long-running job that returns an identifier rather than results, requiring polling. It omits failure semantics (per-URL errors, partial failures), which keeps it from a 5.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two short sentences, front-loaded with what the tool does and followed by the required next step. Nothing is wasted and no filler is present.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description appropriately discloses the return value (a job id) and the retrieval path (poll crawl_status). It is nearly complete for a complex, heavily nested async tool; only error/failure reporting and result payload shape are left unaddressed.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is only 50%, and the large `options` object contains many undocumented properties (mobile, parsers, profile, lockdown, redactPII, includeTags/excludeTags, storeInCache, etc.). The description only says 'with the same options' without explaining any of them or pointing to pagesift_scrape's option semantics explicitly as the reference.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Scrape a list of known URLs as one job'), and the words 'list of known URLs' distinguish it from crawl-based URL discovery. It does not explicitly contrast with the single-URL sibling pagesift_scrape, so differentiation is implied rather than stated.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives the follow-up workflow explicitly: the call is asynchronous and should be polled with pagesift_crawl_status. It offers no explicit when-not guidance or direction to pagesift_scrape for single URLs, but the usage context is clear enough to act on.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

pagesift_crawlCrawl a siteA

Start a crawl job that follows links from a URL and scrapes every page found. Returns a job id right away; poll it with pagesift_crawl_status. Results can be large, so set a conservative limit and use includePaths/excludePaths to stay on topic.

NameRequiredDescriptionDefault
urlYesThe URL to start crawling from.
delayNoSeconds to wait between requests to the same host.
limitNoMaximum number of pages to crawl.
queryNoCrawl the pages most relevant to this query first.
promptNo
sitemapNo
webhookNo
adaptiveNoWith query: stop once the collected pages cover it.
attachmentsNoWith query: also fetch PDF/Office documents linked from relevant pages.
excludePathsNoRegular expressions for paths to skip.
includePathsNoRegular expressions for paths to keep.
scrapeOptionsNoScrape options applied to every crawled page.
maxConcurrencyNo
webhookHeadersNo
allowSubdomainsNo
attachmentLimitNo
crawlEntireDomainNo
maxDiscoveryDepthNo
allowExternalLinksNo
ignoreQueryParametersNo
deduplicateSimilarURLsNo

TDQS

A3.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=false and openWorldHint=true, and the description adds genuinely new behavioral facts: the call is asynchronous and returns a job id immediately, and result payloads can be large. It stops short of explaining rate-limit behavior, cost, or what happens on partial failure, but the async contract is the critical disclosure and it is present.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three sentences, zero filler, front-loaded with the async job contract before the tuning advice. Every sentence delivers distinct, actionable information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For an open-world async crawl tool with no output schema, the description covers the essentials an agent needs to invoke it safely: the job-id-then-poll pattern and size containment. However, with 21 parameters and nested scrapeOptions, an agent still has no description-level signal for webhook delivery, sitemap modes, query-driven adaptive crawling, or depth controls.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 21 parameters and only 43% schema description coverage, the description should compensate but mentions only three of them (limit, includePaths, excludePaths) plus an indirect nod to delay. High-impact undocumented parameters like sitemap, webhook, prompt, maxDiscoveryDepth, crawlEntireDomain, and adaptive receive no explanation anywhere in the schema or description.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('start a crawl job that follows links from a URL and scrapes every page found'), which implicitly distinguishes it from the single-page pagesift_scrape. It does not explicitly name the sibling tools it supersedes, so the differentiation is inferential rather than stated.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives concrete operational guidance: poll the returned job id with pagesift_crawl_status, set a conservative limit, and use includePaths/excludePaths to stay on topic. It never states when to prefer pagesift_map or pagesift_scrape over this tool, so the routing guidance is incomplete.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

pagesift_crawl_statusCrawl statusA
Read-onlyIdempotent

Get the status, progress and pages collected so far for a crawl or batch scrape job. Read-only.

NameRequiredDescriptionDefault
idYesJob id returned by pagesift_crawl or pagesift_batch_scrape.

TDQS

A3.7/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint, idempotentHint and destructiveHint=false, so 'Read-only' in the description adds nothing. What it does add beyond annotations is the shape of the result — status, progress, pages collected — which is useful given there is no output schema. It does not say what the status values mean or how completion is signalled.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two short sentences, front-loaded with the verb and resource. The trailing 'Read-only' clause is redundant with annotations and is the only wasted fragment.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a single-parameter polling tool with no output schema, the description covers what the call returns (status, progress, collected pages) adequately. Missing are the meaning of status values and what the agent should do on completion, which would help an agent drive a crawl loop.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

One parameter with 100% schema description coverage, and the schema already explains that 'id' is the job id returned by pagesift_crawl or pagesift_batch_scrape. The description adds no further meaning, so the baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb ('Get the status, progress and pages collected') and a specific resource (a crawl or batch scrape job), and scopes it to jobs produced by the two job-creating siblings. An agent can distinguish it from pagesift_crawl and pagesift_batch_scrape without opening any schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied by the resource: poll this after starting a crawl or batch scrape. There is no explicit statement of when to use it versus re-calling the creating tool, no guidance on polling cadence, and no mention of terminal states.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

pagesift_mapMap a siteA
Read-only

List the URLs of a website from its sitemaps, robots.txt and homepage links, without fetching each page. Use it to find the right page before scraping, or to plan a batch scrape.

NameRequiredDescriptionDefault
urlYesThe site to map.
limitNo
searchNoOnly return URLs related to this term.
sitemapNo
includeSubdomainsNo
ignoreQueryParametersNo

TDQS

A4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, openWorldHint=true and destructiveHint=false, so the safety profile is covered. The description adds genuinely useful behavior beyond that: URL discovery comes from sitemaps/robots.txt/homepage links and no pages are fetched, which tells the agent this is a cheap, non-rendering discovery call. It doesn't mention output volume, ordering, or rate-limit behavior, so not a 5.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, zero filler, with the mechanism and the no-page-fetch guarantee front-loaded before the usage hint. Every clause earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description should at least imply the shape of the result (a list of URLs) and help with the ambiguous parameters. It does convey the return is URLs, but leaves the sitemap mode enum, limit semantics and subdomain/query-parameter flags unexplained for a 6-parameter tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Four of six parameters (limit, sitemap, includeSubdomains, ignoreQueryParameters) have no schema description at all, and the description mentions none of them. The 'sitemap' enum with include/skip/only modes is semantically load-bearing and completely unexplained, forcing the agent to guess.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb ('List the URLs') and resource ('of a website'), plus the mechanism (sitemaps, robots.txt, homepage links) and the key constraint 'without fetching each page'. That last clause is exactly what separates it from pagesift_scrape and pagesift_crawl without needing to open their schemas.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

'Use it to find the right page before scraping, or to plan a batch scrape' gives clear when-to-use context and implicitly routes agents toward scrape/batch_scrape as the next step. It never names those siblings explicitly or says when not to use this tool, so it stops short of a 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

pagesift_scrapeScrape a pageA
Read-only

Fetch one URL and return clean content: markdown by default, or html, rawHtml, links, images and metadata. Use it when you know the page you need. For several known URLs use pagesift_batch_scrape; to find URLs on a site use pagesift_map; to collect a whole site use pagesift_crawl. pagesift respects robots.txt and reports blocked pages as errors instead of returning their content.

NameRequiredDescriptionDefault
urlYesThe page to scrape.
proxyNoOnly basic and auto are supported; pagesift does not bypass bot protection.
engineNopagesift extension: fetch with plain HTTP, a headless browser, or decide automatically (default).
maxAgeNoAccept a cached copy at most this many milliseconds old; 0 forces a live fetch.
mobileNo
actionsNoBrowser actions to run before capturing the page.
formatsNoOutput formats. Defaults to ["markdown"].
parsersNo
profileNo
timeoutNoTimeout in milliseconds.
waitForNoMilliseconds to wait before capturing the page.
locationNo
lockdownNo
redactPIINo
pdfOptionsNo
excludeTagsNo
ifNoneMatchNopagesift extension: ETag for a conditional request.
includeTagsNo
jsonOptionsNoOptions for the json format.
queryOptionsNoOptions for the query format.
storeInCacheNo
ifModifiedSinceNopagesift extension: HTTP date for a conditional request.
onlyMainContentNoKeep only the main content. Defaults to true.
screenshotOptionsNo
zeroDataRetentionNo
removeBase64ImagesNo
skipTlsVerificationNo

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly, non-destructive, open-world behavior, so the description is not obligated to restate safety. It adds genuinely new context: robots.txt is respected and blocked pages surface as errors rather than content, which prevents an agent from misreading an empty result as a successful scrape. It does not cover caching behavior or rate/latency characteristics.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three tight sentences: capability first, sibling routing second, a behavioral caveat last. No filler, and the most important information is front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a complex, 27-parameter tool with no output schema, the description covers purpose, alternatives, default output, and the robots.txt failure mode, which is enough to call it correctly in the common case. Gaps remain around caching/conditional-request semantics and browser actions, but they are secondary for the typical single-page scrape.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 27 parameters and only ~48% schema description coverage, the description needs to compensate and largely does not. It only clarifies the formats parameter (markdown is the default, plus html/rawHtml/links/images/metadata); high-impact options like proxy restrictions, engine, actions, redactPII, lockdown, zeroDataRetention, storeInCache, and conditional-request fields are left to the schema or undocumented entirely.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource (fetch one URL, return clean content) and enumerates the default and alternative output formats. It also names each sibling it is not, so an agent can distinguish single-page scraping from batch, map, and crawl without opening any schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives an explicit selection rule: use this when you already know the page, use pagesift_batch_scrape for several known URLs, pagesift_map to discover URLs, pagesift_crawl to collect a whole site. Every alternative is named with the condition that selects it.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 5 tool updatesv0.81.2
    • First observedpagesift_batch_scrape
    • First observedpagesift_crawl
    • First observedpagesift_crawl_status
    • First observedpagesift_map
    • First observedpagesift_scrape

TDQS

A4/5.0

Scored across 5 tools

Disambiguation5/5

Each tool has a clearly distinct role: single-page scrape, URL discovery via map, whole-site crawl, job status polling, and batch scraping. Descriptions explicitly cross-reference the alternatives, making selection straightforward.

Naming Consistency4/5

All names use the pagesift_ prefix and snake_case, which is highly consistent. The only slight deviation is crawl_status, which is a noun/state phrase rather than a verb_noun action like the others.

Tool Count5/5

Five tools are well-scoped for a web scraping service and each earns its place. There are no redundant tools, and the set covers the core workflows without bloat.

Completeness4/5

The surface covers single-page scraping, batch scraping, URL mapping, crawling, and crawl status. Minor gaps include no explicit job cancellation or deletion operation, but agents can work around these limitations.

Related MCP Connectors

Related MCP Servers

  • F
    license
    Not graded
    quality
    Not graded
    maintenance
    An MCP server for web content extraction that converts HTML pages into clean, LLM-optimized Markdown using Mozilla's Readability. It supports batch processing, intelligent multi-page crawling, and configurable caching while respecting robots.txt standards.
    15 npm
    -
  • F
    license
    Not graded
    quality
    D
    maintenance
    A locally-hosted MCP server that provides AI assistants with advanced web crawling capabilities, including structured data extraction, deep site crawling, and page screenshots. It enables users to convert single or multiple URLs into clean Markdown content for processing by LLMs without requiring external API keys for basic features.
    -
  • A
    license
    Not graded
    quality
    D
    maintenance
    Open-source web scraper and extraction MCP server with JavaScript rendering, markdown output, PDF/DOCX parsing, structured errors, and validated extraction contract diagnostics for agents.
    2
    AGPL 3.0
  • A
    license
    Not graded
    quality
    F
    maintenance
    An open-source web retrieval MCP server that fetches, crawls, and searches the web, returning clean markdown for AI agents. It integrates with Claude MCP, LangChain, and other frameworks for agentic web access.
    2
    MIT