PageSift
PageSift is a self-hosted MCP server for web and document ingestion: it scrapes, maps, crawls, batch-scrapes, and interacts with web pages while returning clean content and managing asynchronous jobs.
Scrape a single URL into markdown, HTML, raw HTML, links, images, metadata, extracted structured data, chunks, screenshots, or PDF pages; supports caching, conditional requests, browser actions, query filtering, and PDF parser options.
Map a site to list URLs from sitemaps, robots.txt, and homepage links without fetching every page.
Crawl a site by following links from a seed URL, with limits, path filters, depth controls, sitemap options, query relevance, and optional attachment fetching.
Batch scrape up to 10,000 known URLs as one job with shared scrape options.
Poll or cancel crawl/batch jobs and retrieve collected pages, progress, errors, robots-blocked URLs, and expiry details.
Keep browser pages open after scrape and interact with them using browser actions or plain-language prompts, then close the session to free memory.
Respect robots.txt and report blocked pages, with configurable fetch engine, proxy/egress, headers, timeouts, rendering wait behavior, and ad blocking.
Provides LangChain integration so LangChain applications can use PageSift to scrape URLs, crawl sites, and parse documents into clean Markdown or structured JSON.
pagesift 1.6.0
Local-first web & document ingestion engine in Rust — HTTP/browser capture, layout-aware PDF/Office/OCR parsing, persistent sync, verifiable data graphs, and evidence-grounded AI context.
One self-hosted Rust binary with a CLI, Firecrawl v1/v2 & Jina Reader HTTP APIs, an MCP server, a C ABI, and native Python / Node.js packages.
English · 简体中文 · 日本語 · Español
Overview
pagesift turns live web pages, uploaded binary documents, archives, and long-lived monitored sources into clean Markdown, schema-validated JSON, provenance-tracked DataGraph relations, and budget-compiled ContextPack evidence inside a single memory-bounded Rust binary. HTTP fetching, native document readers, HTML extraction, adaptive sync state, incremental context indexing, and evidence storage run in-process without requiring Python, JVM, or Node workers; a headless browser pool, NVIDIA GPU OCR, and user-configured LLM endpoints attach only when requested.
Drop-in or companion for: Firecrawl (self-hosted v1/v2 API), Jina Reader (/r/ and /s/), Crawl4AI, trafilatura, Readability, pymupdf4llm, Docling, Kreuzberg, and xberg.
Related MCP server: Crawl4AI MCP Server
At a glance
Layered runtime: Entry surfaces (
CLI/HTTP/MCP/ FFI) → policy & SSRF-guarded net → adaptive strategy (HTTP/ embedded JSON / browser) → native format readers →DataGraph,Context& append-only store → cgroup-sized NUMA worker pool (pagesift-flow)Inputs: URLs, HTML, PDF (text + optional PP-OCRv6 / formulas), uploaded images (OCR when enabled), Office & OpenDocument, RTF/WordPerfect/HWP, e-books, emails, archives (
zip,tar,7z), SQLite/CSV, 373 code languages, and WARC (format matrix)Outputs: Markdown, HTML, raw HTML, links, images/attachments, RAG chunks, schema JSON,
DataGraph(graph.jsonl, citations, claim lifecycles, conflicts,fact_clusters,?asOf=),ContextPack(KnowledgeUnit,ContextDelta,provenanceManifest), plus structured warnings on partial capturesInterfaces: CLI · Firecrawl v1/v2 HTTP API · Jina Reader (
/r/,/s/) · MCP (stdio / HTTP) · Python · Node.js / TypeScript · Rust crate · C ABI (Go, Java, Kotlin, C#, Ruby, PHP, Lua, Perl, Dart, Swift, Zig) · LangChain & LlamaIndexPlatforms & ops: Single binary (
serve --role gateway,worker,standby), Linux x86_64/aarch64 (glibc ≥ 2.28 & musl), macOS arm64, Windows x86_64, Dockerghcr.io/yolow92/pagesift:1.6.0Index & release: Machine-readable index llms.txt · Apache-2.0 or commercial support · 1.6.0 (release notes)
Install and quick start
Surface | Install | Minimal call |
CLI |
|
|
Python |
|
|
Node.js |
|
|
Server |
|
|
MCP | — |
|
CLI
pagesift scrape https://example.com # Clean Markdown to stdout
pagesift scrape --graph https://example.com # Markdown + DataGraph provenance & claims
pagesift scrape report.pdf # Local PDF, Office, e-book, email, archive, code or WARC
pagesift --ocr scrape scan.pdf # OCR scanned PDF pages or uploaded images
pagesift crawl https://docs.rs --limit 500 --out ./data/ # Crawl to disk (add --graph for graph.jsonl)
pagesift graph inspect ./data/graph.jsonl # Inspect citations, lineage, claims and conflicts
pagesift sync init --id docs --seed https://example.com # Initialize and run a persistent Loop sync source
pagesift context pack --file report.pdf "rate limits" # Compile a token-budgeted ContextPack with citations
pagesift extract --schema schema.json https://example.com # Schema JSON with CSS selector relocation
pagesift doctor # Inspect local browsers, GPU and modelsPython & Node.js
import pagesift
page = pagesift.scrape("https://example.com", formats=["markdown", "links"])
print(page.markdown, page.metadata["title"], page.links)
with pagesift.Scraper(store_path="./data/store") as s:
pack = s.context(action="pack", source_ids=["docs"], query="rate limits", budget_tokens=1024)
print(pack["pack"]["compiledContext"])import { Scraper } from '@pagesift/node'
const scraper = new Scraper({ storePath: './data/store' })
const page = await scraper.scrape('https://example.com', { formats: ['markdown', 'links'] })
const pack = await scraper.context({ action: 'pack', sourceIds: ['docs'], query: 'rate limits', budgetTokens: 1024 })
console.log(page.markdown, pack.pack)HTTP server (Firecrawl v1/v2, Jina Reader, /v2/sources, /v2/graph, /v2/context) & MCP
docker run -d -p 3002:3002 -v pagesift-data:/data -e PAGESIFT_API_KEYS=sk-test ghcr.io/yolow92/pagesift:1.6.0
curl -s localhost:3002/v2/scrape -H 'Authorization: Bearer sk-test' -H 'Content-Type: application/json' \
-d '{"url":"https://example.com","formats":["markdown"]}'
curl -s localhost:3002/r/https://example.com -H 'Authorization: Bearer sk-test'
curl -s localhost:3002/v2/parse -H 'Authorization: Bearer sk-test' -F file=@report.pdf -F 'options={"formats":["markdown"]}'
curl -s localhost:3002/v2/context/pack -H 'Authorization: Bearer sk-test' -H 'Content-Type: application/json' \
-d '{"sourceIds":["docs"],"query":"rate limits","budgetTokens":1024}'
pagesift mcp # stdio for a single agent
pagesift mcp http --port 3003 # shared local MCP daemonMCP exposes pagesift_scrape, pagesift_batch_scrape, pagesift_crawl, pagesift_poll_job, pagesift_cancel_job, pagesift_map, pagesift_interact, pagesift_interact_stop, pagesift_sync, pagesift_graph_query, pagesift_context, and (with a configured model) pagesift_agent / pagesift_agent_status. See Shared MCP service and Context reference.
Capabilities
Organized by architectural plane—showing implemented surfaces, data paths, and reference docs:
Plane | Implemented capabilities | Reference |
Interfaces & compatibility | Self-hosted CLI, Firecrawl v1/v2 HTTP API ( | |
Acquisition & safety | Conditional HTTP ( | |
Native parsing & OCR | Pure-Rust readers down to container/file level ( | |
Extraction & agents | Main-content Markdown with math/code/tables ( | |
Continuous Loop & monitors | Persistent | |
DataGraph & provenance | Cross-page and cross-source evidence graph ( | |
Context & compilation | Cross-version |
Performance at a glance
Benchmarks are tracked separately so this page stays focused on system structure; all recorded baselines, hardware details, caveats, and competitor comparisons live in docs/benchmarks.md:
Latency (recorded benchmarks, single-worker, lower is better): PDF → Markdown
2.5–2.58 ms/page(1,269 pages); HTML → Markdown0.23–0.24 ms/page(600 docs); main-content extraction1.0 ms/page(597 pages); ContextBoundedUotreconcile63.9 µsandContextPackcompile67–225 µs(192/384-token budgets).Throughput, quality & memory (recorded benchmarks): Office in-process
81.9–210.0 files/s(higher is better); HTML → Markdown1.000mean F1; WCXB0.866F1; RAG query filter0.929recall (higher is better); ContextSearch + Pack2,708.8–24,434.4 QPSacross1–16threads (25.9–28.8 MiBRSS, higher QPS / lower RSS is better);17–395 MBpeak memory/worker without a browser across domain runs.Full scorecards & evaluations: benchmarks.md · Full benchmark report · Latest Context & Loop release evaluation
Architecture and core flows
The core runs as one compiled Rust binary with zero managed interpreter runtimes behind parsing, storage, or scheduling. Memory, CPU workers, and browser tabs pass through shared cgroup-aware admission, and internal runtime limits adapt only when online statistics clear significance thresholds (δ = 0.05).
Request and ingestion pipeline
flowchart TD
CLI["CLI / Rust caller"] --> Engine["pagesift-engine: scrape, crawl, Loop sync, Context"]
HTTP["HTTP API: Firecrawl v1/v2, Jina, /v2/sources, /v2/graph, /v2/context"] --> Engine
Bind["Python / Node / MCP / C ABI"] --> Engine
Engine --> Policy["pagesift-hooks & pagesift-policy<br/>robots, host rate limits, backoff, identity leases"]
Policy --> Net["pagesift-net + pagesift-compress (yolowzstd)<br/>HTTP/conditional fetch, SSRF guards, TLS/proxy"]
Net -.-> ITNA["pagesift-itna (opt-in)<br/>barrier class + shared budget"]
ITNA -.-> CDP
Net --> Plan{"pagesift-plan<br/>HTTP, JSON or browser?"}
Plan -->|HTTP or embedded JSON| DOM["pagesift-html (yolowhtml-dom): single-pass arena DOM"]
Plan -->|Browser escalation| CDP["pagesift-cdp: Lightpanda / moli / Chrome"]
CDP --> DOM
Net -->|PDF| PDF["pagesift-pdf (yolowpdf + yolow64)"]
Net -->|Office / e-book / email / archive / text| Office["pagesift-office (yolowdoc)"]
Net -->|Image / scanned PDF| OCR["pagesift-ocr, pagesift-infer, pagesift-gpu"]
Office -->|member / attachment| PDF
Office -->|source code| Syntax["pagesift-syntax (yolowsyntax)"]
PDF -->|scanned page| OCR
DOM --> Frontier["pagesift-frontier: URL scoring"]
Frontier -.->|next crawl / sync candidates| Engine
DOM --> Content["pagesift-content (yolowhtml) & pagesift-extract<br/>main content, selectors, evidence, chunks"]
Content --> Graph["pagesift-graph (DataGraph)<br/>citations, provenance, claims, conflicts, clusters, asOf"]
PDF --> Graph
Office --> Graph
Graph --> Ctx["Context (pagesift-graph + pagesift-extract + pagesift-stat)<br/>KnowledgeUnit (BoundedUot), ContextDelta, Search, ContextPack, VOI Action & Learn"]
Graph --> Out["pagesift-sink & pagesift-store<br/>graph.jsonl, offloaded snapshots, SyncStore, ContextStore (ns::CONTEXT + chis), Experience"]
Ctx <--> Out
Ctx -.->|VOI + 4D shadow-price bounded acquisition| Engine
Out -.->|priors, URL rewrites, checkpoints| PlanContinuous Sync closed loop (Observe → Learn → Decide → Execute → Verify)
Long-lived sources registered via pagesift sync or /v2/sources run an adaptive closed loop per tick without discarding prior verified evidence on transient uncertainty, and optionally reconcile changed documents incrementally into ContextStore:
flowchart LR
Obs["Observe & Schedule<br/>pagesift-store + pagesift-stat<br/>Non-aliased periodicity, Gamma-Poisson<br/>rates, Lagrange water-filling, Hub pull-forward"]
Dec["Decide Modality<br/>pagesift-plan<br/>304/HTTP/Browser cost + tab shadow price,<br/>bounded recovery probes, tenant transfer"]
Exe["Execute & Screen<br/>pagesift-engine/sync/capture<br/>HTTP / JSON synthesis / CDP / 304;<br/>censor decode:, truncation & bad 304s"]
Ver["Verify & Commit<br/>pagesift-graph + pagesift-store<br/>Active/Superseded/Withdrawn/Unverified claims,<br/>ContextDelta & unit lineage, snapshot offload"]
Obs --> Dec --> Exe --> Ver --> ObsEvidence-checked /v2/agent and multi-server fleet topology
flowchart LR
subgraph Agent["Bounded /v2/agent research loop"]
Q[Question + Schema] --> M[User LLM step]
M --> T[search / map / scrape / extract<br/>enforced robots & SSRF policy]
T --> Ev[Archived evidence]
Ev --> Check{Complete?}
Check -->|budget left| M
Check -->|done| Verify[Schema match + field grounding + warnings]
endflowchart LR
subgraph Fleet["Single binary multi-node cluster (pagesift-cluster)"]
C[Clients] --> G[Gateway<br/>API, jobs, webhooks]
G -->|rendezvous host hash| W1[Worker]
G --> W2[Worker]
G -.->|fenced leases & checkpoints| S[Standby]
endWorkspace crate map
Layer | Crates |
Entry surfaces & FFI |
|
Orchestration, strategy & graph |
|
Network, policy & browser |
|
Web extraction & discovery |
|
Native document & OCR readers |
|
Runtime, math & persistence |
|
See Architecture reference for cgroup sizing, statistical defaults (δ = 0.05 USL/survival/Thompson tuning), incremental checkpoint semantics, and byte-range verified media archives.
When to use it, and when not
Good fit
Feeding live web pages, PDFs, Office suites, archives, or source code into AI agents, RAG pipelines, or downstream analytics with zero external runtime workers
Compiling token-budgeted, provenance-cited
ContextPackprompts that preserve table headers, qualifiers, units, temporal validity, and opposing multi-source conflictsSelf-hosting Firecrawl- or Jina Reader-compatible APIs and shared MCP tooling without per-page SaaS fees
Continuous monitoring/sync of changing sources where cost-aware browser escalation, lineage tracking, and claim-level conflict detection matter
Pick something else when
You want a managed SaaS cloud.
pagesiftruns self-hosted on your infrastructure.You need automatic CAPTCHA solving. Interactive challenges are classified and paused for human resolution (
[itna]recovery is opt-in and does not bypass CAPTCHAs).You only process standalone image URLs without local uploads or OCR. Uploaded/local images require
--ocr([ocr] enabled = true); image URLs and OCR-off uploads returnUNSUPPORTED_OPTION.
License
Apache-2.0. For dedicated support, SLAs, or enterprise agreements, see commercial license and support or write to licensing@pagesift.dev.
What's new in 1.6.0
PageSift 1.6.0 introduces Context (docs/context.md) across CLI (pagesift context), HTTP (/v2/context), MCP (pagesift_context), Python (Scraper.context / acontext), Node.js (Scraper.context), and C ABI (pagesift_context), alongside upgrading upstream yolowhtml / yolowhtml-dom to 0.1.2 (<caption> table headings, hidden="until-found" article retention, and input_truncated capture reliability screening): adds cross-version KnowledgeUnit identity and explicit ambiguity/split/merge lineage (boundedUot, greedyDice, exactAnchorOnly); persists atomic dependency-governed ContextDelta updates, IndexVisibility lag metrics, and idempotent anti-ABA consumer checkpoints in pagesift-store (ns::CONTEXT, immune to cache eviction); provides multi-strategy lexical/symbol/temporal (asOf + effectiveAt)/lineage Search with co-retrieved conflict counterparts; compiles token- and MustKeep-constrained ContextPack payloads (submodularClosure, boundedDpp, mmrGreedy, topKGreedy, edgeSalience) with provenanceManifest; and adds bounded Action (context, research, verify, watch, changes) driven by gated Good–Turing / Chapman coverage diagnosis, Howard Value of Information, 4D admission shadow prices, and CUSUM / BernoulliBocpd drift-guarded isotonic Learn calibration.
Release notes · 1.5.6 · 1.5.5 · 1.5.4 · 1.5.3 · 1.5.2 · 1.5.1 · all releases · CHANGELOG
Community
Questions or feedback: Discord · Issues · Discussions
Citation
@software{pagesift2026,
title = {PageSift: Rust Web and Document Extraction Engine for AI Pipelines},
author = {YOLOW92},
year = {2026},
url = {https://github.com/YOLOW92/pagesift},
note = {Version 1.6.0}
}Star History
Available Tools
8 toolspagesift_batch_scrapeBatch scrapeA
Scrape a list of known URLs as one job with the same options. Use pagesift_scrape for one URL and pagesift_crawl when the URLs are not known yet. Invalid URLs are returned in invalidURLs instead of failing the job; a page that fails is recorded in the job's errors while the other pages still return. All requests share the server's concurrency and per-host rate limits. urls accepts up to 10000 entries and options applies to every URL (no per-URL overrides). Returns a job id; poll it with pagesift_poll_job.
Inputs: urls (required), options. Returns: id, next, status, invalidURLs
pagesift_cancel_jobCancel a jobAIdempotent
Cancel a running pagesift_crawl or pagesift_batch_scrape job. Use pagesift_poll_job to read a job and pagesift_cancel_job to stop it. No new pages are scheduled; pages already collected stay available through pagesift_poll_job, and work that finishes after the cancel is discarded. A completed, failed or already cancelled job is left unchanged and its current status is returned, so repeated calls are safe. An unknown or expired id is an error.
Inputs: id (required). Returns: total, status, success, completed
pagesift_crawlCrawl a siteA
Start a crawl job that follows links from a URL and scrapes every page found. Use pagesift_scrape when you need one page and pagesift_map when you only need URLs. By default the crawl includes sitemap URLs, follows links within the seed's path scope, has no depth limit and uses per-host concurrency; maxDiscoveryDepth and maxConcurrency lower these. Pages blocked by robots.txt or failing are reported in the job's errors and the crawl continues. Returns a job id right away; poll it with pagesift_poll_job. Results can be large, so set a conservative limit and use includePaths/excludePaths to stay on topic.
Inputs: url (required), delay, limit, query, +16 more. Returns: id, next, status, invalidURLs
pagesift_interactAct on a kept pageADestructive
Continue on a browser page kept by pagesift_scrape with keepSession. Send either prompt (plain-language instruction; needs an LLM on the server) or actions (a list of browser actions), not both. Actions can permanently change the page or lose unsaved state. State carries over between calls. Returns the page afterwards in data; with a prompt also output (the answer) and steps. Finish with pagesift_interact_stop to free the page. No live view or CDP URL is offered.
Inputs: scrapeId (required), prompt, actions, timeout. Returns: data, error, steps, killed, +8 more
pagesift_interact_stopClose a kept pageADestructive
Close a browser page kept by pagesift_scrape keepSession or pagesift_interact and free its memory. Data already returned stays with the caller; the scrapeId becomes invalid, so later calls with it fail. Calling stop twice or after expiry returns an error. Pages expire automatically after 10 minutes if never stopped, and are reclaimed sooner when idle under memory pressure.
Inputs: scrapeId (required). Returns: success, sessionDurationMs
pagesift_mapMap a siteARead-onlyIdempotent
List a website's URLs from its sitemaps, robots.txt and homepage links without fetching each page. sitemap include (default) uses both sitemaps and page links, skip ignores sitemaps, only uses sitemaps without reading the homepage. If no sitemap is found it falls back to homepage links; limit truncates after search ranking. Use pagesift_scrape for one page and pagesift_crawl to fetch many pages.
Inputs: url (required), limit, search, sitemap, +3 more. Returns: links
pagesift_poll_jobPoll a jobARead-onlyIdempotent
Get the status, progress and pages collected so far for a pagesift_crawl or pagesift_batch_scrape job. status is scraping, completed, cancelled or failed; completed, cancelled and failed are final. While scraping, poll again after a few seconds; stop when the status is final. Pages collected so far are in data and remain available for 24 hours. Use pagesift_cancel_job to stop a running job.
Inputs: id (required). Returns: data, total, errors, status, +3 more
pagesift_scrapeScrape a pageARead-onlyIdempotent
Fetch one URL and return clean content: markdown by default, or html, rawHtml, links, images and metadata. Use it when you know the page you need. For several known URLs use pagesift_batch_scrape; to find URLs on a site use pagesift_map; to collect a whole site use pagesift_crawl. pagesift respects robots.txt and reports blocked pages as errors instead of returning their content. Results are cached for 48 hours by default; set maxAge to 0 to force a live fetch. With keepSession the browser page stays open for pagesift_interact and its id is returned as metadata.scrapeId; kept pages expire after 10 minutes (sooner when idle under memory pressure). Advanced options: actions runs browser steps before capture, chunking splits Markdown, queryFilter keeps blocks relevant to a query, and parsers selects PDF pages.
Inputs: url (required), proxy, egress, engine, +26 more. Returns: html, json, links, pages, +12 more
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 8 tools
Single-page scrape, batch scrape, crawl, and map each target distinct URL-discovery or fetching modes, while poll/cancel and interact/stop form clear operational pairs. The only overlap is that scrape can run pre-capture actions while interact continues an existing kept page, but the descriptions clearly separate 'capture now' from 'continue session'.
All tools use the pagesift_ prefix and snake_case, mostly following a verb or verb_noun pattern such as scrape, crawl, poll_job, and cancel_job. pagesift_interact_stop breaks the verb-first ordering, and batch_scrape uses a modifier, but the set remains predictable and readable.
Eight tools is well-scoped for a web scraping and crawling server: single scrape, batch scrape, map, crawl, two job controls, and two session controls. Each tool has a clear role, with no obviously redundant or missing lifecycle entry.
The surface covers URL discovery with map, single and batch fetching, whole-site crawling, asynchronous job status and cancellation, and browser session continuation and cleanup. For the stated domain, there are no obvious dead ends: job results remain retrievable via poll, and kept pages are managed through interact_stop or expiry.
Maintenance
Related MCP Connectors
Scrape, crawl, map & search the web. Open-source, self-hostable Rust crawler & search for AI agents.
Pages, crawls and URL batches as clean Markdown and JSON for AI agents, with an Evidence Record.
Web pages and whole sites to clean Markdown for RAG and AI agents. Crawls links and sitemaps.
Turn public web pages into clean Markdown for LLMs, RAG and AI agents
Related MCP Servers
- FlicenseNot gradedqualityNot gradedmaintenanceAn MCP server for web content extraction that converts HTML pages into clean, LLM-optimized Markdown using Mozilla's Readability. It supports batch processing, intelligent multi-page crawling, and configurable caching while respecting robots.txt standards.48 npm-
- FlicenseNot gradedqualityDmaintenanceA locally-hosted MCP server that provides AI assistants with advanced web crawling capabilities, including structured data extraction, deep site crawling, and page screenshots. It enables users to convert single or multiple URLs into clean Markdown content for processing by LLMs without requiring external API keys for basic features.-
- AlicenseNot gradedqualityDmaintenanceOpen-source web scraper and extraction MCP server with JavaScript rendering, markdown output, PDF/DOCX parsing, structured errors, and validated extraction contract diagnostics for agents.2AGPL 3.0
- AlicenseNot gradedqualityDmaintenanceAn open-source web retrieval MCP server that fetches, crawls, and searches the web, returning clean markdown for AI agents. It integrates with Claude MCP, LangChain, and other frameworks for agentic web access.2MIT