Skip to main content
Glama

pagesift 1.6.0

Local-first web & document ingestion engine in Rust — HTTP/browser capture, layout-aware PDF/Office/OCR parsing, persistent sync, verifiable data graphs, and evidence-grounded AI context.
One self-hosted Rust binary with a CLI, Firecrawl v1/v2 & Jina Reader HTTP APIs, an MCP server, a C ABI, and native Python / Node.js packages.

License: Apache-2.0 Commercial license Release crates.io PyPI npm PageSift MCP server – quality and maintenance score on Glama Discord

English · 简体中文 · 日本語 · Español

Overview

pagesift turns live web pages, uploaded binary documents, archives, and long-lived monitored sources into clean Markdown, schema-validated JSON, provenance-tracked DataGraph relations, and budget-compiled ContextPack evidence inside a single memory-bounded Rust binary. HTTP fetching, native document readers, HTML extraction, adaptive sync state, incremental context indexing, and evidence storage run in-process without requiring Python, JVM, or Node workers; a headless browser pool, NVIDIA GPU OCR, and user-configured LLM endpoints attach only when requested.

Drop-in or companion for: Firecrawl (self-hosted v1/v2 API), Jina Reader (/r/ and /s/), Crawl4AI, trafilatura, Readability, pymupdf4llm, Docling, Kreuzberg, and xberg.

Related MCP server: Crawl4AI MCP Server

At a glance

  • Layered runtime: Entry surfaces (CLI / HTTP / MCP / FFI) → policy & SSRF-guarded net → adaptive strategy (HTTP / embedded JSON / browser) → native format readers → DataGraph, Context & append-only store → cgroup-sized NUMA worker pool (pagesift-flow)

  • Inputs: URLs, HTML, PDF (text + optional PP-OCRv6 / formulas), uploaded images (OCR when enabled), Office & OpenDocument, RTF/WordPerfect/HWP, e-books, emails, archives (zip, tar, 7z), SQLite/CSV, 373 code languages, and WARC (format matrix)

  • Outputs: Markdown, HTML, raw HTML, links, images/attachments, RAG chunks, schema JSON, DataGraph (graph.jsonl, citations, claim lifecycles, conflicts, fact_clusters, ?asOf=), ContextPack (KnowledgeUnit, ContextDelta, provenanceManifest), plus structured warnings on partial captures

  • Interfaces: CLI · Firecrawl v1/v2 HTTP API · Jina Reader (/r/, /s/) · MCP (stdio / HTTP) · Python · Node.js / TypeScript · Rust crate · C ABI (Go, Java, Kotlin, C#, Ruby, PHP, Lua, Perl, Dart, Swift, Zig) · LangChain & LlamaIndex

  • Platforms & ops: Single binary (serve --role gateway, worker, standby), Linux x86_64/aarch64 (glibc ≥ 2.28 & musl), macOS arm64, Windows x86_64, Docker ghcr.io/yolow92/pagesift:1.6.0

  • Index & release: Machine-readable index llms.txt · Apache-2.0 or commercial support · 1.6.0 (release notes)

Install and quick start

Surface

Install

Minimal call

CLI

cargo install pagesift --locked

pagesift scrape https://example.com

Python

pip install pagesift

pagesift.scrape("https://example.com").markdown

Node.js

npm install @pagesift/node

(await new Scraper().scrape("https://example.com")).markdown

Server

docker run -p 3002:3002 ghcr.io/yolow92/pagesift:1.6.0

curl localhost:3002/r/https://example.com

MCP

—

claude mcp add pagesift -- pagesift mcp

CLI

pagesift scrape https://example.com                          # Clean Markdown to stdout
pagesift scrape --graph https://example.com                  # Markdown + DataGraph provenance & claims
pagesift scrape report.pdf                                   # Local PDF, Office, e-book, email, archive, code or WARC
pagesift --ocr scrape scan.pdf                               # OCR scanned PDF pages or uploaded images
pagesift crawl https://docs.rs --limit 500 --out ./data/     # Crawl to disk (add --graph for graph.jsonl)
pagesift graph inspect ./data/graph.jsonl                    # Inspect citations, lineage, claims and conflicts
pagesift sync init --id docs --seed https://example.com      # Initialize and run a persistent Loop sync source
pagesift context pack --file report.pdf "rate limits"        # Compile a token-budgeted ContextPack with citations
pagesift extract --schema schema.json https://example.com    # Schema JSON with CSS selector relocation
pagesift doctor                                              # Inspect local browsers, GPU and models

Python & Node.js

import pagesift

page = pagesift.scrape("https://example.com", formats=["markdown", "links"])
print(page.markdown, page.metadata["title"], page.links)

with pagesift.Scraper(store_path="./data/store") as s:
    pack = s.context(action="pack", source_ids=["docs"], query="rate limits", budget_tokens=1024)
    print(pack["pack"]["compiledContext"])
import { Scraper } from '@pagesift/node'

const scraper = new Scraper({ storePath: './data/store' })
const page = await scraper.scrape('https://example.com', { formats: ['markdown', 'links'] })
const pack = await scraper.context({ action: 'pack', sourceIds: ['docs'], query: 'rate limits', budgetTokens: 1024 })
console.log(page.markdown, pack.pack)

HTTP server (Firecrawl v1/v2, Jina Reader, /v2/sources, /v2/graph, /v2/context) & MCP

docker run -d -p 3002:3002 -v pagesift-data:/data -e PAGESIFT_API_KEYS=sk-test ghcr.io/yolow92/pagesift:1.6.0

curl -s localhost:3002/v2/scrape -H 'Authorization: Bearer sk-test' -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com","formats":["markdown"]}'
curl -s localhost:3002/r/https://example.com -H 'Authorization: Bearer sk-test'
curl -s localhost:3002/v2/parse -H 'Authorization: Bearer sk-test' -F file=@report.pdf -F 'options={"formats":["markdown"]}'
curl -s localhost:3002/v2/context/pack -H 'Authorization: Bearer sk-test' -H 'Content-Type: application/json' \
  -d '{"sourceIds":["docs"],"query":"rate limits","budgetTokens":1024}'

pagesift mcp                                                 # stdio for a single agent
pagesift mcp http --port 3003                                # shared local MCP daemon

MCP exposes pagesift_scrape, pagesift_batch_scrape, pagesift_crawl, pagesift_poll_job, pagesift_cancel_job, pagesift_map, pagesift_interact, pagesift_interact_stop, pagesift_sync, pagesift_graph_query, pagesift_context, and (with a configured model) pagesift_agent / pagesift_agent_status. See Shared MCP service and Context reference.

Capabilities

Organized by architectural plane—showing implemented surfaces, data paths, and reference docs:

Plane

Implemented capabilities

Reference

Interfaces & compatibility

Self-hosted CLI, Firecrawl v1/v2 HTTP API (/v2/scrape, /v2/crawl, /v2/parse, /v2/monitor, /v2/sources, /v2/graph, /v2/context, /v2/agent, /interact), Jina Reader (/r/, /s/), MCP (stdio/HTTP), Python, Node.js/TS, Rust library, C ABI bindings

parse-api.md · context.md · mcp-service.md

Acquisition & safety

Conditional HTTP (ETag/Last-Modified), SSRF & DNS-rebinding guards, robots.txt & per-host rate limits, identity leases, embedded JSON synthesis (__NEXT_DATA__/__NUXT_DATA__/JSON-LD), optional CDP browser pool (Lightpanda/moli/Chrome) + interactive /interact sessions, opt-in [itna] access-barrier classification

interact.md · http-sessions.md · bot.md

Native parsing & OCR

Pure-Rust readers down to container/file level (yolowhtml, yolowpdf, yolowdoc, yolowsyntax, yolowzstd, yolow64) covering web, Office/ODF, e-books, email, nested archives, SQLite/CSV, 373 code languages (AST function/class splits for 32+ languages), WARC I/O, and PP-OCRv6 + PP-FormulaNet on CPU/GPU

parse-api.md

Extraction & agents

Main-content Markdown with math/code/tables (<caption> & hidden="until-found"), RAG queryFilter & chunking, selector-memory schema extraction with automatic DOM relocation, and model-planned /v2/agent research verified against archived page evidence

structured-extract.md · agent-api.md

Continuous Loop & monitors

Persistent pagesift sync / /v2/sources sources with anti-aliased periodic detection, hierarchical Gamma-Poisson shrinkage & Lagrange water-filling under tick budgets, hub co-change propagation, modality cost & tab shadow pricing, bounded DirectBrowser recovery probes, surprise forgetting, tenant-isolated cold-start transfer, capture reliability censoring, and /v2/monitor webhooks & substantive block diffs

monitor.md · incremental-collection.md

DataGraph & provenance

Cross-page and cross-source evidence graph (graph.jsonl, graph_summary.json, /v2/graph/claims, /history, /conflicts, ?asOf=): persisted document citations, redirect/canonical/attachment provenance, contextual unit/currency disambiguation, OriginRelation independent roots, fact_clusters, and claim lifecycles (Active, Superseded, Withdrawn, Unverified, EvidenceDegraded)

architecture.md

Context & compilation

Cross-version KnowledgeUnit identity & ambiguity tracking (boundedUot / greedyDice / exactAnchorOnly), atomic dependency-governed ContextDelta & idempotent consumer checkpoints, multi-strategy lexical/symbol/temporal/lineage Search, budget- & MustKeep-constrained ContextPack (submodularClosure / boundedDpp / edgeSalience) with provenanceManifest, and bounded VOI + shadow-price Action with drift-guarded online Learn

context.md

Performance at a glance

Benchmarks are tracked separately so this page stays focused on system structure; all recorded baselines, hardware details, caveats, and competitor comparisons live in docs/benchmarks.md:

  • Latency (recorded benchmarks, single-worker, lower is better): PDF → Markdown 2.5–2.58 ms/page (1,269 pages); HTML → Markdown 0.23–0.24 ms/page (600 docs); main-content extraction 1.0 ms/page (597 pages); Context BoundedUot reconcile 63.9 µs and ContextPack compile 67–225 µs (192/384-token budgets).

  • Throughput, quality & memory (recorded benchmarks): Office in-process 81.9–210.0 files/s (higher is better); HTML → Markdown 1.000 mean F1; WCXB 0.866 F1; RAG query filter 0.929 recall (higher is better); Context Search + Pack 2,708.8–24,434.4 QPS across 1–16 threads (25.9–28.8 MiB RSS, higher QPS / lower RSS is better); 17–395 MB peak memory/worker without a browser across domain runs.

  • Full scorecards & evaluations: benchmarks.md · Full benchmark report · Latest Context & Loop release evaluation

Architecture and core flows

The core runs as one compiled Rust binary with zero managed interpreter runtimes behind parsing, storage, or scheduling. Memory, CPU workers, and browser tabs pass through shared cgroup-aware admission, and internal runtime limits adapt only when online statistics clear significance thresholds (δ = 0.05).

Request and ingestion pipeline

flowchart TD
    CLI["CLI / Rust caller"] --> Engine["pagesift-engine: scrape, crawl, Loop sync, Context"]
    HTTP["HTTP API: Firecrawl v1/v2, Jina, /v2/sources, /v2/graph, /v2/context"] --> Engine
    Bind["Python / Node / MCP / C ABI"] --> Engine

    Engine --> Policy["pagesift-hooks & pagesift-policy<br/>robots, host rate limits, backoff, identity leases"]
    Policy --> Net["pagesift-net + pagesift-compress (yolowzstd)<br/>HTTP/conditional fetch, SSRF guards, TLS/proxy"]
    Net -.-> ITNA["pagesift-itna (opt-in)<br/>barrier class + shared budget"]
    ITNA -.-> CDP

    Net --> Plan{"pagesift-plan<br/>HTTP, JSON or browser?"}
    Plan -->|HTTP or embedded JSON| DOM["pagesift-html (yolowhtml-dom): single-pass arena DOM"]
    Plan -->|Browser escalation| CDP["pagesift-cdp: Lightpanda / moli / Chrome"]
    CDP --> DOM

    Net -->|PDF| PDF["pagesift-pdf (yolowpdf + yolow64)"]
    Net -->|Office / e-book / email / archive / text| Office["pagesift-office (yolowdoc)"]
    Net -->|Image / scanned PDF| OCR["pagesift-ocr, pagesift-infer, pagesift-gpu"]
    Office -->|member / attachment| PDF
    Office -->|source code| Syntax["pagesift-syntax (yolowsyntax)"]
    PDF -->|scanned page| OCR

    DOM --> Frontier["pagesift-frontier: URL scoring"]
    Frontier -.->|next crawl / sync candidates| Engine
    DOM --> Content["pagesift-content (yolowhtml) & pagesift-extract<br/>main content, selectors, evidence, chunks"]

    Content --> Graph["pagesift-graph (DataGraph)<br/>citations, provenance, claims, conflicts, clusters, asOf"]
    PDF --> Graph
    Office --> Graph

    Graph --> Ctx["Context (pagesift-graph + pagesift-extract + pagesift-stat)<br/>KnowledgeUnit (BoundedUot), ContextDelta, Search, ContextPack, VOI Action & Learn"]
    Graph --> Out["pagesift-sink & pagesift-store<br/>graph.jsonl, offloaded snapshots, SyncStore, ContextStore (ns::CONTEXT + chis), Experience"]
    Ctx <--> Out
    Ctx -.->|VOI + 4D shadow-price bounded acquisition| Engine
    Out -.->|priors, URL rewrites, checkpoints| Plan

Continuous Sync closed loop (Observe → Learn → Decide → Execute → Verify)

Long-lived sources registered via pagesift sync or /v2/sources run an adaptive closed loop per tick without discarding prior verified evidence on transient uncertainty, and optionally reconcile changed documents incrementally into ContextStore:

flowchart LR
    Obs["Observe & Schedule<br/>pagesift-store + pagesift-stat<br/>Non-aliased periodicity, Gamma-Poisson<br/>rates, Lagrange water-filling, Hub pull-forward"]
    Dec["Decide Modality<br/>pagesift-plan<br/>304/HTTP/Browser cost + tab shadow price,<br/>bounded recovery probes, tenant transfer"]
    Exe["Execute & Screen<br/>pagesift-engine/sync/capture<br/>HTTP / JSON synthesis / CDP / 304;<br/>censor decode:, truncation & bad 304s"]
    Ver["Verify & Commit<br/>pagesift-graph + pagesift-store<br/>Active/Superseded/Withdrawn/Unverified claims,<br/>ContextDelta & unit lineage, snapshot offload"]

    Obs --> Dec --> Exe --> Ver --> Obs

Evidence-checked /v2/agent and multi-server fleet topology

flowchart LR
    subgraph Agent["Bounded /v2/agent research loop"]
        Q[Question + Schema] --> M[User LLM step]
        M --> T[search / map / scrape / extract<br/>enforced robots & SSRF policy]
        T --> Ev[Archived evidence]
        Ev --> Check{Complete?}
        Check -->|budget left| M
        Check -->|done| Verify[Schema match + field grounding + warnings]
    end
flowchart LR
    subgraph Fleet["Single binary multi-node cluster (pagesift-cluster)"]
        C[Clients] --> G[Gateway<br/>API, jobs, webhooks]
        G -->|rendezvous host hash| W1[Worker]
        G --> W2[Worker]
        G -.->|fenced leases & checkpoints| S[Standby]
    end

Workspace crate map

Layer

Crates

Entry surfaces & FFI

pagesift (CLI & library), pagesift-api (HTTP server & schemas), pagesift-mcp (MCP stdio/HTTP), pagesift-py, pagesift-node, pagesift-ffi (C ABI)

Orchestration, strategy & graph

pagesift-engine, pagesift-plan, pagesift-hooks, pagesift-llm, pagesift-search, pagesift-graph

Network, policy & browser

pagesift-net, pagesift-policy, pagesift-rustls, pagesift-cdp, pagesift-media, pagesift-itna

Web extraction & discovery

pagesift-html (yolowhtml-dom), pagesift-content (yolowhtml), pagesift-extract, pagesift-diff, pagesift-frontier

Native document & OCR readers

pagesift-pdf (yolowpdf), pagesift-office (yolowdoc), pagesift-syntax (yolowsyntax), pagesift-ocr, pagesift-infer, pagesift-gpu

Runtime, math & persistence

pagesift-flow (NUMA workers & cgroup admission), pagesift-stat, pagesift-base64 (yolow64), pagesift-compress (yolowzstd), pagesift-sys, pagesift-store, pagesift-sink, pagesift-cluster

See Architecture reference for cgroup sizing, statistical defaults (δ = 0.05 USL/survival/Thompson tuning), incremental checkpoint semantics, and byte-range verified media archives.

When to use it, and when not

Good fit

  • Feeding live web pages, PDFs, Office suites, archives, or source code into AI agents, RAG pipelines, or downstream analytics with zero external runtime workers

  • Compiling token-budgeted, provenance-cited ContextPack prompts that preserve table headers, qualifiers, units, temporal validity, and opposing multi-source conflicts

  • Self-hosting Firecrawl- or Jina Reader-compatible APIs and shared MCP tooling without per-page SaaS fees

  • Continuous monitoring/sync of changing sources where cost-aware browser escalation, lineage tracking, and claim-level conflict detection matter

Pick something else when

  • You want a managed SaaS cloud. pagesift runs self-hosted on your infrastructure.

  • You need automatic CAPTCHA solving. Interactive challenges are classified and paused for human resolution ([itna] recovery is opt-in and does not bypass CAPTCHAs).

  • You only process standalone image URLs without local uploads or OCR. Uploaded/local images require --ocr ([ocr] enabled = true); image URLs and OCR-off uploads return UNSUPPORTED_OPTION.

License

Apache-2.0. For dedicated support, SLAs, or enterprise agreements, see commercial license and support or write to licensing@pagesift.dev.

What's new in 1.6.0

PageSift 1.6.0 introduces Context (docs/context.md) across CLI (pagesift context), HTTP (/v2/context), MCP (pagesift_context), Python (Scraper.context / acontext), Node.js (Scraper.context), and C ABI (pagesift_context), alongside upgrading upstream yolowhtml / yolowhtml-dom to 0.1.2 (<caption> table headings, hidden="until-found" article retention, and input_truncated capture reliability screening): adds cross-version KnowledgeUnit identity and explicit ambiguity/split/merge lineage (boundedUot, greedyDice, exactAnchorOnly); persists atomic dependency-governed ContextDelta updates, IndexVisibility lag metrics, and idempotent anti-ABA consumer checkpoints in pagesift-store (ns::CONTEXT, immune to cache eviction); provides multi-strategy lexical/symbol/temporal (asOf + effectiveAt)/lineage Search with co-retrieved conflict counterparts; compiles token- and MustKeep-constrained ContextPack payloads (submodularClosure, boundedDpp, mmrGreedy, topKGreedy, edgeSalience) with provenanceManifest; and adds bounded Action (context, research, verify, watch, changes) driven by gated Good–Turing / Chapman coverage diagnosis, Howard Value of Information, 4D admission shadow prices, and CUSUM / BernoulliBocpd drift-guarded isotonic Learn calibration.

Release notes · 1.5.6 · 1.5.5 · 1.5.4 · 1.5.3 · 1.5.2 · 1.5.1 · all releases · CHANGELOG


Community

Questions or feedback: Discord · Issues · Discussions

Citation

@software{pagesift2026,
  title  = {PageSift: Rust Web and Document Extraction Engine for AI Pipelines},
  author = {YOLOW92},
  year   = {2026},
  url    = {https://github.com/YOLOW92/pagesift},
  note   = {Version 1.6.0}
}

Star History

Star History Chart

Available Tools

8 tools
pagesift_batch_scrapeBatch scrapeA

Scrape a list of known URLs as one job with the same options. Use pagesift_scrape for one URL and pagesift_crawl when the URLs are not known yet. Invalid URLs are returned in invalidURLs instead of failing the job; a page that fails is recorded in the job's errors while the other pages still return. All requests share the server's concurrency and per-host rate limits. urls accepts up to 10000 entries and options applies to every URL (no per-URL overrides). Returns a job id; poll it with pagesift_poll_job.

Inputs: urls (required), options. Returns: id, next, status, invalidURLs

pagesift_cancel_jobCancel a jobA
Idempotent

Cancel a running pagesift_crawl or pagesift_batch_scrape job. Use pagesift_poll_job to read a job and pagesift_cancel_job to stop it. No new pages are scheduled; pages already collected stay available through pagesift_poll_job, and work that finishes after the cancel is discarded. A completed, failed or already cancelled job is left unchanged and its current status is returned, so repeated calls are safe. An unknown or expired id is an error.

Inputs: id (required). Returns: total, status, success, completed

pagesift_crawlCrawl a siteA

Start a crawl job that follows links from a URL and scrapes every page found. Use pagesift_scrape when you need one page and pagesift_map when you only need URLs. By default the crawl includes sitemap URLs, follows links within the seed's path scope, has no depth limit and uses per-host concurrency; maxDiscoveryDepth and maxConcurrency lower these. Pages blocked by robots.txt or failing are reported in the job's errors and the crawl continues. Returns a job id right away; poll it with pagesift_poll_job. Results can be large, so set a conservative limit and use includePaths/excludePaths to stay on topic.

Inputs: url (required), delay, limit, query, +16 more. Returns: id, next, status, invalidURLs

pagesift_interactAct on a kept pageA
Destructive

Continue on a browser page kept by pagesift_scrape with keepSession. Send either prompt (plain-language instruction; needs an LLM on the server) or actions (a list of browser actions), not both. Actions can permanently change the page or lose unsaved state. State carries over between calls. Returns the page afterwards in data; with a prompt also output (the answer) and steps. Finish with pagesift_interact_stop to free the page. No live view or CDP URL is offered.

Inputs: scrapeId (required), prompt, actions, timeout. Returns: data, error, steps, killed, +8 more

pagesift_interact_stopClose a kept pageA
Destructive

Close a browser page kept by pagesift_scrape keepSession or pagesift_interact and free its memory. Data already returned stays with the caller; the scrapeId becomes invalid, so later calls with it fail. Calling stop twice or after expiry returns an error. Pages expire automatically after 10 minutes if never stopped, and are reclaimed sooner when idle under memory pressure.

Inputs: scrapeId (required). Returns: success, sessionDurationMs

pagesift_mapMap a siteA
Read-onlyIdempotent

List a website's URLs from its sitemaps, robots.txt and homepage links without fetching each page. sitemap include (default) uses both sitemaps and page links, skip ignores sitemaps, only uses sitemaps without reading the homepage. If no sitemap is found it falls back to homepage links; limit truncates after search ranking. Use pagesift_scrape for one page and pagesift_crawl to fetch many pages.

Inputs: url (required), limit, search, sitemap, +3 more. Returns: links

pagesift_poll_jobPoll a jobA
Read-onlyIdempotent

Get the status, progress and pages collected so far for a pagesift_crawl or pagesift_batch_scrape job. status is scraping, completed, cancelled or failed; completed, cancelled and failed are final. While scraping, poll again after a few seconds; stop when the status is final. Pages collected so far are in data and remain available for 24 hours. Use pagesift_cancel_job to stop a running job.

Inputs: id (required). Returns: data, total, errors, status, +3 more

pagesift_scrapeScrape a pageA
Read-onlyIdempotent

Fetch one URL and return clean content: markdown by default, or html, rawHtml, links, images and metadata. Use it when you know the page you need. For several known URLs use pagesift_batch_scrape; to find URLs on a site use pagesift_map; to collect a whole site use pagesift_crawl. pagesift respects robots.txt and reports blocked pages as errors instead of returning their content. Results are cached for 48 hours by default; set maxAge to 0 to force a live fetch. With keepSession the browser page stays open for pagesift_interact and its id is returned as metadata.scrapeId; kept pages expire after 10 minutes (sooner when idle under memory pressure). Advanced options: actions runs browser steps before capture, chunking splits Markdown, queryFilter keeps blocks relevant to a query, and parsers selects PDF pages.

Inputs: url (required), proxy, egress, engine, +26 more. Returns: html, json, links, pages, +12 more

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A4.6/5.0

    Scored across 8 tools

    Disambiguation5/5

    Single-page scrape, batch scrape, crawl, and map each target distinct URL-discovery or fetching modes, while poll/cancel and interact/stop form clear operational pairs. The only overlap is that scrape can run pre-capture actions while interact continues an existing kept page, but the descriptions clearly separate 'capture now' from 'continue session'.

    Naming Consistency4/5

    All tools use the pagesift_ prefix and snake_case, mostly following a verb or verb_noun pattern such as scrape, crawl, poll_job, and cancel_job. pagesift_interact_stop breaks the verb-first ordering, and batch_scrape uses a modifier, but the set remains predictable and readable.

    Tool Count5/5

    Eight tools is well-scoped for a web scraping and crawling server: single scrape, batch scrape, map, crawl, two job controls, and two session controls. Each tool has a clear role, with no obviously redundant or missing lifecycle entry.

    Completeness5/5

    The surface covers URL discovery with map, single and batch fetching, whole-site crawling, asynchronous job status and cancellation, and browser session continuation and cleanup. For the stated domain, there are no obvious dead ends: job results remain retrievable via poll, and kept pages are managed through interact_stop or expiry.

    Maintenance

    ActivityNo data
    ResponsivenessOutdated data

    Related MCP Connectors

    Related MCP Servers

    • F
      license
      Not graded
      quality
      Not graded
      maintenance
      An MCP server for web content extraction that converts HTML pages into clean, LLM-optimized Markdown using Mozilla's Readability. It supports batch processing, intelligent multi-page crawling, and configurable caching while respecting robots.txt standards.
      48 npm
      -
    • F
      license
      Not graded
      quality
      D
      maintenance
      A locally-hosted MCP server that provides AI assistants with advanced web crawling capabilities, including structured data extraction, deep site crawling, and page screenshots. It enables users to convert single or multiple URLs into clean Markdown content for processing by LLMs without requiring external API keys for basic features.
      -
    • A
      license
      Not graded
      quality
      D
      maintenance
      Open-source web scraper and extraction MCP server with JavaScript rendering, markdown output, PDF/DOCX parsing, structured errors, and validated extraction contract diagnostics for agents.
      2
      AGPL 3.0
    • A
      license
      Not graded
      quality
      D
      maintenance
      An open-source web retrieval MCP server that fetches, crawls, and searches the web, returning clean markdown for AI agents. It integrates with Claude MCP, LangChain, and other frameworks for agentic web access.
      2
      MIT