SnoopScan

Turn any public page into clean markdown, structured JSON, or a schema you define — including the pages that block everything else. REST API, MCP server, and SDKs.

Why

• Gets the page. Six fetch tiers on the hosted API, from a plain HTTP request up to a stealth browser on a residential exit (see below for what this repository includes). The engine starts at the cheapest tier a domain has historically needed and climbs only when it has to.

• Knows when it failed. A 200 carrying a challenge page, a consent wall or a generated decoy is a failure, not a success. Four layers of block detection, including statistical comparison against the domain's own baseline.

• Extraction that is routed, not guessed. Page type is classified first, then articles take the heuristic path while forums, listings, products and tables take the structured path. Confidence is scored 0–1 and reported.

• Asks, rather than crawls, where it can. A Shopify store, a WordPress blog or a Substack is served from its own public JSON — one request instead of a crawl.

• You only pay for what worked. Failed requests carry no cost, and every response reports the tier that succeeded and what it consumed.

Related MCP server: CrawlForge MCP Server

This repository and the hosted API

This is SnoopScan's open core: the REST API, extraction, block detection, the crawl frontier, the MCP server, the first two fetch tiers (plain HTTP and browser-grade TLS), storage, and both SDKs. It runs on its own.

Some parts are not here: the proxy layer, the browser and stealth tiers, the anti-bot knowledge base, and the lead-gen pipeline. They run only on the hosted API at snoopscan.com. So a self-hosted copy fetches with the first two tiers, and a page that needs a real browser or a residential exit comes back as blocked (reported honestly, never as a false success). The hosted API climbs all six tiers, and its free plan includes 1,500 credits a month.

Quick start

curl -X POST https://api.snoopscan.com/v1/scrape \
  -H "Authorization: Bearer $SNOOPSCAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/", "formats": ["markdown", "links"]}'

from snoopscan import SnoopScan

snoop = SnoopScan(api_key="...")
print(snoop.scrape("https://example.com/").markdown)

pip install snoopscan

import { SnoopScan } from 'snoopscan';

const snoop = new SnoopScan({ apiKey: '...' });
console.log((await snoop.scrape('https://example.com/')).markdown);

npm install snoopscan

MCP

Every endpoint is also an MCP tool, so an agent calls them directly with no glue code.

claude mcp add --transport http snoopscan https://api.snoopscan.com/mcp \
  --header "Authorization: Bearer $SNOOPSCAN_API_KEY"

Apps that sign in rather than take a key (the Claude app, ChatGPT) connect to https://api.snoopscan.com/mcp-oauth and sign in with a SnoopScan account.

Any MCP client works the same way — Cursor, VS Code, Windsurf, Zed, Codex, or anything speaking streamable HTTP. Tools: scrape, fetchMore, crawl, crawlStatus, crawlPages, map, search, extract, checkChanges, listProducts, listPosts, domain, company, findContacts, hiring, people, findLeads and leadsStatus (the company, contact, people, hiring and lead tools need the hosted API; this repository answers them with a plain "not available on this deployment"). Every tool carries a title and read-only / destructive labels, and none of them submits, posts or buys anything.

Context discipline is the governing constraint. Every content tool carries a maxChars budget, truncation is always visible and returns a continuation token, and crawl tools never inline page bodies. Guardrails cap pages per session, concurrent crawls, crawl size and bandwidth, and every refusal explains the limit so an agent adapts instead of retrying blindly. executeJavascript is not exposed over MCP at any level.

Endpoints

POST /v1/scrape      one URL -> markdown, html, links, screenshot, json
                     (+ `actions`: click, type, scroll, then read the result)
POST /v1/crawl       a site, with a frontier, robots and politeness
POST /v1/map         every URL on a site, ordered by relevance
POST /v1/search      the web, with the results scraped
POST /v1/extract     your JSON schema — or a named template — filled from the page
GET  /v1/templates   the named templates (product, article, jobPosting, …) and their fields
POST /v1/batch       many URLs, one job
POST /v1/parse       a PDF, DOCX or XLSX into text
POST /v1/monitor     watch URLs on a schedule, webhook on change
POST /v1/products    a Shopify or WooCommerce catalogue, from its own API
POST /v1/posts       WordPress, Substack, Squarespace or Discourse posts
POST /v1/company     enrich one company from its own site
POST /v1/domain      what a domain runs, and who it belongs to
POST /v1/places      Maps listings, details and enrichment

Extract a schema:

curl -X POST https://api.snoopscan.com/v1/extract \
  -H "Authorization: Bearer $SNOOPSCAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
        "urls": ["https://example.com/product/1"],
        "schema": {"type": "object",
                   "properties": {"name":  {"type": "string"},
                                  "price": {"type": "string"}}}
      }'

Watch a page and get a webhook when it changes:

curl -X POST https://api.snoopscan.com/v1/monitor \
  -H "Authorization: Bearer $SNOOPSCAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://example.com/pricing"],
       "everyMinutes": 60,
       "webhookUrl": "https://your.app/hook"}'

Every page carries metadata.platform saying what built it, and every response reports the tier that succeeded, the tiers attempted, and the extraction path.

Development

uv venv --python 3.12 && uv pip install -e ".[dev]"
uv pip install -e sdk/python --python .venv/bin/python
cp .env.example .env && createdb scraping_engine && .venv/bin/alembic upgrade head
.venv/bin/python tools/create_key.py "local-dev" --rpm 600
.venv/bin/uvicorn engine.api.app:app --reload --port 8099

The JS/TS SDK lives in sdk/js and builds independently:

cd sdk/js && npm install && npm run build && npm test

.venv/bin/pytest engine/tests -q
.venv/bin/ruff check engine tools && .venv/bin/mypy --strict engine/core
.venv/bin/python tools/smoke.py          # live, not part of CI

Built to the specification set in docs/spec/. Specs 01 (API surface) and 02 (data model) are the contract; everything else implements behind them.

The licence gate

Every dependency must be MIT, Apache-2.0, BSD, ISC or MPL-2.0. The gate is blocking — a build pulling a forbidden licence fails rather than warns — and it runs weekly, because a transitive dependency can relicense underneath you.

.venv/bin/pip-licenses --format=json > licences.json \
  && .venv/bin/python tools/check_licences.py licences.json

Two consequences already visible here: psycopg2 (LGPL) is not used — Alembic migrates through asyncpg instead — and tld, pulled in transitively, carries a recorded allowlist entry naming which branch of its tri-licence we elect.

Licence

The engine is AGPL-3.0. Run a modified version as a network service and section 13 obliges you to offer its source to your users — the running instance does this itself at GET /v1/source rather than relying on a document a fork would forget to update.

Both SDKs are MIT, deliberately — the Python one and the JS/TS one. An AGPL client library would push the copyleft into every application importing it, which is the opposite of what a client is for.

Some modules are proprietary and are not covered by the AGPL: the proxy layer, the browser and stealth tiers, the anti-bot knowledge base, and the lead-gen pipeline. They are listed in LICENSE-PROPRIETARY, generated from engine/licensing.py, and the exclusion is enforced in CI — tools/check_split.py fails the build if any public module depends on one.

Three things that are not negotiable

1. Clean-room implementation. No code from any AGPL/GPL/SSPL project. The API deliberately mirrors the category's option naming — interface compatibility is legitimate — but the implementation is independent.

2. The licence gate is blocking. Not advisory.

3. No personal or identifying data in the repo. Configuration comes from environment variables only. Every fixture uses example.com or .invalid.