agenticschema
Recognizes __NEXT_DATA__ and discovers /_next/data/...json endpoints as machine twins, enabling structured data extraction from Next.js sites.
Discovers Shopify .json endpoints as machine twins, enabling structured product and store data extraction from Shopify-powered pages.
Discovers and queries WordPress REST API (wp-json) endpoints as same-origin machine twins, enabling structured JSON data extraction from WordPress sites.
This server does not currently have a deployable release on Glama. Follow the repository's setup instructions to install and configure it for your MCP client, including any required credentials and local dependencies.
AgenticSchema ·

Turn any webpage into typed JSON, callable MCP / WebMCP tools, or AI-native Epistemic Context — in 3 ms, using 2.7× to 28× fewer tokens than a text scraper and up to 1,449× fewer than raw HTML.
Human scrapers treat a webpage as a flat text document. An LLM doing Live Search experiences the web completely differently: it already has language in its pre-trained weights, suffers from Transformer self-attention dilution on rhetorical filler, is vulnerable to temporal drift on old articles, and needs active navigation toward hidden APIs and primary sources. AgenticSchema closes that gap for both sides of the web:
For AI Live Search, scrapers & agent pipelines (
@agenticschema/core,@agenticschema/server): Extract normalized entities, Dual-Plane Canonical Evidence Graphs (evidence) with zero-bloat Inline Epistemic Sigils ([✓S1],[~S2],[#S3],[§S4]), Multi-Facet Epistemic Sufficiency Verdicts (sufficiency:ANSWERABLE|PARTIAL_ESCALATE|ABSTAIN), Surprisal-Distilled Epistemic Atoms (atoms), Relativistic Temporal Cones (temporal), Shadow JSON APIs (machineTwins), Information-Foraging Frontier Hops (frontier), AdaptiveL0 -> L1 -> L2Wavefronts (forageUrls), and Copy-Discounted Bayesian Truth Fusion (fuseUrls).For website owners (
@agenticschema/browser): Drop one<script>tag onto your site to expose your existing markup as W3C WebMCP tools ondocument.modelContext. You write no new API and run no backend.
Website / HTML (442 KB mean)
│
JSON-LD · Microdata · RDFa · OpenGraph · <dl>/<table> · __NEXT_DATA__
▼
AgenticSchema (3 ms)
│
┌───────────────────┬───────────┴───────────┬───────────────────┐
▼ ▼ ▼ ▼
L0 Pulse Submodular b=120 L1 Epistemic MCP / WebMCP
(99 tokens) (110 tokens) (413 tokens) (1,013 tokens)
1,449× vs HTML 1,305× vs HTML 348× vs HTML 142× vs HTML
└───────────────────┴───────────┬───────────┴───────────────────┘
▼
AI Agent (or npx @agenticschema/server --live)Why AgenticSchema beats normal scraping: the numbers
Everything below is measured and reproducible from this repository (npm run test:corpus and npm run bench:report) across 177 real pages pulled from Common Crawl (shops, news, recipes, reference, books) and 120 agent trials.
1. Token & payload compression vs. HTML and text scrapers
Tokens per page, averaged across all 177 real pages (442.0 KB mean raw HTML), counted with o200k_base:
What the model reads | Mean tokens / page | Mean size / page | vs. Raw HTML | vs. Competent text scraper |
Raw HTML, as served | 143,770 | 442.0 KB | — | — |
Extracted text (Readability-style scraper: | 2,752 | ~11.2 KB | 52× | — |
AgenticSchema MCP tools ( | 1,013 | 3.0 KB | 142× | 2.7× |
AgenticSchema | 413 | 1.2 KB | 348× | 6.7× |
AgenticSchema | 110 | 0.4 KB | 1,305× | 25× |
AgenticSchema | 99 | 276 B | 1,449× | 28× |
Nobody serious feeds raw HTML to a model: a competent scraper strips the markup first, accounting for 52× of the reduction. Against a competent text scraper, AgenticSchema cuts token consumption by another 2.7× (full MCP tools), 6.7× (L1 Epistemic Core), 25× (Multi-Facet Submodular Query Bottleneck under a strict 120-token ceiling), and 28× (L0 Holographic Pulse) — while returning machine-typed fields and verified epistemic claims instead of unstructured paragraphs:
Vertical | Pages | Raw HTML | Text scraper | MCP |
| Submodular ( |
|
book (Goodreads) | 2 | 287,454 | 11,534 | 191 (61×) | 460 (25×) | 100 (115×) | 95 (121×) |
reference (Wikipedia, MDN) | 50 | 62,730 | 3,699 | 208 (18×) | 550 (6.7×) | 108 (34×) | 121 (31×) |
ecommerce (IKEA, Patagonia, Backcountry) | 75 | 240,151 | 2,912 | 1,567 (1.9×) | 412 (7.1×) | 112 (26×) | 99 (29×) |
news (Al Jazeera) | 25 | 90,530 | 882 | 528 (1.7×) | 259 (3.4×) | 110 (8.0×) | 67 (13×) |
recipe (Marmiton) | 25 | 58,454 | 1,549 | 1,514 (1.0×) | 292 (5.3×) | 109 (14×) | 89 (17×) |
All corpus pages (mean) | 177 | 143,770 | 2,752 | 1,013 (2.7×) | 413 (6.7×) | 110 (25×) | 99 (28×) |
(Multipliers in parentheses are vs. a competent text scraper on the same vertical.)
2. AI-Native Live Search Signals extracted across the 177 real pages
Beyond schema normalization, toContext(), forageUrls(), and fuseUrls() extract seven AI-native layers that traditional text scrapers discard or miss completely:
AI-Native Layer | Measured on 177-page Corpus | What it gives the LLM during Live Search |
Dual-Plane Evidence Graph & Inline Sigils ( | 100% of structured & tabular facts | Typed |
Multi-Facet Sufficiency & Absence Taxonomy ( | 110 tokens mean (at | Enforces $\text{ANSWERABLE} \iff \forall f_i \in F_{\text{required}}, \text{Evidence}(f_i) \ge \tau_i$, and diagnoses every facet as |
Active Inference EVI Controller ( | $< 0.1\text{ ms}$ decision | Computes residual query entropy $H(Q \mid S)$ and Expected Value of Information ($\text{EVI}$) to choose |
Surprisal-Distilled Epistemic Atoms ( | 679 verified atoms (3.8 / page) | Strips zero-surprise rhetorical filler and requires a non-parametric anchor (quantity/unit, constraint/negation, or temporal date) subordinated to query relevance. |
Relativistic Temporal Light-Cone ( | 125 / 177 pages (71%) | Anchors |
Shadow APIs & Machine Twins ( | 102 endpoints discovered | Declaratively discovers same-origin JSON/API twins ( |
Information-Foraging Frontier & Adaptive Wavefront ( | 361 high-scent hops | Scans all $N$ pages at |
3. Answer accuracy & hallucination rate (sonnet, 120 trials)
Fewer tokens only matter if the agent answers at least as well. Over 120 isolated trials (5 questions per page: single-fact, multi-part, and unanswerable questions designed to catch hallucinations), comparing an agent given the scraped page text against the same agent given only AgenticSchema MCP tools:
Benchmark metric | Competent text scraper ( | AgenticSchema ( | Delta |
Overall agreement (mappable pages, neutral referee) | 85% | 89% | +4 pt |
E-commerce questions (including | 100% | 100% | 11/11 prices |
Recipe questions ( | 60% | 80% | +20 pt |
Two-part questions (e.g. "What is the rating, and what does it cost?") | 83% | 92% | +9 pt |
Hallucinations on unanswerable questions ( | 10% (1/10) | 0% (0/10) | Zero inventions |
Mean cost per trial (API-equivalent USD) | $0.076 | $0.050 | −34% cost |
Why the tools arm wins on complex and recipe questions: prose scatters facts across the page (or omits recipeYield from body text while declaring it in JSON-LD), causing a reader model to drop one half of a two-part question or lose a product name in boilerplate. Two targeted tool calls return exact fields with zero prose noise.
4. Runtime speed & web-scale coverage
Metric | Value | Why it matters |
Core pipeline & Zero-DOM | 3 ms / page (0 ms DOM parse) | Server/Edge |
Full DOM construction ( | ~38–41 ms / page | Only paid on pages that require a full DOM tree walk (excluding network fetch time). |
Streaming Early-Abort ( | Up to 90% less bandwidth | Aborts the HTTP stream right after |
Domains publishing | 10,000,000+ | Over 10M domains already declare machine-readable search endpoints (Google Crawl stats, 2026-07). |
Browser CDN bundle ( | 30.6 KB gzip | Single self-contained classic IIFE script, W3C WebMCP polyfill included. |
Related MCP server: neuron-inspector
Contents
Quick start: 3 ways to use it
1. Scrape any URL to structured JSON or MCP tools from the terminal
No browser and no configuration. Given a real product page:
world.openfoodfacts.org/product/3017620422003
read get_web_site
read get_organization
action search_web_site(search_term_string)search_web_site is an executable tool derived automatically from the page's SearchAction. Run it as an MCP server for Claude Desktop / Cursor / Claude Code / Antigravity, or dump the structured JSON / Epistemic Context directly for Live Search:
# Start a persistent Live Search MCP server (exposes fetch_evidence, forage_urls, fuse_sources for ANY URL):
npx @agenticschema/server --live
# Or start a page-bound MCP server over stdio for one or more URLs (fetched concurrently):
npx @agenticschema/server https://world.openfoodfacts.org/product/3017620422003
# Or print a 1-turn Hybrid LLM Context (Structured Facts + Specs + Inline Sigils + Deduplicated Prose):
npx @agenticschema/server --context --early-abort https://en.wikipedia.org/wiki/Backpack
# Multi-Facet Submodular Bottleneck: project a page down to a strict 120-token ceiling (~110 tok mean):
npx @agenticschema/server --context --query "waterproof warranty weight" --budget 120 https://en.wikipedia.org/wiki/Backpack
# Holographic Multi-Resolution: get a ~99-token L0 Semantic DNA Pulse or ~413-token L1 Epistemic Core:
npx @agenticschema/server --context --resolution L0 https://en.wikipedia.org/wiki/Backpack
# Adaptive Foveated Wavefront: explore N search results (L0 -> L1 -> automatic L2 escalation on missing facets):
npx @agenticschema/server --forage --query "price weight battery" https://example.com/p1 https://example.com/p2 https://example.com/p3
# Copy-Discounted Knowledge Graph Fusion: fuse N search results via SimHash independence + Bayesian Consensus:
npx @agenticschema/server --fuse --query "price weight battery" https://example.com/p1 https://example.com/p2
# Or scrape any page directly to structured JSON (with OpenGraph/meta/form fallback):
npx @agenticschema/server --json --fallback https://en.wikipedia.org/wiki/BackpackDrop-in Agent Skill included: Point Claude Code, Antigravity, Cursor, or Codex at
skills/agentic-live-search/SKILL.md(also mirrored in.agent/skills/agentic-live-search/SKILL.md) so your AI coding/research agent automatically usesagenticschema(--liveMCP or CLI--context/--forage/--fuse) instead of rawWebFetch.
2. AI-Native Live Search (toContext, forageUrls & fuseUrls) & Edge Middleware
Human scrapers treat a web page as a static text document. An LLM doing Live Search (WebFetch, OAI-SearchBot, Perplexity, RAG) experiences the web completely differently: it already has language in its pre-trained weights, suffers from Transformer attention dilution on rhetorical filler, is vulnerable to temporal drift on old articles, and needs active navigation toward hidden APIs and primary sources.
toContext(), forageUrls(), and fuseUrls() implement 7 AI-native paradigms:
Dual-Plane Canonical Evidence Graph (
evidence) & Zero-Bloat Inline Epistemic Sigils:Exposes a full typed
EvidenceNode[]graph in code with a 4D Epistemic Confidence Vector $C = (C_{\text{extraction}}, C_{\text{agreement}}, C_{\text{source}}, C_{\text{freshness}})$ while embedding 2-token Inline Epistemic Sigils in the LLM prompt:[✓✓S0]: Cross-origin consensus corroborated across $\ge 2$ independent SimHash clusters infuseUrls()($P(v \mid \mathcal{D}) \ge 0.80$).[✓S1]: Intra-page cross-representation consistency — structured Schema.org/Microdata/RDFa datum verified in visible DOM text ($C_{\text{agreement}} = 0.92$, ruling out SEO cloaking or template drift, without claiming 1.0 ontological truth from a single origin).[~S2]: Unverified structured markup not echoed in visible prose ($C_{\text{agreement}} = 0.78$).[#S3]: Tabular<dl>/<table>specification or SSR hydration state (__NEXT_DATA__).[§S4]: High-surprisal quantitative/constraint/temporal prose atom.[!CONFLICT]: Explicit contradiction between markup and visible text, or across sources.
Multi-Facet Sufficiency Calculus (
sufficiency) & Counterfactual Absence Taxonomy:Decomposes multi-part queries into orthogonal Query Facets (
facets), maximizes marginal epistemic utility $U(f, q) = R(f, q) \cdot C(f) \cdot T(f) \cdot N(f \mid S)$, and enforces the strict rule: $$\text{ANSWERABLE} \iff \forall f_i \in F_{\text{required}},\quad \text{Evidence}(f_i) \ge \tau_i \land \neg\text{Conflicted}(f_i)$$Diagnoses every facet in
sufficiency.facetDiagnosticsasSATISFIED,NOT_PUBLISHED(structured entity exists but omits the property),NOT_FOUND(absent everywhere),CONFLICTED, orUNVERIFIABLE(expired/weak $< \tau_i$), preventing the LLM from confusing absence with non-existence.
Active Inference & Expected Value of Information Controller (
controller):Computes residual query uncertainty $H(Q \mid S)$ and Expected Value of Information $\text{EVI}(a) = \frac{\mathbb{E}[\Delta I(a)] \times \text{Impact}(a)}{\text{Cost}{\text{tokens}}(a) + \text{Cost}{\text{latency}}(a)}$ to output the optimal next step:
STOP_AND_ANSWER,ESCALATE_L2,QUERY_MACHINE_TWIN,FOLLOW_FRONTIER,REPORT_CONFLICT, orABSTAIN.
Adaptive Holographic Resolution (
L0 -> L1 -> L2) & Foveated Wavefront (foragePages/forageUrls):L0Pulse (~99 tokens/page — 1,449× smaller than HTML, 28× smaller than text): A 1-line "Semantic DNA strand" ([L0 Product] Name | [✓S1]price=... | t=2025-11-10 | atoms: [§S4] ... | twin=...).L1Epistemic Core (~413 tokens/page — 6.7× smaller than text): Schema + Specs + Surprisal-Distilled Epistemic Atoms + Temporal Cone + Machine Twins + Frontier (zero narrative prose).forageUrls()Adaptive Escalation: Scans all $N$ URLs atL0, promotes the highest-resonance pages toL1, checkssufficiency.missingFacets, and automatically escalates fromL1toL2Deep Inspection only when a query facet remains unresolved inL1.
Surprisal-Driven Epistemic Distillation (
atoms) & Relativistic Temporal Light-Cone (temporal): Strips zero-surprise rhetorical scaffolding, requires a non-parametric anchor (quantity, constraint, or date) subordinated to query relevance, flags expired offers (⚠️EXPIRED_OFFER), and anchors indexical time words ("currently", "this year").Declarative Shadow API & Machine-Twin Autodiscovery (
machineTwins): Detects same-origin JSON/API twins (wp-json,/llms.txt,/_next/data/...json, Shopify.json, feeds) while blocking private/loopback SSRF addresses, credentials, and cross-origin hosts.Claim-Level Copy-Discounted Log-Odds Bayesian Fusion (
fusePages/fuseUrls): Evaluates both page-level and claim-neighborhood SimHash similarity (so a comparator copying a manufacturer's price is discounted onpricewhile retaining full independence on its own lab-measuredweight), and computes calibrated Log-Odds Bayesian Posteriors $P(v \mid \mathcal{D}) \in (0, 0.99]$.
import { contextPages, forageUrls, fuseUrls, createLiveSearchServer, createAgentMiddleware } from '@agenticschema/server';
// A. Adaptive Foveated Multi-Page Exploration (L0 -> L1 -> automatic L2 escalation on missing facets):
const wavefront = await forageUrls(
['https://lab.example/solid-state', 'https://blog.example/tips', 'https://news.example/trends'],
{ query: 'fast charge cycles at -30°C', foveaCount: 1 }
);
console.log(wavefront.context); // Unified Foveated Wavefront with [✓S1]/[§S4] sigils
console.log(wavefront.sufficiency); // { verdict, facetDiagnostics: [{ facet, status, maxConfidence }] }
console.log(wavefront.controller); // { action: 'STOP_AND_ANSWER' | 'FOLLOW_FRONTIER', evi, target }
// B. Single-page Live Search with Multi-Facet Submodular Bottleneck or Holographic Resolution:
const [page] = await contextPages(['https://en.wikipedia.org/wiki/Backpack'], {
query: 'internal frame load capacity',
resolution: 'L1', // 'L0' (~99 tok pulse) | 'L1' (~413 tok epistemic) | 'L2' (full hybrid)
});
console.log(page.evidence); // Typed EvidenceNode[] with 4D confidence (extraction, agreement, source, freshness)
console.log(page.controller); // Active Inference EVI decision plan
// C. Claim-Level Copy-Discounted Log-Odds Bayesian Fusion:
const fused = await fuseUrls(
['https://store-a.example/camera', 'https://review-b.example/camera'],
{ query: 'price sensor weight', tokenBudget: 400 }
);
// D. Edge Middleware (Cloudflare Workers / Next.js / Hono):
const agentMiddleware = createAgentMiddleware({ mcpEndpoint: '/mcp' });3. Expose your own website to in-browser AI agents (W3C WebMCP)
Two tags in development, one in production. The first reads your page's markup and registers WebMCP tools on document.modelContext; the second bridges them to local desktop MCP clients during development:
<!-- ① registration: read this page's Schema.org markup, publish it as WebMCP tools -->
<script src="https://cdn.jsdelivr.net/npm/@agenticschema/browser@latest"></script>
<!-- ② transport (development only): bridge those tools to a local MCP relay -->
<script src="https://cdn.jsdelivr.net/npm/@mcp-b/webmcp-local-relay@4/dist/browser/embed.js"></script>Point your MCP client (Claude Desktop, Cursor, Claude Code) at the relay:
{
"mcpServers": {
"webmcp-local-relay": {
"command": "npx",
"args": ["-y", "@mcp-b/webmcp-local-relay@latest"]
}
}
}Open the page, and the tools appear in your client. Verify with webmcp_list_sources (or inspect Chrome DevTools → Application → WebMCP in Chrome 146+).
Four things worth knowing before you paste that in:
Order matters. Put the relay embed tag after the registration tag.
Leave off
type="module". The script-tag build is a classic IIFE script so it works out of the box in Google Tag Manager and Cloudflare Zaraz and preservesdocument.currentScript. See How the adapter finds its own tag.Tag ② is for development only. Shipping it to real visitors makes their browsers probe
127.0.0.1. See Keep the relay out of production.Pin your versions in production.
Try it in the browser with nothing installed:
Interactive Playground → — Paste any JSON-LD or try the hostile sample to see what the security guard refuses and why.
Live WebMCP Demo Page → — A live page carrying the script tag end to end.
Read this first: registration is not transport
This is the single thing that trips people up on the browser path, so it comes before the browser reference.
Getting a page's data to an external desktop agent takes two steps, and @agenticschema/browser only does the first one:
① REGISTRATION ② TRANSPORT
@agenticschema/browser a WebMCP-capable browser,
reads the page's Schema.org an extension, or a local relay
markup and registers tools carries those tools to the agent
on document.modelContext
│ │
└──────────► document.modelContext ◄─────┘
(the meeting point)@agenticschema/browser writes tools into document.modelContext. It does not open a connection to anything, because a browser tab cannot listen on a port. See Three constraints.
If you only want the browser's own built-in AI agent (Chrome WebMCP) to call your tools, step ① is all you need. If you want an external desktop program (Claude Desktop, Cursor) to call them, you also need a transport (step ②). See Choosing a transport.
The symptom of forgetting step ② is very specific: the tools show up in Chrome DevTools (Application → WebMCP) but your external MCP client reports zero sources. DevTools reads document.modelContext in-process; your external MCP client is a separate program that cannot.
The script tag, in full
Where the tag goes
The build is a single classic script (dist/cdn/auto.js, ~30.6 KB gzipped) with no bundler and no package install behind it:
Platform | Where |
WordPress | The theme's header template, or any plugin that inserts scripts into |
Shopify |
|
React, Next.js, Astro | The shared layout or document component, alongside your other third-party tags. |
Plain HTML | In |
Google Tag Manager, Cloudflare Zaraz | A Custom HTML tag. See Through a tag manager. |
Put it in the shared layout rather than on one page. In a single-page app the adapter follows history.pushState, replaceState, popstate, and DOM mutations automatically, so a single tag covers every route.
How the adapter finds its own tag
To read its data-* options the adapter tries three things, in order:
document.currentScript, set while a classic script runs (including one injected by a tag manager), andnullin a module script,script[data-agenticschema], an explicit marker,script[src*="agenticschema"], thesrcof the standard snippet.
Since the build is an IIFE, the plain snippet takes rule 1 and everything works with no marker:
<script data-max-tools="8"
src="https://cdn.jsdelivr.net/npm/@agenticschema/browser@latest"></script>Adding type="module" gives up rule 1. If you self-host under a custom filename and use type="module", add data-agenticschema:
<script type="module" data-agenticschema data-max-tools="8"
src="/assets/webmcp-bundle.js"></script>Through a tag manager
Google Tag Manager and Cloudflare Zaraz inject a plain <script src> without type="module". Because the CDN bundle is an IIFE, it runs cleanly in any tag manager and reads data-* attributes via document.currentScript.
Fire it on every page, as early as possible.
Keep the local relay embed (
embed.js) out of tag managers.
Every attribute
All options are optional:
Attribute | Values | Default | What it does |
| present / absent | absent | Marks the tag so the adapter can find it when |
|
| actions generated | Turns off executable tools entirely. Read tools are unaffected. |
| integer > 0 |
| Ceiling on generated tools. |
|
| watching on | Stops the adapter following DOM changes and History API navigations. |
| comma-separated hosts | page origin only | Extra hosts an action's destination may point at, beyond the page's own origin. Read Security first. |
| present / | off | Synthesizes Schema.org entities from OpenGraph, Twitter Cards, |
| comma-separated origins | none | Cross-origin parent origins allowed to discover and invoke tools when embedded in an iframe (forwarded to |
A page that uses all of them:
<script data-agenticschema
data-actions="off"
data-max-tools="8"
data-watch="off"
data-fallback
data-allow-hosts="api.example.com, search.example.com"
data-exposed-to="https://partner.example.com"
src="https://cdn.jsdelivr.net/npm/@agenticschema/browser@latest"></script>Content-Security-Policy
A page with a CSP has to allow cdn.jsdelivr.net in script-src, or self-host dist/cdn/auto.js:
Content-Security-Policy: script-src 'self' https://cdn.jsdelivr.net;The development relay embed (embed.js) also creates a blob: iframe and opens a local WebSocket:
Content-Security-Policy: script-src 'self' https://cdn.jsdelivr.net;
frame-src blob:;
connect-src ws://127.0.0.1:9333;Choosing a transport
Transport | Who calls the tools | Setup | Good for |
Native browser WebMCP | the browser's own agent | none: the browser provides | the end state (Chrome 146+ flag / Chrome 149+ origin trial) |
Chrome origin trial | the browser's own agent, on visitors' Chrome | register a token for your origin and serve it | running the native path on real traffic today |
Local relay ( | desktop MCP clients: Claude Desktop, Cursor, Claude Code | one script tag + | development, testing, personal automation |
Browser extension | whatever the extension is wired to | install the extension | using tools across sites you do not control |
Node MCP server ( | any MCP client over stdio or HTTP |
| server-side agents & scraping without a browser |
Keep the relay out of production
Gate embed.js on your build's development flag so real visitors never probe 127.0.0.1:9333:
// Next.js / React
{process.env.NODE_ENV === 'development' && (
<script src="https://cdn.jsdelivr.net/npm/@mcp-b/webmcp-local-relay@4/dist/browser/embed.js" />
)}---
// Astro
---
{import.meta.env.DEV && (
<script src="https://cdn.jsdelivr.net/npm/@mcp-b/webmcp-local-relay@4/dist/browser/embed.js"></script>
)}The JavaScript API & Universal Scraper
Universal scraping with scrape() and extractMeta() (@agenticschema/core)
@agenticschema/core has zero runtime dependencies and runs in Node, Bun, Deno, Cloudflare Workers, or the browser:
import { scrape, extractMeta } from '@agenticschema/core';
import { schemaOrgProfiles } from '@agenticschema/profiles';
const result = scrape(htmlOrDocument, {
...schemaOrgProfiles,
baseUrl: 'https://shop.example.com/p/backpack',
fallback: true, // enabled by default in scrape()
});
result.primary; // Primary entity as clean JSON (e.g. { type: 'Product', name: '...', ... })
result.entities; // All non-action entities materialized on the page
result.meta; // { title, description, canonical, language, openGraph, twitter }
result.actions; // [{ name: 'search_web_site', type: 'SearchAction', parameters: ['q'], required: ['q'] }]
result.tools; // Ready-to-call ToolDescriptor[] (with execute())
result.diagnostics; // Diagnostic[]Filtering and paginating list_* group tools
When a page publishes multiple secondary entities of the same type (e.g. 30 related products or reviews), AgenticSchema collapses them into a single list_<type> tool instead of flooding the agent with 30 tools. Every list_* tool supports built-in filtering and pagination:
const listProducts = result.tools.find((t) => t.name === 'list_product');
// Filter case-insensitively by keyword and paginate:
const page1 = await listProducts.execute({
query: 'waterproof',
limit: 5,
offset: 0,
});Browser programmatic API (@agenticschema/browser)
import { start } from '@agenticschema/browser';
const handle = await start({
maxTools: 12,
actions: 'auto',
fallback: true,
allowedHosts: ['api.example.com'],
exposedTo: ['https://partner.example.com'],
});
handle.tools(); // ToolDescriptor[]: what is currently registered
handle.diagnostics(); // Diagnostic[]: what the pipeline skipped, and why
await handle.refresh(); // remap now; a no-op if the markup has not changed
handle.stop(); // unregister everything and stop watchingstart() accepts every pipeline option plus five browser-specific options:
Option | Type | Default | What it does |
|
| the page's own | The document to read. |
|
|
| Follow route changes and markup edits (including container insertions and |
|
|
| How long to wait after a DOM change before remapping. |
|
|
| Cross-origin parent origins allowed to call tools when embedded in an iframe. |
|
|
| The WebMCP surface to register on. Injectable for tests. |
In compliance with the W3C WebMCP specification, @agenticschema/browser passes strictly opt-in annotations (readOnlyHint: true, untrustedContentHint: true, consequentialHint: true when applicable, never : false) to document.modelContext.registerTool().
The core pipeline
import { toTools } from '@agenticschema/core';
const { tools, diagnostics, graph } = toTools(documentOrHtmlString, options); ┌──────────────── @agenticschema/core ─────────────────┐
Document │ HTML │ │
│ JSON-LD ───────► │ extract ──► normalize ──► select ──► map ──► guard │ ──► ToolDescriptor[]
└──────────────────────────────────────────────────────┘
│
┌───────────────────┴───────────────────┐
▼ ▼
@agenticschema/browser @agenticschema/server
document.modelContext stdio / fetch handler / CLI JSON
(script tag, WebMCP) (works with any MCP client today)Stage | Does |
extract | Pulls out JSON-LD, Microdata ($O(N)$), and RDFa ($O(N)$), plus optional OpenGraph / Twitter Card / |
normalize | Flattens |
select | Identifies the primary subject of the page ( |
map | Applies type profiles (case-insensitively, following dotted paths like |
guard | Validates tool names, strips HTML/control characters, caps payloads with a UTF-8 fast path, and wraps |
Every pipeline option
Category | Option | Type | Default | What it does |
Extraction |
|
| all three | Which structured data syntaxes to read. |
Extraction |
|
|
| Synthesize entities from OpenGraph, Twitter Cards, |
Normalisation |
|
| page URL in browser | Base for resolving relative |
Normalisation |
|
|
| Maximum nesting depth before emitting |
Mapping |
|
| generic profile only | Type profile registry ( |
Mapping |
|
| none | Schema.org 924-class hierarchy resolver from |
Mapping |
|
|
| Ceiling on generated tools ( |
Actions |
|
|
|
|
Actions |
|
| derived from | Origin that action URLs are vetted against. |
Actions |
|
|
| Extra hosts allowed beyond |
Actions |
|
|
| Per-request timeout on action |
Actions |
|
| global | Custom |
Guard |
|
|
| Maximum length of a tool description. |
Guard |
|
|
| Maximum UTF-8 byte length of a tool result ( |
Custom |
|
|
| Hand-declared tools ( |
The Node server & CLI scraper
@agenticschema/server fetches pages concurrently (with scheme, credential, and SSRF validation), parses HTML with linkedom when needed (skipping DOM construction entirely on JSON-LD-only pages), and speaks MCP 2026-07-28 over stdio or HTTP — or outputs Epistemic Context / scraped JSON directly.
npx @agenticschema/server [<url>...] [options]Flag | Default | What it does |
| off | Start a persistent Live Search MCP server over stdio exposing |
| off | Generate a 1-turn Hybrid LLM Context ( |
| off | Adaptive Foveated Multi-Page Wavefront ( |
| off | Multi-Page Knowledge Graph Fusion ( |
|
| Holographic resolution level ( |
| none | Focus extraction via the Multi-Facet Submodular Query Bottleneck and compute |
| none | Hard BPE-calibrated token ceiling ( |
| off | Scrape all given URLs concurrently and print structured JSON ( |
| off | Abort the HTTP response stream right after |
| off | Synthesize Schema.org entities and search actions from OpenGraph, Twitter Cards, |
|
| Cap on generated tools per page. |
| actions on | Do not build executable tools from |
| none | Extra host allowed for actions. Repeatable. |
| off | Serve MCP over HTTP instead of stdio. Binds |
|
| Port for |
| off | Keep diagnostics off stderr. |
| n/a | Print usage and exit. |
Over HTTP / Cloudflare Workers
import { createHttpHandler } from '@agenticschema/server';
const handler = await createHttpHandler(['https://example.test/product'], {
fallback: true,
});
export default { fetch: (request) => handler.fetch(request) };Results of tools/list, resources/list, and resources/read include the MCP 2026-07-28 ttlMs (300000 ms default) and cacheScope (public for tool listings, private for page resources) cache hints.
Deep dive: the 177-page corpus & 120-trial benchmark
How the corpus is built and measured
The 177 pages in the corpus are pulled from an immutable Common Crawl snapshot (CC-MAIN-2026-30) via byte-range WARC requests so reruns are 100% reproducible and never touch live origin servers:
npm run corpus:fetch # fetch from Common Crawl into fixtures/local
npm run corpus:report # inspect raw page types independent of AgenticSchema
npm install --no-save gpt-tokenizer # optional: enable o200k_base token columns
npm run test:corpus # run all 177 pages through the pipelineWhat improves token efficiency so sharply across the corpus:
Transitive entity deduplication (
markMaterialized): Whenget_recipeorget_productmaterializes child entities inline (Offer,AggregateRating,NutritionInformation,Person,HowToSection,ImageObject) up to depth 2, those child entities are marked consumed and never re-emitted as separate duplicate tools. This eliminated 149 redundant tools across the corpus (from 900 down to 751 tools) and brought recipes from3,787tokens (0.4×vs text) down to1,514tokens (1.0×vs text).Boilerplate & machinery suppression: Navigation
ItemLists,WPHeader/WPFooter,BreadcrumbList, and protocol nodes (EntryPoint,SearchAction) are suppressed when a primary subject exists, soSearchActionbecomes a callablesearch_*action tool and never a noisyget_search_actionread tool.Dotted-path resolution (
ReadSpec.from): Real-world e-commerce pages store variant prices inhasVariant.offers(46 of 75 shop pages in the corpus areProductGroups) and group ratings inisVariantOf.aggregateRating. Following multi-hop paths took e-commerce price accuracy from8/11to11/11(100%).
Full details are in docs/corpus.md and docs/bench.md.
Security
The library takes untrusted page content and puts it into an AI model's context. Six defences are enforced in core and inherited by every adapter:
Prompt injection isolation: Page text never enters a tool's
nameordescription— only its data payload.@typeis used for naming only when it matches/^[A-Za-z][A-Za-z0-9]{0,39}$/; hostile@typestrings containing instructions fall back toThing. All descriptions and materialized data strings are stripped of HTML tags and control characters in guaranteed $O(N)$ linear time.Untrusted content annotations: Browser tools carry
untrustedContentHint: trueondocument.modelContextso the host agent knows tool outputs originate from third-party page markup.SSRF & exfiltration prevention on
urlTemplate:Only idempotent action types (
SearchAction,FindAction,ReadAction,ViewAction) withGETare auto-exposed. Side-effecting actions (OrderAction,POST, etc.) are skipped with anaction-skippeddiagnostic.Destinations must be
http:/https:, carry no embedded credentials (user:pass@), and matchpageOrigin(orallowedHosts) both before and after RFC 6570 level-1 template expansion.Placeholders in the scheme, host, port, or protocol-relative authority (
//{host}/...) are rejected at map time.Action requests pass
redirect: 'error'so a3xxredirect cannot escape the origin check.
HTML action response cleaning & structured HTTP errors: When an action endpoint returns an HTML page instead of JSON,
<script>,<style>,<svg>,<noscript>, and markup tags are stripped before payload capping. HTTP4xx/5xxresponses returnisError: truewith structured recovery hints (code,retryable,suggestion).Prototype pollution immunity: JSON-LD, Microdata, and RDFa extractors and normalizers block
__proto__,constructor, andprototypekeys and checkObject.hasOwn()everywhere, preventing crashes or pollution from properties namedtoStringorvalueOf.Tool & payload budgets: Tool count is capped at
maxTools(default24) and output size atmaxPayloadBytes(default32,000bytes, cut cleanly on UTF-8 code-point boundaries).
Read SECURITY.md for the complete threat model.
Custom tools
For actions with side effects (POST, checkout, booking) or private API endpoints, declare custom tools via custom / defineTool():
import { start } from '@agenticschema/browser';
start({
custom: [{
name: 'check_stock',
description: 'Check in-store availability for a postal code.',
annotations: { readOnlyHint: true },
inputSchema: {
type: 'object',
properties: { postalCode: { type: 'string', description: '5-digit postal code' } },
required: ['postalCode'],
additionalProperties: false,
},
execute: async ({ postalCode }) => ({
content: [{ type: 'text', text: await (await fetch(`/api/stock?cap=${postalCode}`)).text() }],
}),
}],
});Custom tools pass through the same guard (name validation, description sanitization, payload cap, and async error boundary) and take precedence over generated tools on name clashes.
Troubleshooting
Symptom | Cause | Fix |
Tools visible in DevTools → Application → WebMCP, but external MCP client shows 0 sources | Registration done, transport missing | Add the relay embed tag in dev, or use |
Page has no JSON-LD / Microdata / RDFa and produces 0 tools | Site only publishes OpenGraph or | Pass |
A |
| Drop |
The script tag never runs | CSP blocks | Allow it in |
No action tool from a | Cross-origin target, | Check |
| Same-type secondary entities collapse into one filterable tool | Pass |
Packages
Package | Purpose |
Extraction, normalization, selection, mapping, guarding, and | |
20+ curated Schema.org type profiles + compact 924-class Schema.org inheritance hierarchy. | |
W3C WebMCP adapter ( | |
MCP 2026-07-28 server over stdio and HTTP ( |
Development
npm install
npm test # 286 tests across 16 test files in < 1s
npm run typecheck # strict TypeScript check across all workspaces
npm run build # build all 4 packages + CDN IIFE bundle
npm run size # enforces single-file IIFE < 30 KB (30,720 B) gzippedContributions welcome. See CONTRIBUTING.md and CODE_OF_CONDUCT.md.
Status and disclaimer
Early, pre-1.0, API not stable. WebMCP is a W3C specification in active browser rollout (available in Chrome via chrome://flags/#enable-web-model-context / #enable-webmcp-testing and Chrome 149+ origin trial; polyfilled automatically everywhere else).
Provided as is, with no warranty of any kind, express or implied. See LICENSE and SECURITY.md.
MIT.
This server cannot be deployed
Maintenance
Related MCP Connectors
Web scraping, browser automation, screenshots and AI data extraction for AI agents via MCP.
Turn any public website into an MCP server for agents to search, read and navigate.
Web scraping, extraction and search for AI agents. Hosted MCP with workspace API keys.
Unified 15-tool SEO, SERP, clean Markdown & GEO (LLM citation Share-of-Voice) MCP server.
Related MCP Servers
- AlicenseAqualityCmaintenanceMCP server that extracts structured JSON from public URLs for AI agents using schemas like product, article, and company.645 npmMIT
- AlicenseBqualityBmaintenanceTurns Chrome into a local MCP server, giving AI agents 51 browser tools to inspect DOM, automate clicks, run JavaScript, audit security and accessibility, mock APIs, extract data, record demos, and more from any MCP client.92615 npmMIT
- FlicenseNot gradedqualityAmaintenanceEnables language models to perform AI-powered web scraping, extraction, search, crawling, schema generation, and scheduled monitoring through remote or local MCP integration.-
- AlicenseNot gradedqualityAmaintenanceConnects MCP-compatible AI agents to an anti-detection browser server, exposing dozens of tools for navigation, page interaction, search, content extraction, downloads, multi-tab handling, and persistent sessions. It lets agents automate and scrape the web at scale using accessibility snapshots, CSS-selector fallbacks, saved profiles, and batch workflows over stdio or authenticated HTTP transport.971 npmMIT