AI-Native Website Agent Layer

An infrastructure layer between websites and AI agents.

Point it at a URL. It crawls the site, extracts a typed knowledge graph, publishes machine-readable context files, generates callable agent actions with a permission model, exposes the whole thing over REST and MCP, scores how ready the site is for AI systems, and keeps watching for changes.

The goal is that an agent answering a question about a business reads one small JSON file instead of crawling and re-parsing HTML — and that when it does, the answer is correct, sourced, and admits what the site does not say.

Website URL
     │
     ▼
  Crawler ──── robots.txt · sitemap · priority frontier · optional JS rendering
     │
     ▼
  Extraction ── schema.org > labelled facts > prose heuristics > (optional) LLM
     │
     ▼
  Knowledge graph ── stable entity keys, typed relations
     │
     ▼
  Retrieval index ── hashed TF-IDF vectors + FTS5/BM25, fused with RRF
     │
     ▼
  Agent actions ── generated from real capabilities, JSON Schema, permission tiers
     │
     ▼
  AI context ── /ai/*.json · llms.txt · OpenAPI 3.1 · MCP resources
     │
     ▼
  Continuous updates ── change detection → context regeneration

Contents — How to use · What gets produced · Design decisions · Real-world runs · Configuration · Known limits

How to use

1. Install and start

cd ai-agent-layer
npm install
npm run demo

npm run demo does three things: serves a built-in fixture website, runs the full pipeline against it, and starts the platform on http://localhost:8787. It prints a dashboard API key — copy it.

For a real deployment use npm start instead (no fixture), and run the migration once to mint your first admin key:

node src/db/migrate.js     # prints an admin key, once
npm start

It runs with zero API keys. Extraction is rule-based by default. Before anything public, set API_KEY_PEPPER in .env to a random secret — rotating it later invalidates every key you have issued.

Related MCP server: Scout MCP Server

2. Analyse a website

From the dashboard: open http://localhost:8787, paste your API key top-right and click Connect, then enter a URL and click Analyse. The row updates live while the pipeline runs.

From the CLI:

node bin/cli.js analyse https://example.com

From the API:

curl -X POST http://localhost:8787/api/v1/sites \
  -H "authorization: Bearer $KEY" \
  -H "content-type: application/json" \
  -d '{"url":"https://example.com","name":"Example"}'

It returns immediately with "pipeline":"started". Poll /api/v1/sites/{slug} until status is ready.

Tuning the crawl

Real sites vary enormously. Pass a config object when creating the site (or PATCH it later):

{
  "maxPages": 45,       // page budget — spent on the highest-value pages first
  "maxDepth": 3,        // link depth from the entry point
  "concurrency": 2,     // lower for slow or fragile origins
  "delayMs": 700,       // politeness gap between requests
  "timeoutMs": 45000,   // raise for slow hosts
  "render": "never",    // auto | never | always
  "respectRobots": true,
  "recrawlIntervalMs": 21600000
}

JavaScript rendering needs Playwright, which is optional:

npm install playwright && npx playwright install chromium

Without it, render: "auto" falls back to static HTML and the readiness score flags any page that needed JS as a finding.

3. Read the results

node bin/cli.js score example-com --full   # score + ranked fixes
node bin/cli.js show example-com           # knowledge graph tree
node bin/cli.js ask example-com "what is the refund policy?"
node bin/cli.js search example-com "warranty length"

If the site doesn't state an answer you get told that, not a guess. The dashboard's AI readiness, Knowledge graph and Playground tabs do the same things interactively.

4. Point an AI agent at it

This is the actual product. Three ways in.

A. Context files (simplest)

Any agent that can fetch a URL:

https://your-host/ai/{site}/context.json

Start there — it is a compact index, usually enough on its own. It links to products.json, policies.json, faqs.json and the rest when more detail is needed. llms.txt is the same content as prose. All served with strong ETags, so a repeat fetch costs a 304.

B. MCP (for Claude and other agent runtimes)

Claude Desktop — add to claude_desktop_config.json:

{
  "mcpServers": {
    "example": {
      "command": "node",
      "args": ["C:/absolute/path/to/ai-agent-layer/src/mcp/stdio.js", "example-com"]
    }
  }
}

Restart Claude Desktop. The site's actions appear as tools, its context files as resources.

Claude Code:

claude mcp add example -- node /absolute/path/to/ai-agent-layer/src/mcp/stdio.js example-com

Over HTTP (any client supporting the streamable transport): POST https://your-host/mcp/{site}

The stdio server defaults to the restricted tier because it runs on your own machine. To expose a read-only server to something less trusted, set AGENTLAYER_MCP_TIER=public.

C. REST actions

curl https://your-host/ai/example-com/actions.json          # discover

curl -X POST https://your-host/ai/example-com/actions/search_products \
  -H "content-type: application/json" \
  -d '{"query":"router","maxPrice":1000}'                   # call

openapi.json describes the same set for anything that speaks OpenAPI.

5. Permissions

Every action has a tier. Read actions are public; anything that creates a record needs a key.

Issue a key for an agent:

curl -X POST http://localhost:8787/api/v1/keys \
  -H "authorization: Bearer $ADMIN_KEY" \
  -H "content-type: application/json" \
  -d '{"name":"partner-agent","tier":"authenticated","scopes":["read"],"rateLimit":600}'

The plaintext key is returned once. Also supported: "siteId" to restrict a key to one site, "expiresAt", and "rateLimit" for a per-minute ceiling.

Tighten or disable an individual action:

curl -X PATCH http://localhost:8787/api/v1/sites/example-com/actions/request_quote \
  -H "authorization: Bearer $ADMIN_KEY" -H "content-type: application/json" \
  -d '{"permission":"restricted"}'      # or {"enabled": false}

The Agent actions dashboard tab has a toggle and dropdown per action.

6. Collect what agents submit

Write actions record a submission:

curl -H "authorization: Bearer $KEY" \
  http://localhost:8787/api/v1/sites/example-com/submissions

To forward to your CRM instead, set a webhook:

curl -X PATCH http://localhost:8787/api/v1/sites/example-com \
  -H "authorization: Bearer $KEY" -H "content-type: application/json" \
  -d '{"config":{"webhooks":{"default":"https://your-crm.example/hook"}}}'

Per-action webhooks work too ({"webhooks":{"request_quote":"https://…"}}). Submissions are stored locally even when forwarding succeeds.

7. Keep it current

The scheduler recrawls every site on MONITOR_INTERVAL_MS (default 6 h) and regenerates context when something changed.

node bin/cli.js sweep --force        # force a sweep now
node bin/cli.js changes example-com  # see what moved
curl http://localhost:8787/ai/example-com/changes   # public, for agents

Critical changes — price, availability, policy, product withdrawal — invalidate any cached AI answer that quoted the old value. If you cache agent answers, poll this endpoint and evict on severity: "critical".

Re-derive without re-crawling (after a config change or an extractor upgrade):

curl -X POST http://localhost:8787/api/v1/sites/example-com/pipeline \
  -H "authorization: Bearer $KEY" -H "content-type: application/json" \
  -d '{"rebuildOnly":true}'

8. See how AI systems use it

node bin/cli.js analytics example-com

Or the AI traffic tab. The number to act on is unanswered questions — queries where an agent asked and your site had no answer. Each is a content gap with a known audience.

9. Add a model (optional)

Everything above works with no API key. Adding one improves extraction on sites where facts are written in prose rather than markup:

# .env
ANTHROPIC_API_KEY=sk-ant-...
ANTHROPIC_MODEL=claude-sonnet-5

Then rebuildOnly to re-extract without re-crawling. The model only fills fields the rules left empty — it never overwrites a fact that came from schema.org markup. answer_question also changes: with a model it synthesises a cited answer; without one it returns the best source passage verbatim and says so.

For semantic retrieval (matching "broken unit" to "RMA"), also set EMBEDDING_PROVIDER=voyage and EMBEDDING_API_KEY.

10. Multi-tenant

Each organisation is isolated: its own sites, knowledge, keys, analytics and audit log.

curl -X POST http://localhost:8787/api/v1/orgs \
  -H "authorization: Bearer $ADMIN_KEY" -H "content-type: application/json" \
  -d '{"name":"Acme Corp","slug":"acme"}'

Mint that org an admin key and hand it over. A cross-tenant lookup returns 404, not 403, so one tenant cannot even confirm another's sites exist.

Troubleshooting

status: error after analysing — read lastError on GET /api/v1/sites/{slug}. Usually a timeout (raise timeoutMs) or robots.txt blocking the crawler.

Score is low but the site looks fine — open the AI readiness tab; every failing check names what it measured and what to change. A common one on modern sites is "Content is present without running JavaScript".

Few entities extracted — check the page-type breakdown at GET /api/v1/sites/{slug}/pages. If most pages are other, the classifier could not read them; see Non-English sites below.

Search returns nothing for an obvious query — expected when the local lexical embeddings have no vocabulary overlap. Reported honestly as a miss rather than as bad results. A hosted embedding model fixes it.

Non-English sites — the crawler, indexer, actions, context generation and MCP layer are language-neutral, and retrieval works on any script. The classifier and fact extractors are English-lexicon-based, so expect good structure and thin facts. The sitemap-group signal recovers page types in any language; recovering facts needs a configured model. See Real-world runs.

Port already in use — set PORT in .env.

What gets produced

For every site, at /ai/{site}/:

Plus live endpoints:

GET  /ai/{site}/search?q=…          hybrid retrieval
GET  /ai/{site}/entities            knowledge graph entities
GET  /ai/{site}/graph               full graph
GET  /ai/{site}/changes             change feed
POST /ai/{site}/actions/{action}    invoke an action
     /mcp/{site}                    MCP (streamable HTTP)
GET  /.well-known/ai-agent-layer    discovery across all sites

Design decisions worth knowing

Crawl budget: coverage beats depth

The page budget is almost always far smaller than the site, so which pages get spent matters more than how many. Two mechanisms:

Priority frontier — /pricing and /about outrank page 7 of the blog.

Diversity penalty — each fetch of a given kind makes the next one of that kind less attractive. This is not a refinement; without it the crawler is useless on large sites. WooCommerce's sitemap lists thousands of /document/… pages, all legitimate depth-1 entries, and a pure priority queue spent the whole 40-page budget on documentation without ever reaching /pricing.

Pages compete inside a bucket: the sitemap group the publisher declared, or the first path segment. One large section can no longer crowd out every other one.

Sitemap groups as a language-neutral signal

Where a URL appeared in the sitemap is publisher-declared structure, and it survives when the URL and title are in a script the classifier cannot read. WordPress splits by post type (wp-sitemap-posts-post-1.xml), Yoast by post-sitemap.xml — both are mapped to page types.

This is what lets a Persian news article with a percent-encoded slug be classified correctly when every English lexical rule finds nothing. It is weighted like a strong URL rule, so it never overrides schema.org or an unmistakable path like /pricing.

Extraction

Sources are merged in descending confidence, and the merge direction is one-way:

schema.org markup  >  labelled facts ("Employees: 82")  >  prose regex  >  LLM

A model never overwrites a fact that came from structured data, and only fills fields the rules left empty. A hallucinating model therefore degrades quality rather than corrupting verifiable facts. Every extraction prompt is framed as "report what this text says" with an explicit instruction that an omitted field is correct and a guessed one is a failure.

The heuristic path is the floor, not a stub — the platform produces a complete knowledge base with no model configured at all.

Explicit negatives

The extractor specifically hunts for what a business does not do ("we do not sell to consumers", "not covered by the warranty"). These are unusually valuable: they let an agent answer no with a citation instead of hedging or guessing yes. They appear in company.json as doesNotOffer.

Numeric limits

Policies are reduced to structured limits — {kind: 'window', value: 30, unit: 'days'} — not just prose. An agent can compare against a number; it cannot reliably compare against a paragraph.

Retrieval

Entity cards are the primary indexed unit: a compact rendering of everything known about one thing. A single retrieved chunk is usually a complete answer rather than a fragment. Page passages are indexed too but carry a lower prior.

Dense (hashed TF-IDF, dependency-free) and sparse (FTS5/BM25) rankings are fused with Reciprocal Rank Fusion rather than score averaging — the two scores are on incomparable scales, and RRF stays sane when one retriever returns nothing.

A miss is reported as a miss. If not one content term of a query appears in the site's corpus, search returns zero results with reason: no_vocabulary_overlap rather than confidently ranking unrelated chunks. Those misses are recorded and surface in analytics as unanswered questions — the highest-value content gaps.

Swapping in a hosted embedding model is a config change (EMBEDDING_PROVIDER=voyage|openai).

Grounded answers

answer_question returns answered: false when the site does not state the answer. With no model configured it returns the top source passage verbatim and labels itself extractive: true — inventing prose from a lexical match is exactly the failure this platform exists to prevent.

Actions

Actions are derived from capabilities the site demonstrably has. A site with no cart gets no create_order; a site with no booking language gets no book_appointment. Offering an action that cannot succeed is worse than offering none, because an agent will call it and report the failure to a user as a fact about the business.

Permission tiers follow blast radius:

Write actions always return a disclaimer that they record a request — never a confirmed order or booking.

All enforcement lives in one place (src/actions/execute.js), so REST, MCP, the CLI and the dashboard cannot diverge.

Change detection

Two layers. Page-level hashing is cheap but noisy. Entity-level diffing against the previous knowledge graph is what produces signal:

• price change → critical

• availability change → critical

• policy update → critical

• new/removed product → notable

Entity identity is a stable slug key, so a product keeps its id, relations and history across recrawls — that is what makes a diff meaningful rather than a comparison of two unrelated snapshots.

Scoring

Four weighted pillars, each a set of independent checks:

Every check returns a 0–1 score plus the evidence it used and a concrete fix. Recommendations are ranked by the points a fix would actually recover. Nothing scores tone or "writing quality" — those cannot be measured reliably and would make the number drift on an unchanged site.

The score also runs a contradiction check (same product with two prices, warranty stated two ways). Contradictions are worse than omissions because they look authoritative.

MCP details

Setup is in How to use §4B. Beyond that:

• Every site is its own MCP server. Actions become tools, context files become resources, plus a changes and a graph resource and an entity/{type}/{key} resource template.

• HTTP transport is stateless — the site scope is in the path, so any node serves any request and there is no session table. Accept: text/event-stream switches the response to SSE framing.

• The initialize response carries grounding instructions telling the model to answer only from tool results and to say so when the site is silent.

• A denied or failed tool call comes back as a result with isError: true, not a JSON-RPC error — that is what lets the model see and react to it.

CLI

agentlayer analyse <url> [--max-pages N] [--render auto|never|always] [--no-llm]
agentlayer list                        every site with its score
agentlayer show <site>                 summary + knowledge graph tree
agentlayer score <site> [--full]       readiness score and what to fix
agentlayer search <site> <query>       hybrid retrieval
agentlayer ask <site> <question>       grounded answer with citations
agentlayer actions <site>              generated actions
agentlayer call <site> <action> [json] invoke one
agentlayer context <site> [file]       print a context file
agentlayer changes <site>              detected changes
agentlayer analytics <site>            AI traffic report
agentlayer keys --create NAME --tier T mint an API key
agentlayer sweep [--force]             run the recrawl sweep now

Multi-tenancy & security

• Every row of website-derived data is scoped by org_id and usually site_id.

• No query reaches the database without one of the two.

• API keys are stored as HMAC digests; the plaintext is returned exactly once.

• Keys carry a tier, scopes, an optional site restriction, an optional rate limit and an optional expiry. A key scoped to one site cannot reach another even within its own org.

• Cross-tenant lookups return 404, not 403, so the existence of another tenant's site is not leaked.

• Rate limits are per-key (per-IP when anonymous), with RateLimit-* headers.

• Every action invocation and mutation is written to an audit log with actor, IP and user agent. Personal data in action payloads is redacted before it is written there.

• Only an admin key can mint another admin key.

Configuration

All optional — see .env.example. The defaults run without any of it.

JavaScript rendering is optional: npm install playwright && npx playwright install chromium. Without it the crawler still works, it just cannot see content that only exists after hydration — and the readiness score will flag that as a finding.

Tests

npm test

117 tests across three files:

• test/unit.test.js — parsing, robots, classification, sitemap-group hints, frontier diversity, HTML extraction, contact normalisation, validation, embeddings, schema hoisting, contradiction detection

• test/pipeline.test.js — full pipeline against a fixture site, then extraction assertions, retrieval quality, permission enforcement, context generation, and change detection driven by actually editing the fixture and recrawling

• test/api.test.js — HTTP surface, tenancy isolation, rate limiting, audit redaction, MCP over both HTTP and stdio

The fixture (test/fixture-site/) is a realistic 18-page B2B site with JSON-LD, comparison tables, accordion FAQs, five policy pages and deliberate edge cases (a product named FW10 that a naive price parser reads as €10, €1,200 that a naive parser reads as €1.20, the same office stated two different ways).

Real-world runs

Two live WordPress sites, crawled with a 40–45 page budget, no model configured. Both surfaced design flaws that the fixture site could not.

kut.ac.ir — Persian university site (RTL, fa-IR)

The crawl, index, action and context layers were unaffected by the language. The classifier was not: 39 of 45 pages were Persian news articles on percent-encoded slugs, and every English lexical rule found nothing — all 39 landed in other.

The fix was not more vocabulary. WordPress had already declared the post type in wp-sitemap-posts-post-1.xml; the crawler was discarding that provenance. Carrying it through to classification moved all 39 to blog_post and left zero unclassified — a signal that costs nothing and works in any script.

What remains thin is facts, not structure: no description, founding date or locations, because those extractors read English prose. That is the documented boundary — configure a model and it reads the prose directly.

Two extraction bugs also came from this site:

• Facebook share links were being collected as the organisation's social profiles. Every article carried share buttons, so the site appeared to have dozens of profiles that were really links back to itself.

• mailto: addresses padded with spaces (name @ example.com, weak scraper defence) were published verbatim. They are now normalised — including Persian-digit local parts — and anything still malformed is dropped rather than presented as a contact.

woocommerce.com — English commerce site

Scored 72.8 (C) with structured data on 40/40 pages. But 35 of those 40 were /document/subscriptions/…. WooCommerce's sitemap lists thousands of documentation URLs; each is a legitimate depth-1 entry, so the priority queue spent the entire budget on docs and never reached /pricing or /about.

That is the diversity penalty described above. On a small fixture site the flaw is invisible; on any real site it makes the crawler useless.

Layout

src/
  crawler/     robots, sitemap, fetcher (+ optional Playwright), classification
  extract/     schema.org, company, products, policies, FAQ, prompts
  knowledge/   graph construction and reading
  search/      embeddings, hybrid retrieval
  context/     /ai/*.json generation, schema hoisting
  scoring/     readiness pillars, checks, recommendations
  actions/     generation, validation, execution
  mcp/         protocol core, HTTP transport, stdio transport
  api/         server, routes, auth, rate limiting, observability
  monitor/     change detection, scheduler
  pipeline/    orchestration
  llm/         provider router (Anthropic, DeepSeek, none)
public/        dashboard (no build step)
bin/cli.js     command line interface

Known limits

Stated plainly, because a platform about not overclaiming should not overclaim:

• The classifier and fact extractors are English-lexicon-based. Everything else — crawler, chunker, index, actions, context generation, MCP — is language-neutral, and retrieval works on any script. On a non-English site expect correct structure and thin extracted facts. The sitemap-group signal recovers page types in any language; recovering the facts needs a configured model. Measured on a Persian university site: see Real-world runs.

• Local embeddings are lexical. They will not match "broken unit" to "RMA". Set EMBEDDING_PROVIDER to a hosted model for semantic recall.

• Scoring is heuristic. It measures what is mechanically checkable. It does not predict whether a given assistant will actually cite you.

• estimatedTokensSaved is an estimate, labelled as one everywhere it appears.

• Availability is as-of-last-crawl, not live inventory. Every response that reports it says so.

• SQLite and in-memory rate limiting suit a single node. Multi-node needs Postgres and a shared limiter store; the interfaces are small.

• answer_question without a model is extractive, returning a source passage rather than a synthesised answer. That is deliberate.