Agent-Ready Web Profile

Ermitteln, wie eine Website tatsächlich von Agents genutzt werden kaun.

Moderne Websites können HTML, llms.txt, Datensätze, Retrival-Indexes, OpenAP, Agent Skills, MCP, A2A, OAuth-Ressourcenmetadaten, agents.json und weitere Discovery-Oberlüchen expnen. Ein Client should no longer sitespezifschen Code benötigen – or bypass, which that Manifest necessary should be – understand them.

ARWP will gibt es now in in in zwei komplementären Teilen:

1. ARWP Profile – ein experimenteile, by publischende service-Landschluss at /s/site-profile.json.

2. ARWP Resolver – eine Intropomitentengine, die ARWP plus previously upstream/community Discovery eing, Evidence and conflict works where, and for one coverenten selected.

Das Profil ist nützlich. Es ist not not required to use Resolver, and it'not designed to replace upstream.

Öffentliche Project-Site: https://dkharlanau.gitHub.io/agent-reacent-web-profile/

Profille Contract: experimental v0.1. Released Validator/Action: v01.. The current main toolchain version 0.2; npm publication remains discrete release gateway and must not be called complete before it succeeds.

The problem

A website can legitimately publish multiple of independent discovery surfaces in open:

                         WEBSITE
                            |
       +--------------------+---------------------+
       |          |         |        |            |
   ARWP profile  agents.*  API     A2A Card   Agent Skills
                          Catalog
       |                    |                     |
       +------------- MCP / OAuth / web ----------+
                            |
                       ARWP RESOLVER
                            |
                 evidence-backed service map
                            |
               +------------+-------------+
               |            |             |
             read         search         tools
             data       structured       agent

The resolver does not ask every ecosystem to converge on one. He answers:

What actually diese Website ber https, then came from, do the claims conflict, and which interface should the client for this task use?

Related MCP server: SBCF MCP Server

Resolve the website

node bin/arwp.mjs resolve https://example.com

Machine-readable output:

node bin/arwp.mjs resolve https://example.com --json

The resolver currently uses normalizes from:

• ordinary web discovery through the bounded scanner;

• valid ARWP profiles;

• /agents.txt and /agents.json as CC;

• RFC 2 /.well-known/api-catalog;

• RFC 3 Protected Resource Metadata;

• A2A /.well-known/agent-card.json;

• Agent Skills /.well-known/agent-skills/index.json;

• experiments of MCP-AI-Catalog/Server-Card-Discovery.

Experimental/community sources remain explicitly tagged. Static metadata is never automatically elevated to LA runtime.

See docs/RESOLVER.md.

Explain what was found

node bin/arwp.mjs explain https://example.com

Example:

Example Knowledge Site
Canonical: https://example.com/
Evidence: 5/8 discovery sources resolved; 1 conflict(s).

Content: 2
Retrieval: 1
APIs: 2
Tools: 1

Conflicts:
- MCP declarations differ between agents.txt and agents.json.

Recommended interfaces:
- read: https://example.com/llms.txt
- search: https://example.com/search.json
- structured: https://example.com/openapi.json
- tools: https://example.com/mcp

Plan an intention

node bin/arwp.mjs plan https://example.com --intent=search

Supported intentions:

• read

• search

• structure

• tools

• agent

The planner delivers the selected interface, theirs with evidence and fallbacks. He uses deterministic routing heuristics—no hidden quality score.

Source remains in indicator

Resolver output differs evidence/identity:

Authority is not authorization. Security and permission decisions are joined at the real protocol/untime.

ARWP Profile

Publishers the one explicit Servicenmap still to publish:

/ai/site-profile.json

Minimal profile:

{
  "$schema": "https://raw.githubusercontent.com/dkharlanau/agent-ready-web-profile/v0.1.0/schema/site-profile.schema.json",
  "profileVersion": "0.1",
  "id": "example-knowledge-site",
  "name": "Example Knowledge Site",
  "canonicalUrl": "https://example.com/",
  "description": "A reviewed public knowledge library.",
  "web": {
    "sitemap": "https://example.com/sitemap.xml",
    "llms": "https://example.com/llms.txt"
  }
}

Optial HTML announcement:

<link rel="describedby" type="application/json" href="/ai/site-profile.json">

That is ARWP contract, no established .well-known-ocal. See SPEC.md.

Adopt ARWP by existing website

node bin/arwp.mjs scan https://example.com
node bin/arwp.mjs init https://example.com
node bin/arwp.mjs validate ai/site-profile.json
node bin/arwp.mjs verify https://example.com/ai/site-profile.json
node bin/arwp.mjs health https://example.com

scan observes on public evidence. init creates conservative profile and does not invent unverified MCP, WebM, Skills or A2A potentials.

A reusable validation Action available:

- name: Validate Agent-Ready Web Profile
  uses: dkharlanau/agent-ready-web-profile@v0.1.0
  with:
    profile: ai/site-profile.json

templates/github-actions/propose-arwp-profile.yml creates an opt-in profile update PR workflow.

Geded hosted discovery service

The same server runtime only allows fixed, cost operations:

GET  /health
POST /scan
POST /resolve
POST /explain
POST /plan

It includes HTTPS-only-target rules, rejection private networks, redirects re-check, response/query limits, explicit browser Origin-Allow-Listing and common rate limiting. It'not an arbitrary URL proxy.

ARWP_SCANNER_ALLOWED_ORIGINS=https://dkharlanau.github.io \
npm run scanner:http

A container object is in scanner-service/. Public hosting remains an external deploy gate.

Resolver as MCP

Agents can use the resolver itself:

npm run resolver:mcp

Tools:

• resolve_site

• explain_site

• plan_site_interface

But the `gate-vant ARWP profile gateway remains separately available to read the declared resources of a profile:

ARWP_PROFILE=https://example.com/ai/site-profile.json npm run mcp:start

Remote Streamable HTTP Gate:

ARWP_PROFILE=https://example.com/ai/site-profile.json \
ARWP_HTTP_ALLOWED_HOSTS=mcp.example.com \
npm run mcp:http

Check docs/GATEWAY.md.

directory and federation

The initial five-site ARWP directory remains a reference/adoption register:

node bin/arwp.mjs directory
node bin/arwp.mjs directory --capability=retrieval
node bin/arwp.mjs federated-search "outside view"

Public directory JSON:

https://dkharlanau.github.io/agent-ready-web-profile/directory.json

The next federation step is resolver-supported: sites should eventually be portable even when they do not publish a ARWP profile but echo usable upstream discovery.

See docs/DIRECTORY.md.

Real reference suite

Five published knowledge site architectures currently publish ARWP profiles and are live verified:

• Dzmitryi Kharlanau — SAP Knowledge;

• Brali Practical Knowledge Library;

• Cognitive Biases Knowledge Library;

• CBT Cards;

• Metkagram.

They intentionally cover different combinations of data, retrieval, OpenAPI, Agent Skills, MCP and trust metadata. Owned references are implementation evidence not independent adoption evidence.

Benchmark before marketing claims

npm run benchmark:resolver

The current benchmark is internal deterministic regression coverage. Compares HTML-only, llms.txt, ARWP-only, agents-only, upst-native against **n**.

It does explicitly not demonstrate token savings, latency savings, ranking, adoption or answer quality.

The next milestone for evidence is a reproducible corpus of 20–50 external sites that measures:

• requests and bytes until a usable interface is identified;

• correct/missed interface selection;

• false-positive capabilities;

• conflicts detected;

• canonical identity/provenance preservation;

• fallbacks need.

Negative results are made public. See docs/BENCHMARK.md.

WhatARWP doesnot are replace

ARWP should Key to preserve upstream semantics, not that implement.

Do not befolder ARWP native:

• RFC 9727 / API Catalog;

• RFC 9728 / Protected Resource Metadata;

• A2A Agent Cards;

• MCP runtime discovery / Server Car;

• Agent Skills;

• crawler AI use preferences;

• payment/commerce-Protokolls.

Project rule:

UPSTREAM EXISTS
      ↓
resolve / validate / normalize it

UPSTREAM DOES NOT EXIST
      ↓
collect a real interoperability failure

ONLY THEN
      ↓
consider an ARWP-specific extension

Security boundaries

• public HTTPS goals only;

• private goals / link-local will be rejected;

• redirect targets are requested;

• requests and responses restricted;

• no URL credentials;

• metadata never grants authorization;

• URL accessibility never proves MCP / WebMCP / A2A runtime conformance;

• conflicts remain visible instead of being covered by score.

Development direction

The North Star is:

How many external websites can ARWP correctly resolve and route without site-specific code?

Current Iteration 2 priorities:

1. publication/installation/deployment of the 0.2.x resolver toolchain;

2. build external benchmark corpus;

3. obtain three independent adopters;

4. static MCP evidence with server/discover live behavior reconcile;

5. verify A2A card signatures when available;

6. build resolver snapshots and drift/conflict monitoring;

7. use evidence to decide whether any new ARWP profile contract version is needed at all.

See ROADMAP.md.

"Repository description"

• [SPEC.md]?` — experimental ARWP contract;

• scheale/site-profile.schema.json — profile JSON-Schema;

• [bin/arwp.mjs]bin/arwp.mjs) — CLI;

• lib/scanner.mjs — bounded website scanner;

• lib/resolver.mjs — multi-standard resolver and planner;

• lib/resolver-adapters.mjs — upstream/community adapters;

• lib/public-fetch.mjs — network-transmitted public-HTTP fetch primitives;

• resolver/server.mjs — MCP server for resolver;

• scanner-service/ — hosted scan/resolve service with boundaries;

• gateway/ — generic ARWP profile gateway;

• router/ — directory federation;

• registry/ — first public ARWP dictionary;

• benchmarks/ — resolver regression/evidence work;

• docs/RESOLVER.md — resolver model;

• docs/BENCHMARK.md — benchmark rules;

• ROADMAP.md — evidence-driven next iteration.

License

Apache License 2.0. See Dashboard.