Skip to main content
Glama

site-kg

ci License: MIT Python 3.11+

Give it a website URL or a local directory; it builds a knowledge graph from docs and code, and serves it to AI agents over MCP.

中文文档:README.zh-CN.md

What it does

URL ──► manifest probe ──► crawl ──► markdown corpus ──► graph.json + search.json ──► MCP server
        (objects.inv →          (httpx+bs4+html2text,   (link-derived edges,          (search / get_page /
         sitemap.xml → BFS)      robots.txt, rate-limited)  inverted index, verdict)     neighbors / site_stats)

local dir ──► walk (.gitignore-aware) ──► same corpus format ──► same graph + MCP
              (md/txt/wiki docs, code files;
               edges: md links, [[wikilinks]], imports, README hubs)
  • Manifest probe — Sphinx sites are enumerated from objects.inv (the authoritative page list), Doxygen sites from navtreeindex0.js (their index.html is a JS shell with no static links), then sitemap.xml, then same-site BFS. No guessing from local filenames.

  • Edges come from the corpus only. If a site has no cross-references the verdict is NOT-READY and the graph layer says so instead of drawing a decorative hairball. Never invents links.

  • No LLM required for the structural layer: search, page bodies and graph traversal all work keyless. The optional ask Q&A layer runs on any OpenAI-compatible gateway (SITE_KG_LLM_BASE + SITE_KG_LLM_KEY).

  • Wheels, not rewrites: MCP via the official mcp SDK; optional JS-heavy crawling via Crawl4AI; optional semantic layer via cognee. Only the glue (manifest → corpus → graph → MCP) is ours.

Related MCP server: read-website-fast

Quickstart

uv venv && source .venv/bin/activate
uv pip install -e ".[crawl]"

# ingest a site (bounded)
python -m site_kg.cli ingest https://example.org/docs/index.html --max-pages 200

# or ingest a local directory (docs wiki + code repo)
python -m site_kg.cli ingest-local /path/to/repo-or-wiki

# serve MCP over streamable HTTP (for network clients)
python -m site_kg.cli serve --transport http --port 8766

# or stdio, for agent hosts that launch subprocesses
python -m site_kg.cli serve

MCP client config (streamable HTTP):

{"mcpServers": {"site-kg": {"url": "http://127.0.0.1:8766/mcp"}}}

> The HTTP transport has no authentication: bind 127.0.0.1 (the default). To expose the server beyond loopback you must set `SITE_KG_ALLOW_REMOTE=1` for the server process, accepting the risk.

MCP tools

tool

purpose

ingest_url(url, max_pages, max_depth, relations)

crawl + build graph; returns site_id and verdict

ingest_local(path, max_files, respect_gitignore, relations)

local directory (docs + code) → graph; same outputs

list_sites()

ingested sites with verdicts

site_stats(site_id)

doc/edge counts, edge types, top hubs

search(site_id, query)

idf-weighted full-text search

get_page(site_id, page_id)

full markdown body + source URL

neighbors(site_id, page_id, depth)

subgraph expansion (multi-hop context)

ask(site_id, question)

LLM Q&A with citations (needs LLM config)

render_site(site_id)

self-contained 3D Three.js graph viewer (human inspection): degree-sized glowing nodes, hover tooltip, click-to-focus neighbourhood, chapters aggregate view (click a sphere to isolate), instant search with focus, live status bar (nodes/edges/view%/FPS)

render_build(site_id)

self-contained WebGPU build-replay viewer: the graph constructs itself live — nodes born in corpus order onto 3D chapter shells, then evidence edges (ref/imports/contains), then one flying wave per derived relation dimension; every edge shoots out of its source node and binds to the target with a pulse while a HUD counts nodes/edges/per-dimension edges. Speed controls 0.25×–16×, replay/pause/skip. Needs a WebGPU-capable browser

Resource: site://<site_id>/graph — the full graph JSON.

Local directories (docs + code)

ingest_local walks a directory into the same corpus format the crawler produces, so every downstream tool (search/get_page/neighbors/ask/render) works unchanged.

  • Docs (.md/.markdown/.mdx/.txt/.wiki/.rst/.adoc) keep their body. Edges come from relative markdown links (extension-less and root-absolute forms included) and [[wikilinks]] (Obsidian/MediaWiki style, case/underscore-insensitive basename match). Unresolvable references are counted in dropped_refs, never turned into edges.

  • Code (37 extensions: py, js/ts, go, rust, c/c++, java/kotlin, ruby, … plus the kernel/build ecosystem — yaml, asm .S/.s, perl, awk, linker scripts, makefile fragments, devicetree .dts/.dtsi) becomes searchable docs whose body starts with a prose header (path, language, detected top-level symbols) so identifiers hit the inverted index. Well-known extension-less build files (Makefile, Kconfig, Kbuild, COPYING, LICENSE, README, MAINTAINERS, …) are ingested by basename with the same treatment. Edges come from real dependency statements: Python import/from…import (including relative and parenthesized forms), JS/TS import/require, C/C++ quoted #include, Go import (go.mod-aware), Rust mod/use crate::, JVM import, Ruby require_relative.

  • README/index files are hubs: contains edges to their directory siblings, mirroring the site-side hub semantics.

  • .gitignore is honored (via pathspec) plus a builtin noise list (node_modules, __pycache__, lockfiles, *.min.js, …); >512 KB and binary files are skipped. Every skip is attributed, not silent: scan_stats counts skipped_noise_file / skipped_ext / skipped_oserror separately (their sum stays in skipped_kind), and NOISE_DIRS pruning reports pruned_noise_dirs plus a sample of the pruned paths — a coverage claim you cannot audit is a wrong claim.

  • Page bodies are sharded, not monolithic: data/sites/<site_id>/docs/<bucket>.json holds 1–256 sha-keyed buckets (count scaled to corpus size, recorded as docBuckets), so get_page reads one small bucket whatever the corpus shape (measured on a 75k-doc kernel tree: reading the single 975 MB docs.json took 8.3 s, a sharded cold page read takes ~12 ms; a drivers/ chapter with 31k files cannot recreate one giant shard). Sites built before sharding keep their docs.json and are still read transparently. Re-ingesting replaces the shards and drops the old single file in the same pass.

Env knobs (read per call, so a long-lived MCP server picks up changes without a restart):

var

effect

SITE_KG_MAX_BYTES

per-file size cap in bytes (default 512 KiB)

SITE_KG_NOISE_EXEMPT

comma-separated posix relpath prefixes whose dirs escape NOISE_DIRS pruning (e.g. drivers/target — real source trees collide with noise names)

SITE_KG_RELATIONS

derived relation dimensions, selected per node: auto (default) / llm / all / off / comma list — see Relation dimensions

SITE_KG_RELATION_ENTITY_PAGES

LLM entity-extraction budget per ingest (default 120)

SITE_KG_RELATION_EMBED_PAGES

embedding-similarity budget per ingest (default 300)

SITE_KG_RELATION_ROUTE_PAGES

LLM router budget per ingest (default 120)

  • PageRank runs over the edge list (aider repo-map style) and lands on each node as r, so site_stats/the viewer can rank structurally central files.

Design choices, after surveying the field (aider, Serena, CocoIndex, gitingest; GraphRAG, LightRAG, cognee, khoj; GraphCoder/CGM papers): the structural layer stays LLM-free and honest — every edge traces to an explicit link or import statement. LLM-extracted entity/relation edges (the GraphRAG/LightRAG approach) buy recall at the cost of hallucinated edges and build cost, so they stay in the optional cognee semantic layer. Compiler-grade precision (LSP/SCIP, Serena-style) and tree-sitter symbol graphs are roadmap items as optional extras, not hard dependencies.

Measured (2026-10-09, dogfooding this repo): 22 files (7 docs + 15 code) → 30 imports + 2 ref + 2 contains edges, verdict READY; identifier search (pagerank, gitignore) lands on the right source file.

Measured at corpus scale (2026-10-13, a 115,598-doc / 7-site DriveOS SDK source tree): re-ingest and report parity verified site-by-site; get_page on the largest site (75,415 docs) reads one ~3.8 MB shard in ~12 ms where the single-file format needed 8.3 s for the same page. Shard/meta caches are keyed by file mtime+size, so a re-ingest from any process (cli, crawl job, another agent) is served fresh with no invalidation hook; all writes are tmp-file + os.replace, and a 4-thread read hammer during repeated re-ingest saw zero torn reads.

Live viewer

site-kg live starts a local, address-first live session: give it the address first; the browser then asks which scene to build in.

# URL: prints a loopback URL; the launcher is prefilled with the address
python -m site_kg.cli live https://docs.example.com

# skip the launcher with a scene: crawl -> stream -> settle in one view
python -m site_kg.cli live https://docs.example.com --scene neural

# local docs/code directory
python -m site_kg.cli live --local ./docs --scene planets

# replay an ingested site from its stored graph
python -m site_kg.cli live my-site --replay

The page streams the build over SSE while the server crawls and lays the graph out incrementally: nodes arrive in corpus order, edges bind as pages are parsed, and derived relation dimensions stay hidden by default and never participate in layout springs, so they cannot distort the picture. Scenes: Planets (1) chapters as worlds in orbit, Neural (2) a dense field of synapses, City (3) one page per tower — floors follow the page's rank, and the lit windows are the links it emits. #scene=, the last-picked scene in localStorage, and the MCP scene= argument are presets once a scene has been chosen — they never replace the address-first start.

Limits and fallbacks:

  • The scale floor is the measured 75,415-node / 61,615-edge corpus; layout runs on the server (no browser-side force layout at that size). --max-nodes trims the preview only.

  • Labels are budgeted at most 40 chapter / 12 tick labels, and the scene is merged point/line batches — no per-node objects or DOM labels.

  • The live scene draws on WebGPU first (hand-written WGSL; the same layout, birth, drag and finale state feeds both backends, so the two cannot drift). A browser with no navigator.gpu or no usable adapter falls back to WebGL2 on the same canvas, and without WebGL2 to a canvas-2D view capped at 5,000 nodes that points at the offline archive for the full graph. WebGL context loss is restored in place.

  • The archive viewer (site-kg render <site_id>) is a single offline HTML file (binary payloads embedded by default; --external-data writes sk-*.bin beside it) and replays the same scenes.

Security: the server binds 127.0.0.1 only, sends no CORS headers, rejects non-loopback Host values, refuses path traversal, serves no corpus files, and offers an optional shared token (--token). With a token, the printed URL carries ?token=… and the page forwards it on snapshot, SSE and start calls. The Three.js r186 bundle is committed and pinned; normal development and offline use need no npm — see tools/build_viewer_bundle.mjs for the deliberate upgrade path.

Relation dimensions

Link and import edges are honest but sparse: most pages in most corpora are unconnected because nobody linked them. --relations (CLI), relations= (MCP) or SITE_KG_RELATIONS (env) adds derived dimensions, chosen per node from that node's own content — not one fixed extractor over everything:

kind

needs

meaning

mentions_symbol

code

this file mentions an identifier defined in another file

similar_to

prose ≥ 220 chars

sparse tf-idf cosine over prose + title tokens: same topic, no explicit link

sibling_of

1–12 dir mates

same source directory / URL folder cohort

supersedes

version in id/URL

2.0 supersedes 1.4 of the same page family

shares_heading

≥ 3 headings

document-skeleton overlap (Jaccard ≥ 0.35 over heading sets): same-generator API pages

shares_term

≥ 2 curated terms

shared writer-marked vocabulary: inline code spans, headings, declared symbols

duplicate_of

frontmatter sha256

identical content (exact-hash tier, chained)

coupled_with

evidence links

bibliographic coupling: both pages cite the same targets

co_cited_with

evidence links

co-citation: the same sources cite both pages (relates stubs)

shares_entity

chat model

pages share ≥ 2 LLM-extracted named entities

semantic_similar

embed model

embedding cosine over title + body head

A stub gets no lexical neighbours, a prose page gets no symbol edges, a release note gets version lineage, a source file gets symbol references. Modes: auto (keyless, default), llm (auto + an LLM router that may only narrow each node's set, plus the two model-backed dimensions), all, off, or an explicit comma list.

Two invariants keep this from becoming a hairball generator: a pair the corpus already links is never re-labelled with a derived kind, and everything is per-node gated and fan-out capped. Content-derived dimensions run even on a corpus with zero cross-references — content is the only signal such a corpus has — but the verdict still reports NOT-READY (no link structure), and a stub corpus still gates to zero edges. No LLM or embedding model is required; without one those dimensions are skipped with a note.

Design and the literature it rests on (AutoSchemaKG arXiv:2505.23628, KGGen arXiv:2502.09956, EDC arXiv:2404.03868, Adaptive-RAG arXiv:2403.14403): docs/RELATIONS.md.

Measured on a 2,000-file DriveOS SDK slice: 132 evidence edges → +142 mentions_symbol, +3,560 similar_to, +427 sibling_of, 4.1 s end-to-end.

Verified

Measured on the embedded vendor 7.0.3 docs site (static Sphinx): 40-page bounded ingest → 40 docs, 32 ref edges, 1709 indexed terms, verdict READY; all seven MCP tools exercised over real streamable-HTTP sessions (2026-10-06). The same origin site proves raw websites — not just wiki exports — carry enough link structure to graph.

ask verified against an on-prem vLLM gateway (qwen3.5-122b chat + bge-m3 embed + bge-reranker-v2-m3): an English question returns a cited step-by-step answer; a Chinese cross-lingual question still answers when keyword recall fails on the ASCII index, because embedding similarity over title + body head is unioned with keyword hits on every query.

Second-site check on the Doxygen API reference (drive-os-linux-sdk-api-ref, 300-page bounded ingest): the first attempt correctly reported NOT-READY (JS shell => BFS discovered 1 page); after the navtree manifest source landed, 301 docs / 527 ref edges, verdict READY, and ask returned a correct cited answer across header/source pages.

Site-type coverage (measured)

engine

manifest tier used

result

Sphinx (embedded vendor guide)

objects.inv

READY, 40 docs / 32 edges

Doxygen (vendor platform API reference)

navtreeindex0.js

READY, 301 docs / 527 edges

MkDocs Material (squidfunk)

sitemap.xml

READY, 60 docs / 2801 edges

Docusaurus (Checkly)

sitemap.xml index → child flattening

READY, 60 docs / 460 edges

VitePress (Vue guide)

sitemap.xml

READY, 50 docs / 2402 edges

MediaWiki (Arch Wiki)

same-site BFS

READY, 44 docs / 1201 edges

robots-disallowed (docs.astral.sh, docusaurus.io)

n/a

reported blocked by robots.txt, never a silent empty graph

Every pretty-URL shape is handled (trailing slash, extension-less, /title/X); a leaf seed that discovers <=2 pages returns a seed_hint telling you to seed the docs root.

ask uses block-level embedding selection, not head truncation: the same question on the same corpus went from "insufficient information" to a fully cited answer after that fix (the answer paragraph sat mid-page in a long FAQ).

Client-rendered SPAs: pages that fetch as empty framework shells are detected (empty #app/#root root, or near-zero visible text and no internal links) and re-rendered with headless Chromium (Playwright). Static fetch always runs first — only shell pages pay for a browser. Verified with a local CSR fixture: BFS discovers the JS-injected links, the graph reaches READY, and the JS-rendered body lands in the corpus.

FAQ

  • The ingest says blocked by robots.txt. The site disallows this crawler's user agent. Ask the owner, or pass respect_robots=false only if you are authorized to crawl it. For deep crawls of sites you own or are authorized to test, use --dev-crawl (CLI) / dev=true (MCP): robots.txt and politeness delays are skipped and the defaults deepen (--max-pages 2000 --max-depth 8, concurrency 8).

  • It found only 1-2 pages. Your seed URL is probably a section leaf; re-ingest with the docs root (the seed_hint field says so). If the site is a client-rendered SPA, the Playwright fallback handles shell pages automatically.

  • Do I need an LLM? No for crawl/search/graph/neighbors. Only ask needs SITE_KG_LLM_BASE + SITE_KG_LLM_KEY (any OpenAI-compatible gateway; vLLM/new-api verified).

  • NOT-READY verdict. The site has no internal cross-references — it is a directory tree, not a graph. You still get full-text search, page bodies (with extracted headings and description in the frontmatter), and the keyless content dimensions (similar_to, shares_heading, …) still connect pages that share substance; what you do not get is a link-structure claim.

Deployment

Local-first. A sudo-free systemd user unit:

# ~/.config/systemd/user/site-kg.service
[Service]
WorkingDirectory=%h/develop/site-kg
ExecStart=%h/develop/site-kg/.venv/bin/python -m site_kg.cli serve --transport http --port 8766
Restart=on-failure
[Install]
WantedBy=default.target

Roadmap

  • Semantic enrichment (optional): cognee cognify over the crawled corpus when SITE_KG_LLM_KEY is present — entity/relation edges alongside the structural ones.

  • Cross-file symbol edges done: mentions_symbol (regex layer, keyless). tree-sitter symbol/reference edges (aider-style tags) remain an optional extra for precision.

  • Crawl4AI deep-crawl for JS sites that defeat the Playwright shell-fallback (infinite scroll, heavy anti-bot), via the .[crawl] extra.

  • Static graph visualization done: render_site emits a self-contained 3D Three.js viewer per site (bundle embedded, works offline; seeded layout computed at render time). render_build adds a WebGPU replay of the construction itself — nodes and per-dimension edge waves animating onto the shells with a live HUD.

License

MIT. Crawled content remains the property of its respective owners.

Available Tools

10 tools
askB

LLM Q&A grounded in the site's pages: keyword retrieval + graph-neighborhood expansion, then embedding rerank and a cited answer. Requires SITE_KG_LLM_BASE and SITE_KG_LLM_KEY in the server environment.

NameRequiredDescriptionDefault
top_kNo
site_idYes
questionYes

TDQS

B3.2/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full disclosure burden. It usefully reveals the retrieval pipeline and the required LLM credentials, which is real operational context an agent needs. It omits what happens when those env vars are absent (failure mode), expected latency/cost of LLM calls, and whether the call is purely read-only.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two tight sentences, front-loaded with the core purpose before the pipeline detail and the env prerequisite. No filler, though the pipeline jargon is dense for the value it adds to tool selection.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

There is no output schema, and the description does say the result is a cited answer, which covers the most important return-value question. However, the response shape (answer string vs. object with citations), parameter meanings, and failure behavior under missing credentials are all left to guesswork for a 3-parameter tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate and it largely does not: site_id, question, and top_k (default 4) are all undocumented. Only 'site_id' is obliquely covered by 'the site's pages'; top_k's meaning (retrieval candidates? reranked results?) is left entirely unexplained.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description gives a specific verb+resource: answers a question grounded in the site's pages, returning a cited answer, and it names the mechanism (keyword retrieval + graph-neighborhood expansion, embedding rerank). This is far more informative than the bare name 'ask'. It never explicitly contrasts with siblings like search or neighbors, though the 'cited answer' framing implies the distinction.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is only implied: an agent can infer it should use this when it wants a synthesized, cited answer rather than raw pages from search/get_page. The one explicit guideline is an environmental prerequisite (SITE_KG_LLM_BASE / SITE_KG_LLM_KEY), not a when-to-use rule. No exclusions or alternative routing are stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_pageB

Full markdown body of one page, with title and source URL.

NameRequiredDescriptionDefault
page_idYes
site_idYes

TDQS

B3/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations exist, so the description carries the full burden. It does disclose the return payload (markdown body plus title and source URL), which is useful given there is no output schema, but it says nothing about truncation, missing/invalid page IDs, permission scope, or whether 'full' is actually full.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single compact clause with the most important information (full markdown content) front-loaded and no filler. It borders on under-specification rather than verbosity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With two undocumented parameters and no output schema, the description should at minimum explain how to obtain the IDs and what the response looks like. It partially covers the return shape but leaves the input contract entirely implicit.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0% — both site_id and page_id are bare strings with no descriptions. The description mentions only 'one page' and adds nothing about ID formats, how site_id scopes the lookup, or whether IDs come from search/list_sites results.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

Names the resource precisely (one page) and the payload (full markdown body, title, source URL), which distinguishes it from bulk siblings like search and list_sites. It lacks an explicit verb ("get"/"retrieve") and never names an alternative, but the scope is unambiguous.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No indication of when to use this versus search, neighbors, or render_site, and no preconditions stated. The agent must infer from the name alone that this is the single-page fetch after an ID is known.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

ingest_localA

Build a knowledge graph from a local directory of docs and/or code files.

Docs (md/markdown/mdx/txt/wiki/rst/adoc) are indexed as-is with edges from markdown links and [[wikilinks]]; code files contribute import/include edges and searchable symbol headers. Returns site_id plus a graph-quality verdict: READY (traversable edges found) or NOT-READY (no links/imports; use search).

relations selects the derived relation dimensions, chosen per node: "auto" (keyless: mentions_symbol/similar_to/sibling_of/supersedes/ shares_heading/shares_term/duplicate_of/coupled_with/co_cited_with), "llm" (auto + an LLM router plus entity/embedding similarity), "all", "off", or an explicit comma list. Omit it to use $SITE_KG_RELATIONS, or "auto" when that is unset. Content-derived edges also run on corpora with no cross-references; the verdict still reports NOT-READY for those.

NameRequiredDescriptionDefault
pathYes
max_filesNo
relationsNo
respect_gitignoreNo

TDQS

A3.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and does substantial work: it explains what edges each file class contributes, defines the READY/NOT-READY verdict and its one caveat (content-derived edges still run when the verdict is NOT-READY), and discloses the return payload (site_id + verdict). Gaps remain around cost/duration for large directories, idempotency of re-ingestion, and failure behavior on bad paths.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The purpose sentence is front-loaded and the file-type/edge breakdown follows logically. The relations paragraph is dense but each clause carries distinct information. Slight over-density in the closing sentence about content-derived edges and the verdict, which restates the caveat twice.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 4-parameter mutation tool with no annotations and no output schema, the description covers the return value and the relation model well. It falls short on `max_files` (what happens when exceeded — truncation? error?), `respect_gitignore` semantics, and whether ingesting the same path twice duplicates or replaces the graph — all decision-relevant for an agent.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate. It does so thoroughly for `relations`, enumerating every accepted value ('auto', 'llm', 'all', 'off', explicit comma list) and the env-var fallback chain — information found nowhere in the schema. However, `path`, `max_files` (default 2000), and `respect_gitignore` (default true) get zero explanation, leaving half the surface undocumented in both places.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Specific verb + resource + scope: 'Build a knowledge graph from a local directory of docs and/or code files.' The word 'local' implicitly separates it from the ingest_url sibling, and it enumerates the supported file types, so an agent knows exactly what corpus this accepts.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is useful conditional guidance inside the output contract — 'NOT-READY ... use search' tells the agent what to do after a link-less corpus — and the relations defaulting rules ($SITE_KG_RELATIONS, then 'auto') are spelled out. But there is no explicit when-to-use-this-vs-ingest_url statement and no stated preconditions (does the site need to exist first, does it replace or append to an existing graph?).

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

ingest_urlA

Crawl a website, build its knowledge graph, and register it as a queryable site.

Returns site_id for later tools plus a graph-quality verdict: READY (traversable edges found) or NOT-READY (tree without links; graph traversal useless).

relations selects the derived relation dimensions, chosen per node: "auto" (keyless dimensions), "llm" (LLM router + entity/embedding similarity), "all", "off", or an explicit comma list. Omit it to use $SITE_KG_RELATIONS, or "auto" when that is unset.

dev is the development/testing mode for sites you own or are authorized to test: it ignores robots.txt and politeness delays so a deep crawl of an authorized target finishes fast. It forces respect_robots=False.

NameRequiredDescriptionDefault
devNo
urlYes
max_depthNo
max_pagesNo
relationsNo
concurrencyNo
respect_robotsNo

TDQS

A3.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and discloses important behavior: it returns a READY/NOT-READY graph-quality verdict, explains the `relations` modes, and warns that `dev` ignores robots.txt and politeness delays and forces `respect_robots=False`. It stops short of describing registration side effects, idempotency, or permissions beyond the authorization warning for `dev`.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is front-loaded with purpose and return values, then covers the two highest-risk parameters. It is somewhat long, but each paragraph adds useful operational context rather than repeating structured fields.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a complex seven-parameter ingestion tool with no annotations and no output schema, the description gives solid high-level context, return behavior, and safety notes for `dev`. It remains incomplete because several crawl-scope and concurrency parameters are undocumented, which matters for correct invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate for all seven parameters. It adds meaningful detail for `relations` and `dev`, but leaves `url`, `max_depth`, `max_pages`, `concurrency`, and `respect_robots` largely or entirely unexplained beyond the schema's names and defaults.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb and resource: it crawls a website, builds a knowledge graph, and registers it as a queryable site. This clearly distinguishes the tool from siblings such as ingest_local, and it also states the return values (site_id and graph-quality verdict).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The intended use is implied by 'Crawl a website' and by the note that `dev` is for sites the caller owns or is authorized to test. However, it does not explicitly say when to choose this tool over ingest_local or other siblings, nor does it state when not to use it.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_sitesA

List ingested sites with their verdicts and graph sizes.

NameRequiredDescriptionDefault

No parameters

Output Schema

NameRequiredDescription
resultYes

TDQS

A3.5/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full disclosure burden; it does reveal a useful scope constraint (only already-ingested sites are listed) and the fields returned. It says nothing about ordering, result size, pagination, or cost, which matters for an unfiltered list over an unknown corpus.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single ten-word sentence that front-loads the verb and resource and adds only the fields worth knowing. Nothing wasted, nothing buried.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a zero-parameter list tool with an output schema covering return values, the description is nearly sufficient: what is listed and what accompanies each entry. The only missing piece is ordering/result-size guidance, which is minor here.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool takes zero parameters, so there is no parameter semantics for the description to explain and the baseline of 4 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (List) and resource (ingested sites) and even names the returned fields (verdicts, graph sizes), so the agent knows exactly what it gets. It does not explicitly differentiate itself from the sibling site_stats, which likely covers similar ground, so it falls short of a 5.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no statement of when to call this versus site_stats (aggregate stats?) or search (filtered lookup). The agent must infer usage entirely from the name and one-line purpose.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

neighborsB

Subgraph around a page up to depth hops (capped at 2). Use after search to expand context.

NameRequiredDescriptionDefault
depthNo
page_idYes
site_idYes

TDQS

B3.1/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations exist, so the description carries the full disclosure burden. It usefully reveals a behavior the schema does not: depth is capped at 2, a real constraint an agent needs. However it says nothing about what the subgraph contains (node properties, edge direction, truncation of large neighborhoods) or cost/latency, which matters for traversal tools.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two short sentences with the core behavior first and the routing hint second — no filler. It is arguably under-written rather than bloated, but structurally it is tight and front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no annotations, no output schema, and 0% parameter coverage, two sentences are not enough: the return shape of the subgraph, the meaning of the identifiers, and the depth default are all absent. Adequate to guess at the call, insufficient to call it confidently.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so all meaning must come from the description, and it only addresses `depth` — giving the hop semantics and the cap of 2. `site_id` and `page_id` remain bare names with no indication of format, and the depth default of 1 is not mentioned.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb-in-effect (retrieve the subgraph) and resource (pages neighboring a given page), scoped by `depth` hops. An agent can tell it apart from get_page (single page) and search (query-based retrieval). It never spells out 'neighbors of' in the schema sense, but the wording is concrete enough to act on.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

'Use after search to expand context' gives one concrete triggering condition and names the preceding sibling tool. There is no guidance on when not to use it (e.g. versus get_page for a single known page) or on what the result is good for beyond generic 'context'.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

render_buildB

Render the graph construction process as a self-contained WebGPU viewer (human inspection path). Replays the build: nodes are born in corpus order onto chapter rings, then evidence edges (ref/imports/contains), then one flying wave per derived relation dimension -- every edge grows from its source node and binds to its target with a pulse, while a HUD counts nodes, edges and per-dimension edges as they land. Returns the file path; AI-facing queries stay on search/neighbors/ask. Needs a WebGPU-capable browser.

NameRequiredDescriptionDefault
site_idYes

TDQS

B3.4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and does disclose useful traits: it produces a file and "Returns the file path," and it requires a WebGPU-capable browser. It does not say whether the output file is overwritten, whether rendering is slow/expensive, or whether it replays ingestion or only existing data.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The purpose is front-loaded in the first clause, then the visual sequence, then the return value and the environment requirement. Dense but every sentence carries information; the middle replayed-build sentence is long but justified by the tool's unusual visual semantics.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Return value is stated (file path) and the browser prerequisite is called out, which covers the two things an agent most needs for a render tool with no output schema. Gaps remain around the site_id parameter, cost/runtime, and how it relates to render_site, making it adequate but not complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0% and the sole required parameter site_id is never mentioned in the description, so nothing is added beyond the bare schema field name. The agent must infer that site_id comes from list_sites and what a 'site' means here.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

Specific verb+resource: "Render the graph construction process as a self-contained WebGPU viewer," with concrete detail on what is visualized (chapter rings, evidence edges, per-dimension waves, HUD counters). It does distinguish itself from the query siblings by scoping itself to human inspection, but it never names or contrasts with render_site, the closest sibling.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly frames itself as the "human inspection path" and states that "AI-facing queries stay on search/neighbors/ask," which is real when-not-this-tool guidance. The obvious remaining alternative, render_site, is not mentioned, so the agent must infer which renderer to pick.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

render_siteB

Render the graph as a self-contained HTML viewer (human inspection path). Archive mode returns the file path; mode="live" replays the stored graph over a loopback SSE server and returns {live_url, run_id, last_seq}. AI-facing queries stay on search/ask/neighbors.

NameRequiredDescriptionDefault
modeNoarchive
sceneNoplanets
site_idYes
max_nodesNo

TDQS

B3.4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations exist, so the description carries the full burden. It does well on output shape (archive returns a file path; mode=live returns {live_url, run_id, last_seq} over a loopback SSE server), but says nothing about side effects such as file writes, server lifetime/cleanup, or whether max_nodes truncates silently.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three tight sentences, front-loaded with the purpose and the human-vs-AI routing decision, with no filler. Dense jargon ('loopback SSE server', 'last_seq') is used without unpacking, which costs a bit of readability.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no annotations and no output schema, the description does the heavy lifting on purpose, mode semantics, and return values. It still leaves three of four parameters undefined and omits operational details (server lifecycle, truncation, permissions), which matters for a tool that both writes a file and opens a local server.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0% and the schema supplies only bare titles and defaults, so the description must compensate. It fully explains `mode` (archive vs live) and ties it to distinct return values, but site_id, scene, and max_nodes get no meaning at all — max_nodes in particular could imply truncation behavior that is left unstated.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Render the graph as a self-contained HTML viewer') and frames it as the human inspection path, which separates it from the search/ask/neighbors tools. It does not distinguish itself from the sibling render_build, so an agent cannot tell which renderer to pick from the description alone.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly routes AI-facing queries away to search/ask/neighbors, giving a clear when-not-to-use rule alongside the 'human inspection' use case. The missing piece is a comparison to render_build, the other rendering sibling, which is left to inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

site_statsB

Graph health for one site: doc/edge counts, edge types, verdict, top hubs.

NameRequiredDescriptionDefault
site_idYes

TDQS

B3.1/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full behavioral burden. It usefully discloses the returned metrics and that the scope is one site, but does not state that the operation is read-only, what permissions are needed, or any cost/rate-limit behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

One front-loaded sentence with a compact colon list; every element names a return metric or scope and there is no filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple one-parameter stats tool with no output schema, the description enumerates the returned fields, which is helpful. However, it leaves the site_id source/format unexplained and, with no annotations, does not cover read-only behavior or access requirements.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

There is a single required site_id parameter with 0% schema description coverage, and the description only says 'one site' without explaining the ID format, source (e.g., list_sites), or validation. It adds almost no parameter-level meaning beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

Names the resource (graph health for one site) and enumerates the exact metrics returned: doc/edge counts, edge types, verdict, top hubs. It is clearly distinct from siblings like get_page, neighbors, search, and list_sites, though it uses a noun phrase rather than an explicit verb.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Implies the tool is for inspecting graph health metrics of a single site, but gives no explicit when-to-use guidance, no exclusions, and no alternatives such as neighbors or search for related queries. An agent must infer the selection context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 10 tool updatesv0.1.0
    • First observedask
    • First observedget_page
    • First observedingest_local
    • First observedingest_url
    • First observedlist_sites
    • First observedneighbors
    • First observedrender_build
    • First observedrender_site
    • First observedsearch
    • First observedsite_stats

TDQS

B3.4/5.0

Scored across 10 tools

Disambiguation4/5

The tools mostly target distinct operations: ingestion, listing/stats, search, page retrieval, neighbor expansion, rendering, and Q&A. The only mild overlap is render_site vs render_build, since both produce visual viewers, but their descriptions distinguish archive/live graph viewing from build-process replay.

Naming Consistency3/5

All names use lowercase snake_case, which keeps them readable. However, the set mixes verb_noun names (list_sites, ingest_local, get_page, render_site) with bare nouns (site_stats, neighbors) and bare verbs (search, ask), so there is no fully predictable naming pattern.

Tool Count5/5

Ten tools is well-scoped for a knowledge-graph ingestion and query server. Each tool covers a distinct phase: ingest, list/inspect, query, expand, render, or ask.

Completeness4/5

The surface covers the core lifecycle: build graphs from local/URL sources, list sites, inspect graph health, search/get pages, expand neighbors, render, and ask grounded questions. Minor gaps remain around site deletion, re-ingestion/update management, or explicit graph export, but normal query workflows are covered.

Maintenance

ActivityMaintained
ResponsivenessUnresponsive

Related MCP Connectors

Related MCP Servers

  • A
    license
    Not graded
    quality
    B
    maintenance
    Enables AI agents to read public web content as clean Markdown, search a private library of previously read documents, and access these tools via MCP, REST API, CLI, and WebMCP.
    47 npm
    20
    MIT
  • A
    license
    Not graded
    quality
    A
    maintenance
    Enables MCP clients and AI agents to fetch individual webpages or crawl sites and receive clean, link-preserving Markdown with caching and robots.txt-aware rate limiting, reducing token usage for downstream pipelines.
    MIT
  • A
    license
    Not graded
    quality
    A
    maintenance
    Enables AI agents and MCP clients to fetch web pages and crawl sites locally, converting content into clean Markdown with preserved links, caching, robots.txt compliance, and minimal token usage.
    MIT