Mycelium

An on-call agent for data incidents, built on a bio-inspired context protocol.

An agent should not read the catalogue. It should forage in it.

Built for Build with DataHub: The Agent Hackathon. Apache-2.0. Runs offline against a real DataHub datapack — no server required to try it.

pip install mycelium-datahub
mycelium triage "freshness SLA missed on ORDER_DETAILS"

P1  CUSTOMERS fails the default quality failing assertions Test(s) and is 50.1x past its
    SLA — 35 assets downstream, 1 watched
    page now — a governed consumer is serving wrong or missing data

Suspects (4 found)
  2 hop CUSTOMERS (origin)
      · test: failing DataHub Test(s) on the data: default quality failing assertions
      · freshness: last written 1203.0h ago against a Daily SLA (50.12x the window)

Who   no owner is recorded on the asset; this is the escalation path the catalogue implies
      · escalate via data product Customer Analytics: bryan.prosser@datahub.com

Next  1. Check the job that writes CUSTOMERS: it has not produced data in 1203h.
      2. Notify consumers — 35 assets downstream, starting with datahub_order_entries.

Not determined
      · the evidence comes from DataHub Tests rather than assertions, so which rows are
        affected is not recorded

severity, blast radius, ownership and timeline computed from DataHub; narrative by
deepseek. 57 DataHub calls, 734 context tokens, 1 model call, 15,005 ms.

Or open the page: docker run -p 7860:7860 ghcr.io/dihannahdi/mycelium.

The severity, the affected assets, the owners and the timeline are computed from DataHub. A model only writes the prose. That order is the whole design: an engineer at 03:00 can check every number against the catalogue and read the rule that produced the P-level — and with no API key at all the report loses its narrative and keeps every fact. A triage tool that goes down because a third-party inference API is rate-limiting you is not an operations tool.

The problem

Most of what a catalogue gets asked is a lookup — who owns this, is it fresh, what are the columns. An agent with a generic tool loop spends thousands of tokens on each one, because the loop has no way to tell a lookup from an investigation. In our measurements, a documented-practice DataHub MCP tool loop puts 8,154 tokens in front of the model to answer "who owns ORDER_HISTORY?" — a median of 8,696 across the nine lookup questions in the benchmark. Mycelium answers them with none.

The harder questions have the opposite failure. Ask "why is the Order Entry Dashboard showing stale order totals?" and the agent can load everything it might need — schemas, lineage, owners, queries, assertions — most of it about tables with no bearing on the problem. Or it can load too little, answer from the first three tables it found, and be confident and wrong.

Both failures share a root: the agent is handed context instead of acquiring it. And every session starts from zero — the fiftieth investigation of the same pipeline costs exactly what the first one did, though any human on the team would have learned the path by now.

Mycelium is a protocol layer between an agent and DataHub that changes the acquisition model. Four mechanisms, each borrowed from a system that already solved a version of this problem.

Related MCP server: AIOps MCP

The job it does

The protocol exists to serve a task, and the task is incident triage: an alert arrives and an on-call engineer needs six things inside a minute.

Plus a seventh that separates a tool from a demo: what could not be determined. A report that never says "I don't know" is not being careful, it is being quiet.

Three surfaces, one report: mycelium triage (with --output json|markdown|slack and severity exit codes for a webhook), the mycelium_triage MCP tool, and POST /api/triage.

Worth knowing about the shipped catalogue: DataHub's published showcase-ecommerce pack contains no failing assertions at all. Every strong finding rests on failing DataHub Tests, which the engine reads as first-class evidence and splits by category — a failing __default_quality_* Test is an incident, a failing __default_governance_documentation is real and is not why the dashboard is stale. Conflating them once put "failing tests include governance ownership and cost change percentile" into a paragraph explaining a freshness breach.

The four mechanisms

1. Hyphal foraging

A fungal mycelium has no map and no planner. It pushes hyphae outward; where one finds nutrients the network thickens that channel, and branches that found nothing wither. Tero et al. (Science, 2010) showed this literally — Physarum polycephalum on a map of Tokyo, oat flakes at the stations, reproducing the real rail topology overnight.

Applied to a catalogue: instead of reading the schema, the agent releases cheap spores along lineage edges. Each edge carries a pheromone weight. Trails that reach an asset the agent ends up citing get pheromone deposited, weighted towards the discovery end. Everything evaporates a little each cycle, so the network never fossilises around last quarter's incidents.

The frontier score for a candidate reached over edge e at depth d:

score = w_τ · norm(τ(e))         learned:   has this path paid off before?
      + w_s · similarity(goal,n) lexical:   does this node look relevant?
      + w_p · n.prior()          intrinsic: is it busy, broken, recently changed?
      + w_a · affinity(n)        history:   has it ended up in answers?
      − w_h · d                  cost:      distance is not free

n.prior() treats a low quality score as attractive. A heavily-queried table with a failing assertion is exactly where a debugging agent should look. The signals come straight from DataHub: usageFeatures, assertionsSummary, and the dataQualityScore structured property.

Weights live in SQLite and persist across sessions — you can open the file and sort by tau to read off which paths your team's investigations have carved.

2. Semantic zoom

Human working memory holds about seven chunks (Miller, 1956). We read a book by holding chapters and gists, and zoom in only when a detail is needed. Three resolutions per node, with measured costs:

The agent does not pick its own resolution. A planner allocates the expensive seats — at most four nodes at Leaf — and simulates the cost against the token budget before anything is fetched. Truncation is always announced: a table rendered with 40 of 200 columns says so, because silently dropping columns is how an agent comes to believe a column does not exist.

3. Synaptic pruning

A maturing brain actively destroys rarely-used synapses. The pruning is not damage; it is what makes the rest fast.

Miller's seven is cited here and it is worth being precise about what actually binds: in practice the resident set settles at three, not seven, because the three-step recency rule claims a chunk before the slot limit ever does. The seven-slot ceiling is real but rarely the constraint — so the famous number is the framing, and the recency rule is the mechanism.

Every resident chunk carries an activation that decays each step and refreshes on reference. Then: anything untouched for 3 steps is evicted regardless of activation; anything below the demote threshold loses a zoom level (Leaf → Tree → Forest), shedding most of its cost without disappearing; anything below the evict threshold leaves, lowest-activation first.

Eviction writes to long-term storage first. recall(urn) brings it back at full fidelity with fresh activation — remembering is itself an act of attention. Nothing is destroyed; it is only set down.

Measured over a 40-step loop admitting a 600-token chunk every step:

resident tokens    600 → 1,200 → 1,800 → 1,800 … 1,800   (budget 6,000)
resident chunks    settles at 3
admitted           40
accounted for      40 of 40   ← 3 resident + 37 in long-term store, all at original size

Flat, and nothing lost. The binding constraint turns out to be the three-step recency rule, not the seven-slot capacity.

4. Dual-process routing

Pull your hand off a hot pan and the spinal cord handled it. Work out why the pan was hot and the cortex is involved. Kahneman's System 1 and System 2.

A router classifies with zero model calls — a pattern table (English and Indonesian) plus entity resolution against DataHub search. Lookups get a template and a handful of DataHub calls — one to eight in the benchmark, and never a token of context. Investigations get the foraging loop.

The load-bearing part is knowing when it doesn't know. Two genuinely different assets sharing a name is an ambiguity, so the reflex refuses and says why. But a dbt model and the warehouse table it builds are siblings in DataHub's own model — one asset, two platforms — so those collapse instead of escalating. Getting that distinction right is what took the reflex from answering 1 of 5 lookups on the real catalogue to answering 5 of 5.

Measurements

Three strategies, same questions, same catalogue, same token estimator. Every arm makes real DataHub calls through one counted client and gets the same system prompt. Reproduce with python -m bench.run; full table in bench/results/.

Evidence surfaced asks whether the expected asset appeared in the assembled context — model-free, so the comparison does not depend on prose quality or on which model ran. All three surface the evidence on all thirteen scored questions. This is a comparison of cost at equal recall, not of one strategy finding things the others missed.

Where the saving actually comes from

Split by question type, and the picture is more specific than the headline:

So: on diagnostic questions Mycelium is level with a plain search-and-render dump (1.0×) and 2.8× cheaper than the MCP tool loop. The overall 6.4× comes almost entirely from routing — recognising that most catalogue questions are lookups and answering them with no model call and no context at all.

That is a narrower claim than "foraging beats search", and it is the one this catalogue supports. On 197 entities, search usually lands on the culprit directly, so walking the lineage graph buys accuracy that search already had.

DataHub API calls are a wash: 27.2 for both Mycelium and the MCP loop. Tokens are 18.2× cheaper by median and 8.8× by mean; round trips are not cheaper at all. A protocol that advertises frugality should say which axis it is frugal on — and which average it is quoting.

Does the advantage widen with the catalogue? Yes — measured

This was a hypothesis in an earlier draft of this README. python -m bench.scale now tests it. The showcase-ecommerce topology is replicated into N tenants over one shared set of org-wide entities (people, tags, glossary — one company, many teams). A broken table is planted in exactly one tenant. Dataset names collide across tenants on purpose, because every team really does have an order_details; dashboards carry the team name, because BI assets are named that way. So the culprit is reachable by lineage and not by name, and the question names one tenant's dashboard.

✓ / ✗ is whether the broken table reached the context at all. Three different shapes:

• The dump stops finding the answer. Not because it was given a stingy top-8 — the culprit simply sinks in the ranking as its identically-named twins multiply: rank 6 → 12 → 36 → 132 → 260. Dumping down to rank 260 at leaf resolution costs 42,342 tokens, against Mycelium's 2,840. That is 14.8× at equal recall, and it is the fair comparison, because the 1,391 in the table above is the cost of not finding it.

• The MCP loop keeps recall and pays for it in width, 8,896 → 14,851, then flattens once the lineage neighbourhood it reads is fully explored. 5.2× Mycelium at the same recall.

• Mycelium is flat at 2,840 from 656 entities upward, because its cost is bounded by the context budget rather than by what the catalogue happens to return.

So the honest summary of both experiments: at 197 entities the foraging is level with search and the win comes from routing; by 20,000 entities search-and-dump has stopped working and foraging is the cheapest strategy that still answers. That said, this is replicated topology, not a second real catalogue — it holds lineage depth and branching constant and grows only the number of competing names. It isolates the variable the hypothesis was about and cannot stand in for a genuinely different organisation's mess.

It also found a real bug. The walk's diminishing-returns rule had a patience of two weak waves, with a comment explaining that a diagnostic path "often crosses a thin BI layer before reaching the warehouse table that actually broke". That layer is two hops deep — chart, then embedded BI view — so a patience of two stopped the walk one hop short of the answer, every time. The main benchmark never noticed, because its dashboard questions are answered at the dashboard, which is five months stale and genuinely its own cause. Patience is now three, and the main benchmark's numbers did not move at all.

What repetition is worth

python -m bench.run --warm 4 reinforces each question four times before measuring:

Recognition fires when a remembered asset is a seed of the new walk and is still unhealthy. Both halves are required: affinity alone would make the network answer every question with whatever it answered last, and the health check is what sends it back to work once an asset is fixed. It engages on two of the seven diagnostics (−60% and −38% context) and two get dearer by 580 and 597 tokens, because a warmed frontier reorders which neighbours are worth expanding and sometimes that means expanding more of them. Where the conditions do not hold, repetition costs exactly what the first run did — which is why the aggregate improvement is 7% and the best per-question one is 60%.

Corrections made to this table

Worth stating, because the numbers moved a long way and always downward:

• An earlier draft claimed 21× on diagnostics. That came from diagnostic questions being short-circuited into a cheap lookup path they did not belong in.

• Then 8.3×, against an MCP baseline that rendered every lineage node at full leaf detail — which DataHub's own skill guidance explicitly tells agents not to do. Rewriting that arm to follow the documented practice (batched tree-level enrichment, leaf only on the seeds) took it from 22,195 to 8,718 median tokens, and the ratio to 2.8×.

• Five of the questions were co-developed with the router, so the reflex rate they produced (5 of 5) was tuning on the test set. Four held-out questions — different subject area, one with no asset name in it, one naming an asset that does not exist — bring it to 7 of 9 that cost no context. Strictly, 6 of 9 reach the reflex; the seventh is the nonexistent asset, which escalates and then costs nothing anyway because the router's failure to resolve it is the answer. Both framings are in the table; this one is the honest way to say it.

Reproduce any of it with python -m bench.run (or python -m bench.scale for the scaling table); artefacts land in bench/results/ — bench-cold.md, bench-warm.md, scale.md. Both run in CI, so a change that quietly breaks a measurement fails the build rather than the README.

What is asserted, not just described

pytest -q → 520 tests, one skipped (the live smoke test, which needs a running DataHub). CI runs the suite, both benchmarks, and the CLI on 3.11 and 3.12; a second workflow builds the container image, starts it, and asks it a question before pushing.

The last three exist because each was a bug that every mechanism test passed through. Citations were being trimmed with rstrip(".,;:)"), which eats the closing parenthesis that ends every DataHub dataset URN — so nothing ever matched, nothing was ever reinforced, and the learning claim was quietly false. Findings were admitted to a seven-slot memory without bound, so the last seven survived rather than the best seven. The learning test itself asserted only second <= first * 1.2 — a non-regression check wearing the name of an improvement check, which passed while repetition made no difference at all. And pyproject.toml declared a mycelium console script pointing at a module that did not exist, so pip install produced an executable that crashed on every invocation.

Each was invisible in isolation, and obvious the moment the whole thing ran against a real catalogue — or the moment someone typed the tool's own name.

Giving the learning back

Weights in a local SQLite file are a private optimisation. The point of a context platform is that knowledge accrues where the team can see it, so Mycelium writes its hot paths back into DataHub as a knowledge-base document and marks the assets involved with a mycelium.pathWeight structured property.

The document is written for a human: which paths repeated investigations actually traversed and cited, with the caveats stated — a weight reflects traffic, not correctness, and low-quality assets attract the walk on purpose.

Off by default. Writing into someone's catalogue is not a side effect of asking a question: it takes MYCELIUM_WRITEBACK_ENABLED=true and the live backend.

Running it

Triage an incident

mycelium triage "freshness SLA missed on ORDER_DETAILS"
mycelium triage "the exec dashboard shows last week's numbers" --output slack
mycelium triage "assertion failed" --subject 'urn:li:dataset:(...)' --output json --exit-code

--subject skips name resolution, which is what a real alerting integration does — the monitor already knows which asset fired. --exit-code returns 10/11/12 for P1/P2/P3 so a webhook script can branch on severity without parsing prose. --writeback puts the report into DataHub as a document and tags the assets involved, which is what makes the next incident on that asset say "this is the third time, here is what the last two said".

Reasoning backends

Nothing is required. With no key, the narrative is composed by a deterministic reporter that restates the evidence and says so on screen; every fact in the report is unchanged. With a key, a model writes the prose over the same computed facts.

export DEEPSEEK_API_KEY=...     # or ANTHROPIC_API_KEY
mycelium triage "..." --llm deepseek

Claude wins when both resolve. Forcing a backend that has no credential raises rather than degrading quietly — an operator who believes a model reasoned over their incident when nothing did is worse off than one who got an error.

The page

docker run -p 7860:7860 ghcr.io/dihannahdi/mycelium
# or
uv venv && uv pip install -e ".[dev]" && python -m mycelium.web.app

Watch the lineage graph light up along a narrow path while the token meter stays flat. Click run it again to see the second investigation take the reinforced trail.

The image is public and carries the catalogue inside it, so that first command needs no clone, no API key and no DataHub. curl localhost:7860/api/health should report "entities": 197 — if it ever reports 0, the catalogue did not ship and every answer will be "no asset matching". To host the same image as a public demo, see deploy/huggingface/DEPLOY.md.

As an MCP server, in your agent

claude mcp add mycelium -- uvx --from mycelium-datahub mycelium-mcp

Add -e MYCELIUM_BACKEND=live -e DATAHUB_GMS_URL=... -e DATAHUB_GMS_TOKEN=... to point it at your own DataHub. Nine tools: triage first when something is broken, then ask, forage, zoom, step, recall, reinforce, stats and writeback.

As a DataHub Skill

skills/mycelium-forage/SKILL.md follows the datahub-skills format and teaches an agent the discipline: route before you think, forage before you read, zoom only the top three, let things go, reinforce what paid off. It extends /datahub-search, /datahub-lineage, and /datahub-quality rather than replacing them — underneath, it is calling the same DataHub tools, just far fewer of them.

Two backends, one contract

Both implement the same DataHubClient protocol and are exercised by the same contract tests. The snapshot is real DataHub metadata — 197 entities, 546 edges, with the generated documentation, usage statistics and quality properties the mechanisms depend on — not a hand-written mock.

Timestamps are anchored to the present the way datahub datapack load does, so an offline run does not make the whole catalogue look half a year stale. Be aware of what that means: the freshest asset in the pack always reads as written just now, because the shift is derived from it. Relative staleness across the catalogue is faithful; the absolute figures are anchored, not observed.

Reasoning is optional

System 1 never calls a model. System 2 uses Claude when a credential resolves, and otherwise a deterministic reporter that composes findings from the evidence the protocol gathered. It is labelled as such everywhere it appears — it restates, it does not reason. It exists so the mechanism can be inspected end to end without an API key, and so the benchmark can compare context cost without model variance confounding it.

Honest limitations

• The foraging is not what wins. On diagnostic questions it is level with a plain search-and-render dump at this catalogue size. The measured advantage comes from routing — from most questions not needing an investigation at all. The lineage walk earns its place by producing a bounded answer with a citable trail, not by being cheaper than search.

• Mechanism 3 has never run against a live model. The loop iterates only when the model asks for something (NEED: ZOOM / NEED: RECALL), and the deterministic backend never asks — so with no API key every run finishes in one step and decay never ticks. The loop, the three-step prune rule, the attention bookkeeping, the step ceiling and both failure paths are now driven by a scripted model in the test suite, which is honest about what it proves: the mechanism is correct, and it is unmeasured on a real incident with a real model deciding what to keep.

• The suspect is chosen by a second ranking. The forager ranks by relevance; _pick_suspect then re-ranks the shortlist by severity of evidence. Two evaluations rather than two rankings: the first is a cheap guess made before paying for a health call, the second the same judgement once the call is paid for. They now pivot on one shared threshold (staleness_suspect_hours) and the polarity of the clock follows the question — silence is suspicious for "why is this stale", recency for "what changed upstream". Before that they contradicted each other outright: the walk rewarded the freshest tables on a staleness question and penalised the four-day-old culprit. Fixing it moved the mean context from 1,126 to 1,090 with the same evidence recall, which is a small win on a small catalogue and the right shape of change regardless.

• Round trips are not reduced. 27.2 DataHub calls against the MCP loop's 27.2. Only tokens are cheaper.

• One catalogue, one subject area. Every measured number here comes from the 197-entity showcase-ecommerce datapack. The live backend was exercised against a real DataHub v1.6.0 holding 1,245 entities — enough to prove the GraphQL path, not enough to claim the mechanism scales. Nothing above that size has been tried.

• The live backend's GraphQL field names were checked against DataHub's own schema for datasets, dashboards, charts, containers, domains, tags, users and groups; for dataJob, dataFlow, dataProduct and glossaryTerm a few annotation fields follow the schema's consistent pattern rather than having been individually re-read. A wrong guess there degrades one entity type's detail, not the run.

• The showcase-ecommerce pack ships no query entities, so queries_for returns empty against it. The parsing path exists and is covered by a synthetic test, but is unverified against real DataHub query output.

• Census.domains[].entity_count is free for the snapshot backend and would cost a query per domain live, so the live backend reports zero.

• Answer quality is only measured with --llm. The default benchmark mode measures context cost and evidence recall, which is what this project claims to improve.

Documentation

• docs/ARCHITECTURE.md — design of record, and why each constant is what it is

• docs/DEMO_SCRIPT.md — the shot plan for the video

• src/mycelium/config.py — every tuning knob, with its rationale

References

• Tero, A. et al. (2010). Rules for Biologically Inspired Adaptive Network Design. Science 327(5964), 439–442.

• Dorigo, M. (1992). Optimization, Learning and Natural Algorithms. PhD thesis, Politecnico di Milano.

• Miller, G. A. (1956). The Magical Number Seven, Plus or Minus Two. Psychological Review 63(2), 81–97.

• Kahneman, D. (2011). Thinking, Fast and Slow.

• Huttenlocher, P. R. (1979). Synaptic density in human frontal cortex. Brain Research 163(2), 195–205.

Licence

Apache-2.0. See LICENSE.