ML On-Call Agent

"Why did last night's run regress?" — a multi-agent system that answers it by correlating a drift report, an eval run and a deploy log, and cites every claim it makes.

Exposed over MCP, so the tools work from any MCP client.

Status: 65 tests. The diagnosis is computed deterministically from weighted evidence, so every number below is an assertion, not a demo.

The real problem

An ML system degrades silently. When somebody finally notices, the evidence is scattered across three places that do not talk to each other:

• a drift report — did the input distribution move?

• an eval run — did the offline scores regress, and on which slices?

• a deploy log — did anyone ship anything?

Correlating them is a job, and it is a job people do badly at 3am because it means holding three JSON files in your head at once.

Those three artifacts are not invented for this repo. They are what model-drift-monitor, llm-eval-pipeline and ai-code-review-bot already produce. The reader was written against a real artifact, not against documentation.

The result

python -m oncall.cli evaluate

 scenario         truth           verdict          conf  margin  steps  action
 healthy          healthy         healthy          0.57     2.0      9  no action
 data_shift       data_shift      data_shift       0.71     8.0      9  retrain on recent data
 bad_deploy       bad_deploy      bad_deploy       0.66     8.5      9  roll back
 concept_drift    concept_drift   concept_drift    0.81     9.0      9  retrain with fresh labels
 pipeline_break   pipeline_break  pipeline_break   0.62     5.0      9  fix the upstream pipeline
 flaky_eval       noise           noise            0.62     3.0      9  rerun the evaluation

task success    : 100%
action accuracy : 100%
mean steps      : 9.0

Ground truth is known by construction, so the agent is scored rather than admired. "The agent produced a plausible incident report" is not a result — plausible is what language models do, including when the evidence supports nothing.

The scenario that is the point

concept_drift: the inputs are statistically identical, nothing was deployed, and the model's ranking has inverted (AUC 0.729 → 0.326, worse than random).

There is nothing to point at. The answer comes from combining two negatives and one positive — no drift, no deploy, ranking collapsed — which is exactly what a summary of any single artifact would miss. It is also the blind spot model-drift-monitor documents about itself, diagnosed one layer up.

The design decision everything rests on

The diagnosis is deterministic. The language model only narrates.

The obvious build is to paste three JSON files into a prompt and ask what went wrong. It produces something fluent every time — including when the evidence supports nothing — and it is untestable, because you cannot assert on prose or tell a correct answer from a lucky one.

So the reasoning is ordinary Python:

• each specialist reads one artifact and emits Findings

• every finding carries a citation — file, field, value. Finding requires one, so a claim without a source cannot be constructed

• each finding names which root causes it supports and which it rules out

• diagnose() sums the weights — a pure function, unit-tested against truth

| finding                                                    | source                     |
|------------------------------------------------------------|----------------------------|
| input drift is 'none': the scored population is             | `drift:severity=none`      |
| statistically the same as training                          |                            |
| roc_auc is 0.326 - WORSE THAN RANDOM. The model's ranking   | `evals:metrics.roc_auc     |
| has inverted, which is a changed relationship               |  =0.326`                   |
| 1 change(s) landed but none touch model behaviour           | `changes:changes[].files`  |

test_the_narration_does_not_change_the_diagnosis asserts the verdict is identical with and without a model. If it ever fails, the model has started doing the reasoning — and the reasoning stops being testable the moment it does.

Negative evidence is where most of the diagnostic power is. "No drift" and "nothing was deployed" are findings, with weights. An LLM summarising the drift JSON would skip them, because nothing happened.

The graph

   SUPERVISOR ──► drift ──┐
        ▲   ├──► evals ───┤   one specialist per artifact, each consulted once
        │   └──► changes ─┤
        │                 ▼
        │             DIAGNOSE          sum the weighted findings
        │                 ▼
        └──────────────  CRITIC         "is this conclusion supported?"
           (bounded)      ▼
                        REPORT

Two properties a straight-line pipeline does not have:

The critic can send work back. If the verdict rests on two artifacts while a third was never read, control returns to the supervisor rather than publishing a conclusion drawn from half the evidence.

The cycle is bounded. MAX_REVISIONS = 2 caps it and recursion_limit catches anything that escapes. An agent that can loop is an agent that can loop forever, and a runaway on-call agent generates pages instead of answering them.

It runs with no LLM and no API key — the supervisor routes by a plain Python rule — so the routing, the delegation and the critic's loop-back are all unit-tested offline. test_the_graph_and_the_plain_loop_agree asserts a LangGraph run and a plain for loop reach identical verdicts on every scenario, which proves the graph adds orchestration, not reasoning.

What each artifact is actually worth

python -m oncall.cli ablate

An honest negative result, and it is asserted so it cannot be quietly forgotten. Removing the change log costs nothing on this scenario set — bad_deploy is already separable from the drift and eval evidence alone. The change log earns its place by naming the commit, which is what a human needs in order to act, not by changing the diagnosis.

test_the_change_log_currently_changes_no_verdicts pins that. If a future scenario makes it load-bearing, the test fails and this table has to change. That is what the assertion is for.

Ablation is the only way to discover that a source you spent a week integrating changes no verdicts.

And when an artifact goes missing

Jobs fail, buckets are empty, paths change. The interesting question is not whether the agent still answers — it will — but whether it notices:

  missing drift     -> verdict bad_deploy   flagged=True
  missing evals     -> verdict bad_deploy   flagged=True
  missing changes   -> verdict bad_deploy   flagged=True

An agent that quietly diagnoses from two of three files is worse than one that refuses, because nobody knows to distrust it.

The MCP server

python -m oncall.mcp_server                 # stdio, for Claude Desktop et al
python -m oncall.mcp_server --http --port 8931

Seven tools — list_incidents, get_drift_report, get_eval_run, get_changes, investigate_incident, compare_incidents, list_specialists — plus a resource and a templated resource.

The tools return evidence, not prose. Each hands back the artifact or the structured diagnosis, so the client's model reasons over data with citations rather than over a paragraph somebody already summarised. A summary is where the detail goes to die.

Written against mcp 2.0.0, which is a breaking rewrite

Worth stating, because almost everything published is 1.x and does not work. Each of these was verified by running it:

Two more that cost real time:

• @server.tool() must be called. Bare @server.tool raises a TypeError that says so.

• A bare -> dict produces no structuredContent at all. The text content is still there, so it looks fine in a chat client and silently breaks any client reading the structured field. Every tool here returns a parameterised generic (Dict[str, Any]) for that reason, and there is a test.

Testing a protocol server without a subprocess

Client(server) accepts a server object directly, so the entire protocol round trip runs in process — no subprocess, no port, no flakiness from either. That affordance is the only reason protocol tests are cheap here.

Quickstart

git clone https://github.com/kanishqtanwar35-hub/ml-oncall-agent
cd ml-oncall-agent
pip install -r requirements.txt
export PYTHONPATH=src

python -m oncall.cli incidents                  # what can be investigated
python -m oncall.cli investigate concept_drift  # the full report
python -m oncall.cli evaluate                   # score it against truth
python -m oncall.cli ablate                     # what each artifact is worth
pytest -q                                       # 65 tests

Point it at real artifacts:

python -m oncall.cli write ./artifacts --scenario bad_deploy
# then: Evidence.load("./artifacts") reads a real pipeline's output unchanged

Everything runs with no API key. investigate --narrate adds a model-written opening paragraph if GEMINI_API_KEY is set, and changes nothing else.

Bugs worth reading about

A hardcoded tolerance swallowed the flaky-eval case. regressions() used a 0.02 threshold, so a 0.006 move never registered and the noise check never ran — the agent said healthy where the truth was noise. That is exactly the folklore-threshold mistake model-drift-monitor argues against, reproduced one repo over. Detection and significance are two questions: the floor is now 0.005 ("anything a dashboard would show") and the harness's own measured noise_std does the discriminating.

The recovery test could not fail. It faked a skipped specialist by pre-seeding consulted — but the critic compares available against consulted, so marking something consulted hid the gap from the very check being tested. It reported recovered: False and looked like a broken critic. The fault has to be injected in the routing, not the bookkeeping; build_graph(skip_first_pass=...) does that, and the critic now demonstrably catches the gap and sends the supervisor back.

Limitations, stated plainly

• The scenarios are synthetic. Deliberately — real incidents do not come with a labelled root cause, which is precisely why diagnosing them is hard. The numbers characterise the method, not any production system.

• Six scenarios is a small set. 100% on six is not 100% in general, and the next scenario added is more likely to break it than confirm it.

• Calibration is unmeasurable here. With no wrong answers there is nothing to compare confidence against. The CLI prints n/a and says why rather than inventing a figure — test_calibration_is_honestly_unmeasurable_here pins that.

• The weights are hand-set. They encode my judgement about what evidence is worth, and a larger labelled incident set would let them be fitted instead — with a held-out split, because fitting them on six scenarios would be memorisation.

• Tool-selection accuracy is trivially 1.0 with three always-present artifacts. The metric is here because it starts mattering at the fourth tool, and adding it retroactively is how you discover the agent has been calling everything for months.

• No live integrations. It reads artifacts from disk. Wiring it to a real warehouse, CI system and git host is deployment work, not reasoning work.

• The critic checks completeness and support, not correctness. It cannot tell you the weights are wrong, only that the evidence was thin.

Roadmap

1. A larger labelled incident set, and fit the weights on a held-out split.

2. More specialists — serving latency, cost ledger, feature-store freshness — which is where tool-selection accuracy starts to mean something.

3. Wire the MCP server to real artifact sources instead of a scenario builder.

4. Multi-incident correlation: three services regressing at once is one incident, not three.

Licence

MIT.