agent-viz

A living asset that extends communication between humans and AI agents from chat to visuals (graphs, tree diagrams, heatmaps, flowcharts). Shared across Kaggle / AtCoder Heuristic / trading model development.

Design policy (based on 2026-08-24 research)

• Backbone of records = MLflow (local file store). Agents write, humans view via mlflow ui

• Custom figures = self-contained HTML (Plotly). Rendered inline in MLflow's artifact view

• What we maintain ourselves is only a thin layer: "common trial ledger schema", "report components", and "domain adapters"

Source of research: KnowledgeBase 00_Inbox/人間とAIエージェントのビジュアル意思疎通ツール 調査メモ

Related MCP server: NetSpectre WebMCP Challenge Edition

Phase 0 (implemented)

• agentviz.schema — common trial ledger schema. 1 trial = TrialRecord, case = seed / fold / period

• agentviz.ledger — TrialLedger. log_trial / fetch_trials to MLflow

• agentviz.report — build_report. Self-contained HTML with trial ledger table + metric trends + per-case × per-trial relative score heatmap

Usage

# セットアップ
.venv\Scripts\python.exe -m pip install -e .[dev]

# テスト
.venv\Scripts\python.exe -m pytest

# デモ（合成AHCデータで台帳→レポート→MLflow記録）
.venv\Scripts\python.exe demo\generate_demo.py

# UI（共有ストアを表示）
.venv\Scripts\python.exe -m mlflow ui --backend-store-uri "<store path>"

Default store is %AGENTVIZ_STORE%, or ~\dev\Projects\agent-viz\store if unset.

Phase 1 (implemented)

• agentviz.adapters.ahc — ingestion of our own AHC runner measurement format. from_results_json (results/*.json) and from_experiments_jsonl (1 line = 1 experiment. Broken lines are returned as errors and processing continues; lines with empty metrics are rescued by recomputing from per-seed results; absolute paths from other machines are resolved by file name via results_dir)

• agentviz.adapters.kaggle — from_cv(fold_scores, lb_score=...). Case = fold, LB is lb_score metric. For competitions where the metric is defined as the average of per-label scores, such as macro AUC / macro F1, it has from_per_label(label_scores, label_meta=..., metric_name="macro_auc") (case = label). The primary metric is not called cv_mean to avoid confusing the variance between folds (measurement noise) with the gap between labels (difference in true ability). Only the latter can be acted upon. label_meta goes into the case's meta and becomes material for stratification by label density or positive count.

• agentviz.adapters.trade — from_walkforward(windows, ...). Case = walk-forward window. OOS is oos_score metric, tear sheet HTML is attached via log_trial(artifact_paths=...)

• agentviz.report — added generalization gap scatter plot (auto-displayed when there are 2+ trials with lb_score / oos_score. Treats CV vs LB = IS vs OOS as the same shape)

• agentviz.replay — build_replay(frames, infos, events). Self-contained HTML that generalizes the skeleton of the ahc069 custom replay (seek bar, play, frame advance, ←→ keys, event click jump) to be domain-independent

Confirmed with real data: 1133 trials ingested from AtCoder\ahc\ahc069\experiments.jsonl (1191 lines), per-seed case resolution for 1130 trials (examples/ingest_ahc069.py).

Phase 2 (implemented) — bidirectional

• agentviz.feedback — canonical store for stratified feedback (append-only JSONL, store/feedback.jsonl). add / list / resolve

• agentviz.panel — Gradio panel and MCP server. Humans view the trial ledger and heatmap, post stratified feedback (target trial, target case, instruction, priority); agents read it via MCP tools, act on it, and close it with resolve_feedback

# パネル起動（http://127.0.0.1:7861、ポートは AGENTVIZ_PANEL_PORT で変更）
.venv\Scripts\python.exe -m agentviz.panel

# Claude Code への登録（パネル起動中に）
claude mcp add --transport http agentviz http://127.0.0.1:7861/gradio_api/mcp/

Even without MCP, you can read/write directly via gradio_client or agentviz.feedback.FeedbackStore. In environments where UI dropdown selections are not reflected, the "Reload" button is a reliable fallback.

Phase 3 (implemented) — decision points

While feedback handles a single round-trip of issue → response ("this stratum is weak, fix it"), decisions handles points of contention where you cannot proceed until a choice is made. They are separated because their shapes differ.

• agentviz.decisions — store for decision points (append-only JSONL, store/decisions.jsonl). propose / decide / supersede

• Options have a measured flag. If unmeasured options cannot be displayed alongside measured ones, "best among measured" is misread as "best"

• Dependencies between decisions are held via blocks. ready() returns only those whose dependencies have been resolved

• Each option points to ledger trials via evidence_trials

Division of canonical sources: the description of finalized decisions is canonical in the KnowledgeBase Vault. decisions holds only the working surface (options, evidence links, status) and points to the Vault side via vault_ref. The same text is not kept in both.

decide is the mouth for recording human judgment. The agent lays out options (propose_decision), and the human chooses. chosen is limited to registered keys; free-form text is not accepted (because it would become impossible to trace mechanically later). revise_option revises only the state of the evidence (evidence_trials / measured / note) with a reason (the evidence at registration time always remains in history as an event).

Aligning viewpoints (implemented) — agent → human

feedback is human → agent, decisions is the ledger of points. One more direction was missing: a way to align, on the human's screen, which comparison the agent is currently looking at when it speaks.

Even if the agent says "when narrowed to 9 labels, it's 8 wins and 1 loss," the numbers won't match if the human is looking at a different comparison. Re-explaining the conditions in words is a game of telephone, and in this round-trip the ambiguity of "did you exclude them or look at all of them" actually arose.

• agentviz.viewstate — store for pointing (append-only JSONL, store/viewstate.jsonl). point / clear / current / history

• MCP tool point_at_comparison — sends the baseline, candidate, and cases excluded from aggregation to the panel, and returns the numbers for that comparison in the same call (fetching them separately can lead to discrepancies). note is required. A screen change without a reason is, from the human's perspective, nothing more than "it changed on its own"

• MCP tool clear_comparison_pointer / the panel's "Clear pointer" button

It does not rewrite the ledger or decisions. It is not an observation or a judgment, but a pointer to align viewpoints. Not silently overwriting the human's choice is the key design point; when applying, the panel always shows "who specified what, when, and for what purpose," and provides a way to clear it.

Decision views (implemented)

Because "you cannot decide from the average ranking table alone" repeatedly emerged in practice, the skeleton of decision-making was componentized. Everything is provided in two faces: human = figure / agent = JSON.

• Paired difference paired_diff — per-case difference between two trials. Warns when the sign of the mean and the case majority disagree (if they disagree, you cannot claim a ranking. This has fired multiple times on real data). cases can limit the aggregation to a subset. This is the mouth for not mixing cases whose conditions are not the same across both trials into the mean; in RSNA, the differences of the 3 labels without gradients were noise, yet they diluted the mean of 12 labels, and the mean -0.040 vs median -0.104 differed by 3x. Excluded cases always go into excluded_cases and remain in the figure in gray rather than being removed (if removed, readers cannot distinguish whether a convenient subset was chosen or cases with different conditions were excluded). The panel also has a selection field for "cases to exclude from aggregation (multiple allowed)"; selecting updates the figure and statistics on the spot (same function as cases in MCP compare_trials and the third element of pairs in build_report)

• Stratified means strata_means — shows "the ranking flips in this stratum". The definition of strata (domain knowledge) is held by the caller

• Decision leverage decision_leverage — which decision to settle first. Two premises (1 decision = 1 factor, only live options) are always included as premises

• Per-case details case_scores / dot strip plot — passively brings into view, as sort order, the "absolute difficulty of cases" that disappears in the relative heatmap

• Oracle headroom headroom — for constraint-relaxation ideas (e.g., scheduled sampling), measure the upper bound with an oracle run before implementation. Include in premises: oracle not adoptable, it is an upper bound, and if below threshold, systematically shelve

• Generalization gap scatter plot — auto-displayed when there are 2+ trials with lb_score / oos_score

Operational components

• Trial archive set_archived / archive_trial — reversibly hides trials that have been settled as phases progress, preserving visualization resolution (does not delete; history remains in MLflow)

• Dark mode — reports support prefers-color-scheme (Plotly figures follow via relayout)

• 17 panel MCP tools (13 read + writes are add_feedback family, decide / archive_trial, point_at_comparison / clear_comparison_pointer. The last two do not change the ledger; they only move the comparison the human's screen is viewing)

Real-world application examples (case studies)

• kaggle-store-sales-workflow — time-series validation design. Bookkept 12 decisions as decision points, running split design, baseline, features, and adoption decisions all as "measure then decide". Up to LB transfer analysis of CV improvements

• kaggle-house-prices-workflow — nested-CV model selection. First application where the per-case heatmap detected a stratified reversal hidden behind the average ranking

• rsna-knee-abnormality-detection — weakly supervised 12-label classification (macro ROC-AUC). First application of from_per_label. When labels are stratified by label density, the dense 8 labels are 0.751 while the 4 labels with sparse supervision are 0.525, revealing that 1/3 of the metric is effectively unlearned beneath the average of 0.6807. Subsequent comparisons also showed the need to restrict the aggregation of paired differences (across all 12 labels the mean is -0.040, but narrowed to labels with gradients, the effect size is -0.090. If adoption had been decided by the mean alone, it would have been misjudged by more than half)

• examples/ingest_ahc069.py — ingestion of 1133 trials from the AHC custom runner's measurement logs

Roadmap

• Viewing the residual correlation matrix (hand-built for blend diversity decisions; candidate for componentization)

• Named stratum store (persisting the human's "pointing")

• run alias (referencing the same measurement from multiple decision contexts; from the lesson that trial reuse broke visibility)

• pahcer format adapter (to be added when real output becomes available)

• Report size optimization (about 4.9MB per report with Plotly embedded; measured to not hinder agent reading)