# echo-agent

An LLM agent that analyses echocardiography cases with tools. It segments the heart, measures left-ventricle volumes, computes the **ejection fraction**, checks whether the segmentation can be trusted using the model's **uncertainty**, and writes a short report in which every number is checked against the tool outputs.

The segmentation tool is the U-Net from [cardioseg](https://github.com/zubairzulfiqar/cardioseg). The tools are also exposed over the **Model Context Protocol (MCP)**, so any MCP client can call them.

![good vs shadowed case](docs/good_vs_shadow.png)
*Top: a clean image, segmented reliably. Bottom: an acoustic shadow hides part of the wall. The segmentation drifts, the uncertainty lights up along that border, and the agent marks the measurement as unreliable.*

## Example

```text
$ echoagent run p001 --cases cases --show-tools
> check_quality({"case_id": "p001", "phase": "ED"})
  {"quality": "poor", "uncertain_border_fraction": 0.137, "reasons": ["14% of the LV border is uncertain ..."]}
> check_quality({"case_id": "p001", "phase": "ES"})
  {"quality": "poor", "uncertain_border_fraction": 0.117, ...}
> compute_ejection_fraction({"case_id": "p001"})
  {"edv_ml": 119.4, "esv_ml": 60.6, "ef_percent": 49.2, "category": "mildly reduced (41-49%)"}
> save_overlay({"case_id": "p001", "phase": "ED"})

# Echo report: case p001
## Measurements
- End-diastolic volume: 119.4 mL
- End-systolic volume: 60.6 mL
- Ejection fraction: 49.2% (mildly reduced (41-49%))
## Image quality
- ED: poor (14% of the LV border is uncertain (threshold 5%); possible acoustic shadow or dropout)
- ES: poor (12% of the LV border is uncertain (threshold 5%); possible acoustic shadow or dropout)
## Conclusion
The segmentation is unreliable on ED and ES, so the ejection fraction above should not be used
without review by a cardiologist or a repeat acquisition.
```

## How it works

```text
request ─► LLM ──tool calls──► tools ──results──┐
            ▲                                    │
            └────────────────────────────────────┘
            │ no more calls
            ▼
         report ─► grounding check ─► one rewrite if needed ─► report + JSONL trace
```

| Tool | What it does |
|---|---|
| `list_cases` | cases available |
| `check_quality` | share of the LV border where the model's uncertainty (normalised entropy) is above 0.5; flags `poor` above 5% or if the cavity is split |
| `measure_lv` | LV area, long-axis length (principal axis), volume by the single-plane **area-length** method, V = 8A² / (3πL) |
| `compute_ejection_fraction` | EDV, ESV, EF = (EDV − ESV) / EDV, and heart-failure category (reduced ≤40%, mildly reduced 41–49%, preserved ≥50%) |
| `save_overlay` | PNG of the contours and the uncertainty map |

**Safety rails**
- **Grounding check:** every number in the final report must appear in some tool result. Otherwise the model is asked to rewrite once, and if it still invents numbers, the report is returned flagged `grounded=False`.
- **Error handling:** tool arguments are validated against the JSON Schema, and errors go back to the model as data so it can fix its call.
- **Step limit**, so a confused model can't loop forever.
- **Audit trail:** every step is written to a JSONL trace.

**Models and backends**
- **LLM:** any OpenAI-compatible API with tool calling (OpenAI, Mistral, Groq, or a local Ollama model). An offline **scripted planner** follows the same loop, so everything runs and is tested without a key.
- **Segmentation:** a trained cardioseg checkpoint with Monte-Carlo dropout, or an **oracle** backend that simulates a model from the ground-truth masks. The oracle places its errors and uncertainty inside acoustic shadows, which gives known answers to evaluate against.

## Quick start

```bash
git clone https://github.com/zubairzulfiqar/echo-agent && cd echo-agent
pip install -e ".[dev]"
pytest -q

echoagent make-cases --out cases --n 20      # synthetic ED/ES pairs with known EF
echoagent run p001 --cases cases --show-tools
echoagent eval --n 30
```

**With a real LLM:**

```bash
# local: ollama pull qwen2.5:7b
export ECHOAGENT_LLM_BASE_URL=http://localhost:11434/v1 ECHOAGENT_LLM_MODEL=qwen2.5:7b ECHOAGENT_LLM_API_KEY=ollama
echoagent run p001 --cases cases --llm openai
echoagent eval --n 30 --llm openai
```

**With the trained U-Net** (see cardioseg for training):

```bash
pip install torch && pip install -e ".[cardioseg]"
echoagent run p001 --cases cases --backend cardioseg --checkpoint path/to/best.pt
```

**As an MCP server:** register this command as a stdio server in your MCP client.

```bash
echoagent mcp --cases /absolute/path/to/cases
```

## Evaluation

`echoagent eval` runs the agent on synthetic cases and scores what it did against ground truth. EF error is measured against the EF computed from the true masks.

Offline scripted planner with the oracle backend, 30 cases, 9 with an acoustic shadow (seed 123):

| Metric | Good images | Poor images |
|---|---|---|
| EF absolute error (points) | 0.0 | 11.5 |
| Correct HF category | 100% | 56% |

- Poor-quality flag: recall 100%, precision 100%
- Warning written in every flagged report: 100%
- Reports fully grounded in tool results: 100%
- Tool calls per case: 4.0

**How to read this:** the oracle is perfect on clean images by construction, so the 0.0 only shows that the measurement pipeline is exact. The useful result is on the poor images. The EF is off by 11.5 points on average and the category is wrong almost half the time, yet the agent flags 100% of those cases and warns in the report. It doesn't hide the unreliable measurement. Running `eval --llm openai` measures the same things for a real LLM (whether it calls the quality check, stays grounded, and warns).

The tests include a model that deliberately invents a number (`HallucinatingLLM`) to check that the grounding rewrite catches it.

## Project layout

```text
echoagent/
  cases.py       synthetic ED/ES cases with known EF, case storage
  measure.py     LV area, long axis, area-length volume, EF and category
  backends.py    oracle and cardioseg (U-Net + MC dropout) segmentation
  tools.py       the tools, JSON schemas, argument validation
  llm.py         OpenAI-compatible tool-calling client, offline scripted planner
  agent.py       loop, grounding check, step limit, JSONL trace
  mcp_server.py  MCP server over stdio (JSON-RPC 2.0), no SDK needed
  evaluate.py    end-to-end evaluation against ground truth
  cli.py         make-cases | run | eval | mcp
tests/           22 tests (one needs PyTorch + cardioseg and runs in CI)
```

## Limitations

- Single-plane area-length from the 4-chamber view only. Clinical practice prefers biplane Simpson's method with the 2-chamber view as well.
- The quality threshold (5% of the border uncertain) is tuned on the oracle backend and has to be re-calibrated for a real model on real data.
- Synthetic cases are a test bed, not clinical evidence.
- This is a research prototype, not a medical device.

## License

MIT