# Model Card Chat

**An MCP server that turns the Hugging Face Hub into a conversational model picker — with a tool-use eval harness to prove the agent actually uses it correctly.**

[![CI](https://github.com/farnoosh-afshinrad/model-card-chat-mcp/actions/workflows/ci.yml/badge.svg)](https://github.com/farnoosh-afshinrad/model-card-chat-mcp/actions/workflows/ci.yml)
[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue)](https://www.python.org/)
[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)
[![Live demo](https://img.shields.io/badge/%F0%9F%A4%97%20live%20demo-try%20it-ff9d00)](https://huggingface.co/spaces/Farnooshrad/model-card-chat)

### ▶ [Try it live — no signup, runs in your browser](https://huggingface.co/spaces/Farnooshrad/model-card-chat)

Ask *"what's a good small model for sentiment analysis?"* and the agent searches the Hub live, compares candidates on **size, license and published benchmark scores**, and recommends one — grounded entirely in tool output, never in the model's memory.

<!-- Add your demo GIF here — recruiters skim, so lead visually.
![demo](docs/demo.gif) -->

---

## Why this exists

Choosing a model on Hugging Face means opening a dozen tabs and manually reconciling parameter counts, licenses and eval numbers. This exposes that workflow as five tools an LLM agent can call, and — the part that makes it engineering rather than a demo — **measures whether the agent picks the right tool, passes the right arguments, and stays grounded.**

## Architecture

```
                    ┌──────────────────────────────────┐
  Claude Desktop ──▶│  server.py      (MCP / stdio)    │
  LangGraph agent ─▶│  langchain_tools.py              │──┐
  LlamaIndex agent ▶│  llamaindex_tools.py             │  │
                    └──────────────────────────────────┘  │
                                                          ▼
                                            ┌──────────────────────────┐
                                            │  core.py — tool registry │
                                            │  one implementation,     │
                                            │  three bindings          │
                                            └───────────┬──────────────┘
                                                        │
                  ┌─────────────────┬───────────────────┼──────────────────┐
                  ▼                 ▼                   ▼                  ▼
          ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐
          │ hf_client.py │  │formatting.py │  │benchmarks.py │  │    rag.py    │
          │ cache, retry │  │   context    │  │ model-index  │  │ chunk + BM25 │
          │ rate limits  │  │ engineering  │  │   parsing    │  │  / dense     │
          └──────────────┘  └──────────────┘  └──────────────┘  └──────────────┘
```

The tool logic lives once in `core.py`; MCP, LangChain and LlamaIndex are thin bindings over it. A tool description tuned against the eval suite improves every binding at once.

## Tools

| Tool | Answers |
|---|---|
| `search_models(query, task, limit)` | "What models exist for X?" |
| `get_model_card(model_id)` | Structured facts: license, size, downloads, **benchmarks** |
| `compare_models(model_ids)` | Side-by-side table of 2–6 models |
| `get_trending_models(task, limit)` | "What's popular right now?" |
| `ask_about_model(model_id, question)` | **RAG** over the written card: training data, limitations, intended use |

## Engineering highlights

### 1. Context engineering — 90% fewer tokens per tool call

Tool results are consumed by an LLM, not a browser. `formatting.py` converts raw API payloads into the minimum text that still supports a correct recommendation, dropping fields no decision depends on (`siblings`, `spaces`, base64 `verifyToken` blobs) and routing-noise tags like `region:us`.

Measured against live payloads with `tiktoken/cl100k_base`:

```
Sample                                            raw   pruned  formatted   vs raw
-------------------------------------------------------------------------------
distilbert-base-uncased-finetuned-sst-2-english  7778     7754        256    96.7%
search(whisper, automatic-speech-recognition)    4095     3849        293    92.8%
search(sentiment, text-classification)           2028     1795        319    84.3%
-------------------------------------------------------------------------------
TOTAL                                           18016    17167       1805    90.0%

Mean tokens per tool call: 226
```

Reproduce with `python scripts/measure_context_savings.py`. Note `raw` is already the `expand[]`-narrowed response, so this understates the saving versus a naive wrapper.

### 2. Benchmark-aware selection

Most Hub wrappers rank by downloads — a popularity contest. This parses the `model-index` block for published evaluation results, and distinguishes **HF-verified** metrics from **self-reported** ones (marked `*`), because self-reported numbers aren't comparable across authors:

```
Benchmarks (published in model card):
  - glue (sst2): Accuracy 91.06%, Precision 89.78%, Recall 93.02%, AUC 97.17%
```

Metrics where lower is better (WER, perplexity, MAE) are recognised so comparisons don't rank a high error rate as the winner.

### 3. RAG over model cards

Model card READMEs run 5–40 KB — too large to paste into context to answer one question. Cards are chunked on Markdown structure (so each chunk is topically coherent and carries its heading as a citation), then retrieved.

Two design decisions worth noting:

- **Headings are indexed with their body.** A question about "limitations" often shares no vocabulary with the prose underneath, matching only the heading. Indexing headings separately was a real bug caught by the test suite.
- **Fenced code blocks are tracked**, so `# Load the model` inside a Python example isn't parsed as a section heading.

BM25 is the default (pure Python, no dependencies, no model download). Install `.[rag]` to switch to dense embeddings for paraphrase-style questions.

### 4. Production concerns

- **TTL cache** — `compare_models` and follow-up turns re-request the same models constantly; caching keeps the anonymous rate limit (500 req / 5 min) out of reach.
- **Retry with backoff** on 429s and transient network errors.
- **Tools never raise.** A raised exception breaks the agent loop; a returned message lets the model recover and explain.
- **Correct error semantics** — the Hub returns `401`, not `404`, for missing models (so private repo names can't be probed).

## Evaluation

Building a tool server is half the job. The half that matters is whether an LLM uses it correctly. `evals/` measures three things across 22 cases in five groups, **programmatically — no LLM-as-judge**, so runs are deterministic and cheap:

| Metric | What it catches |
|---|---|
| **Tool-selection accuracy** | Right tool first; forbidden tools avoided |
| **Parameter accuracy** | Correct Hub task tag (models love inventing `sentiment-analysis` for `text-classification`) |
| **Groundedness** | Every model ID in the answer appeared in a tool result — anything else is a hallucination |

Case groups: `routing` (the floor), `disambiguation` (two tools plausibly apply), `parameters` (arguments are the hard part), `grounding` (tempting the model to answer from memory), `robustness` (bad IDs, vague asks).

The harness is provider-agnostic, so the same cases and the same scoring run against any backend — including free ones. Credentials load from a gitignored `.env`.

```bash
python evals/run_evals.py --list-providers
python evals/run_evals.py --provider github --output evals/results-gpt4o.json   # free
python evals/run_evals.py --provider anthropic --output evals/results-claude.json
```

### Results

22 cases. Raw outputs are committed in `evals/` — every number below is reproducible from them.

| Metric | Claude Sonnet 5 | GPT-4o-mini |
|---|---|---|
| Tool-selection accuracy | 94.7% | 94.7% |
| Parameter accuracy | 100% | 100% |
| Groundedness | 100% | 100% |
| Cases fully passed | **95.5%** | 90.9% |

Tool-selection accuracy by group:

| Group | Claude Sonnet 5 | GPT-4o-mini |
|---|---|---|
| routing | 100% | 100% |
| disambiguation | 80% | 80% |
| parameters | 100% | 100% |
| grounding | 100% | 100% |
| robustness | 100% | 100% |


### What tuning the tool descriptions actually changed

The descriptions in `core.TOOLS` were rewritten against measured failures — each
negative clause ("do not call get_model_card repeatedly", "do not use this for
license questions") traces to a specific case that failed.

**GPT-4o-mini improved substantially:** tool-selection 78.9% → 94.7%, parameter
accuracy 87.5% → 100%, cases fully passed 81.8% → 90.9%.

**Claude did not improve — it was already at the ceiling** these cases can
measure (94.7%), and finished exactly where it started. Reporting that honestly
matters more than the headline: prompt tuning has diminishing returns against a
model that is already choosing correctly, and the remaining 5.3% is one
genuinely ambiguous case rather than a fixable defect.

**One intermediate version actively regressed Claude, 94.7% → 89.5%.** A clause
added to `ask_about_model` to fix a GPT-4o failure — *"call get_model_card first
if the model may not exist"* — caused Claude to run a metadata lookup before
*every* prose question, breaking two disambiguation cases that previously
passed. Narrowing the condition recovered it and lifted disambiguation 60% →
80%.

The lesson worth taking from this: **tool-description changes are not monotonic
across models.** A clause that repairs a weaker model's behaviour can degrade a
stronger one's, and without a per-model eval the regression is invisible.

### Three bugs the evals found

**1. A false-positive groundedness metric.** The first run scored Claude at 20%
groundedness, which read as a serious hallucination problem. It wasn't: model
IDs were extracted with a naive `word/word` regex, so ordinary prose —
`positive/negative`, `size/hardware`, `POS/NEG` — counted as fabricated IDs. The
metric punished whichever model wrote the most explanatory prose. Claude's true
score was 100%. A metric that makes a good model look bad is more dangerous than
no metric, because it reads as a finding.

**2. Scoring that punished good behaviour.** `forbid_tool` originally meant
"never call this tool". But calling the *right* tool first and then fetching
supporting detail is good agent behaviour, and it was being marked wrong. It now
means "must not be called *instead of* the expected tool" — scored on order, not
presence. This moved Claude 89.5% → 94.7% and left both GPT models unchanged,
confirming it corrected a specific mis-scoring rather than inflating everything.

**3. A real crash in the tool layer.** `gpt-4o-mini` sent `limit="10"` as a
string, and `min(limit, MAX)` raised `TypeError` comparing `str` to `int`. The
agent saw tool crashes, gave up, and returned an empty answer — surfacing as a
groundedness failure. Tool arguments come from a language model, not a type
checker, so `core.py` now coerces them; `compare_models` likewise accepts a
comma-separated string where a list was specified. This bug was reachable from
Claude Desktop and no unit test would have found it.

### Keeping the eval honest

`core.TOOLS` is the single source of truth for tool *and* parameter
descriptions. It was not always: the eval scored one set of strings while the
MCP server advertised its docstrings, so tuning against the evals changed
nothing in production. The MCP server now takes both from the registry (via
`Field` annotations, since the SDK does not surface docstring `Args:` blocks in
the schema), the eval derives its schemas from the same place, and
`tests/test_core.py` asserts they cannot drift apart.

`evals/rescore.py` re-applies current scoring to past runs by replaying their
recorded tool calls, so fixing a metric costs nothing and old numbers stay
comparable:

```bash
python evals/rescore.py evals/*.json --write
```

**Caveat worth stating:** n=22, with 3–5 cases per group, so one case moves a
group by 20–33 points. Group-level numbers are directional, not statistically
solid.

## Setup

```bash
git clone https://github.com/farnoosh-afshinrad/model-card-chat-mcp.git
cd model-card-chat-mcp
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest
```

### Claude Desktop

Add to `claude_desktop_config.json` (macOS: `~/Library/Application Support/Claude/`, Windows: `%APPDATA%\Claude\`):

```json
{
  "mcpServers": {
    "model-card-chat": {
      "command": "/absolute/path/to/model-card-chat-mcp/.venv/bin/python3",
      "args": ["-m", "model_card_chat.server"]
    }
  }
}
```

Restart Claude Desktop; five tools should appear. Claude Desktop has **no `cwd` option**, which is why `pip install -e .` matters — it makes the package importable regardless of the launch directory.

### Other runtimes

```bash
mcp dev model_card_chat/server.py            # MCP Inspector UI
python scripts/smoke_test.py                 # verify tools are advertised
python -m model_card_chat.server \
  --transport streamable-http --host 0.0.0.0 --port 7860   # as a service
```

### Web UI

`static/` is a zero-dependency page over the same five tools — the shape anyone
can actually try, since an MCP endpoint needs a client rather than a browser.
It calls the Hugging Face API straight from the browser (the Hub allows CORS),
so there is no backend, no API key and no cold start.

```bash
python -m http.server 8000 --directory static     # http://localhost:8000
```

Deployed at
**[huggingface.co/spaces/Farnooshrad/model-card-chat](https://huggingface.co/spaces/Farnooshrad/model-card-chat)**.
Static Spaces are free, whereas Gradio and Docker Spaces now require PRO:

```bash
pip install huggingface_hub && hf auth login
python deploy/push_to_space.py
```

### Docker

```bash
docker build -t model-card-chat .
docker run -p 7860:7860 model-card-chat
```

The image runs the streamable-HTTP transport bound to `0.0.0.0` — the FastMCP
default of `127.0.0.1` would start healthily and still be unreachable from
outside the container. CI builds the image and asserts it answers an MCP
handshake, so that failure mode can't ship silently.

See [deploy/README.md](deploy/README.md) for free hosting (Hugging Face Spaces,
Render, Fly.io) and a note on why a public MCP endpoint is a weaker portfolio
asset than the demo video.

### As LangChain / LlamaIndex tools

```python
from langgraph.prebuilt import create_react_agent
from langchain.chat_models import init_chat_model
from model_card_chat.integrations.langchain_tools import get_langchain_tools

agent = create_react_agent(
    init_chat_model("claude-sonnet-5", model_provider="anthropic"),
    get_langchain_tools(),
)
```

## Try these prompts

- "What's a good small model for sentiment analysis?"
- "Compare bert-base-uncased, distilbert-base-uncased and roberta-base."
- "What's trending for text-to-image right now?"
- "What data was `openai/whisper-large-v3` trained on?" *(hits the RAG path)*
- "What are the known limitations of Whisper?" *(retrieves the limitations section)*

## Project layout

```
static/             # zero-dependency web UI (Hugging Face Space)
model_card_chat/
  core.py           # tool implementations + registry (single source of truth)
  server.py         # MCP server; tool docstrings are the prompt
  hf_client.py      # HTTP: caching, retries, rate limits
  formatting.py     # context engineering
  benchmarks.py     # model-index parsing
  rag.py            # chunking + BM25 / dense retrieval
  integrations/     # LangChain + LlamaIndex bindings
evals/              # tool-use eval harness + cases
tests/              # 58 tests, fully mocked
scripts/            # token measurement, MCP smoke test
```

## Optional extras

| Extra | Adds |
|---|---|
| `.[rag]` | Dense embedding retrieval (`sentence-transformers`) |
| `.[evals]` | Eval harness (`anthropic`, `pyyaml`) |
| `.[langchain]` / `.[llamaindex]` | Framework adapters |
| `.[dev]` | pytest, respx, ruff, tiktoken |

## Roadmap

- Eval results tracked over time in CI
- Re-run the GPT-4o column once the free quota resets
- Dataset and Spaces tools

## License

MIT