Model Card Chat

An MCP server that turns the Hugging Face Hub into a conversational model picker — with a tool-use eval harness to prove the agent actually uses it correctly.

▶ Try it live — no signup, runs in your browser

Ask "what's a good small model for sentiment analysis?" and the agent searches the Hub live, compares candidates on size, license and published benchmark scores, and recommends one — grounded entirely in tool output, never in the model's memory.

Why this exists

Choosing a model on Hugging Face means opening a dozen tabs and manually reconciling parameter counts, licenses and eval numbers. This exposes that workflow as five tools an LLM agent can call, and — the part that makes it engineering rather than a demo — measures whether the agent picks the right tool, passes the right arguments, and stays grounded.

Related MCP server: Hugging Face MCP Server

Architecture

                    ┌──────────────────────────────────┐
  Claude Desktop ──▶│  server.py      (MCP / stdio)    │
  LangGraph agent ─▶│  langchain_tools.py              │──┐
  LlamaIndex agent ▶│  llamaindex_tools.py             │  │
                    └──────────────────────────────────┘  │
                                                          ▼
                                            ┌──────────────────────────┐
                                            │  core.py — tool registry │
                                            │  one implementation,     │
                                            │  three bindings          │
                                            └───────────┬──────────────┘
                                                        │
                  ┌─────────────────┬───────────────────┼──────────────────┐
                  ▼                 ▼                   ▼                  ▼
          ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐
          │ hf_client.py │  │formatting.py │  │benchmarks.py │  │    rag.py    │
          │ cache, retry │  │   context    │  │ model-index  │  │ chunk + BM25 │
          │ rate limits  │  │ engineering  │  │   parsing    │  │  / dense     │
          └──────────────┘  └──────────────┘  └──────────────┘  └──────────────┘

The tool logic lives once in core.py; MCP, LangChain and LlamaIndex are thin bindings over it. A tool description tuned against the eval suite improves every binding at once.

Tools

Engineering highlights

1. Context engineering — 90% fewer tokens per tool call

Tool results are consumed by an LLM, not a browser. formatting.py converts raw API payloads into the minimum text that still supports a correct recommendation, dropping fields no decision depends on (siblings, spaces, base64 verifyToken blobs) and routing-noise tags like region:us.

Measured against live payloads with tiktoken/cl100k_base:

Sample                                            raw   pruned  formatted   vs raw
-------------------------------------------------------------------------------
distilbert-base-uncased-finetuned-sst-2-english  7778     7754        256    96.7%
search(whisper, automatic-speech-recognition)    4095     3849        293    92.8%
search(sentiment, text-classification)           2028     1795        319    84.3%
-------------------------------------------------------------------------------
TOTAL                                           18016    17167       1805    90.0%

Mean tokens per tool call: 226

Reproduce with python scripts/measure_context_savings.py. Note raw is already the expand[]-narrowed response, so this understates the saving versus a naive wrapper.

2. Benchmark-aware selection

Most Hub wrappers rank by downloads — a popularity contest. This parses the model-index block for published evaluation results, and distinguishes HF-verified metrics from self-reported ones (marked *), because self-reported numbers aren't comparable across authors:

Benchmarks (published in model card):
  - glue (sst2): Accuracy 91.06%, Precision 89.78%, Recall 93.02%, AUC 97.17%

Metrics where lower is better (WER, perplexity, MAE) are recognised so comparisons don't rank a high error rate as the winner.

3. RAG over model cards

Model card READMEs run 5–40 KB — too large to paste into context to answer one question. Cards are chunked on Markdown structure (so each chunk is topically coherent and carries its heading as a citation), then retrieved.

Two design decisions worth noting:

• Headings are indexed with their body. A question about "limitations" often shares no vocabulary with the prose underneath, matching only the heading. Indexing headings separately was a real bug caught by the test suite.

• Fenced code blocks are tracked, so # Load the model inside a Python example isn't parsed as a section heading.

BM25 is the default (pure Python, no dependencies, no model download). Install .[rag] to switch to dense embeddings for paraphrase-style questions.

4. Production concerns

• TTL cache — compare_models and follow-up turns re-request the same models constantly; caching keeps the anonymous rate limit (500 req / 5 min) out of reach.

• Retry with backoff on 429s and transient network errors.

• Tools never raise. A raised exception breaks the agent loop; a returned message lets the model recover and explain.

• Correct error semantics — the Hub returns 401, not 404, for missing models (so private repo names can't be probed).

Evaluation

Building a tool server is half the job. The half that matters is whether an LLM uses it correctly. evals/ measures three things across 22 cases in five groups, programmatically — no LLM-as-judge, so runs are deterministic and cheap:

Case groups: routing (the floor), disambiguation (two tools plausibly apply), parameters (arguments are the hard part), grounding (tempting the model to answer from memory), robustness (bad IDs, vague asks).

The harness is provider-agnostic, so the same cases and the same scoring run against any backend — including free ones. Credentials load from a gitignored .env.

python evals/run_evals.py --list-providers
python evals/run_evals.py --provider github --output evals/results-gpt4o.json   # free
python evals/run_evals.py --provider anthropic --output evals/results-claude.json

Results

22 cases. Raw outputs are committed in evals/ — every number below is reproducible from them.

Tool-selection accuracy by group:

What tuning the tool descriptions actually changed

The descriptions in core.TOOLS were rewritten against measured failures — each negative clause ("do not call get_model_card repeatedly", "do not use this for license questions") traces to a specific case that failed.

GPT-4o-mini improved substantially: tool-selection 78.9% → 94.7%, parameter accuracy 87.5% → 100%, cases fully passed 81.8% → 90.9%.

Claude did not improve — it was already at the ceiling these cases can measure (94.7%), and finished exactly where it started. Reporting that honestly matters more than the headline: prompt tuning has diminishing returns against a model that is already choosing correctly, and the remaining 5.3% is one genuinely ambiguous case rather than a fixable defect.

One intermediate version actively regressed Claude, 94.7% → 89.5%. A clause added to ask_about_model to fix a GPT-4o failure — "call get_model_card first if the model may not exist" — caused Claude to run a metadata lookup before every prose question, breaking two disambiguation cases that previously passed. Narrowing the condition recovered it and lifted disambiguation 60% → 80%.

The lesson worth taking from this: tool-description changes are not monotonic across models. A clause that repairs a weaker model's behaviour can degrade a stronger one's, and without a per-model eval the regression is invisible.

Three bugs the evals found

1. A false-positive groundedness metric. The first run scored Claude at 20% groundedness, which read as a serious hallucination problem. It wasn't: model IDs were extracted with a naive word/word regex, so ordinary prose — positive/negative, size/hardware, POS/NEG — counted as fabricated IDs. The metric punished whichever model wrote the most explanatory prose. Claude's true score was 100%. A metric that makes a good model look bad is more dangerous than no metric, because it reads as a finding.

2. Scoring that punished good behaviour. forbid_tool originally meant "never call this tool". But calling the right tool first and then fetching supporting detail is good agent behaviour, and it was being marked wrong. It now means "must not be called instead of the expected tool" — scored on order, not presence. This moved Claude 89.5% → 94.7% and left both GPT models unchanged, confirming it corrected a specific mis-scoring rather than inflating everything.

3. A real crash in the tool layer. gpt-4o-mini sent limit="10" as a string, and min(limit, MAX) raised TypeError comparing str to int. The agent saw tool crashes, gave up, and returned an empty answer — surfacing as a groundedness failure. Tool arguments come from a language model, not a type checker, so core.py now coerces them; compare_models likewise accepts a comma-separated string where a list was specified. This bug was reachable from Claude Desktop and no unit test would have found it.

Keeping the eval honest

core.TOOLS is the single source of truth for tool and parameter descriptions. It was not always: the eval scored one set of strings while the MCP server advertised its docstrings, so tuning against the evals changed nothing in production. The MCP server now takes both from the registry (via Field annotations, since the SDK does not surface docstring Args: blocks in the schema), the eval derives its schemas from the same place, and tests/test_core.py asserts they cannot drift apart.

evals/rescore.py re-applies current scoring to past runs by replaying their recorded tool calls, so fixing a metric costs nothing and old numbers stay comparable:

python evals/rescore.py evals/*.json --write

Caveat worth stating: n=22, with 3–5 cases per group, so one case moves a group by 20–33 points. Group-level numbers are directional, not statistically solid.

Setup

git clone https://github.com/farnoosh-afshinrad/model-card-chat-mcp.git
cd model-card-chat-mcp
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest

Claude Desktop

Add to claude_desktop_config.json (macOS: ~/Library/Application Support/Claude/, Windows: %APPDATA%\Claude\):

{
  "mcpServers": {
    "model-card-chat": {
      "command": "/absolute/path/to/model-card-chat-mcp/.venv/bin/python3",
      "args": ["-m", "model_card_chat.server"]
    }
  }
}

Restart Claude Desktop; five tools should appear. Claude Desktop has no cwd option, which is why pip install -e . matters — it makes the package importable regardless of the launch directory.

Other runtimes

mcp dev model_card_chat/server.py            # MCP Inspector UI
python scripts/smoke_test.py                 # verify tools are advertised
python -m model_card_chat.server \
  --transport streamable-http --host 0.0.0.0 --port 7860   # as a service

Web UI

static/ is a zero-dependency page over the same five tools — the shape anyone can actually try, since an MCP endpoint needs a client rather than a browser. It calls the Hugging Face API straight from the browser (the Hub allows CORS), so there is no backend, no API key and no cold start.

python -m http.server 8000 --directory static     # http://localhost:8000

Deployed at huggingface.co/spaces/Farnooshrad/model-card-chat. Static Spaces are free, whereas Gradio and Docker Spaces now require PRO:

pip install huggingface_hub && hf auth login
python deploy/push_to_space.py

Docker

docker build -t model-card-chat .
docker run -p 7860:7860 model-card-chat

The image runs the streamable-HTTP transport bound to 0.0.0.0 — the FastMCP default of 127.0.0.1 would start healthily and still be unreachable from outside the container. CI builds the image and asserts it answers an MCP handshake, so that failure mode can't ship silently.

See deploy/README.md for free hosting (Hugging Face Spaces, Render, Fly.io) and a note on why a public MCP endpoint is a weaker portfolio asset than the demo video.

As LangChain / LlamaIndex tools

from langgraph.prebuilt import create_react_agent
from langchain.chat_models import init_chat_model
from model_card_chat.integrations.langchain_tools import get_langchain_tools

agent = create_react_agent(
    init_chat_model("claude-sonnet-5", model_provider="anthropic"),
    get_langchain_tools(),
)

Try these prompts

• "What's a good small model for sentiment analysis?"

• "Compare bert-base-uncased, distilbert-base-uncased and roberta-base."

• "What's trending for text-to-image right now?"

• "What data was openai/whisper-large-v3 trained on?" (hits the RAG path)

• "What are the known limitations of Whisper?" (retrieves the limitations section)

Project layout

static/             # zero-dependency web UI (Hugging Face Space)
model_card_chat/
  core.py           # tool implementations + registry (single source of truth)
  server.py         # MCP server; tool docstrings are the prompt
  hf_client.py      # HTTP: caching, retries, rate limits
  formatting.py     # context engineering
  benchmarks.py     # model-index parsing
  rag.py            # chunking + BM25 / dense retrieval
  integrations/     # LangChain + LlamaIndex bindings
evals/              # tool-use eval harness + cases
tests/              # 58 tests, fully mocked
scripts/            # token measurement, MCP smoke test

Optional extras

Roadmap

• Eval results tracked over time in CI

• Re-run the GPT-4o column once the free quota resets

• Dataset and Spaces tools

License

MIT