Model Card Chat

An MCP server that turns the Hugging Face Hub into a conversational model picker — with a tool-use eval harness to prove the agent actually uses it correctly.

Ask "what's a good small model for sentiment analysis?" and the agent searches the Hub live, compares candidates on size, license and published benchmark scores, and recommends one — grounded entirely in tool output, never in the model's memory.

Why this exists

Choosing a model on Hugging Face means opening a dozen tabs and manually reconciling parameter counts, licenses and eval numbers. This exposes that workflow as five tools an LLM agent can call, and — the part that makes it engineering rather than a demo — measures whether the agent picks the right tool, passes the right arguments, and stays grounded.

Related MCP server: Hugging Face MCP Server

Architecture

                    ┌──────────────────────────────────┐
  Claude Desktop ──▶│  server.py      (MCP / stdio)    │
  LangGraph agent ─▶│  langchain_tools.py              │──┐
  LlamaIndex agent ▶│  llamaindex_tools.py             │  │
                    └──────────────────────────────────┘  │
                                                          ▼
                                            ┌──────────────────────────┐
                                            │  core.py — tool registry │
                                            │  one implementation,     │
                                            │  three bindings          │
                                            └───────────┬──────────────┘
                                                        │
                  ┌─────────────────┬───────────────────┼──────────────────┐
                  ▼                 ▼                   ▼                  ▼
          ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐
          │ hf_client.py │  │formatting.py │  │benchmarks.py │  │    rag.py    │
          │ cache, retry │  │   context    │  │ model-index  │  │ chunk + BM25 │
          │ rate limits  │  │ engineering  │  │   parsing    │  │  / dense     │
          └──────────────┘  └──────────────┘  └──────────────┘  └──────────────┘

The tool logic lives once in core.py; MCP, LangChain and LlamaIndex are thin bindings over it. A tool description tuned against the eval suite improves every binding at once.

Tools

Engineering highlights

1. Context engineering — 90% fewer tokens per tool call

Tool results are consumed by an LLM, not a browser. formatting.py converts raw API payloads into the minimum text that still supports a correct recommendation, dropping fields no decision depends on (siblings, spaces, base64 verifyToken blobs) and routing-noise tags like region:us.

Measured against live payloads with tiktoken/cl100k_base:

Sample                                            raw   pruned  formatted   vs raw
-------------------------------------------------------------------------------
distilbert-base-uncased-finetuned-sst-2-english  7778     7754        256    96.7%
search(whisper, automatic-speech-recognition)    4095     3849        293    92.8%
search(sentiment, text-classification)           2028     1795        319    84.3%
-------------------------------------------------------------------------------
TOTAL                                           18016    17167       1805    90.0%

Mean tokens per tool call: 226

Reproduce with python scripts/measure_context_savings.py. Note raw is already the expand[]-narrowed response, so this understates the saving versus a naive wrapper.

2. Benchmark-aware selection

Most Hub wrappers rank by downloads — a popularity contest. This parses the model-index block for published evaluation results, and distinguishes HF-verified metrics from self-reported ones (marked *), because self-reported numbers aren't comparable across authors:

Benchmarks (published in model card):
  - glue (sst2): Accuracy 91.06%, Precision 89.78%, Recall 93.02%, AUC 97.17%

Metrics where lower is better (WER, perplexity, MAE) are recognised so comparisons don't rank a high error rate as the winner.

3. RAG over model cards

Model card READMEs run 5–40 KB — too large to paste into context to answer one question. Cards are chunked on Markdown structure (so each chunk is topically coherent and carries its heading as a citation), then retrieved.

Two design decisions worth noting:

• Headings are indexed with their body. A question about "limitations" often shares no vocabulary with the prose underneath, matching only the heading. Indexing headings separately was a real bug caught by the test suite.

• Fenced code blocks are tracked, so # Load the model inside a Python example isn't parsed as a section heading.

BM25 is the default (pure Python, no dependencies, no model download). Install .[rag] to switch to dense embeddings for paraphrase-style questions.

4. Production concerns

• TTL cache — compare_models and follow-up turns re-request the same models constantly; caching keeps the anonymous rate limit (500 req / 5 min) out of reach.

• Retry with backoff on 429s and transient network errors.

• Tools never raise. A raised exception breaks the agent loop; a returned message lets the model recover and explain.

• Correct error semantics — the Hub returns 401, not 404, for missing models (so private repo names can't be probed).

Evaluation

Building a tool server is half the job. The half that matters is whether an LLM uses it correctly. evals/ measures three things across 22 cases in five groups, programmatically — no LLM-as-judge, so runs are deterministic and cheap:

Case groups: routing (the floor), disambiguation (two tools plausibly apply), parameters (arguments are the hard part), grounding (tempting the model to answer from memory), robustness (bad IDs, vague asks).

The harness is provider-agnostic, so the same cases and the same scoring run against any backend — including free ones. Credentials load from a gitignored .env.

python evals/run_evals.py --list-providers
python evals/run_evals.py --provider github --output evals/results-gpt4o.json   # free
python evals/run_evals.py --provider anthropic --output evals/results-claude.json

Results

22 cases, measured runs (not estimates). Raw outputs are committed in evals/.

Tool-selection accuracy by group — the averages hide where the real weaknesses are:

What the results say

Every model scored 100% on groundedness. No model invented a Hub ID that wasn't in a tool result — which is the property that makes a model-recommender trustworthy at all.

Parameter accuracy validated a design decision. Enumerating the exact Hub task tags inline in the tool description was meant to stop models inventing sentiment-analysis instead of text-classification. Both larger models got 100%; only gpt-4o-mini slipped to 87.5%. The failure mode the enumeration was written to prevent is the one that mostly disappeared.

Disambiguation is the weakest group across all three (60–80%) — deciding between get_model_card (structured facts) and ask_about_model (prose). That boundary is a tool-description problem, and it's where the next round of tuning should go.

GPT-4o's 33% on robustness is the sharpest single finding. Given a misspelled model ID it called no tool at all and answered from memory; given a nonexistent model it reached for the RAG tool instead of the metadata lookup. Claude and even gpt-4o-mini handled both correctly.

A bug the evals found in the evals

The first run scored Claude at 20% groundedness, which looked like a serious hallucination problem. It wasn't. The metric extracted model IDs with a naive word/word regex, so ordinary prose — positive/negative, size/hardware, languages/accents, POS/NEG — was being counted as fabricated model IDs. The metric systematically punished whichever model wrote the most explanatory prose.

Fixed by filtering candidates against a prose-word list and requiring a model-name shape. Claude's groundedness went 20% → 100%; GPT-4o was unaffected, having written tersely enough to dodge the false positives. Those exact strings are now regression tests in tests/test_evals_scoring.py.

evals/rescore.py re-applies current scoring to past runs by replaying their recorded tool calls, so a metric fix costs nothing and old numbers stay comparable:

python evals/rescore.py evals/*.json --write

Worth stating plainly: a metric that makes a good model look bad is more dangerous than no metric, because it reads as a finding.

Setup

git clone https://github.com/farnoosh-afshinrad/model-card-chat-mcp.git
cd model-card-chat-mcp
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest

Claude Desktop

Add to claude_desktop_config.json (macOS: ~/Library/Application Support/Claude/, Windows: %APPDATA%\Claude\):

{
  "mcpServers": {
    "model-card-chat": {
      "command": "/absolute/path/to/model-card-chat-mcp/.venv/bin/python3",
      "args": ["-m", "model_card_chat.server"]
    }
  }
}

Restart Claude Desktop; five tools should appear. Claude Desktop has no cwd option, which is why pip install -e . matters — it makes the package importable regardless of the launch directory.

Other runtimes

mcp dev model_card_chat/server.py            # MCP Inspector UI
python scripts/smoke_test.py                 # verify tools are advertised
python -m model_card_chat.server \
  --transport streamable-http --host 0.0.0.0 --port 7860   # as a service

Docker

docker build -t model-card-chat .
docker run -p 7860:7860 model-card-chat

The image runs the streamable-HTTP transport bound to 0.0.0.0 — the FastMCP default of 127.0.0.1 would start healthily and still be unreachable from outside the container. CI builds the image and asserts it answers an MCP handshake, so that failure mode can't ship silently.

See deploy/README.md for free hosting (Hugging Face Spaces, Render, Fly.io) and a note on why a public MCP endpoint is a weaker portfolio asset than the demo video.

As LangChain / LlamaIndex tools

from langgraph.prebuilt import create_react_agent
from langchain.chat_models import init_chat_model
from model_card_chat.integrations.langchain_tools import get_langchain_tools

agent = create_react_agent(
    init_chat_model("claude-sonnet-5", model_provider="anthropic"),
    get_langchain_tools(),
)

Try these prompts

• "What's a good small model for sentiment analysis?"

• "Compare bert-base-uncased, distilbert-base-uncased and roberta-base."

• "What's trending for text-to-image right now?"

• "What data was openai/whisper-large-v3 trained on?" (hits the RAG path)

• "What are the known limitations of Whisper?" (retrieves the limitations section)

Project layout

model_card_chat/
  core.py           # tool implementations + registry (single source of truth)
  server.py         # MCP server; tool docstrings are the prompt
  hf_client.py      # HTTP: caching, retries, rate limits
  formatting.py     # context engineering
  benchmarks.py     # model-index parsing
  rag.py            # chunking + BM25 / dense retrieval
  integrations/     # LangChain + LlamaIndex bindings
evals/              # tool-use eval harness + cases
tests/              # 58 tests, fully mocked
scripts/            # token measurement, MCP smoke test

Optional extras

Roadmap

• Docker image + deployment of the streamable-HTTP transport

• Eval results tracked over time in CI

• Dataset and Spaces tools

License

MIT