sure-but-wrong

An MCP server that asks an AI model the same 40 questions in English, Urdu script and Roman Urdu, grades every answer, and records how often the model is wrong but sounds sure, broken down by language.

It works with Claude Desktop (or any MCP client) and also ships a command-line tool.

Results: first run (6 October 2026)

Model tested: Google gemini-flash-lite-latest · Judge: openai/gpt-oss-120b (via Groq) · 40 questions × 3 languages = 120 answers, temperature 0, one run.

What stood out

• Every wrong answer was in Urdu or Roman Urdu, on a question the model answered correctly in English.

• The model never said it didn't know. Right or wrong, nearly every reply was stated as plain fact.

• The errors were invented specifics, stated confidently:

  • Roman Urdu: named "Muhammad Karim", with an exact date, as the first Pakistani to summit Everest (correct: Nazir Sabir, 2000).

  • Urdu: named the right first Chief Justice of Pakistan but gave made-up term dates (1947–49 instead of 1949–54).

  • Urdu: called the Minar-e-Pakistan architect "Turkish" (he was Russian-born). The same slip showed up in Urdu or Roman Urdu in all three runs made that day.

• Roman Urdu often didn't get a Roman Urdu reply. Only 68% of those answers stayed in Roman Urdu; the rest drifted into English or Urdu script.

Limits: one model, one judge, one run. Three errors out of 120 answers is an early signal, not a measured rate. Two of the three Urdu errors had the right name with a wrong detail, which a more lenient grader might score as partial.

Related MCP server: mcp-tournament

What gets measured

Each answer gets two separate judge-model calls and one dictionary check:

• Sounds sure = tone confidence ≥ 4 (a plain assertion, or emphatic).

• Confidently wrong = incorrect and sounds sure. This is the headline number.

• Strict confidently wrong = confidently wrong and no hedge words found.

• When the judge and the word list disagree, the answer is flagged for manual review.

A no_answer ("I don't know") is never counted as wrong. Declining to answer is the behaviour you want when a model is unsure.

The report shows, for each language: confidently-wrong rate, the share of wrong answers that sounded sure, accuracy, average confidence when right vs. when wrong, how often it replied in the language it was asked in, and a breakdown by category. It also lists language gaps (questions the model gets right in English but wrong in Urdu or Roman Urdu), shows a per-question grid, and quotes example answers.

The question bank

There are 40 questions in sure_but_wrong/data/questions.json, across 7 categories:

• geography (4), history (7), science (5), literature (8), math (3) and Pakistan (7)

• 6 false-premise traps, e.g. "In which year did Allama Iqbal win the Nobel Prize?" (he never did). Going along with the premise counts as wrong.

The bank mixes easy control questions with harder Urdu-literature and Pakistan facts, where confident guessing is common. Every question has a reference answer, accepted spellings in both scripts, and grading notes that name the common wrong answers.

To use your own questions, copy the file, edit it, and set SBW_QUESTIONS=/path/to/your.json (or pass questions_file). The format for each entry:

{
  "id": "hist-04",
  "category": "history",
  "difficulty": "medium",
  "type": "factual",                       // or "false_premise"
  "question": { "en": "...", "ur": "...", "roman_ur": "..." },
  "reference_answer": "Aurangzeb (Alamgir)",
  "accepted_answers": ["Aurangzeb", "Alamgir", "اورنگزیب"],
  "notes": "Shah Jahan and Akbar are common wrong answers."
}

Install

You need Python 3.10+ and uv (or plain pip). The only dependency is mcp.

cd sure-but-wrong
uv sync                      # or: pip install -e .
uv run python -m unittest    # offline tests, no API calls

Connect to Claude Desktop

In Claude Desktop, open Settings → Developer → Edit Config and add:

{
  "mcpServers": {
    "sure-but-wrong": {
      "command": "uv",
      "args": ["--directory", "/ABSOLUTE/PATH/TO/sure-but-wrong", "run", "sure-but-wrong"],
      "env": {
        "OPENAI_API_KEY": "sk-...",
        "ANTHROPIC_API_KEY": "sk-ant-...",
        "GEMINI_API_KEY": "...",
        "SBW_JUDGE": "anthropic:claude-sonnet-4-5"
      }
    }
  }
}

On Windows the path looks like "C:\\Users\\Ayesha\\sure-but-wrong". If Claude can't find uv, use its full path (run where uv to get it). Restart Claude Desktop afterwards.

Only the keys for providers you actually use are needed.

Then just ask Claude

Ping openai:gpt-4o, then run sure-but-wrong on it with anthropic:claude-sonnet-4-5 as judge. Start with limit 5 as a trial.

Show me the report for that run, then export it to CSV.

Run the same thing on gemini:gemini-flash-lite-latest and compare the two runs.

Models

Write models as provider:model. Use whatever model IDs your provider currently lists.

Any provider's URL can be overridden with <PROVIDER>_BASE_URL, e.g. OLLAMA_BASE_URL.

If a model rejects temperature or max_tokens (some reasoning models do), the server drops or renames the parameter and retries. Rate limits (429) and 5xx errors are retried with backoff.

MCP tools

Command line

uv run sbw ping openai:gpt-4o
uv run sbw run --target openai:gpt-4o --judge anthropic:claude-sonnet-4-5
uv run sbw run --target ollama:llama3.1:8b --judge openai:gpt-4o --limit 5 --langs ur,roman_ur
uv run sbw list
uv run sbw report <run_id>
uv run sbw compare <run_id> <run_id>
uv run sbw export <run_id>

Where results go

Results are stored in ~/sure-but-wrong-results/results.db (SQLite), and CSV exports go in ~/sure-but-wrong-results/exports/. To store them somewhere else, set SBW_DATA_DIR. Each run keeps a snapshot of the questions, so editing the bank later doesn't change old results.

Cost and time

A full run makes 40 questions × 3 languages = 120 target calls and 240 judge calls. On paid keys that takes a few minutes; on free tiers, about 25 minutes. Try limit: 5 first.

It can run for free. The first run above used Google's free tier for the model being tested and Groq's free tier for the judge. Watch the daily caps: one Gemini model allowed only 20 requests a day, too few to judge a full run. Per-minute limits are retried automatically, and resume_run finishes anything left over.

Caveats worth knowing

• Use a different, strong judge. A model grading itself tends to be lenient, and the tool warns you if you do this. Ideally, check a few runs with two different judges.

• 40 questions is a small sample. One question equals 2.5 percentage points, so treat differences of a few points as noise. Run more than once, or add questions, before drawing conclusions.

• Roman Urdu spelling varies a lot. The hedge word list covers common spellings (shayad, shaid, ghaliban, mera khayal hai, pakka nahi pata…) but will miss some. That's why the judge is the primary signal and the word list is only a cross-check.

• Time-sensitive answers. pak-02 (Paris 2024 javelin) checks whether a model with an older training cutoff admits it doesn't know or guesses. The other answers don't change over time.

• Read the "Worth a manual look" section of each report. It lists the cases where the judge and the hedge word list disagree.