Skip to main content
Glama

Ask Claude, Cursor or any AI assistant about a spreadsheet and you usually get a confident answer with figures in it. Some of those figures were never calculated: the model pattern-matched them. verified-analyst closes that gap with a deterministic checker: every number in an answer must trace back to a value a real SQL query returned, or the answer is sent back for re-analysis.

It ships four ways:

what it is

for

MCP server

describe_schema, run_sql, check_answer tools for any MCP client

making Claude Desktop / Claude Code / Cursor check their own numbers

Agent + CLI

a tool-calling analyst that cannot finish until its prose is grounded

va ask "..." on a folder of CSVs

Agent skill

a SKILL.md that makes a coding agent compute in SQL and run va check on its draft

Claude Code, Codex and other agents that load skills, with no MCP setup

Leaderboard

the same oracle-scored eval run across many models

answering "how often does each model invent numbers?"

An invented number is caught, a grounded one passes

Real output of examples/catch_a_lie.py. The first draft sounds plausible and even names the wrong leader; the checker rejects it because neither figure was ever returned by a query.

flowchart LR
    Q[Question] --> A[Model]
    A -- run_sql --> D[(DuckDB, read-only)]
    D -- rows --> A
    A -- draft answer --> V{check_answer:<br/>every number in a result?}
    V -- "no: 999 was never returned" --> A
    V -- yes --> O[Answer with every number traced]

Use it from Claude or Cursor (MCP)

pip install "verified-analyst[mcp] @ git+https://github.com/QasimJaffry/verified-analyst"
va data --out ~/va-demo            # optional: a synthetic subscription-app dataset to try it on

(Not on PyPI yet, so the install comes straight from GitHub.)

Claude Code:

claude mcp add verified-analyst -- va mcp --data /absolute/path/to/your/csv-folder

Claude Desktop / Cursor (mcpServers in the client's config file):

{ "mcpServers": { "verified-analyst": { "command": "va", "args": ["mcp", "--data", "/absolute/path/to/your/csv-folder"] } } }

Every .csv in the folder becomes a table. The server tells the assistant to compute figures in SQL and to call check_answer on its draft; check_answer returns which numbers were never returned by a query in the session:

{ "grounded": false, "ungrounded": ["999"], "numbers_checked": ["1234", "999"],
  "note": "These numbers were not returned by any query in this session. Compute them with run_sql ..." }

The data folder is fixed when the server starts, queries are SELECT-only with file access disabled, and the checker calls no model.

Related MCP server: duckdb-analytics-mcp

Use it as an agent skill, or from a script

npx skills add QasimJaffry/verified-analyst      # installs skills/verified-numbers

The skill tells the agent to compute each figure in SQL and to run one stateless command on its draft before answering:

va check --data ./data \
  --sql "SELECT platform, COUNT(*) AS users FROM users GROUP BY platform" \
  --answer "iOS leads with 4,812 users."
# prints JSON with "ungrounded": ["4,812"] and exits 1

Exit code 0 means every number came from one of the given queries, 1 means some did not, 2 means a query was rejected. That makes it usable in CI or any script, with no model involved.

The leaderboard: how often does each model invent numbers?

export OPENROUTER_API_KEY=...       # one key, many models
va data
va leaderboard --models <model-id>,<model-id>,<model-id> --runs 3
# model ids: any tool-calling model from openrouter.ai/models (ids change often, so copy current ones)
# -> results/leaderboard/LEADERBOARD.md, leaderboard.json, index.html (+ per-model runs.csv)

For each model it reports:

  • invented a number: share of runs whose first answer stated a figure none of its own queries returned (raw model behaviour, before the guard steps in)

  • answered the unanswerable: on questions the data cannot answer, how often it produced a number anyway

  • correct + grounded: final answers that match the oracle and pass the checker, with the guard on

  • saved by guard: runs that invented a number, got bounced back, then passed

  • median cost per question (from OpenRouter's price list) and latency

No real-model results are published yet. The first run will be committed under results/ with its date, versions and data hashes. Nothing on this page is a model's score.

Until then, here is the pipeline run against deterministic reference solvers (scripted, not real models). This proves the metrics measure what they claim (va leaderboard --models reference:oracle,reference:self_correct,reference:overconfident,reference:tampered_prose,reference:hallucinate --runs 1):

Leaderboard of scripted reference solvers

solver

invented a number

answered the unanswerable

correct + grounded

saved by guard

oracle (always faithful)

0%

0%

100%

0

self_correct (fudges, then fixes when rejected)

100%

0%

100%

21

overconfident (answers impossible questions)

19%

100%

81%

0

tampered_prose (nudges numbers)

100%

0%

19%

0

hallucinate (never queries)

100%

100%

0%

0

Use it standalone

pip install "verified-analyst @ git+https://github.com/QasimJaffry/verified-analyst"   # add [anthropic] for the Anthropic provider
va data                                 # synthetic dataset in ./data
va ask "Which channel converts best after the trial?" --show-sql --chart-out chart.json
va eval --provider openrouter --model <model-id> --runs 5 --out results/my-model

chart.json is a Vega-Lite spec built only from rows the last query returned: the model picks the chart type and columns, never the data points.

What is different from the many "chat with your CSV" tools

  1. A numeric-claim verifier on the final prose (verify.py). Not an LLM judge: rounding-, percent- and suffix-aware matching of each number against returned cells. Arithmetic in prose is rejected on purpose, so the figure must be SELECTed.

  2. Oracle scoring, no LLM judge (scoring.py). Expected answers come from executing each task's gold SQL. A run passes only if the value is correct and the prose is grounded.

  3. Questions it should refuse. 5 of 26 tasks have no valid numeric answer (missing data, or too vague). Inventing a number there is a failure.

  4. The harness is tested against its own mutants. Deliberately wrong solvers must fail; a faithful one must pass 100%. CI enforces it.

Related open-source work: several DuckDB/SQL MCP servers give assistants query access, andon-verify re-checks numbers in finished reports, and text-to-SQL agents abound. This project's narrower claim is the in-loop, prose-level number check plus a refusal-aware oracle eval and a cross-model leaderboard.

Harness validation (no model involved)

python scripts/harness_table.py

reference solver

answerable passed (of 21)

refusal/clarify passed (of 5)

what it proves

oracle

21

5

a faithful solver passes everything

wrong_value

0

5

right prose, wrong value: the scorer catches it

tampered_prose

0

5

value right, prose number nudged: the verifier catches it

hallucinate

0

0

never queries, invents a number: verifier catches it; also fails refusals

overconfident

21

0

answers vague/impossible questions with a number: scorer catches it

refuse_all

0

3

blanket refusal cannot game the eval (the 2 vague tasks need needs_clarification)

self_correct

21

5

a model the guard rescues still gets credit, and is counted as "saved by guard"

How the verifier decides

answer says

query returned

verdict

1,616 Android users

1616

grounded

Average is 4.5

4.4567

grounded (rounded to the digits shown)

40.8%

0.408 or 40.8

grounded

about 41.6k

41581.04

grounded

Average is 4.47

4.4567

rejected (wrong at the second decimal)

3000 users in total

rows 1616, 1384

rejected: 3000 was never returned, so SELECT the sum

signed up on 2025-03-14

n/a

dates and clock times are not claims

Limit worth stating plainly: the verifier proves the numbers came from a query, not that the query answered the question. A real cell quoted for the wrong metric passes the verifier and is caught only by the oracle eval. See DESIGN.md.

SQL safety (does not rely on the model behaving)

  • exactly one statement, and DuckDB's own parser must classify it as SELECT

  • external access is switched off after the data loads and the configuration is locked, so read_csv('/etc/passwd'), COPY, ATTACH, SET ... all fail

  • 100-row result cap and a 10 s timeout that cancels runaway queries

  • no code-execution tool (see DESIGN.md for why)

Eval design

  • Dataset: synthetic subscription mobile app (users, subscriptions, payments), seeded and reproducible. It is synthetic; nothing describes a real app.

  • Tasks: 26 in evals/tasks.jsonl: lookup 5, aggregate 6, join 6, time 4, unanswerable 3, ambiguous 2.

  • Reporting: pass rate with a Wilson 95% interval, latency as p50/p95 (never the mean), median steps and tokens, unverified-number rate, per-category breakdown, runs.csv with every run. Metadata (date, versions, dataset and task hashes) is written next to the numbers.

  • Scale caveat: 26 tasks is a small, single-schema set, so treat the leaderboard as a pilot. Wilson intervals are reported precisely because they are wide.

Layout

src/verified_analyst/
  agent.py        loop, tools, verification gate (records the first draft before the guard)
  verify.py       numeric-claim extraction + grounding
  mcp_server.py   MCP tools for Claude / Cursor (core is transport-independent)
  db.py           DuckDB session, SELECT-only guard, schema description
  charts.py       Vega-Lite spec from returned rows
  providers/      anthropic, openai-compatible (stdlib HTTP)
  datagen.py      seeded synthetic dataset
  evals/          tasks, oracle scoring, stats, runner, reference solvers, leaderboard
evals/tasks.jsonl
tests/            129 tests, incl. a real MCP stdio round trip
skills/           verified-numbers agent skill (uses `va check`)

Limitations

  • Verified-by-query is not verified-correct: a real cell quoted for the wrong metric passes the checker and is caught only by the oracle eval.

  • Small integers can be grounded by coincidence because rounding tolerance is real.

  • In MCP mode the checker only helps if the assistant calls it; the server's instructions ask it to, but clients decide.

  • One schema and 26 tasks: the leaderboard is a pilot. More schemas and tasks are the next step.

  • Provider code is tested at the wire-format level, not against live APIs in CI.

MIT licensed.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    • Query, join, profile, clean and convert CSV/JSON/Parquet with server-side DuckDB over MCP.

    • Text-to-SQL MCP server: read-only queries on PostgreSQL, MySQL and SQL Server with your real schema

    • ParseHub.dev is a hosted document-to-records service with a remote MCP server. Agents can upload a CSV, PDF, or Word file, summarize and analyze CSV data (filter, sort, group_by, and more), read text from PDF and Word files, extract attendees, agenda items, and action items from meeting notes, ask a question about an uploaded document, and store files as database records to list and read later. Each account sees only its own files and records. **Endpoint:** `https://parsehub.dev/mcp` (Streamable HTTP, stateless) **Auth:** API key in the `X-API-Key` header (keys start with `mcp_`), or `Authorization: Bearer <API key or JWT>`. Every request without credentials returns 401. **Plans:** Free (100 API requests/month, 1 document upload), or 5,000 requests/month on the $9/month or $42/year plan. Every MCP request counts toward the allowance. **Tools:** upload_csv, summarize_csv, analyze_csv, upload_document, read_document, extract_meeting_notes, ask_document, database_status, list_database_documents, get_database_document, store_document, extract_stored_meeting_notes, export_database (administrators only). Not affiliated with ParseHub (parsehub.com), the web-scraping tool. ParseHub.dev does not scrape websites. By Poietes LLC. Docs: https://parsehub.dev/guide#connect-via-mcp

      https://parsehub.dev/mcp
    • Query your warehouse or a CSV with Claude/ChatGPT over MCP, governed by table-level ACL + audit.

      https://mcp.contextflo.com/mcp

    Related MCP Servers

    • A
      license
      A
      quality
      Not graded
      maintenance
      An MCP server that enables AI assistants to load, query, and analyze local CSV files using tools for filtering, aggregation, and grouping. It provides capabilities to describe schemas, calculate statistics, and sample data directly from CSV files.
      6
      -
    • A
      license
      A
      quality
      D
      maintenance
      A read-only DuckDB MCP server offering context-efficient analytics tools (list_datasets, describe_table, profile_column, explain, query) with a semantic layer for business rules, security guards, and disclosed truncation to help LLMs produce correct answers while minimizing token usage.
      5
      MIT
    • A
      license
      A
      quality
      C
      maintenance
      A minimal MCP server that provides a persistent DuckDB SQL engine to AI assistants, enabling natural-language querying of CSVs, Parquet, and cloud data with 12 tools and optional read-only mode.
      12
      MIT