Skip to main content
Glama
mekala27-45

readback

by mekala27-45

readback

Every number my agent says is read back to the warehouse before you see it.

readback is an analyst agent over three of my own data builds, served as an MCP server at protocol revision 2026-07-28 and graded against answers that were already known. You ask a question about housing (doorstep), flights (turnaround) or a fictional bank (ledger); the agent finds the table, writes the SQL, and before the answer leaves, a gate made of plain code re-runs every number it is about to say. A number that does not read back is repaired once, then refused. The live site answers with floor (no model run beats the floor on the development split).

Status: 15 of the eighteen items in the definition of done are done. Not done yet: item 10, item 15, item 18. The checklist below says what each is waiting on.

What it does, in numbers

  • The warehouses. 163 published tables from doorstep, turnaround and ledger, fetched at their pinned tags and checked file by file against SHA-256 digests in the lock. Nothing is recomputed; the builds are the record.

  • The catalog. 1,105 measure columns, every one with a unit (1,105), 657 with a definition lifted from the build's own documents and cited to the line. The rest sit in a quarantine of 2,129 named rows rather than getting a definition I made up.

  • The protocol. 7 tools, each with an output schema; stateless; public hour long cache hints; discovery; trace context carried through the request metadata; conformance passed over in memory, stdio, streamable http.

  • The golden set. 150 questions whose answers were known before the agent existed, split 50 development and 100 test.

  • The floor, the planner with no model in it: 98.0% on the test split (interval 93.0% to 99.4%), every one of its 174 shipped numbers read back. It shares the measures table with the golden truth, so read its row as a ceiling for the families it was written for, not as a rival.

  • The red team. 15 of 15 injection cases passed on the floor. The first run passed fewer, and DECISIONS.md says what it caught.

  • Models graded so far: gemini, groq, openrouter, sambanova, ollama have not run yet. A provider's row appears when the live eval has recorded its cassettes; it never extrapolates from a partial run.

Related MCP server: mnemiq

How it works

  doorstep, turnaround, ledger at v0.1.0   (fetched by digest, read only)
        v  catalog: units, cited definitions, synonyms, a lexical index, the quarantine
        v  DuckDB, three read only schemas, the SQL guard in front
   MCP SERVER   warehouses  describe  search_catalog  run_sql  published  readback  lookup
        ^  the agent is a client of its own server (in memory in tests, HTTP in production)
   AGENT        classify  plan  act  gate  repair  refuse  interrupt  answer   (LangGraph)
        |  router: gemini, groq, openrouter, sambanova, ollama, the floor; a budget each
        v
   EVALUATION   golden set, graders in code, red team, cassettes   TRACING  one trace per turn

The agent calls the same seven tools any host gets, so a bug in a tool shows up in the eval and a bug in the protocol shows up in the separate client. The gate is code, because a model checking a model is a vote, and a vote is not a verification. The floor exists because a free tier can vanish in a week, and a product that only works with a key is a demo. The known truth is the manifests, because three builds spent a day each proving their numbers. ARCHITECTURE.md has the rest.

The skills, and where each lives

Skill

Where it lives

MCP server design

packages/mcp_server, docs/protocol.md, results/conformance.json

MCP client

packages/agent (the client), tests over the in memory transport

Agent orchestration

packages/agent: the LangGraph graph, interrupts, the checkpointer

Tool use and structured outputs

the Answer schema, the tool schemas, the output mode per provider

Natural language to SQL, verified

packages/planner, run_sql, the readback tool

Evaluation engineering

packages/evals, evals/golden, the scorecard

Known truth and graders in code

catalog/manifest_index.parquet, packages/evals graders

Red teaming and injection defense

evals/redteam, the gate, docs/security.md

Observability

packages/tracing, docs/observability.md, the /traces page

Cost control and model routing

providers/registry.yaml, the budget ledger, the cassettes

Resilience without a key

the floor, the recorded session on the site

Durable execution and human in the loop

the Postgres checkpointer, the interrupt on an ambiguous name

Context engineering

prompts/registry.yaml, search then describe, tokens per question

Semantic layer and data catalog

catalog/, the units, the synonym table, catalog/measures.yaml

Security

the SQL guard, the read only connection, redaction, the identifier scan, the client budget

APIs and persistence

packages/api, scripts/check_persistence.py

Serverless deployment

Dockerfile.vercel, the cold start measurement

Spec driven, agent assisted engineering

AGENTS.md, .claude/skills, docs/SPEC.md, .mcp.json, DECISIONS.md

Interoperability standards

server.json, the connect page

Frontend for agents

web/: the streaming desk, the chips, the trace viewer

Statistics for evaluation

Wilson intervals, Benjamini-Hochberg, kappa

Communication

report/briefing.md, the pushback paragraph on every page

Reproducibility

the claim gate, the rederive, the cassettes

The definition of done

Item

Status

Evidence

1

done

data/warehouses.lock.json pins 163 tables by digest; catalog/quarantine.json publishes every rule; data/PROVENANCE.md restates the terms

2

done

evals/golden holds 150 questions split 50 and 100; results/golden_first_ten.json has the hand checked ten

3

done

results/conformance.json: revision 2026-07-28, discovery, 7 tools with output schemas, every transport, trace context

4

done

tests/warehouse/test_guard.py, tests/mcp/test_tools.py (readback, lookup round trip), tests/api/test_client_budget.py

5

done

providers/registry.yaml; tests/providers; the floor graded on both splits in results/evals/runs/floor.json

6

done

tests/agent/test_graph.py; evals/registration.json holds the plan and prompt hashes

7

done

results/deploy/verification.json from AJAY, Windows, PowerShell 5.1.26100.9711, plain JSON-RPC

8

done

tests/api/test_http.py; scripts/check_persistence.py wrote results/persistence.json

9

done

results/traces/*.parquet; the /traces page renders the waterfall

10

not done

needs a model graded on the test split and the clarity judge, which need a key

11

done

the default is floor: no model run beats the floor on the development split

12

done

AGENTS.md, CLAUDE.md, .github/copilot-instructions.md, .mcp.json, three skills (results/skills.json), docs/SPEC.md, server.json, docs/connect.md

13

done

the desk on GitHub Pages, checked in a browser in results/live_check.json

14

done

traces, evals, catalog, connect, playbook and report pages, in results/live_check.json

15

not done

results/latency.json with the cold start; results/live_check.json

16

done

results/rederive.json: no drift

17

done

DECISIONS.md has 30 dated entries, 2 reversals, 6 protocol decisions; notebooks 01 to 03

18

not done

coverage 85.4, mypy clean, ruff clean, commits 36, rederive no drift

Run it

uv sync
uv run python scripts/fetch_warehouses.py
uv run readback-mcp                      # the server over stdio
uv run uvicorn readback_api.main:app     # the API and /mcp over HTTP
uv run readback eval --provider floor    # the golden set through the floor

No key is needed. AGENTS.md has the tests and the gates.

What the platform engineer would push back on

The known answers are this portfolio's own published numbers, so the eval measures whether the agent reads them correctly, not whether they are right; each build proved its own. The gate checks that a number agrees with the query it cites, not that the query was the right one, which the graders check separately. The live eval runs on free quotas, so a model's row covers the share of the test split its budget allowed, and says so. The floor only answers the families it was written for, and refuses the rest.

How this was built

I wrote the specification and made the calls; an AI pair (Claude) wrote much of the code with me, and the commits carry its co-author trailer. Every number above comes from a query or a test the repository runs, and the claim gate fails the build if this file says anything else.

Answers are drawn from the published tables of three portfolio builds by Ajay Mekala: doorstep (public housing data from FHFA, the Census Bureau, HUD, the CFPB and Freddie Mac), turnaround (public flight data from the Bureau of Transportation Statistics, the FAA and OurAirports, with weather from Open-Meteo under CC BY 4.0) and ledger (a fictional bank, Harborline, with public HMDA, CFPB, FDIC and macro series). No person appears in any table, and questions are redacted before a model sees them. Every number is read back to the warehouse before it is shown. This is a portfolio demonstration, not financial, travel or lending advice.

Available Tools

7 tools
describeA
Read-onlyIdempotent

Describe one table: its grain, every column with its type, unit and definition (cited to the build's definitions file where one exists), the manifest figures that read it, and its first five rows sorted by the grain. Call it for one table at a time, after search_catalog has named the table.

NameRequiredDescriptionDefault
tableYes
warehouseYes

Output Schema

NameRequiredDescription
rowsYes
grainYes
tableYes
columnsYes
examplesYesthe first five rows sorted by the grain
statementNo
warehouseYes
source_stepYes
manifest_keysYes

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly, idempotent, non-destructive and closed-world, so the safety profile is covered. The description adds real substance beyond that — that definitions are cited to a build definitions file 'where one exists' (signalling the citation may be absent) and that the row sample is grain-sorted and limited to five. It stops short of anything on cost, latency, or size of wide tables.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, no filler, with the returned payload front-loaded and the invocation constraint in the second sentence. Every clause carries information an agent needs.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so the return shape is documented structurally, and the description is thorough about sequencing against search_catalog. The only real hole is the undisclosed meaning of the `warehouse` enum, which matters because table names may not be unique across the three warehouses.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must carry both parameters and only carries one: it clarifies that `table` is a single table name sourced from search_catalog. The `warehouse` parameter is never mentioned, and its enum values (doorstep, turnaround, ledger) are opaque without any explanation of what each warehouse contains.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Describe one table') and enumerates exactly what comes back: grain, every column with type/unit/definition, manifest figures that read it, and first five grain-sorted rows. That makes it distinguishable from search_catalog (naming) and run_sql (querying) without opening either schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

'Call it for one table at a time, after search_catalog has named the table' gives an explicit prerequisite and ordering relative to a named sibling. The one-at-a-time constraint tells the agent to loop rather than batch, which is the key usage decision for this tool.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

lookupA
Read-onlyIdempotent

Resolve a name to the way the warehouses spell it: a metro (doorstep), a carrier or airport (turnaround), an applicant group, an AML rule or a county (ledger), a year or a quarter. When a name matches more than one entity (Portland, Springfield, Columbus) the server asks the client to choose; a client that cannot be asked receives the candidates with needs_choice set, and must ask the person.

NameRequiredDescriptionDefault
kindYes
nameYes

Output Schema

NameRequiredDescription
kindYes
queryYes
resolvedYes
statementNo
warehouseYes
candidatesYes
needs_choiceYestrue when the name matches more than one entity and the client could not be asked

TDQS

A3.9/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare the operation read-only, idempotent, and non-destructive, while the description adds meaningful behavioral detail about ambiguous matches: the server prompts the client to choose, and non-interactive clients receive candidates with needs_choice set. This goes beyond the annotations, though it does not describe no-match handling or broader failure behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The definition is two sentences, front-loaded with the core purpose, and every clause contributes useful information about either entity kinds or ambiguity handling. There is no redundant restatement of the schema or annotations.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has only two required parameters, rich annotations, and an output schema, so the description does not need to explain return values. It covers the key ambiguity behavior but leaves sibling-relative usage and no-match behavior implicit, leaving a small completeness gap.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 0% schema description coverage, the description carries the burden and does add meaning by enumerating the kind values and mapping them to contexts such as metro/doorstep, carrier/airport/turnaround, and county/ledger. It also explains that name is resolved to canonical warehouse spelling and may produce ambiguous candidates, though individual enum values remain only loosely defined.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb and resource: resolving a name to the canonical warehouse spelling, with several entity kinds enumerated. It is clear what the tool does, though it does not explicitly distinguish itself from sibling tools such as search_catalog or describe.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied by the need to resolve names and by the ambiguity-handling guidance, but there is no explicit statement of when to prefer lookup over alternatives or when not to use it. The ambiguity behavior gives useful context but does not fully cover tool selection.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

publishedA
Read-onlyIdempotent

Read values from a warehouse's published manifest, the numbers its own documents print, by exact key or by a glob such as ch5.* (at most 50, sorted by key). Each value comes with its unit, its published rounding, its interval where one was published, and its display string.

NameRequiredDescriptionDefault
warehouseYes
key_or_patternYes

Output Schema

NameRequiredDescription
valuesYes
matchesYes
patternYes
statementNo
warehouseYes

TDQS

A3.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly, idempotent, non-destructive, closed-world, so the safety profile is covered. The description adds genuinely non-redundant behavior: a hard 50-result cap, key-sorted ordering, glob matching semantics, and the shape of each returned value (unit, published rounding, interval, display string).

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two dense sentences with no filler; the core action and scope lead, and the return-value detail follows. Nothing repeats the schema or annotations.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so describing return values was optional but adds useful orientation; the glob semantics and limit make invocation unambiguous. The remaining gap is the unlabeled warehouse enum and the absent comparison to sibling retrieval tools.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description carries the burden. It handles key_or_pattern well — exact key, glob syntax with a concrete example, and the result cap. It says nothing about the warehouse parameter beyond implying it selects a manifest, leaving the three enum values (doorstep, turnaround, ledger) undifferentiated.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource: read values from a warehouse's published manifest, with a clarifier ('the numbers its own documents print') that pins down what 'published' means. It does not explicitly contrast with siblings like readback, lookup, or run_sql, so an agent must infer the boundary rather than being told.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Retrieval mechanics are clear (exact key or glob such as ch5.*, capped at 50, sorted by key), which implies the usage pattern. But there is no statement of when to choose this over readback, lookup, search_catalog, or run_sql, and no excluded cases (e.g., values that were never published).

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

readbackA
Read-onlyIdempotent

Re-verify one number before you say it. Give the value, its unit, the display string you will show, and exactly one source: a result_id from run_sql in this request chain (with the column and row), a SQL statement to run, a manifest key, or a catalog key for a count of tables, rows, metrics or manifest values. The server re-runs or looks up the source, converts units only where the catalog allows (fraction and percent), and returns agree or disagree with the difference and the tolerance: half a unit of the published rounding for a manifest key, half a unit of your display rounding for a query. Every number in an answer must have an agreeing readback.

NameRequiredDescriptionDefault
claimYes

Output Schema

NameRequiredDescription
claimYes
reasonYes
verdictYes
evidenceYes
statementNo
toleranceYes
claim_unitYes
differenceYes
unit_sourceYes
evidence_refYesthe manifest key, or the digest of the result that was re-run
evidence_unitYes

TDQS

A4.6/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the read-only/idempotent annotations, the description explains what the server actually does: it re-runs or looks up the source, converts units only where the catalog permits (fraction and percent), and returns agree/disagree plus the difference and the tolerance. It even defines the tolerance rule per source type (half a unit of published rounding for manifest keys, half a unit of display rounding for queries).

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The critical constraint, re-verify one number before stating it, is front-loaded, and the sources and tolerance rules follow logically. The middle sentence is dense with nested clauses and could be split, but no sentence is filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has an output schema, so the description doesn't need to detail return shape, and it still summarizes the verdict and tolerance semantics. Combined with the source enumeration and the 'every number must have an agreeing readback' requirement, an agent has everything needed to invoke it correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The description explains the four mutually exclusive source modes (result_id with column and row, sql, manifest_key, catalog_key) and the unit-conversion constraint, which is more than the schema conveys in prose. It leaves warehouse, definition_id and evidence_unit unexplained in the description, relying on their schema descriptions instead.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb and resource: re-verify one number, given a value, its unit, the display string and exactly one source. It is unmistakable from sibling tools like run_sql or lookup because it explicitly consumes their outputs (a result_id from run_sql in this request chain).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It states when to use it ('before you say it', 'Every number in an answer must have an agreeing readback') and restricts the input to exactly one of four source types, which is strong routing guidance. It never names a when-not condition or an alternative tool for a case it can't handle, so it stops short of a 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

run_sqlA
Read-onlyIdempotent

Run one read only DuckDB statement against one warehouse schema. Only a single SELECT or WITH runs; writes, ATTACH, COPY, INSTALL, LOAD, PRAGMA, SET, file reading functions and tables outside the named schema are refused with the clause named. Unqualified table names are qualified with the warehouse. The limit is capped at 500 rows and the statement stops after 10 seconds. Returns the statement as it ran, the columns with units where the catalog knows them, the rows, the row count before the limit, and a result_id to cite in readback.

NameRequiredDescriptionDefault
sqlYes
limitNo
warehouseYes

Output Schema

NameRequiredDescription
sqlYesthe statement as it ran, normalized, with the limit applied
rowsYes
limitYes
digestYes
columnsYes
result_idYesa handle for readback in the same request chain
row_countYesrows the statement returns before the limit
statementNo
warehouseYes
elapsed_msYes

TDQS

A4.5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Goes well beyond the readOnlyHint/idempotentHint annotations: it names what gets refused and how (clause named in the error), states the 500-row cap and 10-second timeout, and describes unqualified-name qualification. This is exactly the enforcement detail an agent needs before composing a query.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Dense and front-loaded: the single-statement rule leads, then restrictions, then limits, then returns. Sentences are long and clause-heavy but every one carries distinct information; minor trimming possible in the return-value list.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given an output schema exists, the description needn't enumerate return fields, yet doing so confirms the result_id/readback workflow, which is genuinely useful. The only real gap is the missing contrast with sibling query/catalog tools for an agent deciding whether to write SQL at all.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, so the description must compensate. It does for warehouse (scopes to one named schema; enum values are self-evident) and limit (capped at 500), and sql's purpose is obvious from the tool itself. It stops short of clarifying whether a caller-supplied limit below 500 is honored or how it interacts with the cap.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Run one read only DuckDB statement against one warehouse schema') plus exact scope limits (single SELECT/WITH, one schema). An agent can tell this apart from siblings like search_catalog, describe and readback without opening the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Clearly defines the allowed case (single SELECT or WITH against a named warehouse schema) and the excluded cases (writes, ATTACH, COPY, INSTALL, LOAD, PRAGMA, SET, file functions, out-of-schema tables). It also routes the agent forward by telling it the result_id is meant to be cited in readback, though it never explicitly contrasts when to prefer search_catalog or describe over raw SQL.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

search_catalogA
Read-onlyIdempotent

Search the catalog of the three warehouses for tables, columns, metrics and published manifest values, ranked by a lexical index after the query is expanded through the synonym table (house price finds value_anchored). Returns the expanded query and up to k hits (at most 20), each with its unit and one definition sentence. Pass kind to search one kind only (definition finds the entries of each build's definitions document). Use it to find where a number lives before writing SQL.

NameRequiredDescriptionDefault
kNo
kindNo
queryYes
warehouseNo

Output Schema

NameRequiredDescription
hitsYes
queryYes
expandedYes
statementNo

TDQS

A4.1/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Adds substantial behavior beyond annotations: lexical ranking, synonym-table query expansion with an example, return of the expanded query, capped hits (at most 20), per-hit unit and definition sentence, and kind-specific behavior for definitions. Annotations already confirm it is read-only, idempotent, non-destructive, and closed-world; the description adds rich operational context.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loads the main action immediately and remains dense but purposeful. The synonym example and kind clarification earn their place, and there is no filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the output schema and annotations, the description covers the core search semantics, ranking, result shape, and kind filtering well. The main remaining gap is that the warehouse parameter is not explained despite the low schema description coverage.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 0% schema description coverage, the description must carry parameter meaning. It explains query expansion, k's maximum of 20, and kind's one-kind-only behavior, including the definition kind, but it does not explain the warehouse parameter or its allowed values beyond implying the three warehouses exist.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource: search the catalog of the three warehouses for tables, columns, metrics, and manifest values. The purpose is clear, but sibling differentiation is only implicit through the 'before writing SQL' context; no sibling tool is named as an alternative.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives a clear usage context: use it to find where a number lives before writing SQL, and notes that passing kind restricts the search to one kind. It does not state when not to use it or explicitly name alternatives such as lookup or describe.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

warehousesA
Read-onlyIdempotent

List the three warehouses (doorstep, turnaround, ledger) with the tag and commit each is pinned at, how many tables and rows each has, the period each covers, and the statement. Call this first when you do not know which warehouse a question belongs to.

NameRequiredDescriptionDefault

No parameters

Output Schema

NameRequiredDescription
statementNo
warehousesYes

TDQS

A4.2/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint, idempotentHint, non-destructive and closed-world, so the safety profile is fully covered. The description adds that the result is a fixed set of three warehouses with pin/tag and row/table counts, which is useful context but largely mirrors the output schema.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, front-loaded with what is returned and followed by the routing rule; nothing wasted.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With zero parameters and an output schema present, the description only needs to frame purpose and timing, which it does well. Minor gap: it does not indicate whether the returned set is fixed or workspace-dependent.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool takes no parameters, so the baseline is 4. The description correctly implies no inputs are needed to enumerate the warehouses.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb ("List") and resource ("the three warehouses") and even enumerates them (doorstep, turnaround, ledger), plus the attributes returned. An agent can distinguish this from search_catalog/run_sql immediately.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly says "Call this first when you do not know which warehouse a question belongs to," giving a clear selection rule for a discovery step. It does not name the sibling tools that take over once the warehouse is known, so it falls just short of a full when/when-not treatment.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 7 tool updatesv0.1.0
    • First observeddescribe
    • First observedlookup
    • First observedpublished
    • First observedreadback
    • First observedrun_sql
    • First observedsearch_catalog
    • First observedwarehouses

TDQS

A4/5.0

Scored across 7 tools

Disambiguation4/5

Each tool has a distinct role in the workflow (orient via warehouses, find via search_catalog/lookup, inspect via describe, query via run_sql, cite via published, verify via readback), and descriptions explicitly state call order. The only soft overlaps are search_catalog also returning published manifest values and describe vs. search_catalog both surfacing definitions, but the ranked-search vs. exact-key distinction is articulated well enough to steer selection.

Naming Consistency3/5

Conventions are mixed: verb_noun for search_catalog and run_sql, bare verbs for describe and lookup, and nouns/adjectives for warehouses, published and readback. Names are individually readable and domain-idiomatic, but there is no predictable pattern an agent can generalize from.

Tool Count5/5

Seven tools is well within the sweet spot, and each one covers a genuine stage of the read-verify pipeline rather than duplicating another. Nothing looks like padding or an afterthought.

Completeness4/5

The lifecycle is covered end to end: discovery (warehouses, search_catalog, describe, lookup), retrieval (run_sql, published), and verification (readback). Minor gaps remain, such as no direct way to enumerate all tables or columns in a warehouse schema without first naming them through search, which forces extra round trips.

Maintenance

ActivityMaintained
ResponsivenessNo issues

Related MCP Connectors

Related MCP Servers

  • A
    license
    Not graded
    quality
    A
    maintenance
    Enables AI agents to query governed data warehouses through plain language, returning grounded answers with SQL, confidence grades, and signed receipts while enforcing access, testing, and audit controls.
    437 PyPI
    16
    Apache 2.0
  • A
    license
    A
    quality
    B
    maintenance
    Answers natural-language questions against Postgres, Oracle, Snowflake, Databricks, DuckDB and SQLite via db_read(question) and get_schema(), each separately grantable. The model only proposes SQL — shape, access, dialect and query plan are checked deterministically before any rows are read, and the server refuses rather than returning an unverified answer.
    3
    81
    Apache 2.0
  • A
    license
    Not graded
    quality
    C
    maintenance
    Enables an LLM to inspect a relational database schema and answer financial questions by running validated, read-only SQL queries against a synthetic SQLite dataset.
    MIT
  • A
    license
    A
    quality
    B
    maintenance
    Enables agents to query curated metrics through a registry-backed semantic layer, preventing ad-hoc SQL and enforcing consistent definitions. Every numeric claim is verified against the underlying query result before it is allowed to be sent.
    4
    MIT