Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@metrics-mcp show me active accounts for the last 3 months"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
A semantic layer an agent has to go through, and a check that it did.
Metrics are defined once, in YAML. The SQL comes from dbt. An agent asking a question over MCP cannot write its own query and cannot invent its own definition of "active account" — and every number it puts in a sentence is verified against the result it came from before that sentence is allowed out.
uv sync --extra dev
uv run mx build # generates the dataset, builds the dbt warehouse
uv run mx query MX-001 # or: uv run mx serve, for the MCP serverNo key and no network for any of that. The dataset is generated from a fixed seed, the warehouse is a DuckDB file, and the whole CI run — build, tests, eval — happens offline.
MX-001 - Active accounts count
┌─────────┬───────┐
│ month │ value │
├─────────┼───────┤
│ 2026-06 │ 168 │
│ 2026-07 │ 176 │
│ 2026-08 │ 186 │
│ 2026-09 │ 185 │
└─────────┴───────┘
2026-09 is incomplete - not comparable with the months before it.That last line is the shape of the whole repo: the thing most likely to be misread is said out loud, by the layer that knows, rather than left for the reader to notice.
Point an agent at a warehouse with a SQL tool and it will answer every question you ask. Some of those answers will be wrong in a way nobody catches, because the two failures that matter do not look like failures:
It writes its own definition. Asked for active accounts twice in one conversation, it can produce two different numbers — one counting any event, one counting meaningful ones — and both look right.
It states a number that is not in the data. A figure near the real one is indistinguishable from the real one at a glance, and it only has to happen once for the whole surface to stop being trusted.
This repo closes both, structurally rather than by asking nicely in a prompt.
Related MCP server: horizon-mcp-demo
20 questions, 15 of which have an answer in the registry and 5 of which do not
(evals/cases.yaml). Three things are measured separately, because they fail
for different reasons.
keyword baseline |
| |
routing — right metric chosen | 40.0% | 68.4% |
refusal — declined the 5 unanswerable | 80.0% | 100.0% |
false refusal — declined an answerable one | 40.0% | 31.7% |
grounding — every stated number verified | 100.0% | 100.0% |
The model column is the mean of four runs. Routing ranged 60.0–80.0% across them and false refusal 20.0–40.0%, which is itself worth saying: a single eval run is not a measurement. Refusal and grounding were 100% in every run.
Grounding at 100% means nothing on its own — a check that always passes scores 100% too. So two stubs deliberately answer wrongly, in the two different ways an answer can be wrong, and the check is scored on catching them.
stub, and how it lies | membership check | scoped to the named month |
hallucinator — a number that is nowhere in the data | 100% caught | 100% caught |
misattributor — a real number, from the wrong month | 0% caught | 90% caught |
That table is the argument for scoping. An invented number is easy: it is absent from the result and any check finds it. A real number attached to the wrong month passes a membership check every single time, and nothing about the sentence looks invented to a reader either.
The residual 10% is months that happen to share a value, where naming the wrong one is undetectable by this method — a property of the data, not a bug with a fix. Both numbers are gated in CI: if an invented number ever survives, or if misattribution catching drops below 90%, the eval exits non-zero.
What the model's errors actually were. Every one of them was a false refusal — declining a question it could have answered, never answering the wrong one. Across four runs it picked a wrong metric zero times. That is the failure you want: a system that says "no metric covers that" is recoverable, one that confidently reports the wrong quantity is not.
The baseline is there to make the other columns mean something. It is word overlap against the label and definition, refusing below two matching words. It gets 40% because "MAU" does not appear in the phrase "Monthly active users" and nothing about word counting fixes that.
A metric is defined once. metrics/_catalog.yml holds the definition, the
grain, the model, the columns and the caveats. Nothing downstream — not the
CLI, not the MCP server, not the eval — carries a second copy. When two honest
definitions of "active" exist, the registry names both (MX-001 counts any
event, MX-011 counts meaningful ones) so a conversation can say which it
means, rather than one quietly winning.
A ratio is a pair of columns, never a stored rate. sum(numerator) / sum(denominator) survives aggregation; avg(rate) does not, and it is the one
people reach for. Storing activation as a per-account rate would give an
account with one user the same weight as one with forty, and the number would
stop being what its name says while still looking fine. The registry cannot
express a ratio any other way, and a test asserts the two roll-ups disagree —
so the reason the rule exists is demonstrated, not asserted.
│ month │ plan │ value │ numerator │ denominator │
│ 2026-08 │ enterprise │ 0.6667 │ 2.0 │ 3.0 │
│ 2026-08 │ free │ 0.0741 │ 2.0 │ 27.0 │
│ 2026-08 │ pro │ 0.5 │ 8.0 │ 16.0 │The components come back with the value, so a wider period can be rolled up correctly by whoever needs one.
The registry is checked against the warehouse, not trusted. mx check
resolves every model and every column reference — 43 of them — and fails if one
is missing. A renamed column breaks CI instead of silently returning a wrong
number to whoever asks next.
OK 11 metrics resolve against the warehouse (43 column references checked)Four tools, and the absence of a fifth is the design:
tool | |
| what exists |
| what one means, caveats included |
| the numbers, plus the SQL that produced them |
| whether a sentence is allowed to be sent |
There is no run_sql. A test asserts the exposed tool set is exactly these
four, so adding one is a deliberate act with a red build attached.
The agent supplies a metric id, a period, and at most one dimension from that metric's allow-list. Every identifier reaching SQL comes from the registry file, which has itself been resolved against the warehouse. A cut that is not on the list is refused rather than approximated:
{"error": "MX-001 cannot be cut by 'account_id'. Allowed: plan, country."}verify_answer pulls every numeral out of the prose and asks whether each one
is in the result. Two details do most of the work:
A claim that names a month is checked against that month. Membership in the result as a whole is not enough — it accepts a real number attached to the wrong period every time (measured above: 0% caught by membership, 90% once scoped). That is the error that survives review, because nothing about the sentence looks invented.
Percentages and rounding are handled where they are ambiguous, and nowhere
else. A stored 0.238 may be written 23.8%; a printed 12.3 may stand for
a stored 12.34. Comparison happens at the precision the text used, so 12.5
is still rejected.
It does not check meaning. "Activation fell" next to a correct, rising number will pass. This narrows failure to the numeric kind — the kind that destroys trust fastest, and the only kind a machine can settle on its own.
The first version of the number regex ended with (?![\d.]), to stop it
matching inside a longer number. It also stopped it matching a number followed
by a full stop — which is to say, the last number in almost every sentence
anyone writes. The verifier reported "no numbers found", concluded there was
nothing to disprove, and passed everything.
Nothing failed. The tests were green, because they happened to use numbers mid-sentence. The eval found it: the hallucinator was being caught 60% of the time instead of 100%, and there was no reason for the gap. A guard that silently stops guarding looks exactly like a guard that is working, and the only thing that tells them apart is a case you expect to fail. That is why the stubs are in CI.
Both themes are generated from one definition by docs/make_diagram.py — two hand-drawn files drift.
The CLI and the MCP server share the query builder, so what a reviewer sees on the command line is what an agent gets.
The warehouse is 7 models over 4 sources, with 29 dbt tests. Staging cleans exactly two things — exact duplicate events, and timestamps from client clocks set in the future — and the tests are the contract: remove the dedupe and uniqueness fails; remove the clock filter and the range test fails. A further test asserts the raw data still contains both defects, so the two guards cannot pass by having nothing to catch.
Nothing is repaired further up. A metric should not have to know its source needs cleaning.
- id: MX-012
label: Weekly contributing accounts
definition: Accounts that took a terminal action in the week.
grain: month
unit: count
model: fct_account_month
type: count_distinct
measure: account_id
filter: n_terminal_events > 0
dimensions: [plan, country]
owner: product
status: active
caveats: Terminal actions only - a view is not a contribution.That is the whole change. mx check resolves it on the next run; if
n_terminal_events is not on fct_account_month, CI says so by name.
filter accepts a bare column or a column compared to a number, and nothing
else. It is written into SQL, so its grammar is the security boundary — there
are tests for 1=1 or true, a trailing ; drop table, and a subquery.
The dataset is synthetic. Deliberately, so CI needs no download and no credential, and so a test can assert on exact numbers. It is not a real product's traffic and no conclusion about user behaviour should be drawn from it. The subject here is the contract between registry, warehouse and agent.
Routing is the weakest link, at 68%. All of its errors are false refusals, which is the safe direction, but a metric whose label does not contain the asker's word ("MAU") gets missed. Synonyms in the registry would likely fix most of it and are not implemented.
One dimension at a time. Cutting by plan and country is not supported.
Month grain only. The spine, the facts and the tools all assume it.
verify_answer holds results in memory, so a result_id does not
survive a restart.
Four eval runs is enough to see variance, not to bound it.
Grounding is checked per claim, not per paragraph. A sentence naming two months is checked against both, so a number correct for either passes.
data/generate.py seeded synthetic product usage
warehouse/ dbt project on DuckDB - 7 models, 29 tests
metrics/_catalog.yml the registry: 11 metrics
src/metrics_mcp/
registry.py load, validate, resolve against the warehouse
warehouse.py registry entry -> SQL -> result
verify.py the number check
server.py MCP: 4 tools
cli.py mx build | check | list | query | serve
evals/ 20 cases, 4 drivers
tests/ 55 tests, offlineuv sync --extra dev
uv run mx build # generate + dbt build
uv run mx check # registry resolves against the warehouse
uv run mx list # the registry
uv run mx query MX-007 --by plan --sql
uv run mx serve # MCP over stdiouv run pytest -q # 55 tests, no key
uv run python evals/run_eval.py --llm keyword --compare hallucinator --compare misattributor
uv run python evals/run_eval.py --llm misattributor --check membership # the gap
uv run python evals/run_eval.py --llm anthropic --compare keyword # needs a keyTo point a client at it, run mx serve and register the process as an MCP
server over stdio.
It is the pattern I run in production at a proptech SaaS — a registry of 155 metrics across four dashboard domains, a 124-model dbt project, and an LLM commentary layer whose numbers are checked against the fact set before they ship — rebuilt small, on open ground, so the parts can be read. That code is not mine to publish. This is.
The full definition of one metric, including its caveats.
Read the caveats before describing a number to anyone. They are where the reasons a figure is easy to misread are written down — a denominator that is not what a reader would assume, a most-recent month that is not yet complete.
| Name | Required | Description | Default |
|---|---|---|---|
| metric_id | Yes |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It discloses that the returned definition includes caveats and warns about common misinterpretations (denominators, incomplete months), which is valuable context. However, it does not address error behavior, permissions, or side effects, though the 'get' verb implies a safe read operation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The core purpose is front-loaded in the first sentence, and the second paragraph adds meaningful guidance on why caveats matter. Every sentence earns its place; there is no fluff or repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The output schema covers return values, and the tool is simple. However, the 0% parameter coverage and lack of any pointer to list_metrics as the source for metric_id leave a gap in correctly invoking the tool. The caveat emphasis is helpful but does not compensate for the missing parameter guidance.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The single parameter metric_id is not mentioned or explained in the description, and schema coverage is 0%. The description provides no guidance on how to obtain a valid ID, its format, or its relationship to the returned definition, leaving the agent completely dependent on the bare schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool returns 'the full definition of one metric, including its caveats', identifying a specific resource and scope. This distinguishes it from siblings like list_metrics (which lists metrics) and query_metric (which queries values).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The instruction to 'read the caveats before describing a number' implies this tool is the source for caveats, but the description never explicitly compares it to list_metrics, query_metric, or verify_answer, nor states when not to use it. Usage is implied but not directly guided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
List the metrics this warehouse can answer for.
Returns id, label, a one-line definition and the dimensions each may be cut by. Start here: a metric that is not on this list cannot be computed, and the honest answer to a question about it is that it does not exist yet.
| Name | Required | Description | Default |
|---|---|---|---|
| owner | No |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the transparency burden. It discloses the output shape (id, label, definition, dimensions) and the semantic rule about nonexistent metrics. However, it does not explain the effect of the optional owner parameter or any pagination/permission behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Every sentence earns its place: the purpose, the return payload, and the usage rule. There is no filler and the key guidance is front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a listing tool with an output schema, the core use case and existence-check rule are well covered. The main gap is the unexplained owner parameter, which leaves the input model incomplete for an agent trying to understand all calling options.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has one parameter, owner, with 0% description coverage, and the tool description never mentions it. The agent is left with only the parameter name and default null to infer its meaning.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a clear verb and resource: 'List the metrics this warehouse can answer for.' It also states what is returned and establishes that this is the existence-check tool, allowing an agent to distinguish it from get_metric and query_metric.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly says 'Start here' and explains that a metric not on the list cannot be computed, which tells the agent to use this tool before querying or verifying metrics. It does not name sibling alternatives directly, but the when-to-use guidance is clear.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Compute a metric over a range of months, optionally cut by one dimension.
Months are 'YYYY-MM' and both ends are inclusive. dimension must be one
of the values listed for the metric; anything else is refused rather than
approximated.
The reply carries a result_id. Pass it to verify_answer together with
whatever you are about to say, and say nothing whose numbers it rejects.
Ratios come back with their numerator and denominator so a wider period can
be rolled up by summing those, never by averaging the rates.
| Name | Required | Description | Default |
|---|---|---|---|
| dimension | No | ||
| end_month | Yes | ||
| metric_id | Yes | ||
| start_month | Yes |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and does so thoroughly. It discloses that invalid dimensions are refused rather than approximated, that replies carry a result_id that must be verified, and that ratios include numerator/denominator so aggregations must use sums, not averages. These are non-obvious behavioral expectations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact and front-loaded: purpose first, then month semantics, then dimension constraint, then verification and rollup rules. Every sentence adds operational value; there is no filler or repetition of schema content.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool complexity, absence of annotations, and presence of an output schema, the description covers the essential operational details: inclusive date range, dimension validation, the result_id/verify_answer contract, and correct ratio aggregation. An agent has enough to invoke query_metric correctly and integrate its result into a response.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must compensate. It explains start_month/end_month as 'YYYY-MM' with inclusive bounds, and dimension as restricted to values listed for the metric, with refusal behavior. metric_id is less elaborated, but its role as the metric selector is inferable from the primary sentence and sibling tools.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: 'Compute a metric over a range of months, optionally cut by one dimension.' This clearly distinguishes query_metric from siblings like list_metrics (listing metrics) and get_metric (likely retrieving metadata), and from verify_answer (checking results).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives clear invocation context: month format, inclusive ranges, dimension validation, and the required hand-off to verify_answer. It does not explicitly state when not to use this tool or name alternatives, but the usage flow is strongly implied by the sibling names and the verification instruction.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Check that every number in text came from the result it cites.
Call this on the answer you intend to give, before giving it. If it comes back with unverified numbers, the answer is wrong somewhere: fix it or say less. Do not restate a rejected number with different wording.
| Name | Required | Description | Default |
|---|---|---|---|
| text | Yes | ||
| result_id | Yes |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the burden, and it does reasonably well. It discloses that the tool checks provenance of numbers, that it can return unverified numbers, and that those numbers should not be paraphrased. It does not describe exact output structure, but an output schema exists and is not the description's primary job.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact and front-loaded. The first sentence states the core purpose, the second gives direct usage instructions, and the third adds a critical behavioral constraint. Every sentence earns its place without fluff or redundant explanation.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the simple two-parameter shape, the presence of an output schema, and the lack of annotations, the description covers the essential calling context and post-call behavior reasonably. It could more explicitly explain how `result_id` is obtained, but the cited-result language and sibling context make it inferable.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It does: `text` is the answer text to verify, and 'the result it cites' implies `result_id` is the identifier of the result that the text references. This is sufficient for a two-parameter tool, though it does not explicitly spell out the parameter names.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and resource: 'Check that every number in `text` came from the result it cites.' This clearly distinguishes the tool from its metric-listing/querying siblings, since verify_answer is about validating an answer rather than retrieving metrics.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explicitly tells the agent when to call the tool: 'Call this on the answer you intend to give, before giving it.' It also explains what to do with the result, such as fixing the answer or saying less, and warns against restating rejected numbers. It does not name alternative tools for exclusion, but the usage context is clear.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.1.0get_metriclist_metricsquery_metricverify_answerScored across 4 tools
Each tool has a clearly distinct role: listing available metrics, retrieving full definitions with caveats, computing metric values, and verifying drafted answers. There is no overlap or ambiguity about which tool to call for a given step.
All tool names follow the same snake_case verb_noun pattern: list_metrics, get_metric, query_metric, verify_answer. The pattern is consistent and predictable, with only the expected singular/plural variation for the metric resource.
Four tools is well-scoped for a read-only metrics warehouse surface. Each tool covers a necessary step in the workflow and none feel redundant or missing.
The tool set covers the full analytical workflow: discover what metrics exist, understand their exact definitions and caveats, query them, and verify any final answer before it is delivered. There are no obvious dead ends or missing operations for the stated purpose.
The grounded data layer for any LLM: governed SQL, metrics, lineage and catalog over your data.
Deterministic fact verification for AI agents — checksums & curated data, not guesses.
Verified, sourced, real-time intelligence layer for AI agents.
Define, ship & query your analytics tracking from one source of truth, trusted by humans and agents.
