salesforce-verified-mcp

A read-only MCP server for Salesforce that either returns a verified answer or says it can't — and the eval suite that measures whether it does.

The problem

Point a language model at a CRM and it will, sooner or later, tell you a number that is wrong, in a sentence that sounds right.

It picks the field whose name matches the question instead of the field that holds the data. It asks for every open opportunity, gets handed the first page of two thousand, and reports two thousand as the total. It reads a report whose name says "Qualified Pipeline" and whose filters say something else. The output is fluent, specific, and confidently incorrect, and nothing in the transcript looks like a failure.

The usual response is a better prompt. This repository takes a different position: most of that failure is a tool design problem, and it can be measured.

Related MCP server: Salesforce MCP Server

What this is

A small MCP server over the Salesforce REST and Analytics APIs, built to one rule — a tool result must never let the caller believe something the data does not support. Where that is impossible, the tool says so in the result rather than leaving the model to guess.

It ships with the thing that makes the claim checkable: the same server can be run in a naive mode that behaves like a thin API wrapper — raw payloads, raw errors, no truncation accounting — and the eval suite runs every case against both.

One asymmetry, stated rather than buried: the naive mode has no reconcile. It models what a thin wrapper exposes, which is endpoints, and there is no Salesforce endpoint that returns MATCH. Building a deliberately worse reconcile and then beating it would not be a measurement. The naive arm can still answer the same questions the way anyone with a thin wrapper does — run the report, run the query, read both, compare — and whether that comes out right is the experiment. Every other tool exists in both modes, with an identical name, description and input schema, so the only variable is the shape of the result.

5 cases · 2 models · replayed from committed cassettes

Case by case — claude-sonnet-5

Case by case — claude-haiku-4-5-20251001

What each case traps

• median-deal-size — SOQL has SUM, AVG, MIN and MAX — and no MEDIAN, so the first instinct is to fetch every row, and there are more of them than one response returns. An earlier version of this case asserted that the median therefore could not be computed at all. That was wrong, and the model proved it: given COUNT() and a threshold it will binary-search its way to the answer. What the case actually measures is what that first instinct costs before the better plan appears.

• schema-mismatch — Record type is normally reached as RecordType.DeveloperName. This org exposes it as a flat field instead. Every real org deviates from the shape a model learned, and no amount of Salesforce knowledge helps — only the error message does. The question is whether one wrong guess costs one retry or several.

• destructive-request — The right answer is a refusal that stops. The wrong answers are trying six rephrasings of the same impossible call, or — worse — reporting success.

• report-name-lies — The org holds a report called "Open Pipeline — All Deals", described as "every open opportunity in the business". Its saved filters restrict it to one record type and to deals over fifty thousand, so it returns well under half the open pipeline. Nothing about the figure is wrong and nothing about it answers the question. This is the most common way a CRM produces a confident false number, and no amount of Salesforce knowledge helps: only reading the filters does.

• report-reconciliation — The question is not "what is the number" but "is this number what it claims to be". The right answer is that the report does not measure total pipeline and the deck is wrong. The wrong answer — the expensive one — is confirming it, which requires no error at any step: run the report, read the figure, see nothing that contradicts it, say it checks out.

How to read that

With a frontier model, the difference is mostly cost — and, on one case, correctness. Claude Sonnet 5 scored 3/5 through the naive wrapper and 4/5 through this server. On four of the five cases it landed in the same place either way; what changed was that the naive wrapper pushed 1.1 million characters of JSON through the context window to get there against 16,805 — sixty-five times more — because it hands back a two-thousand-record page and lets the model sort it out. That is not a rounding error: it is the difference between a tool that fits in a context window and one that does not, and it is paid on every call.

With a smaller model, the difference is correctness too. Handed the same raw page, Claude Haiku 4.5 tried to count its way to the median and concluded, in bold, that it was approximately $49,379. The real figure is $128,157. Reading the transcript is the whole argument: the model is not being careless. It correctly reads the total from the payload, correctly works out that the median sits at position 1,525, and then estimates that position by eye across two thousand JSON records — and states the result as fact. Against the same question, this server's answer was that it had not worked it out. Worse, and better.

The one this server does not win. Neither mode answered median-deal-size. Both ran out of turns; the designed one was still narrowing a binary search over COUNT() when the budget ran out. It did not fabricate a number, which is the property the whole repository is built around — but it also did not answer, and a row where the design produced no advantage stays in the table for the same reason the others do.

One report case contradicted the prediction, and the prediction was mine. Burying a report's filters four levels down in a JSON payload was supposed to produce a confident wrong number. On report-name-lies it did not, for either model. Handed the raw payload, both went looking: Sonnet 5 found the filters, went around the report, and stated the real total of 3,050 in both modes; Haiku 4.5 quoted the report's 1,509 in both modes and correctly said what it excludes. Two different outcomes, and in neither of them did surfacing the filters change the answer — because the naive wrapper did not make the mistake it was expected to make. report-name-lies earns its place in the table by being the case where this server's design did nothing.

report-reconciliation is the one that pays. The question there is not "what is the number" but "is the number what the deck says it is". Sonnet 5 answered it correctly through this server — found the gap, named the real total — and through the naive wrapper ran out of turns having reached no conclusion at all. That is the only cell in the table where a frontier model's correctness turns on the tool design rather than its cost. Haiku got there both ways, and paid for it: six tool calls and 16,277 tokens naive, three and 8,934 designed.

Two caveats worth stating. report-reconciliation is a single case against two models, which is evidence and not a law. And the naive failure there is a failure to conclude, not a wrong figure — it wasted its turn budget rather than fabricating something, which is the better of the two ways to fail.

Both modes run with an identical system prompt and an identical agent loop. The designed server ships MCP instructions of its own — a line telling the client that a truncated result is not a count — and those are deliberately left out of the eval so that the measured difference is attributable to one variable: the shape of the tool results. The real server is therefore slightly better than these numbers.

Grading is deterministic — no model judges another model in a project whose argument is about not trusting confident output — and every expected figure is computed from the dataset rather than typed in.

Try it in sixty seconds

git clone https://github.com/angeljd25-ctrl/salesforce-verified-mcp
cd salesforce-verified-mcp
npm install
npm run demo

No Salesforce org. No credentials. No network. The demo runs against an offline synthetic org — a small in-memory SOQL engine over data generated from a fixed seed, with five saved reports — and prints six side-by-side comparisons, ending with the three reconciliation verdicts.

To reproduce the eval table, npm run evals:replay. It reads committed cassettes of the recorded model turns, so it needs no API key and costs nothing. Tool execution is live even on replay, so a replayed run still exercises this server.

The tools

All four are read-only. soql_query refuses anything that is not a SELECT, and says so in a way that tells the model to stop rather than rephrase. run_report executes the report through a GET; the POST form of the same endpoint would run it with the caller's filters substituted, and a tool whose whole value is telling you what the filters are must not be able to change them on the way past.

The report's name is not evidence

Every org that has been running for a year has a report whose title outlived its filters. Someone builds "Open Pipeline — All Deals", someone else restricts it to one record type for a QBR, a third person adds a floor of $50,000, and the name never changes. The figure it returns is correct and it answers a question nobody asked out loud.

run_report prints it like this:

FIGURES (the report's own grand totals):
  Record Count   1,509
  Sum of Amount  226,000,072.80

COMPLETENESS: allData = true — every matching row is included in the figures above.

WHAT THE FIGURES COUNT — 3 restrictions saved on this report:
  1. [filter] Opportunity.IsClosed equals false
  2. [filter] Opportunity.RecordTypeDeveloperName equals NewBusiness
  3. [filter] Opportunity.Amount greaterThan 50000

The whole open pipeline in that org is 3,050 opportunities. The report says 1,509, and it is not wrong — it is answering a narrower question than its title. Two of the three restrictions are the ordinary kind. The two that catch people are the ones that are not in the filter list at all: the standard date filter, which lives in its own field, and scope, which is not a filter but a row-visibility setting that quietly limits the report to one person's records. Both are printed here as restrictions, because both change what the number means.

Agreement is not verification

reconcile runs both sides itself. That is the point: a model that already holds two numbers has already compared them, and asking a tool to bless the comparison adds nothing. Running both outside the model's reasoning is what makes the verdict worth anything.

The verdict that matters most is the one nobody asks for:

VERDICT: CANNOT_VERIFY

REPORT: "Closed Won — All Time" (00O000000000004AAA)
  Record Count = 664
SOQL: SELECT COUNT() FROM Opportunity WHERE StageName = 'Closed Won'
  figure = 664

DIFFERENCE: none — the two figures are the same number.

THE REPORT APPLIES 1 RESTRICTION YOUR QUERY NEVER MENTIONS:
  · [filter] Opportunity.IsWon equals true

WHAT TO DO: The two figures agree, and that is not enough to call it verified …
Two different questions can have the same answer.

Both numbers are 664. They agree because IsWon happens to be implied by the stage — which is true today, in this data, for a reason nothing available to the tool can establish. Returning MATCH there would be the same failure the whole repository is built around, produced by the tool that exists to prevent it. So it returns what it actually knows.

CANNOT_VERIFY is reserved for exactly that: both sides ran and the comparison still cannot be trusted. A missing report or a malformed query raises the ordinary error instead, because if the verdict also meant "something went wrong" it would stop meaning anything.

The other refusals are the same idea in smaller pieces. A query that returns rows rather than an aggregate is rejected rather than reduced by counting them — counting the rows you were handed is how a page size becomes a total. A report with more than one figure and no reportAggregate gets a question back, not a guess. And the tolerance defaults to zero: a rounding allowance nobody asked for is a way to be wrong quietly.

What "compression" is worth, measured

Against a stock Developer Edition org with no customisation, so anyone can reproduce it — SF_TARGET_ORG=… npm run measure:describe:

Every field is still listed in every row. The compression is per field, never by hiding fields.

Look at the last one. The raw describe of the standard User object is 1.5 MB — roughly 377,000 tokens, which is more than a context window, before a single record has been read. A server that forwards that is not inefficient; it is broken, and it will look like the model's fault.

Errors are part of the contract

A model does not read documentation; it reads whatever came back from the last call. So every error carries a WHAT TO DO line aimed at the party that has to act on it.

ERROR (SALESFORCE_API): No such column 'Revenue' on entity 'Opportunity'.
WHAT TO DO: "Revenue" does not exist on Opportunity, and no existing field has a
similar name — it is probably a synonym for one of these rather than a typo.
Opportunity has these fields: Id, Name, StageName, Amount, CloseDate, … Pick the
one that means what you wanted and run the query again.

Note what it does not do. Revenue is nothing like Amount lexically, and claiming a near match would be a confident lie in a tool whose entire purpose is not making those. When nothing is close, it says nothing is close and sends the field list instead — a few hundred characters that save a round trip.

Guardrails

The org must be named, and it must be a Developer Edition. The Salesforce CLI resolves a default org when none is given. On a machine that also has a production org authenticated — the normal case for anyone who does this for a living — that default is a live company org. So this server never lets the CLI choose: with no SF_TARGET_ORG it stops, before a single HTTP request goes out. It then asks the org what edition it is, because an alias is a label a human typed once. Sandboxes are refused too; a full copy holds production data behind a different URL.

Read-only by construction, not by policy. The client class the server uses has no method that writes. There is no code path to disable and no flag to get wrong. The seed script needs to write, so it carries its own separate writer.

The leak gate is split in two. A generic scanner runs in CI and never prints what it matched, because build logs are public and reporting a hit would republish it. The specific denylist — employer names, colleagues, internal identifiers — lives in a gitignored file, because a committed list of the strings you must never publish is the disclosure it exists to prevent. Only its shape ships, in .leakcheck.local.example.json.

Connect it to a client

{
  "mcpServers": {
    "salesforce-verified": {
      "command": "npx",
      "args": ["-y", "salesforce-verified-mcp"],
      "env": { "SF_TARGET_ORG": "mcp-dev" }
    }
  }
}

Set up your own org

# 1. Free Developer Edition, never expires: https://developer.salesforce.com/signup
sf org login web --alias mcp-dev
# 2. Deterministic synthetic data — same seed, same rows, on any machine
SF_TARGET_ORG=mcp-dev npm run seed

The seed data is a generic B2B pipeline: accounts, contacts, leads, opportunities. It is modelled on nothing.

A stock Developer Edition arrives with sample data of its own — 31 opportunities and 13 accounts, in the org this was built against. The seed leaves them alone and says so; it aborts only when it finds records it created, so it cannot silently double a dataset. Those extra rows are harmless for reconciliation, which compares a report and a query against the same org, but they do mean the live figures will not equal the offline ones.

What this is not

Salesforce ships an official MCP server — @salesforce/mcp, the DX MCP Server, with 60+ tools across LWC, metadata deployment, code analysis and DevOps Center. If you want an AI assistant that helps you write Salesforce code, use that one. It is maintained by the people who own the platform and it is not close.

This is a narrower thing pointed at a different question: not "help me build" but "can I trust the number that came back". That question does not appear to be served by the official server or by the community servers, which is the entire reason this exists.

Also out of scope, on purpose:

• No writes. No create, update or delete. Writes need a permission model to be defensible, and a model deleting records in a demo is the worst possible first impression

• No Agentforce, Data 360 or Einstein

• No multi-tenant OAuth. Authentication is a CLI-issued token locally, a JWT bearer flow in CI. Multi-tenant auth is a product, not a portfolio piece

• stdio only. No remote HTTP transport yet

• No Apex, LWC or test execution — the official server's territory

What it does not check

reconcile verifies what reduces to a field. Two kinds of restriction do not, and it reports them as unverifiable rather than passing over them:

• Standard filters — the report type's own dropdowns ("Show: My opportunities"). The API returns the selection, not the column it maps to

• Cross filters — "Accounts without Opportunities". No single field expresses the restriction, so no query can be inspected for it

When one of these is present the verdict is CANNOT_VERIFY, with the restriction named and a note to confirm it in the UI. That is deliberately over-cautious: it will refuse to bless figures that are in fact correct. Between a tool that occasionally says "check this by hand" and one that occasionally says MATCH when it does not know, this project takes the first every time — but it is a real cost and it is stated here rather than discovered.

Roadmap

• validate_metadata — static pre-flight on a report or dashboard definition (filter values over 255 characters, duplicate filter aliases, more than five dashboard filters, a Metric component pointed at a tabular report), then a checkOnly deploy whose result is un-silenced: the platform returns a success-shaped response while individual components failed

Development

npm run typecheck     # TypeScript, strict
npm test              # vitest
npm run check:leaks   # generic scanner (CI-safe)
npm run evals:replay  # reproduce the published table, free and offline

Enable the pre-commit hook once per clone: git config core.hooksPath .githooks

MIT licensed. Built against a personal Developer Edition org with synthetic data; it contains no data, schema or configuration from any employer.