elliot-mcp
Elliot is an MCP verification server that checks AI agents' "done" claims and audits whether a repo's docs match its actual code, backing every verdict with deterministic proof.
Audit documentation claims (
elliot_audit_start,elliot_record_claim,elliot_record_behavior,elliot_audit_finish): compare README/docs/security pages against code, verifying every quote and absence search against a pinned commit, then writereport.html,report.mdandclaims.jsonintoelliot-report/.Independent review (
elliot_review_packet,elliot_record_review): hand findings to a fresh reviewer that confirms, downgrades or rejects them, with optional isolated reviewer-process mode viaELLIOT_ROLEandELLIOT_REVIEW_TOKEN.Prove files really changed (
elliot_snapshot,verify_changes): fingerprint the working tree, then diff a later state or HEAD to show what a task actually modified.Run functional checks (
verify_outcome): execute a command in a disposable repo copy inside an OS sandbox with network blocked; only exit code 0 is VERIFIED.Check a deployment is live (
verify_url): verify HTTP status and expected text, refusing loopback/private-network addresses.Verify GitHub claims (
verify_github): confirm a PR is merged, CI checks passed, or a commit/branch was pushed.Verify npm publishes (
verify_npm): check a package version exists on the registry, optionally published after a given time.Verify signed receipts (
verify_attestation): check Ed25519 signatures and digests, flagging signer identity untrusted without an independently supplied public key.Report agent honesty (
elliot_report): summarize recent receipts showing how many claims were checked and how many held up.Integrate anywhere:
initregisters the server with Claude Code, Cursor and Codex; also ships a Claude Code plugin and a GitHub Action that checks PR descriptions.
Allows auditing GitHub pull requests by checking claims in the PR description against the code and commenting with proof of verification or falsehood.
Quick start
In the repository you want to check:
npx -y github:EmanueleCell4/elliot-mcp initRestart your agent, then type:
/elliot-auditA minute or three later the report is in elliot-report/<date>-<commit>/ at the root of the repo: report.html, report.md and claims.json. There is nothing to open, no dashboard and no account.
Requirements: Node 20 or newer, git, and Claude Code, Cursor or Codex.
As a Claude Code plugin
Inside Claude Code:
/plugin install elliot --marketplace EmanueleCell4/elliot-mcpThat adds the Elliot tools, a /elliot:audit command, and a skill that makes Claude verify its own "done" claims before reporting them.
On every pull request (GitHub Action)
Elliot reads the pull request description, checks each claim against the code ("fixes X", "adds tests for Y", "no behaviour change"), and comments with the proof. If an agent wrote the PR, this is where its "done" gets checked.
# .github/workflows/elliot.yml
name: Elliot
on:
pull_request:
types: [opened, edited, synchronize]
permissions:
contents: read
pull-requests: write
jobs:
check:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: EmanueleCell4/elliot-mcp@v0.5.0
with:
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}The comment looks like this:
Verdict | Claim | Proof |
❌ FALSE | Adds unit tests for the comment builder in test/comment.test.js | searched for |
✅ VERIFIED | The badge is only offered when the audit has no findings |
|
It runs Claude Code with your API key. One check usually takes 1–2 minutes. Nothing is sent anywhere except Anthropic's API and GitHub.
README badge
When every claim Elliot checks holds up, the report includes a badge you can paste into your README:
Related MCP server: agentseed
Why
Coding agents say "Done ✅" when the work isn't done. They delete failing tests, skip checks, describe security that doesn't exist, and write READMEs that promise what the code never does. CI stays green because tests check the code the agent wrote, not the promises the product makes.
I found this out the hard way. My own legal-tech app had a security page promising role-based access enforced in the database. The data layer bypassed it by design, and every permission check was a function that ignored its input. The agent had told me it was done.
The usual fix is to ask a second AI to review the first one. That gives you another opinion from the same kind of system that made the mistake. In the first automatic audit I ran, the agent doing the checking cited line numbers that don't exist in 4 of 17 submissions.
Elliot works the other way around: the agent proposes, Elliot verifies. Every quote must really be in the file, at that line, at that commit: the audit fingerprints the tree when it starts (tracked changes and untracked files included) and refuses evidence once anything changes, and symlinks that lead outside the repository are not evidence. Every "this doesn't exist" is searched by Elliot itself. A fresh reviewer then tries to prove each finding wrong before it reaches the report.
What it found
Elliot 0.3 ran unattended inside Claude Code on 9 popular open-source tools from the AI coding-agent ecosystem (October 2026). A separate reviewer that had not seen the audits then checked every finding against the code.
Findings in the reports | Confirmed correct | Left unconfirmed by the reviewer | Wrong | Time per repo |
13 | 12 | 1 | 0 | 1 to 3 minutes |
"Wrong" means a factual error: the code does not show what the finding says.
Some of what it caught:
A tool whose README says "No Telemetry: no data collection or tracking" ships analytics that are on by default, with consent pre-set to "yes". (public issue)
An app where logging in turns telemetry back on after the user switched it off, then identifies the user by email. (public issue)
A security page that calls a script sandbox "read-only" while the code can give scripts file writes and MCP tools. Confirmed by the project's triage bot, with a docs fix sent upstream. (issue, PR)
Docs saying an API key is stored encrypted while the code writes it as plain JSON.
A popular agent tool that sends users' full AI prompts and responses to a third-party error tracker by default, under an option named "anonymousTelemetry". Reported privately to the maintainers; details will be published after a fix.
What it still misses: on the same 9 repos it found 6 of the 10 serious issues I had found by hand beforehand (a small, hand-picked set, so read it as a rough coverage figure, not a recall estimate). The most common miss is a missing authorization check (for example, a download endpoint that doesn't check who is asking). Elliot doesn't look for those yet; see the roadmap.
Benchmark: Claude with Elliot vs Claude alone
Eight widely used SDKs, 18 broken promises planted by flipping a value behind a documented default, two runs per condition, same model and instructions. A blind judge checked every finding that was not a planted bug.
Claude alone | Claude + Elliot | |
Planted bugs found (2 runs) | 28 / 36 (78%) | 30 / 36 (83%) |
False alarms (judged not real) | 21% of findings | 6% of findings |
Real unplanted problems found | 8 | 5 |
Elliot raises about a third as many false alarms and finds slightly more planted bugs; Claude alone casts a wider net. When the two blind judges (Claude and Codex) disagreed, reading the code showed Codex had misstated the code 9 times: a second model is not a check unless something re-reads the source. Method, limits and all raw outputs: benchmark/planted-promises.
Benchmark: can anyone re-check what the agent says it verified?
56 real bug fixes (28 tasks, Opus and Haiku). Every fix was correct and no agent claimed "done" falsely. But in 19 of 56 runs (34%) the check the agent said it ran could not be re-run by anyone else: a deleted scratch script, a check described in prose, a path on the agent's own machine. The work was right; the evidence was gone. benchmark/done-claims
How it works
Risk sweep (Elliot). Before the agent reads anything, Elliot scans the docs for risky promises (telemetry, privacy, encryption, access control, sandboxes, "never", "by default") and the code for signals that often contradict them: analytics SDKs, permission bypasses such as
--dangerously-skip-permissions, privileged database clients, secrets written to disk. The audit cannot finish until every one is addressed or skipped with a reason.Read and propose (your agent). Your agent traces each claim into the code and proposes a verdict, quoting the docs and the code.
Verify the proof (Elliot). Each quote is matched against the file at the pinned commit, within two lines of where the agent said it is. Each "absent" claim is searched by Elliot with
git grep. Citations that don't check out are rejected, and the agent has to fix them.Materiality test. A FALSE or PARTIAL verdict must say what a user who trusts the docs would get wrong. Omissions, naming details and tiny delays become notes, not findings. Changelogs, roadmaps and internal developer docs are rejected as sources, because they are not promises to users.
Independent review. A fresh subagent that never saw the audit tries to prove every finding wrong, and confirms, downgrades or retracts it. Elliot will not write the report until every finding has been reviewed.
Report. HTML, Markdown and JSON in
elliot-report/, plus a tamper-evident receipt for every check in.elliot/receipts.jsonl.
If you wrote code in the same session, the protocol tells the agent to run the audit in a fresh subagent, so the auditor is not the author.
Example report
The full report is in examples/lex-ai-report.html. It was produced by an earlier version, before the independent review step existed.
Commands
Set-up: init
npx -y github:EmanueleCell4/elliot-mcp init # every agent it detects
npx -y github:EmanueleCell4/elliot-mcp init --claude # only Claude Code
npx -y github:EmanueleCell4/elliot-mcp init --cursor # only Cursor
npx -y github:EmanueleCell4/elliot-mcp init --codex # only Codex
npx -y github:EmanueleCell4/elliot-mcp init --all # all threeinit is idempotent: running it twice changes nothing, and it never overwrites a command file you edited. It writes:
Agent | MCP server registered in | Command added |
Claude Code |
|
|
Cursor |
|
|
Codex |
|
|
Audit a repo: /elliot-audit
Run it in your agent from the repository root. In Claude Code and Codex, anything you type after the command is passed to the agent as extra instructions, for example /elliot-audit focus on the security page.
Check a "done" claim
The same server also checks individual claims while the agent works. It ships instructions telling the agent to use them before it says "done".
The agent says | Tool | What Elliot looks at |
"I fixed / cleaned / refactored the code" |
| the content hash of every file before and after the task |
"The behavior works" |
| runs the requested test command in a temporary OS sandbox; only exit code 0 is VERIFIED |
"It's deployed / live" |
| the HTTP status and the text really served at the URL |
"PR merged", "CI is green", "pushed" |
| the GitHub API: merge state, check runs, whether the commit exists |
"Published to npm" |
| the npm registry: version exists, published after a given time |
"What did my agent really do today?" |
| your receipts: claims checked, how many held up |
These checks answer VERIFIED, NOT_VERIFIED or INCONCLUSIVE. Inconclusive means Elliot could not confirm, for example because of a GitHub rate limit. It never guesses.
All MCP tools
Tool | Used by | What it does |
| the agent | Pins the commit, lists the files that make claims, runs the risk sweep, returns the protocol. |
| the agent | Records one claim with a verdict. Checks the source quote, every evidence quote and every absence search. |
| the agent | Records risky code behaviour the docs never mention, with checked code quotes. |
| the agent | Returns a self-contained prompt for a fresh reviewer subagent. |
| the reviewer | Confirms, downgrades or retracts a finding. A retraction needs a code quote. |
| the agent | Refuses to finish while risk-sweep items or reviews are missing, then writes the report. |
| the agent | Fingerprints the working tree before a task. |
| the agent | Compares the tree with a snapshot, or with HEAD. |
| the agent | Runs a command and arguments against a disposable copy, with writes confined to the copy and network blocked. |
| anyone | Checks an Ed25519 signature and receipt digest, optionally against a separately trusted public key. |
| the agent | Checks status and expected text at a URL. |
| the agent | Checks |
| the agent | Checks a package version on the registry. |
| you | Summarises recent receipts. |
Verdicts in an audit
Verdict | Meaning |
| The code implements the claim. A small gap that misleads nobody is kept as a note. |
| Implemented, with a gap that would make a user who relies on it do something wrong. |
| The code contradicts the claim, or the feature, command, flag or option does not exist. |
| Depends on a deployment, a contract or a service outside the repo. |
| Risky behaviour in the code that the docs never mention. |
Manual installation
Use this if you'd rather not run init.
Claude Code
claude mcp add elliot -- npx -y github:EmanueleCell4/elliot-mcpCodex (~/.codex/config.toml)
[mcp_servers.elliot]
command = "npx"
args = ["-y", "github:EmanueleCell4/elliot-mcp"]Cursor (.cursor/mcp.json) and any other MCP client
{ "mcpServers": { "elliot": { "command": "npx", "args": ["-y", "github:EmanueleCell4/elliot-mcp"] } } }Then add the command yourself, or ask your agent to "run an Elliot claim audit of this repository". For agents that support skills, also add skills/elliot/SKILL.md.
Configuration
All settings are optional environment variables.
Variable | Default | What it does |
| unset | Used by |
|
| Where receipts and audit state are stored. |
|
| GitHub API base URL, for GitHub Enterprise or tests. |
|
| Registry used by |
| unset | Path to an Ed25519 PKCS#8 private key. When set, every saved receipt is signed. Keep it outside the repository. |
| public-key fingerprint | Human-readable signer label included in signed receipts. The label is not an identity proof by itself. |
| unset | Set to |
| unset | Enables isolated review mode; configure the same non-empty secret on auditor and reviewer server processes. |
Network access and data
Elliot has no telemetry and no server of its own. What it touches:
Audit (
/elliot-audit,/elliot:audit): reads files in the repository and runsgitlocally. It makes no network calls.verify_url: sends one GET request to the URL the agent passes in. Loopback, private-network and link-local addresses (including cloud metadata) are refused on every redirect hop unless you setELLIOT_ALLOW_PRIVATE_URLS=1; one 15-second deadline covers the whole request and the body stops at 2 MiB.verify_github: callsapi.github.com(orELLIOT_GITHUB_API). If a GitHub token is set (theGITHUB_TOKENenv var, or the plugin's optionalgithub_tokensetting), it is sent only to that API.verify_npm: callsregistry.npmjs.org(orELLIOT_NPM_REGISTRY).GitHub Action: runs Claude Code in your CI with your
ANTHROPIC_API_KEY, reads the PR description and code, and posts one comment through the GitHub API.On disk: receipts in
.elliot/(orELLIOT_DIR) and reports inelliot-report/, both inside your repository.
The Claude Code plugin runs the server bundled in this repository (node src/index.js). Claude Code installs its two npm dependencies from package-lock.json when you install the plugin.
Output
your-repo/
├── elliot-report/
│ └── 2026-10-22-dd84474/
│ ├── report.html # open in a browser
│ ├── report.md # paste into an issue or PR
│ └── claims.json # everything, for scripts and CI
└── .elliot/
├── receipts.jsonl # one tamper-evident receipt per check
└── audits/ # audit stateAdd .elliot/ to .gitignore. Commit elliot-report/ if you want the report to live with the code.
A receipt looks like this:
{
"elliot": "0.3.0",
"id": "rcpt_3f9a1c2b7d10",
"check": "claim",
"claim": "Role-based access is enforced at the database layer through row-level security.",
"target": "lex-ai@dd84474",
"verdict": "FALSE",
"reason": "All data access uses the service-role client, which bypasses RLS.",
"evidence": { "source": { "file": "app/security/page.tsx", "line_start": 14 }, "evidence": [ { "file": "lib/supabase/index.ts", "line_start": 10 } ] },
"checked_at": "2026-10-21T19:42:11.000Z",
"checker": "local",
"digest": "sha256:…"
}The digest detects changes to a receipt. For third-party verification, configure a private key outside the repository:
openssl genpkey -algorithm Ed25519 -out "$HOME/.elliot-signing-key.pem"
chmod 600 "$HOME/.elliot-signing-key.pem"
export ELLIOT_SIGNING_KEY="$HOME/.elliot-signing-key.pem"
export ELLIOT_SIGNER_ID="acme-ci"Saved receipts then include the Ed25519 public key, its SHA-256 fingerprint, signer label and signature. A verifier checks the digest and signature with verify_attestation; valid is true only when a separately trusted public key is supplied and matches. Without that key, the verifier reports cryptographic validity while marking signer identity untrusted. The embedded key proves which key made the signature, not that the key belongs to a particular person or organization. Publish or distribute the trusted public key through a channel your verifiers already trust. Protect and back up the private key outside the repo. Receipts created without ELLIOT_SIGNING_KEY remain unsigned for compatibility.
Isolated review
For enforced reviewer separation, run Elliot twice with the same ELLIOT_DIR and a private ELLIOT_REVIEW_TOKEN (generate one with openssl rand -hex 32). The auditor process uses ELLIOT_ROLE=auditor; the independent reviewer process uses ELLIOT_ROLE=reviewer. The auditor server omits elliot_record_review; the reviewer server exposes only that tool. elliot_review_packet gives the reviewer the finding, checked quotes, impact and an audit-scoped capability, without repository access or the auditor's explanation. Elliot encrypts the capability in audit state, checks its hash and the reviewer server credential on submission, and rejects cross-audit reuse and duplicate decisions. Keep the shared ELLIOT_REVIEW_TOKEN in the two host configurations; only the single-audit capability travels in the review packet.
{
"mcpServers": {
"elliot-auditor": {
"command": "node", "args": ["/path/to/elliot-mcp/src/index.js"],
"env": { "ELLIOT_ROLE": "auditor", "ELLIOT_REVIEW_TOKEN": "<same-random-secret>", "ELLIOT_DIR": "/path/to/project/.elliot" }
},
"elliot-reviewer": {
"command": "node", "args": ["/path/to/elliot-mcp/src/index.js"],
"env": { "ELLIOT_ROLE": "reviewer", "ELLIOT_REVIEW_TOKEN": "<same-random-secret>", "ELLIOT_DIR": "/path/to/project/.elliot" }
}
}
}The host must give the reviewer only the packet and the reviewer MCP connection. The default unconfigured mode retains the original combined tool surface for existing MCP clients. Separate processes and tool permissions enforce the boundary; deployments where the auditor can read reviewer host configuration or impersonate its MCP connection do not provide a separate trust domain.
Functional outcome checks
Call verify_outcome with an executable and argument array, for example command: "npm", args: ["test"]. Elliot copies the working tree to a temporary directory, then runs the command in the OS sandbox (sandbox-exec on macOS or bwrap on Linux), with network access blocked and writes limited to the temporary copy. A zero exit code is VERIFIED; a failing check is NOT_VERIFIED. If the OS sandbox is unavailable, the result is INCONCLUSIVE and the command is not run. Dependencies already present in node_modules are copied into the temporary tree. No shell expansion is performed; pass flags as separate arguments.
FAQ
Isn't this just an LLM judging an LLM?
No. The agent proposes verdicts, but every piece of evidence is checked deterministically: quotes against files at a pinned commit, absences with Elliot's own git grep. A second agent that never saw the audit reviews every finding, and a retraction has to be backed by a code quote that Elliot also checks.
Does it send my code anywhere?
Elliot itself makes no network calls during an audit. It reads files and runs git locally. Your agent reads the code as it normally does, with the model you already use. The verify_url, verify_github and verify_npm checks call those services only when the agent uses them.
Does it change my code?
No. It writes only .elliot/ and elliot-report/, and init writes the config files listed above.
What does it cost? Nothing. Elliot is MIT-licensed and the audit runs on the agent and model you already pay for. An audit of a medium-sized repo took 1 to 3 minutes in our tests.
Can it audit private repos?
Yes. It runs where your code is. Only verify_github needs GITHUB_TOKEN for private repos.
Can I trust a PASS?
Treat a clean report as "nothing found among the claims checked", not as a certificate. The claim audit is a static reading of the code; it does not run your app (use verify_outcome for that). Authorization checks are covered only where sibling handlers check the caller and one does not; ownership checks are not covered yet.
I found a false positive.
Please open an issue with the report's claims.json entry. False positives are the number Elliot is built to keep at zero.
Roadmap
File, URL, GitHub and npm checks with local receipts
/elliot-audit: claim audit with checked citations, report inelliot-report/Risk sweep, undisclosed behaviours, materiality test and independent review
Authorization gaps: handlers that skip the auth check their sibling handlers perform
Ownership checks: handlers that check the caller is logged in but not that they own the record
Run functional outcome checks in an OS sandbox
Hooks that run Elliot automatically when the agent stops (Claude Code, Codex)
A GitHub Action that checks the claims in each pull request description
Ed25519 signed receipts with offline verification
Development
git clone https://github.com/EmanueleCell4/elliot-mcp && cd elliot-mcp
npm install
npm test # 35 tests, including end-to-end runs that drive the server over stdio with the MCP clientThe server is plain Node (ES modules) with two dependencies: the MCP SDK and zod. The main pieces:
File | What it does |
| MCP server, tool definitions, |
| Audit state, citation checks, review gate, verdict resolution |
| Risk sweep: risky doc promises and code signals |
| The protocol the agent follows |
| HTML and Markdown reports |
| Registers Elliot in Claude Code, Cursor and Codex |
|
|
Issues, false positives and stories of agents telling you they were done are very welcome.
License
MIT © 2026 Emanuele Cella
Available Tools
14 toolselliot_audit_finishFinish the audit and write the reportA
Close the audit and write report.html, report.md and claims.json into elliot-report/-/ at the repository root.
| Name | Required | Description | Default |
|---|---|---|---|
| out_dir | No | Override the output folder (relative to the repository root). | |
| skipped | No | Risk-sweep items you looked at and decided need no claim, each with a reason. They are listed in the report. | |
| audit_id | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations only declare readOnlyHint=false and openWorldHint=false; the description usefully adds what is created and where (three named files under elliot-report/<date>-<commit>/). It does not disclose whether existing files are overwritten, whether the audit is locked afterwards, or what happens with unrecorded candidates, so it stops short of a full behavioral picture.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single front-loaded sentence with no filler; the action and its concrete outputs come first and nothing is redundant.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a state-mutating finalization tool with no output schema, the description covers the produced artifacts but omits the preconditions for a successful call, the effect on audit state, and what is returned on success or failure.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 67%, and the description says nothing about any of the three parameters. In particular the significant 'skipped' array (risk-sweep items with reasons that appear in the report) and the 'out_dir' override are left entirely to the schema, so the description does not compensate for the coverage gap.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb ('Close the audit') and enumerates the exact artifacts produced (report.html, report.md, claims.json) plus their destination path. It is distinguishable from most siblings, but it does not clarify how it differs from the sibling elliot_report, which plausibly also emits a report.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is only implied: 'Close the audit' suggests this is the terminal step of an audit workflow, but there is no explicit statement of prerequisites (an open audit_id from elliot_audit_start, claims already recorded) or of when to prefer this over elliot_report.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
elliot_audit_startStart a claim auditA
Start auditing whether this repository does what its README, docs and security/pricing pages say. Pins the commit, lists the files that make claims, and returns the protocol to follow.
| Name | Required | Description | Default |
|---|---|---|---|
| path | No | Any path inside the repository (default: the server's working directory). | |
| sources | No | Audit only the claims made in these repo-relative files, e.g. a pull request description saved as .elliot-pr/PR.md. Turns off the repo-wide risk sweep. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations declare readOnlyHint=false and openWorldHint=false, and the description corroborates the mutating nature by disclosing that it 'pins the commit' and returns a protocol to follow. It does not say whether the pinned state is persistent, whether concurrent audits conflict, or what cleanup is required against elliot_audit_finish. Still, the key side effect is surfaced, which is real value beyond the annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single dense sentence whose first clause states the core purpose and whose remaining clauses front-load the concrete outputs. No filler, no repetition of the title, nothing that could be cut without losing information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a no-required-param session-start tool with full schema coverage, no output schema and meaningful annotations, the description covers purpose, side effects, and the returned artifact ('the protocol to follow'). What is missing is lifecycle context: how long an audit stays open and its relationship to elliot_audit_finish. That is a modest gap, not a blocking one.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% and both parameters are richly documented in the schema, including the default working directory and the fact that 'sources' disables the repo-wide risk sweep. The description adds nothing about either parameter, so it sits at the baseline. The phrase 'lists the files that make claims' loosely gestures at the sources scoping but no more than the schema already says.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description gives a specific verb and resource (audit whether the repo matches its README/docs/security/pricing claims) and enumerates the concrete effects: pinning the commit, listing claim-bearing files, returning a protocol. That is far more than a restatement of the name. It does not name elliot_audit_finish as the counterpart, so sibling differentiation is only implied by the word 'Start'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The verb 'Start' plus 'returns the protocol to follow' implies this is the entry point of a multi-step audit before elliot_record_claim / elliot_audit_finish, but that sequencing is never stated. There are no explicit when-not conditions or named alternatives. The agent must infer the workflow position rather than being told it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
elliot_record_behaviorRecord undisclosed risky behaviorA
Record something risky the code does that the docs never mention (for example, it sends usage events, stores a key in plain text, or skips permission prompts) and no doc sentence promises otherwise. If a doc sentence does promise otherwise, use elliot_record_claim on that sentence instead. Never attach this to an unrelated sentence.
| Name | Required | Description | Default |
|---|---|---|---|
| signal | No | The code signal id from the risk sweep that led here, if any. | |
| audit_id | Yes | ||
| behavior | Yes | What the code does, plainly, e.g. 'Sends the user email to PostHog on login'. | |
| evidence | Yes | Code that does it. | |
| severity | Yes | ||
| user_impact | Yes | Who is affected and how. | |
| docs_searched | Yes | Where you looked in the docs and found nothing about it. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations only tell the agent this writes to a closed-world target; the description adds the substantive preconditions (no doc sentence may promise otherwise, must belong to a specific sentence) that govern a valid record. It stops short of saying what recording does downstream — dedup, ID returned, effect on the audit — so it is not fully transparent.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with the core purpose and only three sentences long; the examples and the sibling routing each earn their space. The closing instruction 'Never attach this to an unrelated sentence' is slightly oblique since no sentence parameter exists in the schema, costing a point.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 7-parameter write tool with no output schema, the description covers the conceptual bar for behavior/docs_searched but omits what a successful call produces and how audit_id or signal should be sourced. Adequate to call correctly in principle, but an agent must infer the plumbing from the schema alone.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 71%, and the description adds real meaning for two of the trickiest fields: what counts as 'behavior' (with concrete examples) and the evidentiary standard behind docs_searched ('the docs never mention... no doc sentence promises otherwise'). It says nothing about audit_id, severity, or signal, leaving those entirely to the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Specific verb+resource ('record something risky the code does that the docs never mention') with three concrete examples of qualifying behavior. It also names the sibling elliot_record_claim and the exact condition that separates the two, so an agent can route without opening a schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly states when to use this tool (undocumented behavior with no doc promise), when to use the alternative instead ('If a doc sentence does promise otherwise, use elliot_record_claim on that sentence'), and an exclusion ('Never attach this to an unrelated sentence').
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
elliot_record_claimRecord one audited claimA
Submit one claim with its verdict and evidence. Elliot checks that the claim's source quote is really in the docs at those lines, that every evidence quote is really in the code at those lines, and runs every 'absent' search itself. If anything does not check out the claim is rejected: fix the citation and submit again.
| Name | Required | Description | Default |
|---|---|---|---|
| note | No | For VERIFIED: a minor gap worth mentioning that misleads nobody. | |
| claim | Yes | The claim, quoted or tightly paraphrased, as a user would read it. | |
| source | Yes | Where the project makes the claim. | |
| verdict | Yes | VERIFIED: the code implements it. PARTIAL: implemented with a material gap or narrower than claimed. FALSE: the code contradicts it or the feature does not exist. UNVERIFIABLE: depends on a deployment, contract or service outside the repo. | |
| audit_id | Yes | ||
| evidence | Yes | Code that implements or contradicts the claim. Use type 'absent' when the claimed thing does not exist in code. | |
| severity | No | Required for PARTIAL and FALSE: how much a user relying on the claim is hurt. | |
| explanation | Yes | One or two plain sentences: what the code actually does. | |
| user_impact | No | Required for PARTIAL and FALSE: what a user who relies on this sentence would get wrong, concretely. If there is no real consequence, use VERIFIED with a note. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Goes well beyond the annotations by disclosing the validation contract: source quotes are checked against the docs at those lines, evidence quotes against the code, and 'absent' searches are executed server-side. It does not say whether a rejected claim is persisted, whether resubmission replaces a prior claim, or what a successful submission returns, which are the remaining behavioral unknowns for a mutating call.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three tight sentences, front-loaded with the action, then the validation mechanics, then the failure remedy. Nothing is repeated from the schema and no sentence is filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Adequate for a 9-parameter nested-schema tool whose fields are 89% documented: the description covers the one thing the schema cannot, namely that quotes and absence checks are independently verified. It stops short of describing the success response, which matters slightly since there is no output schema.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 89%, so the baseline is 3; the description earns an extra point by clarifying that for 'absent' evidence the agent supplies only a pattern and Elliot performs the search, which is non-obvious from the schema alone.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ("Submit one claim with its verdict and evidence") and names the payload triple, so an agent can distinguish it from elliot_record_behavior and elliot_record_review without opening any schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives clear retry guidance once validation fails ("fix the citation and submit again"), which is the main decision point for a per-claim submitter. It does not explicitly route against sibling recording tools, so the alternative-selection half of the guidance is left implicit.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
elliot_record_reviewRecord an independent review decisionA
For the reviewer process: confirm, downgrade or reject one finding. In isolated mode provide the audit-scoped reviewer_token from the evidence packet. Confirm needs evidence you checked yourself (a code quote or an absent search); reject needs a code quote showing the claim holds. All evidence is checked against the files.
| Name | Required | Description | Default |
|---|---|---|---|
| reason | Yes | ||
| finding | Yes | The finding id, e.g. 3 or b1. | |
| outcome | Yes | confirm: real and material (pass severity to change it, e.g. low). downgrade: the mismatch is real but immaterial and becomes a note; do not pass a severity. reject: the claim actually holds. | |
| audit_id | Yes | ||
| evidence | No | ||
| severity | No | Corrected severity, when the auditor's is wrong. | |
| reviewer_token | No | Per-audit capability included in the evidence packet when isolated review is enabled. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Goes well beyond the annotations by disclosing the enforcement rule that all evidence is verified against the actual files, which implies submissions can fail validation on unverifiable quotes. It also states the per-outcome evidence obligations. It stops short of saying what happens on a failed check or whether a decision can be revised later.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Four short sentences, front-loaded with the purpose before the mode prerequisite and evidence rules. The 'For the reviewer process:' opening is slightly clipped but no sentence is wasted.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
There is no output schema, so the description must carry the result story, and it does not say what the tool returns or what an agent sees after a successful or rejected review. For a state-mutating review tool with 7 parameters, the mechanics of failure and the returned decision record are a material gap.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 57% and the description adds real meaning for outcome by spelling out what counts as valid evidence for confirm versus reject. However, it restates reviewer_token already documented in the schema, never mentions the severity parameter, and introduces an 'absent search' evidence kind that the schema's evidence.type const ('quote' only) cannot express.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (record/review) and resource (one finding) with the exact decision vocabulary (confirm, downgrade, reject), scoped to 'the reviewer process'. It is clearly distinct from sibling recorders like elliot_record_claim or elliot_record_behavior, though it never names those siblings to reinforce the split.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives a clear situational frame ('for the reviewer process') plus a mode-specific prerequisite: supply the audit-scoped reviewer_token from the evidence packet in isolated mode. It does not route the agent away from alternatives (e.g. elliot_review_packet) or say what to do when a finding is not yet reviewable, so the exclusion half is missing.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
elliot_reportHonesty reportARead-only
Summarise recent receipts: how many claims were checked and how many held up. Show this to the user when they ask what the agent really did.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | How many recent receipts (default 50). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare this as read-only and closed-world, so the safety profile is covered. The description adds that the report summarises recent receipts and includes counts of checked and upheld claims, which is useful context but not rich behavioral detail such as data range or return format.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two tight sentences with no wasted words. The purpose is front-loaded, and the usage instruction follows naturally.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple read-only summary tool with one fully documented parameter and no output schema, the description provides the essential purpose and usage context. It could clarify the scope or meaning of 'recent' beyond the limit parameter, but otherwise it is sufficiently complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, and the sole limit parameter is fully documented in the schema with its default and range. The description does not add meaning beyond the schema, so the baseline of 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and resource: summarising recent receipts, including the metrics 'claims checked' and 'held up'. It is clear what the tool does, but it does not explicitly differentiate itself from sibling tools such as elliot_review_packet or the verify_* tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives a clear usage trigger: show this when the user asks what the agent really did. It does not provide explicit exclusions or name alternative tools, so it is not a full 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
elliot_review_packetGet the independent review packetA
Returns only findings and checked evidence for review. Give it to a fresh reviewer process. With ELLIOT_REVIEW_TOKEN configured, the auditor endpoint cannot record review decisions; connect the reviewer only to a separate ELLIOT_ROLE=reviewer server.
| Name | Required | Description | Default |
|---|---|---|---|
| audit_id | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations only declare readOnlyHint=false and openWorldHint=false, so the description carries real weight and delivers: the payload is limited to findings and checked evidence, an ELLIOT_REVIEW_TOKEN is required for the configuration described, and the auditor endpoint cannot record review decisions. That is genuinely useful operational context an agent would not get from the annotations alone.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three short sentences, front-loaded with the return contract before the security caveat. Every sentence carries information; no filler, though the security sentence is dense and could be split.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema, the description says little about the shape of the packet beyond 'findings and checked evidence', which a fresh reviewer process must parse. The critical security/role guidance is present, but the caller lacks detail on packet structure and on audit_id semantics.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
There is one required parameter (audit_id) and schema description coverage is 0%, and the description never mentions it. The schema's regex pattern at least signals the audit_<hex> format, but nothing in the prose explains where to obtain the id or what it identifies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Specific verb+resource: it returns findings and checked evidence, and the word 'only' scopes the payload away from decisions/summaries that sibling tools like elliot_report or elliot_record_review presumably handle. It is clear what the tool produces, though it never explicitly names which sibling it replaces.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
'Give it to a fresh reviewer process' states the intended consumer directly, and the closing sentence gives an actionable deployment constraint (use a separate ELLIOT_ROLE=reviewer server). There is no explicit 'do not use this when...' exclusions, but the intended use case is unambiguous.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
elliot_snapshotSnapshot the working treeA
Fingerprint every tracked and untracked file in the git repository before starting a task, so verify_changes can later prove what really changed.
| Name | Required | Description | Default |
|---|---|---|---|
| path | No | Any path inside the repository (default: the server's working directory). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations declare readOnlyHint=false and openWorldHint=false, so the agent already knows this is a bounded, non-read-only operation. The description adds the scope (tracked + untracked files) but never explains what state the non-read-only behavior creates — where the snapshot persists, whether a prior snapshot is overwritten, or how expensive it is on a large repo.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
One sentence with the action and the downstream purpose front-loaded, and no filler. Every clause carries information an agent needs.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With zero required parameters, no output schema and a trivial schema, the definition covers purpose, timing and downstream linkage adequately. The remaining gap is what the fingerprint operation actually stores and returns, which matters somewhat given readOnlyHint=false.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The single `path` parameter is fully documented in the schema, including that any path inside the repo works and that the server's working directory is the default. The description adds no syntax or edge-case detail beyond that, which is acceptable given 100% schema coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
It names a concrete action and resource — fingerprinting every tracked and untracked file in a git repository — and ties that output to a specific consumer, verify_changes. That is enough for an agent to distinguish it from the other verify_* and elliot_record_* siblings.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The phrase 'before starting a task' plus the reference to verify_changes later gives a clear trigger and pairing. It stops short of stating exclusions (e.g. don't run mid-task or on non-git directories), so it is clear context rather than a full when/when-not rule.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
verify_attestationVerify a signed receiptARead-only
Verify an Ed25519-signed Elliot receipt and its digest. Returns valid=true only when the signature passes and an explicitly supplied public key matches; without one it reports cryptographic validity but marks signer identity untrusted.
| Name | Required | Description | Default |
|---|---|---|---|
| receipt | Yes | ||
| public_key | No | Public key independently trusted by the verifier. Omit to check only mathematical signature validity; valid remains false. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no output schema, the description carries the return-value burden and does so well: it discloses that valid=true requires both signature pass and an explicitly supplied matching key, and that omitting the key yields crypto validity flagged as untrusted signer. That is substantive semantics beyond the readOnly/openWorld annotations and does not contradict them.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two tight sentences, front-loaded with the core action and then the conditional valid/public-key rule. No wasted text and the key caveat arrives early.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema exists, and the description adequately covers return semantics and the security caveat, making the tool safe to call correctly. The only shortfall is silence on the shape of the required nested receipt object, which an agent may need before invoking.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
At 50% schema coverage, the public_key behavior is reinforced by the description, but the required 'receipt' parameter — a nested free-form object — gets no explanation of its expected structure or fields. Marginal value added over the schema, no compensation for the gap.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource — 'Verify an Ed25519-signed Elliot receipt and its digest' — plus the exact cryptographic scheme. This clearly distinguishes it from siblings like verify_changes, verify_url, or verify_outcome, which handle other artifact types.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explains the decision condition for supplying public_key (trusted signer identity vs. math-only check), which is real usage guidance. However, it never states when to reach for this tool versus the many sibling verify_* tools, leaving tool-level routing to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
verify_changesVerify files actually changedARead-only
Check whether files really changed, compared with a snapshot taken before the task (or with HEAD if no snapshot is given). Use before claiming code was fixed, cleaned, refactored or written.
| Name | Required | Description | Default |
|---|---|---|---|
| path | No | Any path inside the repository. | |
| claim | No | The claim being checked, in the agent's own words, e.g. 'Deployed the landing page'. | |
| paths | No | Only count changes under these repo-relative paths. | |
| snapshot_id | No | Id returned by elliot_snapshot. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true and openWorldHint=false, so the description's added value is the comparison semantics: it compares against a prior snapshot or, failing that, HEAD. That is real behavioral context an agent cannot get from the annotations or schema alone, though it says nothing about what a negative or ambiguous result looks like.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, zero filler: the action and its baseline come first, the usage trigger second. Nothing could be removed without losing information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a zero-required-parameter, read-only verification tool this covers what to compare, against what, and when to call it. The one remaining gap is the result shape — whether it returns a boolean, a per-file diff, or a mismatch report — which is not covered by an output schema.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so path, claim, paths and snapshot_id are all documented in the schema itself. The description only reinforces the snapshot_id-vs-HEAD fallback; it adds no meaning for the paths filter or the claim string beyond what the schema already says. Baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource: checks whether files actually changed, with the comparison baseline named (snapshot taken before the task, or HEAD by default). This clearly separates it from verify_url, verify_github, verify_npm, which verify external artifacts rather than local file diffs.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly says to use it "before claiming code was fixed, cleaned, refactored or written", which anchors the tool to a concrete moment in the workflow. It also states the default baseline (HEAD when no snapshot is given), but does not name any sibling tool as an alternative or exclusion.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
verify_githubVerify a GitHub claimARead-only
Check what GitHub actually says: whether a PR is merged, whether CI checks passed on a PR or commit, or whether a commit/branch was really pushed. Set GITHUB_TOKEN for private repos.
| Name | Required | Description | Default |
|---|---|---|---|
| pr | No | Pull request number. | |
| ref | No | Commit sha or branch name. | |
| repo | Yes | owner/name | |
| check | Yes | ||
| claim | No | The claim being checked, in the agent's own words, e.g. 'Deployed the landing page'. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true and openWorldHint=true, so the safety and external-call profile is covered structurally. The description adds a genuinely useful auth prerequisite ('Set GITHUB_TOKEN for private repos'), but says nothing about behavior on missing refs, rate limits, or whether a failed check is an error or a negative verdict.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, front-loaded with the core purpose, and every clause earns its place by naming a distinct check mode or a prerequisite. The colon-separated enumeration is dense but scannable, with no filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description carries the burden of explaining what comes back, and it never says whether the result is a boolean, a verdict with evidence, or an error on a failed check. The check-to-parameter mapping is also left implicit, so a caller can still mis-invoke it despite otherwise adequate coverage.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 80%, so the schema already documents pr, ref, repo, and claim; the baseline of 3 applies. The description adds modest value by implying which check each input answers (PR-based vs. ref-based), but never states explicitly that `pr` is required for pr_merged/checks_passing while `ref` is required for commit_exists/checks_passing — a pairing the agent must infer.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a specific verb and resource ('Check what GitHub actually says') and then enumerates the three exact checks it performs — PR merge status, CI check status, and commit/branch existence — which map directly onto the check enum. This clearly differentiates it from siblings like verify_npm and verify_url, which cover other verification surfaces.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The enumerated checks give clear context for when each mode applies (verifying a merge claim vs. a CI claim vs. a pushed-commit claim), and the private-repo token note signals the prerequisite. There is no explicit when-not guidance or named alternative for overlapping checks, so it falls short of a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
verify_npmVerify an npm package was publishedARead-only
Check the npm registry for a package and version. Use before claiming a package or release was published.
| Name | Required | Description | Default |
|---|---|---|---|
| name | Yes | Package name, e.g. 'zod' or '@scope/pkg'. | |
| claim | No | The claim being checked, in the agent's own words, e.g. 'Deployed the landing page'. | |
| version | No | Exact version; defaults to the latest dist-tag. | |
| published_after | No | ISO time the version must have been published after (proves it is a new publish). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true and openWorldHint=true, so the agent knows this is a safe external read. The description adds the pre-claim verification intent, but says nothing about what a failed check looks like (error vs. negative result) or how the claim parameter influences the response — meaningful gaps for a verification gate with no output schema.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, zero waste, with the action stated first and the usage rule second. Nothing to trim and nothing buried.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a four-parameter verification tool with no output schema, the description never indicates the shape of the verdict the agent should expect or how to interpret a failure, which is exactly what an agent needs to act on the result. Adequate but with a clear gap.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so all four parameters are already documented, including the published_after proof-of-new-publish semantics. The description only echoes name and version, so baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource: check the npm registry for a package and version. The npm-registry scope implicitly separates it from verify_github and verify_url, but no sibling is named explicitly to confirm that distinction.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
"Use before claiming a package or release was published" gives a concrete trigger condition tied to the other elliot_record_claim-style tools. It lacks an explicit when-not or a named alternative, which keeps it below a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
verify_outcomeRun a functional check in isolationARead-only
Run a command and arguments in a disposable repository copy under the host OS sandbox. Writes stay in the copy, network is blocked, and VERIFIED requires exit code zero. For a shell pipeline use command bash with args ["-c", "..."]; Elliot adds set -o pipefail so a failing step is not hidden by the last one. Put every file the check needs in the repository: files elsewhere (such as /tmp scripts) do not exist inside the sandbox. If the OS sandbox cannot start, returns INCONCLUSIVE without running the command.
| Name | Required | Description | Default |
|---|---|---|---|
| args | No | Executable arguments, e.g. ["test"] for npm test. | |
| path | No | Repository path (default: server working directory). | |
| claim | No | The claim being checked, in the agent's own words, e.g. 'Deployed the landing page'. | |
| command | Yes | Executable, for example npm, node or pytest. No shell is used. | |
| timeout_ms | No | Timeout, default 120000ms. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Substantially exceeds the annotations: it discloses disposable-copy isolation, network blocking, that writes are confined to the copy, the exit-code-zero rule for VERIFIED, pipefail injection for bash, and the INCONCLUSIVE fallback when the sandbox cannot start. This also explains why readOnlyHint=true holds despite commands that write files, which is genuinely additive context.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with the core action and scope, then progressively narrower operational caveats in four tight sentences with no filler. Each sentence carries distinct information (isolation, shell recipe, file placement, failure mode).
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 5-parameter tool with no output schema, it covers the important unknowns: isolation, network policy, success criterion, and one failure terminal state. It does not enumerate the full result vocabulary beyond VERIFIED/INCONCLUSIVE, which is the only remaining gap an agent might hit when interpreting output.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3, but the description adds real meaning beyond the schema: the bash + ["-c", "..."] pattern for pipelines (the schema only says 'No shell is used'), and the constraint that every needed file must live in the repository, which implicitly scopes path. It says nothing about claim or timeout_ms, leaving those to the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The first sentence states a specific verb and execution context: run a command and arguments in a disposable repository copy under the host OS sandbox. It is far more precise than the title alone and makes clear this is a functional-check executor rather than a metadata/URL/npm verifier. It never names the sibling verifiers (verify_changes, verify_npm, verify_url) it is distinguished from, so it stops short of a 5.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is implied by the constraints: it explains the shell-pipeline recipe (command bash, args ["-c", "..."]) and warns that files outside the repository do not exist in the sandbox. There is no explicit 'use this instead of verify_changes/verify_npm when...' routing, and no stated exclusions, so the agent must infer when to reach for this tool versus its siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
verify_urlVerify a URL is liveARead-only
Check that a URL (a deploy, a preview, a page) actually responds with the expected status and, optionally, contains expected text. Use before claiming something is deployed or live.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The URL that should be live. | |
| claim | No | The claim being checked, in the agent's own words, e.g. 'Deployed the landing page'. | |
| expect_text | No | Text that must appear in the response body, e.g. a headline you just shipped. | |
| expect_status | No | Expected HTTP status (default 200). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true and openWorldHint=true, so the safety and external-network profile is covered. The description adds that it checks status and optionally body text, but says nothing about timeouts, redirects, authentication, or what happens on failure. With annotations carrying the safety burden, this is adequate but not rich.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two tight sentences, front-loaded with what the tool does and followed by the decision context. Every sentence earns its place with no redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool is simple and all parameters are well documented in the schema. However, there is no output schema, and the description does not explain what a verification result looks like (e.g., pass/fail, observed vs expected status, text-match outcome). For a verification tool, that return-value gap keeps it at minimum viable completeness.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so every parameter is already documented in the schema. The description restates the purpose of the URL, status, and text checks but adds no syntax or format detail beyond the schema. Baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb ('check/verify') and resource ('a URL') and clarifies the scope: response status and optional expected body text. It does not explicitly differentiate itself from narrower siblings like verify_github or verify_npm, leaving the agent to infer that this is the generic URL checker.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives a clear usage context: 'Use before claiming something is deployed or live.' This tells the agent when the tool is appropriate. It does not name alternatives or state when not to use it, which keeps it below a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
14 tool updates
v0.6.0- First observed
elliot_audit_finish - First observed
elliot_audit_start - First observed
elliot_record_behavior - First observed
elliot_record_claim - First observed
elliot_record_review - First observed
elliot_report - First observed
elliot_review_packet - First observed
elliot_snapshot - First observed
verify_attestation - First observed
verify_changes - First observed
verify_github - First observed
verify_npm - First observed
verify_outcome - First observed
verify_url
TDQS
Scored across 14 tools
Most tools target clearly different resources: verify_changes (file diffs), verify_url (HTTP), verify_outcome (sandboxed command), verify_github, verify_npm, verify_attestation are unambiguous. Weaker spots are the report-family (elliot_report summarises receipts, elliot_review_packet packages findings, elliot_audit_finish writes report files) and elliot_record_claim vs elliot_record_behavior, which are distinguished only by careful reading of the descriptions.
There is a coherent, if dual, convention: elliot_* for audit-state tools and verify_* for external-evidence checks, each with readable verb/noun content. Minor deviations exist (elliot_audit_start / elliot_audit_finish use noun_verb ordering while elliot_record_claim, elliot_record_behavior, elliot_record_review use verb_noun).
14 tools sits comfortably in the reasonable band, and each verify_* variant covers a genuinely different evidence source rather than padding. The only mild bloat is the three report/output tools, though each serves a different consumer (user summary, reviewer process, on-disk artifacts).
The audit lifecycle is well covered end to end: snapshot, audit_start, claim/behavior recording, review_packet, record_review, audit_finish, report. Gaps are minor: no tool to list or inspect individual previously recorded claims/receipts, and no ability to retract an already-accepted claim, plus verification is limited to GitHub and npm among external registries.
Maintenance
Related MCP Connectors
Research-backed lint + generation for agent context files (CLAUDE.md, AGENTS.md) + prompt enhancer.
Website checker for AI coding agents. Measures your site in a real browser, with evidence.
Checks AI-written code against your rules as it is written, and keeps a record of what it found.
Lints + auto-fixes how AI coding agents discover any new product. 24 rules, 6 tools, score 0-100.
Related MCP Servers
- AlicenseAqualityDmaintenanceValidates AI-generated code against actual codebases to catch hallucinations, dead code, and API mismatches before runtime.1280 npm1MIT
- AlicenseNot gradedqualityAmaintenanceEnables AI coding agents to enforce spec-driven development and verify code before it is marked done, using six tools that catch invented APIs, scan for hallucinated content, check plugin conformance, sandbox-run tests, validate schemas, and record audit evidence.771 npm8PolyForm Noncommercial 1.0.0
- AlicenseNot gradedqualityCmaintenanceEnables auditing and drafting of AI agent files (CLAUDE.md, AGENTS.md, Cursor rules) and turning project descriptions into full build specs, with local and API-based tools.11 npmMIT
- AlicenseNot gradedqualityBmaintenanceEnables coding agents to run deterministic completeness audits on code before declaring 'done'—catching unwired modules, placebo tests, swallowed errors, and stubs via AST analysis, entirely offline and free.MIT