This server (Umbra) audits code and tests to find untested blind spots, verifies failing-test proofs, generates reports, and provides project glossary definitions. It supports Python, TypeScript, JavaScript, Java, Rust, and Go, and can run fully offline/deterministic or use optional model backends.
umbra_audit: Analyze a source module and its test file to identify unexercised branches, untested input regions, unasserted return contracts, and unasserted exception paths. Returns coverage metrics and Silence index, and can generate isolated proof requests or model-backed proof generation (off/codex/openai with explicit confirmation where applicable).
umbra_verify_proposal: Submit a proposed failing test through the execution gate. Retains only proposals that genuinely fail or cause errors, rejects malformed proposals with correction hints, and discards passing proposals.
umbra_write_report: Generate portable Markdown and self-contained HTML reports from the audit and retained proofs. Runs locally with no external data sent.
umbra_glossary: Retrieve authoritative definitions for Umbra-specific terms or list all defined terms; deterministic and offline.
Reveals what your tests are not telling you.
Umbra is an apophaticdefinition code auditor. Every other tool reports what it found: coverage says which lines ran, a linter says which lines smell, a green suite says all the tests pass everything they can see. None of them report what the tests are structurally unable to see. That blind region, the umbra, is where agent-written code fails.
Umbra names the shadow and proves it. For a Python, TypeScript, JavaScript, Java, Rust, or Go module and its test file(s). Umbra maps what the tests actually exercise, finds the reachable and meaningful cases nothing asserts, and reports a Silence index: the honest complement to coverage. Higher means more blind spots. Not what ran. What nothing could see. It then writes failing tests that will close the gap, and examines your functions for purity.
Built with Codex, using GPT-5.6. Track: Developer Tools. GPT-5.6 runs only where structure runs out and only meaning remains.
Umbra is the newest instrument in the Slop Audit (Open Honest Foundation), the same catch-what-metrics-miss-by-construction discipline taken from an enterprise codebase down to a single module and its test suite. Both rest on one premise: name what the instrument cannot see. Umbra's page: slopaudit.org/umbra.html.
Umbra runs two ways: as a Codex plugin, its home, auditing code the moment the agent writes it (see "Use from Codex" below); and as a standalone CLI. Either way it audits six languages (Python, TypeScript, JavaScript, Java, Rust, Go).
The CLI is a Python package, run with uv. From a clone:
uv tool install . # or: uv run umbra --helpRelated MCP server: Veris
Umbra runs two ways. Its home is the Codex plugin; the CLI is the standalone path. Both are shown here.
Once the plugin is registered (see "Use from Codex" below for the one-time setup and the full tool reference), you drive it in plain language. The surface is four tools, and here is what you can ask for:
Audit a module and its tests. "Audit this module and its test file with Umbra." Codex calls umbra_audit, reports coverage beside the Silence index, and names every blind spot. Deterministic, and needs no API key.
Prove the gaps. For each blind spot the audit returns one isolated proof request. Codex writes a single failing test from it and calls umbra_verify_proposal; Umbra's execution gate keeps only a test that genuinely fails or makes the function error, and rejects a malformed one with the exact field to fix. Umbra never writes into your test file, so adopting a surviving proof is your choice.
Get the report. "Write the report." Codex calls umbra_write_report and presents the self-contained HTML alongside the portable Markdown. Local and deterministic; sends nothing anywhere.
Ask what a term means. "What does Umbra mean by the Silence index?" Codex calls umbra_glossary and relays Umbra's own fixed definition instead of improvising one. Omit the term to list every defined term.
Stay fully offline. The audit is deterministic by default (model_backend: off) and makes zero API calls. Only proof generation can use a model, and only the host sampler unless you opt in: the OpenAI backend stays off until you set confirm_openai (or the disclosed UMBRA_OPENAI_FALLBACK).
Point Umbra at a source module and its test file, in any supported language:
uv run umbra audit path/to/module.py --tests path/to/test_module.pyUmbra prints the Coverage-versus-Silence report and writes the generated failing test using the audited language's file extension. It reads each test file to identify its framework and runs the matching one: pytest for Python; Vitest or Node's built-in node:test for TypeScript and JavaScript; JUnit or a throw-on-failure main harness for Java; the #[test] suite for Rust, whether inline in the module or in a separate file; and go test for Go, including named and external test packages. Coverage comes from each language's own tool: coverage.py, Vitest or Node V8, JaCoCo, cargo-llvm-cov, and the Go cover profile. A framework Umbra does not recognize, or a run it cannot complete, is reported as not measured with the reason attached, never a guessed number.
umbra audit [MODULE] — audit a single source file (with --tests), a Python project directory, or a Git diff (--since).
Flag | Default | Effect |
| — | A Python, TypeScript, JavaScript, Java, Rust, or Go source file, or a Python project directory. Omit when using |
| — | Test file for a single audited source. Project and diff modes discover the suite themselves. |
| — | Audit only the module and test pairs changed since this Git ref; writes |
| — | Diff mode only: exit non-zero when the aggregate Silence index rises by more than N points. |
| off | Run fully offline, with zero API calls. |
| off | Emit one machine-readable audit JSON object to stdout; the human report goes to stderr. |
| — | Also write that JSON object to a file. |
| off | Turn the audit into a convergence loop (see below). |
| 12 | Maximum model calls across the whole harden run. |
| 4 | Maximum re-audit passes. |
| 1 | Model calls before each re-audit. |
umbra benchmark — run the reproducible proof-cycle benchmark on the fixed Python gap set.
Flag | Default | Effect |
| 3 | Independent trials per fixed gap. |
| 4 | Maximum proposal attempts per gap trial. |
|
| Markdown scorecard path. |
Environment variables (CLI and plugin)
Variable | Effect |
| Enables the model layer for proof generation (CLI), or the disclosed OpenAI backend (plugin). |
| Set to |
| Per-subprocess time limit for audited and generated code (default 10). |
| Directory holding the JDK, JaCoCo agent, and JUnit console launcher for Java audits; falls back to the repository |
Add --harden to turn the audit into a loop. Umbra generates the tests that close each open facet, keeps the passing ones as regression guards and the failing ones as bug proofs, re-audits, and repeats until no closeable silence remains. The Silence index is the loop's convergence target, so "done" is definable, unlike coverage-chasing.
uv run umbra audit path/to/module.py --tests path/to/test_module.py --hardenHarden generates tests, so it uses the model layer (or Codex). The call, iteration, and per-iteration budgets are bounded by --harden-call-budget, --harden-iteration-budget, and --harden-calls-per-iteration. Run without --harden for a fully deterministic audit.
Give audit a repository directory to audit every real Python source module against the entire discovered pytest suite. Umbra runs that suite once under branch coverage, attributes resolvable calls from every test file to their source module, and writes umbra-project-report.md with per-module results plus the repository aggregate. Re-export-only __init__.py files are skipped. A module that cannot be parsed, or whose suite does not complete, is listed as unaudited and is excluded from the aggregate.
uv run umbra audit path/to/repository --no-modelProject mode is deterministic and sends no data to any API. It currently supports Python repositories with flat or src/ layouts and conventional tests/ or test/ filenames. A suite with failing tests remains measurable when coverage exists: the report names the pass count, fail count, and exit status. A run that produces no coverage is explicitly unmeasured and excluded from the index.
Umbra gives each subprocess that runs audited or generated code a 10-second limit. It stops the whole process group, so a timed-out test or replay cannot leave child processes behind. Set UMBRA_TIMEOUT_SECONDS to a positive number of seconds to adjust that limit:
UMBRA_TIMEOUT_SECONDS=30 uv run umbra audit path/to/module.py --tests path/to/test_module.pyA timed-out replay becomes an honesty_unverified result with the timeout reason. A timed-out coverage run is explicitly unmeasured and does not receive a Silence index.
Reports made for humans. Not the usual wall of text a test report buries you in: a short, readable account of what your suite cannot see, with the one failing test that proves it.
For scripts and CI, add --json. Standard output is one JSON object only; the human report is sent to standard error, while the Markdown and HTML reports are still written as usual.
uv run umbra audit examples/cart.py --tests examples/test_cart.py --no-model --jsonThe stable version-1 keys are schema_version, module_path, tests_path, language, coverage_percent, coverage_status, silence_index, silences, unverified_properties, generated_proofs, honest_maps, and model. silence_index contains the percent, numerator, denominator, and facet counts. Every silence records its function, kind, location, explanation, and whether it is closeable. Every generated proof records its gap, status, rendered test, captured output, and proposal. For a language whose coverage is not measured, coverage_status is "not_measured" and coverage_percent is null.
Use --json-out path/to/audit.json to write the same document to a file. It can be combined with --json.
In project mode, --json emits one aggregate object with audit_kind: "project", the suite status, per-module audited or unaudited status, aggregate Silence index counts, and the by-kind breakdown.
When OPENAI_API_KEY is present and the model layer runs, Umbra sends OpenAI one isolated function source, its type contract, and one located gap description for each proposed proof. It does not send the repository, test files, a whole-module site list, or any other functions.
Run fully offline, with zero API calls:
uv run umbra audit path/to/module.py --tests path/to/test_module.py --no-model--deterministic is an alias for --no-model.
Audit only the conventional module and test pairs touched since a Git ref:
uv run umbra audit --since origin/main --no-modelUmbra writes umbra-diff-report.md. It reports each pair's prior and current Silence index, the newly introduced blind spots, an aggregate delta, and files it could not pair, including deletions and unconventional test names. With the model enabled, only newly located gaps receive isolated proof requests.
The repository action is action.yml. The sample pull-request workflow is umbra-diff.yml: it sets up Python and uv, comments with umbra-diff-report.md, and can fail when the aggregate Silence index increase exceeds silence-threshold. Python is the guaranteed runner. Other supported-language files are reported only when their existing structural path can run; the comment names exactly which languages Umbra audited.
This repository ships a local Codex MCP plugin at plugins/umbra-codex. Use it whichever way you run Codex.
The plugin exposes exactly four tools. The usual loop is umbra_audit → umbra_verify_proposal (once per returned proof request) → umbra_write_report; umbra_glossary is independent and answers term questions at any time.
Tool | Parameters | Returns |
|
| Coverage percent and status, the Silence index with its located silences, up to |
|
| An |
|
| Absolute paths to both written artifacts: a plain, portable Markdown report and a self-contained, shareable HTML report. The server includes only gate-retained proofs for that pair. Deterministic; sends nothing to any API. |
|
| Umbra's fixed authoritative definition and sources for the term, or the full term index. An unknown term is named plainly, never guessed. No model call; sends nothing anywhere. |
Proof backends (umbra_audit → model_backend):
off (default): fully deterministic, zero API calls, no key required. Returns the isolated proof requests for Codex to write and verify itself.
codex: requests host-side MCP sampling, so Codex writes each proof under its own reasoning.
openai: uses a locally configured OPENAI_API_KEY, and only after confirm_openai: true. Never pass a key as a tool argument. The disclosed UMBRA_OPENAI_FALLBACK=1 is a separate opt-in, used only when host sampling is unavailable.
If a model is reached at all, every backend sends OpenAI only one isolated function source, its type contract, and one located gap per request — never the repository, test files, or other functions.
Terms umbra_glossary defines. Silence index, closeable facet, closeable silence, undeclared domain, model-ready gap, runtime property, render failure, honesty_unverified, unexercised branch, candidate input region, unasserted return contract, exception path, mutation, determinism, purity, idempotency, apophatic, umbra, Slop Audit, call-stack map, boundary_in, orchestrator, pure, boundary_out, .hd format, execution gate, and coverage versus silence.
In the Codex CLI, register Umbra's MCP server directly (add --env UMBRA_JAVA_TOOLS=/your/java/tools for Java JUnit audits; see Platform support):
codex mcp add umbra -- sh -c 'cd /path/to/umbra/plugins/umbra-codex && uv run --project ./pyproject.toml python ./server.py'In Codex inside ChatGPT, register the repository as a plugin marketplace (its manifest is ./.agents/plugins) and install Umbra Codex from the Plugins page, then restart Codex. The plugin exposes umbra_audit, which takes module_path, tests_path, and an optional proof backend. Its deterministic audit needs no API key. It returns capped proof_requests in structured output. Each request contains only one isolated function source, its type contract, and one located gap description. It never sends the repository, the test file, or other functions.
The plugin bundles an Umbra wheel and a plugin-local uv project. It can start from the Codex plugin cache or any copied directory without importing this repository. Its first launch creates a local uv environment and may download transitive dependency wheels; later launches reuse that environment.
For each proof request the audit returns, Codex writes one candidate failing test from that request alone and calls umbra_verify_proposal with the proposal and the request index. Umbra runs the verification gate and retains the proof only when it rendered into a runnable test that genuinely fails or makes the audited function error during execution. A malformed input or property is rejected with the field to correct, so Codex can resubmit it instead of presenting it as proof. Generating and verifying a proof runs in a sandbox and changes nothing: Umbra proves a gap, it never writes the test into your suite. Adopting a surviving proof is your own choice.
Codex writes each proof with its own reasoning through the verify step above, so no host-side MCP sampling is required (Codex does not support it). Requesting the inline path with generate_proof: true instead asks the host to run the model, finds sampling unavailable, and returns the deterministic result. A separate inline OpenAI backend is off by default: set UMBRA_OPENAI_FALLBACK=1 and configure OPENAI_API_KEY only if you explicitly authorize it. When used, Umbra reports that the fallback was authorized and that each request sent only one isolated function source, its type contract, and one located gap. It never sends the repository, test files, or other functions.
After the proofs, Codex generates both report artifacts with umbra_write_report and presents the self-contained HTML visual. Report writing is local and deterministic, sends no data to any API, and never gates proof generation; it returns absolute paths for you to open.
Never paste an API key into a Codex message or MCP tool call. Configure OPENAI_API_KEY in the local environment that starts Codex or the Umbra MCP server, then select model_backend: "openai" and confirm_openai: true in umbra_audit. Without that explicit confirmation, the plugin makes zero API calls. Alternatively, UMBRA_OPENAI_FALLBACK=1 authorizes the disclosed fallback only after Codex host sampling is unavailable. The same isolated function, contract, and gap boundary applies.
Umbra runs on macOS and Linux. Two things bound where it runs and what it can measure, and it names both rather than pretending otherwise:
macOS and Linux today, not Windows. Umbra runs each piece of audited code in its own process group and stops the whole group on timeout, using POSIX process groups (os.killpg, a new session, SIGKILL). Windows has no equivalent path yet, so a timeout would fail there. A Windows port using job objects is future work.
Non-Python coverage needs that language's own toolchain, and providing it is the user's job, not the plugin's. Python audits need only Python. TypeScript and JavaScript need Node; a Vitest suite is resolved from the audited project's own node_modules, while a node:test suite needs only Node. Java needs a JDK, the JaCoCo agent, and, for JUnit suites, the JUnit console launcher; all three are found through the directory named by the UMBRA_JAVA_TOOLS environment variable, falling back to the repository tools/ directory. Rust needs cargo-llvm-cov and LLVM; Go needs go. Where a toolchain, or a detected framework's tool, is absent, that language's suite is reported as unmeasured with the reason, never guessed.
Apophasis cuts both ways: the tool that names the shadow in your tests names its own.
It proves. It does not fix. Umbra writes the failing test that proves a gap. It never edits your code and never decides what the code should do. The fix is yours.
It finds one kind of blindness, not every bug. A quiet Umbra means nothing structurally locatable is unseen. It is not a promise your code is correct.
On open inputs it tests the edges, not everything. Bounded types are enumerated; unbounded ones (any int, any string) get boundary cases, not totality.
A generated test's expectation is a plausible guess, kept only when it fails. A surviving failure proves a real discrepancy exists; it does not claim to know your intended contract. You judge which side is wrong.
Runtime honesty is not uniform across languages yet. Line coverage and branch facets are now measured for all six languages, each via its native tool (coverage.py, Vitest V8, JaCoCo, cargo llvm-cov, go test), and a branch that the coverage data cannot attribute is marked honesty_unverified rather than guessed. What is not uniform is runtime honesty: mutation, purity, and idempotency are verified for Python. JavaScript verifies argument mutation and same-input output determinism, and attempts idempotency only when the suite asserts a re-application. TypeScript, Java, Rust, and Go report those properties as unverified rather than inventing them.
Bundled examples run in one command:
uv run umbra audit examples/cart.py --tests examples/test_cart.pyThe suite is green at 100% coverage; Umbra reports the silence and ends on a real failing test it wrote.
The Java fixture is equally self-contained:
uv run umbra audit examples/Cart.java --tests examples/CartTest.javaThe Rust fixture is equally self-contained and needs no Cargo project:
uv run umbra audit examples/cart.rs --tests examples/cart_test.rsexamples/delivery.rs shows the bounded-enum rule: each enum variant is a finite input region, so Umbra can name every untested variant rather than sampling one.
The Go fixture runs without a module or test framework:
uv run umbra audit examples/cart.go --tests examples/cart_check.goUmbra's proof loop is a measurement instrument, and umbra benchmark turns it into a fixed, reproducible, model-agnostic benchmark: given one located gap under a strict schema and an execution gate, how many debugging cycles does a model need to produce a failing test that genuinely runs and fails?
Twenty fixed Python gaps span input regions, unexercised branches, and unasserted exception paths. Two NaN gaps are labeled inherently unprovable, since no valid Python literal expresses NaN, and are excluded from the model's rates. Every outcome comes from real execution, not a self-report.
Run the reference backend (GPT-5.6) with OPENAI_API_KEY set:
uv run umbra benchmark --runs 3Reference run (GPT-5.6, 60 trials): 100% first-try valid, 0.00 mean debugging cycles. That is the number to beat. To run it against your own model, see benchmark/README.md.
SUBMISSION.md: problem, solution, approach, and how Codex and GPT-5.6 were used.
docs/BUILD-NOTES.md: the Codex collaboration log and the honest velocity count.
docs/SPEC.md: the technical target Codex builds to.
docs/HONEST-CODE-PRINCIPLES.md, docs/HONEST-TEST.md: the standards Umbra is built on.
benchmark/README.md: the reproducible proof-cycle benchmark.
Code is Apache-2.0, matching the Honest Framework: see LICENSE.
Documentation content (docs/, SUBMISSION.md, and the explanatory prose here) is CC-BY-NC-4.0: see LICENSE-DOCS. Code examples inside the documentation are part of the software and carry the Apache 2.0 licence.
Public repo. (Fallback if kept private: share with testing@devpost.com and build-week-event@openai.com.)
Track: Developer Tools
Entrant: Open Honest Foundation (incorporated Missouri nonprofit); representative: Adam Z. Wasserman
Deadline: 2026-07-21, 5:00 PM PT (7:00 PM CDT)
Codex /feedback session ID (core build): 019f62bc-61bc-7631-8402-dfba41870f89
Repo: public, Apache-2.0 (github.com/openhonest/umbra)
Apophatic means describing something by what it is not. It comes from apophatic theology (the via negativa), which approaches the divine only by negation. Umbra applies the same move to tests: it reports the cases they cannot see, not the ones they catch.
↩Locate test-suite blind spots in one Python, TypeScript, JavaScript, Java, Rust, or Go module and its test file. Umbra deterministically finds unexercised branches, untested input regions, unasserted return contracts, and unasserted exception paths, then reports coverage beside a transparent Silence index. It can return isolated proof requests or generate and execute failing-test proofs; the execution gate keeps only runnable tests that genuinely fail or make the audited function error. The result identifies two local report artifacts: a plain portable Markdown report and a rich, shareable, self-contained HTML visual. After it returns, write and verify a failing-test proof for each returned request, then generate both reports with module_path and tests_path only: the server includes only proofs its gate retained. Umbra proves gaps and never edits the test file. The optional OpenAI backend reads only a locally configured key after explicit confirmation; never pass a key as tool input.
| Name | Required | Description | Default |
|---|---|---|---|
| proof_cap | No | Maximum isolated proof requests returned or sent to a selected backend. | |
| tests_path | Yes | Path to that module's test file. | |
| module_path | Yes | Path to one Python, TypeScript, JavaScript, Java, Rust, or Go module. | |
| model_backend | No | off returns proof requests, codex requests host sampling, openai uses a locally configured key after confirmation. | off |
| confirm_openai | No | Required before the openai backend can send an isolated proof request. | |
| generate_proof | No | Compatibility alias for model_backend: codex. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the annotations, the description discloses important behaviors: deterministic gap detection, an execution gate that keeps only runnable tests that genuinely fail or error, explicit statement that it 'never edits the test file,' and that the OpenAI backend reads only a locally configured key after confirmation. This provides rich context about side effects and security.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is lengthy but well-structured and front-loaded with the core purpose. Each sentence contributes useful details (capabilities, workflow, constraints). It could be slightly trimmed, but the overall structure is appropriate for the tool's complexity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Even though there is no output schema, the description explains what the tool returns (coverage report, Silence index, proof requests, HTML and Markdown report artifacts) and describes execution side effects and backend behavior. It is complete for a complex tool with no output schema.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
With 100% schema description coverage, the schema already explains all six parameters. The tool description adds workflow context (e.g., using only module_path and tests_path for report generation) but does not significantly enrich the meaning of individual parameters beyond what the schema provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific action: 'Locate test-suite blind spots' and names the exact languages and gap types (unexercised branches, untested input regions, unasserted return contracts, unasserted exception paths). This clearly distinguishes the tool from siblings like umbra_verify_proposal, umbra_write_report, and umbra_glossary.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives clear context and a workflow: 'After it returns, write and verify a failing-test proof for each returned request, then generate both reports with module_path and tests_path only.' It also provides safety constraints like 'never pass a key as tool input.' However, it does not explicitly name alternative tools for when-not-to-use scenarios, so it falls short of a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Return Umbra's authoritative fixed definition for one precise, non-standard Umbra term, or list all defined terms with short glosses. Use this whenever a user asks what an Umbra term means. Unknown terms are identified plainly and never guessed. This deterministic local tool makes no model call and sends no data anywhere.
| Name | Required | Description | Default |
|---|---|---|---|
| term | No | Optional Umbra term. Omit it to list every authoritative glossary term with a short gloss. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the annotations (readOnly, non-destructive, closed-world), the description reveals that the tool is 'deterministic local' and 'makes no model call and sends no data anywhere,' adding privacy guarantees. It also commits to 'never guessed' for unknown terms, setting accuracy expectations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is three sentences, each earning its place: the function, the usage trigger, and behavioral safeguards. No redundant phrasing.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple one-parameter glossary tool with no output schema, the description covers the two operation modes, usage scenario, and behavioral guarantees. It is complete enough for an agent to select and invoke correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema already describes the optional `term` parameter and its default listing behavior (100% coverage), so the baseline is 3. The description adds context that the term is 'precise, non-standard' and that unknown terms are handled plainly, enhancing the semantics.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with 'Return Umbra's authoritative fixed definition for one precise, non-standard Umbra term, or list all defined terms with short glosses,' which clearly identifies the verb and resource. It also distinguishes itself from sibling tools (audit, verify, write) by focusing exclusively on term definitions.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explicitly instructs 'Use this whenever a user asks what an Umbra term means,' providing a clear trigger. It does not explicitly name alternative tools or when not to use it, but the context is unambiguous given the sibling tools' distinct purposes.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Run one user-approved Codex proposal through Umbra's execution gate. Use proof_index from umbra_audit's structured proof_requests. Umbra retains only a runnable test that genuinely fails or makes the audited function error. It rejects malformed concrete_input or expected_property fields with a correction reason so Codex can resubmit. A passing proposal is discarded. Re-run umbra_audit after an approved test write to measure the updated suite.
| Name | Required | Description | Default |
|---|---|---|---|
| proposal | Yes | ||
| tests_path | Yes | ||
| module_path | Yes | ||
| proof_index | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the annotations (readOnlyHint=false, destructiveHint=false), the description reveals key behavioral traits: it retains only a test that genuinely fails or errors, rejects malformed concrete_input or expected_property fields with correction reasons, and discards passing proposals. This substantive detail helps the agent anticipate outcomes and side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is five sentences long, each contributing distinct information: primary action, input source, retention rule, rejection behavior, and follow-up step. There is no fluff or redundancy, making it compact and front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers the tool's purpose, behavioral rules, prerequisites, and successor action, offering a solid operational flow for a complex tool with a nested object. However, it omits details about return values or error response structure (no output schema exists), leaving some gaps for agents needing precise interface knowledge.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
With schema description coverage at 0%, the description must compensate for parameter meanings, but it only briefly mentions proof_index and the malformed fields. It does not explain the expected format of concrete_input, expected_property, or the role of module_path, tests_path, and plain_explanation beyond the schema's bare type declarations.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: 'Run one user-approved Codex proposal through Umbra's execution gate.' This clearly distinguishes the tool from siblings like umbra_audit and umbra_write_report by focusing on executing a proposal rather than auditing or reporting.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides workflow guidance by instructing to use proof_index from umbra_audit's structured proof_requests and to re-run umbra_audit after a test write. It establishes clear context for when to use the tool, though it does not explicitly state exclusions or alternatives beyond the implicit sequencing.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
After the proof loop, write both Umbra report artifacts locally with module_path and tests_path only. The persistent server automatically includes only proofs retained by umbra_verify_proposal for that same pair. retained_proofs remains an optional explicit override for compatible callers. This deterministic action re-audits locally, sends no data to any API, and does not run proof generation.
| Name | Required | Description | Default |
|---|---|---|---|
| tests_path | Yes | ||
| module_path | Yes | ||
| output_path | No | Optional local base path or .md/.html report path. Umbra writes both sibling files with .md and .html suffixes. Defaults to umbra-report.md and umbra-report.html in the server working directory. | |
| retained_proofs | No | Optional compatibility override: exactly the proof objects returned with outcome retained by umbra_verify_proposal. Omit it to use this server session's retained proofs for the same module and test paths. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the annotations, the description discloses determinism ('deterministic action'), privacy ('sends no data to any API'), and what it does not do ('does not run proof generation'). It also explains the server's automatic proof inclusion behavior, adding rich context about side effects and state.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three concise sentences, each earning its place: the first states the main action and required params, the second explains the default proof source, and the third discloses deterministic and privacy behavior. No fluff or repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers purpose, usage timing, parameter override, and behavioral traits. It does not mention return values, but since the tool writes files and has no output schema, this is a minor gap. Overall, it is sufficient for an agent to select and invoke the tool correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 50%, and the description adds context for retained_proofs (explicit override behavior) but does not explain what module_path and tests_path are or their expected formats. It clarifies the usage pattern ('with module_path and tests_path only') but leaves the required parameter semantics to be inferred from names.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action: 'write both Umbra report artifacts locally with module_path and tests_path only.' It specifies the verb, resource, and scope, and distinguishes from siblings by referencing umbra_verify_proposal as the source of proofs. The phrase 'After the proof loop' also situates it in the workflow.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It provides clear temporal context ('After the proof loop') and indicates that the primary usage is with module_path and tests_path only, while retained_proofs is an optional override. However, it does not explicitly name alternatives or specify when not to use the tool, relying on implicit workflow positioning.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Each tool has a completely distinct purpose: umbra_audit discovers blind spots, umbra_verify_proposal runs proposals through an execution gate, umbra_write_report generates report artifacts, and umbra_glossary defines terms. There is no functional overlap between any of these tools.
All tools share the 'umbra_' prefix and mostly follow a verb_noun pattern (audit, verify_proposal, write_report). The only deviation is umbra_glossary, which uses a noun rather than a verb, but it remains clear and predictable within the naming scheme.
With only 4 tools, the server is well-scoped for its focused purpose of test-suite auditing. Each tool serves a necessary and distinct function, and there are no redundant or superfluous entries.
The tools cover the complete workflow: audit to identify gaps, verify to test proposals, write_report to produce outputs, and glossary for reference. There are no obvious missing operations or dead ends for the stated domain.
Generates unit tests for Python code with coverage before/after reports and concrete edge cases.
Deterministic context layer for your codebase: change impact, blast radius, answers with receipts.
Pre-commit code quality guardian. Detects semantic drift in AI-generated code.
Ground-truth code graph for your codebase: exact callers, callees, symbols & dependencies.
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/openhonest/umbra'
If you have feedback or need assistance with the MCP directory API, please join our Discord server
