The OrangePro MCP server integrates with AI coding agents to analyze codebases, generate tests, assess quality risks, and manage the hosted platform. Here's what you can do:
Agent Management
List, inspect, run, and monitor OrangePro agents (orangepro_list_agents, orangepro_get_agent, orangepro_run_agent)
Review run history, fetch logs, and diagnose health/connectivity issues (orangepro_list_agent_runs, orangepro_get_agent_logs, orangepro_get_agent_health)
Knowledge Graph & Coverage Analysis
Resolve user stories, requirements, or features against the Knowledge Graph to find grounded entities and coverage gaps (orangepro_resolve_story)
View a heatmap of critical, partial, and healthy test coverage zones across your application (get_coverage_gaps)
Test Generation
Generate test cases for user stories or features lacking coverage (generate_missing_coverage)
Convert bug reports into regression tests to prevent recurrence (convert_bug_to_tests)
Build focused regression test suites for specific feature areas or risky changes (build_regression_pack)
Convert generated test cases into executable scripts for Playwright, Cypress, Selenium, or Puppeteer (generate_test_scripts)
Risk & Release Assessment
Score pull requests for quality risk, identify coverage gaps, and get recommended tests before merging (analyze_pr_risk)
Get a risk explanation using coverage heatmaps and 30-day trend data (explain_quality_risk)
Receive a ship/review/block recommendation with confidence score and risk areas for release decisions (analyze_release_readiness)
Supports generating and running tests with the Jest framework.
Supports generating and running tests with the Mocha framework.
Enables local test generation with Ollama models.
Enables test generation using OpenAI's models.
Supports generating and running tests with the pytest framework.
Supports generating and running tests with the Vitest framework.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@OrangePro MCP Check my code for untested behaviors and generate tests"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
OrangePro maps every public function in your repository, links each one to the tests that actually call it, and ranks what's left by what it can break. For a test you care about, it breaks the function in an isolated copy and checks that the test fails. It runs on your machine, uses no AI model for scoring, and sends nothing to OrangePro. Model calls happen only if you add your own key, for optional test generation and suggestions.
npx -y @orangepro/mcp-server@latest start .Related MCP server: mcp-codebase-oracle
Every run writes two reports and a spreadsheet of tests to .orangepro/:
File | For | What's in it |
| Leads, managers, anyone in a hurry | One page: what to do with the 20 highest-priority functions, the top five with the reason and the tests drafted for each, what changed since the last report, and how to reproduce the run. |
| Developers | Worklist: every ranked function with what to do, filters, and a side panel with the evidence, why it ranks, two drafted tests and the proof command. Map: where tests reach, by area. Run details: inputs, counts, proofs, fingerprints and settings. |
| QA, test managers | One row per test slot (preconditions, steps, test data, expected result, priority, folder), ready for Excel, Google Sheets, TestRail, Jira Xray or Zephyr Scale. Export tests (CSV) in the detailed report downloads it, and its Run details show how the columns map (Importing tests.csv). |
Every function gets one of four things to do:
What it means | What to do | |
Done | A test fails when this code breaks. | Nothing. It moves back to Prove if the code changes. |
Prove | A test runs it, or a coverage run executes it, but no test is proven to catch a break. | Run the proof command. |
Check | A test file or test name points here, but no call was traced. | Read that test file first; prove it if it drives the function, otherwise write one. |
Write | No test catches a break here yet. | Write a test (two are drafted for the top 20), then prove it. |
Both from OrangePro run on its own repository at commit 64a5213, without a model key, so no tests were drafted. With a key, each of the top 20 shows two drafted tests, and the side panel shows their steps and code.
It tells you what a test actually checks, not just what it runs. A test that calls a function isn't necessarily checking it. opro prove replaces the function body with a fixed return value in an isolated copy and reruns your own test, unchanged. If the test still passes, it wasn't protecting that function.
It puts consequences first. One filter shows the code paths that reach a delete or purge (cache evictions don't count); others show code that changes often, has many callers or runs on a schedule.
It's honest about evidence. A function is "linked" only when a test calls that exact function. A test with a similar name is a lead, never coverage. Tests that only check mocks don't count.
It's repeatable. Same commit, full git history, same config and same version give the same ranking. Each report records fingerprints, so you can tell when a change in results came from the code and when it came from the tool.
Your coding agent can drive it. As an MCP server it gives Claude Code, Cursor, Copilot, Codex and others the ranked gaps, the suggested test location and the proof step in one loop.
It's free, local and open source (MIT). No account, no API key for analysis, and no upload.
cd /path/to/your/repo
# install the repo's own dependencies first (npm ci, uv sync, go mod download, ...)
npx -y @orangepro/mcp-server@latest start .
open .orangepro/short_behavior-coverage.htmlAnalysis, ranking and proof need no model key. Test generation is optional and uses your own key (see Model setup).
Tips for the most accurate run:
Use a full clone, not a shallow one. Change history drives part of the ranking, and the report says when history was partial.
Exclude what isn't product code with rank_exclude_paths (see Configuration).
Rerun after a change. Each report says what changed since the last one. When nothing changed (same commit, files, settings, coverage and proofs), start reuses the last reports at once and says so; --fresh runs in full anyway.
What's written:
.orangepro/
├── short_behavior-coverage.html ← one-page summary
├── behavior-coverage.html ← detailed report: worklist, map, run details
├── tests.csv ← drafted tests, one row per test
├── report-data.json ← the data both reports show (used to compare and reuse runs)
├── run-key.json, last-start.json ← what the last run saw, so an unchanged repo is not analysed again
├── graph.json ← the evidence graph (deterministic)
├── ledger.json ← proof certificates
├── rtm.md ← traceability matrix
└── config.json ← optional per-repo settings
orangepro_generated/ ← generated tests (only with a model key); your files are never editedEvery function gets exactly one evidence label. The four things to do above are read from it.
Evidence | What it means | To do |
Proven | A test passed on the original code and failed at its own assertion when this function was broken in an isolated copy. | Done |
Runs under tests | A coverage tool you ran executed this code. | Prove |
Linked | A test calls this exact function. That's a structural link, not proof. | Prove |
Unconfirmed | A test file or a test with a similar name points here, but no call was traced. That's a lead, not evidence. | Check |
No test found | Nothing links a test to it. | Write |
A proof that ran and did not catch the break turns the function into Write; a proof made before the code changed turns it back into Prove.
What OrangePro doesn't count, so linked numbers are a floor, not a coverage percentage:
tests that only assert on mocks;
calls made over HTTP or from end-to-end suites;
in Python, objects that reach a test only through a fixture.
Check the tests before writing new ones for a flagged path.
"Proven 0" is normal on a first run. Proof runs your tests, so it happens only for the functions you choose, or within the attempt budget of
opro start.
Each unproven function gets an OrangePro Risk Score, ORS = P × I × D:
P: how likely it is to change. Change history, fan-out, new code and size.
I: what it can break. Incoming references, entry-point position, data sensitivity, and a floor for paths that reach a delete or purge within two calls.
D: how hard a break would be to notice. Evidence tier (a linked test or a coverage run makes a break easier to notice than a name match or nothing), and whether it runs unattended (jobs and schedulers).
The score sets the order of work. It is not a defect probability. The weights are fixed and no AI model is involved. The report shows the inputs behind every row.
# Python: the mutation value is derived from the function (return annotation or its observed result)
opro prove-loop --target-symbol 'sym:app/billing/invoices.py#void_invoice' \
--test 'tests/test_invoices.py::test_void_marks_invoice_void' --replacement sentinel
# TypeScript / JavaScript: give the inert body to substitute
opro prove-loop --target-symbol 'sym:src/orders.service.ts#OrdersService.cancel' \
--test src/orders.service.spec.ts --replacement 'return null;'Proven: the test passes on the original code and fails at its own assertion on the broken copy.
Not proven: the test still passes on the broken copy, so it doesn't protect that function. That's a finding too.
Unrunnable: setup failed. This is never counted either way.
Find weak tests without a model key:
opro roast . # passing tests whose targeted mutant still survivesOrangePro runs as an MCP server. Add it to your client's config:
{
"mcpServers": {
"orangepro-local": {
"command": "npx",
"args": ["-y", "@orangepro/mcp-server@latest", "mcp"]
}
}
}Client | Where to put it |
Claude Code |
|
Cursor |
|
VS Code / Copilot | MCP settings |
Codex / OpenCode / Windsurf |
|
A prompt that runs the whole loop:
"Use
orangepro_start, thenorangepro_find_test_gaps. For the top gap, write a test at the suggested path, run it, then callorangepro_prove_loopand tell me whether it was proven."
Optional. Put it in .orangepro/config.json in the repository, or in ~/.orangepro/config.json for defaults across repositories. Every setting that changes the ranking is shown in the report.
{
"classification": {
"rank_exclude_paths": ["ui/**", "docs/**", "scripts/**"],
"test_support_paths": ["internal/testutil/**"],
"destructive_sinks": ["archive*"]
},
"tuning": { "churn_window_days": 180 },
"proof": { "python_runner": "auto", "attempt_limit": 20 },
"overrides": [
{ "symbol": "sym:src/legacy/shim.ts#shim", "action": "suppress", "reason": "generated shim, not product code" }
]
}rank_exclude_paths removes paths from the ranking. They are still counted as behaviors.
destructive_sinks adds delete-like calls for your codebase.
Overrides (suppress, pin, reclassify) each require a reason, and the report lists them.
Weights and tiers can't be configured.
Language | Map functions | Link tests that call them | Draft tests | Mutation proof |
TypeScript / JavaScript | ✓ | ✓ | ✓ Jest, Vitest, Mocha | ✓ Jest, Vitest, Mocha |
Python | ✓ | ✓ | ✓ pytest | ✓ pytest |
Go | ✓ | ✓ | ✓ | ✓ |
Java | ✓ | ✓ | ✓ JUnit 4 and 5 | ✓ Maven with JUnit 5 only |
Kotlin, Rust, PHP, C#, Ruby, Swift, C, C++ | ✓ | name match only | not yet | not yet |
"Name match only" means a test file or test name that looks like the function makes it Unconfirmed (Check). No call is traced, so these languages never reach Linked or Proven today.
TypeScript and JavaScript are parsed with the TypeScript compiler; the other languages with tree-sitter.
Python proof runs pytest through the repository's own environment: an existing .venv or venv, then uv or poetry when their lock files are present, else python -m pytest. Set proof.python_runner to use your own command.
Java proof needs Maven (Surefire) and JUnit 5. Gradle and JUnit 4 projects are mapped and linked, but not proven yet.
Not traced yet, so tested code can show as Unconfirmed: calls through Go test-suite methods and helper constructors, dependency injection that is wired at runtime, reflection, and calls made over HTTP or RPC.
Node.js 20 or newer, on macOS or Linux. Windows isn't tested: proofs use POSIX paths and symlinks.
Git, for change history. Without it, or in a shallow clone, the ranking uses structure only and the report says so. A folder that isn't a git checkout always runs in full.
Install the repository's dependencies first (npm ci, uv sync, go mod download, mvn -q install, …). Proof runs your own tests in an isolated copy and needs them to pass there.
What's read: source and test files, skipping anything in .gitignore or .orangeproignore (negations with ! aren't supported), the usual build and dependency folders (node_modules, vendor, dist, build, target, coverage, .venv, …), files over 1 MB, and generated code (files marked Code generated … DO NOT EDIT). Up to 100,000 files; ORANGEPRO_MAX_FILES lowers that and ORANGEPRO_MAX_ANALYZE_MS sets a time limit for the scan.
Test drafting covers the 20 highest-priority functions, two tests each, in the order the worklist shows, and the reports show those 20. Drafts are kept until the function's code changes.
What counts as "nothing changed" for reusing a run: the commit, every modified or untracked file that git doesn't ignore, OrangePro's settings and ignore files, coverage files, the proof ledger, the start options, whether a model key is set, and the analysis version. A run whose test drafting failed on a provider error (out of credits, rate limit) runs again in full.
The CSV column mapping follows each tool's CSV importer (see Importing tests.csv). Check the import preview the first time.
Nothing about your code or your runs is sent anywhere. There is no usage telemetry.
Source is read in-process and never stored or uploaded. Reports and the evidence graph contain metadata, not your source.
Your source files are never edited. Proofs run in an isolated copy.
Model calls happen only if you configure a key, and they go directly from your machine to the provider you chose. Keys are read from your environment or from a .env.provider.local, .env.local or .env file in the repository, and OrangePro never writes them anywhere. opro setup saves only the provider and model name.
With a key, the reports and tests.csv include the tests your model drafted, and the prompts include the function names, files and evidence they're based on.
Reports load nothing from the network. The only outbound links are ones you click.
Reports include a Give feedback link and a This looks wrong link on each finding. Once in a while, after the findings, they also ask whether the report helped.
The links carry nothing about your project. The form sends only what you review and submit.
You stay anonymous unless you leave an email.
opro feedback # print the link and your settings
opro feedback off # stop the "Did this help?" question (links stay)
ORANGEPRO_FEEDBACK_URL=off # hide every feedback linkThe question appears at most once every 14 days (ORANGEPRO_FEEDBACK_COOLDOWN_DAYS). That preference is stored only on your machine. MCP clients get the link as optional result metadata (_meta["ai.orangepro/feedback_url"]), and your agent is never asked to prompt you for feedback.
opro # same as opro start .
opro start . --no-ai --no-auto # analyze + reports, no model calls, no proof attempts
opro start . --fresh # run in full even when nothing changed since the last run
opro start --base main # scope to a branch diff
opro analyze # build the evidence graph and reports
opro gaps --limit 10 # top unproven behaviors, ranked
opro prove-loop ... # mutation proof for one function (see above)
opro roast . # find tests whose mutant survives (no key needed)
opro doctor --proof # why top targets aren't proven yet
opro generate --base main # tests for what this branch changed (needs a model key)
opro rtm # traceability matrix
opro export # metadata-only evidence pack
opro coverage # find or generate runtime coverage artifacts
opro feedback [on|off] # feedback link and invitation setting
opro mcp # run as an MCP server (stdio)Add --json to any read command for machine output. Run opro help for every flag.
Tool | What it does |
| Analyze, write both reports and |
| Build or refresh the evidence graph |
| Unproven behaviors ranked by ORS |
| Setup + mutation proof + report refresh for one behavior |
| Mutation proof only |
| Grounded tests for gaps (model key required) |
| What a diff touches |
| Workspace state without running anything |
| What evidence to add next |
| Graph readiness (0–100) |
| Traceability matrix |
| Aggregate statistics |
| Record a test run result |
| Why a test was generated |
| Metadata-only evidence pack |
| Incremental graph update |
| Weak behavior→code suggestions (optional AI, never evidence) |
| Candidate flows (optional AI, never evidence) |
Run your own unit and integration coverage first, then opro start. OrangePro ingests Go coverprofiles, lcov, coverage.py XML and JaCoCo XML, and keeps unit, integration and unclassified coverage separate. To label artifacts, add .orangepro/coverage-suites.json:
{
"artifacts": {
".orangepro/coverage/unit.coverprofile": { "suite": "unit", "command": "make unit-test-coverage" },
".orangepro/coverage/integration.coverprofile": { "suite": "integration", "command": "make integration-test-coverage" }
}
}Provider | Environment variable |
OpenAI-compatible |
|
Anthropic |
|
Ollama (local, no key) |
|
Auto-detect order: OpenAI → Ollama → Anthropic. Override with --provider and --model, or run opro setup.
Model output never changes an evidence tier. Only the mutation proof can mark a behavior Proven.
Generated tests land in orangepro_generated/ with the files and existing tests they're grounded on, and where and how to run them.
Drafts whose code fails the compile check are kept as plans ("Plan only"), with the reason named. Without a key, each of the top 20 shows two empty slots with a prompt you can copy into your coding agent.
One row per test, UTF-8. The first 12 columns are for import; the rest keep the priority, evidence and proof command with each test. Priority is High for the five highest-ranked functions and Medium for the rest.
Column in | TestRail | Jira Xray | Zephyr Scale |
ID | Test Case Identifier | (leave Key empty) | |
Title | Title | Summary | Name |
Section ( | Sections Hierarchy | ||
Folder ( | Test Repository Path | Folder | |
Priority | Priority | Priority | Priority (map Medium to Normal) |
Test type | Type | Labels | Labels |
Preconditions | Preconditions | (separate issues in Xray; add to Description if you want them) | Precondition |
Steps | Manual Test Step: Action | ||
Test data | Manual Test Step: Data | ||
Steps with data | Steps | Test Script (Steps) - Step | |
Expected result | Expected Result | Manual Test Step: Expected Result | Test Script (Steps) - Expected Result |
References | References | Description | Objective |
TestRail: Import from CSV with the Test Case (Text) template, one row per test case.
Jira Xray: Test Case Importer; each test imports as one manual step.
Zephyr Scale: it has no test-data field, so Steps with data carries it.
Check the importer's preview before you confirm.
opro generate --base main # tests for what this branch changed (read-only git diff)
opro generate --changed # current branch vs its base
opro generate --pr 1234 # checks out PR #1234 (asks first; refuses on a dirty tree)This repository is the free local tool. The OrangePro platform adds:
a persistent graph across PRs and repositories;
CI gates on evidence and risk changes;
requirement and incident correlation;
team dashboards.
git clone https://github.com/OrangeproAI/orangepro-mcp.git
cd orangepro-mcp && npm ci && npm run build
npm testPRs welcome. Please open an issue first for large changes.
Analyze a pull request for quality risk. Returns overall risk score (0-100), risk drivers, impacted categories, similar historical bugs, coverage gaps, and recommended tests to run. Use before merging to catch regressions.
| Name | Required | Description | Default |
|---|---|---|---|
| pr_title | Yes | Pull request title. | |
| changed_files | No | List of changed file paths (e.g., ['src/checkout.ts', 'src/payment.ts']). | |
| pr_description | No | Pull request body, summary, or description of changes. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true, idempotentHint=true, destructiveHint=false, so description does not need to disclose safety. Adds output details (risk score, drivers, etc.) but no further behavioral traits like rate limits or auth needs.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences: first states purpose, second adds usage and output highlights. No redundancy or fluff.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with 3 parameters (all well-documented in schema) and no output schema, the description gives a solid overview of the return value and usage context. Could mention prerequisites like repository access, but overall complete enough.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so each parameter is already documented. Description adds no additional parameter meaning beyond what schema provides, but it does describe output structure which is not part of parameter semantics.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description states 'Analyze a pull request for quality risk' with a specific verb and resource, and distinguishes from siblings like analyze_release_readiness and generate_missing_coverage by focusing on risk analysis.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly states 'Use before merging to catch regressions,' providing clear context. Does not mention when not to use or specific alternatives, but sibling list offers alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Get a tenant-wide release readiness assessment. Returns a ship/review/block recommendation with confidence score, coverage analysis, execution summary, script readiness, risk areas, recent failures, and recommended actions. Use before deciding whether to release.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations indicate readOnlyHint=true, so the description properly confirms a read operation. It adds value by detailing the contents of the assessment (coverage analysis, execution summary, etc.), leaving no ambiguity about what the tool returns.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description consists of two efficient sentences: the first defines purpose and output, the second gives usage. No unnecessary words; content is front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite lacking an output schema, the description enumerates all key return components (recommendation, confidence score, coverage analysis, etc.), fully informing the agent of what to expect. This compensates for the missing schema.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters and schema coverage is 100%. No parameter information is needed, and the description adds nothing about parameters, which is appropriate. Baseline for 0 parameters is 4.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: 'Get a tenant-wide release readiness assessment.' It lists specific outputs (recommendation, confidence score, coverage analysis, etc.), making it distinct from sibling tools like analyze_pr_risk which focus on individual PR risk.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides explicit usage guidance: 'Use before deciding whether to release.' This clearly indicates when to use the tool, though it does not explicitly mention when not to use it or alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Generate a focused regression test pack for a feature area or recent change. Use after a refactor, migration, or risky change to ensure the area stays stable. Returns a set of test cases targeting the specified area.
| Name | Required | Description | Default |
|---|---|---|---|
| area | Yes | Feature area, system, or flow to protect (e.g., 'checkout', 'user authentication', 'payment processing'). | |
| context | No | Recent change description or additional risk context (e.g., 'migrated payment provider from Stripe to Adyen'). |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already indicate idempotentHint=true and destructiveHint=false. The description adds that the tool 'Returns a set of test cases', which is a fair behavioral note. However, it does not elaborate on permissions, rate limits, or other nuances. With annotations covering safety, a 3 is appropriate.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is three sentences with no waste: purpose, usage guideline, and output. Front-loaded with the core action. Every sentence earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has only 2 parameters, no output schema, and annotations present, the description is adequately complete. It explains the output as a 'set of test cases', which is sufficient. Could mention idempotency explicitly, but annotations cover it. Slightly vague on output format, but not a major gap.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Both parameters (area, context) are fully described in the input schema with 100% coverage. The description adds minimal extra meaning beyond repeating schema descriptions. Baseline 3 is correct as the schema handles most of the parameter documentation.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Generate' and the resource 'regression test pack'. It specifies the focus on a feature area or recent change, distinguishing it from sibling tools like 'generate_missing_coverage' or 'analyze_pr_risk' which serve different purposes.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly states when to use: 'after a refactor, migration, or risky change'. Provides clear context with the example. While it doesn't explicitly state when not to use or directly compare with siblings, the guidance is strong and sufficient for most agents.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Analyze a bug report and generate durable regression tests to prevent recurrence. Provide a detailed bug description for best results. Returns root cause analysis, affected areas, and generated test cases with steps.
| Name | Required | Description | Default |
|---|---|---|---|
| severity | No | Bug severity hint — affects test priority ranking. | |
| bug_description | Yes | Detailed bug report: what happened, expected behavior, actual behavior, and reproduction steps. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already indicate idempotent (idempotentHint=true) and non-destructive (destructiveHint=false). The description adds value by stating the tool returns 'root cause analysis, affected areas, and generated test cases with steps,' clarifying behavioral outcomes beyond what annotations provide.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two concise sentences, front-loaded with the core action and outputs, with no extraneous words. Every sentence adds value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the moderate complexity (2 parameters, 100% schema coverage, no output schema), the description sufficiently explains inputs and outputs. It could mention any prerequisites (e.g., existence of a test harness) but is broadly complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3. The description adds semantics: it explains that 'severity' influences 'test priority ranking,' which is not in the schema description. This extra context helps the agent use the parameter effectively.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb ('analyze' and 'generate') and resource ('bug report' and 'regression tests'), making the tool's purpose clear. It does not explicitly differentiate from siblings like 'build_regression_pack' or 'generate_test_scripts', but the focus on bug-to-test conversion is distinct.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description advises providing a 'detailed bug description for best results,' which gives a usage hint but does not specify when to use this tool versus alternatives (e.g., when to use 'generate_test_scripts' instead) or when not to use it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Get a quality risk assessment using coverage heatmap, execution history, and 30-day trend data. Identifies high-risk and medium-risk areas. Use to answer questions like 'are we safe to ship?' or 'what areas need more tests?'
| Name | Required | Description | Default |
|---|---|---|---|
| area | No | Focus the risk assessment on categories matching this string. | |
| question | No | Specific quality question to answer (e.g., 'Is the checkout flow well-tested?'). |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already indicate readOnlyHint=true and destructiveHint=false, so the agent knows this is safe. The description adds behavioral context: it uses coverage heatmap, execution history, and 30-day trend data to identify high- and medium-risk areas. This goes beyond annotations, though it does not detail data sources or outputs fully.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences, front-loaded with the tool's core action and data sources. Every sentence serves a purpose: the first explains what and how, the second gives usage examples. No extraneous words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema, the description partially compensates by stating it identifies high- and medium-risk areas. Parameters are well-documented. However, it does not specify the output format (e.g., a list, scores, or visualization) or how the risk assessment is presented, which could leave the agent uncertain about return structure.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, with both parameters described sufficiently in the input schema. The description adds example questions but does not elaborate on the 'area' parameter beyond what the schema says. It provides minimal additional semantic value over the schema, so a baseline of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool retrieves a quality risk assessment using specific data sources (coverage heatmap, execution history, 30-day trend data) and identifies high/medium risk areas. Example questions like 'are we safe to ship?' further clarify purpose. The name 'explain_quality_risk' matches, and it is distinct from siblings like 'analyze_pr_risk' or 'analyze_release_readiness' which focus on different contexts.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides explicit usage guidance via example questions ('are we safe to ship?', 'what areas need more tests?'), indicating when to use this tool. It does not explicitly state when not to use it or mention alternative tools, but the sibling list implies alternatives exist and the description sets a clear context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Generate test cases for a user story or feature that needs better coverage. Submits a test generation job and polls for results (up to 2 minutes). Returns categorized test cases with steps and expected results.
| Name | Required | Description | Default |
|---|---|---|---|
| app_domain | No | Application domain (e.g., 'E-Commerce', 'Banking', 'Healthcare') to tailor test patterns. | |
| user_story | Yes | User story, feature description, or coverage gap to generate tests for. | |
| app_context | No | Application overview or technical context to improve test relevance. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already indicate readOnlyHint=false (mutation) and destructiveHint=false. The description adds that the tool submits a job and polls for results up to 2 minutes, which is key behavioral context beyond annotations. It does not cover failure handling or rate limits, but the polling timeout is valuable.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences that front-load the core purpose and then describe the process and output. Every word adds value; no fluff or repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema, the description informs about the return format (categorized tests with steps and expected results). It also mentions the polling timeout. This is fairly complete for a job-submission tool, though it could mention potential error states or job failure behavior.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and each parameter already has a description. The tool description does not add new semantics beyond summarizing the overall functionality. Baseline 3 is appropriate as the schema already does the heavy lifting.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it generates test cases for stories/features needing coverage, specifies the job submission and polling process with a 2-minute timeout, and mentions the output format (categorized tests with steps and expected results). This is specific and distinct from sibling tools like generate_test_scripts or convert_bug_to_tests.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage 'for a user story or feature that needs better coverage,' but does not explicitly compare to sibling tools or provide when-not-to-use guidance. For example, it doesn't contrast with 'generate_test_scripts' or 'convert_bug_to_tests,' leaving the agent to infer the best context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Convert test cases from a completed test generation job into executable test scripts. Requires a source_job_id from a prior generate_missing_coverage or convert_bug_to_tests call. Generates scripts for Playwright, Cypress, Selenium, or Puppeteer. Use this as the second step after generating test cases to get runnable automation code.
| Name | Required | Description | Default |
|---|---|---|---|
| framework | No | Target test framework. 'all' generates for all frameworks. Defaults to playwright. | |
| app_domain | No | Application domain for context (e.g., 'E-Commerce'). | |
| source_job_id | Yes | Job ID from a completed test generation run (from generate_missing_coverage output). | |
| test_case_ids | No | Specific test case IDs to convert. If omitted, converts all test cases from the source job. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations indicate a non-readOnly and non-destructive operation. The description adds that it generates runnable automation code and mentions specific frameworks, but does not disclose rate limits, authentication needs, or details about side effects beyond the conversion.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise with four sentences, front-loading the purpose. Every sentence adds value without redundancy or wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given 4 parameters, 1 required, and no output schema, the description adequately covers the workflow: source job, framework selection, and optional test case filtering. It omits return format but is sufficient for agent invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with descriptions for all 4 parameters. The description adds minimal extra meaning, e.g., tying framework to the four options and mentioning the default. Since schema already does the heavy lifting, a 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool converts test cases into executable test scripts, specifies the required source_job_id, and lists supported frameworks. It distinguishes itself from siblings like generate_missing_coverage and convert_bug_to_tests by positioning itself as the second step.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly requires a source_job_id from a prior generate_missing_coverage or convert_bug_to_tests call, and frames this as the second step. It does not explicitly exclude any scenarios or mention alternatives, but the context is clear.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Find application areas lacking test coverage. Returns a heatmap of critical (red), partial (yellow), and healthy (green) coverage zones with test counts. Use to identify where to generate additional tests.
| Name | Required | Description | Default |
|---|---|---|---|
| area | No | Filter results to categories matching this string (case-insensitive). |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already mark the tool as read-only and non-destructive. The description adds minor behavioral context (color coding, test counts) but does not need to go beyond annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two efficient sentences convey purpose, output, and usage without waste.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple tool with one optional parameter and no output schema, the description fully covers what the tool does and what to expect.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema already provides full description for the single parameter 'area' (case-insensitive filter). The tool description adds no further parameter details, so baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool finds application areas lacking test coverage and returns a heatmap with colored zones and test counts. It distinguishes from siblings like analyze_pr_risk or generate_missing_coverage.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explicitly says 'Use to identify where to generate additional tests,' providing clear context. However, it does not mention when not to use or compare directly with sibling tools like generate_missing_coverage.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Get full detail, configuration, timeline, and recent runs for a specific OrangePro agent. Use after orangepro_list_agents to inspect a particular agent. Returns agent config, run history, and current status.
| Name | Required | Description | Default |
|---|---|---|---|
| agent_id | Yes | The agent_id from orangepro_list_agents. | |
| tenant_id | No | OrangePro tenant id. Defaults to ORANGEPRO_TENANT_ID env var. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true and destructiveHint=false, so the agent knows this is a safe read-only operation. The description adds that it returns 'agent config, run history, and current status,' which is helpful but does not go beyond what annotations guarantee. No contradictions with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences long, each serving a distinct purpose: first sentence defines the function, second sentence gives usage guidance and return summary. No filler words, front-loaded with the key action.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given that there is no output schema, the description adequately covers what the tool returns ('agent config, run history, and current status'). The tool is a single-agent getter, and the sibling tools cover more specialized queries, so this description is complete enough for its role.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% with both agent_id and tenant_id described. The description adds value by clarifying that agent_id comes 'from orangepro_list_agents' and that tenant_id 'Defaults to ORANGEPRO_TENANT_ID env var,' which is useful context beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: 'Get full detail, configuration, timeline, and recent runs for a specific OrangePro agent.' It uses a specific verb ('Get') and a well-defined resource ('OrangePro agent'), and differentiates itself from sibling tools like orangepro_list_agents by focusing on inspecting a single agent in depth.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly says 'Use after orangepro_list_agents to inspect a particular agent.' This provides clear context for when to use the tool. It does not explicitly state when not to use it or list alternatives, but the sibling tools (e.g., orangepro_get_agent_health, orangepro_get_agent_logs) suggest specialized uses, and the description implies this is a general inspection tool.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Read health and connectivity status for an OrangePro agent. Use to diagnose why an agent is failing — checks source config, auth, and runtime status.
| Name | Required | Description | Default |
|---|---|---|---|
| agent_id | Yes | The agent_id to check health for. | |
| tenant_id | No | OrangePro tenant id. Defaults to ORANGEPRO_TENANT_ID env var. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint and destructiveHint. The description adds valuable context about what is checked (source config, auth, runtime status), surpassing the annotations. No contradictions.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, no wasted words. First sentence concisely states purpose, second offers usage guidance. Well-structured and front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple health check tool with only 2 parameters and no output schema, the description fully covers purpose, usage, and behavior. No gaps.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so parameters are well-documented. The description does not add parameter-level details but provides context for the tool's overall function. Baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Read' and resource 'health and connectivity status', and specifies it diagnoses agent failure by checking source config, auth, and runtime status. It effectively distinguishes from siblings like orangepro_get_agent or orangepro_get_agent_logs.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly states 'Use to diagnose why an agent is failing', providing a clear context. However, it does not explicitly mention when not to use or list alternatives, though sibling tools are available.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Read recent log lines for an OrangePro agent. Use to debug failures, check processing details, or verify what an agent did during a run. Returns timestamped log lines.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | Max number of log lines to return. Default 100. | |
| offset | No | Number of log lines to skip for pagination. | |
| agent_id | Yes | The agent_id to get logs for. | |
| tenant_id | No | OrangePro tenant id. Defaults to ORANGEPRO_TENANT_ID env var. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true and destructiveHint=false, so description's additional mention of 'timestamped log lines' adds minor value beyond annotations. No contradictions.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, front-loaded with verb and resource. Every sentence is informative and there is no waste.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With annotations and full schema coverage, the description adequately explains the tool's purpose and behavior. Lacks explicit details on pagination, but schema covers limit/offset. Sufficient for the complexity level.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, and description does not add significant meaning beyond what the schema already provides for each parameter. Baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states verb ('Read'), resource ('log lines for an OrangePro agent'), and purpose with specific use cases (debug, check details, verify). Distinguishes from siblings like 'orangepro_get_agent' or 'orangepro_list_agent_runs'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly states when to use: 'to debug failures, check processing details, or verify what an agent did during a run.' Does not provide explicit exclusions or compare to siblings, but context is clear.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
List recent runs for a specific OrangePro agent. Use to check run history, find failed runs, or verify a recent run completed. Returns run_id, status, start time, duration, and records processed.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | Max number of runs to return. Default 20. | |
| offset | No | Number of runs to skip for pagination. | |
| agent_id | Yes | The agent_id to list runs for. | |
| tenant_id | No | OrangePro tenant id. Defaults to ORANGEPRO_TENANT_ID env var. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true and destructiveHint=false, so the description adds value by listing return fields (run_id, status, etc.). However, it does not disclose pagination behavior or ordering, which would be useful.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences, front-loaded with purpose, and no wasted words. Every sentence adds value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema, the description includes return fields. It covers the main usage but could mention default ordering (e.g., recent first) or pagination hints. Overall adequate.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents all parameters. The description does not add significant extra meaning beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'List recent runs for a specific OrangePro agent' with a specific verb and resource. It distinguishes from sibling tools like orangepro_list_agents (lists agents) and orangepro_run_agent (run agent).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides explicit use cases: 'check run history, find failed runs, or verify a recent run completed.' It does not explicitly mention when not to use or alternatives, but the context is clear.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
List all configured OrangePro agents for a tenant. Use this first to discover available agents before getting details or triggering runs. Returns agent_id, name, type, status, and last run timestamp for each agent.
| Name | Required | Description | Default |
|---|---|---|---|
| tenant_id | No | OrangePro tenant id. Defaults to ORANGEPRO_TENANT_ID env var. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true and destructiveHint=false, indicating a safe read operation. The description confirms this by stating it returns a list of agents with specific fields, but does not add behavioral context beyond what annotations provide.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences, front-loaded with the action and purpose, and contains no unnecessary words. It efficiently conveys the primary use case and return fields.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a list tool with no output schema, the description compensates by listing the returned fields (agent_id, name, type, status, last run timestamp). It does not mention pagination or ordering, but for a simple discovery tool, it is sufficiently complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has 100% coverage for its single optional parameter (tenant_id) with a description. The tool description does not add additional parameter guidance beyond what is in the schema, so the baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it lists all configured OrangePro agents for a tenant, and specifies the returned fields (agent_id, name, type, status, last run timestamp). It distinctly differentiates itself from sibling tools like orangepro_get_agent (single agent) and orangepro_list_agent_runs.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly advises to use this tool first to discover available agents before getting details or triggering runs, which provides clear guidance on when to use it. It does not list when not to use it or alternatives, but the context from sibling names implies the workflow.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Resolve a user story, requirement, or feature description against the OrangePro Knowledge Graph. Returns grounded entities, matched concepts, and confidence scores. Use to verify story coverage or find KG gaps.
| Name | Required | Description | Default |
|---|---|---|---|
| top_k | No | Number of top matches to return. Defaults to 5. | |
| tenant_id | No | OrangePro tenant id. Defaults to ORANGEPRO_TENANT_ID env var. | |
| input_kind | No | Input type: 'story', 'requirement', or 'feature'. Defaults to 'story'. | |
| story_text | Yes | The user story, requirement, or feature text to resolve against the KG. | |
| source_type | No | Source type: 'manual', 'jira', or 'github'. Defaults to 'manual'. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare idempotentHint=true, destructiveHint=false. The description adds that it returns entities and scores, but does not disclose additional traits like required permissions or rate limits. With annotations covering safety, a 3 is appropriate.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is three sentences, front-loading the action and outcome. Every sentence is informative and there is no redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given 5 parameters and no output schema, the description mentions return types (entities, concepts, confidence scores) which aids completeness. It could elaborate more on 'resolve' semantics or interpretation of scores, but overall it is sufficient.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3. The description provides no additional detail on parameters beyond what the schema already describes. No extra meaning is added.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb 'resolve' and resource 'OrangePro Knowledge Graph', clearly stating the tool's function. It lists return values (grounded entities, matched concepts, confidence scores) and allows it to be distinguished from sibling tools that deal with agents or test generation.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly says 'Use to verify story coverage or find KG gaps', providing clear usage context. However, it does not mention when to avoid this tool or suggest alternatives among siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Start an OrangePro agent run. Safe to retry — the API returns the active run if one is already in progress. Use this to trigger data ingestion, KG sync, or test generation agents.
| Name | Required | Description | Default |
|---|---|---|---|
| agent_id | Yes | The agent_id to run. Get this from orangepro_list_agents. | |
| tenant_id | No | OrangePro tenant id. Defaults to ORANGEPRO_TENANT_ID env var. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description confirms the idempotentHint annotation by stating it's safe to retry and explains the API returns the active run if in progress, adding behavioral context beyond annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two short sentences, front-loaded with the action and safety note. No wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with 2 parameters and no output schema, the description covers purpose, idempotency, common uses, and parameter origins. Nothing is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with descriptions for both parameters. The description adds valuable context: agent_id is obtained from list_agents, tenant_id defaults to an environment variable.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses specific verbs ('Start'), identifies the resource ('OrangePro agent run'), and lists concrete use cases (data ingestion, KG sync, test generation). It clearly distinguishes from sibling tools like orangepro_list_agents.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly states it is safe to retry and provides example use cases. While it doesn't list when not to use it or mention alternatives, the context and sibling tool list imply appropriate usage.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.2.0analyze_pr_riskanalyze_release_readinessbuild_regression_packconvert_bug_to_testsexplain_quality_riskgenerate_missing_coveragegenerate_test_scriptsget_coverage_gapsorangepro_get_agentorangepro_get_agent_healthorangepro_get_agent_logsorangepro_list_agent_runsorangepro_list_agentsorangepro_resolve_storyorangepro_run_agentScored across 15 tools
Tools are mostly distinct with clear purposes, though 'explain_quality_risk' and 'analyze_release_readiness' both address shipping safety and could be confused. The orangepro_ group is well-differentiated.
Mixing snake_case verbs (analyze_pr_risk, build_regression_pack) with lowercase phrases (explain_quality_risk) and prefixed names (orangepro_list_agents) creates inconsistency. Some tools lack a consistent verb_noun pattern.
15 tools is slightly above the ideal range but still reasonable for a quality and release management domain. Each tool serves a distinct purpose without unnecessary redundancy.
Covers risk analysis, test generation, and agent management, but lacks test execution, result reporting, and agent creation/deletion. Gaps exist in the full lifecycle of quality assurance.
Generates unit tests for Python code with coverage before/after reports and concrete edge cases.
Discover software problems, analyze evidence, and create implementation-ready Build plans.
Deterministic context layer for your codebase: change impact, blast radius, answers with receipts.
AI Agent with Architectural Memory. Impact analysis (free), tests and code from the graph (pro).
