Skip to main content
Glama

Advanced Zebrunner MCP Server

version

A Model Context Protocol (MCP) server that brings advanced analytics, reporting, failure forensics, and safe TCM mutations to Zebrunner. Designed to coexist with the official Zebrunner MCP (beta) โ€” see the "Dual-MCP setup" section below.

Tool naming: every tool on this server is registered under the canonical adv_<name> form (e.g. adv_create_test_case, adv_list_test_runs) so it never collides with the official Zebrunner MCP. The legacy names are kept as deprecated aliases so prompts/scripts that called the old names continue to work for now; aliases will be removed in the next major release.

๐Ÿ†• v9.2.8 โ€” New adv_analyze_test_impact + /test-impact prompt for PR/code-change test planning (regression + coverage gaps). See TEST_IMPACT_WORKFLOW.md and release notes.

๐Ÿ†• v9.2.7 โ€” New adv_scaffold_test_case wizard (hybrid form/conversational) to author test cases from best practices with an automatic warn-only similar-case check, plus configurable projectAliases in zebrunner-config.json. See release notes.

๐Ÿ†• v9.2.5 โ€” adv_get_test_authoring_trend (TAM template 7) completes 22/22 dashboard widget MCP coverage. Hub tools, pass-rate views, period modes. See TEST_PROMPTS.md ยง18 and change-logs.md.

v9.2.1 โ€” LLM-visible metrics (include_call_metrics, session breakdown), compact expansion (+19 tools), format/truncation bugfixes. See Token efficiency guide and change-logs.md.

v9.2.0 โ€” Opt-in token/cost optimizations: format:'compact', detail:'summary', adv_batch_get_test_cases, report inline:false. Defaults unchanged. See change-logs.md.

v9.1.0 โ€” Launch mutations: adv_rerun_launch_failures, adv_start_launch (Jenkins Build Now), plus /relaunch-regression-failures and /feature-scoped-launch prompts. See GitHub Release v9.1.0.

๐Ÿ“– Need help with installation? Check out our Step-by-Step Install Guide for detailed setup instructions.

๐Ÿš€ Installing via npm? See our MCP NPM Installation Guide for Claude Desktop, Cursor, IntelliJ IDEA, and ChatGPT Desktop configuration.

๐Ÿ“š Test vs Test Case vs Launch? See Terminology & Glossary โ€” definitions, counting rules, and how metrics are calculated.

๐Ÿ”ฅ Why This Server

This is the Advanced Zebrunner MCP Server โ€” built to go well beyond basic test case management and help QA teams work smarter and faster with AI. Compared to the official Zebrunner MCP (beta, ~70 tools spanning Public REST + Reporting/TAM/Launcher), this server provides 69 analytics-focused tools (adv_* prefix) and is safe to run side-by-side with the official server:

  • Reporting & Analytics โ€” dashboards, pass-rate trends, regression stability reports, runtime efficiency analysis, bug reviews, and weekly delta tracking

  • Test Coverage & Analysis โ€” coverage gaps, automation readiness scoring, and cross-suite analysis

  • Intelligent Rules System โ€” 3-tier quality rules with 100+ automated checkpoints for test case validation and improvement

  • Screenshot & Video Analysis โ€” OCR, image processing, and video analysis powered by sharp, ffmpeg, and tesseract

  • MCP Resources & Prompts โ€” @ context injection and / workflow commands for streamlined daily operations

  • Multi-Transport & Multi-Auth โ€” STDIO, Streamable HTTP, self-service OAuth, Okta SSO, and token exchange (Modes 1โ€“5)

Related MCP server: MCP Universal Test Suite Generator

๐Ÿ“‘ Table of Contents

  1. ๐Ÿ”ฅ Why This Server

  2. ๐ŸŽฏ What is this tool?

  3. ๐Ÿง  Intelligent Rules System

  4. ๐Ÿ“‹ Prerequisites

  5. ๐Ÿš€ Quick Start Guide

  6. ๐Ÿ”ง Usage Methods

  7. ๐Ÿ› ๏ธ Available Tools

  8. ๐Ÿ“Ž MCP Resources & Prompts (NEW)

  9. ๐ŸŽญ Role-Specific Prompts & Workflows

  10. ๐Ÿ“– Output Formats

  11. โš™๏ธ Configuration Options

  12. ๐Ÿงช Testing Your Setup

  13. ๐Ÿ” Troubleshooting

  14. ๐ŸŽฏ Example Workflows

  15. ๐Ÿ”ง Advanced Features

  16. ๐Ÿ“š Additional Documentation

  17. ๐Ÿค Contributing

  18. ๐Ÿ“„ License

  19. ๐ŸŽ‰ You're Ready!


๐ŸŽฏ What is this tool?

This tool allows you to:

  • Retrieve test cases and test suites from Zebrunner

  • Analyze test coverage and generate test code

  • Get test execution results and launch details

  • Validate test case quality with automated checks using intelligent rules

  • Generate reports and insights from your test data

  • Improve test cases with AI-powered suggestions and automated fixes

All through natural language commands in AI assistants!

โฌ†๏ธ Back to top

๐Ÿง  Intelligent Rules System

What Makes This Tool Special

Our MCP server includes a sophisticated 3-tier rules system that transforms how you work with test cases:

๐ŸŽฏ Test Case Review Rules (test_case_review_rules.md)

  • Purpose: Core quality standards and writing guidelines

  • What it does: Defines fundamental principles for writing high-quality test cases

  • Key areas: Independence, single responsibility, comprehensive preconditions, complete step coverage

  • Used by: adv_validate_test_case and adv_improve_test_case tools

โœ… Test Case Analysis Checkpoints (test_case_analysis_checkpoints.md)

  • Purpose: Detailed validation checklist with 100+ checkpoints

  • What it does: Provides granular validation criteria for thorough test case analysis

  • Key areas: Structure validation, automation readiness, platform considerations, quality assurance

  • Used by: adv_validate_test_case for comprehensive scoring and issue detection

โš™๏ธ MCP Zebrunner Rules (mcp-zebrunner-rules.md)

  • Purpose: Technical configuration for test generation and coverage analysis

  • What it does: Defines framework detection patterns, code templates, and coverage thresholds

  • Key areas: Framework detection, test generation templates, coverage thresholds, quality standards

  • Used by: adv_generate_draft_test_by_key and adv_get_enhanced_test_coverage_with_rules tools

How the Rules Work Together

graph TD
    A[Test Case] --> B[adv_validate_test_case]
    B --> C[test_case_review_rules.md]
    B --> D[test_case_analysis_checkpoints.md]
    B --> E[Validation Result + Issues]
    
    E --> F[adv_improve_test_case]
    F --> G[AI-Powered Improvements]
    
    A --> H[adv_generate_draft_test_by_key]
    H --> I[mcp-zebrunner-rules.md]
    H --> J[Generated Test Code]
    
    A --> K[adv_get_enhanced_test_coverage_with_rules]
    K --> I
    K --> L[Coverage Analysis + Rules Validation]

Why This Matters

  1. Consistency: All team members follow the same quality standards

  2. Automation: Reduce manual review time with automated validation

  3. Learning: New team members learn best practices through AI feedback

  4. Customization: Adapt rules to your project's specific needs

  5. Continuous Improvement: AI suggests improvements based on proven patterns

Customizing Rules for Your Project

You can customize any of the three rules files:

# Copy default rules to customize
cp test_case_review_rules.md my-project-review-rules.md
cp test_case_analysis_checkpoints.md my-project-checkpoints.md
cp mcp-zebrunner-rules.md my-project-technical-rules.md

# Use custom rules in validation
"Validate test case PROJ-123 using custom rules from my-project-review-rules.md"

Example customizations:

  • Mobile projects: Add mobile-specific validation rules

  • API projects: Focus on API testing patterns and data validation

  • Different frameworks: Customize code generation templates

  • Company standards: Align with your organization's testing guidelines

โฌ†๏ธ Back to top

๐Ÿ“‹ Prerequisites

What you need to know

  • Basic command line usage (opening terminal, running commands)

  • Your Zebrunner credentials (login and API token)

  • Basic understanding of test management (test cases, test suites)

Software requirements

  • Node.js 18 or newer - Download here

  • npm (comes with Node.js)

  • Access to a Zebrunner instance with API credentials

How to check if you have Node.js

Open your terminal/command prompt and run:

node --version
npm --version

If you see version numbers, you're ready to go!

โฌ†๏ธ Back to top

๐Ÿš€ Quick Start Guide

๐Ÿ’ก Want more detailed instructions? Check out our More Detailed Step-by-step Install Guide with troubleshooting tips and platform-specific instructions.

Step 1: Get the code

Choose one of these methods:

git clone https://github.com/maksimsarychau/mcp-zebrunner.git
cd mcp-zebrunner

Option B: Download and extract

Download the project files and extract them to a folder.

Step 2: Install dependencies

npm install

Step 3: Configure your Zebrunner connection

Create a .env file in the project folder with your Zebrunner details:

# Your Zebrunner instance URL (without trailing slash)
ZEBRUNNER_URL=https://your-company.zebrunner.com/api/public/v1

# Your Zebrunner login (usually your email)
ZEBRUNNER_LOGIN=your.email@company.com

# Your Zebrunner API token (get this from your Zebrunner profile)
ZEBRUNNER_TOKEN=your_api_token_here

# Optional: Enable debug logging (default: false)
DEBUG=false

# Optional: Enable intelligent rules system (auto-detected if rules file exists)
ENABLE_RULES_ENGINE=true

How to get your Zebrunner API token:

  1. Log into your Zebrunner instance

  2. Go to your profile settings

  3. Find the "API Access" section

  4. Generate a new API token

  5. Copy the token to your .env file

Step 4: Build the project

npm run build

Step 5: Test your connection

npm run test:health

If you see "โœ… Health check completed", you're ready to go!

๐Ÿ”„ Updating to New Version

Check current version

# Check your current version
npm run version
# or manually check package.json
cat package.json | grep '"version"'

Update steps

# 1. Pull latest changes from master branch
git pull origin master

# 2. Install any new dependencies
npm install

# 3. Rebuild the project
npm run build

# 4. Test your connection (requires valid .env file)
npm run test:health

Important Notes:

  • โœ… Your .env file must be properly configured for the health check to work

  • โœ… Restart Claude Desktop/Code after updating to reload the MCP server

  • โœ… Check release notes for any breaking changes before updating

If the health check fails, verify your .env configuration and Zebrunner credentials.

โฌ†๏ธ Back to top

๐Ÿ”ง Usage Methods

Add this configuration to your Claude Desktop or Claude Code settings. Important: You must use the full absolute path to your project folder.

{
  "mcpServers": {
    "mcp-zebrunner": {
      "command": "node",
      "args": ["/full/absolute/path/to/mcp-zebrunner/dist/server.js"],
      "env": {
        "ZEBRUNNER_URL": "https://your-company.zebrunner.com/api/public/v1",
        "ZEBRUNNER_LOGIN": "your.email@company.com",
        "ZEBRUNNER_TOKEN": "your_api_token_here",
        "DEBUG": "false",
        "ENABLE_RULES_ENGINE": "true",
        "DEFAULT_PAGE_SIZE": "100",
        "MAX_PAGE_SIZE": "100"
      }
    }
  }
}

Example paths:

  • Windows: C:\\Users\\YourName\\Projects\\mcp-zebrunner\\dist\\server.js

  • macOS/Linux: /Users/YourName/Projects/mcp-zebrunner/dist/server.js

Alternative: Command Line Integration (Claude Code)

You can also add the server using the command line:

claude mcp add mcp-zebrunner \
  --env ZEBRUNNER_URL="https://your-company.zebrunner.com/api/public/v1" \
  --env ZEBRUNNER_LOGIN="your.email@company.com" \
  --env ZEBRUNNER_TOKEN="your_api_token_here" \
  --env DEBUG="false" \
  --env ENABLE_RULES_ENGINE="true" \
  -- node /full/absolute/path/to/mcp-zebrunner/dist/server.js

Important: Replace /full/absolute/path/to/mcp-zebrunner/ with the actual full path to your project folder.

Method 2: Run as standalone server

Development mode (with auto-reload)

npm run dev

Production mode

npm start

Method 3: Smart URL-Based Analysis ๐Ÿš€

NEW in v5.4.1+: Claude can automatically detect Zebrunner URLs and analyze them with optimal settings!

Just paste a Zebrunner URL in your conversation, and Claude will automatically:

  • Parse the URL to extract project, launch, and test IDs

  • Call the appropriate analysis tool

  • Use recommended settings (videos, screenshots, AI analysis enabled)

๐Ÿ“‹ Supported URL Patterns

1. Test Analysis URLs

https://your-workspace.zebrunner.com/projects/PROJECT/automation-launches/LAUNCH_ID/tests/TEST_ID

What happens:

  • Claude automatically calls adv_analyze_test_failure

  • Extracts: projectKey, testRunId (launch ID), testId

  • Enables: includeVideo: true, analyzeScreenshotsWithAI: true, all diagnostics

Example:

User: "Analyze https://your-workspace.zebrunner.com/projects/MCP/automation-launches/120911/tests/5455386"

Claude automatically calls:
{
  projectKey: "MCP",
  testRunId: 120911,
  testId: 5455386,
  includeVideo: true,
  analyzeScreenshotsWithAI: true,
  includeLogs: true,
  includeScreenshots: true,
  analyzeSimilarFailures: true,
  screenshotAnalysisType: "detailed",
  format: "detailed"
}

2. Launch Analysis URLs

https://your-workspace.zebrunner.com/projects/PROJECT/automation-launches/LAUNCH_ID

What happens:

  • Claude automatically calls adv_detailed_analyze_launch_failures

  • Extracts: projectKey, testRunId (launch ID)

  • Enables: includeScreenshotAnalysis: true, comprehensive analysis

Example:

User: "Analyze https://your-workspace.zebrunner.com/projects/MCP/automation-launches/120911"

Claude automatically calls:
{
  projectKey: "MCP",
  testRunId: 120911,
  filterType: "without_issues",
  includeScreenshotAnalysis: true,
  screenshotAnalysisType: "detailed",
  format: "summary",
  executionMode: "sequential"
}

โœจ Advanced Usage

Override Default Settings

Claude understands natural language overrides:

User: "Analyze https://...url... but without screenshots"
โ†’ Claude sets: analyzeScreenshotsWithAI: false

User: "Analyze https://...url... in jira format"
โ†’ Claude sets: format: "jira"

User: "Quick analysis of https://...url..."
โ†’ Claude sets: format: "summary", screenshotAnalysisType: "basic"

Multiple URLs

Analyze multiple tests/launches in one request:

User: "Compare these failures:
https://your-workspace.zebrunner.com/projects/MCP/automation-launches/120911/tests/5455386
https://your-workspace.zebrunner.com/projects/MCP/automation-launches/120911/tests/5455390"

โ†’ Claude analyzes both sequentially and compares results

Cross-Workspace Support

โš ๏ธ URLs from different workspaces will show a warning but still attempt analysis:

User: "Analyze https://other-workspace.zebrunner.com/..."
โ†’ Claude warns: "URL is from 'other-workspace.zebrunner.com' but configured workspace is 'your-workspace.zebrunner.com'"
โ†’ Proceeds with analysis using available credentials

๐Ÿ“– URL Pattern Reference

Component

Example

Extracted As

Used In Tool

Workspace

your-workspace.zebrunner.com

Validation only

N/A

Project Key

MCP

projectKey

All tools

Launch ID

120911

testRunId

All tools

Test ID

5455386

testId

adv_analyze_test_failure only

๐ŸŽฏ Why Use URL-Based Analysis?

โœ… Faster: No need to manually specify IDs
โœ… Convenient: Copy-paste URLs directly from Zebrunner UI
โœ… Optimized: Automatic use of recommended settings
โœ… Smart: Claude detects intent and adjusts parameters
โœ… Flexible: Natural language overrides work seamlessly

๐Ÿ’ก Pro Tips

  1. Direct from Zebrunner: Copy URL directly from your browser while viewing a test/launch

  2. Batch Analysis: Paste multiple URLs separated by newlines

  3. Custom Settings: Add natural language instructions to override defaults

  4. Quick Checks: URLs work great for quick "what happened here?" questions

  5. Reports: Combine with format requests: "Generate JIRA ticket for https://...url..."

โฌ†๏ธ Back to top

๐Ÿค Dual-MCP setup (official zebrunner + Advanced)

Zebrunner now offers an official hosted MCP (beta) at https://{workspace}.zebrunner.com/api/mcp. It is safe to run both servers in the same Cursor / Claude session: every tool on this server is exposed under the adv_<name> prefix, so it never collides with the official tools.

Example .cursor/mcp.json:

{
  "mcpServers": {
    "zebrunner": {
      "url": "https://your-workspace.zebrunner.com/api/mcp",
      "headers": {
        "X-Zebrunner-Username": "your.name@company.com",
        "X-Zebrunner-Api-Token": "<your-zebrunner-api-token>"
      }
    },
    "mcp-zebrunner": {
      "command": "npx",
      "args": ["-y", "mcp-zebrunner"],
      "env": {
        "ZEBRUNNER_URL": "https://your-workspace.zebrunner.com/api/public/v1",
        "ZEBRUNNER_LOGIN": "your.name@company.com",
        "ZEBRUNNER_TOKEN": "<your-zebrunner-api-token>"
      }
    }
  }
}

Rule of thumb when both are connected:

Use zebrunner (official) for

Use mcp-zebrunner (Advanced) for

list_projects, shared steps, test run replace/close, custom field create

Failure forensics, runtime + flakiness analytics

Server-side filter DSL; file upload + screenshot content in triage

Reporting, dashboards, regression analyzers

Bulk launch/test comments, issue linking, launcher/preset CRUD, launch_tests

Preview/confirm TCM mutations; Jenkins Build Now / rerun failures

Inside chat, ask adv_about_mcp_tools with mode: "routing" or open the zebrunner://mcp-routing resource for the full live-generated diff and routing table.

๐Ÿ› ๏ธ Available Tools

Once connected, you can use these tools through natural language in your AI assistant. This section highlights all 69 tools organized by category. For the complete catalog with natural-language examples for every tool, see TOOLS_CATALOG.md.

Dashboard widgets: All 22 Zebrunner dashboard templates have MCP coverage (v9.2.5). See docs/TEST_PROMPTS.md ยง18 for example prompts and npm run test:api verification.

Token-efficient reads

Large TCM exports can consume significant context. Use these opt-in parameters (defaults stay backward-compatible). Full guide: docs/TOKEN_EFFICIENCY.md.

Technique

Example

When to use

format: 'compact'

adv_get_all_tcm_test_cases_by_project with format:'compact'

Minified JSON (~22% smaller than pretty json)

detail: 'summary'

adv_get_test_cases_by_suite_smart with detail:'summary'

After filtering โ€” returns id, key, title, priority, automationState, webUrl only

adv_batch_get_test_cases

{ case_keys: ["MCP-1","MCP-2"], detail:"summary", format:"compact" }

Fetch a shortlist without N round-trips

adv_generate_report inline: false

Writes HTML/PNG to disk, returns paths

Huge dashboards in chat clients

count_only: true

Any bulk TCM/suite read

Metrics without payload

Env flags (off)

MCP_COMPACT_DEFAULTS=true, MCP_SUMMARY_DEFAULTS=true

Server-wide default flips after eval passes

Workflow: list/filter with detail:'summary' + format:'compact' โ†’ adv_get_test_case_by_key for full body (steps, preconditions) before create/update.

Example prompts for your assistant:

  • "List test cases in project MCP with summary detail and compact JSON."

  • "Fetch MCP-1 and MCP-2 in one batch call with summary and compact format."

  • "Generate a quality dashboard for MCP with inline false โ€” save to disk."

Tool naming: All tools are exposed as adv_<name> (e.g. adv_get_test_case_by_key). Use these names in prompts, scripts, and when both the official and Advanced MCP are connected. Legacy short names (get_test_case_by_key, etc.) are deprecated and only registered when ZEBRUNNER_REGISTER_LEGACY_ALIASES=true.

๐Ÿ“‹ Test Case Management

Core Test Case Tools

Tool

Description

Example Usage

Best For

adv_get_test_case_by_key

Get detailed test case information

"Get test case MCP-123 details"

All roles

adv_batch_get_test_cases

Fetch multiple cases by key (partial success)

"Get MCP-1 and MCP-2 with summary detail"

All roles

adv_get_test_cases_advanced

Advanced filtering with automation states, dates

"Get test cases created after 2025-01-01 with automation state 'Manual'"

QA, SDETs

adv_get_test_cases_by_automation_state

Filter by specific automation states

"Show me all 'Not Automated' test cases in project MCP"

SDETs, Managers

adv_get_test_case_by_title

Search test cases by title (partial match)

"Find test cases with title containing 'login functionality'"

All roles

adv_get_test_case_by_filter

Advanced filtering by suite, dates, priority, automation state

"Get test cases from suite 491 created after 2025-01-01 with high priority"

QA, Managers

adv_get_automation_states

List available automation states

"What automation states are available for project MCP?"

All roles

adv_get_automation_priorities

List available priorities with IDs

"Show me all priority levels for project MCP"

All roles

Batch Test Case Operations

Tool

Description

Example Usage

Best For

adv_get_all_tcm_test_cases_by_project

Get ALL test cases (handles pagination)

"Get all test cases for project MCP"

Managers, Leads

adv_get_all_tcm_test_cases_with_root_suite_id

All test cases with hierarchy info

"Get all test cases with their root suite information"

Analysts

adv_get_test_cases_by_suite_smart

Smart suite test case retrieval with root/child auto-detection

"Get test cases from suite 18824 in project MCP using smart mode"

QA, Analysts

Test Case Change History

Most test case tools support optional change history enrichment โ€” fetching the audit log of modifications for each test case. This is useful for understanding when steps were changed, when automation state transitioned, or when a test case was deprecated.

Parameter

Type

Default

Description

include_history

boolean

false

Attach a history array of parsed change entries to each test case

history_filter

steps_only | events_only | all

steps_only

Filter: step/precondition diffs only, lifecycle events only, or all changes

history_limit

number (1โ€“100)

20

Max history entries per test case

Supported on: adv_get_test_case_by_key, adv_get_test_cases_advanced, adv_get_test_cases_by_automation_state, adv_get_test_case_by_title, adv_get_test_case_by_filter, adv_get_all_tcm_test_cases_by_project, adv_get_test_cases_by_suite_smart

Example prompts:

  • "Get test case MCP-29 with change history" โ†’ sets include_history=true

  • "Show me MCP-29 history, events only" โ†’ sets include_history=true, history_filter='events_only'

  • "Get all automated test cases with their last 5 changes" โ†’ sets include_history=true, history_limit=5

Detected events: became_automated, became_manual, became_deprecated, became_undeprecated, steps_changed, preconditions_changed, postconditions_changed, and dynamically generated became_<state_name> for any project-specific automation state.

๐ŸŒณ Test Suite Hierarchy & Organization

Suite Management

Tool

Description

Example Usage

Best For

adv_list_test_suites

List suites with pagination

"List test suites for project MCP"

All roles

adv_get_suite_hierarchy

Hierarchical tree view

"Show me the hierarchy of test suites with depth 3"

Managers, QA

adv_get_root_suites

Get top-level suites

"Show me all root suites for project MCP"

Managers

adv_get_all_subsuites

Get all child suites

"Get all subsuites from root suite 18697"

QA, Analysts

Suite Analysis Tools

Tool

Description

Example Usage

Best For

adv_get_tcm_suite_by_id

Get suite by ID (simple: fast direct API, full: hierarchy-enriched)

"Show me suite 20421 in project MCP"

All roles

adv_get_tcm_test_suites_by_project

Comprehensive suite listing

"Get all suites for project MCP with hierarchy"

Managers

adv_get_all_tcm_test_case_suites_by_project

Get all TCM test case suites with pagination

"Get all TCM test case suites for project MCP"

Managers, Analysts

adv_get_root_id_by_suite_id

Find root suite for any suite

"What's the root suite for suite 12345?"

Analysts

๐Ÿ”ง Mutation Tools (Beta)

Safety Model: Every mutation tool follows a two-call confirmation gate. The first call returns a preview; only after user approval should confirm: true be passed to execute the mutation. All mutations are audit-logged to ~/.mcp-zebrunner-audit.jsonl. Use dry_run: true for raw payload inspection.

Next-step steering (v7.2.2): After every successful mutation, the server appends a Tip: block guiding the LLM to the most useful next action (e.g., "validate quality", "publish the draft", "populate the test run"). Hints are conditional -- they are suppressed when redundant. For example, the quality-check hint is omitted if review: true was already used. Created test cases are always forced to draft=true, and the hint always reminds the LLM to publish via adv_update_test_case. This approach is inspired by the Strands Agents steering pattern and delivers just-in-time guidance without bloating system prompts.

Suite Mutations

Tool

Description

Example Usage

Best For

adv_create_test_suite

(Beta) Create a new Test Suite

"Create root suite 'Payments' in project MCP"

QA, Managers

adv_update_test_suite

(Beta) Update an existing Test Suite (full PUT)

"Rename suite 18697 to 'Login & Registration'"

QA, Managers

Test Case Mutations

Tool

Description

Example Usage

Best For

adv_create_test_case

(Beta) Create a new Test Case with runtime validation of priorities, automation states, and custom fields. Accepts {file_path} in attachments. Optional source_case_key to pre-populate from an existing test case.

"Create test case 'Verify login' in suite 17470 for project MCP"

QA, SDETs

adv_update_test_case

(Beta) Partially update a Test Case by ID or key (PATCH). Accepts {file_path} in attachments for local file upload.

"Attach /Users/me/screenshot.png to test case MCP-42"

QA, SDETs

๐Ÿ” Test Coverage & Analysis

Test Impact (PR / code changes) (v9.2.8)

Tool

Description

Example Usage

Best For

adv_analyze_test_impact

Rank Zebrunner cases affected by code changes from compact semantic context

"Analyze test impact for diary edit + serving size changes on PROJ2"

Developers, SDETs

Use the /test-impact MCP prompt (optional pr_url) for guided workflow. See TEST_IMPACT_WORKFLOW.md. Configure repositoryProjectMap via .env.example.

Coverage Analysis

Tool

Description

Example Usage

Best For

adv_get_test_coverage_by_test_case_steps_by_key

Analyze implementation coverage

"Analyze coverage for MCP-123 against this code: [paste code]"

Developers, SDETs

adv_get_enhanced_test_coverage_with_rules

Rules-based coverage analysis

"Enhanced coverage analysis for MCP-123 with framework detection"

SDETs, Leads

Duplicate Analysis

Tool

Description

Example Usage

Best For

adv_analyze_test_cases_duplicates

Find and group similar test cases by step similarity

"Analyze suite 12345 for duplicates with 80% similarity threshold"

QA Managers, SDETs

adv_analyze_test_cases_duplicates_semantic

Advanced semantic analysis with LLM-powered step clustering

"Semantic analysis of suite 12345 with step clustering and medoid selection"

Senior QA, Test Architects

๐Ÿ”— Clickable Links Feature: Both duplicate analysis tools support clickable links to Zebrunner web UI:

  • Add include_clickable_links: true to make test case keys clickable in markdown output

  • JSON/DTO formats automatically include webUrl fields when enabled

  • Links are generated from your ZEBRUNNER_URL environment variable

  • Example: "Analyze suite 17585 for duplicates with clickable links enabled"

๐Ÿงช Test Code Generation & Validation

AI-Powered Tools

Tool

Description

Example Usage

Best For

adv_generate_draft_test_by_key

Generate test code with framework detection

"Generate Java/Carina test for MCP-123 based on this implementation"

SDETs, Developers

adv_validate_test_case

Quality validation with improvement

"Validate test case MCP-123 and suggest improvements"

QA, Managers

adv_improve_test_case

Dedicated improvement tool

"Improve test case MCP-123 with specific suggestions"

QA, SDETs

๐Ÿš€ Launch & Execution Management

Launch Operations โญ Essential for Managers

Tool

Description

Example Usage

Best For

adv_get_launch_details

Comprehensive launch information; optional includeJobParameters for Jenkins Build Now discovery

"Get launch details for launch 118685 with job parameters"

Managers, Leads, SDETs

adv_rerun_launch_failures

๐Ÿ†• v9.1.0 Rerun failed/aborted tests for one or many launches (preview/confirm)

"Rerun failures for launch 132522 in project android"

Managers, SDETs

adv_start_launch

๐Ÿ†• v9.1.0 Trigger Jenkins Build Now (not Launch Launchers); preview/confirm

"Build now regression for android milestone 26.19.0 build 50977"

Managers, SDETs

adv_get_launch_summary

Quick launch overview

"Show me summary for launch 118685"

Managers

adv_get_all_launches_for_project

List individual launch executions with pagination

"List launches for project MCP from last month"

Managers, Leads

adv_get_all_launches_with_filter

Search launches by milestone/build/name

"Find launches for milestone 2.1.0 and build 'mcp-app-2.1.0'"

Managers, Leads

adv_generate_weekly_regression_stability_report

Weekly regression stability report with WoW delta, linked issues, and strict Jira-ready output. Supports launch list or build-based auto-discovery (version-segment build lookup with launch.build validation when needed).

"Weekly stability report for MCP: (120906 vs 120814), (120901 vs 120809)" or "Weekly stability report for builds 9117 vs 48886"

Managers, Leads

adv_analyze_regression_runtime

Regression Runtime Efficiency โ€” per-launch elapsed time, attempt/re-run breakdown, configurable duration classification (Short/Medium/Long), dual metrics for both Tests and Test Cases (Average Runtime, WRI), duration distribution with test case counts, and baseline comparison with delta tracking.

"Analyze regression runtime for the iOS project on the latest milestone. Show WRI and WRI per test case." or "Compare runtime for latest vs previous milestone"

Managers, Leads, SDETs

adv_analyze_regression_runtime โ€” Usage Examples:

  • "Analyze regression runtime for the iOS project on the latest milestone. Show both average runtime per test and per test case, plus WRI and WRI per test case."

  • "Run regression runtime analysis for all three projects on their latest milestones. Compare how many test cases fall into Short vs Medium vs Long buckets per team."

  • "Analyze regression runtime for the Android project, latest vs previous milestone. Show WRI and WRI per test case for both. Has efficiency improved or degraded?"

  • "Analyze regression runtime with medium_threshold_seconds: 120 and long_threshold_seconds: 300. Show which tests fall into each bucket."

  • "What is the average time cost per test case in each duration bucket? Are long-running tests covering proportionally more test cases?"

๐Ÿ“Š Reporting & Analytics

Test Failure Analysis ๐Ÿ†• Game Changer

Tool

Description

Example Usage

Best For

adv_analyze_test_failure

Deep forensic analysis of failed tests with logs, screenshots, error classification, and recommendations. NEW in v5.11.0: Compare with last passed execution! Shows what changed (logs, duration, environment). Also: format: 'jira' generates ready-to-paste Jira tickets with auto-priority, labels, and clickable ๐ŸŽฅ video links!

"Analyze test failure 5451420 in launch 120806 and compare with last passed execution"

QA Engineers, SDETs, Managers

adv_get_test_execution_history

๐Ÿ†• NEW in v5.11.0! Track test execution trends across launches. View pass/fail history, find last passed execution, calculate pass rate. Critical Detection: Highlights when test failed in all recent runs!

"Show execution history for test 5478492" or "Has test 5478492 been failing consistently?"

QA Engineers, SDETs, Managers

adv_detailed_analyze_launch_failures

๐Ÿš€ Enhanced v4.12.1 Analyze failures WITHOUT linked issues with Claude-level intelligence + Jira format support. Auto-deep-dive with executive summary, timeline, patterns, priorities. NEW: Generate Jira-ready tickets for entire launches!

"Analyze failures in launch 120806" or with format: 'jira'

QA Managers, SDETs, Team Leads

โœ… FIXED in v5.2.4! Improved Reliability & Video Links

  • ๐ŸŽฅ Video URLs fixed: Now uses test-sessions API (/api/reporting/v1/launches/{id}/test-sessions) for reliable video artifact extraction

  • ๐Ÿ›ก๏ธ Comprehensive error handling: Gracefully handles missing screenshots/logs (returns empty arrays instead of throwing)

  • ๐Ÿšซ No more "no result received" errors: All API calls have proper try-catch blocks with fallbacks

  • ๐Ÿ› Better debugging: Enhanced logging when debug: true is enabled in config

  • ๐Ÿ“Š Schema updates: Supports both old and new API structures for backward compatibility

๐ŸŽซ NEW in v4.12.1! Jira-Ready Ticket Format

  • Use format: 'jira' to generate ready-to-paste Jira tickets

  • Auto-calculated priority based on stability and impact

  • Smart labels: test-automation, locator-issue, flaky-test, etc.

  • Complete Jira markup: Tables, panels, code blocks, clickable links

  • ๐ŸŽฅ Prominent video links: Beautiful panels + links section

  • Copy-paste ready: No manual formatting needed

  • Saves 5-10 minutes per ticket with consistent quality

๐Ÿง  Enhanced in v4.11.1! adv_detailed_analyze_launch_failures provides automatic deep synthesis like Claude would manually provide:

  • ๐ŸŽฏ Executive Summary: Key findings, patterns, and stability indicators

  • ๐Ÿ“… Timeline Analysis: When failures first appeared, progression tracking

  • ๐Ÿ”ฌ Pattern Analysis: Groups by root cause with affected tests and stability %

  • ๐ŸŽฏ Priority-Based Recommendations: ๐Ÿ”ด HIGH / ๐ŸŸก MEDIUM / ๐ŸŸข LOW with impact analysis

  • ๐Ÿ“‹ Enhanced Test Details: Full error messages, stack traces, timestamps

  • โ“ Smart Follow-up Questions: Guides next investigation steps

  • Smart filtering: Analyzes only tests WITHOUT linked issues by default

  • Optional AI screenshot analysis for all tests

  • No manual follow-up needed - get complete picture in one call!

๐Ÿ†• NEW in v5.11.0! Test Execution History & Comparison

  • ๐Ÿ“Š Track execution trends: View pass/fail history across launches with adv_get_test_execution_history

  • ๐Ÿ”„ Compare with last passed: New compareWithLastPassed parameter in adv_analyze_test_failure

    • Compare logs (new errors detection)

    • Compare duration (performance regression)

    • Compare environment (device/platform changes)

    • Compare screenshots (visual differences)

  • โš ๏ธ Critical detection: Automatically highlights when test failed in all recent executions

  • ๐ŸŽฏ Regression analysis: See exactly what changed between passed and failed runs

  • ๐Ÿ“ˆ Pass rate metrics: Calculate test stability over time

  • See TOOLS_CATALOG.md for example prompts!

Screenshot Analysis & Visual Forensics ๐ŸŽฏ Enhanced in v4.11.0

Tool

Description

Example Usage

Best For

adv_download_test_screenshot

Download protected screenshots from Zebrunner with authentication

"Download screenshot from https://your-workspace.zebrunner.com/files/abc123 for test 5451420"

QA Engineers, Automation Engineers

adv_analyze_screenshot

Visual analysis with OCR, UI detection, and Claude Vision

"Analyze screenshot https://your-workspace.zebrunner.com/files/abc123 with OCR and detailed analysis"

QA Engineers, SDETs, Developers

๐Ÿ“ธ Enhanced! Screenshot analysis now integrated directly into adv_analyze_test_failure and analyze_launch_failures - no need to call separately! See Screenshot Analysis Guide for details.

Platform & Results Analysis โญ Critical for Management

Tool

Description

Example Usage

Best For

adv_get_platform_results_by_period

Aggregated test results, pass rate, and statistics for a project over a time period

"Get results for MCP during last 7 days" or "Show pass rate for MCP"

Managers, Leads

adv_get_top_bugs

Most frequent defects

"Show me top 10 bugs from last week"

Managers, Developers

adv_get_bug_review

Detailed bug review with failure analysis, priority breakdown, and automatic detail fetching

"Get bug review with full failure details for top 10 bugs"

Managers, QA, Developers

adv_get_bug_failure_info

Comprehensive failure info by hashcode (alternative to auto-fetch)

"Get failure info for hashcode 1051677506"

Developers, SDETs

adv_get_project_milestones

Available milestones

"Get all milestones for project MCP"

Managers, PMs

Period behavior for reporting widgets: period is passed to Zebrunner widgets as-is (for example Today, Month, Last 30 Days). If a selected widget does not support a specific period label, the API will return an error.

Project Discovery

Tool

Description

Example Usage

Best For

adv_get_available_projects

Discover all accessible projects

"What projects can I access?"

All roles

adv_test_reporting_connection

Test API connectivity

"Test my connection to Zebrunner"

All roles

adv_about_mcp_tools

Summarize tools, prompts, resources, or show session metrics. Modes: summary, tool, prompts, resources, metrics

"Using Zebrunner MCP make a summary of all tools with examples" or "Show me tool usage metrics for this session"

All roles

๐Ÿƒ Test Run Management

Public API Test Runs โญ Powerful for Analysis

Tool

Description

Example Usage

Best For

adv_list_test_runs

Advanced test run filtering

"Get test runs from last 30 days with status 'FAILED'"

Managers, SDETs

adv_get_test_run_by_id

Detailed test run information

"Get details for test run 12345"

Managers, QA

adv_list_test_run_test_cases

Test cases in a specific run

"Show me all test cases in test run 12345"

QA, Analysts

Configuration Management

Tool

Description

Example Usage

Best For

adv_get_test_run_result_statuses

Available result statuses

"What result statuses are configured for project MCP?"

QA, SDETs

adv_get_test_run_configuration_groups

Configuration options

"Show me configuration groups for project MCP"

SDETs, Leads

๐Ÿ“Ž MCP Resources & Prompts (NEW)

Full guide: docs/RESOURCES_AND_PROMPTS.md โ€” detailed usage, examples, reference tables, and contributor guide.

In addition to 69 tools, the server now provides 14 resources and 17 prompts that improve discoverability and automate complex workflows.

Resources โ€” @ Context Injection

Resources are read-only reference data you attach to your conversation via the @ menu. They help the AI use exact parameter values instead of guessing.

Resource

What it provides

@zebrunner://reports/types

6 report types with params, defaults, and examples

@zebrunner://periods

12 valid time period values (case-sensitive)

@zebrunner://charts

Chart formats, types, and 17 supported tools

@zebrunner://formats

5 output format families with valid values

@zebrunner://projects

All accessible projects with keys and IDs

@zebrunner://projects/{key}/suites

Root test suites for a project

@zebrunner://projects/{key}/suite-hierarchy

Full suite tree with parent-child relationships

@zebrunner://projects/{key}/automation-states

Automation state names and IDs

@zebrunner://projects/{key}/priorities

Priority levels and IDs

@zebrunner://projects/{key}/milestones

Active and completed milestones

@zebrunner://projects/{key}/result-statuses

Test run result statuses

@zebrunner://projects/{key}/configuration-groups

Test run config groups and options

@zebrunner://projects/{key}/fields

System and custom field definitions

Quick example:

@ zebrunner://reports/types
@ zebrunner://projects
Generate an executive dashboard for all starred projects.

Prompts โ€” / Workflow Commands

Prompts are pre-built workflow instructions triggered via the / command menu. Each prompt guides the AI through a tested multi-tool orchestration.

Prompt

Parameters

What it does

/pass-rate

projects

Cross-platform pass rate with target comparison

/runtime-efficiency

projects

Runtime metrics with milestone delta

/automation-coverage

projects

7-metric coverage + automation intake rate

/executive-dashboard

projects

5-section standup-ready dashboard

/release-readiness

project, milestone?

Go/No-Go assessment with evidence

/suite-coverage

projects

Per-suite coverage tables (TOTAL + TOTAL REGRESSION)

/regression-summary

project, milestone?, build?

Regression results overview, new bugs, top bugs, slowest tests

/review-test-case

case_key

Validate + improve a test case

/launch-triage

project

Post-regression failure analysis

/relaunch-regression-failures

projects, milestone?, build?, period?

๐Ÿ†• v9.1.0 Discover failed launches and batch-rerun failures

/feature-scoped-launch

project, feature, suite_name?, build?

๐Ÿ†• v9.1.0 Feature keyword โ†’ test_run_rules โ†’ Build Now per root suite

/flaky-review

project

Flaky test detection + stabilization plan

/find-duplicates

project, suite_id?

Structural + semantic duplicate analysis

/daily-qa-standup

projects

Daily standup summary with action items

/automation-gaps

projects

Automation backlog prioritization

/project-overview

project

Comprehensive project health card

/session-metrics

(none)

Show tool usage stats for the current session

Quick example:

/executive-dashboard
  projects: android,ios,web

โฌ†๏ธ Back to top

๐ŸŽฏ Management-Focused Quick Commands

๐Ÿ“ˆ Daily Standup Reports

# Get yesterday's results
"Get platform results for last 7 days for project MCP"

# Check recent failures  
"Show me top 5 bugs from last week"

# Review recent launches
"Get all launches for project MCP from last 3 days"

๐Ÿ” Test Suite Optimization

# Basic duplicate analysis
"Analyze suite 12345 for duplicates with 80% similarity threshold"

# Advanced semantic analysis with step clustering
"Semantic analysis of suite 12345 with 85% step clustering and medoid selection"

# Analyze specific test cases for duplicates
"Analyze test cases MCP-123, MCP-124, MCP-125 for duplicates"

# Project-wide duplicate analysis (use with caution - large datasets)
"Analyze project MCP for test case duplicates with 85% similarity"

# Get detailed similarity matrix with pattern types
"Analyze suite 12345 for duplicates with similarity matrix included"

# Two-phase clustering with semantic insights
"Semantic duplicate analysis with step clustering threshold 90% and insights enabled"

# Enable clickable links for easy navigation
"Analyze suite 17585 for duplicates with clickable links enabled"

๐Ÿ“Š Weekly Management Reports

# Comprehensive project health
"Get all launches for project MCP with milestone filter"

# Platform performance analysis
"Get iOS and Android test results for the last month"

# Quality metrics
"Get all test cases by automation state for project MCP"

๐ŸŽฏ Milestone & Release Planning

# Milestone tracking
"Get project milestones for MCP with completion status"

# Build-specific results
"Get launches for build 'mcp-app-2.1.0-release' and milestone '2.1.0'"

# Release readiness
"Get automation readiness for all test cases in project MCP"

๐Ÿž Issue Analysis & Troubleshooting

# Bug analysis
"Show me top 10 most frequent bugs with issue links"

# Failure investigation
"Get test run 12345 details with all test cases"

# Platform-specific issues
"Get Android test results for last 7 days with failure analysis"

โฌ†๏ธ Back to top

๐ŸŽญ Role-Specific Prompts & Workflows

๐Ÿ‘ฉโ€๐Ÿ’ป Manual QA Engineers

Daily Test Case Review

"Get test case MCP-45 details and validate its quality"
"Show me all test cases in suite 18708 that need improvement"
"Validate test case MCP-67 and suggest specific improvements"
"Find test cases with title containing 'login' to review authentication tests"
"Get test cases from suite 491 with high priority for today's testing"

Test Case Creation & Improvement

"I'm writing a test case for login functionality. What should I include based on our quality standards?"
"Improve test case MCP-89 - it's missing some preconditions"
"Check if test case MCP-12 is ready for manual execution"

Test Suite Organization

"Show me the hierarchy of test suites for project MYAPP to understand the structure"
"Get all subsuites from Authentication suite to review test coverage"
"List test cases in suite 18708 and identify which ones need validation"
"Find test cases with title containing 'payment' to organize payment testing"
"Get all high priority test cases from suite 491 for release testing"

Coverage Analysis

"I executed test case MCP-34 manually. Here's what I did: [paste your execution notes]. Analyze coverage against the documented steps."
"Compare test case MCP-56 with this manual testing session: [paste session details]"

๐Ÿค– Test Automation Engineers & SDETs

Automation Readiness Assessment

"Validate test case MCP-78 for automation readiness"
"Get all test cases in suite 18708 and identify which ones are ready for automation"
"Check test case MCP-23 - does it have clear, unambiguous steps for automation?"
"Find test cases with title containing 'API' to prioritize API automation"
"Get automation priorities to understand which test cases to automate first"
"Get test cases from suite 491 with 'Not Automated' state for automation planning"

Test Code Generation

"Generate Java/Carina test code for MCP-45 based on this existing framework: [paste framework code]"
"Create JavaScript/Jest test for MCP-67 using this test structure: [paste test example]"
"Generate Python/Pytest code for MCP-89 with these page objects: [paste page object code]"

Coverage Analysis & Validation

"Analyze test coverage for MCP-34 against this automated test: [paste test code]"
"Enhanced coverage analysis for MCP-56 with rules validation - here's my implementation: [paste code]"
"Compare test case MCP-78 steps with this Selenium test: [paste selenium code]"

Framework Integration

"Generate test code for MCP-45 using our Carina framework with these page objects: [paste existing code]"
"Create test automation for MCP-67 that integrates with this CI/CD pipeline: [paste pipeline config]"
"Generate API test for MCP-89 using this RestAssured setup: [paste API test framework]"

Batch Automation Analysis

"Validate all test cases in Authentication suite for automation readiness"
"Generate coverage report for all test cases in project MYAPP"
"Identify test cases in suite 18708 that have automation blockers"
"Find test cases with title containing 'regression' for automation sprint planning"
"Get test cases from suite 491 created after 2025-01-01 with high priority for next automation cycle"
"Get automation priorities and states to create automation roadmap"

๐Ÿ‘จโ€๐Ÿ’ป Developers

Test Case Understanding

"Get test case MCP-45 details to understand what I need to implement"
"Show me test cases related to login functionality in project MYAPP"
"Explain test case MCP-67 requirements in developer-friendly format"
"Find test cases with title containing 'authentication' for my feature development"
"Get high priority test cases from suite 491 that I need to implement"

Implementation Validation

"I implemented this feature: [paste code]. Analyze coverage against test case MCP-34"
"Here's my API implementation: [paste code]. Check coverage against test case MCP-56"
"Validate my UI implementation against test case MCP-78: [paste component code]"

Code Generation for Testing

"Generate unit tests for test case MCP-45 using Jest framework"
"Create integration tests for MCP-67 based on this API: [paste API code]"
"Generate test data setup for MCP-89 using this database schema: [paste schema]"

Bug Analysis

"Get test execution results for launch 118685 to understand recent failures"
"Show me top bugs from last week related to my feature area"
"Get detailed bug review for Android project from last 14 days"
"Show me comprehensive failure information for hashcode 1051677506"
"What are the top 50 bugs affecting our project this month?"
"Give me a summary of bug failures with reproduction dates from last 7 days"
"Analyze test case MCP-34 - why might it be failing in automation?"

๐Ÿ‘” Test Managers & Team Leads

Team Quality Metrics

"Get quality metrics for all test cases in project MYAPP"
"Show me test cases that need improvement in suite 18708"
"Generate quality report for test cases created this month"
"Find test cases with title containing 'critical' to assess critical path quality"
"Get automation priorities to align team efforts with business priorities"
"Get test cases from suite 491 with high priority that need quality improvements"

Test Suite Analysis

"Show me the complete test suite hierarchy for project MYAPP"
"Analyze test coverage across all suites in project MYAPP"
"Get automation readiness status for all test cases in Authentication suite"

Team Performance & Planning

"Get test execution results by platform for the last 30 days"
"Show me top 10 most frequent bugs to prioritize fixes"
"Analyze test case quality trends in project MYAPP"
"Get all launches for project MYAPP from last 30 days with milestone tracking"
"Show me platform results for last 7 days to track team performance"
"Get test runs with status 'FAILED' from last week for team retrospective"

Process Improvement

"Validate all test cases in suite 18708 to identify common quality issues"
"Generate improvement recommendations for test cases created by junior team members"
"Analyze which test cases are consistently failing automation"
"Get top 10 bugs from last month to identify process improvements"
"Show me test runs with detailed failure analysis for process optimization"
"Get automation readiness metrics across all test cases"
"Find test cases with title containing 'flaky' to address test stability"
"Get test cases from suite 491 with medium priority that could be automated"
"Get automation priorities to optimize team resource allocation"

Reporting & Stakeholder Communication

"Generate comprehensive test coverage report for project MYAPP in markdown format"
"Get test execution summary for launch 118685 for stakeholder presentation"
"Show me test quality metrics and improvement suggestions for quarterly review"
"Get platform results by period for executive dashboard"
"Create milestone progress report with test execution data"
"Generate weekly team performance report with launch and bug metrics"

Daily Management Tasks

"Get all launches for project MYAPP from yesterday"
"Show me top 5 bugs from last 7 days with issue links"
"Get platform results for iOS and Android from last week"
"Check automation readiness for upcoming release milestone"
"Get test run details for failed runs from last 24 hours"
"Show me project milestones and their completion status"

๐Ÿข Project Owners & Product Managers

Project Health Overview

"Get overall test coverage status for project MYAPP"
"Show me test execution results by platform for the last quarter"
"Generate project testing health report in markdown format"
"Get all launches for project MYAPP with milestone and build tracking"
"Show me platform results summary for executive review"
"Get project milestones with completion status and testing metrics"

Feature Testing Status

"Get test cases related to [feature name] in project MYAPP"
"Show me test execution results for [feature name] functionality"
"Analyze test coverage for [epic/story] requirements"
"Get launches filtered by milestone for feature release tracking"
"Show me test runs for specific build versions"

Quality Assurance Metrics

"Get quality metrics for all test cases in project MYAPP"
"Show me test case validation results and improvement areas"
"Generate testing quality report for stakeholder presentation"
"Get top bugs analysis for quality trend assessment"
"Show me automation vs manual testing ratio across the project"
"Find test cases with title containing 'smoke' to assess smoke test coverage"
"Get automation priorities to communicate testing strategy to stakeholders"
"Get test cases from suite 491 with critical priority for risk assessment"

Risk Assessment

"Show me top 10 most frequent bugs in project MYAPP"
"Get test cases that are not ready for automation and assess risk"
"Analyze test execution trends to identify potential quality risks"
"Get platform-specific failure rates for the last month"
"Show me test runs with high failure rates for risk mitigation"
"Get milestone-based testing progress for release risk assessment"

Planning & Resource Allocation

"Get automation readiness assessment for all test cases in project MYAPP"
"Show me test cases that need quality improvement and estimate effort"
"Analyze test suite structure to identify optimization opportunities"
"Get testing resource utilization by platform and time period"
"Show me milestone testing progress for sprint planning"
"Get comprehensive launch analysis for capacity planning"
"Find test cases with title containing 'performance' to plan performance testing"
"Get automation priorities to allocate automation resources effectively"
"Get test cases from suite 491 created in last month to plan review sessions"

Executive Reporting

"Generate executive dashboard with platform results and bug trends"
"Get quarterly testing metrics with milestone progress"
"Show me ROI analysis of automation vs manual testing efforts"
"Create board-ready testing status report with key metrics"
"Get testing velocity trends for project timeline assessment"

โฌ†๏ธ Back to top

๐Ÿ“– Output Formats

All tools support multiple output formats:

  • json - Structured data (default)

  • markdown - Rich formatted output with sections and tables

  • string - Human-readable text summaries

  • dto - Raw data objects

Example:

"Get test case PROJ-123 in markdown format"
"Show me test suites as JSON"

โฌ†๏ธ Back to top

โš™๏ธ Configuration Options

Environment Variables

# Required (STDIO mode) / Optional (HTTP selfauth mode โ€” see below)
ZEBRUNNER_URL=https://your-instance.zebrunner.com/api/public/v1
ZEBRUNNER_LOGIN=your.email@company.com
ZEBRUNNER_TOKEN=your_api_token

# Optional - HTTP Authentication Mode
MCP_AUTH_MODE=selfauth             # Enable per-user credential form (Mode 3)
MCP_TRANSPORT=http                 # Use HTTP transport
PORT=3000                          # HTTP server port
TOKEN_STORE_KEY=your-secret        # Encryption key for stored credentials
TOKEN_STORE_PATH=./tokens.enc      # Path for encrypted credential store

# Optional - Basic Settings
DEBUG=false                        # Enable detailed logging (default: false)
DEFAULT_PAGE_SIZE=100             # Default items per page (optional)
MAX_PAGE_SIZE=100                 # Maximum items per page (optional)

# Optional - Intelligent Rules System
ENABLE_RULES_ENGINE=true          # Enable intelligent rules (auto-detected if rules file exists)
MCP_RULES_FILE=custom-rules.md    # Custom technical rules file (optional)
MIN_COVERAGE_THRESHOLD=70         # Minimum coverage percentage (optional)
REQUIRE_UI_VALIDATION=true        # Require UI validation in tests (optional)
REQUIRE_API_VALIDATION=true       # Require API validation in tests (optional)

Instance Configuration File (zebrunner-config.json)

The MCP server ships with a zebrunner-config.json in the project root that contains instance-specific settings. Customize this file to adapt the server to your Zebrunner workspace. If the file is missing or contains invalid values, built-in defaults are used automatically.

{
  "projectAliases": {
    "web": "WEB",
    "android": "AND",
    "ios": "IOS",
    "api": "WEB"
  },
  "testConnectionProjectKey": "MCP",
  "widgetTemplates": {
    "RESULTS_BY_PLATFORM": 8,
    "TOP_BUGS": 4,
    "BUG_REVIEW": 9,
    "FAILURE_INFO": 6,
    "FAILURE_DETAILS": 10
  },
  "dashboardNames": {
    "weeklyResults": "Weekly results",
    "bugsReproRate": "Bugs repro rate"
  },
  "platformMap": {
    "web": [],
    "api": ["api"],
    "android": [],
    "ios": ["ios"]
  },
  "featureAreaKeywords": {
    "quicklog": "Search & Quick Log",
    "search": "Search & Quick Log",
    "notification": "Notifications"
  },
  "localeTestRunRules": {
    "enabled": true,
    "projectKeys": ["AND", "IOS", "WEB"],
    "enUsOnlyFeatureSuites": ["Plans", "Workout Routines", "Recipe Discovery"],
    "suiteNameMatch": "includes"
  },
  "relaunchFailures": {
    "excludeLaunchNamePatterns": ["Performance"],
    "maxLaunchesPerPlatform": 50
  }
}

Key

Description

projectAliases

Maps short names (web, android, etc.) to actual Zebrunner project keys. Update these to match your projects.

testConnectionProjectKey

Project key used by the adv_test_reporting_connection tool when no env var is set.

widgetTemplates

Numeric IDs for SQL widget templates used by reporting tools. These IDs are tenant-specific โ€” check your Zebrunner instance if reports return empty data.

dashboardNames

Dashboard display names used by widget SQL queries. Must match dashboard names in your Zebrunner workspace.

platformMap

Maps platform aliases to widget SQL PLATFORM filter values.

featureAreaKeywords

Keyword-to-label mapping used by regression stability reports to bucket test names into feature areas. Customize for your application's feature structure.

localeTestRunRules

Project-scoped Build Now rules โ€” see below. Used by adv_start_launch when locale โ‰  en_US.

relaunchFailures

Project-scoped rerun rules โ€” see below. Used by /relaunch-regression-failures and referenced by adv_rerun_launch_failures.

Individual keys can be omitted โ€” only the keys you include will override the defaults.

Project-specific automation rules (localeTestRunRules & relaunchFailures)

These optional blocks configure launch mutation workflows per project. They do not affect read-only tools, TCM tools, or projects outside the configured scope.

localeTestRunRules โ€” adv_start_launch (Jenkins Build Now) when the effective locale is not en_US:

Sub-key

Description

enabled

Master switch. Set false to disable locale-based NOT_TAGS logic entirely.

projectKeys

Zebrunner project keys where rules apply (e.g. ["MCP", "DEF"]). Other projects are unchanged.

enUsOnlyFeatureSuites

TCM feature suite names that are English-only; auto-excluded via NOT_TAGS in test_run_rules on preview.

suiteNameMatch

"exact" or "includes" โ€” how suite names are matched when discovering feature suite IDs.

relaunchFailures โ€” /relaunch-regression-failures prompt and batch rerun guidance:

Sub-key

Description

excludeLaunchNamePatterns

Launch names to skip when discovering failures (case-insensitive substring match). Default: ["Performance"].

maxLaunchesPerPlatform

Cap per platform in the prompt workflow (default 50, matches adv_rerun_launch_failures max).

/feature-scoped-launch does not use zebrunner-config.json for Jenkins suite_path โ€” the prompt resolves it from args, recent launches, or by asking the user.

Examples

CUSTOMER-style deployment (shipped defaults in repo zebrunner-config.json):

"localeTestRunRules": {
  "enabled": true,
  "projectKeys": ["MCP", "DEF", "WEB"],
  "enUsOnlyFeatureSuites": ["Plans", "Workout Routines", "Recipe Discovery"],
  "suiteNameMatch": "includes"
},
"relaunchFailures": {
  "excludeLaunchNamePatterns": ["Performance"],
  "maxLaunchesPerPlatform": 50
}

Generic / non-CUSTOMER deployment (e.g. only project MCP for demos, no locale exclusions):

"projectAliases": { "demo": "MCP" },
"testConnectionProjectKey": "MCP",
"localeTestRunRules": { "enabled": false },
"relaunchFailures": {
  "excludeLaunchNamePatterns": [],
  "maxLaunchesPerPlatform": 50
}

Minimal override via env (Docker/K8s without mounting the file):

ZEBRUNNER_CONFIG_JSON='{"localeTestRunRules":{"enabled":false},"relaunchFailures":{"excludeLaunchNamePatterns":["Benchmark"]}}'

See also: docs/RESOURCES_AND_PROMPTS.md for how prompts and tools consume these settings.

Per-User Zebrunner URL (v8.1.0+)

When running in HTTP mode with MCP_AUTH_MODE=selfauth and without setting ZEBRUNNER_URL, each user provides their own Zebrunner instance URL on the login form. This enables multi-tenant hosting where a single MCP server serves users across different Zebrunner organizations.

  • Login form shows a Zebrunner URL field (e.g., https://mcp.zebrunner.com)

  • Users can update their URL, credentials, or disconnect at /settings

  • Per-user URLs are stored encrypted alongside credentials in the token store

  • When ZEBRUNNER_URL IS set as an environment variable, the URL field is hidden and the env value is used globally (existing behavior, no change)

Rules System Configuration

The rules system automatically detects and uses rules files in your project root:

Automatic Detection

If you have a mcp-zebrunner-rules.md file in your project root, the rules engine will automatically enable itself.

Custom Rules Files

You can customize the three types of rules:

  1. Test Case Review Rules (test_case_review_rules.md)

# Custom Test Case Review Rules

## Rule 1: Title Quality
- Titles must be descriptive and specific
- Minimum length: 10 characters
- Should not contain vague terms like "test", "check"

## Rule 2: Test Steps
- Each step must have clear action and expected result
- Steps should be numbered and sequential
- Avoid combining multiple actions in one step
  1. Analysis Checkpoints (test_case_analysis_checkpoints.md)

# Custom Analysis Checkpoints

## Independence Assessment
- [ ] Can this test case run independently?
- [ ] Are all preconditions explicitly stated?
- [ ] No dependencies on other test cases?

## Automation Readiness
- [ ] All steps are unambiguous?
- [ ] Technical feasibility confirmed?
- [ ] Stable selectors available?
  1. Technical Rules (mcp-zebrunner-rules.md)

# Technical Configuration

## Coverage Thresholds
- Overall Coverage: 80%
- Critical Steps: 95%
- UI Validation Steps: 85%

## Framework Detection
**Java/TestNG**:
- Keywords: @Test, TestNG, WebDriver
- File patterns: *Test.java, *Tests.java

โฌ†๏ธ Back to top

๐Ÿงช Testing Your Setup

Run health checks

npm run test:health

Test API connection

npm run smoke

Run full test suite

npm test

Run specific test types

npm run test:unit           # Fast unit tests
npm run test:integration    # API integration tests
npm run test:e2e           # End-to-end tests

โฌ†๏ธ Back to top

๐Ÿ” Troubleshooting

Common Issues

"Authentication failed" or 401 errors

  • โœ… Check your ZEBRUNNER_LOGIN and ZEBRUNNER_TOKEN

  • โœ… Verify your API token is still valid

  • โœ… Ensure your user has proper permissions in Zebrunner

"Project not found" or 404 errors

  • โœ… Check the project key spelling (e.g., "MYAPP", not "myapp")

  • โœ… Verify you have access to the project in Zebrunner

  • โœ… Some endpoints may not be available on all Zebrunner instances

"Connection timeout" errors

  • โœ… Check your ZEBRUNNER_URL is correct

  • โœ… Ensure your network can reach the Zebrunner instance

  • โœ… Try increasing timeout in configuration

MCP integration not working

  • โœ… Verify the path to dist/server.js is correct

  • โœ… Check that the project built successfully (npm run build)

  • โœ… Ensure environment variables are set in MCP configuration

  • โœ… Look at Claude Desktop/Code logs for error messages

Rules engine not working

  • โœ… Check that ENABLE_RULES_ENGINE=true in your .env file

  • โœ… Verify rules files exist and have meaningful content

  • โœ… Restart the MCP server after changing rules files

  • โœ… Check debug logs for rules parsing errors

Debug Mode

Enable detailed logging to troubleshoot issues:

DEBUG=true

This will show:

  • API requests and responses

  • Error details and stack traces

  • Performance metrics

  • Feature availability

  • Rules parsing and validation details

Getting Help

  1. Check the logs - Enable debug mode and look for error messages

  2. Test your connection - Run npm run test:health

  3. Verify your configuration - Double-check your .env file

  4. Check Zebrunner permissions - Ensure your user has proper access

  5. Validate rules files - Ensure rules files have meaningful content

  6. Report issues - If you find a bug, please open an issue at https://github.com/maksimsarychau/mcp-zebrunner/issues

โฌ†๏ธ Back to top

๐ŸŽฏ Example Workflows

Workflow 1: Test Case Review (Manual QA)

1. "Get test case PROJ-123 details"
2. "Validate test case PROJ-123"
3. "Improve test case PROJ-123 with specific suggestions"
4. "Check if test case PROJ-123 is ready for manual execution"

Workflow 2: Test Automation (SDET)

1. "Validate test case PROJ-456 for automation readiness"
2. "Generate Java/Carina test code for PROJ-456"
3. "Analyze coverage between test case and my implementation"
4. "Get automation readiness assessment"

Workflow 3: Implementation Validation (Developer)

1. "Get test case PROJ-789 details to understand requirements"
2. "Analyze coverage for PROJ-789 against my implementation"
3. "Generate unit tests based on test case requirements"
4. "Validate implementation completeness"

Workflow 4: Quality Management (Team Lead)

1. "Get quality metrics for all test cases in project MYAPP"
2. "Show me test cases that need improvement"
3. "Generate team quality report"
4. "Identify automation readiness across the project"

Workflow 5: Project Health (Product Manager)

1. "Get overall test coverage status for project MYAPP"
2. "Show me test execution results by platform"
3. "Generate project testing health report"
4. "Identify quality risks and improvement opportunities"

โฌ†๏ธ Back to top

๐Ÿ”ง Advanced Features

Batch Operations

Process multiple test cases at once:

"Validate all test cases in suite 18708"
"Generate coverage report for all test cases in project MYAPP"
"Improve all test cases that have quality issues"

Custom Output Formats

Get data in the format you need:

"Get test cases as JSON for API integration"
"Show test suite hierarchy in markdown for documentation"
"Generate quality report in markdown for stakeholder presentation"

Find exactly what you need:

"Get test cases created after 2025-01-01"
"Find test cases with automation state 'Manual'"
"Show me test cases that are not ready for automation"

Rules-Based Analysis

Leverage intelligent validation:

"Validate test case PROJ-123 using custom rules from my-project-rules.md"
"Enhanced coverage analysis with framework-specific rules"
"Generate improvement suggestions based on team quality standards"

๐Ÿ“š Additional Documentation

๐Ÿ“– Tool References

๐Ÿง  Intelligent Rules System

๐Ÿ“‹ Rules Files (Customizable)

๐Ÿ” Specialized Guides

๐Ÿ› ๏ธ Feature Documentation

โฌ†๏ธ Back to top

๐Ÿค Contributing

  1. Fork the repository

  2. Create a feature branch

  3. Make your changes with appropriate tests

  4. Ensure all tests pass: npm test

  5. Submit a pull request

๐Ÿ“„ License

License

This project is licensed under AGPL-3.0 to ensure that all improvements to the MCP server remain open-source, especially when the software is deployed as a network service.

If you modify and run the server in a way that users interact with it over a network (e.g., Claude Desktop / Cursor clients), you must make the full modified source code available to those users.

Commercial usage is allowed. For closed-source modifications or enterprise licensing, please contact the author. See LICENSE.md file for details.


๐ŸŽ‰ You're Ready!

Once you've completed the setup:

  1. Test your connection with npm run test:health

  2. Configure your AI assistant with the MCP server

  3. Start asking questions about your test cases!

Example first commands to try:

  • "List test suites for project [YOUR_PROJECT_KEY]"

  • "Get test case [YOUR_TEST_CASE_KEY] details"

  • "Validate test case [YOUR_TEST_CASE_KEY]"

  • "Show me the test suite hierarchy"

The intelligent rules system will help ensure your test cases meet quality standards and are ready for both manual execution and automation. Happy testing! ๐Ÿš€

Available Tools

49 tools
aggregate_test_cases_by_featureA

๐Ÿ” Find ALL test cases related to a specific feature across the project. Searches in title, description, preconditions, and test steps (case-insensitive, partial match). Groups results by Root Suite and Feature Suite, avoiding duplicates. Output formats: detailed (full hierarchy), short (summary), dto (JSON), test_run_rules (for automation tags)

ParametersJSON Schema
NameRequiredDescriptionDefault
project_keyYesProject key (e.g., 'MCPAND', 'MCP')
feature_keywordYesFeature keyword to search for (case-insensitive, partial match)
output_formatNoOutput format: detailed, short, dto, or test_run_rulesshort
tags_formatNoTAGS output format: by_root_suite (separate TAGS line per root suite, default) or single_line (all combined on one line)by_root_suite
max_resultsNoMaximum test cases to process

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively describes key behaviors: it searches across multiple text fields, uses case-insensitive partial matching, groups results by Root Suite and Feature Suite while avoiding duplicates, and offers multiple output formats. This covers search scope, matching logic, result organization, and output options, though it doesn't mention rate limits, permissions, or error handling.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is efficiently structured with four sentences that each add value: the core purpose, search scope and matching logic, result grouping method, and output formats. It uses emojis and formatting for readability without wasting space, and every sentence contributes essential information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's moderate complexity (search across multiple fields with hierarchical grouping), no annotations, and no output schema, the description does well to cover purpose, behavior, and output options. It could be more complete by mentioning what the different output formats contain or any limitations, but it provides sufficient context for an agent to understand when and how to use this tool effectively.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all 5 parameters thoroughly. The description adds some context by mentioning 'case-insensitive, partial match' for the feature_keyword parameter and listing output formats, but doesn't provide additional semantic meaning beyond what's in the schema descriptions. This meets the baseline for high schema coverage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the specific action ('Find ALL test cases related to a specific feature'), the resource ('test cases'), and scope ('across the project'). It distinguishes from siblings by specifying it searches across multiple fields (title, description, preconditions, test steps) and groups results hierarchically, unlike tools like 'get_test_case_by_title' or 'get_test_cases_by_suite_smart' which have narrower scopes.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear context for when to use this tool: when searching for test cases related to a feature across the entire project with case-insensitive partial matching. It doesn't explicitly state when not to use it or name specific alternatives, but the context is sufficient to differentiate from siblings that filter by suite, key, or other criteria.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

analyze_screenshotB

๐Ÿ” Analyze test screenshot with OCR and visual analysis - returns image to Claude Vision for detailed analysis

ParametersJSON Schema
NameRequiredDescriptionDefault
screenshotUrlNoScreenshot URL to download and analyze
screenshotPathNoLocal path to screenshot file
testIdNoTest ID for context
enableOCRNoEnable OCR text extraction (slower)
analysisTypeNobasic=metadata+OCR only, detailed=includes image for Claude Visiondetailed
expectedStateNoExpected UI state for comparison

TDQS

B3.1/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions OCR and visual analysis, and that it 'returns image to Claude Vision for detailed analysis,' which gives some insight into processing behavior. However, it fails to disclose critical traits: whether this is a read-only or mutating operation, performance implications (e.g., 'slower' is noted in schema but not description), authentication needs, rate limits, or error handling. For a tool with 6 parameters and no annotations, this is a significant gap.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is appropriately sized and front-loaded: a single sentence efficiently conveys the core functionality (analyze screenshot with OCR/visual analysis) and key behavioral detail (returns to Claude Vision). Every word earns its place with no redundancy or fluff, making it easy to scan and understand quickly.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (6 parameters, no output schema, no annotations), the description is moderately complete. It covers the purpose and high-level behavior but lacks details on when to use it vs. siblings, full behavioral traits, and output expectations. Without annotations or output schema, the description should do more to compensate, but it provides a basic foundation that's adequate for simple use cases.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema description coverage is 100%, so the schema already documents all parameters thoroughly. The description adds minimal value beyond the schema: it implies the tool uses OCR and visual analysis, which relates to 'enableOCR' and 'analysisType' parameters, but doesn't provide additional syntax, format details, or usage examples. With high schema coverage, the baseline is 3, and the description doesn't significantly enhance parameter understanding.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: 'Analyze test screenshot with OCR and visual analysis' specifies the verb (analyze) and resource (test screenshot) with the methods used (OCR and visual analysis). It distinguishes from sibling tools like 'download_test_screenshot' (which only downloads) and 'analyze_test_execution_video' (which analyzes videos). However, it doesn't explicitly mention what makes it unique from other analysis tools in the list.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides minimal guidance: it mentions the tool returns 'image to Claude Vision for detailed analysis,' which implies when detailed visual analysis is needed. However, it lacks explicit when-to-use criteria, doesn't specify alternatives among siblings (e.g., when to use this vs. 'analyze_test_failure'), and offers no exclusions or prerequisites. This leaves usage context largely implied.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

analyze_test_cases_duplicatesB

๐Ÿ” Analyze test cases for duplicates and group similar ones by step similarity (80-90%)

ParametersJSON Schema
NameRequiredDescriptionDefault
project_keyYesProject key (e.g., 'ANDROID', 'IOS')
suite_idNoOptional: Analyze specific test suite ID
test_case_keysNoOptional: Analyze specific test case keys instead of suite
similarity_thresholdNoSimilarity threshold percentage (50-100, default: 80)
formatNoOutput formatmarkdown
include_similarity_matrixNoInclude detailed similarity matrix in output
include_clickable_linksNoInclude clickable links to Zebrunner web UI (markdown format only)

TDQS

B3.4/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries full burden but provides minimal behavioral context. It mentions grouping by step similarity with a threshold range (80-90%), but doesn't disclose what the analysis returns, whether it's read-only or modifies data, performance characteristics, or authentication requirements. The description adds some value but leaves significant gaps.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence with zero waste. It's front-loaded with the core purpose and includes the key constraint (similarity threshold range) in parentheses. Every word earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 7-parameter analysis tool with no annotations and no output schema, the description is insufficient. It doesn't explain what the analysis returns, how results are structured, whether it's a read operation, or any behavioral constraints. The description should do more to compensate for the lack of structured metadata.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema fully documents all 7 parameters. The description mentions 'step similarity (80-90%)' which aligns with the 'similarity_threshold' parameter's default range, but doesn't add meaningful semantics beyond what the schema already provides. Baseline 3 is appropriate when schema does the heavy lifting.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb ('analyze'), resource ('test cases'), and specific purpose ('for duplicates and group similar ones by step similarity (80-90%)'). It distinguishes from siblings like 'analyze_test_cases_duplicates_semantic' by specifying step-based similarity rather than semantic analysis.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for duplicate detection with step similarity, but doesn't explicitly state when to use this tool versus alternatives like 'analyze_test_cases_duplicates_semantic' or 'aggregate_test_cases_by_feature'. No exclusions or prerequisites are mentioned.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

analyze_test_cases_duplicates_semanticC

๐Ÿง  Advanced semantic duplicate analysis using LLM-powered step clustering and two-phase analysis

ParametersJSON Schema
NameRequiredDescriptionDefault
project_keyYesProject key (e.g., 'ANDROID', 'IOS')
suite_idNoOptional: Analyze specific test suite ID
test_case_keysNoOptional: Analyze specific test case keys instead of suite
similarity_thresholdNoTest case similarity threshold percentage (50-100, default: 80)
step_clustering_thresholdNoStep clustering threshold percentage (50-100, default: 85)
analysis_modeNoAnalysis mode: basic (fast), semantic (LLM-powered), hybrid (both)hybrid
use_step_clusteringNoEnable two-phase clustering (step clusters first, then test case clusters)
use_medoid_selectionNoUse medoid-based representative selection instead of heuristic
include_semantic_insightsNoGenerate semantic insights about workflows and patterns
formatNoOutput formatmarkdown
include_similarity_matrixNoInclude detailed similarity matrix in output
include_clickable_linksNoInclude clickable links to Zebrunner web UI (markdown format only)

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden of behavioral disclosure. While it mentions 'advanced semantic duplicate analysis' and 'two-phase analysis,' it fails to describe critical behaviors such as computational intensity, potential rate limits, authentication requirements, or what the output looks like (e.g., clusters, insights). For a complex tool with 12 parameters, this is a significant gap.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence that front-loads key information ('Advanced semantic duplicate analysis') and uses emojis and technical terms appropriately. Every word earns its place, making it highly concise and well-structured for quick comprehension.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (12 parameters, no annotations, no output schema), the description is incomplete. It lacks details on behavioral traits, output format expectations, and usage context. While the schema covers parameters well, the description doesn't address the broader operational context needed for effective tool invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all 12 parameters thoroughly. The description adds no additional meaning about parameters beyond what's in the schema (e.g., it doesn't explain how 'step_clustering_threshold' interacts with 'similarity_threshold' or clarify the 'hybrid' mode). With high schema coverage, the baseline is 3, and the description doesn't compensate with extra insights.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool performs 'semantic duplicate analysis' using 'LLM-powered step clustering and two-phase analysis,' which is a specific verb+resource combination. However, it doesn't explicitly differentiate from its sibling 'analyze_test_cases_duplicates' (which likely performs basic duplicate analysis), leaving some ambiguity about when to choose one over the other.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites, compare with sibling tools like 'analyze_test_cases_duplicates,' or specify scenarios where this advanced analysis is preferred over basic methods. This leaves the agent without context for tool selection.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

analyze_test_execution_videoA

๐ŸŽฌ Download and analyze test execution video with Claude Vision - extracts frames, compares with test case, and predicts if failure is bug or test issue. NEW: Analysis depth modes (quick/standard/detailed), parallel frame extraction, similar failures search, and historical trends analysis!

ParametersJSON Schema
NameRequiredDescriptionDefault
testIdYesTest ID from Zebrunner
testRunIdYesLaunch ID / Test Run ID
projectKeyNoProject key (MCP, etc.)
projectIdNoProject ID (alternative to projectKey)
extractionModeNoFrame extraction mode: failure_focused (10 frames), smart (20 frames), full_test (30 frames)smart
frameIntervalNoSeconds between frames for full_test mode
failureWindowSecondsNoTime window around failure (seconds)
compareWithTestCaseNoCompare with test case steps
testCaseKeyNoOverride test case key
analysisDepthNoAnalysis depth: quick_text_only (no frames, ~10-20s), standard (8-12 frames for failure+coverage, ~30-60s), detailed (20-30 frames with OCR, ~60-120s)standard
includeOCRNoExtract text from frames using OCR (slow, adds 2-3s per frame)
analyzeSimilarFailuresNoFind similar failures in project (last 30 days, top 10)
includeHistoricalTrendsNoAnalyze test stability and flakiness (last 30 runs)
includeLogCorrelationNoCorrelate frames with log timestamps
formatNoOutput formatdetailed
generateVideoReportNoGenerate timestamped report

TDQS

A3.9/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries full burden and does well by disclosing key behavioral traits: it mentions performance characteristics ('quick_text_only (no frames, ~10-20s)'), processing details ('parallel frame extraction'), and additional capabilities ('similar failures search, historical trends analysis'). However, it doesn't mention authentication requirements, rate limits, or error handling scenarios.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is appropriately sized and front-loaded with the core purpose in the first sentence. The feature list in the second sentence is somewhat dense but relevant. Every sentence earns its place by conveying important capabilities, though the exclamation point and 'NEW' tag could be considered slightly promotional rather than purely informative.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (16 parameters, no output schema, no annotations), the description does a good job covering the tool's scope and capabilities. It explains what the tool does, mentions analysis modes, and highlights key features. However, for such a complex tool, it could benefit from more guidance on output format or result interpretation since there's no output schema provided.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the baseline is 3. The description doesn't add meaningful parameter semantics beyond what's already in the schema - it mentions analysis depth modes and parallel frame extraction but doesn't explain parameter interactions or provide usage examples. The schema already thoroughly documents all 16 parameters with descriptions and defaults.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: 'Download and analyze test execution video with Claude Vision - extracts frames, compares with test case, and predicts if failure is bug or test issue.' It uses specific verbs (download, analyze, extracts, compares, predicts) and distinguishes from sibling tools like analyze_screenshot or analyze_test_failure by focusing specifically on video analysis.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage context through its feature list ('NEW: Analysis depth modes...') but doesn't explicitly state when to use this tool versus alternatives like analyze_screenshot or analyze_test_failure. It suggests video analysis is appropriate but doesn't provide guidance on prerequisites or when other tools might be better suited.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

analyze_test_failureA

๐Ÿ” Deep forensic analysis of failed test including logs, screenshots, error classification, and similar failures. ๐Ÿ’ก NEW: Compare with last passed execution to see what changed! ๐Ÿ’ก TIP: Can be auto-invoked from Zebrunner test URLs like: https://workspace.zebrunner.com/projects/PROJECT/automation-launches/LAUNCH_ID/tests/TEST_ID

ParametersJSON Schema
NameRequiredDescriptionDefault
testIdYesTest ID (e.g., 5451420)
testRunIdYesTest Run ID / Launch ID (e.g., 120806)
projectKeyNoProject key (e.g., 'MCP') - alternative to projectId
projectIdNoProject ID - alternative to projectKey
includeScreenshotsNoInclude screenshot links
includeLogsNoInclude log analysis
includeArtifactsNoInclude all test artifacts
includePageSourceNoInclude page source analysis
includeVideoNoInclude video URL
analyzeSimilarFailuresNoFind similar failures in the launch
analyzeScreenshotsWithAINoDownload and analyze screenshots with AI (Claude Vision)
screenshotAnalysisTypeNoScreenshot analysis type: basic (metadata+OCR) or detailed (includes Claude Vision)detailed
formatNoOutput format: detailed, summary, or jira (ready for Jira ticket creation)detailed
compareWithLastPassedNoCompare current failure with last passed execution to identify what changed

TDQS

A3.8/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It mentions the tool's capabilities (logs, screenshots, error classification, similar failures, comparison) and auto-invocation from URLs, which adds useful behavioral context. However, it doesn't disclose operational traits like rate limits, authentication needs, or potential side effects (e.g., whether it triggers downloads or external AI calls). The description is informative but lacks full transparency on such behavioral aspects.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is appropriately sized and front-loaded: the first sentence states the core purpose, followed by new features and a usage tip. Each sentence adds value (e.g., highlighting new comparison capability and auto-invocation). It's efficient with no wasted words, though the emojis and formatting slightly reduce professionalism without harming clarity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (14 parameters, nested objects, no output schema) and no annotations, the description is moderately complete. It covers the purpose, key features, and a usage hint, but lacks details on output format, error handling, or prerequisites. For a forensic analysis tool with many parameters, more context on expected results or limitations would enhance completeness.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all 14 parameters thoroughly. The description adds minimal parameter semantics beyond the schemaโ€”it implies analysis includes logs, screenshots, and comparisons, which aligns with parameters like 'includeLogs' and 'compareWithLastPassed', but doesn't provide additional syntax or format details. With high schema coverage, the baseline score of 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: 'Deep forensic analysis of failed test including logs, screenshots, error classification, and similar failures.' It specifies the verb ('analyze') and resource ('failed test'), plus the scope of analysis. It distinguishes from siblings like 'analyze_screenshot' or 'detailed_analyze_launch_failures' by focusing on a single test's forensic details and comparison features.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear context for when to use it: analyzing failed tests with forensic details and comparisons. It explicitly mentions auto-invocation from Zebrunner URLs, which is helpful. However, it doesn't specify when NOT to use it or name alternatives among siblings (e.g., 'detailed_analyze_launch_failures' for broader analysis), so it's not fully explicit about alternatives.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

detailed_analyze_launch_failuresA

๐Ÿš€ Analyze failed tests WITHOUT linked issues in a launch with grouping, statistics, and recommendations. Automatically analyzes all tests if โ‰ค10, otherwise first 10 (use offset/limit for more). Use filterType: 'all' to include tests with issues. Supports pagination and screenshot analysis. NEW: Jira format with smart grouping - creates combined tickets for similar errors! ๐Ÿ’ก TIP: Can be auto-invoked from Zebrunner launch URLs like: https://workspace.zebrunner.com/projects/PROJECT/automation-launches/LAUNCH_ID

ParametersJSON Schema
NameRequiredDescriptionDefault
testRunIdYesLaunch ID / Test Run ID (e.g., 120806)
projectKeyNoProject key (e.g., 'MCP') - alternative to projectId
projectIdNoProject ID - alternative to projectKey
filterTypeNoFilter: 'without_issues' = only tests without linked Jira tickets (DEFAULT), 'all' = all failed testswithout_issues
includeScreenshotAnalysisNoDownload and analyze screenshots with AI for each test (increases analysis time)
screenshotAnalysisTypeNoScreenshot analysis type if enableddetailed
formatNoOutput format: 'detailed' = full analysis, 'summary' = condensed, 'jira' = ready for Jira tickets with smart groupingsummary
jiraDetailLevelNoJira detail level: 'basic' = fast (no deep analysis), 'full' = comprehensive with deep analysis (DEFAULT, slower but thorough)full
executionModeNoExecution mode: sequential (safe), parallel (fast), or batches (balanced)sequential
batchSizeNoBatch size if executionMode is 'batches' (default: 5)
offsetNoPagination offset - start from test N (e.g., 0 for first 20, 20 for next 20)
limitNoNumber of tests to analyze (default: 20, max recommended: 30)

TDQS

A4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden and does well by disclosing key behavioral traits: automatic analysis limits (โ‰ค10 tests analyzed fully, otherwise first 10), pagination support, screenshot analysis impact on time, Jira smart grouping, and execution mode options affecting speed/safety. It could improve by clarifying output format details or error handling, but covers most critical aspects.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is front-loaded with the core purpose, but contains some redundancy (e.g., mentioning grouping/statistics/recommendations multiple times) and includes promotional elements like emojis and 'NEW' tags that don't add functional clarity. The tip about auto-invocation is useful but could be more concise. Overall, it's adequately structured but could be tighter.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a complex tool with 12 parameters and no output schema, the description provides substantial context about behavior, limitations, and practical usage. It covers analysis scope, pagination, format options, and performance considerations. The main gap is the lack of information about return values or output structure, which would be helpful given the absence of an output schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the baseline is 3. The description adds some value by explaining the practical implications of parameters (e.g., filterType usage, automatic analysis limits, Jira grouping), but doesn't provide significant additional semantics beyond what's already well-documented in the schema. It reinforces parameter purposes without contradicting them.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: analyzing failed tests in a launch with grouping, statistics, and recommendations. It specifies the scope (tests without linked issues by default, with filter options) and distinguishes from siblings by mentioning unique capabilities like Jira format with smart grouping and screenshot analysis, which aren't present in other tools like 'analyze_test_failure' or 'get_launch_summary'.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear context on when to use this tool: for analyzing failed tests in a launch, with explicit guidance on filterType options (e.g., 'without_issues' vs 'all'). It mentions auto-invocation from Zebrunner URLs as a tip. However, it doesn't explicitly state when NOT to use it or name specific alternatives among the many sibling tools, though the unique features imply differentiation.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

download_test_screenshotC

๐Ÿ“ธ Download test screenshot with authentication from Zebrunner

ParametersJSON Schema
NameRequiredDescriptionDefault
screenshotUrlYesScreenshot URL (e.g., 'https://your-workspace.zebrunner.com/files/abc123' or '/files/abc123')
testIdNoTest ID for context
projectKeyNoProject key for context
outputPathNoCustom output path (default: temp directory)
returnBase64NoReturn base64 encoded image

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden. It mentions authentication but doesn't specify required permissions, rate limits, or what happens if the download fails. The behavioral context is minimal for a tool that likely involves network operations and file handling.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence with no wasted words. It's front-loaded with the core action and resource, making it easy to understand at a glance.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with 5 parameters, no annotations, and no output schema, the description is insufficient. It doesn't explain what the tool returns (e.g., file path, base64 data) or error handling, leaving gaps in understanding its full behavior and output.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema fully documents all parameters. The description adds no additional meaning beyond implying authentication is needed, which doesn't clarify parameter usage. Baseline 3 is appropriate as the schema handles parameter documentation.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('Download') and resource ('test screenshot') with the platform ('Zebrunner'), and mentions authentication. However, it doesn't explicitly differentiate from sibling tools like 'analyze_screenshot', which might be for analysis rather than downloading.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance is provided on when to use this tool versus alternatives like 'analyze_screenshot' or other test-related tools. The description lacks context about prerequisites or scenarios for downloading screenshots.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

generate_draft_test_by_keyB

๐Ÿงช Generate draft test code from Zebrunner test case with intelligent framework detection

ParametersJSON Schema
NameRequiredDescriptionDefault
project_keyNoProject key (auto-detected from case_key if not provided)
case_keyYesTest case key (e.g., 'ANDROID-6')
implementation_contextYesImplementation context (existing code, file paths, or framework hints)
target_frameworkNoTarget test framework (auto-detected if 'auto')auto
output_formatNoOutput format for generated testcode
include_setup_teardownNoInclude setup and teardown code
include_assertions_templatesNoInclude assertion templates
generate_page_objectsNoGenerate page object classes
include_data_providersNoInclude data provider templates
include_suite_hierarchyNoInclude featureSuiteId and rootSuiteId information
file_pathNoFile path for saving generated code (optional)

TDQS

B3.3/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries full burden but only mentions 'intelligent framework detection' as a behavioral trait. It doesn't disclose whether this is a read-only operation, if it modifies data, what permissions are needed, rate limits, or what the output looks like. For a code generation tool with 11 parameters, this is insufficient behavioral disclosure.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence that front-loads the core purpose. The emoji adds character but doesn't detract from clarity. Every word earns its place, though it could potentially be more specific about the generation scope.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a complex code generation tool with 11 parameters and no output schema, the description is inadequate. It doesn't explain what the generated output contains, how the framework detection works, what happens when file_path is provided, or the relationship between parameters. With no annotations and rich parameter schema, the description should provide more contextual guidance.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all 11 parameters thoroughly. The description adds no additional parameter semantics beyond what's in the schema, but doesn't need to compensate for gaps. The baseline 3 is appropriate when the schema does all the parameter documentation work.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the specific action ('Generate draft test code') and resource ('from Zebrunner test case'), including the intelligent framework detection feature. It distinguishes itself from sibling tools like 'get_test_case_by_key' or 'improve_test_case' by focusing on code generation rather than retrieval or enhancement.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage context through 'intelligent framework detection' and the tool name suggests it's for creating test drafts from existing cases. However, it lacks explicit guidance on when to use this vs. alternatives like 'improve_test_case' or 'get_test_case_by_key', and doesn't mention prerequisites or exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_all_launches_for_projectB

๐Ÿ“‹ Get all launches for a project with pagination (uses new reporting API)

ParametersJSON Schema
NameRequiredDescriptionDefault
projectYesProject alias (web/android/ios/api), project key, or project ID
pageNoPage number (starts from 1)
pageSizeNoNumber of launches per page (max 100)
formatNoOutput format - 'raw' for full API response, 'formatted' for user-friendly displayformatted

TDQS

B3.4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden. It discloses behavioral traits like pagination support and API version ('new reporting API'), which is helpful. However, it doesn't mention potential rate limits, authentication requirements, error conditions, or what the output looks like (though there's no output schema). For a tool with 4 parameters and no annotations, this is a moderate but incomplete disclosure.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence that front-loads the core purpose. The emoji (๐Ÿ“‹) adds visual cue but minimal informational value. No wasted words, though it could be slightly more structured (e.g., separating pagination note).

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given 4 parameters, 100% schema coverage, no annotations, and no output schema, the description provides basic context but lacks depth. It mentions pagination and API version, which helps, but doesn't cover output format details, error handling, or sibling tool differentiation. For a list/retrieval tool, this is minimally adequate but leaves gaps in full agent guidance.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema description coverage is 100%, with all parameters well-documented in the schema itself. The description adds no additional parameter semantics beyond what's in the schema (e.g., it doesn't explain 'project' aliases or 'format' implications further). Baseline 3 is appropriate when the schema does the heavy lifting, though the description could have enhanced understanding of parameter interactions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb ('Get all launches') and resource ('for a project'), making the purpose unambiguous. It specifies 'with pagination' and 'uses new reporting API', which adds useful context. However, it doesn't explicitly differentiate from sibling tools like 'get_all_launches_with_filter' or 'get_launch_details', missing full sibling differentiation.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage context through 'with pagination' and 'uses new reporting API', suggesting this is for retrieving multiple launches in a paginated manner. However, it doesn't provide explicit guidance on when to use this tool versus alternatives like 'get_all_launches_with_filter' or 'get_launch_details', leaving the agent to infer based on tool names alone.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_all_launches_with_filterC

๐Ÿ” Get launches with filtering by milestone, build number, or launch name (uses new reporting API)

ParametersJSON Schema
NameRequiredDescriptionDefault
projectYesProject alias (web/android/ios/api), project key, or project ID
milestoneNoFilter by milestone name (e.g., '25.39.0')
queryNoSearch query for build number or launch name (e.g., 'your-app-25.39.0-45915' or 'Performance')
pageNoPage number (starts from 1)
pageSizeNoNumber of launches per page (max 100)
formatNoOutput format - 'raw' for full API response, 'formatted' for user-friendly displayformatted

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions the tool 'uses new reporting API' which hints at implementation details but doesn't describe critical behaviors like pagination handling (implied by page/pageSize parameters but not explained), rate limits, authentication requirements, or what happens when no filters are applied. The description is insufficient for a tool with 6 parameters and no annotation coverage.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence that front-loads the main purpose with an emoji for visual emphasis. Every element serves a purpose: the emoji indicates search/retrieval, the verb 'Get' is clear, and the parenthetical adds implementation context. However, the 'new reporting API' mention could be more specific about benefits.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with 6 parameters, no annotations, and no output schema, the description is inadequate. It doesn't explain the relationship between parameters (e.g., how 'query' interacts with 'milestone'), doesn't describe the return format or structure, and provides no error handling or behavioral context. The mention of 'new reporting API' is vague and doesn't compensate for these gaps.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all parameters thoroughly. The description adds minimal value by mentioning filtering capabilities ('by milestone, build number, or launch name') which aligns with the 'milestone' and 'query' parameters, but doesn't provide additional context beyond what's in the schema descriptions. Baseline 3 is appropriate when the schema does the heavy lifting.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('Get launches') and specifies filtering capabilities ('with filtering by milestone, build number, or launch name'), which distinguishes it from simple listing tools. However, it doesn't explicitly differentiate from sibling tools like 'get_all_launches_for_project' or 'get_launch_details' beyond mentioning the 'new reporting API'.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives like 'get_all_launches_for_project' or 'get_launch_details'. It mentions the 'new reporting API' but doesn't explain why this matters or when to prefer this tool over others. No exclusions or prerequisites are stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_all_subsuitesB

๐Ÿ“‹ Get all subsuites from a root suite as flat list with pagination

ParametersJSON Schema
NameRequiredDescriptionDefault
project_keyYesProject key
root_suite_idYesRoot suite ID to get all subsuites from
include_rootNoInclude the root suite in results
formatNoOutput formatjson
pageNoPage number (0-based)
sizeNoPage size (configurable via MAX_PAGE_SIZE env var)

TDQS

B3.1/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries full burden. It mentions pagination but doesn't disclose rate limits, authentication needs, whether it's read-only or destructive, or what happens with invalid inputs. For a tool with 6 parameters and no annotation coverage, this leaves significant behavioral gaps.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence that front-loads the core purpose. Every word earns its place: the emoji adds visual cue, 'Get all subsuites' states the action, 'from a root suite' specifies scope, and 'as flat list with pagination' clarifies output format and behavior.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a read operation with 6 parameters and no output schema, the description is adequate but incomplete. It covers the basic purpose and output structure but lacks behavioral details (rate limits, errors) and sibling differentiation. With no annotations and no output schema, more context would be helpful for safe invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so parameters are well-documented in the schema itself. The description adds minimal value beyond implying the tool operates on a 'root suite' and produces a 'flat list', which aligns with the schema but doesn't provide additional semantic context beyond what's already in parameter descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb ('Get') and resource ('all subsuites from a root suite'), and specifies the output structure ('as flat list with pagination'). However, it doesn't differentiate from sibling tools like 'get_suite_hierarchy' or 'get_root_suites', which likely provide hierarchical or root-only views respectively.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance is provided on when to use this tool versus alternatives. With siblings like 'get_suite_hierarchy' (likely hierarchical) and 'get_root_suites' (likely root-only), the description lacks explicit comparisons or context for choosing this flat-list approach over other suite retrieval methods.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_all_tcm_test_cases_by_projectB

๐Ÿ“‹ Get ALL TCM test cases by project using comprehensive pagination

ParametersJSON Schema
NameRequiredDescriptionDefault
project_keyYesProject key (e.g., 'android' or 'ANDROID')
formatNoOutput formatjson
include_clickable_linksNoInclude clickable links to Zebrunner web UI
max_resultsNoMaximum number of results (configurable limit for performance)

TDQS

B3.1/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries full burden but only mentions 'comprehensive pagination', leaving gaps: it doesn't disclose rate limits, authentication needs, error handling, or whether this is a read-only operation. For a tool with 4 parameters and no annotations, this is insufficient behavioral context.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Single sentence, front-loaded with key information ('Get ALL TCM test cases by project'), and uses an emoji for visual clarity. Every word earns its place with zero waste.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given 4 parameters, 100% schema coverage, no output schema, and no annotations, the description is minimally adequate. It covers the core purpose but lacks details on behavioral traits, usage context, and output expectations, leaving gaps for an AI agent to infer.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema fully documents all 4 parameters. The description adds no additional parameter semantics beyond implying pagination (via 'comprehensive pagination'), which aligns with the 'max_results' parameter but doesn't provide extra syntax or format details.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('Get ALL TCM test cases') and resource ('by project'), with 'comprehensive pagination' hinting at scope. It distinguishes from siblings like 'get_test_case_by_key' or 'get_test_cases_by_suite_smart' by emphasizing 'ALL' and project-level retrieval, though it doesn't explicitly contrast them.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No explicit guidance on when to use this tool versus alternatives like 'get_all_tcm_test_cases_with_root_suite_id' or 'get_test_cases_by_suite_smart'. The description implies it's for bulk retrieval by project, but lacks context on prerequisites, performance trade-offs, or specific use cases.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_all_tcm_test_case_suites_by_projectC

๐Ÿ“‹ Get ALL TCM test case suites by project using comprehensive pagination

ParametersJSON Schema
NameRequiredDescriptionDefault
project_keyYesProject key (e.g., 'android' or 'ANDROID')
include_hierarchyNoInclude hierarchy information (rootSuiteId, parentSuiteName, etc.)
formatNoOutput formatjson

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions 'comprehensive pagination', which hints at handling large datasets, but fails to describe critical traits like whether this is a read-only operation, potential rate limits, authentication requirements, or what the output looks like (e.g., structure, size). This leaves significant gaps for a tool that likely returns extensive data.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence that front-loads the key action ('Get ALL TCM test case suites by project') and adds a useful detail ('using comprehensive pagination'). There's no wasted verbiage, though it could be slightly more structured (e.g., separating purpose from behavioral notes).

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the complexity implied by 'ALL' and 'comprehensive pagination', and the absence of both annotations and an output schema, the description is incomplete. It doesn't address how results are returned (e.g., pagination details, response format), performance considerations, or error handling, which are crucial for an agent to use this tool effectively in a TCM context.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema description coverage is 100%, so the schema already documents all parameters thoroughly (project_key, include_hierarchy, format). The description adds no additional meaning beyond what the schema provides, such as explaining the impact of 'include_hierarchy' or when to use different 'format' options. This meets the baseline for high schema coverage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb ('Get') and resource ('ALL TCM test case suites by project'), making the purpose understandable. It distinguishes itself from sibling tools like 'get_tcm_test_suites_by_project' by emphasizing 'ALL' and 'comprehensive pagination', though it doesn't explicitly name alternatives.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives like 'get_tcm_test_suites_by_project' or 'get_all_subsuites'. It mentions 'comprehensive pagination' but doesn't explain when this is necessary or preferable, leaving the agent to infer usage context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_all_tcm_test_cases_with_root_suite_idC

๐ŸŒณ Get ALL TCM test cases enriched with root suite ID information

ParametersJSON Schema
NameRequiredDescriptionDefault
project_keyYesProject key (e.g., 'android' or 'ANDROID')
formatNoOutput formatjson

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden of behavioral disclosure. It mentions retrieving 'ALL' test cases but doesn't clarify scope limitations, pagination, performance implications, authentication requirements, or what 'enriched with root suite ID information' entails in practice. This leaves significant gaps for a tool that presumably handles large datasets.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence with zero wasted words. It uses an emoji for visual distinction but maintains clarity. Every element ('Get ALL', 'TCM test cases', 'enriched with root suite ID information') serves a purpose.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool that retrieves 'ALL' test cases with no output schema and no annotations, the description is insufficient. It doesn't address critical context like return format details (beyond the 'format' parameter), handling of large result sets, error conditions, or how the enrichment differs from basic test case data. This leaves too many unknowns for effective use.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents both parameters thoroughly. The description doesn't add any meaningful parameter semantics beyond what's in the schema (e.g., it doesn't explain why 'project_key' is required or how 'format' affects the 'enriched' output). Baseline 3 is appropriate when the schema does the heavy lifting.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('Get ALL TCM test cases') and specifies the enrichment ('with root suite ID information'), which is a specific verb+resource combination. However, it doesn't explicitly distinguish this tool from sibling tools like 'get_all_tcm_test_cases_by_project' or 'get_test_cases_advanced', which likely have overlapping functionality.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives. With multiple sibling tools for retrieving test cases (e.g., 'get_all_tcm_test_cases_by_project', 'get_test_cases_advanced'), there's no indication of what makes this tool unique or when it should be preferred over others.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_automation_prioritiesB

๐ŸŽฏ Get available priorities for a project (names and IDs)

ParametersJSON Schema
NameRequiredDescriptionDefault
projectYesProject alias (web/android/ios/api), project key, or project ID
formatNoOutput formatjson

TDQS

B3.1/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions the output includes 'names and IDs' but doesn't describe the return format beyond that, nor does it address authentication requirements, rate limits, error conditions, or whether this is a read-only operation. The description is minimal and leaves critical behavioral aspects unspecified for a tool with parameters.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is extremely concise at just one sentence with an emoji, front-loading the core purpose immediately. Every word earns its place, with no redundant information or fluff. The structure is optimal for quick comprehension while being complete enough for the stated purpose.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's moderate complexity (2 parameters, no output schema, no annotations), the description is minimally complete. It states what the tool does but lacks context about when to use it, what the output looks like beyond 'names and IDs', or how it fits into broader workflows. For a read operation with good schema coverage, this is adequate but leaves gaps in usage guidance and behavioral transparency.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema description coverage is 100%, with both parameters well-documented in the schema itself. The description adds no additional parameter semantics beyond what's already in the schema (e.g., it doesn't explain the relationship between project alias/key/ID or when to use each format). This meets the baseline of 3 for high schema coverage without adding value.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb 'Get' and the resource 'available priorities for a project', including the specific output 'names and IDs'. It distinguishes this tool from siblings by focusing on priorities rather than test cases, launches, or other project data. However, it doesn't explicitly differentiate from similar tools like 'get_automation_states' or 'get_project_milestones' that might also retrieve project metadata.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites, timing considerations, or relationships to other tools like 'get_available_projects' (which might need to be called first) or 'get_automation_states' (which might retrieve related metadata). The agent must infer usage from the tool name and description alone.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_automation_statesB

๐Ÿ”ง Get available automation states for a project (names and IDs)

ParametersJSON Schema
NameRequiredDescriptionDefault
projectYesProject alias (web/android/ios/api), project key, or project ID
formatNoOutput formatjson

TDQS

B3.1/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It states the tool retrieves data but doesn't disclose behavioral traits like whether it's read-only, requires authentication, has rate limits, or what the output looks like (e.g., list format, pagination). This leaves significant gaps for a tool that likely interacts with project data.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence with zero waste. It front-loads the purpose with an emoji for visual clarity, making it easy to scan and understand quickly without unnecessary details.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's moderate complexity (2 parameters, no output schema, no annotations), the description is minimally adequate. It states what the tool does but lacks context on usage, behavioral traits, or output details. With no annotations or output schema, more completeness would be beneficial, but it meets a basic threshold.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema description coverage is 100%, with clear descriptions for both parameters (project and format). The description adds no additional parameter semantics beyond what the schema provides, such as explaining the significance of automation states or output format implications. Baseline 3 is appropriate since the schema does the heavy lifting.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb ('Get') and resource ('automation states for a project'), specifying that it retrieves names and IDs. It distinguishes from siblings like 'get_test_cases_by_automation_state' by focusing on available states rather than test cases, though it doesn't explicitly mention this distinction.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance is provided on when to use this tool versus alternatives. It doesn't mention prerequisites, such as needing a valid project identifier, or compare it to siblings like 'get_available_projects' or 'get_test_cases_by_automation_state' for context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_available_projectsC

๐Ÿ—๏ธ Discover available projects with their keys and IDs for dynamic project selection

ParametersJSON Schema
NameRequiredDescriptionDefault
starredNoFilter by starred projects (true=only starred, false=only non-starred, undefined=all)
publiclyAccessibleNoFilter by public accessibility (true=only public, false=only private, undefined=all)
formatNoOutput format: raw API response or formatted dataformatted
includePaginationInfoNoInclude pagination metadata from projects-limit endpoint

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the tool 'discovers' projects, implying a read-only operation, but doesn't mention critical behaviors like authentication requirements, rate limits, pagination handling (despite a parameter for pagination info), or error conditions. This is inadequate for a tool with multiple parameters and no output schema.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence with no wasted words. It front-loads the core purpose and uses an emoji for visual clarity. Every part of the sentence contributes to understanding the tool's role, making it appropriately concise and well-structured.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (4 parameters, no annotations, no output schema), the description is insufficient. It lacks details on behavioral traits, output format expectations (beyond 'keys and IDs'), and usage context. While the schema covers parameters well, the description doesn't compensate for missing annotations or output schema, leaving gaps in overall understanding.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The description adds no parameter-specific information beyond what the schema provides. Since schema description coverage is 100%, the baseline score is 3. The description's mention of 'dynamic project selection' loosely relates to filtering but doesn't elaborate on parameter usage or interactions, so it doesn't enhance the schema's documentation.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: 'Discover available projects with their keys and IDs for dynamic project selection.' It uses specific verbs ('discover') and identifies the resource ('projects') along with key output attributes ('keys and IDs'). However, it doesn't explicitly differentiate from sibling tools like 'get_project_milestones' or 'get_all_launches_for_project,' which prevents a perfect score.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives. It mentions 'dynamic project selection' but doesn't specify scenarios, prerequisites, or exclusions. With many sibling tools available (e.g., 'get_all_launches_for_project'), the lack of comparative context leaves the agent without clear usage direction.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_bug_failure_infoC

๐Ÿ”ฌ Get comprehensive failure information including failure summary and detailed test runs (SQL widgets, templateId: 6 & 10)

ParametersJSON Schema
NameRequiredDescriptionDefault
projectNoProject alias ('web', 'android', 'ios', 'api'), project key, or numeric projectIdweb
dashboardIdYesDashboard ID from bug review (e.g., 99)
hashcodeYesHashcode from bug review failure link (e.g., '1051677506')
periodNoTime period for failure analysisLast 14 Days
formatNoOutput format: detailed (full info), summary (concise), or json (raw data)detailed

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries full burden. While it mentions what information is retrieved (failure summary, detailed test runs), it doesn't disclose behavioral aspects like whether this is a read-only operation, potential performance characteristics, authentication requirements, error conditions, or what happens with invalid inputs. The mention of 'SQL widgets' and templateIds provides some implementation context but not enough operational transparency.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence that front-loads the core purpose. The emoji adds visual distinction but doesn't detract from clarity. While concise, it could potentially benefit from slightly more structure given the tool's complexity and lack of annotations.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with 5 parameters, no annotations, and no output schema, the description provides adequate basic purpose but leaves significant gaps. It doesn't explain what the output looks like (crucial without an output schema), doesn't cover error handling, and provides no guidance on tool selection among many similar siblings. The mention of specific templateIds (6 & 10) adds some domain context but doesn't compensate for the broader completeness gaps.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 100% schema description coverage, the schema already documents all 5 parameters thoroughly. The description adds no parameter-specific information beyond what's in the schema - it doesn't explain relationships between parameters (e.g., how dashboardId and hashcode work together) or provide usage examples. The baseline of 3 is appropriate when the schema does the heavy lifting.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb ('Get') and resource ('comprehensive failure information') with specific components mentioned (failure summary, detailed test runs, SQL widgets, templateId references). It distinguishes from some siblings like 'analyze_test_failure' by emphasizing retrieval rather than analysis, but doesn't explicitly differentiate from all similar tools like 'get_launch_details' or 'get_test_execution_history'.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives. With many sibling tools dealing with test failures, launches, and bug reviews (e.g., 'analyze_test_failure', 'get_bug_review', 'get_launch_details'), there's no indication of when this specific failure information retrieval tool is preferred over other options.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_bug_reviewC

๐Ÿ” Get detailed bug review with failures, defects, reproduction dates, and optional automatic failure detail fetching (SQL widget, templateId: 9)

ParametersJSON Schema
NameRequiredDescriptionDefault
projectNoProject alias ('web', 'android', 'ios', 'api'), project key, or numeric projectIdweb
periodNoTime period for bug reviewLast 7 Days
limitNoMaximum number of bugs to return (default: 100, max: 500)
include_failure_detailsNoWhen true, automatically fetches detailed failure info for each bug (affected test runs, error details). Enables comprehensive single-call analysis.
failure_detail_levelNoLevel of failure details: none (just bug list), summary (error + count), full (all affected test runs)summary
max_details_limitNoMaximum bugs to fetch detailed failure info for (default: 30, max: 50). Prevents excessive API calls.
templateIdNoOverride templateId if needed (default: 9 for Bug Review)
formatNoOutput format: detailed (full info with markdown links), summary (concise), or json (raw data)detailed

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries full burden for behavioral disclosure. While it mentions 'automatic failure detail fetching' and includes a templateId reference, it doesn't address critical behavioral aspects: whether this is a read-only operation, what permissions are required, rate limits, error handling, or what the output looks like. The SQL widget mention adds some context but doesn't fully compensate for the missing behavioral transparency.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence that front-loads the core purpose. The emoji adds visual distinction but doesn't detract from clarity. The parenthetical note about templateId is somewhat technical but relevant. There's minimal waste, though it could be slightly more structured for optimal readability.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the complexity (8 parameters, no annotations, no output schema), the description is moderately complete. It covers the basic purpose and hints at some functionality but leaves significant gaps: no output description, no behavioral context, and no guidance on usage relative to siblings. The description alone is insufficient for an agent to fully understand how to use this tool effectively.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema description coverage is 100%, so the schema already documents all 8 parameters thoroughly. The description adds minimal value beyond the schema - it mentions 'failures, defects, reproduction dates' which loosely maps to output content rather than input parameters, and references 'automatic failure detail fetching' which corresponds to the 'include_failure_details' parameter. This meets the baseline for high schema coverage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: 'Get detailed bug review with failures, defects, reproduction dates, and optional automatic failure detail fetching'. It specifies the verb ('Get') and resource ('bug review') with specific content details. However, it doesn't explicitly differentiate from sibling tools like 'get_bug_failure_info' or 'get_top_bugs', which prevents a perfect score.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives. It mentions 'optional automatic failure detail fetching' but doesn't explain when this should be enabled or how this tool compares to other bug-related tools in the sibling list. There's no mention of prerequisites, use cases, or exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_enhanced_test_coverage_with_rulesC

๐Ÿ” Enhanced test coverage analysis with configurable rules validation and quality scoring

ParametersJSON Schema
NameRequiredDescriptionDefault
project_keyNoProject key (auto-detected from case_key if not provided)
case_keyYesTest case key (e.g., 'ANDROID-6')
implementation_contextYesActual implementation details (code snippets, file paths, or implementation description)
analysis_scopeNoScope of analysis: steps, assertions, data coverage, or full analysisfull
output_formatNoOutput format: chat response, markdown file, detailed analysis, or all formatsdetailed
include_recommendationsNoInclude improvement recommendations
validate_against_rulesNoValidate coverage against configured rules
show_framework_detectionNoShow detected framework and patterns
include_suite_hierarchyNoInclude featureSuiteId and rootSuiteId in analysis
file_pathNoFile path for adding code comments or saving markdown (optional)

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries full burden for behavioral disclosure. While it mentions 'analysis', 'validation', and 'scoring', it doesn't clarify whether this is a read-only operation, if it modifies data, what permissions are required, or what the output looks like (beyond format options). For a complex 10-parameter tool with no annotation coverage, this leaves significant behavioral gaps unaddressed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence that front-loads the core purpose. The emoji adds visual distinction without being distracting. Every word contributes to understanding the tool's enhanced nature. It could potentially benefit from a second sentence about output characteristics, but as-is it's appropriately concise for the complexity level.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a complex 10-parameter analysis tool with no annotations and no output schema, the description is insufficiently complete. It doesn't explain what 'enhanced' means relative to basic coverage tools, what 'quality scoring' entails, what rules are validated against, or what the analysis output contains. The user must infer these critical details from parameter names alone, which is inadequate for proper tool selection and invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all 10 parameters thoroughly with descriptions, enums, defaults, and requirements. The description adds no specific parameter information beyond what's in the schema - it doesn't explain relationships between parameters (e.g., how 'analysis_scope' affects 'validate_against_rules') or provide usage examples. The baseline 3 is appropriate when the schema does all the parameter documentation work.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool performs 'enhanced test coverage analysis with configurable rules validation and quality scoring', which is a specific verb+resource combination. It distinguishes itself from siblings like 'get_test_coverage_by_test_case_steps_by_key' by emphasizing 'enhanced' analysis with rules validation. However, it doesn't explicitly differentiate from other analysis tools like 'analyze_test_failure' or 'detailed_analyze_launch_failures' beyond the coverage focus.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives. With many sibling tools focused on test analysis (e.g., 'analyze_test_failure', 'get_test_coverage_by_test_case_steps_by_key'), there's no indication of when this 'enhanced' analysis is preferred, what prerequisites exist, or when other tools might be more appropriate. The description assumes the user already knows the context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_launch_detailsB

๐Ÿš€ Get comprehensive launch details including test sessions (uses new reporting API with enhanced authentication)

ParametersJSON Schema
NameRequiredDescriptionDefault
projectKeyNoProject key (e.g., 'android' or 'ANDROID') - alternative to projectId
projectIdNoProject ID (e.g., 7) - alternative to projectKey
launchIdYesLaunch ID (e.g., 118685)
includeLaunchDetailsNoInclude detailed launch information
includeTestSessionsNoInclude test sessions data
formatNoOutput formatjson

TDQS

B3.2/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions 'enhanced authentication', which hints at potential auth requirements, but doesn't specify what those are (e.g., API keys, permissions). It also doesn't describe rate limits, error handling, pagination, or what 'comprehensive' entails beyond test sessions. For a read operation with 6 parameters and no annotations, this leaves significant gaps in understanding how the tool behaves.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence that front-loads the core purpose ('Get comprehensive launch details including test sessions') and adds contextual notes ('uses new reporting API with enhanced authentication'). There's no wasted verbiage, and every part serves a purpose. It loses a point because the emoji (๐Ÿš€) is decorative and doesn't add functional clarity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (6 parameters, no output schema, no annotations), the description is minimally adequate. It covers the basic purpose and hints at authentication needs, but doesn't explain output structure, error cases, or how 'comprehensive' differs from sibling tools. For a tool with moderate complexity and no structured behavioral hints, it should do more to guide effective use.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema description coverage is 100%, meaning all parameters are documented in the schema itself. The description doesn't add any parameter-specific information beyond what's in the schema (e.g., it doesn't explain relationships between projectKey and projectId, or clarify what 'dto' format means). Since the schema does the heavy lifting, the baseline score of 3 is appropriateโ€”the description neither compensates for gaps nor adds extra value.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: 'Get comprehensive launch details including test sessions'. It specifies the verb ('Get'), resource ('launch details'), and scope ('including test sessions'), which distinguishes it from simpler summary tools. However, it doesn't explicitly differentiate from sibling tools like 'get_launch_summary' or 'detailed_analyze_launch_failures', which prevents a perfect score.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides some usage context by mentioning 'uses new reporting API with enhanced authentication', which implies this is a preferred or updated method. However, it doesn't explicitly state when to use this tool versus alternatives like 'get_launch_summary' or 'get_all_launches_for_project', nor does it specify prerequisites or exclusions. The guidance is implied rather than explicit.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_launch_summaryB

๐Ÿ“‹ Get quick launch summary without detailed test sessions (uses new reporting API)

ParametersJSON Schema
NameRequiredDescriptionDefault
projectKeyNoProject key (e.g., 'android' or 'ANDROID') - alternative to projectId
projectIdNoProject ID (e.g., 7) - alternative to projectKey
launchIdYesLaunch ID (e.g., 118685)
formatNoOutput formatjson

TDQS

B3.3/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries full burden for behavioral disclosure. It mentions 'uses new reporting API' which adds some context about implementation, but doesn't describe what 'quick launch summary' contains, whether it's read-only, what permissions are needed, rate limits, or response format. For a tool with no annotation coverage, this leaves significant behavioral gaps.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is perfectly concise - one sentence with an emoji that adds visual context. Every word earns its place: 'Get quick launch summary' states the core purpose, 'without detailed test sessions' provides important differentiation, and 'uses new reporting API' adds implementation context. No wasted words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given no annotations and no output schema, the description provides adequate basic purpose but lacks important context about what the 'quick launch summary' actually contains, response format details, or behavioral constraints. For a tool with rich sibling context and no structured safety/behavior annotations, it should do more to compensate.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all 4 parameters thoroughly. The description adds no additional parameter information beyond what's in the schema. According to guidelines, when schema coverage is high (>80%), the baseline is 3 even with no param info in description.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: 'Get quick launch summary without detailed test sessions' with the specific verb 'Get' and resource 'launch summary'. It distinguishes from siblings like 'get_launch_details' and 'get_launch_test_summary' by emphasizing 'quick' and 'without detailed test sessions'. However, it doesn't explicitly name these alternatives for full differentiation.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies when to use this tool ('quick launch summary without detailed test sessions') versus more detailed alternatives, but doesn't explicitly state when-not-to-use or name specific sibling tools. It mentions 'uses new reporting API' which provides some context, but lacks explicit guidance on prerequisites or comparisons.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_launch_test_summaryA

๐Ÿ“Š Get lightweight launch test summary with statistics (auto-paginated, token-optimized)

ParametersJSON Schema
NameRequiredDescriptionDefault
projectKeyNoProject key (e.g., 'MCP') - alternative to projectId
projectIdNoProject ID (e.g., 7) - alternative to projectKey
launchIdYesLaunch ID (e.g., 119783)
statusFilterNoFilter by status (e.g., ['FAILED', 'SKIPPED'])
minStabilityNoMinimum stability percentage (0-100)
maxStabilityNoMaximum stability percentage (0-100)
sortByNoSort order (stability=most unstable first)stability
limitNoLimit number of tests returned (e.g., 10 for first 10 tests)
summaryOnlyNoReturn only statistics without full test list (most lightweight)
includeLabelsNoInclude labels array (increases token usage)
includeTestCasesNoInclude testCases array (increases token usage)
formatNoOutput formatjson

TDQS

A3.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden and adds valuable behavioral context: it discloses 'auto-paginated' (handling pagination automatically) and 'token-optimized' (efficient for token usage), which aren't obvious from the schema. However, it doesn't mention potential rate limits, authentication needs, or error handling.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence with no wasted words, using emojis and parentheses effectively to convey key points ('lightweight', 'statistics', 'auto-paginated', 'token-optimized'). It's appropriately sized and front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the complexity (12 parameters, no output schema, no annotations), the description is somewhat complete by adding behavioral context, but it lacks details on output format, error cases, or how it differs from siblings like 'get_launch_summary'. It's adequate but has clear gaps for a tool with many parameters.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all 12 parameters thoroughly. The description doesn't add any parameter-specific semantics beyond what's in the schema (e.g., it doesn't explain how 'lightweight' relates to parameters like 'summaryOnly'), meeting the baseline of 3.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb ('Get') and resource ('lightweight launch test summary with statistics'), making the purpose specific and understandable. It distinguishes from some siblings like 'get_launch_details' or 'get_launch_summary' by emphasizing 'lightweight' and 'statistics', though it doesn't explicitly name alternatives.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage through terms like 'lightweight' and 'token-optimized', suggesting it's for quick summaries rather than detailed analysis, but it doesn't provide explicit when-to-use guidance or name specific alternatives from the sibling list (e.g., 'get_launch_details' for more detailed info).

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_platform_results_by_periodC

๐Ÿ“Š Get test results by platform for a given period (SQL widget, templateId: 8)

ParametersJSON Schema
NameRequiredDescriptionDefault
projectNoProject alias ('web', 'android', 'ios', 'api'), project key, or numeric projectIdweb
periodNoTime periodLast 7 Days
platformNoPlatform alias or explicit array for paramsConfig.PLATFORM
browserNoOptional BROWSER filter, e.g., ['chrome'] for web
milestoneNoOptional MILESTONE filter, e.g., ['25.39.0'] for milestone filtering
templateIdNoOverride templateId if needed
dashboardNameNoOverride dashboard title
formatNoOutput format: raw widget response or formatted dataformatted

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries full burden for behavioral disclosure. It mentions 'SQL widget' and 'templateId: 8' which gives some implementation context, but doesn't describe what type of data is returned, whether this is a read-only operation, performance characteristics, or any limitations. For a tool with 8 parameters and no annotations, this is inadequate behavioral transparency.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is extremely concise - a single sentence with an emoji for visual emphasis. Every element earns its place: the emoji signals reporting/analytics, the core function is stated clearly, and the SQL widget context provides implementation specificity without unnecessary elaboration.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with 8 parameters, no annotations, and no output schema, the description is insufficiently complete. It doesn't explain what 'test results' actually means in terms of data structure, doesn't mention return format or typical response content, and provides minimal guidance on parameter usage despite the complex parameter set. The SQL widget reference helps but doesn't compensate for these gaps.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all 8 parameters thoroughly. The description adds minimal value beyond the schema - it mentions 'platform' and 'period' in the main description and references 'templateId' in parentheses, but doesn't provide additional semantic context about how parameters interact or typical usage patterns.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('Get test results') and resource ('by platform for a given period'), with the emoji and SQL widget context adding specificity. However, it doesn't explicitly differentiate from sibling tools like 'get_all_launches_for_project' or 'get_launch_summary' that might also retrieve test data, leaving some ambiguity about its unique scope.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives among the many sibling tools. It mentions 'SQL widget, templateId: 8' which hints at a specific context, but doesn't explain when this is preferred over other test result retrieval tools or what makes it distinct in usage scenarios.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_project_milestonesC

๐ŸŽฏ Get available milestones for a project with pagination and filtering

ParametersJSON Schema
NameRequiredDescriptionDefault
projectNoProject alias ('web', 'android', 'ios', 'api'), project key, or numeric projectIdweb
pageNoPage number for pagination (1-based)
pageSizeNoNumber of milestones per page (max 100)
statusNoFilter by completion status: incomplete (default, excludes overdue), completed, overdue (incomplete but past due date), or allincomplete
formatNoOutput format: raw API response or formatted dataformatted

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries full burden for behavioral disclosure. While 'Get' implies a read operation, the description doesn't address authentication requirements, rate limits, error conditions, or what the response structure looks like. The mention of 'pagination and filtering' is helpful but insufficient for a tool with 5 parameters and no output schema.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence that front-loads the core purpose ('Get available milestones for a project') and adds two key features ('with pagination and filtering'). Every word earns its place with zero redundancy or unnecessary elaboration.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with 5 parameters, no annotations, and no output schema, the description is insufficiently complete. While the schema documents parameters well, the description doesn't address what the tool returns, error handling, authentication needs, or how it differs from similar data retrieval tools in the sibling list. The agent would need to guess about the response format and appropriate usage contexts.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema description coverage is 100%, with each parameter well-documented in the schema itself. The description adds minimal value beyond the schema - it mentions 'pagination and filtering' which aligns with the 'page', 'pageSize', and 'status' parameters, but doesn't provide additional context or usage examples. This meets the baseline expectation when schema coverage is complete.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb ('Get') and resource ('available milestones for a project'), making the purpose immediately understandable. However, it doesn't explicitly differentiate this tool from sibling tools like 'get_all_launches_for_project' or 'get_test_cases_by_filter', which follow similar patterns for different resources.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description mentions 'pagination and filtering' which implies some usage context, but provides no explicit guidance on when to use this tool versus alternatives. With many sibling tools for retrieving project-related data (e.g., 'get_all_launches_for_project', 'get_test_cases_by_filter'), there's no indication of when milestones specifically are needed versus other project data types.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_root_id_by_suite_idB

๐Ÿ” Get root suite ID for a specific suite ID

ParametersJSON Schema
NameRequiredDescriptionDefault
project_keyYesProject key (e.g., 'android' or 'ANDROID')
suite_idYesSuite ID to find root for
formatNoOutput formatjson

TDQS

B3.1/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries full burden. It only states the tool's function without disclosing behavioral traits like error handling, rate limits, authentication needs, or output format details beyond the 'format' parameter. This is inadequate for a tool with no annotation coverage.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence with an emoji for emphasis. It's front-loaded with the core purpose and wastes no words, making it easy for an agent to parse quickly.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given no annotations and no output schema, the description is minimal but covers the basic purpose. However, it lacks details on behavior, error cases, or output structure, which are important for a tool that likely returns hierarchical data. It's adequate but has clear gaps.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema fully documents all parameters. The description adds no additional meaning beyond what's in the schema, such as explaining relationships between 'suite_id' and the returned root ID. Baseline 3 is appropriate when schema does all the work.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('Get root suite ID') and target resource ('for a specific suite ID'), using an emoji for visual emphasis. It specifies the verb and resource but doesn't differentiate from sibling tools like 'get_root_suites' or 'get_suite_hierarchy', which prevents a perfect score.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites, exclusions, or compare to sibling tools like 'get_root_suites' or 'get_suite_hierarchy', leaving the agent without context for tool selection.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_root_suitesB

๐ŸŒณ Get root suites (suites with no parent) from project

ParametersJSON Schema
NameRequiredDescriptionDefault
project_keyYesProject key (e.g., 'android' or 'ANDROID')
formatNoOutput formatjson

TDQS

B3.1/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It implies a read-only operation ('Get') but doesn't specify permissions, rate limits, pagination, or error handling. The description adds minimal context beyond the basic action, leaving gaps in understanding how the tool behaves in practice.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence with no wasted words. It front-loads the core purpose ('Get root suites') and includes a relevant emoji for visual clarity. Every element earns its place, making it easy to scan and understand quickly.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a read-only tool with 2 parameters and 100% schema coverage but no output schema, the description is minimally adequate. It states what the tool does but lacks details on return values, error cases, or integration with sibling tools. Without annotations or output schema, more behavioral context would improve completeness, but it meets the basic threshold.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema fully documents both parameters (project_key and format). The description doesn't add any meaning beyond what the schema provides, such as explaining the significance of 'root suites' in relation to parameters. Baseline 3 is appropriate as the schema handles parameter documentation adequately.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('Get') and resource ('root suites from project'), specifying that root suites are those with no parent. It distinguishes from siblings like 'get_all_subsuites' or 'get_suite_hierarchy' by focusing on root-level suites only. However, it doesn't explicitly contrast with these siblings in the text.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives like 'get_all_subsuites', 'get_suite_hierarchy', or 'list_test_suites'. It lacks context about prerequisites, such as needing a valid project key, and doesn't mention any exclusions or specific scenarios where this tool is preferred.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_suite_hierarchyB

๐ŸŒณ Get hierarchical test suite tree with configurable depth

ParametersJSON Schema
NameRequiredDescriptionDefault
project_keyYesProject key
root_suite_idNoStart from specific root suite
max_depthNoMaximum tree depth
formatNoOutput formatjson
include_clickable_linksNoInclude clickable links to Zebrunner web UI

TDQS

B3.4/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries full burden. It mentions 'configurable depth' which hints at a behavioral trait, but doesn't disclose important aspects like whether this is a read-only operation, potential performance impacts with deep trees, authentication needs, rate limits, or what the hierarchical output looks like. The description is minimal and leaves key behavioral questions unanswered.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is perfectly concise - a single sentence with an emoji that reinforces the tree concept. Every word earns its place: 'Get' (action), 'hierarchical test suite tree' (resource and structure), 'with configurable depth' (key capability). No wasted words or redundant information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with 5 parameters, no annotations, and no output schema, the description is insufficient. It doesn't explain what the hierarchical output looks like, how to interpret the tree structure, performance considerations with depth, or relationships with sibling tools. The 100% schema coverage helps, but the description should provide more context about the tool's behavior and output.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all 5 parameters thoroughly. The description adds no additional parameter semantics beyond what's in the schema - it doesn't explain relationships between parameters (e.g., how root_suite_id interacts with project_key) or provide usage examples. Baseline 3 is appropriate when schema does the heavy lifting.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('Get hierarchical test suite tree') and the resource ('test suite'), with the emoji reinforcing the tree concept. It distinguishes from siblings by specifying 'hierarchical' and 'tree', unlike other tools like 'get_all_subsuites' or 'list_test_suites' that might return flat lists.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for retrieving hierarchical structures with depth control, but doesn't explicitly state when to use this versus alternatives like 'get_all_subsuites' or 'get_root_suites'. It provides some context through 'configurable depth' but lacks explicit when/when-not guidance or named alternatives.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_tcm_suite_by_idC

๐Ÿ” Find TCM suite by ID with comprehensive search

ParametersJSON Schema
NameRequiredDescriptionDefault
project_keyYesProject key (e.g., 'android' or 'ANDROID')
suite_idYesSuite ID to find
only_root_suitesNoSearch only in root suites
formatNoOutput formatjson
include_clickable_linksNoInclude clickable links to Zebrunner web UI

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries full burden. It mentions 'comprehensive search' but doesn't disclose behavioral traits like whether this is a read-only operation, if it requires authentication, rate limits, error handling, or what 'comprehensive' entails (e.g., depth of data returned). This leaves significant gaps for a tool with 5 parameters and no output schema.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence with zero waste. It's front-loaded with the core purpose and uses an emoji for visual emphasis without adding fluff. Every word earns its place, making it highly concise and well-structured.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given 5 parameters, no annotations, and no output schema, the description is incomplete. It doesn't explain the return format, what 'comprehensive search' includes, or behavioral aspects like safety or performance. For a tool with this complexity, more context is needed to guide effective use.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so parameters are well-documented in the schema. The description adds no additional meaning beyond implying ID-based search, which is already clear from the schema. Baseline 3 is appropriate as the schema does the heavy lifting, but the description doesn't compensate with extra context like parameter interactions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a clear purpose: 'Find TCM suite by ID with comprehensive search' specifies the verb ('Find'), resource ('TCM suite'), and method ('by ID'). It distinguishes from siblings like 'get_all_tcm_test_case_suites_by_project' (which lists all suites) by focusing on ID-based retrieval, but doesn't explicitly name alternatives or contrast scope beyond the 'comprehensive search' hint.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No explicit guidance on when to use this tool versus alternatives is provided. The description mentions 'comprehensive search' but doesn't clarify what that entails or differentiate from siblings like 'get_suite_hierarchy' or 'get_root_suites'. Usage context is implied (ID-based lookup) but lacks exclusions or prerequisites.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_tcm_test_suites_by_projectC

๐Ÿ“‹ Get TCM test suites by project with pagination (Java methodology)

ParametersJSON Schema
NameRequiredDescriptionDefault
project_keyYesProject key (e.g., 'android' or 'ANDROID')
max_page_sizeNoMaximum page size for pagination
page_tokenNoPage token for pagination
formatNoOutput formatjson

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries full burden but only mentions pagination and Java methodology. It doesn't disclose whether this is a read-only operation, potential rate limits, authentication requirements, error conditions, or what the output looks like (e.g., structure of returned test suites).

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Extremely concise single sentence with zero waste. Every element (verb, resource, pagination note, methodology) serves a purpose. The emoji adds visual distinction without compromising clarity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with 4 parameters, no annotations, and no output schema, the description is insufficient. It doesn't explain what 'TCM test suites' are, what the Java methodology entails, what the output contains, or how pagination works in practice. The agent would struggle to use this effectively without additional context.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all parameters thoroughly. The description adds no additional parameter semantics beyond what's in the schema, maintaining the baseline score of 3 for adequate coverage through structured data alone.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb 'Get' and resource 'TCM test suites by project', specifying pagination and Java methodology. It distinguishes from some siblings like 'get_all_tcm_test_cases_by_project' by focusing on test suites rather than test cases, though it doesn't explicitly contrast with 'get_root_suites' or 'list_test_suites'.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when to use this tool versus alternatives like 'get_root_suites', 'list_test_suites', or 'get_all_tcm_test_case_suites_by_project'. The description mentions pagination but doesn't explain when this is necessary or preferable over non-paginated options.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_test_case_by_filterC

๐Ÿ” Get test cases using advanced filtering options with exact matching

ParametersJSON Schema
NameRequiredDescriptionDefault
project_keyYesProject key
test_suite_idNoFilter by exact test suite ID
created_afterNoFilter test cases created after this date (ISO format: '2025-01-01T00:00:00Z')
created_beforeNoFilter test cases created before this date (ISO format: '2025-12-31T23:59:59Z')
last_modified_afterNoFilter test cases last modified after this date (ISO format: '2025-01-01T00:00:00Z')
last_modified_beforeNoFilter test cases last modified before this date (ISO format: '2025-12-31T23:59:59Z')
priority_idNoFilter by priority ID (use get_automation_priorities to see available priorities)
automation_state_idNoFilter by automation state ID (use get_automation_states to see available states)
max_page_sizeNoMaximum number of results per page
page_tokenNoToken for pagination (from previous response _meta.nextPageToken)
get_allNoGet all matching test cases across all pages
formatNoOutput formatjson
include_clickable_linksNoInclude clickable links to Zebrunner web UI

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions 'advanced filtering options with exact matching', which hints at query behavior, but fails to cover critical aspects like pagination handling (implied by parameters but not described), rate limits, authentication needs, or whether it's a read-only operation. This leaves significant gaps for an agent to understand how to use it effectively.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence with an emoji for visual emphasis. It's front-loaded with the core action and includes no redundant information, making it highly concise and well-structured for quick comprehension.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the complexity (13 parameters, no annotations, no output schema), the description is insufficient. It doesn't explain the tool's behavior, output format (beyond the 'format' parameter), error handling, or how results are structured (e.g., pagination details). For a tool with many parameters and no structured guidance, more descriptive context is needed to ensure proper usage.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema description coverage is 100%, providing detailed parameter documentation. The description adds minimal value beyond the schema by emphasizing 'advanced filtering options with exact matching', which loosely aligns with parameters like test_suite_id and date filters. However, it doesn't explain parameter interactions or provide additional context, so it meets the baseline for high schema coverage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb ('Get') and resource ('test cases') with the method ('using advanced filtering options with exact matching'), making the purpose evident. However, it doesn't explicitly distinguish this tool from similar siblings like 'get_test_cases_advanced' or 'get_test_cases_by_suite_smart', which likely offer overlapping functionality.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives, such as 'get_test_cases_advanced' or 'get_test_case_by_key'. It mentions 'advanced filtering options' but doesn't specify what makes it advanced or when it's preferred over simpler tools, leaving the agent without clear usage context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_test_case_by_keyB

๐Ÿ” Get detailed test case by key (โœ… Verified Working)

ParametersJSON Schema
NameRequiredDescriptionDefault
project_keyNoProject key (e.g., 'android' or 'ANDROID') - auto-detected from case_key if not provided
case_keyYesTest case key (e.g., 'ANDROID-29', 'IOS-2')
formatNoOutput formatjson
include_debugNoInclude debug information in markdown
include_suite_hierarchyNoInclude featureSuiteId and rootSuiteId with suite hierarchy path
include_clickable_linksNoInclude clickable links to Zebrunner web UI

TDQS

B3.1/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries full burden for behavioral disclosure. It mentions the tool is 'Verified Working', which hints at reliability, but fails to describe critical behaviors: whether it's a read-only operation, what happens if the key doesn't exist, authentication requirements, rate limits, or response format details. The description adds minimal value beyond the basic purpose.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Extremely concise and front-loaded: the core purpose is stated in the first few words. The verification note is brief and doesn't distract. Every element earns its place with no wasted words or redundant information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a read operation with 6 parameters and no output schema, the description is minimally adequate. It states what the tool does but lacks details about return values, error handling, or behavioral constraints. With no annotations and no output schema, more context would be helpful, but the high schema coverage partially compensates.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema fully documents all 6 parameters. The description adds no additional parameter semantics beyond what's in the schema (e.g., no examples of case_key formats beyond what's implied, no explanation of when to use different formats). Baseline 3 is appropriate since the schema does the heavy lifting.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('Get detailed test case') and resource ('by key'), making the purpose immediately understandable. It distinguishes from siblings like 'get_test_case_by_title' by specifying the key-based lookup. The emoji and verification note add flavor but don't detract from clarity.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No explicit guidance on when to use this tool versus alternatives like 'get_test_case_by_title', 'get_test_case_by_filter', or 'get_test_cases_advanced'. The description implies it's for retrieving a single test case by its identifier, but lacks context about prerequisites, error conditions, or comparison with sibling tools.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_test_case_by_titleC

๐Ÿ” Get test cases by title using partial match search with pagination support

ParametersJSON Schema
NameRequiredDescriptionDefault
project_keyYesProject key
titleYesTitle to search for (partial match)
max_page_sizeNoMaximum number of results per page
page_tokenNoToken for pagination (from previous response _meta.nextPageToken)
get_allNoGet all matching test cases across all pages
formatNoOutput formatjson
include_clickable_linksNoInclude clickable links to Zebrunner web UI

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden of behavioral disclosure. While it mentions 'partial match search' and 'pagination support', it doesn't describe important behavioral aspects like authentication requirements, rate limits, error conditions, what happens when no matches are found, or the structure of returned data. For a tool with 7 parameters and no annotation coverage, this is insufficient.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is extremely concise - a single sentence that efficiently communicates the core functionality. Every word earns its place: 'Get test cases by title' establishes the purpose, 'partial match search' clarifies the search behavior, and 'pagination support' indicates scalability features. No wasted words or redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with 7 parameters, no annotations, and no output schema, the description is inadequate. It doesn't explain what the tool returns (test case objects, their structure, or error formats), doesn't mention authentication or rate limiting, and provides no context about the Zebrunner system it operates on. The description should do more to compensate for the lack of structured metadata.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema description coverage is 100%, so all parameters are well-documented in the schema itself. The description adds minimal value beyond the schema - it mentions 'partial match search' (implied by the 'title' parameter description) and 'pagination support' (implied by 'page_token' and 'max_page_size'). This meets the baseline of 3 when schema coverage is high.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: 'Get test cases by title using partial match search with pagination support.' It specifies the verb ('Get'), resource ('test cases'), and key functionality ('partial match search', 'pagination support'). However, it doesn't explicitly differentiate from sibling tools like 'get_test_case_by_key' or 'get_test_case_by_filter', which would be needed for a score of 5.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives. With many sibling tools for retrieving test cases (e.g., 'get_test_case_by_key', 'get_test_case_by_filter', 'get_test_cases_advanced'), there's no indication of when this partial-title-search approach is preferred or what distinguishes it from other search methods.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_test_cases_advancedA

๐Ÿ“Š Advanced test case retrieval with filtering and pagination (โœจ Enhanced with automation state and date filtering) โš ๏ธ IMPORTANT: Use 'suite_id' for direct parent suites, 'root_suite_id' for root suites that contain sub-suites. ๐Ÿ’ก TIP: Use 'get_test_cases_by_suite_smart' for automatic suite type detection!

ParametersJSON Schema
NameRequiredDescriptionDefault
project_keyYesProject key
suite_idNoFilter by direct parent suite ID (for child suites)
root_suite_idNoFilter by root suite ID (includes all sub-suites)
include_stepsNoInclude detailed test steps
automation_statesNoFilter by automation state(s). Can be: single name ('Not Automated'), single ID (10), array of names (['Not Automated', 'To Be Automated']), array of IDs ([10, 12]), or mixed array (['Not Automated', 12])
created_afterNoFilter test cases created after this date (ISO format: '2025-01-01' or '2025-01-01T10:00:00Z')
created_beforeNoFilter test cases created before this date (ISO format: '2025-12-31' or '2025-12-31T23:59:59Z')
modified_afterNoFilter test cases modified after this date (ISO format: '2025-01-01' or '2025-01-01T10:00:00Z')
modified_beforeNoFilter test cases modified before this date (ISO format: '2025-12-31' or '2025-12-31T23:59:59Z')
formatNoOutput formatjson
pageNoPage number (0-based)
sizeNoPage size (configurable via MAX_PAGE_SIZE env var)
include_clickable_linksNoInclude clickable links to Zebrunner web UI

TDQS

A3.8/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions 'advanced retrieval with filtering and pagination' which implies read-only behavior, but doesn't explicitly state whether this is a read operation, what permissions might be required, rate limits, or what the return format looks like. The description adds some context about suite filtering but lacks comprehensive behavioral details.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is efficiently structured with emoji icons and clear sections (purpose, important notes, tip). While slightly verbose with the emoji decorations, every sentence earns its place by providing distinct value. The information is front-loaded with the core purpose stated first.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a complex tool with 13 parameters and no output schema, the description provides adequate context about filtering capabilities and sibling alternatives. However, it doesn't describe the return format, pagination behavior, or error conditions. With no annotations and no output schema, more comprehensive behavioral context would be helpful for this advanced retrieval tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all 13 parameters thoroughly. The description adds minimal parameter semantics beyond what's in the schema - it only clarifies the suite_id vs root_suite_id distinction. This meets the baseline expectation when schema coverage is high.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose as 'Advanced test case retrieval with filtering and pagination' with specific enhancements mentioned (automation state and date filtering). It distinguishes itself from basic retrieval tools but doesn't explicitly differentiate from all sibling tools beyond the one mentioned alternative.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides explicit guidance on when to use this tool vs alternatives: it specifies to use 'suite_id' for direct parent suites and 'root_suite_id' for root suites, and explicitly recommends using 'get_test_cases_by_suite_smart' for automatic suite type detection. This gives clear when/when-not/alternative instructions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_test_cases_by_automation_stateA

๐Ÿค– Get test cases filtered by automation state (๐Ÿ’ก Use get_automation_states to see available states)

ParametersJSON Schema
NameRequiredDescriptionDefault
project_keyYesProject key
automation_statesYesAutomation state(s) to filter by. Examples: 'Not Automated', ['Not Automated', 'To Be Automated'], [10, 12], or 'Automated'
suite_idNoOptional: Filter by specific suite ID
created_afterNoOptional: Filter test cases created after this date (ISO format: '2025-01-01')
formatNoOutput formatjson
pageNoPage number (0-based)
sizeNoPage size
include_clickable_linksNoInclude clickable links to Zebrunner web UI

TDQS

A3.5/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It only mentions filtering by automation state and references another tool for state options. It lacks critical behavioral details such as pagination behavior (implied by 'page' and 'size' params but not explained), rate limits, authentication needs, or what happens with invalid states. For a read operation with 8 parameters, this is insufficient.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence with no wasted words. It front-loads the core purpose and includes a helpful tip in parentheses. The emojis ('๐Ÿค–', '๐Ÿ’ก') add visual cues without compromising brevity. Every element serves a clear purpose.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a read operation with 8 parameters and no output schema, the description is minimally adequate. It covers the primary filtering logic and references a related tool, but lacks details on pagination, output structure, error handling, or performance considerations. With no annotations to fill gaps, it leaves the agent with incomplete operational context.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema fully documents all 8 parameters. The description adds no parameter-specific information beyond implying filtering by automation state, which is already covered in the schema. It references 'get_automation_states' for state options, but this doesn't enhance parameter understanding beyond the schema's examples. Baseline 3 is appropriate when schema does the heavy lifting.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb ('Get') and resource ('test cases') with the specific filtering criterion ('by automation state'). It distinguishes itself from siblings like 'get_test_cases_advanced' or 'get_test_case_by_filter' by specifying the automation state filter, though it doesn't explicitly compare to them. The emojis add visual emphasis but don't detract from clarity.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides explicit guidance on when to use this tool by mentioning 'get_automation_states' to see available states, which helps the agent understand prerequisites. However, it doesn't specify when NOT to use it or compare it to alternative filtering tools like 'get_test_cases_advanced', leaving some ambiguity in tool selection.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_test_cases_by_suite_smartA

๐Ÿง  Smart test case retrieval by suite ID - automatically detects if suite is root suite and uses appropriate filtering with enhanced pagination

ParametersJSON Schema
NameRequiredDescriptionDefault
project_keyYesProject key (e.g., 'MCP')
suite_idYesSuite ID to get test cases from
include_stepsNoInclude detailed test steps for first few cases
formatNoOutput formatjson
get_allNoGet all test cases (true) or paginated results (false)
include_sub_suitesNoInclude test cases from sub-suites (if any)
pageNoPage number (0-based, only used if get_all=false)
sizeNoPage size (only used if get_all=false)

TDQS

A3.5/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It adds some context: 'smart' behavior with automatic root suite detection and 'enhanced pagination' hints at adaptive filtering and pagination features. However, it lacks details on permissions, rate limits, error handling, or what 'enhanced pagination' entails (e.g., performance implications). The description doesn't contradict annotations, but it's insufficient for a mutation-like tool (implied retrieval).

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence that front-loads key information: smart retrieval, suite ID focus, automatic detection, and enhanced pagination. Every element earns its place with no redundant or vague phrasing, making it highly concise and well-structured.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (8 parameters, smart logic) and lack of annotations or output schema, the description is moderately complete. It covers the core purpose and hints at behavioral traits but lacks details on return values, error cases, or how the 'smart' detection works in practice. For a retrieval tool with many parameters, more context would be helpful, but it's minimally adequate.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all 8 parameters thoroughly. The description adds no parameter-specific semantics beyond implying that 'suite_id' triggers smart detection logic. It doesn't explain interactions between parameters (e.g., how 'get_all' affects pagination) or provide usage examples. Baseline 3 is appropriate as the schema does the heavy lifting.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: 'Smart test case retrieval by suite ID' specifies the verb (retrieval) and resource (test cases). It distinguishes itself by mentioning 'smart' behavior with automatic detection of root suites and enhanced pagination. However, it doesn't explicitly differentiate from sibling tools like 'get_test_cases_advanced' or 'get_all_tcm_test_cases_by_project' beyond the 'smart' aspect.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage context through 'automatically detects if suite is root suite and uses appropriate filtering,' suggesting this tool handles root vs. non-root suite scenarios intelligently. However, it doesn't explicitly state when to use this tool versus alternatives like 'get_test_cases_advanced' or 'get_all_tcm_test_cases_with_root_suite_id,' nor does it provide exclusion criteria or prerequisites.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_test_coverage_by_test_case_steps_by_keyC

๐Ÿ” Analyze test case coverage against actual implementation with recommendations

ParametersJSON Schema
NameRequiredDescriptionDefault
project_keyNoProject key (auto-detected from case_key if not provided)
case_keyYesTest case key (e.g., 'ANDROID-6')
implementation_contextYesActual implementation details (code snippets, file paths, or implementation description)
analysis_scopeNoScope of analysis: steps, assertions, data coverage, or full analysisfull
output_formatNoOutput format: chat response, markdown file, code comments, or all formatschat
include_recommendationsNoInclude improvement recommendations
include_suite_hierarchyNoInclude featureSuiteId and rootSuiteId in analysis
file_pathNoFile path for adding code comments or saving markdown (optional)
include_clickable_linksNoInclude clickable links to Zebrunner web UI

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries full burden for behavioral disclosure. It mentions 'recommendations' but lacks critical details: whether this is a read-only analysis or modifies data, what the output looks like (format/content), error handling, or performance characteristics. For a 9-parameter analysis tool, this is insufficient.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence that front-loads the core purpose. It uses an emoji for visual emphasis but maintains focus. No wasted words, though it could potentially benefit from slightly more structure for a complex tool.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a complex analysis tool with 9 parameters and no output schema, the description is inadequate. It doesn't explain what the analysis produces, how recommendations are structured, or the tool's behavioral characteristics. With no annotations and rich parameter schema, the description should provide more context about outputs and usage patterns.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema fully documents all 9 parameters. The description doesn't add any parameter-specific context beyond what's in the schema (e.g., explaining relationships between parameters or usage patterns). Baseline 3 is appropriate when the schema does all the work.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: analyzing test case coverage against implementation with recommendations. It uses specific verbs ('analyze') and identifies the resource ('test case coverage'), but doesn't explicitly differentiate from sibling tools like 'get_enhanced_test_coverage_with_rules' or 'get_test_case_by_key', which prevents a perfect score.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance is provided on when to use this tool versus alternatives. The description doesn't mention prerequisites, appropriate contexts, or exclusions, leaving the agent to infer usage from the tool name and parameters alone.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_test_execution_historyB

๐Ÿ“Š Get execution history for a test across multiple launches - shows pass/fail history, last passed execution, and pass rate

ParametersJSON Schema
NameRequiredDescriptionDefault
testIdYesTest ID
testRunIdYesTest Run ID / Launch ID containing the test
projectKeyNoProject key (e.g., 'MCP') - alternative to projectId
projectIdNoProject ID - alternative to projectKey
limitNoNumber of history items to return (default: 10, max: 50)
formatNoOutput format: dto (structured), json, or string (markdown table)string

TDQS

B3.2/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries full burden but only states what data is returned (pass/fail history, last passed execution, pass rate). It lacks critical behavioral details: whether this is a read-only operation, authentication requirements, rate limits, pagination behavior, or error conditions. The description is insufficient for a tool with 6 parameters and no output schema.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence that front-loads the core purpose and key return data. Every element earns its place with no wasted words, though it could benefit from additional behavioral context.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with 6 parameters, no annotations, and no output schema, the description is incomplete. It doesn't address authentication, error handling, rate limits, or the structure of returned data. While it mentions what data is included, it doesn't provide enough context for reliable agent usage given the complexity.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, providing complete parameter documentation. The description adds no specific parameter semantics beyond implying historical data retrieval across launches. It doesn't explain relationships between parameters (e.g., projectKey vs projectId alternatives) or clarify the 'format' parameter's practical implications.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb ('Get') and resource ('execution history for a test across multiple launches'), specifying the scope of data retrieval. It distinguishes from siblings by focusing on historical pass/fail data rather than current status or analysis, though it doesn't explicitly name alternatives.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage when historical test execution data is needed, but provides no explicit guidance on when to choose this tool over similar siblings like 'get_launch_test_summary' or 'get_test_run_result_statuses'. No exclusions or prerequisites are mentioned.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_test_run_by_idC

๐Ÿ” Get detailed Test Run information by ID from Public API

ParametersJSON Schema
NameRequiredDescriptionDefault
idYesTest Run ID
projectNoProject alias ('web', 'android', 'ios', 'api') or project keyweb
formatNoOutput format: raw API response or formatted dataformatted

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description must fully disclose behavioral traits. It states the tool retrieves 'detailed Test Run information' from a 'Public API', implying a read-only operation, but lacks critical details such as authentication requirements, rate limits, error handling, or the nature of the returned data (e.g., structured vs. raw). This leaves significant gaps in understanding the tool's behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise and front-loaded with the core purpose in a single sentence, using an emoji for visual emphasis. There's no wasted text, but it could be slightly more structured by explicitly separating key details (e.g., 'Retrieves detailed Test Run data by ID from the Public API. Supports project aliases and output formatting.'), keeping it efficient but not perfectly optimized.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (3 parameters, no annotations, no output schema), the description is incomplete. It doesn't explain what 'detailed information' includes, how the output varies by 'format', or potential side effects like API rate limits. For a tool interacting with an external API and returning unspecified data, more context is needed to ensure reliable use.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The description adds minimal parameter semantics beyond the input schema, which has 100% coverage. It implies the 'id' parameter is for a Test Run, but doesn't clarify the meaning of 'project' or 'format' choices. Since the schema already documents all parameters well, the baseline score of 3 is appropriate, as the description doesn't significantly enhance understanding.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('Get detailed Test Run information') and resource ('by ID from Public API'), making the purpose specific and understandable. However, it doesn't explicitly differentiate this tool from sibling tools like 'get_launch_details' or 'get_test_case_by_key', which also retrieve specific entities by identifier, so it doesn't reach the highest score.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites (e.g., needing a valid Test Run ID), exclusions, or comparisons to sibling tools like 'list_test_runs' for browsing or other 'get_*' tools for different entity types, leaving usage context unclear.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_test_run_configuration_groupsC

Get list of Configuration Groups and their Options for a project. These are used to configure Test Runs.

ParametersJSON Schema
NameRequiredDescriptionDefault
projectYesProject alias ('web', 'android', 'ios', 'api') or project key
formatNoOutput formatformatted

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the tool retrieves a list but does not describe key behavioral traits such as pagination, rate limits, authentication requirements, error handling, or the structure of the returned data. The mention of 'format' parameter hints at output variations, but without details, this is insufficient for a mutation-free read operation.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise and front-loaded, consisting of two sentences that directly state the tool's purpose and context. There is no unnecessary verbiage, and each sentence contributes meaning. However, it could be slightly more structured by explicitly separating purpose from usage context.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (a read operation with 2 parameters) and the absence of annotations and output schema, the description is minimally adequate. It covers the basic purpose and hints at usage context but lacks details on behavior, output format, or error handling. The high schema coverage helps, but the description does not fully compensate for the missing behavioral and output information.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has 100% description coverage, with clear documentation for both parameters ('project' and 'format'). The description adds no additional parameter semantics beyond what the schema provides, such as explaining the significance of 'project' values or the differences between 'raw' and 'formatted' outputs. Since schema coverage is high, the baseline score of 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: 'Get list of Configuration Groups and their Options for a project.' It specifies the verb ('Get'), resource ('Configuration Groups and their Options'), and scope ('for a project'), which is specific and actionable. However, it does not explicitly differentiate from sibling tools, as none appear to be direct alternatives for retrieving configuration groups.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides minimal usage guidance: it mentions that configuration groups 'are used to configure Test Runs,' implying a context for test configuration. However, it lacks explicit instructions on when to use this tool versus alternatives, prerequisites, or exclusions. No sibling tools are named as alternatives, and there is no clear differentiation from other tools in the list.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_test_run_result_statusesB

Get list of Result Statuses configured for a project. These statuses are used when assigning results to Test Cases.

ParametersJSON Schema
NameRequiredDescriptionDefault
projectYesProject alias ('web', 'android', 'ios', 'api') or project key
formatNoOutput formatformatted

TDQS

B3.1/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden of behavioral disclosure. It describes a read operation ('Get list'), which implies it's non-destructive, but doesn't mention any behavioral traits such as authentication requirements, rate limits, error handling, or response format details. For a tool with no annotations, this is a significant gap in transparency.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise and well-structured, consisting of two sentences that directly state the tool's purpose and usage context without unnecessary details. It's front-loaded with the core action and resource, making it easy to understand quickly. Every sentence earns its place by adding value.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's moderate complexity (2 parameters, no output schema, no annotations), the description is minimally adequate. It covers the basic purpose and context but lacks details on behavioral aspects, output expectations, and usage guidelines. Without annotations or an output schema, the description should do more to compensate, but it falls short of being fully complete for effective tool selection and invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has 100% description coverage, with clear documentation for both parameters ('project' and 'format'). The description doesn't add any parameter-specific semantics beyond what the schema provides, such as explaining the difference between 'raw' and 'formatted' outputs or detailing project key formats. Given the high schema coverage, a baseline score of 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: 'Get list of Result Statuses configured for a project.' It specifies the resource (Result Statuses) and the action (Get list), and mentions their usage context ('used when assigning results to Test Cases'). However, it doesn't explicitly differentiate from sibling tools that might also retrieve project-related data, such as 'get_project_milestones' or 'get_automation_states', which slightly reduces clarity.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives. It mentions that the statuses are 'used when assigning results to Test Cases,' which implies a context but doesn't specify prerequisites, exclusions, or direct comparisons to sibling tools like 'get_automation_states' or 'get_test_run_by_id.' This lack of explicit usage instructions limits its effectiveness for an AI agent.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_top_bugsC

๐Ÿž Top N most frequent defects with optional issue links (SQL widget, templateId: 4)

ParametersJSON Schema
NameRequiredDescriptionDefault
projectNoProject alias ('web', 'android', 'ios', 'api'), project key, or numeric projectIdweb
periodNoTime periodLast 7 Days
limitNoHow many bugs to return
templateIdNoOverride templateId if needed
issueUrlPatternNoe.g., 'https://yourcompany.atlassian.net/browse/{key}'
platformNoOptional platform filter; defaults to [] for this widget
milestoneNoOptional MILESTONE filter, e.g., ['25.39.0'] for milestone filtering
formatNoOutput format: raw widget response or formatted dataformatted

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries full burden for behavioral disclosure. It mentions 'SQL widget' and 'templateId: 4' which gives some implementation context, but doesn't describe what the tool returns (format, structure), whether it's read-only or has side effects, authentication requirements, rate limits, or error behavior. The description is insufficient for a tool with 8 parameters.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence that gets straight to the point. It uses emoji and parenthetical notes effectively without unnecessary verbosity. However, the 'SQL widget, templateId: 4' note feels somewhat cryptic and could be better integrated.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with 8 parameters, no annotations, and no output schema, the description is inadequate. It doesn't explain what the tool returns (critical since there's no output schema), doesn't provide usage context beyond a cryptic SQL widget reference, and offers minimal behavioral transparency. The description should do much more to compensate for the lack of structured metadata.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all 8 parameters thoroughly. The description adds minimal value beyond the schema - it mentions 'optional issue links' which relates to the 'issueUrlPattern' parameter, and 'templateId: 4' which matches the default value in schema. This meets the baseline for high schema coverage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: retrieving 'Top N most frequent defects' with 'optional issue links'. It specifies the resource (defects/bugs) and action (get top N), but doesn't explicitly differentiate from sibling tools like 'get_bug_failure_info' or 'get_bug_review' that might handle bugs differently.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives. It mentions 'SQL widget, templateId: 4' which hints at a specific implementation context, but doesn't explain when this tool is appropriate compared to other bug-related tools in the sibling list or what prerequisites might exist.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

improve_test_caseC

๐Ÿ”ง Analyze and improve a test case with detailed suggestions and optional automatic fixes

ParametersJSON Schema
NameRequiredDescriptionDefault
projectKeyYesProject key (e.g., 'android' or 'ANDROID')
caseKeyYesTest case key (e.g., 'ANDROID-29')
rulesFilePathNoPath to custom rules markdown file
checkpointsFilePathNoPath to custom checkpoints markdown file
formatNoOutput formatmarkdown
applyHighConfidenceChangesNoAutomatically apply high-confidence improvements
include_clickable_linksNoInclude clickable links to Zebrunner web UI

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries full burden but only vaguely mentions 'optional automatic fixes' without detailing what changes are made, permissions required, or side effects. It doesn't specify if improvements are saved automatically, require review, or affect other test cases, leaving key behavioral traits unclear.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence that front-loads the core purpose. It uses an emoji for visual emphasis but avoids unnecessary elaboration, though it could be slightly more structured by separating analysis from fixes.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with 7 parameters, no annotations, and no output schema, the description is minimally adequate. It covers the basic purpose but lacks details on behavior, output format implications, or integration with sibling tools, leaving gaps in understanding how to effectively use it.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema fully documents all 7 parameters. The description adds no additional meaning beyond what's in the schema, such as explaining how 'rulesFilePath' or 'checkpointsFilePath' influence analysis. Baseline 3 is appropriate since the schema does the heavy lifting.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('analyze and improve') and resource ('test case'), with additional detail about providing 'detailed suggestions and optional automatic fixes'. It distinguishes from siblings like 'validate_test_case' or 'get_test_case_by_key' by focusing on enhancement rather than retrieval or validation, though it doesn't explicitly name alternatives.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance is provided on when to use this tool versus alternatives like 'validate_test_case' or 'get_enhanced_test_coverage_with_rules'. The description implies usage for test case improvement but lacks context on prerequisites, constraints, or specific scenarios where it's most appropriate.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_test_runsB

๐Ÿƒ List Test Runs from Public API with advanced filtering

ParametersJSON Schema
NameRequiredDescriptionDefault
projectNoProject alias ('web', 'android', 'ios', 'api') or project keyweb
pageTokenNoToken for pagination (from previous response)
maxPageSizeNoNumber of test runs per page (max 100)
nameFilterNoFilter by test run name (partial match)
milestoneFilterNoFilter by milestone ID (use get_project_milestones to find ID) or milestone name (will be converted to ID)
buildNumberFilterNoFilter by build number (searches in configurations, title, and description)
closedFilterNoFilter by closed status (true=closed, false=open)
sortByNoSort order: -createdAt (newest first), createdAt (oldest first), -title (Z-A), title (A-Z)-createdAt
formatNoOutput format: raw API response or formatted dataformatted

TDQS

B3/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries full burden but only mentions 'advanced filtering' without detailing behavioral aspects. It doesn't disclose pagination behavior (implied by pageToken), rate limits, authentication requirements, response format expectations, or whether this is a read-only operation. The description adds minimal value beyond what's obvious from the name.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single sentence with an emoji, which is reasonably concise but front-loads style over substance. While it efficiently conveys the core purpose, it could be more structured by separating key capabilities. The emoji adds character but doesn't enhance functional understanding.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with 9 parameters, no annotations, and no output schema, the description is insufficient. It doesn't explain what constitutes a 'test run', what data is returned, how pagination works, or any error conditions. The lack of behavioral context and output information leaves significant gaps for an agent to use this tool effectively.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 100% schema description coverage, the input schema thoroughly documents all 9 parameters. The description adds no specific parameter information beyond 'advanced filtering', which is already evident from the parameter names. The baseline score of 3 reflects adequate coverage through schema alone, with no additional value from the description.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('List') and resource ('Test Runs'), and specifies the source ('from Public API') and capability ('with advanced filtering'). It distinguishes from siblings like 'get_test_run_by_id' by indicating it returns multiple items with filtering, but doesn't explicitly contrast with other list-like tools like 'get_all_launches_for_project'.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage when needing filtered test runs from the public API, but doesn't explicitly state when to choose this tool over alternatives like 'get_all_launches_for_project' or 'list_test_run_test_cases'. The parameter descriptions provide some context (e.g., referencing 'get_project_milestones' for milestone IDs), but no explicit guidance on tool selection is given.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_test_run_test_casesC

๐Ÿ“ List all Test Cases in a Test Run from Public API

ParametersJSON Schema
NameRequiredDescriptionDefault
testRunIdYesTest Run ID
projectNoProject alias ('web', 'android', 'ios', 'api') or project keyweb
formatNoOutput format: raw API response or formatted dataformatted

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries full burden. It mentions 'Public API' which hints at external access, but doesn't disclose critical behavioral traits like authentication requirements, rate limits, pagination behavior, error handling, or whether this is a read-only operation. For a tool with no annotation coverage, this leaves significant gaps.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence that gets straight to the point. The emoji adds visual distinction but doesn't detract from clarity. It's appropriately sized for a straightforward list operation.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a read operation with good schema coverage but no annotations or output schema, the description is adequate but incomplete. It specifies the scope ('all Test Cases in a Test Run') but lacks context about authentication, pagination, error scenarios, or relationship to sibling tools. The absence of output schema means the description should ideally hint at return format.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so parameters are well-documented in the schema itself. The description adds no additional parameter semantics beyond what's already in the schema descriptions. This meets the baseline expectation when schema does the heavy lifting.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('List all Test Cases') and resource ('in a Test Run from Public API'), making the purpose immediately understandable. It doesn't explicitly differentiate from sibling tools like 'get_test_case_by_key' or 'get_test_cases_by_suite_smart', but the scope is well-defined.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives. With many sibling tools for retrieving test cases (e.g., 'get_test_case_by_key', 'get_test_cases_by_suite_smart'), there's no indication of when this list-all approach is preferred or what prerequisites might exist.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_test_suitesC

๐Ÿ“‹ List test suites for a project (โœ… Verified Working)

ParametersJSON Schema
NameRequiredDescriptionDefault
project_keyYesProject key (e.g., 'android' or 'ANDROID')
project_idNoProject ID (alternative to project_key)
formatNoOutput formatjson
include_hierarchyNoInclude hierarchy information
pageNoPage number (0-based)
sizeNoPage size (configurable via MAX_PAGE_SIZE env var)
page_tokenNoPage token for pagination
include_clickable_linksNoInclude clickable links to Zebrunner web UI

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It only mentions that the tool is 'Verified Working' but fails to describe pagination behavior (implied by page/size parameters), authentication requirements, rate limits, or what happens when no test suites exist. For a listing tool with 8 parameters, this leaves significant gaps in understanding how it behaves.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is appropriately brief with a single sentence that states the core purpose. The emoji and verification status add minimal clutter. However, the verification note ('โœ… Verified Working') doesn't provide actionable information for tool selection and could be considered slightly extraneous.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a listing tool with 8 parameters, pagination capabilities, and no output schema, the description is insufficient. It doesn't explain what format the listing returns, how pagination works with page_token, or what hierarchy information includes. With no annotations and no output schema, the description should provide more context about the tool's behavior and results.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The description mentions 'project' which aligns with the required 'project_key' parameter, but adds no additional semantic context beyond what the schema already provides. With 100% schema description coverage, the baseline is 3 - the description doesn't compensate for any gaps because there are none in the schema, but it also doesn't add meaningful parameter insights.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb ('List') and resource ('test suites for a project'), making the purpose immediately understandable. It distinguishes itself from siblings like 'get_all_tcm_test_suites_by_project' by being a general listing tool rather than TCM-specific. However, it doesn't explicitly differentiate from 'get_root_suites' or 'get_all_subsuites', which prevents a perfect score.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives. With many sibling tools that also retrieve test suites or related data (e.g., 'get_all_tcm_test_suites_by_project', 'get_root_suites', 'get_suite_hierarchy'), there's no indication of when this listing tool is preferred over those more specific options.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

test_reporting_connectionA

๐Ÿ”Œ Test connection to Zebrunner Reporting API with new authentication

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

TDQS

A4.1/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries full behavioral disclosure burden. It indicates this is a test/verification operation (implied read-only) and mentions authentication, but doesn't specify what 'test connection' entails (e.g., whether it validates credentials, endpoint availability, or both), what happens on success/failure, or any rate limits. It provides basic context but lacks operational details.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence that immediately conveys the tool's purpose with no wasted words. It's front-loaded with the core action and includes only essential contextual information about authentication.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a zero-parameter tool with no annotations and no output schema, the description provides adequate basic purpose and context. However, it doesn't explain what constitutes a successful test, what format/values the response might contain, or potential error conditions, leaving some operational ambiguity.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool has zero parameters with 100% schema description coverage, so the baseline is 4. The description appropriately doesn't discuss parameters since none exist, focusing instead on the tool's purpose and authentication context.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the specific action ('Test connection') and target resource ('Zebrunner Reporting API'), with the additional context of 'with new authentication' that distinguishes it from generic connectivity checks. It uses a precise verb and identifies the exact system being tested.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage context ('with new authentication') suggesting this should be used when authentication credentials have changed or need verification. However, it doesn't explicitly state when NOT to use it or name alternative tools for similar purposes among the many sibling tools.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

validate_test_caseC

๐Ÿ” Validate a test case against quality standards and best practices (Dynamic Rules Support + Improvement)

ParametersJSON Schema
NameRequiredDescriptionDefault
projectKeyYesProject key (e.g., 'android' or 'ANDROID')
caseKeyYesTest case key (e.g., 'ANDROID-29')
rulesFilePathNoPath to custom rules markdown file
checkpointsFilePathNoPath to custom checkpoints markdown file
formatNoOutput formatmarkdown
improveIfPossibleNoAttempt to automatically improve the test case
include_clickable_linksNoInclude clickable links to Zebrunner web UI

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries full burden but offers minimal behavioral insight. It mentions 'Dynamic Rules Support + Improvement' which hints at configurable rules and potential auto-improvement, but doesn't disclose critical details like whether this is a read-only analysis or makes changes, what permissions are required, error handling, or rate limits. The 'improveIfPossible' parameter suggests mutation capability, but this isn't explicitly stated in the description.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence that conveys the core purpose. The emoji adds visual distinction without being distracting. However, the parenthetical '(Dynamic Rules Support + Improvement)' could be integrated more smoothly, and the description lacks any structural separation of key concepts.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 7-parameter tool with no annotations and no output schema, the description is inadequate. It doesn't explain what the validation output looks like, what 'quality standards and best practices' entail, how 'improvement' manifests, or the consequences of validation. The agent must rely entirely on parameter names and schema descriptions to understand this tool's behavior.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all 7 parameters thoroughly. The description adds no parameter-specific information beyond what's in the schema. It mentions 'Dynamic Rules Support' which loosely relates to 'rulesFilePath' and 'checkpointsFilePath', but doesn't explain their purpose or format. Baseline 3 is appropriate when schema does the heavy lifting.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('validate') and target ('test case') with additional context about quality standards and best practices. It distinguishes from siblings by mentioning 'Dynamic Rules Support + Improvement', which suggests a specific validation approach not present in tools like 'improve_test_case' or 'get_test_case_by_key'. However, it doesn't explicitly differentiate from all sibling tools that might involve test case analysis.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives. It doesn't mention when validation is needed, what triggers it, or how it differs from similar tools like 'improve_test_case' or 'get_enhanced_test_coverage_with_rules'. The agent must infer usage from the tool name alone.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

TDQS

B3/5.0
Disambiguation2/5

Multiple tools have overlapping purposes, causing significant ambiguity. For example, 'get_all_tcm_test_cases_by_project', 'get_all_tcm_test_cases_with_root_suite_id', 'get_test_case_by_filter', 'get_test_cases_advanced', and 'get_test_cases_by_suite_smart' all retrieve test cases with subtle distinctions that an agent would struggle to differentiate. Similarly, 'analyze_test_cases_duplicates' and 'analyze_test_cases_duplicates_semantic' overlap in purpose, and 'get_launch_details' vs. 'get_launch_summary' vs. 'get_launch_test_summary' are confusingly similar. The descriptions help but don't fully resolve the redundancy.

Naming Consistency3/5

The naming follows a mixed convention with some consistency but notable deviations. Most tools use snake_case (e.g., 'aggregate_test_cases_by_feature', 'get_launch_details'), which is good, but there are inconsistencies like 'list_test_runs' (verb-first) vs. 'get_all_launches_for_project' (get-first), and some tools lack clear verb patterns (e.g., 'test_reporting_connection'). While readable, the mix of 'get', 'list', 'analyze', and other verbs without a strict pattern reduces predictability.

Tool Count2/5

With 49 tools, the count is excessive for a test management server, leading to bloat and confusion. Many tools could be consolidated (e.g., multiple test case retrieval methods, duplicate analysis variants), and the high number suggests poor scoping. This overwhelms agents and makes navigation difficult, as typical MCP servers for similar domains (like GitHub with 10-15 tools) are more manageable. The server's purpose is clear, but the tool surface is unnecessarily fragmented.

Completeness4/5

The tool set is highly comprehensive for test management, covering analysis, retrieval, reporting, and automation tasks with few obvious gaps. It includes CRUD-like operations for test cases, launches, and suites, along with advanced features like failure analysis and duplicate detection. Minor gaps might include direct update or delete tools for test cases, but the server focuses on read and analyze operations, which aligns with its reporting and analytics domain. Overall, the coverage is thorough and supports complex workflows.

Maintenance

ActivityMaintained
ResponsivenessSyncing

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers

  • A
    license
    A
    quality
    B
    maintenance
    Connects AI coding assistants to TestCollab for managing test cases, plans, and suites directly through natural language. It enables users to create, update, and query testing resources within integrated development environments and AI chat clients.
    17
    17
    4
    MIT
  • A
    license
    Not graded
    quality
    D
    maintenance
    Enables AI models (Claude, ChatGPT, GitHub Copilot) to run and analyze local tests, rerun failures, and orchestrate QA workflows using existing UI and API test frameworks.
    19
    MIT

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/maksimsarychau/mcp-zebrunner'

If you have feedback or need assistance with the MCP directory API, please join our Discord server