blacksmith-mcp
Click on "Install Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@blacksmith-mcpShow me the failed tests from the latest workflow run"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Blacksmith MCP
An MCP server that connects Claude to your Blacksmith CI data. Query workflow runs, analyze test failures, detect flaky tests, and monitor usage—all through natural conversation.
Why?
Debugging CI failures usually means clicking through dashboards, copying run IDs, and piecing together information across multiple pages. With this MCP, you can just ask:
"Why did the last CI run fail?"
"Which tests are flaky this week?"
"Compare test failures between main and my PR"
"What's using the most cache storage?"
Claude handles the API calls and gives you actionable insights.
Related MCP server: Langfuse MCP Server
Quick Start
Zero-config if you're logged into Blacksmith in Chrome:
# Add to Claude Code
claude mcp add blacksmith -- npx blacksmith-mcp
# Set your org (run once)
export BLACKSMITH_ORG="your-org-name"The MCP automatically extracts your session from Chrome cookies. No manual token copying needed.
Installation
Option 1: Claude Code CLI
claude mcp add blacksmith -- npx blacksmith-mcpOption 2: Project Configuration
Add to your .mcp.json:
{
"mcpServers": {
"blacksmith": {
"type": "stdio",
"command": "npx",
"args": ["blacksmith-mcp"],
"env": {
"BLACKSMITH_ORG": "your-org-name"
}
}
}
}Option 3: Global Install
npm install -g blacksmith-mcpConfiguration
Authentication
Automatic (recommended): Log into app.blacksmith.sh in Chrome. The MCP extracts your session cookie automatically.
Manual: Set BLACKSMITH_SESSION_COOKIE environment variable with your session cookie value.
Environment Variables
Variable | Required | Description |
| Yes | Your Blacksmith organization name |
| No | Session cookie (auto-extracted from Chrome if not set) |
Available Tools
Workflow Runs
Tool | Description |
| List workflow runs with filters (status, branch, workflow, actor, PR) |
| Get run details including all jobs |
| List jobs for a workflow run |
| Get job details (steps, timing, runner info) |
| Get raw log output for a job |
Test Analytics
Tool | Description |
| Get all test results for a job |
| Get failed tests with full error messages |
| Group failures by error pattern (e.g., "Cannot read properties") |
| Compare failures between two runs (find regressions) |
| Detect tests that fail intermittently |
| Find tests exceeding duration threshold |
| Track a specific test's pass/fail history |
| Analyze trends: duration, failure rate, test count |
Usage & Billing
Tool | Description |
| Current core usage (active vs max) |
| Current billing period amount |
| Billable minutes vs free tier |
| Cache size, entries by repository |
| Detailed cache entries for a repo |
Organization
Tool | Description |
| List accessible organizations |
| Org status (personal, onboarded, region) |
| Search logs across all jobs |
Example Conversations
Debug a Failed Run
You: Why did the last CI run fail?
Claude: Uses
list_runs(status="failure", limit=1)thenget_failed_testsThe last run failed with 34 test failures. The main issues are:
Connection timeout (12 tests) - Tests in
api/are timing out connecting to the test databaseUndefined property (8 tests) -
user.settings.themeis undefined in the new dark mode codeAssertion failures (14 tests) - Snapshot mismatches in UI components
Find Flaky Tests
You: Which tests are flaky?
Claude: Uses
get_flaky_tests(days=7, min_runs=5)Found 3 flaky tests:
Test
Flake Rate
Pattern
should handle concurrent requests40% (4/10 runs)
Fails on CI, passes locally
renders loading state20% (2/10 runs)
Race condition with setTimeout
validates auth token30% (3/10 runs)
Intermittent network timeout
Compare Branches
You: Did my PR introduce new test failures?
Claude: Uses
compare_test_runswith PR run vs base branchYour PR introduced 2 new failures:
UserProfile.test.tsx: "Cannot read property 'avatar' of undefined"
api/users.test.ts: Expected 200, got 4045 previously failing tests now pass (nice!).
Development
# Install dependencies
pnpm install
# Build
pnpm build
# Development mode (watch)
pnpm dev
# Test with MCP Inspector
npx @modelcontextprotocol/inspector node dist/index.jsTroubleshooting
Session Expired
If you see SESSION_EXPIRED, your Blacksmith session has expired. Simply log back into app.blacksmith.sh in Chrome and retry.
Cookie Extraction Failed
The automatic cookie extraction requires:
macOS with Chrome installed
Being logged into Blacksmith in Chrome
Chrome not running with a locked profile
If it fails, set BLACKSMITH_SESSION_COOKIE manually.
No Organization Set
Run list_orgs to see available organizations, then set BLACKSMITH_ORG to your org name.
API Notes
This MCP uses Blacksmith's internal web API, which is undocumented. The API was reverse-engineered from the Blacksmith web app and may change without notice.
License
MIT
Contributing
Contributions welcome! Please open an issue first to discuss proposed changes.
Available Tools
37 toolscompare_test_runsA
Compare test failures between two runs to identify regressions. Shows new failures, fixed tests, and persistent failures. If base_run_id not provided, compares against most recent prior run.
| Name | Required | Description | Default |
|---|---|---|---|
| run_id | Yes | Current GitHub Actions workflow run ID | |
| job_name | Yes | Job name to compare (e.g., "Test (Blacksmith/Self-Hosted)") | |
| base_run_id | No | Base run ID to compare against. If not provided, compares against most recent prior run. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must convey behavioral traits. It discloses the comparison logic and default behavior but does not clarify whether the tool is read-only, what authentication is needed, or any rate limits. The output format is also not described.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is exceptionally concise with two sentences that convey core functionality and a key behavioral detail. Every word is meaningful, no fluff.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema, the description partially explains output ('Shows new failures, fixed tests, and persistent failures') but lacks details on format or structure. For a moderate-complexity tool comparing runs, this is adequate but leaves room for improvement.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, providing baseline at 3. The description reiterates the default behavior for base_run_id, which is already in the schema description, adding no new semantic information beyond what the schema provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states the tool compares test failures between two runs to identify regressions, specifying the types of results (new failures, fixed tests, persistent failures). While the verb 'compare' is specific and the resource is clear, it does not explicitly differentiate itself from sibling tools like get_failed_tests or get_run_tests, which could also relate to test failures.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explains when to use the tool (to identify regressions by comparing runs) and provides a default behavior when base_run_id is omitted. However, it does not specify when not to use the tool or mention alternative tools for similar tasks.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_active_branchesB
Get branches with CI activity. Optionally filter by repository.
| Name | Required | Description | Default |
|---|---|---|---|
| end_date | No | End date (YYYY-MM-DD). Defaults to today. | |
| repository | Yes | Repository name (e.g. "Org/repo"). Required by the API. | |
| start_date | No | Start date (YYYY-MM-DD). Defaults to 7 days ago. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full burden but only offers a high-level statement. It omits critical behavioral details such as pagination, permissions, definition of 'CI activity', or response structure.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence of 10 words with no redundancy. Every word contributes to conveying the core purpose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool has 3 parameters, no output schema, and no annotations. The description does not explain the return type, pagination, or what constitutes 'CI activity', making it insufficient for an agent to fully understand usage.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents all parameters. The description adds minimal value ('optionally filter by repository'), and there is a slight inconsistency where the schema marks repository as required but the description says 'optionally filter'.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it retrieves branches with CI activity and offers optional repository filtering. This verb+resource combination is specific and distinguishes it from siblings like 'get_active_repositories'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies use for listing branches with recent CI runs but does not explicitly state when to use this tool over alternatives or provide exclusions. Given the many sibling tools, more guidance would be helpful.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_active_repositoriesA
Get repositories with CI activity in a period. Shows which repos are actively running workflows.
| Name | Required | Description | Default |
|---|---|---|---|
| end_date | No | End date (YYYY-MM-DD). Defaults to today. | |
| start_date | No | Start date (YYYY-MM-DD). Defaults to 7 days ago. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden. It only states what the tool does but does not disclose behavioral traits such as response format, permission requirements, or any side effects. For a data retrieval tool, more transparency on output structure is needed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences long, front-loaded with the purpose, and contains no superfluous information. Every sentence is necessary.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema, the description should explain the return value. It states 'shows which repos are actively running workflows' but does not specify the format (e.g., list of repo names or objects). With 2 optional parameters and a simple purpose, it is somewhat incomplete but not critically lacking.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so baseline score is 3. The description does not add any additional meaning beyond the schema; it simply reiterates 'in a period' which aligns with the start_date and end_date parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Get' and the resource 'repositories with CI activity in a period'. It distinguishes this tool from numerous sibling 'get_' tools by focusing on active repositories with workflow runs.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for identifying actively running repositories but provides no explicit when-to-use or when-not-to-use guidance, nor does it mention alternatives among the 31 sibling tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_actorsB
Get users who have triggered workflow runs. Useful for filtering runs by actor.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries full burden. It only states it is a read operation, but lacks details on scope, authorization, pagination, or data freshness. For a simple list, this is basic but insufficient.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two concise sentences with the verb first. No wasted words, though it could slightly expand on scope without losing conciseness.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no parameters, output schema, or annotations, the description provides basic purpose and a use case. However, it omits details like organization scope, ordering, or how recent the data is, which would help fully leverage the tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
There are zero parameters, so schema coverage is 100%. Per guidelines, baseline is 4. The description adds no parameter info, but none is needed.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool returns users who triggered workflow runs and specifies a use case. However, it does not explicitly distinguish it from sibling get_* tools, though the resource (actors) is unique enough.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description suggests a use case ('filtering runs by actor'), but does not provide explicit when-to-use or when-not-to-use guidance, nor mention alternatives among siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_cache_entriesA
Get detailed cache entries for a repository. Shows cache keys, sizes, scopes (branches), and last hit times. Useful for debugging cache issues.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | Maximum number of entries to return (default: 20) | |
| repository | Yes | Repository name - try short name first (e.g., "votion"), or full name (e.g., "Votion-Platform/votion") if needed |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so the description must fully disclose behavior. It implies a read operation by saying 'get' and listing fields, but does not mention any potential side effects, authorization requirements, rate limits, or pagination behavior. This is insufficient for a tool with no annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, under 50 words, efficiently conveying the essential purpose and content. No redundant words, front-loaded with the action.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
While the description covers what the tool returns, it lacks details about the output format (e.g., array vs object, pagination) and does not mention any constraints. For a tool with no output schema, more completeness would be beneficial.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and parameter descriptions are already informative. The description adds overall context about returned fields but does not add specific meaning to the parameters beyond what the schema provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('get'), the resource ('cache entries'), and the scope ('for a repository'), listing specific fields returned. It distinguishes well from sibling tools by focusing on cache entries specifically.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description includes 'Useful for debugging cache issues' which gives context, but does not explicitly state when to choose this tool over alternatives like 'get_cache_stats', nor when not to use it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_cache_statsA
Get Blacksmith cache statistics: total size, hit rate, entries by repository. Shows how effectively caching is being used.
| Name | Required | Description | Default |
|---|---|---|---|
| include_history | No | Include historical cache data (default: false) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Although no annotations are provided, the description indicates a read-only operation that returns aggregated statistics, implying no destructive behavior. It does not mention rate limits or permissions, but for a read-only stats tool, this is acceptable.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise with two sentences: the first immediately states the purpose and key outputs, and the second adds a brief insight about the tool's value. No unnecessary words or redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity (one optional boolean, no output schema), the description covers the main outputs and the insight provided. It could mention that including history gives different data, but the schema covers the parameter. Overall, it is sufficiently complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The sole parameter 'include_history' is fully described in the schema with a clear explanation. The tool description does not add any additional semantic context beyond what the schema provides, so it meets the baseline for high schema coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool retrieves Blacksmith cache statistics including total size, hit rate, and entries by repository, which distinguishes it from 'get_cache_entries' that likely returns individual entries. The verb 'Get' and the specific resource are explicit.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description lacks explicit guidance on when to use this tool versus alternatives like 'get_cache_entries'. While the purpose implies it's for aggregated stats, no exclusions or context for choosing between siblings are provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_core_usage_timeseriesA
Get core usage over time. Shows how many cores are being used at each interval. Defaults to last 24 hours with 15-minute intervals.
| Name | Required | Description | Default |
|---|---|---|---|
| hours | No | Number of hours to look back (default: 24). | |
| window_size | No | Data point interval in minutes (default: 15). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden. It only states it shows core usage over time with intervals but lacks disclosure of authentication needs, rate limits, or any destructive behavior. The behavior is minimally defined.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences, front-loads the action, and contains no unnecessary words. It is concise and well-structured.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool has no output schema, so the description should explain what the response contains. It only says 'shows how many cores are being used' but does not describe the format (e.g., array of {timestamp, value}) or any pagination. Adequate for a simple tool but missing return structure details.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with descriptions for both parameters. The description reinforces defaults (24 hours, 15 minutes) but adds no further semantics beyond what the schema provides. Baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it retrieves core usage over time at specified intervals, with defaults of 24 hours and 15-minute intervals. This distinguishes it from siblings like get_current_usage (current snapshot) or get_usage_summary (aggregate).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for time-series core usage data and mentions defaults, but does not explicitly state when to use this tool versus alternatives like get_current_usage or get_usage_summary. No exclusions or alternative suggestions are provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_current_usageA
Get current core usage snapshot (active cores vs max cores).
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided; the description carries full burden. It accurately describes a read-only snapshot but lacks details on authentication, rate limits, or caching behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Single sentence with no waste, front-loaded with the essential information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no parameters and no output schema, the description provides the core purpose. However, additional differentiation from siblings would improve completeness.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
No parameters, schema coverage 100%. The description adds context like 'snapshot' and 'active vs max', but baseline is 3 per guidelines.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb 'Get' and resource 'current core usage snapshot', clearly stating it returns active vs max cores. It distinguishes from sibling tools like 'get_usage_summary' and 'get_core_usage_timeseries'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool vs its siblings. With many similar 'get_*' tools, explicit usage context is missing.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_failed_testsA
Get failed tests for a job with full error details. Use error_lines param to control stack trace length. Returns all failures by default (no limit).
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | Maximum number of failed tests to return (default: all) | |
| suite | No | Filter by test suite name (e.g., "FeatureFlags Middleware") | |
| job_id | Yes | GitHub Actions job ID | |
| run_id | Yes | GitHub Actions workflow run ID | |
| error_lines | No | Number of error lines to include per test (default: 5, max: 50) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must disclose behavioral traits. It notes returns all failures by default (no limit) and error_lines controls stack trace length, but omits potential large output, performance impact, pagination, or authentication needs. For a data retrieval tool, this gap is significant.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences, front-loaded with purpose, and no redundant or filler content. Every word adds value, making it efficient and easy to parse.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema and 5 parameters, the description covers core functionality but lacks details on ordering, pagination, error handling (e.g., no failures), or how it relates to similar tools. Adequate for basic use but not fully comprehensive.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with parameter descriptions. The description adds context about error_lines controlling stack trace length and limit's default (all). This adds marginal value beyond the schema, so baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool retrieves failed tests for a job with full error details, specifying the required parameters (run_id, job_id) and optional controls like error_lines. It effectively distinguishes from siblings like get_flaky_tests or get_slow_tests by focusing on failures.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description mentions controlling stack trace length with error_lines and default behavior, but does not explicitly guide when to use this tool over alternatives like get_failures_by_pattern or get_job_tests. Usage context is implied but not directly compared to siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_failures_by_patternA
Group failed tests by error pattern (e.g., "is not a function", "Cannot read properties"). Shows count, affected suites/files, and sample error for each pattern. Best for quickly identifying root causes.
| Name | Required | Description | Default |
|---|---|---|---|
| top_n | No | Number of top error patterns to return (default: 10) | |
| job_id | Yes | GitHub Actions job ID | |
| run_id | Yes | GitHub Actions workflow run ID |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description fully carries the burden of behavioral disclosure. It explains the output includes count, affected suites/files, and sample error, giving a clear picture of what the tool returns without stating side effects (which are none apparent for a read operation).
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three sentences: first states the grouping functionality, second lists output fields, third identifies best use case. Every sentence is informative, no filler words, efficiently front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has no output schema, the description adequately covers the return format (count, suites/files, sample error). It could mention default sorting or ordering but is sufficient for an agent to understand the output structure.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the JSON schema already documents all three parameters (run_id, job_id, top_n). The description does not add any additional parameter semantics beyond what is in the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool groups failed tests by error pattern, using specific examples like "is not a function". This differentiates it from sibling tools like get_failed_tests which likely list individual failures, and get_flaky_tests which focus on flaky behavior.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description says 'Best for quickly identifying root causes,' which provides context but does not explicitly state when not to use this tool, nor does it mention alternatives like get_failed_tests for individual failure details.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_flaky_testsA
Detect flaky tests by analyzing pass/fail patterns across recent runs. Returns tests that fail intermittently (e.g., "failed 3 of 10 runs"). Killer feature for CI stability.
| Name | Required | Description | Default |
|---|---|---|---|
| days | No | Number of days to analyze (default: 7) | |
| job_name | Yes | Job name to analyze (e.g., "Test (Blacksmith/Self-Hosted)") | |
| min_runs | No | Minimum number of runs a test must appear in to be considered (default: 3) | |
| threshold | No | Flakiness threshold 0-1, e.g., 0.2 means test failed 20%+ of runs (default: 0.1) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the burden. It discloses it returns flaky tests with an example pattern but lacks details on edge cases, default behavior, or response format. Adequate but not comprehensive.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two concise sentences: first states purpose, second adds example and value. No redundant words, front-loaded with key information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given moderate complexity, no output schema, but parameters are well-documented. The description covers core behavior and provides a concrete example of output, making it mostly complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
All four parameters are fully described in the schema (100% coverage). The description adds no extra meaning beyond the schema definitions, so baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool detects flaky tests by analyzing pass/fail patterns. It uses specific verb 'detect' and resource 'flaky tests', and distinguishes from siblings like get_failed_tests and get_slow_tests by focusing on intermittent failures.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Implied usage for CI stability monitoring, but no explicit when-to-use or when-not-to-use conditions. No alternatives or exclusions mentioned, leaving the agent to infer from context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_invoice_amountA
Get the current billing period invoice amount.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description bears full burden. It only states the action without disclosing authorization needs, idempotency, rate limits, or what happens if no invoice exists. Minimal behavioral insight.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Single sentence, no extraneous words. Information is front-loaded and efficient.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a zero-parameter tool, the description states the core function adequately. However, it lacks information about return format (e.g., currency, number type) and error cases. Minimal completeness.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
No parameters exist (schema coverage 100% by definition). The description adds meaning by specifying the value retrieved, which goes beyond the empty schema. Baseline 4 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states the verb 'Get', the resource 'invoice amount', and the scope 'current billing period'. This is specific and distinguishes it from sibling tools, which are all CI/test-related and not billing.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No explicit guidance on when to use or when not. However, the tool has no parameters and no siblings overlap in function, so usage is straightforward. Implied but not explicit.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_jobC
Get details of a specific job including steps, runner info, and timing.
| Name | Required | Description | Default |
|---|---|---|---|
| job_id | Yes | GitHub Actions job ID | |
| run_id | Yes | GitHub Actions workflow run ID |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden of disclosing behavioral traits. It only says 'Get details,' implying a read operation, but does not confirm read-only nature, authentication requirements, rate limits, or any side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence that front-loads the tool's purpose and key output fields. Every word earns its place; no unnecessary content.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers the basic return content (steps, runner info, timing) but does not elaborate on structure, edge cases, or error scenarios. Without an output schema, more detail would be helpful for completeness.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Both parameters are fully described in the schema (100% coverage). The description adds context about the return content (steps, runner info, timing) but does not enhance parameter understanding. Baseline 3 is appropriate given high schema coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('Get details of a specific job') and lists included fields (steps, runner info, timing). This differentiates it from sibling tools like 'list_jobs' or 'get_job_logs', though it could be more explicit about what constitutes 'details'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided about when to use this tool vs alternatives like 'get_job_logs' or 'get_job_tests'. The description implies using it when you need job details, but does not exclude other scenarios or mention prerequisites.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_job_duration_distributionA
Get job duration distribution with percentile breakdowns (P50/P90/P95/P99/Max). Shows histogram of how long jobs take and identifies outliers.
| Name | Required | Description | Default |
|---|---|---|---|
| end_date | No | End date (YYYY-MM-DD). Defaults to today. | |
| start_date | No | Start date (YYYY-MM-DD). Defaults to 7 days ago. | |
| bucket_count | No | Number of histogram buckets. Defaults to 50. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It mentions showing a histogram and identifying outliers but lacks details about data range limits, performance, caching behavior, or how outliers are defined. The behavioral transparency is insufficient for a tool with no annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two concise sentences with no fluff. The description is front-loaded with the most important information and efficiently communicates the tool's purpose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
While the description explains the purpose and outputs (percentiles, histogram, outliers), it lacks details on the return format or structure of the histogram. Without an output schema, the description should provide more clarity on what the response looks like. It is partially complete but insufficient for confident invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% (all three parameters have descriptions). The description adds context about percentiles and histogram but does not provide additional semantics for the parameters beyond what the schema already states. Baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Get' and the resource 'job duration distribution', specifies percentile breakdowns (P50/P90/P95/P99/Max), and mentions histogram and outlier identification, which distinguishes it from sibling tools like 'get_job' or 'get_jobs_daily'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for analyzing job durations and outliers but does not explicitly state when to use this tool versus alternatives or provide exclusions. No guidance on prerequisites or context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_job_logsA
Get the logs for a specific job. Returns raw log output.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | Maximum number of log lines (default: 1000) | |
| vm_id | No | VM ID for the job (optional) | |
| job_id | Yes | GitHub Actions job ID |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided. Description states 'Returns raw log output' but omits any side effects, error behavior, or rate limits. Basic transparency for a read operation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, front-loaded with purpose, no redundant information. Highly efficient.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Simple tool with 3 params and no output schema. Description covers main purpose but lacks details on log source (GitHub Actions), pagination behavior, or when to choose alternatives. Adequate but incomplete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so schema already describes all parameters. Description adds no extra meaning beyond schema, meeting baseline but not compensating.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Clear verb 'Get' and resource 'logs for a specific job'. Mentions return type 'raw log output'. Distinguishes from siblings like get_job (job details) and search_logs (searching logs).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No explicit guidance on when to use this tool vs alternatives like search_logs or get_job_tests. Usage is implied by the name, but lacks exclusions or context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_jobs_by_dimensionA
Break down job metrics by dimension: repository, workflow, branch, or runner_type. Shows which repos/workflows consume the most CI time. Example: get_jobs_by_dimension(dimension="repository", metric="duration")
| Name | Required | Description | Default |
|---|---|---|---|
| top_n | No | Number of top results to return. Defaults to 20. | |
| metric | No | Metric to measure. Defaults to jobs. | |
| end_date | No | End date (YYYY-MM-DD). Defaults to today. | |
| dimension | No | Dimension to break down by. Defaults to repository. | |
| start_date | No | Start date (YYYY-MM-DD). Defaults to 7 days ago. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description must cover behavioral traits. It states the tool shows top-consuming repos/workflows, suggesting an aggregate/ranking, but does not disclose pagination, rate limits, data freshness, or error behavior. The description is adequate but minimal.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences plus a code example, front-loaded with action verb. Every word earns its place; no extraneous information. Perfectly concise.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema is provided, and the description does not detail the return format (e.g., structure of results, ordering). For 5 optional parameters, more context on what the output looks like would improve completeness. The example hints at usage but not results.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with parameter descriptions. The description adds an example showing dimension='repository' and metric='duration', but does not elaborate on defaults or interaction beyond what the schema provides. Baseline is 3 due to high schema coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool breaks down job metrics by dimension (repository, workflow, branch, runner_type) and shows which repos/workflows consume the most CI time. It includes a concrete example, and the purpose is distinct from siblings like get_jobs_summary or get_jobs_daily.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies when to use the tool (for dimension breakdowns) but does not explicitly say when not to use it or compare it to alternatives. No guidance on prerequisites or exclusion criteria is provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_jobs_dailyA
Get daily job metrics: counts, durations, and failure rates per day. Great for spotting trends. Defaults to last 7 days.
| Name | Required | Description | Default |
|---|---|---|---|
| end_date | No | End date (YYYY-MM-DD). Defaults to today. | |
| start_date | No | Start date (YYYY-MM-DD). Defaults to 7 days ago. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided; description discloses that it returns daily counts, durations, and failure rates with a default date range. However, it does not mention response structure, pagination, or authentication requirements. For a simple read operation, this is adequate but not beyond minimal.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, front-loaded with purpose, no redundant words. Each sentence earns its place by stating what it does and a use case.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity (2 optional date parameters, no output schema), the description fully covers the needed context. It explains the kind of data returned and the default range, which is sufficient for an agent to decide and invoke correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema already describes both parameters with YYYY-MM-DD format (100% coverage). The description adds value by stating 'Defaults to last 7 days', providing context on default behavior that the schema does not convey.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses specific verb 'Get' and resource 'daily job metrics' with explicit fields (counts, durations, failure rates). It clearly distinguishes from sibling tools like list_jobs (individual jobs) and get_jobs_summary (likely overall summary).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description states 'Great for spotting trends', indicating usage for trend analysis over time. It defaults to last 7 days. While it doesn't explicitly exclude alternatives, the context of sibling tools implies differentiation from other job retrieval functions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_jobs_summaryA
Get aggregate job statistics for a period: total jobs, total duration, average duration, failure rate. The big picture view.
| Name | Required | Description | Default |
|---|---|---|---|
| end_date | No | End date (YYYY-MM-DD). Defaults to today. | |
| start_date | No | Start date (YYYY-MM-DD). Defaults to 7 days ago. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so description carries full burden. Reveals return fields but omits behavioral traits such as read-only nature, data freshness, rate limits, or required permissions. Incomplete for a tool with zero annotation coverage.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two concise sentences, front-loaded with action and key statistics. No fluff, every sentence adds value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Lists some return fields but not the full structure. Without output schema, description should provide more detail on the summary format (e.g., per day or total). Adequate for a simple tool but leaves gaps.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema covers both parameters 100%, so description adds little beyond 'for a period'. Baseline 3 is appropriate as description merely echoes schema content.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states verb 'Get', resource 'aggregate job statistics', and lists specific metrics (total jobs, duration, average duration, failure rate). Distinguishes from siblings like get_job (single) and list_jobs (list).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Implied as a high-level overview ('The big picture view'), but no explicit guidance on when to use vs alternatives like get_jobs_daily or get_trends. No exclusions or prerequisites mentioned.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_job_testsB
Get test results for a job. Optionally filter by status (pass/fail/skip).
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | Maximum number of tests to return when include_tests is true (default: 50) | |
| job_id | Yes | GitHub Actions job ID | |
| run_id | Yes | GitHub Actions workflow run ID | |
| status | No | Filter by test status | |
| include_tests | No | Include individual test details (default: false, returns summary only) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden but only restates basic functionality. It does not disclose behavioral traits like read-only nature, side effects, rate limits, pagination, or default behavior for 'include_tests'. The schema mentions a default for limit and include_tests, but the description does not reiterate these defaults.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single concise sentence with no wasted words. It conveys the core purpose and one key option. However, it could be restructured to better front-load the most important information or add a note about sibling tools.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description is minimally complete for a simple tool with well-described schema. However, it lacks context about the 'include_tests' parameter's effect on output (summary vs. detailed), default values, and how the tool fits into the workflow of fetching test results. Given no output schema, a bit more explanation would be helpful.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and already describes all parameters. The description adds minimal value by mentioning optional status filtering, which matches the enum. It does not elaborate on 'limit', 'include_tests', or the relationship between run_id and job_id, so it contributes little beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Get' and the resource 'test results for a job', and distinguishes this tool from siblings like get_job, get_job_logs, or get_failed_tests by focusing on test results with optional status filtering.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives such as get_failed_tests, get_slow_tests, or get_test_history, which are listed as siblings. A user must infer the tool's purpose without any explicit comparison or exclusion criteria.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_log_filter_optionsA
Get available filter values for log search (e.g., job names, log levels). Use before search_logs to discover valid filter values.
| Name | Required | Description | Default |
|---|---|---|---|
| hours | No | Number of hours to search back (default: 1, max: 24) | |
| property | Yes | Property to get filter options for (e.g., "job_names" to see available job names). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It states the tool retrieves filter values and implies it is a safe read operation by suggesting use before a search, but it does not disclose any behavioral traits like side effects, authentication requirements, or rate limits.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences (22 words), front-loaded with the verb 'Get', and contains no redundant information. Every word earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema and no annotations, the description adequately covers the tool's purpose and usage context. It could briefly mention the output format (e.g., list of values), but the simple nature of the tool makes it mostly complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3. The description mentions examples (job names, log levels) that align with the 'property' parameter enum, but adds no significant meaning beyond the schema's own parameter descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool retrieves available filter values for log search, with examples like job names and log levels. It distinguishes from sibling tools by advising use before search_logs.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly says 'Use before search_logs to discover valid filter values,' providing clear context for when to use the tool. However, it does not explicitly mention when not to use it or list alternatives beyond search_logs.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_log_histogramA
Get log volume over time as a histogram. Useful for spotting error spikes or unusual activity patterns.
| Name | Required | Description | Default |
|---|---|---|---|
| hours | No | Number of hours to search back (default: 1, max: 24) | |
| query | No | Search query to filter logs. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must fully disclose behavior. It mentions returning a histogram of log volume over time, but does not specify the output format, whether the operation is read-only, or any authentication or rate-limiting requirements. The description is functional but lacks depth on expected response and side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise (two sentences) and front-loaded with the main purpose. It efficiently conveys the tool's function and use case without unnecessary words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has two optional parameters, no output schema, and no annotations, the description provides a clear overview. However, it could be more complete by explicitly describing the histogram output (e.g., time buckets, counts). It is not severely lacking but leaves minor ambiguity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has 100% coverage with descriptions for both parameters (hours, query). The function description does not add additional semantics beyond what is in the schema; it merely restates the purpose. Baseline 3 is appropriate as the schema already sufficiently explains the parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it retrieves log volume as a histogram over time, specifying the resource (logs) and the output format (histogram). It also provides a concrete use case (spotting error spikes or unusual activity patterns), distinguishing it from siblings that deal with different metrics or resources.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description suggests when to use the tool ('spotting error spikes or unusual activity patterns'), providing clear context. However, it does not explicitly exclude scenarios or mention alternative tools (e.g., search_logs for raw logs, or get_failures_by_pattern for specific failure patterns).
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_monitorsC
Get alerting/monitoring rules configured for the organization. Shows active monitors and their timeline history.
| Name | Required | Description | Default |
|---|---|---|---|
| days | No | Number of days of timeline history (default: 30). | |
| limit | No | Maximum number of rules to return (default: 25). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full burden. It mentions 'active monitors and their timeline history' but does not disclose side effects, authentication needs, rate limits, or whether the timeline history is mutable. This is insufficient for a monitoring tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise at two sentences with no redundant information. However, it could be better structured with separate lines for clarity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema is present, and the description does not explain the return format or fields of the monitors and timeline history. Given the absence of output schema, the description should provide more context.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and both parameters have clear descriptions. The tool description adds no additional meaning beyond the schema. Baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states 'Get alerting/monitoring rules configured for the organization' which is a specific verb+resource, and mentions 'active monitors' which differentiates it from other get_* tools. However, it could be more precise about what constitutes an 'alerting/monitoring rule'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives like get_org_status or get_failed_tests. There are no when-to-use or when-not-to-use instructions, leaving the agent uninformed about context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_org_settingsA
Get all organization settings: email, alerts, timeouts, feature flags (Docker caching, SSH, branch protection, log ingestion), and PR comment config.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must convey behavioral traits. It indicates a read operation ('Get'), but does not disclose whether special permissions are required, if results are sensitive, or the structure of the response. This is minimally acceptable.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence that efficiently lists the categories of settings. It is front-loaded with the main action and resource, with no wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has no parameters and no output schema, the description adequately conveys what the tool returns. It enumerates the categories of settings, but does not detail the exact structure or nesting. For a simple retrieval, this is sufficient.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
There are no parameters, so the input schema covers 100% of parameter semantics. The description adds value by listing what the settings include, which helps the agent understand the output without needing parameter details.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly specifies the verb 'Get' and the resource 'all organization settings', and lists specific categories such as email, alerts, timeouts, feature flags, and PR comment config. No sibling tool covers settings, so it is easily distinguishable.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description does not explicitly state when to use this tool or provide alternatives. However, since no sibling tool retrieves settings, usage is implicitly clear. No exclusions or context are given.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_org_statusA
Get the status of the current organization (personal org, onboarding, runner region).
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations exist, so the description carries full burden. It implies a read-only operation but does not explicitly state that it is safe, does not require special permissions, or has no side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, concise sentence of 14 words that is front-loaded with the verb and resource. Every word adds value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description lists three aspects of the status but does not explain what they mean or the response format. Given no output schema, additional context would improve completeness.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has no parameters, so the baseline score is 4. The description adds no parameter info, but none is needed.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Get' and the resource 'status of the current organization', listing three specific aspects (personal org, onboarding, runner region). This explicitly defines the tool's purpose and distinguishes it from siblings like get_org_settings or get_runner_types.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives. The description does not mention prerequisites, typical use cases, or situations where other tools are more appropriate.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_runA
Get details of a specific workflow run by ID. Includes list of jobs.
| Name | Required | Description | Default |
|---|---|---|---|
| run_id | Yes | GitHub Actions workflow run ID |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries full burden. It only states what the tool does (get details, include jobs) but omits critical behavioral traits such as authentication requirements, rate limits, side effects, or safety profile.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two concise sentences, front-loaded with the core action and result. No unnecessary words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool has no output schema, so the description should elaborate on what 'details' means (e.g., status, conclusion, timestamps). It only mentions 'list of jobs', leaving ambiguity about the response structure. Adequate but not thorough.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% for the single parameter (run_id), and the description adds no new meaning beyond the schema description. Baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool retrieves details of a specific workflow run by ID and includes a list of jobs. It distinguishes from siblings like 'get_job' (specific job) and 'list_runs' (list of runs).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for obtaining run details by ID but does not explicitly state when not to use it or suggest alternatives, which is lacking given the many sibling tools with overlapping functionality.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_run_filter_optionsA
Get available filter values for workflow runs: statuses, repositories, branches, workflows, and users. Use this to discover valid filter values before calling list_runs.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It correctly describes the tool as a read operation to get filter values, but it does not disclose any potential behavioral aspects like authentication requirements, rate limits, or the exact structure of the output. The description is adequate but lacks depth.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences, front-loading the purpose and usage. Every word is meaningful, with no redundancy or filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's low complexity (0 parameters, no output schema), the description covers the essential aspects: what it returns (filter values for specified categories) and when to use it (before list_runs). It is complete enough for an agent to select and invoke this tool correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has no parameters, and schema coverage is 100%. The description does not need to add parameter-level detail, but it adds value by explaining the purpose of the output. Baseline 4 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: 'Get available filter values for workflow runs: statuses, repositories, branches, workflows, and users.' It specifies the resource (workflow runs) and the categories of filter values, and distinguishes itself from sibling tool 'list_runs' by mentioning it as a prerequisite.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly advises to 'use this to discover valid filter values before calling list_runs,' which is clear usage guidance. However, it does not provide negative examples or when not to use, but given the simplicity, this is sufficient.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_run_histogramB
Get workflow run duration distribution as a histogram. Shows how long runs typically take.
| Name | Required | Description | Default |
|---|---|---|---|
| end_date | No | End date (YYYY-MM-DD). Defaults to today. | |
| start_date | No | Start date (YYYY-MM-DD). Defaults to 7 days ago. | |
| bucket_count | No | Number of histogram buckets. Defaults to 12. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so description must fully convey behavior. It only states the action and result (histogram of durations) without disclosing aspects like read-only nature, required permissions, or performance implications of large date ranges or high bucket counts.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, 17 words. Front-loaded with the action and result. No redundant or extraneous information. Highly efficient.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description is adequate for a simple query tool with fully documented parameters, but it lacks details on the output format (e.g., bucket structure) and does not mention that it is a read operation. Given no annotations or output schema, more context could be beneficial.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so parameters are already well-documented (end_date, start_date, bucket_count with defaults). The description adds no additional semantics beyond what the schema provides, so baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states it retrieves a histogram of workflow run durations ('Get workflow run duration distribution as a histogram'). Verb and resource are specific. However, it does not differentiate from similar siblings like 'get_job_duration_distribution', which could cause confusion.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives (e.g., 'get_run' for single run details, 'get_log_histogram' for log patterns). Does not mention context such as date range selection or typical scenarios.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_runner_typesA
Get runner types used for CI jobs in a period. Shows what machine types are being used.
| Name | Required | Description | Default |
|---|---|---|---|
| end_date | No | End date (YYYY-MM-DD). Defaults to today. | |
| start_date | No | Start date (YYYY-MM-DD). Defaults to 7 days ago. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, and the description does not disclose any behavioral traits such as idempotency, authentication requirements, or potential side effects. It only states what it shows, not how it behaves or what guarantees it offers.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise with two short sentences, no unnecessary words, and no structural issues. It efficiently conveys the core functionality.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given that there is no output schema and no annotations, the description is minimalist. It does not explain the return format, pagination, or any edge cases. For a simple list tool it is adequate but could provide more detail.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the parameters are already well-documented in the input schema. The description adds no additional semantic value beyond summarizing the purpose.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool retrieves runner types (machine types) used for CI jobs, which is a distinct resource not covered by sibling tools. The verb 'get' and noun 'runner types' are specific and unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implicitly indicates usage for viewing runner types within a date period. While it doesn't explicitly exclude alternative tools, the uniqueness of the resource makes guidance straightforward. No when-not-to-use or alternatives are mentioned.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_run_testsA
Get test results for an entire run (all jobs). Simpler than calling get_job_tests per job. Truncates at 500 tests.
| Name | Required | Description | Default |
|---|---|---|---|
| run_id | Yes | GitHub Actions workflow run ID |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so the description carries full burden. It discloses truncation at 500 tests but lacks details on pagination, sorting, or the structure of the response. This is insufficient for a read tool with a data limit.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Extremely concise: two sentences, no filler, front-loaded with the purpose. Every word earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple tool with one parameter and no output schema, the description is adequate. It clarifies the scope (all jobs) and a key behavioral limitation (truncation). However, it could mention whether tests are sorted or filtered, and what happens when truncation occurs.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and the description adds no additional meaning beyond the schema's description of `run_id` as 'GitHub Actions workflow run ID'. Baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states it retrieves test results for an entire run (all jobs), distinguishing it from the sibling `get_job_tests` by noting it is simpler than calling per job.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly mentions it is simpler than calling `get_job_tests` per job, providing clear context for when to use this tool. However, it does not explicitly state when not to use it, such as for per-job breakdowns.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_slow_testsB
Find tests exceeding a duration threshold. Shows slowest tests, their percentage of total test time, and average duration stats.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | Maximum number of slow tests to return (default: 20) | |
| job_id | Yes | GitHub Actions job ID | |
| run_id | Yes | GitHub Actions workflow run ID | |
| threshold_ms | No | Duration threshold in milliseconds (default: 5000ms = 5s) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must disclose behavioral traits. It mentions the tool shows stats but doesn't disclose that it is a read operation, authentication needs, or pagination behavior. No mention of side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
One clear, front-loaded sentence that efficiently conveys purpose. Could be slightly more structured by listing stats, but it's concise and scannable.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema, so the description partially compensates by mentioning stats. However, it lacks specifics on return structure. Parameter details are covered by schema, but overall context for a 4-parameter tool (2 required) is adequate.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3. The description references the duration threshold and stats but adds no extra meaning beyond the schema's parameter descriptions (limit, threshold_ms). Not detrimental, but not additive.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool finds tests exceeding a duration threshold and specifies what stats are shown (percentage, average duration). It distinguishes from siblings like get_failed_tests. The verb 'Find' and resource 'tests exceeding a duration threshold' are specific.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No explicit guidance on when to use vs alternatives. With many sibling test-related tools (e.g., get_failed_tests, get_flaky_tests), a statement about when to choose this tool is missing.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_test_historyA
Get the failure history for a specific test across recent runs. Shows when it passed/failed, on which branches, and error messages for failures.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | Number of historical results to return (default: 10) | |
| suite | No | Test suite name (helps disambiguate if multiple tests have same name) | |
| job_name | Yes | Job name to search in (e.g., "Test (Blacksmith/Self-Hosted)") | |
| test_name | Yes | Name of the test to look up |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so description carries full burden. It discloses the tool reads history and shows pass/fail, branches, and errors, but does not mention data freshness, rate limits, or what constitutes 'recent runs'. Could be more informative about output ordering or default limit.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences: first states purpose, second adds output details. No wasted words, front-loaded with core purpose. Efficient and clear.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema, description covers output fields (pass/fail, branches, errors). Could improve by noting ordering (chronological) and that limit defaults to 10, but overall sufficient for a history tool with moderate complexity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents all 4 parameters. The description does not add any extra meaning beyond what the schema provides (e.g., does not mention that limit defaults to 10). Baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states it retrieves failure history for a specific test across recent runs, listing pass/fail status, branches, and error messages. This differentiates it from siblings like get_failed_tests (list of failing tests) or get_flaky_tests (flaky tests) by focusing on one test's history.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description does not explicitly state when to use this tool versus alternatives like compare_test_runs or get_job_tests. It implies usage when needing historical data for a single test, but lacks guidance on when not to use it or prerequisites.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_trendsA
Track metrics over time: duration (are tests getting slower?), failure_rate (are tests getting flakier?), test_count (are we adding tests?). Returns trend analysis with data points.
| Name | Required | Description | Default |
|---|---|---|---|
| days | No | Number of days to analyze (default: 14) | |
| metric | Yes | Metric to track: duration (job runtime), failure_rate (% tests failing), test_count (total tests) | |
| job_name | Yes | Job name to analyze (e.g., "Test (Blacksmith/Self-Hosted)") | |
| granularity | No | Group data by day or week (default: day) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It indicates the tool returns 'trend analysis with data points' but does not elaborate on how many data points, aggregation method, or whether trends are computed (e.g., regression). This is adequate but not rich.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, front-loaded with the tool's purpose and examples. Every sentence adds value, with no wasted words. Highly concise and well-structured.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema and 4 parameters, the description does not specify the structure of the returned trend analysis (e.g., data point format, pagination, limits). This is a moderate gap for an agent to use correctly, but not severely incomplete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents all parameters and enums. The description adds no additional parameter meaning beyond restating metric examples, which the schema already covers. Baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool tracks specific metrics (duration, failure_rate, test_count) over time for a job, and distinguishes from sibling tools by focusing on trends rather than individual runs or distributions.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no explicit guidance on when to use this tool versus alternatives (e.g., get_job_duration_distribution, get_jobs_daily), leaving the agent to infer context from sibling names alone.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_usage_summaryA
Get usage summary showing billable minutes vs free tier allowance. Shows remaining free minutes and overage.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description states it 'shows' data (implying a read operation) and specifies outputs (remaining free minutes, overage), but does not disclose authentication needs, caching, or any potential side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two concise sentences with no waste; purpose is front-loaded and every sentence adds value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with no parameters and no output schema, the description covers key outputs (billable minutes, free allowance, remaining free, overage) but lacks specifics like units or time period granularity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
No parameters exist (schema coverage 100%), so baseline is 4. Description adds no param info, which is acceptable.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb 'Get' and resource 'usage summary', and clearly distinguishes itself from siblings like 'get_current_usage' and 'get_invoice_amount' by focusing on free tier vs billable minutes, remaining free minutes, and overage.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool vs alternatives (e.g., get_current_usage, get_invoice_amount), nor are there any prerequisites or context for usage.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_jobsA
List all jobs for a specific workflow run. Use this to get job IDs for get_job, get_job_logs, and get_job_tests.
| Name | Required | Description | Default |
|---|---|---|---|
| run_id | Yes | GitHub Actions workflow run ID |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries full burden. It only states 'List all jobs', implying a read operation, but does not disclose any behavioral traits such as pagination, rate limits, or whether the list is exhaustive. No contradiction with annotations since none exist.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, no wasted words. First sentence clearly states the purpose, second provides usage guidance. Perfectly sized for the tool's simplicity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple tool with one parameter and no output schema, the description covers the basic purpose and usage. However, it lacks any mention of what the output contains (e.g., list of job objects with IDs), which would be helpful for an agent to understand the return format.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with the single parameter run_id described as 'GitHub Actions workflow run ID'. The description adds no additional meaning beyond what the schema provides, meeting the baseline for full coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb 'List' and resource 'jobs for a specific workflow run'. It directly distinguishes itself from siblings like get_job, get_job_logs, get_job_tests by stating it provides job IDs for those tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly states when to use this tool: 'Use this to get job IDs for get_job, get_job_logs, and get_job_tests.' It implies the context of needing job IDs before calling those tools, but does not mention when not to use it or alternatives like get_jobs_by_dimension.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_orgsA
List all Blacksmith organizations accessible to your account. Use this first to see available orgs.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full burden. It states 'list all ... accessible to your account,' implying a read-only, safe operation. No side effects or rate limits are mentioned, but for a simple list tool, this is adequate.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences with no wasted words. The description is front-loaded with the action and concise.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
While the tool is simple with no parameters, the description lacks output details. With no output schema, the agent might benefit from knowing what fields are returned (e.g., org IDs or names). This is a minor gap.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
There are no parameters, so the baseline is 4. The description does not need to add parameter details, and schema coverage is 100%.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'List all Blacksmith organizations' with a specific verb and resource. It distinguishes from sibling tools as none of them list organizations directly.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description says 'Use this first to see available orgs,' providing explicit context for when to use the tool. However, it does not mention when not to use it or explore alternatives, which are not needed given no sibling for org listing.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_runsA
List workflow runs with filtering. Filter by status (success/failure/cancelled/skipped/in_progress), branch, workflow name, actor, or PR number. Example: list_runs(status="failure") to find failed runs.
| Name | Required | Description | Default |
|---|---|---|---|
| actor | No | Filter by actor (GitHub username who triggered the run) | |
| limit | No | Maximum number of runs to return | |
| branch | No | Filter by branch name | |
| status | No | Filter by run status: success, failure, cancelled, skipped, or in_progress | |
| end_date | No | End date (YYYY-MM-DD) | |
| pr_number | No | Filter by pull request number | |
| start_date | No | Start date (YYYY-MM-DD) | |
| workflow_name | No | Filter by workflow name |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full burden. It discloses the basic operation (listing and filtering) but omits behavioral details like pagination behavior, default ordering, or whether it returns only recent runs. The limit parameter hints at pagination but is not explained.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences plus an example, all front-loaded and efficient. Every sentence serves a purpose without wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has 8 parameters and no output schema, the description covers the main functionality and filtering options. However, it misses details like result ordering or default limit, which would make it more complete for an agent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema description coverage is 100%, so baseline is 3. The description adds marginal value by listing filters inline and providing an example, but does not add significant meaning beyond what the schema already provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool lists workflow runs with filtering, distinguishing it from sibling tools like get_run (single run) or list_jobs (jobs). The verb 'list' is specific and the resource 'workflow runs' is explicit.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides filtering options and an example usage, giving practical context. However, it lacks explicit guidance on when to use this tool versus alternatives like get_run or compare_test_runs, or when not to use it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
search_logsA
Search logs across all jobs. Filter by query (e.g., "error", "timeout"), log level (INFO/WARN/ERROR/DEBUG), and time range. Great for finding issues across runs.
| Name | Required | Description | Default |
|---|---|---|---|
| hours | No | Number of hours to search back (default: 1, max: 24) | |
| level | No | Filter by log level | |
| limit | No | Maximum number of logs to return (default: 100) | |
| query | No | Search query (e.g., "error", "timeout", "failed"). Leave empty to get all logs. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Implicitly a read operation but not explicitly stated. No side effects mentioned. Lacks detail on pagination or behavior with no results.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, front-loaded with action, no unnecessary words. Concise and to the point.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Adequate for a simple search tool with no output schema. Missing details on return format but sufficient given parameter descriptions and common understanding.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema covers 100% of parameters with descriptions. Tool description adds example query values and mentions 'time range', providing marginal extra value beyond schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states it searches logs across all jobs with filtering options (query, log level, time range). Distinguishes from sibling tools like get_job_logs by specifying 'across all jobs'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides context for when to use ('finding issues across runs') and implies cross-job searching, but does not explicitly exclude alternatives or specify when not to use.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.
37 tool updates
v0.1.1- First observed
compare_test_runs - First observed
get_active_branches - First observed
get_active_repositories - First observed
get_actors - First observed
get_cache_entries - First observed
get_cache_stats - First observed
get_core_usage_timeseries - First observed
get_current_usage - First observed
get_failed_tests - First observed
get_failures_by_pattern - First observed
get_flaky_tests - First observed
get_invoice_amount - First observed
get_job - First observed
get_job_duration_distribution - First observed
get_job_logs - First observed
get_job_tests - First observed
get_jobs_by_dimension - First observed
get_jobs_daily - First observed
get_jobs_summary - First observed
get_log_filter_options - First observed
get_log_histogram - First observed
get_monitors - First observed
get_org_settings - First observed
get_org_status - First observed
get_run - First observed
get_run_filter_options - First observed
get_run_histogram - First observed
get_run_tests - First observed
get_runner_types - First observed
get_slow_tests - First observed
get_test_history - First observed
get_trends - First observed
get_usage_summary - First observed
list_jobs - First observed
list_orgs - First observed
list_runs - First observed
search_logs
TDQS
Most tools have distinct purposes (runs, jobs, tests, usage, logs, etc.), but there are several test-related tools (get_failed_tests, get_failures_by_pattern, get_flaky_tests, get_slow_tests, get_test_history) that could cause confusion despite detailed descriptions.
The predominant pattern is 'get_*', but there are exceptions like 'list_jobs', 'list_orgs', 'list_runs', 'compare_test_runs', and 'search_logs', mixing verbs and styles.
With 37 tools, the server is overloaded for a CI analytics domain. While each tool has a clear purpose, the sheer number can overwhelm agents and suggests insufficient consolidation.
The tool surface covers monitoring, debugging, and analytics (runs, jobs, tests, logs, cache, usage) well, but lacks management actions like triggering runs or updating settings, leaving the domain partially incomplete.
Maintenance
Resources
Unclaimed servers have limited discoverability.
Looking for Admin?
If you are the server author, to access and configure the admin panel.
Related MCP Connectors
Direct access to Cypress tests results and accessibility reports in your AI workflow.
Read-only analytics for Convex apps, queryable via MCP from Claude, Cursor, and other clients.
Query your Betterlytics web analytics from AI agents: traffic, funnels, journeys, errors, uptime.
BuildPulse CI test analytics for AI agents — flaky tests, coverage, and CI run history.
Related MCP Servers
- FlicenseNot gradedqualityCmaintenanceProvides access to PyTorch CI/CD analytics data including workflows, jobs, test runs, and log analysis through an MCP interface.4-
- AlicenseBqualityDmaintenanceEnables querying Langfuse analytics, cost metrics, and usage data across multiple projects. Provides tools for trace analysis, model/service cost breakdowns, and daily usage trends through natural language queries.2492MIT
- FlicenseNot gradedqualityDmaintenanceProvides statistical analysis and usage tracking for Claude Code, enabling users to query daily/monthly stats, compare users, analyze trends, detect anomalies, and generate usage reports through natural language.-
- AlicenseAqualityAmaintenanceEnables querying Rybbit Analytics data directly through MCP-compatible clients like Claude Code. It provides tools for monitoring website statistics, user sessions, error logs, funnels, and performance metrics via natural language.4064MIT
Latest Blog Posts
- Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)By Om-Shree-0709 on .mcpAgent IdentityOAuth 2.1
- Your AI Chatbot Just Exposed Your CEO's Salary to an InternBy Om-Shree-0709 on .Agent IdentityMCP SecurityOAuth Delegation
- Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)By Om-Shree-0709 on .Agentic AiPrompt InjectionWebAssembly
MCP directory API
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/grahamnotgrant/blacksmith-mcp'
If you have feedback or need assistance with the MCP directory API, please join our Discord server