A
licenseA
qualityC
maintenanceProvides tools to analyze and debug GitHub Actions CI failures, including summarizing failures, detecting flaky tests, and suggesting fixes.
10
3 npm
1
ISC
MCP Test Failure Analysis Server
by rudrathkr
MCPサーバーデモ
このプロジェクトには、FastMCPで構築されたQA向けのテスト失敗分析用Python MCPサーバーが含まれています。
現在の構造
mcp-server-demo/
├── FailureAnalysisFromLogs.py
├── README.md
├── logs/
├── pyproject.toml
├── testFailureAnalysis.py
└── uv.lockRelated MCP server: QA Copilot AI
このプロジェクトに含まれるもの
testFailureAnalysis.pyanalyze_test_failurecluster_failuresdetect_flaky_tests
FailureAnalysisFromLogs.pyanalyze_test_failurelogs/フォルダー内のローカルの.logファイルを読み込み、分類します
logs/ログベースのMCPサーバーで使用されるサンプル失敗ログ
pyproject.tomlPythonのバージョンと依存関係の設定
uv.lock再現可能なインストールを行うための固定された依存関係バージョン
.vscode/mcp.jsontest-failure-analysis、test-failure-analysis-from-logs、およびmcp-atlassian用のMCPサーバーエントリ
前提条件
Python
3.11以降uv初回依存関係インストール時のインターネット接続
インストール
リポジトリのルートから以下を実行します：
cd mcp-server-demo
uv syncサーバーの実行
入力ベースのMCPサーバーを起動するには：
uv run python testFailureAnalysis.pyログベースのMCPサーバーを起動するには：
uv run python FailureAnalysisFromLogs.py利用可能なツール
analyze_test_failure
テスト名、スタックトレース、ログを使用して失敗したテストを分析し、考えられる失敗のカテゴリと推奨事項を返します。
FailureAnalysisFromLogs.py の analyze_test_failure
logs/フォルダー内のローカルの.logファイルを分析し、失敗の分類、考えられる根本原因、推奨事項、および重要なエラー行を返します。すべてのログ、特定のテスト名、または正確なログファイル名を指定して分析できます。
cluster_failures
スタックトレースのシグネチャごとに類似の失敗をグループ化し、繰り返されるパターンを見つけやすくします。
detect_flaky_tests
過去の成功/失敗の結果を確認し、不安定な動作（flaky）を示すテストを特定します。
オプションのローカルMCP設定
リポジトリのルートには.vscode/mcp.jsonが含まれており、開発中のローカルサーバー設定のためにMCP対応ツールで使用できます。これにはtest-failure-analysis、test-failure-analysis-from-logs、およびmcp-atlassianのエントリが含まれています。
Atlassian MCP
ローカルのMCP設定には、Jiraアクセス用のmcp-atlassianサーバーエントリが含まれています。
使用方法
MCP対応クライアントで、.vscode/mcp.jsonから設定済みのmcp-atlassianエントリを使用してください。
手動で実行するには、以下を使用します：
JIRA_URL=<your-jira-url> \
JIRA_USERNAME=<your-jira-username> \
JIRA_API_TOKEN=<your-jira-api-token> \
uvx mcp-atlassianトラブルシューティング
uvが利用できない場合は、インストールしてからターミナルを再起動してください。依存関係のインストールが失敗する場合は、Python
3.11+がアクティブであることを確認してください。サーバーが起動しない場合は、
mcp-server-demo内で再度uv syncを実行してください。
image/project-workflow.png にあるワークフロー画像
Available Tools
3 toolsanalyze_test_failureC
Analyze a failed test and suggest likely root cause.
| Name | Required | Description | Default |
|---|---|---|---|
| test_name | Yes | ||
| stack_trace | Yes | ||
| logs | Yes |
TDQS
C2.4/5.0
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations exist, and the description only gives a high-level purpose without detailing behavior (e.g., whether it is safe, requires permissions, or modifies state). The description does not compensate for missing annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness3/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is very short (one sentence), which is concise but lacks necessary detail. It is front-loaded, but could be more informative without losing conciseness.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness1/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With 3 required parameters, no output schema, and no annotations, the description is severely lacking. It does not explain return values, how inputs are used, or any other context needed for effective use.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters1/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%; the description adds no meaning about the three parameters (test_name, stack_trace, logs). They are just named but not explained.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: analyzing a failed test and suggesting a root cause. However, it does not explicitly differentiate from siblings like 'cluster_failures' or 'detect_flaky_tests', though the intent is distinct.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives, no prerequisites or exclusions provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
cluster_failuresB
Group failures by similar error signature.
Each failure should contain test_name and stack_trace.
| Name | Required | Description | Default |
|---|---|---|---|
| failures | Yes |
TDQS
B3.1/5.0
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must cover behavioral traits. It only states grouping by error signature but omits details about output, state modifications, error handling, or performance implications.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences, highly concise, and front-loaded with the core action. Every word adds value without redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
There is no output schema and no description of return values. The tool clusters failures but doesn't indicate what the output looks like (e.g., clusters with centroid signatures). This is a significant gap for a grouping tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The single parameter 'failures' is an array of objects without schema descriptions. The description compensates by specifying that each failure should contain 'test_name' and 'stack_trace', adding meaning beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'group' and resource 'failures by similar error signature', and specifies required fields. It distinguishes from sibling tools like analyze_test_failure and detect_flaky_tests, though not explicitly.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus its siblings. There is no mention of scenarios where clustering is appropriate or when to use alternative tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
detect_flaky_testsB
Detect flaky tests from historical pass/fail data.
Each item should contain test_name and status.
| Name | Required | Description | Default |
|---|---|---|---|
| test_history | Yes |
TDQS
B3.3/5.0
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description bears full responsibility. It does not disclose whether the tool is read-only, destructive, or has side effects. It only hints at input formatting (test_name and status) but omits behavior like output structure or failure modes.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise with two sentences: the first states the tool's purpose, and the second provides an input hint. No unnecessary words; every sentence contributes meaning.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple tool with one array parameter and no output schema, the description covers the core purpose and input requirement. However, it lacks details on output (e.g., what the tool returns) and fails to provide context relative to siblings. It is minimally adequate but not fully complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, but the description adds that each item should contain 'test_name and status', which provides necessary meaning beyond the schema (which only says array of objects with additionalProperties). However, it does not specify data types or expected values, so it adds limited value.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('Detect'), the resource ('flaky tests'), and the data source ('historical pass/fail data'). It distinguishes from sibling tools like 'analyze_test_failure' which focuses on analyzing a specific failure, and 'cluster_failures' which clusters failures, making the tool's purpose distinct.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no explicit guidance on when to use this tool versus alternatives ('analyze_test_failure', 'cluster_failures'). It implies usage for historical data but does not set context for selection, leaving the agent to infer.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
3 tool updates
v0.1.0- First observed
analyze_test_failure - First observed
cluster_failures - First observed
detect_flaky_tests
TDQS
B3.3/5.0
Scored across 3 tools
Disambiguation5/5
Each tool targets a distinct aspect of test failure analysis: root cause analysis for a single failure, grouping failures by error signature, and detecting flaky tests from history. There is no overlap or ambiguity.
Naming Consistency5/5
All tool names follow a consistent verb_noun pattern using snake_case (analyze_test_failure, cluster_failures, detect_flaky_tests), making them predictable and easy to understand.
Tool Count5/5
With exactly 3 tools, the server is well-scoped for test failure analysis. This number is sufficient to cover core functionalities without being overwhelming or too sparse.
Completeness4/5
The tool set covers essential operations for analyzing test failures, clustering, and flakiness detection. However, it lacks tools for data retrieval (e.g., fetching test history) or generating summary reports, which are minor gaps.
Maintenance
ActivityInactive
ResponsivenessNo issues
Related MCP Connectors
Flaky test detection, root cause analysis, and fix suggestions for development teams.
Offline log parsing, CI failures, HTTP capture analysis and bounded JSON configuration validation.
Triage failing GitHub Actions jobs and see what self-heal repaired, in natural language.
BuildPulse CI test analytics for AI agents — flaky tests, CI failures, flakiness, and code coverage.
Related MCP Servers
- AlicenseBqualityDmaintenanceMCP server for AI-powered QA analysis. It enables analyzing test failures, identifying root causes, suggesting fixes, classifying defects, detecting flaky tests, and generating test cases and bug reports.10ISC
- AlicenseNot gradedqualityCmaintenanceEnables AI-driven triage of Allure test results by clustering failures into stable signatures and generating root-cause explanations, so you can turn large failing CI runs into a few actionable categories.MIT
- FlicenseNot gradedqualityBmaintenanceEnables automated analysis of test suite traces to identify timing races and state pollution, helping detect flaky test non-determinism in CI/CD workflows such as Datadog CI or Trunk.8-