find_flaky_tests
Identify unstable tests in CircleCI projects to analyze failures and implement targeted fixes for improved test reliability.
Instructions
This tool retrieves information about flaky tests in a CircleCI project.
The agent receiving this output MUST analyze the flaky test data and implement appropriate fixes based on the specific issues identified.
CRITICAL REQUIREMENTS:
1. Truncation Handling (HIGHEST PRIORITY):
- ALWAYS check for <MCPTruncationWarning> in the output
- When present, you MUST start your response with:
"WARNING: The logs have been truncated. Only showing the most recent entries. Earlier build failures may not be visible."
- Only proceed with log analysis after acknowledging the truncation
Input options (EXACTLY ONE of these THREE options must be used):
Option 1 - Project Slug:
- projectSlug: The project slug obtained from listFollowedProjects tool (e.g., "gh/organization/project")
Option 2 - Direct URL (provide ONE of these):
- projectURL: The URL of the CircleCI project in any of these formats:
* Project URL: https://app.circleci.com/pipelines/gh/organization/project
* Pipeline URL: https://app.circleci.com/pipelines/gh/organization/project/123
* Workflow URL: https://app.circleci.com/pipelines/gh/organization/project/123/workflows/abc-def
* Job URL: https://app.circleci.com/pipelines/gh/organization/project/123/workflows/abc-def/jobs/xyz
Option 3 - Project Detection (ALL of these must be provided together):
- workspaceRoot: The absolute path to the workspace root
- gitRemoteURL: The URL of the git remote repository
Additional Requirements:
- Never call this tool with incomplete parameters
- If using Option 1, make sure to extract the projectSlug exactly as provided by listFollowedProjects
- If using Option 2, the URLs MUST be provided by the user - do not attempt to construct or guess URLs
- If using Option 3, BOTH parameters (workspaceRoot, gitRemoteURL) must be provided
- If none of the options can be fully satisfied, ask the user for the missing information before making the tool callInput Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| params | No |
Implementation Reference
- src/tools/getFlakyTests/handler.ts:18-67 (handler)Primary handler function for the 'find_flaky_tests' tool. Processes input parameters to identify the CircleCI project slug, fetches flaky tests data, and returns either formatted text or writes detailed files if FILE_OUTPUT_DIRECTORY is set.
export const getFlakyTestLogs: ToolCallback<{ params: typeof getFlakyTestLogsInputSchema; }> = async (args) => { const { workspaceRoot, gitRemoteURL, projectURL, projectSlug: inputProjectSlug, } = args.params; let projectSlug: string | null | undefined; if (inputProjectSlug) { projectSlug = inputProjectSlug; } else if (projectURL) { projectSlug = getProjectSlugFromURL(projectURL); } else if (workspaceRoot && gitRemoteURL) { projectSlug = await identifyProjectSlug({ gitRemoteURL, }); } else { return mcpErrorOutput( 'Missing required inputs. Please provide either: 1) projectSlug, 2) projectURL, or 3) workspaceRoot with gitRemoteURL.', ); } if (!projectSlug) { return mcpErrorOutput(` Project not found. Ask the user to provide the inputs user can provide based on the tool description. Project slug: ${projectSlug} Git remote URL: ${gitRemoteURL} `); } const tests = await getFlakyTests({ projectSlug, }); if (process.env.FILE_OUTPUT_DIRECTORY) { try { return await writeTestsToFiles({ tests }); } catch (error) { console.error(error); return formatFlakyTests(tests); } } return formatFlakyTests(tests); }; - Zod schema defining the input parameters for the tool: projectSlug, workspaceRoot, gitRemoteURL, or projectURL (exactly one method to identify the project).
export const getFlakyTestLogsInputSchema = z.object({ projectSlug: z.string().describe(projectSlugDescriptionNoBranch).optional(), workspaceRoot: z .string() .describe( 'The absolute path to the root directory of your project workspace. ' + 'This should be the top-level folder containing your source code, configuration files, and dependencies. ' + 'For example: "/home/user/my-project" or "C:\\Users\\user\\my-project"', ) .optional(), gitRemoteURL: z .string() .describe( 'The URL of the remote git repository. This should be the URL of the repository that you cloned to your local workspace. ' + 'For example: "https://github.com/user/my-project.git"', ) .optional(), projectURL: z .string() .describe( 'The URL of the CircleCI project. Can be any of these formats:\n' + '- Project URL: https://app.circleci.com/pipelines/gh/organization/project\n' + '- Project URL with branch: https://app.circleci.com/pipelines/gh/organization/project?branch=feature-branch\n' + '- Pipeline URL: https://app.circleci.com/pipelines/gh/organization/project/123\n' + '- Workflow URL: https://app.circleci.com/pipelines/gh/organization/project/123/workflows/abc-def\n' + '- Job URL: https://app.circleci.com/pipelines/gh/organization/project/123/workflows/abc-def/jobs/xyz', ) .optional(), }); - src/tools/getFlakyTests/tool.ts:3-41 (registration)Tool definition object registering the tool with name 'find_flaky_tests', detailed usage instructions in description, and input schema reference. Included in the CCI_TOOLS array.
export const getFlakyTestLogsTool = { name: 'find_flaky_tests' as const, description: ` This tool retrieves information about flaky tests in a CircleCI project. The agent receiving this output MUST analyze the flaky test data and implement appropriate fixes based on the specific issues identified. CRITICAL REQUIREMENTS: 1. Truncation Handling (HIGHEST PRIORITY): - ALWAYS check for <MCPTruncationWarning> in the output - When present, you MUST start your response with: "WARNING: The logs have been truncated. Only showing the most recent entries. Earlier build failures may not be visible." - Only proceed with log analysis after acknowledging the truncation Input options (EXACTLY ONE of these THREE options must be used): Option 1 - Project Slug: - projectSlug: The project slug obtained from listFollowedProjects tool (e.g., "gh/organization/project") Option 2 - Direct URL (provide ONE of these): - projectURL: The URL of the CircleCI project in any of these formats: * Project URL: https://app.circleci.com/pipelines/gh/organization/project * Pipeline URL: https://app.circleci.com/pipelines/gh/organization/project/123 * Workflow URL: https://app.circleci.com/pipelines/gh/organization/project/123/workflows/abc-def * Job URL: https://app.circleci.com/pipelines/gh/organization/project/123/workflows/abc-def/jobs/xyz Option 3 - Project Detection (ALL of these must be provided together): - workspaceRoot: The absolute path to the workspace root - gitRemoteURL: The URL of the git remote repository Additional Requirements: - Never call this tool with incomplete parameters - If using Option 1, make sure to extract the projectSlug exactly as provided by listFollowedProjects - If using Option 2, the URLs MUST be provided by the user - do not attempt to construct or guess URLs - If using Option 3, BOTH parameters (workspaceRoot, gitRemoteURL) must be provided - If none of the options can be fully satisfied, ask the user for the missing information before making the tool call `, inputSchema: getFlakyTestLogsInputSchema, }; - src/circleci-tools.ts:58-72 (registration)Central registration mapping tool names to their handler functions, specifically 'find_flaky_tests' to getFlakyTestLogs. The tool object is also imported and included in CCI_TOOLS array.
export const CCI_HANDLERS = { get_build_failure_logs: getBuildFailureLogs, find_flaky_tests: getFlakyTestLogs, get_latest_pipeline_status: getLatestPipelineStatus, get_job_test_results: getJobTestResults, config_helper: configHelper, create_prompt_template: createPromptTemplate, recommend_prompt_template_tests: recommendPromptTemplateTests, run_pipeline: runPipeline, list_followed_projects: listFollowedProjects, run_evaluation_tests: runEvaluationTests, rerun_workflow: rerunWorkflow, analyze_diff: analyzeDiff, run_rollback_pipeline: runRollbackPipeline, } satisfies ToolHandlers; - Supporting helper that writes each flaky test to an individual file in the output directory, creates .gitignore, and returns a list of file paths for analysis.
const writeTestsToFiles = async ({ tests, }: { tests: Test[]; }): Promise<{ content: { type: 'text'; text: string; }[]; }> => { if (tests.length === 0) { return { content: [ { type: 'text' as const, text: 'No flaky tests found - no files created', }, ], }; } const flakyTestsOutputDirectory = getFlakyTestsOutputDirectory(); try { rmSync(flakyTestsOutputDirectory, { recursive: true, force: true }); mkdirSync(flakyTestsOutputDirectory, { recursive: true }); // Create .gitignore to ignore all files in this directory const gitignorePath = join(flakyTestsOutputDirectory, '.gitignore'); const gitignoreContent = '# Ignore all flaky test output files\n*\n'; writeFileSync(gitignorePath, gitignoreContent, 'utf8'); } catch (error) { throw new Error( `Failed to create output directory: ${error instanceof Error ? error.message : String(error)}`, ); } const filePaths: string[] = []; try { tests.forEach((test, index) => { const filename = generateSafeFilename({ test, index }); const filePath = join(flakyTestsOutputDirectory, filename); writeTestToFile({ test, filePath, index }); filePaths.push(filePath); }); return { content: [ { type: 'text' as const, text: `Found ${tests.length} flaky tests that need stabilization. Each file contains test failure data and metadata - analyze these reports to understand what's causing the flakiness, then locate and fix the actual test code.\n\nFlaky test reports:\n${filePaths.map((path) => `- ${path}`).join('\n')}\n\nFiles are located in: ${flakyTestsOutputDirectory}`, }, ], }; } catch (error) { return mcpErrorOutput( `Failed to write flaky test files: ${error instanceof Error ? error.message : String(error)}`, ); } };