find_flaky_tests
Retrieve information about flaky tests in CircleCI projects to analyze test reliability and implement targeted fixes.
Instructions
This tool retrieves information about flaky tests in a CircleCI project.
The agent receiving this output MUST analyze the flaky test data and implement appropriate fixes based on the specific issues identified.
CRITICAL REQUIREMENTS:
1. Truncation Handling (HIGHEST PRIORITY):
- ALWAYS check for <MCPTruncationWarning> in the output
- When present, you MUST start your response with:
"WARNING: The logs have been truncated. Only showing the most recent entries. Earlier build failures may not be visible."
- Only proceed with log analysis after acknowledging the truncation
Input options (EXACTLY ONE of these THREE options must be used):
Option 1 - Project Slug:
- projectSlug: The project slug obtained from listFollowedProjects tool (e.g., "gh/organization/project")
Option 2 - Direct URL (provide ONE of these):
- projectURL: The URL of the CircleCI project in any of these formats:
* Project URL: https://app.circleci.com/pipelines/gh/organization/project
* Pipeline URL: https://app.circleci.com/pipelines/gh/organization/project/123
* Workflow URL: https://app.circleci.com/pipelines/gh/organization/project/123/workflows/abc-def
* Job URL: https://app.circleci.com/pipelines/gh/organization/project/123/workflows/abc-def/jobs/xyz
Option 3 - Project Detection (ALL of these must be provided together):
- workspaceRoot: The absolute path to the workspace root
- gitRemoteURL: The URL of the git remote repository
Additional Requirements:
- Never call this tool with incomplete parameters
- If using Option 1, make sure to extract the projectSlug exactly as provided by listFollowedProjects
- If using Option 2, the URLs MUST be provided by the user - do not attempt to construct or guess URLs
- If using Option 3, BOTH parameters (workspaceRoot, gitRemoteURL) must be provided
- If none of the options can be fully satisfied, ask the user for the missing information before making the tool call
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| params | No |
Implementation Reference
- src/tools/getFlakyTests/handler.ts:18-67 (handler)Primary MCP tool handler function. Resolves project slug from inputs, fetches flaky tests data, handles file output if enabled, otherwise formats for MCP response.export const getFlakyTestLogs: ToolCallback<{ params: typeof getFlakyTestLogsInputSchema; }> = async (args) => { const { workspaceRoot, gitRemoteURL, projectURL, projectSlug: inputProjectSlug, } = args.params ?? {}; let projectSlug: string | null | undefined; if (inputProjectSlug) { projectSlug = inputProjectSlug; } else if (projectURL) { projectSlug = getProjectSlugFromURL(projectURL); } else if (workspaceRoot && gitRemoteURL) { projectSlug = await identifyProjectSlug({ gitRemoteURL, }); } else { return mcpErrorOutput( 'Missing required inputs. Please provide either: 1) projectSlug, 2) projectURL, or 3) workspaceRoot with gitRemoteURL.', ); } if (!projectSlug) { return mcpErrorOutput(` Project not found. Ask the user to provide the inputs user can provide based on the tool description. Project slug: ${projectSlug} Git remote URL: ${gitRemoteURL} `); } const tests = await getFlakyTests({ projectSlug, }); if (process.env.FILE_OUTPUT_DIRECTORY) { try { return await writeTestsToFiles({ tests }); } catch (error) { console.error(error); return formatFlakyTests(tests); } } return formatFlakyTests(tests); };
- Zod schema defining the input parameters for the find_flaky_tests tool.export const getFlakyTestLogsInputSchema = z.object({ projectSlug: z.string().describe(projectSlugDescriptionNoBranch).optional(), workspaceRoot: z .string() .describe( 'The absolute path to the root directory of your project workspace. ' + 'This should be the top-level folder containing your source code, configuration files, and dependencies. ' + 'For example: "/home/user/my-project" or "C:\\Users\\user\\my-project"', ) .optional(), gitRemoteURL: z .string() .describe( 'The URL of the remote git repository. This should be the URL of the repository that you cloned to your local workspace. ' + 'For example: "https://github.com/user/my-project.git"', ) .optional(), projectURL: z .string() .describe( 'The URL of the CircleCI project. Can be any of these formats:\n' + '- Project URL: https://app.circleci.com/pipelines/gh/organization/project\n' + '- Project URL with branch: https://app.circleci.com/pipelines/gh/organization/project?branch=feature-branch\n' + '- Pipeline URL: https://app.circleci.com/pipelines/gh/organization/project/123\n' + '- Workflow URL: https://app.circleci.com/pipelines/gh/organization/project/123/workflows/abc-def\n' + '- Job URL: https://app.circleci.com/pipelines/gh/organization/project/123/workflows/abc-def/jobs/xyz', ) .optional(), });
- src/circleci-tools.ts:68-85 (registration)Registers the 'find_flaky_tests' tool name mapped to its handler function in the central CCI_HANDLERS object.export const CCI_HANDLERS = { get_build_failure_logs: getBuildFailureLogs, find_flaky_tests: getFlakyTestLogs, get_latest_pipeline_status: getLatestPipelineStatus, get_job_test_results: getJobTestResults, config_helper: configHelper, create_prompt_template: createPromptTemplate, recommend_prompt_template_tests: recommendPromptTemplateTests, run_pipeline: runPipeline, list_followed_projects: listFollowedProjects, run_evaluation_tests: runEvaluationTests, rerun_workflow: rerunWorkflow, download_usage_api_data: downloadUsageApiData, find_underused_resource_classes: findUnderusedResourceClasses, analyze_diff: analyzeDiff, run_rollback_pipeline: runRollbackPipeline, list_component_versions: listComponentVersions, } satisfies ToolHandlers;
- src/circleci-tools.ts:37-54 (registration)Includes the getFlakyTestLogsTool in the exported CCI_TOOLS array for tool registration.export const CCI_TOOLS = [ getBuildFailureLogsTool, getFlakyTestLogsTool, getLatestPipelineStatusTool, getJobTestResultsTool, configHelperTool, createPromptTemplateTool, recommendPromptTemplateTestsTool, runPipelineTool, listFollowedProjectsTool, runEvaluationTestsTool, rerunWorkflowTool, downloadUsageApiDataTool, findUnderusedResourceClassesTool, analyzeDiffTool, runRollbackPipelineTool, listComponentVersionsTool, ];
- Core helper function that fetches flaky tests from CircleCI API using insights.getProjectFlakyTests and retrieves detailed test data from jobs.const getFlakyTests = async ({ projectSlug }: { projectSlug: string }) => { const circleci = getCircleCIClient(); const flakyTests = await circleci.insights.getProjectFlakyTests({ projectSlug, }); if (!flakyTests || !flakyTests.flaky_tests) { throw new Error('Flaky tests not found'); } const flakyTestDetails = [ ...new Set( flakyTests.flaky_tests.map((test) => ({ jobNumber: test.job_number, test_name: test.test_name, })), ), ]; const testsArrays = await rateLimitedRequests( flakyTestDetails.map(({ jobNumber, test_name }) => async () => { try { const tests = await circleci.tests.getJobTests({ projectSlug, jobNumber, }); const matchingTest = tests.find((test) => test.name === test_name); if (matchingTest) { return matchingTest; } console.error(`Test ${test_name} not found in job ${jobNumber}`); return tests.filter((test) => test.result === 'failure'); } catch (error) { if (error instanceof Error && error.message.includes('404')) { console.error(`Job ${jobNumber} not found:`, error); return undefined; } else if (error instanceof Error && error.message.includes('429')) { console.error(`Rate limited for job request ${jobNumber}:`, error); return undefined; } throw error; } }), ); const filteredTestsArrays = testsArrays .flat() .filter((test) => test !== undefined); return filteredTestsArrays; };