find_flaky_tests
Retrieve flaky test data in CircleCI projects to identify and address intermittent failures. Supports project slugs, direct URLs, or workspace detection for accurate results.
Instructions
This tool retrieves information about flaky tests in a CircleCI project.
The agent receiving this output MUST analyze the flaky test data and implement appropriate fixes based on the specific issues identified.
CRITICAL REQUIREMENTS:
1. Truncation Handling (HIGHEST PRIORITY):
- ALWAYS check for <MCPTruncationWarning> in the output
- When present, you MUST start your response with:
"WARNING: The logs have been truncated. Only showing the most recent entries. Earlier build failures may not be visible."
- Only proceed with log analysis after acknowledging the truncation
Input options (EXACTLY ONE of these THREE options must be used):
Option 1 - Project Slug:
- projectSlug: The project slug obtained from listFollowedProjects tool (e.g., "gh/organization/project")
Option 2 - Direct URL (provide ONE of these):
- projectURL: The URL of the CircleCI project in any of these formats:
* Project URL: https://app.circleci.com/pipelines/gh/organization/project
* Pipeline URL: https://app.circleci.com/pipelines/gh/organization/project/123
* Workflow URL: https://app.circleci.com/pipelines/gh/organization/project/123/workflows/abc-def
* Job URL: https://app.circleci.com/pipelines/gh/organization/project/123/workflows/abc-def/jobs/xyz
Option 3 - Project Detection (ALL of these must be provided together):
- workspaceRoot: The absolute path to the workspace root
- gitRemoteURL: The URL of the git remote repository
Additional Requirements:
- Never call this tool with incomplete parameters
- If using Option 1, make sure to extract the projectSlug exactly as provided by listFollowedProjects
- If using Option 2, the URLs MUST be provided by the user - do not attempt to construct or guess URLs
- If using Option 3, BOTH parameters (workspaceRoot, gitRemoteURL) must be provided
- If none of the options can be fully satisfied, ask the user for the missing information before making the tool call
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| params | No |
Input Schema (JSON Schema)
{
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"properties": {
"params": {
"additionalProperties": false,
"properties": {
"gitRemoteURL": {
"description": "The URL of the remote git repository. This should be the URL of the repository that you cloned to your local workspace. For example: \"https://github.com/user/my-project.git\"",
"type": "string"
},
"projectSlug": {
"description": "The project slug from listFollowedProjects tool (e.g., \"gh/organization/project\").",
"type": "string"
},
"projectURL": {
"description": "The URL of the CircleCI project. Can be any of these formats:\n- Project URL: https://app.circleci.com/pipelines/gh/organization/project\n- Project URL with branch: https://app.circleci.com/pipelines/gh/organization/project?branch=feature-branch\n- Pipeline URL: https://app.circleci.com/pipelines/gh/organization/project/123\n- Workflow URL: https://app.circleci.com/pipelines/gh/organization/project/123/workflows/abc-def\n- Job URL: https://app.circleci.com/pipelines/gh/organization/project/123/workflows/abc-def/jobs/xyz",
"type": "string"
},
"workspaceRoot": {
"description": "The absolute path to the root directory of your project workspace. This should be the top-level folder containing your source code, configuration files, and dependencies. For example: \"/home/user/my-project\" or \"C:\\Users\\user\\my-project\"",
"type": "string"
}
},
"type": "object"
}
},
"type": "object"
}
Implementation Reference
- src/tools/getFlakyTests/handler.ts:18-67 (handler)The primary handler function for the 'find_flaky_tests' tool. Resolves the CircleCI project slug from inputs, fetches flaky tests using the helper function, and either writes detailed reports to files or returns formatted text output.export const getFlakyTestLogs: ToolCallback<{ params: typeof getFlakyTestLogsInputSchema; }> = async (args) => { const { workspaceRoot, gitRemoteURL, projectURL, projectSlug: inputProjectSlug, } = args.params ?? {}; let projectSlug: string | null | undefined; if (inputProjectSlug) { projectSlug = inputProjectSlug; } else if (projectURL) { projectSlug = getProjectSlugFromURL(projectURL); } else if (workspaceRoot && gitRemoteURL) { projectSlug = await identifyProjectSlug({ gitRemoteURL, }); } else { return mcpErrorOutput( 'Missing required inputs. Please provide either: 1) projectSlug, 2) projectURL, or 3) workspaceRoot with gitRemoteURL.', ); } if (!projectSlug) { return mcpErrorOutput(` Project not found. Ask the user to provide the inputs user can provide based on the tool description. Project slug: ${projectSlug} Git remote URL: ${gitRemoteURL} `); } const tests = await getFlakyTests({ projectSlug, }); if (process.env.FILE_OUTPUT_DIRECTORY) { try { return await writeTestsToFiles({ tests }); } catch (error) { console.error(error); return formatFlakyTests(tests); } } return formatFlakyTests(tests); };
- Zod schema defining the input parameters for identifying the CircleCI project: projectSlug, workspaceRoot+gitRemoteURL, or projectURL.export const getFlakyTestLogsInputSchema = z.object({ projectSlug: z.string().describe(projectSlugDescriptionNoBranch).optional(), workspaceRoot: z .string() .describe( 'The absolute path to the root directory of your project workspace. ' + 'This should be the top-level folder containing your source code, configuration files, and dependencies. ' + 'For example: "/home/user/my-project" or "C:\\Users\\user\\my-project"', ) .optional(), gitRemoteURL: z .string() .describe( 'The URL of the remote git repository. This should be the URL of the repository that you cloned to your local workspace. ' + 'For example: "https://github.com/user/my-project.git"', ) .optional(), projectURL: z .string() .describe( 'The URL of the CircleCI project. Can be any of these formats:\n' + '- Project URL: https://app.circleci.com/pipelines/gh/organization/project\n' + '- Project URL with branch: https://app.circleci.com/pipelines/gh/organization/project?branch=feature-branch\n' + '- Pipeline URL: https://app.circleci.com/pipelines/gh/organization/project/123\n' + '- Workflow URL: https://app.circleci.com/pipelines/gh/organization/project/123/workflows/abc-def\n' + '- Job URL: https://app.circleci.com/pipelines/gh/organization/project/123/workflows/abc-def/jobs/xyz', ) .optional(), });
- src/circleci-tools.ts:37-54 (registration)The 'find_flaky_tests' tool (imported as getFlakyTestLogsTool) is registered in the main CCI_TOOLS array used by the MCP server.export const CCI_TOOLS = [ getBuildFailureLogsTool, getFlakyTestLogsTool, getLatestPipelineStatusTool, getJobTestResultsTool, configHelperTool, createPromptTemplateTool, recommendPromptTemplateTestsTool, runPipelineTool, listFollowedProjectsTool, runEvaluationTestsTool, rerunWorkflowTool, downloadUsageApiDataTool, findUnderusedResourceClassesTool, analyzeDiffTool, runRollbackPipelineTool, listComponentVersionsTool, ];
- src/index.ts:57-69 (registration)MCP server registers the tool using server.tool() in a loop over CCI_TOOLS, mapping to the corresponding handler.CCI_TOOLS.forEach((tool) => { const handler = CCI_HANDLERS[tool.name]; if (!handler) throw new Error(`Handler for tool ${tool.name} not found`); if (process.env.debug === 'true') { console.error(`[DEBUG] [Startup] Registering tool: ${tool.name}`); } server.tool( tool.name, tool.description, { params: tool.inputSchema.optional() }, handler as any, ); });
- Core helper function that queries CircleCI Insights API for flaky tests, then fetches detailed test data from relevant jobs using rate-limited requests.const getFlakyTests = async ({ projectSlug }: { projectSlug: string }) => { const circleci = getCircleCIClient(); const flakyTests = await circleci.insights.getProjectFlakyTests({ projectSlug, }); if (!flakyTests || !flakyTests.flaky_tests) { throw new Error('Flaky tests not found'); } const flakyTestDetails = [ ...new Set( flakyTests.flaky_tests.map((test) => ({ jobNumber: test.job_number, test_name: test.test_name, })), ), ]; const testsArrays = await rateLimitedRequests( flakyTestDetails.map(({ jobNumber, test_name }) => async () => { try { const tests = await circleci.tests.getJobTests({ projectSlug, jobNumber, }); const matchingTest = tests.find((test) => test.name === test_name); if (matchingTest) { return matchingTest; } console.error(`Test ${test_name} not found in job ${jobNumber}`); return tests.filter((test) => test.result === 'failure'); } catch (error) { if (error instanceof Error && error.message.includes('404')) { console.error(`Job ${jobNumber} not found:`, error); return undefined; } else if (error instanceof Error && error.message.includes('429')) { console.error(`Rate limited for job request ${jobNumber}:`, error); return undefined; } throw error; } }), ); const filteredTestsArrays = testsArrays .flat() .filter((test) => test !== undefined); return filteredTestsArrays; };