find_flaky_tests

Instructions

This tool retrieves information about flaky tests in a CircleCI project. 

The agent receiving this output MUST analyze the flaky test data and implement appropriate fixes based on the specific issues identified.

CRITICAL REQUIREMENTS:
1. Truncation Handling (HIGHEST PRIORITY):
   - ALWAYS check for <MCPTruncationWarning> in the output
   - When present, you MUST start your response with:
     "WARNING: The logs have been truncated. Only showing the most recent entries. Earlier build failures may not be visible."
   - Only proceed with log analysis after acknowledging the truncation

Input options (EXACTLY ONE of these THREE options must be used):

Option 1 - Project Slug:
- projectSlug: The project slug obtained from listFollowedProjects tool (e.g., "gh/organization/project")

Option 2 - Direct URL (provide ONE of these):
- projectURL: The URL of the CircleCI project in any of these formats:
  * Project URL: https://app.circleci.com/pipelines/gh/organization/project
  * Pipeline URL: https://app.circleci.com/pipelines/gh/organization/project/123
  * Workflow URL: https://app.circleci.com/pipelines/gh/organization/project/123/workflows/abc-def
  * Job URL: https://app.circleci.com/pipelines/gh/organization/project/123/workflows/abc-def/jobs/xyz

Option 3 - Project Detection (ALL of these must be provided together):
- workspaceRoot: The absolute path to the workspace root
- gitRemoteURL: The URL of the git remote repository

Additional Requirements:
- Never call this tool with incomplete parameters
- If using Option 1, make sure to extract the projectSlug exactly as provided by listFollowedProjects
- If using Option 2, the URLs MUST be provided by the user - do not attempt to construct or guess URLs
- If using Option 3, BOTH parameters (workspaceRoot, gitRemoteURL) must be provided
- If none of the options can be fully satisfied, ask the user for the missing information before making the tool call

Tool call template

This tool declares no required parameters. Add optional parameters from the input schema as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {},
    "name": "find_flaky_tests"
  }
}

Implementation Reference

• src/tools/getFlakyTests/handler.ts:18-67 (handler)

  Primary handler function for the 'find_flaky_tests' tool. Processes input parameters to identify the CircleCI project slug, fetches flaky tests data, and returns either formatted text or writes detailed files if FILE_OUTPUT_DIRECTORY is set.

  export const getFlakyTestLogs: ToolCallback<{
    params: typeof getFlakyTestLogsInputSchema;
  }> = async (args) => {
    const {
      workspaceRoot,
      gitRemoteURL,
      projectURL,
      projectSlug: inputProjectSlug,
    } = args.params;
  
    let projectSlug: string | null | undefined;
  
    if (inputProjectSlug) {
      projectSlug = inputProjectSlug;
    } else if (projectURL) {
      projectSlug = getProjectSlugFromURL(projectURL);
    } else if (workspaceRoot && gitRemoteURL) {
      projectSlug = await identifyProjectSlug({
        gitRemoteURL,
      });
    } else {
      return mcpErrorOutput(
        'Missing required inputs. Please provide either: 1) projectSlug, 2) projectURL, or 3) workspaceRoot with gitRemoteURL.',
      );
    }
  
    if (!projectSlug) {
      return mcpErrorOutput(`
            Project not found. Ask the user to provide the inputs user can provide based on the tool description.
  
            Project slug: ${projectSlug}
            Git remote URL: ${gitRemoteURL}
            `);
    }
  
    const tests = await getFlakyTests({
      projectSlug,
    });
  
    if (process.env.FILE_OUTPUT_DIRECTORY) {
      try {
        return await writeTestsToFiles({ tests });
      } catch (error) {
        console.error(error);
        return formatFlakyTests(tests);
      }
    }
  
    return formatFlakyTests(tests);
  };

• src/tools/getFlakyTests/inputSchema.ts:4-32 (schema)

  Zod schema defining the input parameters for the tool: projectSlug, workspaceRoot, gitRemoteURL, or projectURL (exactly one method to identify the project).

  export const getFlakyTestLogsInputSchema = z.object({
    projectSlug: z.string().describe(projectSlugDescriptionNoBranch).optional(),
    workspaceRoot: z
      .string()
      .describe(
        'The absolute path to the root directory of your project workspace. ' +
          'This should be the top-level folder containing your source code, configuration files, and dependencies. ' +
          'For example: "/home/user/my-project" or "C:\\Users\\user\\my-project"',
      )
      .optional(),
    gitRemoteURL: z
      .string()
      .describe(
        'The URL of the remote git repository. This should be the URL of the repository that you cloned to your local workspace. ' +
          'For example: "https://github.com/user/my-project.git"',
      )
      .optional(),
    projectURL: z
      .string()
      .describe(
        'The URL of the CircleCI project. Can be any of these formats:\n' +
          '- Project URL: https://app.circleci.com/pipelines/gh/organization/project\n' +
          '- Project URL with branch: https://app.circleci.com/pipelines/gh/organization/project?branch=feature-branch\n' +
          '- Pipeline URL: https://app.circleci.com/pipelines/gh/organization/project/123\n' +
          '- Workflow URL: https://app.circleci.com/pipelines/gh/organization/project/123/workflows/abc-def\n' +
          '- Job URL: https://app.circleci.com/pipelines/gh/organization/project/123/workflows/abc-def/jobs/xyz',
      )
      .optional(),
  });

• src/tools/getFlakyTests/tool.ts:3-41 (registration)

  Tool definition object registering the tool with name 'find_flaky_tests', detailed usage instructions in description, and input schema reference. Included in the CCI_TOOLS array.

  export const getFlakyTestLogsTool = {
    name: 'find_flaky_tests' as const,
    description: `
      This tool retrieves information about flaky tests in a CircleCI project. 
      
      The agent receiving this output MUST analyze the flaky test data and implement appropriate fixes based on the specific issues identified.
  
      CRITICAL REQUIREMENTS:
      1. Truncation Handling (HIGHEST PRIORITY):
         - ALWAYS check for <MCPTruncationWarning> in the output
         - When present, you MUST start your response with:
           "WARNING: The logs have been truncated. Only showing the most recent entries. Earlier build failures may not be visible."
         - Only proceed with log analysis after acknowledging the truncation
  
      Input options (EXACTLY ONE of these THREE options must be used):
  
      Option 1 - Project Slug:
      - projectSlug: The project slug obtained from listFollowedProjects tool (e.g., "gh/organization/project")
  
      Option 2 - Direct URL (provide ONE of these):
      - projectURL: The URL of the CircleCI project in any of these formats:
        * Project URL: https://app.circleci.com/pipelines/gh/organization/project
        * Pipeline URL: https://app.circleci.com/pipelines/gh/organization/project/123
        * Workflow URL: https://app.circleci.com/pipelines/gh/organization/project/123/workflows/abc-def
        * Job URL: https://app.circleci.com/pipelines/gh/organization/project/123/workflows/abc-def/jobs/xyz
  
      Option 3 - Project Detection (ALL of these must be provided together):
      - workspaceRoot: The absolute path to the workspace root
      - gitRemoteURL: The URL of the git remote repository
  
      Additional Requirements:
      - Never call this tool with incomplete parameters
      - If using Option 1, make sure to extract the projectSlug exactly as provided by listFollowedProjects
      - If using Option 2, the URLs MUST be provided by the user - do not attempt to construct or guess URLs
      - If using Option 3, BOTH parameters (workspaceRoot, gitRemoteURL) must be provided
      - If none of the options can be fully satisfied, ask the user for the missing information before making the tool call
      `,
    inputSchema: getFlakyTestLogsInputSchema,
  };

• src/circleci-tools.ts:58-72 (registration)

  Central registration mapping tool names to their handler functions, specifically 'find_flaky_tests' to getFlakyTestLogs. The tool object is also imported and included in CCI_TOOLS array.

  export const CCI_HANDLERS = {
    get_build_failure_logs: getBuildFailureLogs,
    find_flaky_tests: getFlakyTestLogs,
    get_latest_pipeline_status: getLatestPipelineStatus,
    get_job_test_results: getJobTestResults,
    config_helper: configHelper,
    create_prompt_template: createPromptTemplate,
    recommend_prompt_template_tests: recommendPromptTemplateTests,
    run_pipeline: runPipeline,
    list_followed_projects: listFollowedProjects,
    run_evaluation_tests: runEvaluationTests,
    rerun_workflow: rerunWorkflow,
    analyze_diff: analyzeDiff,
    run_rollback_pipeline: runRollbackPipeline,
  } satisfies ToolHandlers;

• src/tools/getFlakyTests/handler.ts:121-182 (helper)

  Supporting helper that writes each flaky test to an individual file in the output directory, creates .gitignore, and returns a list of file paths for analysis.

  const writeTestsToFiles = async ({
    tests,
  }: {
    tests: Test[];
  }): Promise<{
    content: {
      type: 'text';
      text: string;
    }[];
  }> => {
    if (tests.length === 0) {
      return {
        content: [
          {
            type: 'text' as const,
            text: 'No flaky tests found - no files created',
          },
        ],
      };
    }
  
    const flakyTestsOutputDirectory = getFlakyTestsOutputDirectory();
  
    try {
      rmSync(flakyTestsOutputDirectory, { recursive: true, force: true });
      mkdirSync(flakyTestsOutputDirectory, { recursive: true });
  
      // Create .gitignore to ignore all files in this directory
      const gitignorePath = join(flakyTestsOutputDirectory, '.gitignore');
      const gitignoreContent = '# Ignore all flaky test output files\n*\n';
      writeFileSync(gitignorePath, gitignoreContent, 'utf8');
    } catch (error) {
      throw new Error(
        `Failed to create output directory: ${error instanceof Error ? error.message : String(error)}`,
      );
    }
  
    const filePaths: string[] = [];
  
    try {
      tests.forEach((test, index) => {
        const filename = generateSafeFilename({ test, index });
        const filePath = join(flakyTestsOutputDirectory, filename);
  
        writeTestToFile({ test, filePath, index });
        filePaths.push(filePath);
      });
  
      return {
        content: [
          {
            type: 'text' as const,
            text: `Found ${tests.length} flaky tests that need stabilization. Each file contains test failure data and metadata - analyze these reports to understand what's causing the flakiness, then locate and fix the actual test code.\n\nFlaky test reports:\n${filePaths.map((path) => `- ${path}`).join('\n')}\n\nFiles are located in: ${flakyTestsOutputDirectory}`,
          },
        ],
      };
    } catch (error) {
      return mcpErrorOutput(
        `Failed to write flaky test files: ${error instanceof Error ? error.message : String(error)}`,
      );
    }
  };