find_flaky_tests

Instructions

This tool retrieves information about flaky tests in a CircleCI project. 

The agent receiving this output MUST analyze the flaky test data and implement appropriate fixes based on the specific issues identified.

CRITICAL REQUIREMENTS:
1. Truncation Handling (HIGHEST PRIORITY):
   - ALWAYS check for <MCPTruncationWarning> in the output
   - When present, you MUST start your response with:
     "WARNING: The logs have been truncated. Only showing the most recent entries. Earlier build failures may not be visible."
   - Only proceed with log analysis after acknowledging the truncation

Input options (EXACTLY ONE of these THREE options must be used):

Option 1 - Project Slug:
- projectSlug: The project slug obtained from listFollowedProjects tool (e.g., "gh/organization/project")

Option 2 - Direct URL (provide ONE of these):
- projectURL: The URL of the CircleCI project in any of these formats:
  * Project URL: https://app.circleci.com/pipelines/gh/organization/project
  * Pipeline URL: https://app.circleci.com/pipelines/gh/organization/project/123
  * Workflow URL: https://app.circleci.com/pipelines/gh/organization/project/123/workflows/abc-def
  * Job URL: https://app.circleci.com/pipelines/gh/organization/project/123/workflows/abc-def/jobs/xyz

Option 3 - Project Detection (ALL of these must be provided together):
- workspaceRoot: The absolute path to the workspace root
- gitRemoteURL: The URL of the git remote repository

Additional Requirements:
- Never call this tool with incomplete parameters
- If using Option 1, make sure to extract the projectSlug exactly as provided by listFollowedProjects
- If using Option 2, the URLs MUST be provided by the user - do not attempt to construct or guess URLs
- If using Option 3, BOTH parameters (workspaceRoot, gitRemoteURL) must be provided
- If none of the options can be fully satisfied, ask the user for the missing information before making the tool call

Implementation Reference

• src/tools/getFlakyTests/handler.ts:18-67 (handler)

  Primary handler function for the 'find_flaky_tests' tool. Processes input parameters to identify the CircleCI project slug, fetches flaky tests data, and returns either formatted text or writes detailed files if FILE_OUTPUT_DIRECTORY is set.

  export const getFlakyTestLogs: ToolCallback<{
    params: typeof getFlakyTestLogsInputSchema;
  }> = async (args) => {
    const {
      workspaceRoot,
      gitRemoteURL,
      projectURL,
      projectSlug: inputProjectSlug,
    } = args.params;
  
    let projectSlug: string | null | undefined;
  
    if (inputProjectSlug) {
      projectSlug = inputProjectSlug;
    } else if (projectURL) {
      projectSlug = getProjectSlugFromURL(projectURL);
    } else if (workspaceRoot && gitRemoteURL) {
      projectSlug = await identifyProjectSlug({
        gitRemoteURL,
      });
    } else {
      return mcpErrorOutput(
        'Missing required inputs. Please provide either: 1) projectSlug, 2) projectURL, or 3) workspaceRoot with gitRemoteURL.',
      );
    }
  
    if (!projectSlug) {
      return mcpErrorOutput(`
            Project not found. Ask the user to provide the inputs user can provide based on the tool description.
  
            Project slug: ${projectSlug}
            Git remote URL: ${gitRemoteURL}
            `);
    }
  
    const tests = await getFlakyTests({
      projectSlug,
    });
  
    if (process.env.FILE_OUTPUT_DIRECTORY) {
      try {
        return await writeTestsToFiles({ tests });
      } catch (error) {
        console.error(error);
        return formatFlakyTests(tests);
      }
    }
  
    return formatFlakyTests(tests);
  };

• src/tools/getFlakyTests/inputSchema.ts:4-32 (schema)

  Zod schema defining the input parameters for the tool: projectSlug, workspaceRoot, gitRemoteURL, or projectURL (exactly one method to identify the project).

  export const getFlakyTestLogsInputSchema = z.object({
    projectSlug: z.string().describe(projectSlugDescriptionNoBranch).optional(),
    workspaceRoot: z
      .string()
      .describe(
        'The absolute path to the root directory of your project workspace. ' +
          'This should be the top-level folder containing your source code, configuration files, and dependencies. ' +
          'For example: "/home/user/my-project" or "C:\\Users\\user\\my-project"',
      )
      .optional(),
    gitRemoteURL: z
      .string()
      .describe(
        'The URL of the remote git repository. This should be the URL of the repository that you cloned to your local workspace. ' +
          'For example: "https://github.com/user/my-project.git"',
      )
      .optional(),
    projectURL: z
      .string()
      .describe(
        'The URL of the CircleCI project. Can be any of these formats:\n' +
          '- Project URL: https://app.circleci.com/pipelines/gh/organization/project\n' +
          '- Project URL with branch: https://app.circleci.com/pipelines/gh/organization/project?branch=feature-branch\n' +
          '- Pipeline URL: https://app.circleci.com/pipelines/gh/organization/project/123\n' +
          '- Workflow URL: https://app.circleci.com/pipelines/gh/organization/project/123/workflows/abc-def\n' +
          '- Job URL: https://app.circleci.com/pipelines/gh/organization/project/123/workflows/abc-def/jobs/xyz',
      )
      .optional(),
  });

• src/tools/getFlakyTests/tool.ts:3-41 (registration)

  Tool definition object registering the tool with name 'find_flaky_tests', detailed usage instructions in description, and input schema reference. Included in the CCI_TOOLS array.

  export const getFlakyTestLogsTool = {
    name: 'find_flaky_tests' as const,
    description: `
      This tool retrieves information about flaky tests in a CircleCI project. 
      
      The agent receiving this output MUST analyze the flaky test data and implement appropriate fixes based on the specific issues identified.
  
      CRITICAL REQUIREMENTS:
      1. Truncation Handling (HIGHEST PRIORITY):
         - ALWAYS check for <MCPTruncationWarning> in the output
         - When present, you MUST start your response with:
           "WARNING: The logs have been truncated. Only showing the most recent entries. Earlier build failures may not be visible."
         - Only proceed with log analysis after acknowledging the truncation
  
      Input options (EXACTLY ONE of these THREE options must be used):
  
      Option 1 - Project Slug:
      - projectSlug: The project slug obtained from listFollowedProjects tool (e.g., "gh/organization/project")
  
      Option 2 - Direct URL (provide ONE of these):
      - projectURL: The URL of the CircleCI project in any of these formats:
        * Project URL: https://app.circleci.com/pipelines/gh/organization/project
        * Pipeline URL: https://app.circleci.com/pipelines/gh/organization/project/123
        * Workflow URL: https://app.circleci.com/pipelines/gh/organization/project/123/workflows/abc-def
        * Job URL: https://app.circleci.com/pipelines/gh/organization/project/123/workflows/abc-def/jobs/xyz
  
      Option 3 - Project Detection (ALL of these must be provided together):
      - workspaceRoot: The absolute path to the workspace root
      - gitRemoteURL: The URL of the git remote repository
  
      Additional Requirements:
      - Never call this tool with incomplete parameters
      - If using Option 1, make sure to extract the projectSlug exactly as provided by listFollowedProjects
      - If using Option 2, the URLs MUST be provided by the user - do not attempt to construct or guess URLs
      - If using Option 3, BOTH parameters (workspaceRoot, gitRemoteURL) must be provided
      - If none of the options can be fully satisfied, ask the user for the missing information before making the tool call
      `,
    inputSchema: getFlakyTestLogsInputSchema,
  };

• src/circleci-tools.ts:58-72 (registration)

  Central registration mapping tool names to their handler functions, specifically 'find_flaky_tests' to getFlakyTestLogs. The tool object is also imported and included in CCI_TOOLS array.

  export const CCI_HANDLERS = {
    get_build_failure_logs: getBuildFailureLogs,
    find_flaky_tests: getFlakyTestLogs,
    get_latest_pipeline_status: getLatestPipelineStatus,
    get_job_test_results: getJobTestResults,
    config_helper: configHelper,
    create_prompt_template: createPromptTemplate,
    recommend_prompt_template_tests: recommendPromptTemplateTests,
    run_pipeline: runPipeline,
    list_followed_projects: listFollowedProjects,
    run_evaluation_tests: runEvaluationTests,
    rerun_workflow: rerunWorkflow,
    analyze_diff: analyzeDiff,
    run_rollback_pipeline: runRollbackPipeline,
  } satisfies ToolHandlers;

• src/tools/getFlakyTests/handler.ts:121-182 (helper)

  Supporting helper that writes each flaky test to an individual file in the output directory, creates .gitignore, and returns a list of file paths for analysis.

  const writeTestsToFiles = async ({
    tests,
  }: {
    tests: Test[];
  }): Promise<{
    content: {
      type: 'text';
      text: string;
    }[];
  }> => {
    if (tests.length === 0) {
      return {
        content: [
          {
            type: 'text' as const,
            text: 'No flaky tests found - no files created',
          },
        ],
      };
    }
  
    const flakyTestsOutputDirectory = getFlakyTestsOutputDirectory();
  
    try {
      rmSync(flakyTestsOutputDirectory, { recursive: true, force: true });
      mkdirSync(flakyTestsOutputDirectory, { recursive: true });
  
      // Create .gitignore to ignore all files in this directory
      const gitignorePath = join(flakyTestsOutputDirectory, '.gitignore');
      const gitignoreContent = '# Ignore all flaky test output files\n*\n';
      writeFileSync(gitignorePath, gitignoreContent, 'utf8');
    } catch (error) {
      throw new Error(
        `Failed to create output directory: ${error instanceof Error ? error.message : String(error)}`,
      );
    }
  
    const filePaths: string[] = [];
  
    try {
      tests.forEach((test, index) => {
        const filename = generateSafeFilename({ test, index });
        const filePath = join(flakyTestsOutputDirectory, filename);
  
        writeTestToFile({ test, filePath, index });
        filePaths.push(filePath);
      });
  
      return {
        content: [
          {
            type: 'text' as const,
            text: `Found ${tests.length} flaky tests that need stabilization. Each file contains test failure data and metadata - analyze these reports to understand what's causing the flakiness, then locate and fix the actual test code.\n\nFlaky test reports:\n${filePaths.map((path) => `- ${path}`).join('\n')}\n\nFiles are located in: ${flakyTestsOutputDirectory}`,
          },
        ],
      };
    } catch (error) {
      return mcpErrorOutput(
        `Failed to write flaky test files: ${error instanceof Error ? error.message : String(error)}`,
      );
    }
  };