run_evaluation_tests
Run evaluation tests on CircleCI pipelines by triggering new workflows. Generate configuration files, monitor progress via pipeline URLs, and manage test inputs with project slugs, direct URLs, or workspace detection.
Instructions
This tool allows the users to run evaluation tests on a circleci pipeline.
They can be referred to as "Prompt Tests" or "Evaluation Tests".
This tool triggers a new CircleCI pipeline and returns the URL to monitor its progress.
The tool will generate an appropriate circleci configuration file and trigger a pipeline using this temporary configuration.
The tool will return the project slug.
Input options (EXACTLY ONE of these THREE options must be used):
Option 1 - Project Slug and branch (BOTH required):
- projectSlug: The project slug obtained from listFollowedProjects tool (e.g., "gh/organization/project")
- branch: The name of the branch (required when using projectSlug)
Option 2 - Direct URL (provide ONE of these):
- projectURL: The URL of the CircleCI project in any of these formats:
* Project URL with branch: https://app.circleci.com/pipelines/gh/organization/project?branch=feature-branch
* Pipeline URL: https://app.circleci.com/pipelines/gh/organization/project/123
* Workflow URL: https://app.circleci.com/pipelines/gh/organization/project/123/workflows/abc-def
* Job URL: https://app.circleci.com/pipelines/gh/organization/project/123/workflows/abc-def/jobs/xyz
Option 3 - Project Detection (ALL of these must be provided together):
- workspaceRoot: The absolute path to the workspace root
- gitRemoteURL: The URL of the git remote repository
- branch: The name of the current branch
Test Files:
- promptFiles: Array of prompt template file objects from the ./prompts directory, each containing:
* fileName: The name of the prompt template file
* fileContent: The contents of the prompt template file
Pipeline Selection:
- If the project has multiple pipeline definitions, the tool will return a list of available pipelines
- You must then make another call with the chosen pipeline name using the pipelineChoiceName parameter
- The pipelineChoiceName must exactly match one of the pipeline names returned by the tool
- If the project has only one pipeline definition, pipelineChoiceName is not needed
Additional Requirements:
- Never call this tool with incomplete parameters
- If using Option 1, make sure to extract the projectSlug exactly as provided by listFollowedProjects
- If using Option 2, the URLs MUST be provided by the user - do not attempt to construct or guess URLs
- If using Option 3, ALL THREE parameters (workspaceRoot, gitRemoteURL, branch) must be provided
- If none of the options can be fully satisfied, ask the user for the missing information before making the tool call
Returns:
- A URL to the newly triggered pipeline that can be used to monitor its progress
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| params | No |
Input Schema (JSON Schema)
{
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"properties": {
"params": {
"additionalProperties": false,
"properties": {
"branch": {
"description": "The name of the branch currently checked out in local workspace. This should match local git branch. For example: \"feature/my-branch\", \"bugfix/123\", \"main\", \"master\" etc.",
"type": "string"
},
"gitRemoteURL": {
"description": "The URL of the remote git repository. This should be the URL of the repository that you cloned to your local workspace. For example: \"https://github.com/user/my-project.git\"",
"type": "string"
},
"pipelineChoiceName": {
"description": "The name of the pipeline to run. This parameter is only needed if the project has multiple pipeline definitions. If not provided and multiple pipelines exist, the tool will return a list of available pipelines for the user to choose from. If provided, it must exactly match one of the pipeline names returned by the tool.",
"type": "string"
},
"projectSlug": {
"description": "The project slug from listFollowedProjects tool (e.g., \"gh/organization/project\"). When using this option, branch must also be provided.",
"type": "string"
},
"projectURL": {
"description": "The URL of the CircleCI project. Can be any of these formats:\n- Project URL with branch: https://app.circleci.com/pipelines/gh/organization/project?branch=feature-branch\n- Pipeline URL: https://app.circleci.com/pipelines/gh/organization/project/123\n- Workflow URL: https://app.circleci.com/pipelines/gh/organization/project/123/workflows/abc-def\n- Job URL: https://app.circleci.com/pipelines/gh/organization/project/123/workflows/abc-def/jobs/xyz",
"type": "string"
},
"promptFiles": {
"description": "Array of prompt template files in the ./prompts directory (e.g. <relevant-name>.prompt.yml).",
"items": {
"additionalProperties": false,
"properties": {
"fileContent": {
"description": "The contents of the prompt template file",
"type": "string"
},
"fileName": {
"description": "The name of the prompt template file",
"type": "string"
}
},
"required": [
"fileName",
"fileContent"
],
"type": "object"
},
"type": "array"
},
"workspaceRoot": {
"description": "The absolute path to the root directory of your project workspace. This should be the top-level folder containing your source code, configuration files, and dependencies. For example: \"/home/user/my-project\" or \"C:\\Users\\user\\my-project\"",
"type": "string"
}
},
"required": [
"promptFiles"
],
"type": "object"
}
},
"type": "object"
}