gepa_evaluate_prompt

Instructions

Implementation Reference

• src/mcp/server.ts:832-919 (handler)

  The primary handler function that implements the core logic of the 'gepa_evaluate_prompt' MCP tool. It performs simulated evaluations across multiple tasks with configurable rollouts, computes aggregate performance metrics (success rate, average score, execution time), and integrates the results into the Pareto frontier for prompt optimization.

    private async evaluatePrompt(params: EvaluatePromptParams): Promise<{
      content: { type: string; text: string; }[];
    }> {
      const { promptId, taskIds, rolloutCount = 5, parallel = true } = params;
  
      // Validate required parameters
      if (!promptId || !taskIds || taskIds.length === 0) {
        throw new Error('promptId and taskIds are required');
      }
  
      const evaluationId = `eval_${Date.now()}_${Math.random().toString(36).substring(7)}`;
      const totalEvaluations = taskIds.length * rolloutCount;
  
      try {
        // Simulate prompt evaluation process
        const evaluationResults: any[] = [];
        
        for (const taskId of taskIds) {
          for (let rollout = 0; rollout < rolloutCount; rollout++) {
            const rolloutResult = {
              taskId,
              rollout: rollout + 1,
              success: Math.random() > 0.2, // 80% success rate simulation
              score: Math.random() * 0.5 + 0.5, // Score between 0.5-1.0
              executionTime: Math.random() * 2000 + 500, // 500-2500ms
              details: `Rollout ${rollout + 1} for task ${taskId}`,
            };
            evaluationResults.push(rolloutResult);
          }
        }
  
        // Calculate aggregate metrics
        const successfulEvaluations = evaluationResults.filter(r => r.success);
        const successRate = successfulEvaluations.length / totalEvaluations;
        const averageScore = successfulEvaluations.reduce((sum, r) => sum + r.score, 0) / successfulEvaluations.length;
        const averageExecutionTime = evaluationResults.reduce((sum, r) => sum + r.executionTime, 0) / evaluationResults.length;
  
        // Update prompt candidate in Pareto frontier
        const candidate: GEPAPromptCandidate = {
          id: promptId,
          content: '', // Will be retrieved if needed
          generation: 0,
          taskPerformance: new Map(taskIds.map(taskId => [taskId, averageScore])),
          averageScore: averageScore * successRate, // Combined fitness score
          rolloutCount: totalEvaluations,
          createdAt: new Date(),
          lastEvaluated: new Date(),
          mutationType: 'initial',
        };
  
        this.paretoFrontier.addCandidate(candidate);
  
        return {
          content: [
            {
              type: 'text',
              text: `# Prompt Evaluation Complete
  
  ## Evaluation Details
  - **Evaluation ID**: ${evaluationId}
  - **Prompt ID**: ${promptId}
  - **Tasks Evaluated**: ${taskIds.length}
  - **Rollouts per Task**: ${rolloutCount}
  - **Total Evaluations**: ${totalEvaluations}
  - **Parallel Execution**: ${parallel ? 'Yes' : 'No'}
  
  ## Performance Metrics
  - **Success Rate**: ${(successRate * 100).toFixed(1)}%
  - **Average Score**: ${averageScore.toFixed(3)}
  - **Combined Fitness**: ${(averageScore * successRate).toFixed(3)}
  - **Average Execution Time**: ${averageExecutionTime.toFixed(0)}ms
  
  ## Task Breakdown
  ${taskIds.map(taskId => {
    const taskResults = evaluationResults.filter(r => r.taskId === taskId);
    const taskSuccessRate = taskResults.filter(r => r.success).length / taskResults.length;
    const taskAvgScore = taskResults.filter(r => r.success).reduce((sum, r) => sum + r.score, 0) / taskResults.filter(r => r.success).length || 0;
    return `- **${taskId}**: ${(taskSuccessRate * 100).toFixed(1)}% success, ${taskAvgScore.toFixed(3)} avg score`;
  }).join('\n')}
  
  ✨ Candidate updated in Pareto frontier with performance metrics.`,
            },
          ],
        };
      } catch (error) {
        throw new Error(`Failed to evaluate prompt: ${error instanceof Error ? error.message : 'Unknown error'}`);
      }
    }

• src/mcp/server.ts:147-173 (schema)

  MCP input schema and metadata definition for the 'gepa_evaluate_prompt' tool, used for tool discovery (listTools) and input validation.

  name: 'gepa_evaluate_prompt',
  description: 'Evaluate prompt candidate performance across multiple tasks',
  inputSchema: {
    type: 'object',
    properties: {
      promptId: {
        type: 'string',
        description: 'Unique identifier for the prompt to evaluate'
      },
      taskIds: {
        type: 'array',
        items: { type: 'string' },
        description: 'List of task IDs to evaluate the prompt against'
      },
      rolloutCount: {
        type: 'number',
        default: 5,
        description: 'Number of evaluation rollouts per task'
      },
      parallel: {
        type: 'boolean',
        default: true,
        description: 'Whether to run evaluations in parallel'
      }
    },
    required: ['promptId', 'taskIds']
  }

• src/mcp/server.ts:532-534 (registration)

  Registration and dispatch logic in the central MCP tool request handler switch statement, mapping tool calls to the evaluatePrompt handler method.

  case 'gepa_evaluate_prompt':
    return await this.evaluatePrompt(args as unknown as EvaluatePromptParams);

• src/types/gepa.ts:251-256 (schema)

  TypeScript interface defining the typed parameters for the evaluatePrompt handler function.

  export interface EvaluatePromptParams {
    promptId: string;
    taskIds: string[];
    rolloutCount?: number;
    parallel?: boolean;
  }

• src/mcp/types.ts:267-267 (helper)

  Type-safe constant defining the tool name string for use in code references.

  EVALUATE_PROMPT: 'gepa_evaluate_prompt',