DevOps AI Toolkit

# PRD: AI Provider Performance Comparison & Benchmarking **Status**: Draft **Priority**: Medium **GitHub Issue**: [#139](https://github.com/vfarcic/dot-ai/issues/139) **Created**: 2025-10-04 **Dependencies**: - ✅ [PRD 136: Tool-Based Agentic Architecture](./done/136-tool-based-agentic-architecture.md) - **COMPLETE** (2025-10-05) - ⏳ [PRD 137: OpenTelemetry Tracing](./137-opentelemetry-tracing.md) - **PENDING** ## Executive Summary The DevOps AI Toolkit supports multiple AI providers (Anthropic Claude, OpenAI GPT, Google Gemini) but lacks comprehensive performance and cost comparison data to help users choose the optimal provider for their use case. This PRD outlines the implementation of an automated benchmarking system that captures detailed metrics and generates data-driven comparison reports. ### Problem Statement - **No performance data**: Users don't know which provider is fastest for their workload - **Unknown cost implications**: Actual token usage and costs are not measured - **Quality uncertainty**: No systematic comparison of AI response quality across providers - **Blind provider selection**: Users choose based on preference rather than data - **No reliability metrics**: Test failure rates and error patterns not tracked per provider ### Solution Overview Build an automated benchmarking framework that: 1. Captures comprehensive metrics during integration test execution 2. Stores benchmark results in structured format in the repository 3. Generates markdown comparison reports with performance, cost, and quality data 4. Enables on-demand benchmarking across all supported providers ## Technical Architecture ### Current State Analysis **Existing Capabilities**: - Multi-provider support via `AIProvider` interface (PRD 73) - Integration test suite covering all MCP tools (44 tests) - Basic test timing data in logs - Session files with AI responses **Missing Capabilities**: - Token usage tracking (depends on PRD 137: OpenTelemetry) - Structured benchmark data storage - Automated comparison report generation - Quality assessment metrics - Cost calculations ### Target Architecture ``` benchmarks/ ├── raw-data/ # Raw benchmark results │ ├── anthropic/ │ │ ├── 2025-10-04-run-001.json │ │ └── 2025-10-04-run-002.json │ ├── openai/ │ └── google/ ├── comparisons/ # Generated comparison reports │ ├── latest.md │ ├── 2025-10-04.md │ └── archive/ └── scripts/ ├── run-benchmarks.sh # Execute benchmark suite └── generate-report.ts # Create comparison markdown ``` ### Data Collection Points **From OpenTelemetry (PRD 137)**: - Token counts (input/output per AI call) - Response latency per provider - Request success/failure rates - Retry attempts and fallback usage **From Integration Tests**: - Test execution time per provider - Test pass/fail rates - Error patterns and messages - Resource usage (memory, CPU) **From Session Files**: - AI response samples for quality comparison - Remediation analysis accuracy - Manifest generation quality - Capability inference precision ### Benchmark Data Schema ```typescript interface BenchmarkResult { metadata: { timestamp: string; provider: string; model: string; gitCommit: string; testSuite: string; }; performance: { totalDuration: number; testResults: Array<{ testName: string; duration: number; passed: boolean; error?: string; }>; avgResponseTime: number; p50ResponseTime: number; p95ResponseTime: number; }; tokenUsage: { totalInputTokens: number; totalOutputTokens: number; avgInputTokensPerCall: number; avgOutputTokensPerCall: number; byTool: Record<string, TokenStats>; }; cost: { inputCost: number; outputCost: number; totalCost: number; pricingDate: string; }; reliability: { totalTests: number; passed: number; failed: number; failuresByType: Record<string, number>; }; quality: { sampleResponses: Array<QualitySample>; accuracyMetrics?: AccuracyMetrics; }; } ``` ## Implementation Plan ### Major Milestones #### ✅ Milestone 1: Benchmark Infrastructure **✅ Foundation Already in Place** (from PRD 136): - ✅ Basic metrics infrastructure (`src/core/providers/provider-debug-utils.ts`) - ✅ JSONL logging with `DEBUG_DOT_AI=true` flag - ✅ Token usage tracking (input/output tokens per operation) - ✅ Duration tracking per AI call - ✅ Operation identifiers for categorization - ✅ Provider identification in metrics **Still To Do**: - [ ] Create `benchmarks/` directory structure (raw-data/, comparisons/, scripts/) - [ ] Design comprehensive benchmark result schema (extend existing metrics) - [ ] Enhance data collection to include: - [ ] Test pass/fail results - [ ] Error patterns and messages - [ ] Response time percentiles (p50, p95) - [ ] Per-test breakdown (not just per-operation) - [ ] Add cost calculations using provider pricing - [ ] Add benchmark execution script that: - [ ] Wraps integration tests - [ ] Organizes results by provider - [ ] Generates structured JSON files from JSONL logs **Success Criteria**: - Integration tests save structured benchmark data (✅ basic data already captured) - Each test run produces complete JSON result file (⏳ needs enhancement) - Data includes all metrics from schema: performance (✅ partial), tokens (✅ complete), cost (⏳ needs addition), reliability (⏳ needs addition) - Multiple providers can be benchmarked sequentially (✅ infrastructure supports) **Estimated Duration**: 1-2 days (reduced from original due to foundation from PRD 136) --- #### ✅ Milestone 2: Token Usage Integration - [ ] Integrate with OpenTelemetry tracing (PRD 137) - [ ] Extract token counts from trace spans - [ ] Calculate per-tool token usage statistics - [ ] Add cost calculations using current provider pricing - [ ] Validate token accuracy against provider dashboards **Success Criteria**: - Token counts match provider billing data (±5%) - Cost calculations reflect actual pricing - Per-tool breakdown available for optimization - Historical token trends tracked over time **Blocked By**: PRD 137 (OpenTelemetry Tracing) completion --- #### ✅ Milestone 3: Quality Sampling System - [ ] Extract representative AI responses from session files - [ ] Define quality assessment criteria per tool type - [ ] Capture side-by-side responses for same prompts - [ ] Store quality samples with benchmark results - [ ] Add manual quality rating interface (optional) **Success Criteria**: - Key scenarios have sample responses from all providers - Remediate OOM analysis samples captured - Capability inference examples included - Manifest generation quality preserved - Responses easily comparable side-by-side **Estimated Duration**: 2-3 days --- #### ✅ Milestone 4: Comparison Report Generation - [ ] Build markdown report generator script - [ ] Create report template with structured sections - [ ] Generate performance comparison tables - [ ] Include cost analysis and ROI calculations - [ ] Add quality comparison with code samples - [ ] Generate recommendations based on use case **Success Criteria**: - Reports are clear, actionable, and data-driven - Performance tables show timing across all providers - Cost analysis includes per-test and per-tool breakdowns - Quality section has side-by-side AI responses - Recommendations help users choose optimal provider - Reports stored in `benchmarks/comparisons/` **Estimated Duration**: 2-3 days --- #### ✅ Milestone 5: Automated Benchmark Execution - [ ] Create `make benchmark` command - [ ] Add provider selection flags (`--provider=anthropic,openai,google`) - [ ] Implement clean environment setup per run - [ ] Add report generation post-execution - [ ] Update CI/CD to support benchmark runs - [ ] Document benchmark execution workflow **Success Criteria**: - Single command runs full benchmark suite - Clean Kubernetes cluster state per provider - Benchmark results automatically saved and committed - Latest comparison report generated and updated - CI can run benchmarks on-demand (manual trigger) - Clear documentation in README **Estimated Duration**: 1-2 days --- #### ✅ Milestone 6: Documentation & Public Release - [ ] Create comprehensive benchmarking guide - [ ] Document how to interpret comparison reports - [ ] Add provider selection guidelines - [ ] Include cost optimization recommendations - [ ] Publish initial benchmark results - [ ] Announce benchmarking capability to users **Success Criteria**: - Users understand how to run benchmarks - Comparison reports are easy to interpret - Provider selection guidance is clear - Public benchmark results demonstrate value - Community can contribute benchmark runs - Documentation complete and tested **Estimated Duration**: 1-2 days --- ## Success Metrics ### Technical Metrics - **Data Completeness**: 100% of required metrics captured (performance, tokens, cost, quality) - **Accuracy**: Token counts within 5% of provider billing data - **Automation**: Full benchmark suite runs with single command - **Reproducibility**: Consistent results across multiple runs (±10%) ### User Experience Metrics - **Execution Time**: Complete 3-provider benchmark suite in <30 minutes - **Report Clarity**: Users can choose optimal provider after reading report - **Cost Transparency**: Clear understanding of pricing implications - **Quality Insight**: Users can evaluate AI response quality differences ### Business Impact - **Provider Optimization**: Users save 20-40% on AI costs by choosing optimal provider - **Informed Decisions**: 90%+ of users benchmark before production deployment - **Community Value**: Public benchmarks attract users evaluating the project ## Risk Assessment ### High Priority Risks 1. **Token Tracking Dependency**: OpenTelemetry implementation delays block cost analysis - **Mitigation**: Implement basic benchmarking without token data first, add cost later - **Fallback**: Manual token extraction from provider dashboards for initial reports 2. **Provider API Instability**: Rate limits or API changes during benchmarking - **Mitigation**: Add retry logic, rate limiting, and graceful degradation - **Detection**: Monitor for anomalous results, validate against historical data 3. **Test Environment Variability**: Network latency, cluster state affects consistency - **Mitigation**: Use dedicated test cluster, run multiple iterations, report confidence intervals - **Validation**: Statistical analysis to detect outliers ### Medium Priority Risks 1. **Pricing Changes**: Provider pricing updates make cost data stale - **Mitigation**: Include pricing date in reports, automated pricing update checks - **Monitoring**: Monthly pricing validation against provider websites 2. **Quality Subjectivity**: Hard to objectively measure AI response quality - **Mitigation**: Focus on measurable criteria (correctness, completeness, format) - **Enhancement**: Add manual review workflow for complex quality assessments 3. **Storage Growth**: Benchmark data accumulates over time - **Mitigation**: Archive old results, compress historical data, retention policy - **Threshold**: Alert when benchmarks/ directory exceeds 100MB ## Documentation Impact ### Files to Create - `docs/benchmarking-guide.md` - Complete benchmarking workflow guide - `benchmarks/README.md` - Overview of benchmark system and how to interpret results - `benchmarks/comparisons/latest.md` - Most recent provider comparison report ### Files to Update - `README.md` - Add benchmarking section to features - `docs/ai-providers-guide.md` - Link to benchmark results for provider selection - `CONTRIBUTING.md` - Guidelines for contributing benchmark runs ## Future Considerations ### Phase 2 Enhancements - **Real-time Dashboards**: Live benchmark visualization (Grafana/Prometheus) - **Continuous Benchmarking**: Automated runs on every release - **Model Variants**: Compare different models within same provider (GPT-4o vs GPT-4-turbo) - **Workload Profiles**: Specialized benchmarks for different use cases (simple vs complex) - **Community Benchmarks**: Accept and aggregate benchmark runs from community - **A/B Testing Framework**: Compare provider performance in production ### Integration Opportunities - **Cost Optimizer**: Automatic provider switching based on cost thresholds - **Smart Routing**: Route queries to optimal provider based on workload type - **Performance SLOs**: Alert when provider performance degrades beyond thresholds ## Decision Log ### ✅ Decision: Public Benchmark Results - **Date**: 2025-10-04 - **Decision**: Store benchmark results in public GitHub repository - **Rationale**: - Transparency builds trust with users - Helps potential users evaluate the project - Enables community contributions and validation - No sensitive data in benchmark results - **Impact**: All benchmark data and reports visible in `benchmarks/` directory - **Owner**: Project Maintainers ### ✅ Decision: Static Reports Over Real-time Dashboards - **Date**: 2025-10-04 - **Decision**: Generate markdown reports instead of live dashboards (Phase 1) - **Rationale**: - Simpler implementation, faster time to value - Works with existing documentation workflow - No infrastructure dependencies (no Grafana/Prometheus) - Easier for users to consume (just read markdown) - Dashboards can be added later if demand exists - **Impact**: Milestone 4 focuses on markdown generation, dashboard moved to Phase 2 - **Owner**: Development Team ### ✅ Decision: On-Demand Execution Model - **Date**: 2025-10-04 - **Decision**: Benchmarks run on-demand via `make benchmark`, not automatically on every commit - **Rationale**: - Benchmarks take 20-30 minutes to complete - Expensive (AI API costs for 3 providers) - Most commits don't affect AI provider performance - Manual trigger gives control over when to incur costs - Can be run before releases or when evaluating provider changes - **Impact**: No CI automation in Milestone 5, just manual execution capability - **Owner**: Development Team ## Work Log ### 2025-10-04: PRD Created **Duration**: ~1 hour **Phase**: Planning and Requirements **Activities**: - Analyzed conversation about AI provider comparison needs - Identified dependencies (PRD 136, PRD 137) - Created GitHub issue #139 - Designed benchmark data schema - Defined 6 major milestones with clear success criteria - Documented risks and mitigation strategies **Key Decisions**: - Public benchmark results for transparency - Static reports initially, dashboards in Phase 2 - On-demand execution to control costs **Next Steps**: - Wait for PRD 136 (Tool-Based Architecture) completion - Wait for PRD 137 (OpenTelemetry Tracing) completion - Begin Milestone 1: Benchmark Infrastructure **Current Status**: Draft - Awaiting dependencies --- ### 2025-10-05: PRD 136 Dependency Completed **Duration**: ~15 minutes **Phase**: Dependency Update **Activities**: - Reviewed completed PRD 136 (Tool-Based Agentic Architecture) - Analyzed metrics infrastructure added in PRD 136 - Updated dependencies to mark PRD 136 as ✅ COMPLETE - Updated Milestone 1 to reflect existing foundation **Key Findings** (from PRD 136): - ✅ Metrics infrastructure already in place (`provider-debug-utils.ts`) - ✅ Token usage tracking (input/output) functional - ✅ Duration tracking per AI call - ✅ Operation identifiers for categorization - ✅ JSONL logging with `DEBUG_DOT_AI=true` - ✅ Provider identification in metrics **Impact on Milestone 1**: - Foundation already built - basic metrics collection working - Reduces Milestone 1 scope to enhancement vs. building from scratch - Estimated duration remains 1-2 days (but with less risk) - Can leverage existing JSONL format and debug utilities **Next Steps**: - Wait for PRD 137 (OpenTelemetry Tracing) completion for advanced token metrics - Can begin Milestone 1 work now using existing metrics foundation - Design comprehensive schema building on existing metrics - Add cost calculations, reliability metrics, quality sampling **Current Status**: Draft - One dependency complete (PRD 136), awaiting PRD 137 --- **Last Updated**: 2025-10-05 **Next Review**: After PRD 137 (OpenTelemetry Tracing) is complete **Stakeholders**: DevOps AI Toolkit Users, Contributors, AI Provider Evaluators