DevOps AI Toolkit

# PRD: AI Model Comparison for MCP Platform **Created**: 2025-10-07 **Status**: Draft **Owner**: Viktor Farcic **Last Updated**: 2025-10-07 **GitHub Issue**: [#151](https://github.com/vfarcic/dot-ai/issues/151) **Priority**: Medium **Complexity**: Medium --- ## Executive Summary Test and compare 6 AI models across ALL MCP platform AI interactions using the existing integration test suite as the data generation method. This research will provide comprehensive performance analysis, reliability assessment, and cost-benefit recommendations based on real-world usage patterns across all MCP tools. **Methodology**: Run complete integration test suite with each AI model to generate comprehensive comparison data across all AI-powered features. **Builds on**: [PRD #143 - Tool-Based Remediation Observability](https://github.com/vfarcic/dot-ai/blob/main/prds/143-tool-based-remediation-observability.md) (Extended metrics infrastructure) --- ## Problem Statement ### Current Limitations **1. Limited Model Testing Scope** - Only remediation tool tested with multiple models (Claude, GPT-5, Gemini) - No testing across other AI-powered MCP tools: recommend, answer-question, generate-manifests - Missing coverage of complete user workflows and interactions **2. Incomplete Market Coverage** - Premium segment: Claude, GPT-5 tested for remediation only - Mid-tier segment: Missing (Mistral Large, xAI Grok-4) - Budget segment: Missing (DeepSeek-R1) - No cross-tool consistency analysis **3. Limited Real-World Validation** - Testing focused on single tool (remediation) - No validation of model performance across diverse AI interaction types - Missing data on tool-specific model strengths/weaknesses ### Impact Users must: - Make AI model decisions without comprehensive platform-wide comparison data - Risk choosing models that work well for some tools but poorly for others - Cannot optimize for overall platform cost/performance - Miss opportunities to use different models for different tool types --- ## Solution Overview **Integration Test-Based Model Comparison**: ### Testing Strategy Use existing integration test suite as comprehensive data generation method: - Tests ALL AI-powered MCP tools in realistic scenarios - Captures real user workflows and interaction patterns - Provides consistent testing methodology across tools - Generates extensive metrics across all AI interactions ### Model Selection (6 Total) **Premium Tier (Established Leaders)**: - ✅ Claude Sonnet 4.5 - Current platform recommendation - ✅ OpenAI GPT-5 - High-cost reference standard **Mid-Tier (Regional Champions & Challengers)**: - ⏳ xAI Grok-4 - Emerging challenger (Elon Musk's model) - ⏳ Mistral Large - European champion, cost-effective **Budget Tier (Disruptors)**: - ⏳ DeepSeek-R1 - Ultra-low cost, reasoning-focused **Problematic (Case Study)**: - ✅ Google Gemini 2.5 Pro - Reliability issues documented (remediation) ### MCP Tools Coverage All AI-powered tools will be tested: - **recommend** - Intent analysis and solution recommendations - **remediate** - Investigation and troubleshooting (already tested) - **answer-question** - Solution configuration and enhancement - **generate-manifests** - Kubernetes manifest generation - **version** - AI provider connectivity testing - **platform discovery** - Build platform operations discovery --- ## User Journey ### Before (Current State) ``` 1. User wants to choose AI model for MCP platform 2. Limited to remediation-specific testing data 3. No understanding of cross-tool performance trade-offs 4. May choose model that works poorly for their primary use cases 5. No platform-wide cost optimization guidance ``` ### After (With Comprehensive Testing) ``` 1. User reviews model comparison guide with ALL MCP tools tested 2. Sees performance across their specific tool usage patterns 3. Identifies optimal model for their primary workflows 4. Understands cost implications across all interactions 5. Makes informed decision with confidence in overall platform performance 6. Has fallback recommendations for different tool combinations ``` --- ## Technical Approach ### Testing Infrastructure (Leverage Existing) - ✅ Integration test suite covering all MCP tools - ✅ Extended metrics collection system (Decision 5 from PRD #143) - ✅ Vercel AI SDK provider supporting multiple models - ✅ Debug infrastructure for conversation analysis - ✅ Metrics storage and analysis pipeline ### Integration Test Inventory Based on codebase analysis, AI-powered tests include: **Core AI Tools**: - `tests/integration/tools/recommend.test.ts` - Intent analysis and recommendations - `tests/integration/tools/remediate.test.ts` - Investigation and troubleshooting - `tests/integration/tools/answer-question.test.ts` - Configuration enhancement - `tests/integration/tools/generate-manifests.test.ts` - Manifest generation - `tests/integration/tools/version.test.ts` - AI provider connectivity **Platform Operations**: - Tests involving build platform discovery - Solution selection and configuration workflows - Cross-tool integration scenarios ### Model Integration Requirements **xAI Grok-4**: - Package: `@ai-sdk/xai` (needs installation) - Integration: Add to `PROVIDER_MODELS` in `vercel-provider.ts` - API Key: `XAI_API_KEY` environment variable **Mistral Large**: - Package: `@ai-sdk/mistral` (needs installation) - Integration: Add to `PROVIDER_MODELS` in `vercel-provider.ts` - API Key: `MISTRAL_API_KEY` environment variable **DeepSeek-R1**: - Available through OpenAI-compatible API - Integration: Use existing OpenAI provider with custom endpoint - Model: `deepseek-reasoner` or `deepseek-r1` ### Testing Protocol **Phase 1: Test Suite Analysis** 1. Inventory all integration tests using AI interactions 2. Identify test coverage gaps (if any) 3. Ensure tests capture diverse AI interaction patterns 4. Validate extended metrics collection across all tests **Phase 2: Model Infrastructure Setup** 1. Install required SDK packages for new models 2. Extend provider configuration for all 6 models 3. Set up API keys and environment configuration 4. Validate basic connectivity for each model **Phase 3: Comprehensive Testing** 1. Run COMPLETE integration test suite with each of 6 models 2. Capture extended metrics for every AI interaction 3. Document failure modes and edge cases per tool 4. Save comprehensive baseline data for analysis **Phase 4: Analysis & Recommendations** 1. Analyze metrics across all 6 models and all tools 2. Identify tool-specific model performance patterns 3. Create decision matrix for different use case priorities 4. Document platform-wide recommendations and trade-offs --- ## Success Criteria ### Testing Completion - [ ] All 6 models integrated and tested across complete integration suite - [ ] Extended metrics captured for every AI interaction across all tools - [ ] Reliability assessment completed for each model-tool combination - [ ] Performance comparison analysis across all MCP AI features ### Analysis & Documentation - [ ] Model comparison guide covering all MCP AI interactions - [ ] Tool-specific model recommendations (if patterns emerge) - [ ] Platform-wide cost optimization guide - [ ] Implementation instructions for all supported models ### Quality Assurance - [ ] All model configurations validated across all AI tools - [ ] Test results reproducible and comprehensive - [ ] Failure modes documented per model-tool combination - [ ] Cost estimates for different usage patterns --- ## Milestones ### Milestone 1: Test Suite Analysis & Infrastructure Setup **Goal**: Prepare comprehensive testing infrastructure **Tasks**: - [ ] Inventory all integration tests using AI interactions - [ ] Install required AI SDK packages (`@ai-sdk/xai`, `@ai-sdk/mistral`) - [ ] Extend `vercel-provider.ts` to support 3 additional models - [ ] Set up API keys and environment configuration - [ ] Validate basic connectivity for each model across all tools **Success Criteria**: All 6 models can be used with complete integration test suite ### Milestone 2: Complete Model Testing (All 6 Models × All AI Tools) **Goal**: Generate comprehensive dataset across all AI interactions **Tasks**: - [ ] Run complete integration test suite with Claude Sonnet 4.5 (baseline) - [ ] Run complete integration test suite with OpenAI GPT-5 - [ ] Run complete integration test suite with Google Gemini 2.5 Pro - [ ] Run complete integration test suite with xAI Grok-4 - [ ] Run complete integration test suite with Mistral Large - [ ] Run complete integration test suite with DeepSeek-R1 - [ ] Capture and store extended metrics for all test runs **Success Criteria**: Complete metrics dataset for 6 models across all AI-powered MCP tools ### Milestone 3: Comprehensive Analysis **Goal**: Extract insights from complete testing data **Tasks**: - [ ] Analyze performance patterns across all model-tool combinations - [ ] Identify tool-specific model strengths and weaknesses - [ ] Calculate cost implications for different usage patterns - [ ] Document reliability and consistency patterns - [ ] Create decision framework for model selection **Success Criteria**: Clear insights into optimal model choices for different use cases ### Milestone 4: Documentation & Recommendations **Goal**: Create user-facing guidance based on comprehensive testing **Tasks**: - [ ] Create platform-wide model selection guide - [ ] Document tool-specific recommendations (if applicable) - [ ] Update MCP documentation with model configuration options - [ ] Publish comprehensive comparison analysis - [ ] Archive baseline data for future reference **Success Criteria**: Public documentation with actionable model selection guidance --- ## Dependencies ### External Dependencies - [ ] API access to xAI Grok-4 (API key required) - [ ] API access to Mistral Large (API key required) - [ ] API access to DeepSeek-R1 (API key required) - [ ] NPM packages: `@ai-sdk/xai`, `@ai-sdk/mistral` ### Internal Dependencies - [x] Extended metrics collection system (Decision 5, PRD #143) ✅ - [x] Vercel AI SDK provider (Milestone 2.5, PRD #143) ✅ - [x] Complete integration test suite ✅ - [x] Debug infrastructure and metrics storage ✅ ### Potential Blockers - API rate limits across extensive testing - Cost constraints for running complete suite 6 times - Model availability or API changes during testing --- ## Risks & Mitigations | Risk | Impact | Mitigation | |------|--------|-----------| | Extensive testing exceeds API cost budgets | High | Monitor usage, implement test batching, use smaller test subsets for validation | | New models fail consistently across tools | Medium | Document failures, focus on working models, treat failures as valuable data | | Test suite runtime becomes prohibitive | Medium | Parallel execution, selective test running, optimize test efficiency | | Model behavior inconsistent between test runs | Medium | Multiple runs for statistical significance, version pinning | | Integration test coverage insufficient | Low | Add missing tests if gaps identified, focus on existing comprehensive coverage | --- ## Open Questions 1. **Test Execution Strategy**: Serial vs parallel model testing to manage costs and time? 2. **Statistical Significance**: How many test runs per model needed for reliable data? 3. **Cost Management**: What budget exists for comprehensive testing across 6 models? 4. **Tool-Specific Optimization**: Should we allow different models for different tools? 5. **Baseline Comparison**: Use current production model as baseline across all tests? --- ## Out of Scope ### Deferred to Future Work - [ ] Tool-specific model optimization - [ ] Automated model selection based on interaction type - [ ] Custom model endpoints beyond standard APIs - [ ] Integration with non-Vercel AI SDK providers ### Explicitly Not Included - Model fine-tuning or customization - Testing with different prompt strategies per model - Performance optimization for specific model-tool combinations - Real-time model switching during operations --- ## Decision Log *Decisions will be documented here as the PRD progresses* --- ## Work Log ### 2025-10-07: PRD Creation **Duration**: Initial planning session **Status**: Draft **Context**: - Started with remediation-specific model testing scope - Expanded to full MCP platform coverage based on user feedback - Integration test-based approach chosen for comprehensive real-world validation **Key Insight**: Using integration tests as data generation method provides: - Comprehensive coverage of ALL AI interactions - Realistic usage patterns and workflows - Consistent testing methodology - Extensive metrics across diverse tool types **Scope Defined**: - 6 models × Complete integration test suite = comprehensive comparison data - Focus on platform-wide recommendations rather than tool-specific optimization - Leverage existing testing infrastructure from PRD #143 **Next Steps**: Begin Milestone 1 - Test suite analysis and model infrastructure setup