DevOps AI Toolkit

# PRD: Guided Setup System for DevOps AI Toolkit **Created**: 2025-10-15 **Status**: Planning - Ready to Start **Owner**: Viktor Farcic **Last Updated**: 2025-10-15 **GitHub Issue**: [#165](https://github.com/vfarcic/dot-ai/issues/165) **Priority**: High **Complexity**: Medium --- ## Executive Summary Create an interactive guided setup system that transforms the complex DevOps AI Toolkit configuration process into a simple question-based workflow. Users answer 4-5 questions about their needs and preferences, and the system generates all required configuration files (.env, .mcp.json, docker-compose.yml, setup instructions) with personalized recommendations. **Current Pain**: 9 AI models × 3 embedding providers × 5 deployment methods = overwhelming complexity requiring extensive documentation reading **Solution**: 5-minute guided setup with smart recommendations and automatic file generation **Impact**: Reduce time-to-first-success from 30+ minutes to under 5 minutes, eliminate configuration errors, improve user onboarding experience. --- ## Problem Statement ### Current Setup Complexity **1. Overwhelming Configuration Options** - **27 AI combinations**: 9 AI models × 3 embedding providers - **5 deployment methods**: Docker, Kubernetes, ToolHive, NPX, Development - **Multiple configuration files**: .env, .mcp.json, docker-compose.yml, kubeconfig - **Scattered documentation**: 7+ setup guides across different methods **2. High Cognitive Load** - Users must understand AI model trade-offs (quality vs cost vs speed) - Need to learn embedding provider differences and cost implications - Must choose deployment method without understanding trade-offs - Complex environment variable dependencies and API key management **3. Common Setup Failures** - Wrong AI model for user's primary use case (e.g., choosing GPT-5 Pro which has reliability issues) - Mismatched provider configurations (wrong API keys, incompatible setups) - Deployment method doesn't match user's environment or skill level - Missing or incorrect environment variables causing runtime failures **4. Documentation Fatigue** - Users abandon setup after reading extensive configuration options - Information overload prevents decision-making - Setup guides assume technical knowledge users may not have ### Impact on User Adoption **Friction Points**: - 30+ minutes average setup time with high failure rate - Users confused about which AI model to choose for their needs - Common misconfiguration leading to non-functional deployments - Documentation requires deep reading to understand trade-offs **Abandonment Scenarios**: - Users overwhelmed by choices in setup documentation - Configuration failures leading to "it doesn't work" perception - Unclear cost implications causing budget concerns - Complex setup process doesn't match "quick start" expectations --- ## Solution Overview ### Interactive Guided Setup System **Core Concept**: Transform configuration complexity into a simple conversation - **4-5 simple questions** instead of reading 7+ setup guides - **Smart recommendations** based on proven usage patterns and performance data - **Automatic file generation** with validated configurations - **Personalized cost estimates** and setup instructions ### Question-Based Configuration Flow **Step 1: Use Case Discovery** ``` 🎯 What do you want to use the DevOps AI Toolkit for? □ Kubernetes deployment recommendations □ Troubleshooting and debugging (remediation) □ Documentation testing and validation □ Shared prompts library only □ Everything (full platform capabilities) ``` **Step 2: Environment Assessment** ``` 🏗️ What's your preferred setup method? □ Docker (recommended - works everywhere) □ Kubernetes (I have a K8s cluster) □ NPX (I prefer Node.js tools) □ Development (I want to contribute code) ``` **Step 3: AI Model Preferences** ``` 💰 What's most important to you? □ Best quality (cost is not a concern) □ Best value (balance cost and performance) □ Lowest cost (budget-conscious) □ I have specific API credits to use 🤖 Any AI provider preferences? □ No preference (show me the best option) □ I prefer Anthropic (Claude) □ I prefer OpenAI (GPT) □ I prefer Google (Gemini) □ I want to try xAI (Grok) ``` ### Smart Recommendation Engine **Based on Performance Data**: Use real evaluation results from PRD-151 - Quality leaders: Claude Sonnet 4.5, Grok-4-Fast-Reasoning - Value leaders: Grok-4-Fast-Reasoning, Gemini 2.5 Flash - Budget options: Google models, Mistral Large - Avoid recommendations: GPT-5 Pro, Mistral Large (reliability issues) **Personalized Configuration Example**: ``` 📊 Your Recommended Configuration: For your needs (Full platform, Best value, No preference): ✅ AI Model: Grok-4-Fast-Reasoning → Excellent performance (0.765 score), cost-effective ($0.20/$0.50 per million tokens) ✅ Embedding: Google text-embedding-004 → 5x cheaper than OpenAI, good semantic search performance ✅ Deployment: Docker → All features working, no manual dependencies 💡 Why this setup? • Grok-4-Fast-Reasoning: Best value in our testing, reliable across all tools • Google embeddings: Significant cost savings without performance loss • Docker: Proven setup that works in all environments 💸 Estimated monthly cost: $2-8 for typical usage ``` ### Automatic File Generation **Generated Configuration Files**: 1. **`.env`** - Environment variables with correct API key names 2. **`.mcp.json`** - MCP client configuration for chosen deployment method 3. **`docker-compose.yml`** - If Docker deployment selected 4. **`setup-instructions.md`** - Personalized next steps 5. **`cost-estimate.md`** - Detailed cost breakdown for chosen configuration --- ## User Journey ### Before (Current State) ``` 1. User discovers DevOps AI Toolkit, wants to try it 2. Clicks "Quick Start" → sees 7 different setup methods 3. Reads extensive documentation trying to understand choices 4. Gets overwhelmed by AI model options (9 models × trade-offs) 5. Guesses at configuration, creates .env and .mcp.json manually 6. Setup fails due to wrong API keys or configuration errors 7. Spends 30+ minutes troubleshooting or gives up ``` ### After (With Guided Setup) ``` 1. User discovers DevOps AI Toolkit, wants to try it 2. Runs guided setup: "npx @vfarcic/dot-ai setup" 3. Answers 4-5 simple questions (2 minutes) 4. Reviews personalized recommendation with cost estimate 5. Confirms setup → all files generated automatically 6. Follows generated setup instructions (3 minutes) 7. Successfully using the toolkit within 5 minutes total ``` ### Success Flow Example ``` $ npx @vfarcic/dot-ai setup 🚀 DevOps AI Toolkit - Guided Setup Let's get you up and running in under 5 minutes! 🎯 What do you want to use the toolkit for? [1] Kubernetes deployments [2] Troubleshooting [3] Everything > 3 🏗️ Preferred setup method? [1] Docker (recommended) [2] Kubernetes [3] NPX > 1 💰 What matters most? [1] Best quality [2] Best value [3] Lowest cost > 2 📊 Perfect! Here's your recommended setup: AI Model: Grok-4-Fast-Reasoning (best value, 0.765 quality score) Embedding: Google text-embedding-004 (cost-effective) Deployment: Docker (all features included) Estimated cost: $2-8/month for typical usage ✅ Generate configuration files? (y/n) y Generated files: • .env (with your API key placeholders) • .mcp.json (Docker configuration) • docker-compose.yml (ready to run) • setup-instructions.md (your next steps) 🎉 Setup complete! Follow setup-instructions.md to finish. ``` --- ## Technical Approach ### Implementation Architecture **CLI Tool Structure**: ``` src/ ├── setup/ │ ├── guided-setup.ts # Main setup orchestrator │ ├── question-flow.ts # Interactive questionnaire │ ├── recommendation-engine.ts # AI model recommendations │ ├── file-generators/ # Configuration file generators │ │ ├── env-generator.ts │ │ ├── mcp-generator.ts │ │ └── docker-generator.ts │ └── validators/ # API key and setup validation └── cli/ └── setup-command.ts # CLI entry point ``` **Decision Tree Logic**: ```typescript interface SetupAnswers { useCase: 'deployment' | 'troubleshooting' | 'docs' | 'prompts' | 'full'; deployment: 'docker' | 'kubernetes' | 'npx' | 'development'; priority: 'quality' | 'value' | 'cost'; provider?: 'anthropic' | 'openai' | 'google' | 'xai' | 'none'; } interface RecommendedConfig { aiModel: string; aiProvider: string; embeddingModel: string; embeddingProvider: string; deployment: string; rationale: string; estimatedCost: string; } ``` **Recommendation Engine**: - Use performance data from PRD-151 AI model comparison - Apply cost-benefit analysis based on user priorities - Factor in reliability scores and failure modes - Consider provider-specific strengths for different use cases ### File Generation System **Template-Based Generation**: ```typescript // .env template const envTemplate = ` # AI Model Configuration AI_PROVIDER={{aiProvider}} {{aiApiKey}}={{apiKeyPlaceholder}} # Embedding Provider Configuration EMBEDDINGS_PROVIDER={{embeddingProvider}} {{embeddingApiKey}}={{embeddingKeyPlaceholder}} # Session Configuration DOT_AI_SESSION_DIR=./tmp/sessions # Generated by DevOps AI Toolkit Guided Setup on {{timestamp}} `; ``` **Dynamic MCP Configuration**: - Generate .mcp.json based on deployment method chosen - Include correct command and arguments for selected setup - Add environment variables specific to configuration **Setup Instructions Generator**: - Personalized next steps based on deployment method - API key acquisition instructions for chosen providers - Validation commands to test the setup - Troubleshooting section for common issues ### Validation and Error Handling **API Key Validation**: - Optional API key testing during setup - Provide clear instructions for obtaining API keys - Validate API key format and basic connectivity **Configuration Validation**: - Ensure generated files are syntactically correct - Validate environment variable names and values - Check for common configuration conflicts **Error Recovery**: - Allow users to re-run setup to change configurations - Preserve working configurations when updating - Clear error messages with specific resolution steps --- ## Success Criteria ### User Experience Metrics - [ ] Setup time reduced from 30+ minutes to under 5 minutes - [ ] Setup success rate improved to >95% (vs current ~70%) - [ ] User satisfaction: >90% rate setup as "easy" or "very easy" - [ ] Configuration errors reduced by >80% ### Feature Completeness - [ ] Interactive CLI tool supporting all major setup scenarios - [ ] Smart recommendations based on actual AI model performance data - [ ] Automatic generation of all required configuration files - [ ] Personalized cost estimates and setup instructions - [ ] Comprehensive error handling and validation ### Integration Success - [ ] Generated configurations work with all existing deployment methods - [ ] Backward compatibility with manual setup approaches - [ ] Integration with existing documentation and setup guides - [ ] Support for all current AI model and embedding provider combinations ### Adoption Metrics - [ ] 50%+ of new users use guided setup within 30 days of launch - [ ] Reduced support tickets related to setup and configuration issues - [ ] Improved user retention in first week after setup - [ ] Positive feedback from user community on setup experience --- ## Milestones ### Milestone 1: Core Question Flow & Recommendation Engine **Goal**: Interactive questionnaire with smart AI model recommendations **Tasks**: - [ ] Design and implement interactive question flow system - [ ] Create recommendation engine using PRD-151 performance data - [ ] Implement decision tree logic for AI model selection - [ ] Add cost estimation system based on user priorities - [ ] Build rationale generator explaining recommendations **Success Criteria**: Users can answer questions and receive personalized AI model recommendations with cost estimates ### Milestone 2: Configuration File Generation System **Goal**: Automatic generation of all required setup files **Tasks**: - [ ] Implement .env file generator with correct API key variables - [ ] Create .mcp.json generator for all deployment methods - [ ] Add docker-compose.yml generation for Docker deployments - [ ] Build personalized setup instructions generator - [ ] Create cost breakdown and estimation reporting **Success Criteria**: System generates syntactically correct configuration files for all deployment scenarios ### Milestone 3: CLI Tool Integration & Validation **Goal**: Complete CLI tool with validation and error handling **Tasks**: - [ ] Build CLI command interface (`npx @vfarcic/dot-ai setup`) - [ ] Implement API key format validation and testing - [ ] Add configuration file validation and conflict detection - [ ] Create setup verification and testing system - [ ] Build error recovery and re-configuration support **Success Criteria**: Full CLI tool working with comprehensive validation and error handling ### Milestone 4: Documentation Integration & User Testing **Goal**: Complete documentation and validated user experience **Tasks**: - [ ] Update all setup guides to reference guided setup as primary method - [ ] Create comprehensive guided setup documentation - [ ] Add troubleshooting guide for common guided setup issues - [ ] Conduct user testing with 10+ new users - [ ] Iterate based on user feedback and usage patterns **Success Criteria**: Documentation complete, user testing shows >90% success rate and positive feedback ### Milestone 5: Launch Optimization & Monitoring **Goal**: Production-ready guided setup with monitoring and analytics **Tasks**: - [ ] Add usage analytics and success rate monitoring - [ ] Implement guided setup performance optimization - [ ] Create support documentation for troubleshooting guided setup issues - [ ] Launch guided setup as default setup method - [ ] Monitor adoption rates and user feedback **Success Criteria**: Guided setup launched successfully with >50% adoption rate and <5% support ticket rate --- ## Dependencies ### Internal Dependencies - [ ] AI model performance data from PRD-151 (available) - [ ] Existing deployment method configurations (available) - [ ] Current documentation and setup guides (available) - [ ] MCP client configuration patterns (available) ### External Dependencies - [ ] Node.js CLI framework (inquirer.js or similar) - [ ] Template engine for file generation (handlebars or similar) - [ ] API key validation libraries for different providers - [ ] Cost calculation data from AI provider pricing pages ### Technical Requirements - [ ] Node.js >=18 for CLI tool compatibility - [ ] NPX support for easy installation and execution - [ ] Cross-platform compatibility (Windows, macOS, Linux) - [ ] Integration with existing package.json and npm scripts --- ## Risks & Mitigations | Risk | Impact | Mitigation | |------|--------|-----------| | Generated configurations don't work in all environments | High | Comprehensive testing across all deployment methods, validation system | | AI model recommendations become outdated | Medium | Update recommendation engine when new models added, version configurations | | Users still prefer manual setup | Medium | Keep manual setup as option, gather feedback on why users avoid guided setup | | CLI tool has compatibility issues | Medium | Test across Node.js versions and operating systems, provide fallback options | | API key validation fails for some providers | Low | Make validation optional, provide clear manual verification steps | | Cost estimates become inaccurate | Low | Regular updates from provider pricing, conservative estimates with warnings | --- ## Open Questions 1. **CLI vs Web Interface**: Should we also provide a web-based setup wizard in addition to CLI? 2. **Configuration Updates**: How should users update their configuration when new AI models are added? 3. **Team Setup**: Should guided setup support team/organization-wide configurations? 4. **Advanced Options**: How do we balance simplicity with power-user customization needs? 5. **Multi-Environment**: Should guided setup support different configs for dev/staging/prod? --- ## Out of Scope ### Deferred to Future Versions - [ ] Web-based setup wizard interface - [ ] Advanced configuration customization options - [ ] Team/organization setup workflows - [ ] Integration with external configuration management tools ### Explicitly Not Included - Automatic AI model switching based on task type - Real-time cost monitoring and alerts - Setup configurations for non-MCP clients - Custom AI model endpoint configurations --- ## Success Metrics ### Quantitative Goals - **Setup Time**: Reduce average setup time from 30+ minutes to <5 minutes - **Success Rate**: Increase first-time setup success from ~70% to >95% - **Error Reduction**: Reduce configuration-related support tickets by >80% - **Adoption**: 50%+ of new users adopt guided setup within 30 days ### Qualitative Goals - Users rate setup experience as significantly improved - Reduced complexity and cognitive load during onboarding - Increased confidence in AI model selection decisions - Faster time-to-value for new DevOps AI Toolkit users --- ## Work Log ### 2025-10-15: PRD Creation **Duration**: Initial planning session **Status**: Draft **Context**: Setup complexity has grown significantly with 9 AI models, 3 embedding providers, and 5 deployment methods. Current documentation requires extensive reading and manual configuration, leading to setup failures and user abandonment. **Key Insight**: Transform the configuration problem into a conversation. Instead of asking users to understand 27 possible combinations, ask them about their goals and generate the optimal configuration automatically. **Success Pattern Identified**: Similar tools like create-react-app, Vue CLI, and Angular CLI use guided setup to eliminate configuration complexity. The DevOps AI Toolkit can follow this proven pattern. **Next Steps**: Begin Milestone 1 - Core question flow and recommendation engine development