DevOps AI Toolkit

# PRD: Deployment Documentation & Example-Based Learning **Created**: 2025-11-21 **Status**: Planning - Blocked by dot-ai-controller PRD #4 **Owner**: TBD **Last Updated**: 2025-11-21 **Issue**: #228 **Priority**: High ## Executive Summary Enhance the recommendation system by generating deployment documentation alongside Kubernetes manifests, storing them in Git, and using past deployment examples as few-shot learning context for future recommendations. This creates an organizational memory of deployment decisions and improves AI accuracy over time. **⚠️ PREREQUISITE**: This PRD is blocked by dot-ai-controller PRD #4 (Solution CRD for Deployment Tracking). Work cannot begin until dot-ai-controller #4 is complete, as this feature requires the CRD infrastructure for tracking documentation references. ## Problem Statement ### Current Challenges - **No Deployment History**: AI generates recommendations without knowledge of past successful deployments - **Lost Context**: Deployment decisions, rationale, and trade-offs are not captured - **No Learning from Examples**: Each recommendation starts from scratch without benefiting from organizational patterns - **Generic Solutions**: AI lacks context about how similar applications were deployed in the past - **Missing Documentation**: Generated manifests have no accompanying explanation or rationale ### User Impact - **Inconsistent Deployments**: Similar applications get deployed differently because AI doesn't learn from past examples - **Lost Knowledge**: New team members can't see how previous deployments were configured and why - **Repetitive Work**: Users repeatedly explain the same deployment patterns to the AI - **Lower Accuracy**: AI recommendations could be more accurate if informed by real deployment history ## Goals ### Primary Goals 1. **Generate Documentation with Manifests** - Create comprehensive markdown documentation alongside Kubernetes manifests - Capture intent, decisions, rationale, patterns applied, and key configurations - Make documentation useful for both humans and AI 2. **Store Deployment Examples in Git** - Documentation lives alongside manifests in version control - Build searchable knowledge base of organizational deployments - Enable team collaboration and knowledge sharing 3. **Index Documentation for AI Retrieval** - Store documentation embeddings in Qdrant vector DB - Enable semantic search of past deployments - Track references to Git locations for up-to-date information 4. **Use Examples as Few-Shot Learning** - Retrieve relevant past deployments during new recommendations - Feed examples to AI as context for better accuracy - Learn from organizational deployment patterns over time 5. **Track Example Effectiveness** - Integration with PRD #218 learning system - Track which examples are most helpful - Monitor if example-informed recommendations are accepted more often ## Solution Overview ### High-Level Workflow ``` ┌─────────────────────────────────────────────────────────────┐ │ 1. Generate Manifests + Documentation │ │ - User completes recommendation workflow │ │ - System generates YAML manifests │ │ - System generates markdown documentation │ └────────────────────┬────────────────────────────────────────┘ │ ┌────────────────────▼────────────────────────────────────────┐ │ 2. User Saves to Git │ │ - User commits manifests + docs to repository │ │ - Documentation includes: intent, solution, rationale │ └────────────────────┬────────────────────────────────────────┘ │ ┌────────────────────▼────────────────────────────────────────┐ │ 3. Index Documentation (via PRD #25 CRD) │ │ - User provides Git location to system │ │ - System creates/updates CR with documentation reference │ │ - CR controller embeds docs and stores in Qdrant │ │ - Periodic sync checks for doc changes │ │ - [CRD/controller design: see PRD #25] │ └────────────────────┬────────────────────────────────────────┘ │ ┌────────────────────▼────────────────────────────────────────┐ │ 4. Future Recommendations Use Examples │ │ - New recommendation request arrives │ │ - System retrieves relevant past deployment docs (RAG) │ │ - AI receives examples as few-shot learning context │ │ - Generates solution informed by real deployment history │ └─────────────────────────────────────────────────────────────┘ ``` ### Key Components 1. **Documentation Generation** (enhance `generateManifests` tool) - Generate markdown with solution summary, decisions, rationale - Return docs alongside manifests in response - Details TBD during implementation 2. **Git Integration** (user workflow enhancement) - User saves manifests + docs to Git repo - User provides Git location to system - Details TBD during implementation 3. **CRD-Based Tracking** (dot-ai-controller PRD #4 - PREREQUISITE) - Custom Resource tracks deployment documentation - Stores reference to Git location (URL, path, branch) - Controller periodically checks for changes - Updates Qdrant embeddings when docs change - **Design entirely in dot-ai-controller PRD #4** 4. **Vector DB Storage** (Qdrant collection) - New collection: `deployment-examples` - Stores documentation embeddings for semantic search - Metadata includes: intent, app name, resources, Git reference 5. **RAG Retrieval** (enhance recommendation flow) - Query `deployment-examples` collection with user intent - Retrieve top N relevant past deployments - Feed examples to AI as context - Details TBD during implementation 6. **Metrics Integration** (PRD #218) - Track how often each example is retrieved - Track if example-informed recommendations are accepted - Use metrics to improve example ranking ## Requirements ### Functional Requirements 1. **Documentation Generation** - Generate markdown documentation from solution data - Include: user intent, solution overview, resources, decisions, rationale, patterns applied - Format must be human-readable and AI-parseable - Documentation returned alongside manifests 2. **Vector DB Integration** - Embed documentation content for semantic search - Store metadata for filtering (app type, resources, date) - Reference to Git location for fetching latest version - Efficient retrieval during recommendation workflow 3. **Example Retrieval** - Semantic search of past deployments by intent - Configurable number of examples to retrieve - Examples formatted for AI consumption - Graceful degradation if no examples found 4. **Usage Metrics** - Track retrieval count for each example - Track acceptance rate of recommendations using examples - Integration with PRD #218 metrics system ### Non-Functional Requirements - **Performance**: Example retrieval adds < 2 seconds to recommendation flow - **Storage**: Vector DB scales to thousands of deployment examples - **Reliability**: System handles Git access failures gracefully - **Security**: Git credentials managed securely for private repositories - **Privacy**: No sensitive data in documentation or embeddings ## Dependencies ### Prerequisites (BLOCKING) - **dot-ai-controller PRD #4**: Solution CRD for Deployment Tracking - **MUST BE COMPLETE BEFORE STARTING THIS PRD** - Provides CRD infrastructure for tracking documentation references - Implements controller for syncing docs from Git to Qdrant - This PRD cannot begin until dot-ai-controller #4 is fully implemented - See: https://github.com/vfarcic/dot-ai-controller/issues/4 ### Integration Points - **PRD #218**: Learning system for tracking example effectiveness - **Recommendation tool**: Enhanced to retrieve and use examples - **Generate Manifests tool**: Enhanced to create documentation - **Qdrant**: Vector DB operational and accessible - **Git**: Users have Git repositories for storing manifests/docs ### Future Considerations - **Packaging PRD (TBD)**: Helm/Kustomize packaging may affect documentation format - Consider creating packaging PRD and evaluating impact before implementing this feature ## Implementation Milestones **⚠️ NOTE**: These milestones can only begin after dot-ai-controller PRD #4 is complete. ### Milestone 1: Documentation Generation ⬜ **Goal**: Generate useful documentation alongside manifests **Success Criteria:** - Documentation generated from solution data - Includes intent, resources, decisions, rationale - Human-readable and comprehensive - Returned in `generateManifests` response **Estimated Duration**: TBD during planning ### Milestone 2: Vector DB Integration ⬜ **Goal**: Store and retrieve deployment examples efficiently **Success Criteria:** - `deployment-examples` collection created in Qdrant - Documentation embedded and stored with metadata - Semantic search retrieves relevant examples - Git references stored for up-to-date access **Estimated Duration**: TBD during planning ### Milestone 3: RAG Enhancement ⬜ **Goal**: Use past examples to improve recommendations **Success Criteria:** - Recommendation flow retrieves relevant examples - Examples provided to AI as context - AI generates solutions informed by examples - Measurable improvement in recommendation quality **Estimated Duration**: TBD during planning ### Milestone 4: Metrics & Learning Integration ⬜ **Goal**: Track example effectiveness and improve over time **Success Criteria:** - Examples tracked with retrieval counts - Acceptance rates measured for example-informed recommendations - Integration with PRD #218 metrics system - Data informs example ranking and retrieval **Estimated Duration**: TBD during planning ## Success Criteria - [ ] **Documentation Quality**: Generated docs are comprehensive and useful - [ ] **Example Retrieval**: Relevant past deployments retrieved in < 2 seconds - [ ] **AI Accuracy**: Recommendation quality improves with example context (measurable) - [ ] **User Adoption**: Users regularly commit docs and reference past deployments - [ ] **System Reliability**: CRD controller (from PRD #25) handles 95%+ of doc updates successfully - [ ] **Organizational Memory**: Growing knowledge base of deployment examples ## Risks & Mitigations | Risk | Impact | Probability | Mitigation | |------|--------|-------------|------------| | dot-ai-controller PRD #4 delayed or blocked | High | Medium | Monitor dot-ai-controller #4 progress, prepare documentation generation work in parallel | | Documentation quality varies | Medium | Medium | Define clear template and generation guidelines | | Users don't commit docs to Git | High | Medium | Make workflow seamless, demonstrate value with metrics | | Example retrieval adds latency | Medium | Low | Optimize vector search, cache frequent queries | | Sensitive data in docs | High | Low | Clear guidelines on what to include, sanitization checks | | Examples become stale | Medium | Medium | CRD controller (dot-ai-controller PRD #4) detects changes, users can refresh | ## Open Questions 1. **Documentation Format**: Exact structure and sections for generated documentation? (Discuss during implementation) 2. **File Location Convention**: Where should docs be stored relative to manifests? (Discuss during implementation) 3. **Number of Examples**: How many past deployments to retrieve for context? (3? 5? Configurable?) 4. **Prioritization**: Should recent examples be weighted higher than older ones? 5. **Filtering**: Should examples be filtered by success indicators (still deployed, no failures)? 6. **Git Credentials**: How to securely manage Git access for private repositories? (May be handled by dot-ai-controller PRD #4) 7. **Packaging PRD**: Should packaging (Helm/Kustomize) PRD be created and completed first? ## Future Enhancements - **Automatic Indexing**: Git hooks or CI/CD integration to auto-index new deployments - **Example Templates**: Pre-populate with common deployment examples - **Example Recommendations**: "Here's how you deployed a similar app 3 months ago" - **Cross-Cluster Learning**: Share examples across multiple clusters (anonymized) - **Example Versioning**: Track documentation changes over time - **Community Examples**: Optional sharing of examples across organizations ## Work Log ### 2025-11-21: PRD Creation **Duration**: ~1 hour **Status**: Planning - Blocked by dot-ai-controller PRD #4 **Completed Work**: - Created PRD based on user discussion - Defined four-phase implementation approach - Established hard dependency on dot-ai-controller PRD #4 (blocking) - Documented high-level workflow and components - Kept CRD/controller design scoped to dot-ai-controller PRD #4 **Key Decisions**: - Enhance existing `generateManifests` tool rather than separate system - Rely on dot-ai-controller PRD #4 for CRD and controller implementation - Store examples in Qdrant vector DB for semantic retrieval - Integrate with PRD #218 for usage metrics - Keep implementation details flexible for discussion during development - Make dot-ai-controller PRD #4 a hard blocker - cannot start this until it is complete **Next Steps**: - Monitor dot-ai-controller PRD #4 progress - Can begin planning documentation format in parallel - Begin Milestone 1 only after dot-ai-controller PRD #4 is complete --- ## Appendix ### Example Documentation Output (Draft) ```markdown # Deployment: Payment Service API **Generated**: 2025-11-18 **Intent**: Deploy Go microservice with PostgreSQL database and Redis cache **Status**: Applied successfully ## Solution Overview Deployed as Deployment with 3 replicas, using StatefulSet for PostgreSQL, and Deployment for Redis. Exposed via ClusterIP services internally. ## Key Decisions - **Replicas**: 3 (based on "High Availability" pattern) - **PostgreSQL**: StatefulSet with persistent volume (10Gi) - **Redis**: Deployment without persistence (cache-only use case) - **Resources**: CPU 500m/1000m, Memory 512Mi/1Gi (Go API pattern) ## Patterns Applied - High Availability Pattern (multiple replicas) - Stateful Storage Pattern (PostgreSQL) - Cache Pattern (Redis) ## Resources Deployed - Deployment: payment-api (3 replicas) - StatefulSet: postgresql (1 replica, PVC 10Gi) - Deployment: redis (1 replica) - Service: payment-api-svc, postgresql-svc, redis-svc - ConfigMap: app-config ## Configuration Answers - **Application Name**: payment-api - **Namespace**: production - **Port**: 8080 - **Database Size**: 10Gi - **Enable TLS**: Yes ## Related Resources - Manifests: manifests/payment-api.yaml - Repository: github.com/myorg/payment-service - Solution ID: sol-1762983784617-9ddae2b8 ``` ### Relationship to dot-ai-controller PRD #4 **What dot-ai-controller PRD #4 Provides** (PREREQUISITE): - Custom Resource Definition for deployment tracking - Controller for syncing documentation from Git - Qdrant embedding management - Change detection and update mechanisms - See: https://github.com/vfarcic/dot-ai-controller/issues/4 **What This PRD Adds**: - Documentation generation from solution data - Example-based learning and RAG retrieval - Integration with recommendation workflow - Usage metrics for example effectiveness