CodeGraph CLI MCP Server

--- pdf-engine: lualatex mainfont: "DejaVu Serif" monofont: "DejaVu Sans Mono" header-includes: | \usepackage{fontspec} \directlua{ luaotfload.add_fallback("emojifallback", {"NotoColorEmoji:mode=harf;"}) } \setmainfont[ RawFeature={fallback=emojifallback} ]{DejaVu Serif} --- # CodeGraph Release Management Guide ## Overview This document outlines the comprehensive release management system for CodeGraph, designed to ensure zero-downtime production deployments while maintaining the system's strict performance requirements (sub-50ms P99 latency, >10,000 QPS throughput). ## Architecture Alignment Our release management system is specifically designed around CodeGraph's architecture targets: - **Latency**: Sub-50ms P95 response times maintained during deployments - **Throughput**: >10,000 QPS sustained during rolling updates - **Memory Efficiency**: <500MB footprint for 100k LOC processing - **Cache Performance**: >92% hit rates preserved across deployments - **Availability**: 99.9% uptime with zero-downtime deployments ## Release Pipeline Components ### 1. Automated Release Pipeline (`.github/workflows/release.yml`) **Triggers:** - Git tag pushes (`v*`) - Manual workflow dispatch with environment selection **Key Features:** - Multi-platform binary builds (Linux x86_64/ARM64, macOS, Windows) - Size-optimized container images using `release-size` profile - Crates.io publishing for workspace components - Automated changelog generation **Deployment Strategies:** - **Rolling Updates**: Default for staging, gradual pod replacement - **Blue-Green**: Complete environment switch, instant rollback capability - **Canary**: Traffic-weighted deployments with automatic promotion/rollback ### 2. Emergency Rollback System (`.github/workflows/rollback.yml`) **Safety Features:** - Explicit confirmation requirement (`CONFIRM` input) - Pre-rollback state backup creation - Target version validation against git history - Database migration rollback (when applicable) **Rollback Phases:** 1. **Validation**: Confirm target version exists and is valid 2. **Backup**: Create complete state snapshot (K8s resources, configs, secrets) 3. **Database Rollback**: Handle schema downgrades if needed 4. **Application Rollback**: Execute deployment strategy-specific rollback 5. **Verification**: Comprehensive post-rollback validation 6. **Notification**: Automated incident reporting and stakeholder alerts ### 3. Deployment Validation Framework (`scripts/deploy/validate.sh`) **Comprehensive Health Checks:** - Authentication-aware endpoint validation - HTTP/2 optimization feature testing - GraphQL query execution verification - Prometheus metrics availability validation - Performance benchmarking (response time thresholds) - Concurrent request handling (configurable load testing) - Security header compliance verification **Configuration Options:** ```bash # Basic usage BASE_URL=https://api.codegraph.com scripts/deploy/validate.sh # Advanced configuration BASE_URL=https://staging.codegraph.com \ TIMEOUT_SECS=120 \ CONCURRENT_REQUESTS=50 \ MAX_RESPONSE_TIME_MS=200 \ VERBOSE=true \ scripts/deploy/validate.sh ``` ### 4. Production Monitoring (`monitoring/`) **Prometheus Alerting Rules** (`prometheus-rules.yml`): - **Application Health**: Service availability, endpoint response times - **Performance Monitoring**: Latency percentiles, throughput tracking - **Resource Utilization**: Memory, CPU, disk usage with architecture-aligned thresholds - **Database Health**: RocksDB performance, FAISS index efficiency - **Cache Performance**: Hit rates, eviction patterns - **Security Compliance**: Authentication failures, rate limiting triggers **Grafana Dashboard** (`grafana-dashboard.json`): - Real-time performance metrics visualization - Architecture target compliance tracking - Deployment event correlation - Resource utilization trends - Cache performance analysis ## Release Process Workflows ### Standard Release Process 1. **Pre-Release Validation** ```bash # Run comprehensive checks make ci make load-test make deploy-validate ``` 2. **Release Creation** - Create git tag: `git tag -a v1.2.3 -m "Release v1.2.3"` - Push tag: `git push origin v1.2.3` - Pipeline automatically triggers 3. **Staging Deployment** - Automatic deployment to staging environment - Full validation suite execution - Performance regression testing 4. **Production Deployment** ```bash # Manual workflow dispatch with production target gh workflow run release.yml \ -f version=v1.2.3 \ -f environment=production \ -f deployment_strategy=canary ``` 5. **Post-Deployment Monitoring** - Grafana dashboard monitoring - Prometheus alert verification - Performance metrics validation ### Emergency Rollback Process 1. **Trigger Rollback** ```bash gh workflow run rollback.yml \ -f target_version=v1.2.2 \ -f environment=production \ -f reason="Critical performance regression" \ -f confirmation=CONFIRM ``` 2. **Automatic Execution** - Version validation and backup creation - Database migration rollback (if needed) - Application deployment rollback - Health verification and stakeholder notification ### Deployment Strategies #### Rolling Updates - **Use Case**: Standard releases, low-risk changes - **Characteristics**: Gradual pod replacement, maintains capacity - **Rollback**: Kubernetes native rollback capabilities #### Blue-Green Deployment - **Use Case**: Major releases, database schema changes - **Characteristics**: Complete environment duplication - **Rollback**: Instant traffic switch to previous environment #### Canary Deployment - **Use Case**: High-risk changes, experimental features - **Characteristics**: Traffic-weighted gradual rollout - **Rollback**: Automatic based on performance metrics ## Performance Monitoring Integration ### Key Performance Indicators **Latency Monitoring:** ```yaml # P95 latency threshold - alert: HighLatency expr: histogram_quantile(0.95, http_request_duration_seconds) > 0.05 for: 2m ``` **Throughput Tracking:** ```yaml # QPS threshold monitoring - alert: LowThroughput expr: rate(http_requests_total[5m]) < 10000 for: 5m ``` **Cache Performance:** ```yaml # Cache hit rate monitoring - alert: LowCacheHitRate expr: cache_hit_ratio < 0.92 for: 3m ``` ### Deployment Event Correlation Grafana dashboard includes deployment annotations that correlate releases with performance changes, enabling rapid identification of deployment-related issues. ## Security and Compliance ### Pre-Deployment Security Checks - Container image vulnerability scanning - Dependency audit with `cargo audit` - Security header validation in deployment tests ### Runtime Security Monitoring - Authentication failure tracking - Rate limiting effectiveness - API key usage patterns ## Troubleshooting Guide ### Common Issues and Solutions **Build Failures:** ```bash # Clear build cache and retry cargo clean make build-release ``` **Deployment Validation Failures:** ```bash # Debug with verbose output VERBOSE=true BASE_URL=https://staging.codegraph.com scripts/deploy/validate.sh ``` **Performance Regression:** ```bash # Compare against baseline make perf-regression ``` **Rollback Issues:** - Check target version exists in git history - Verify Kubernetes access credentials - Confirm backup creation succeeded ### Monitoring and Alerting **Critical Alerts:** - Service unavailability (>1 minute) - P95 latency >50ms (>2 minutes) - Cache hit rate <92% (>3 minutes) - Memory usage >500MB (>5 minutes) **Dashboard Access:** - Production: `https://grafana.codegraph.com/d/codegraph-prod` - Staging: `https://grafana.codegraph.com/d/codegraph-staging` ## Disaster Recovery ### Backup Strategy - Automated state backups before each deployment - RocksDB snapshot preservation - Configuration and secret backup - 30-day retention policy ### Recovery Procedures 1. **Service Restoration**: Emergency rollback to last known good version 2. **Data Recovery**: RocksDB restoration from snapshots 3. **Configuration Recovery**: K8s resource restoration from backups 4. **Validation**: Complete deployment validation suite execution ## Continuous Improvement ### Performance Baseline Management ```bash # Update performance baselines after successful releases BASELINE_NAME=v1.2.3 make bench make bench-report ``` ### Release Metrics Tracking - Deployment success rate - Rollback frequency - Time to recovery - Performance impact measurement ### Feedback Integration - Post-incident reviews - Performance regression analysis - Security audit findings integration - Stakeholder feedback incorporation ## Conclusion This release management system provides comprehensive automation, monitoring, and safety mechanisms to ensure CodeGraph maintains its performance targets while enabling rapid, reliable deployments. The system's design prioritizes zero-downtime operations while providing robust rollback capabilities for emergency scenarios. For questions or issues, consult the troubleshooting guide or contact the DevOps team through the established incident response channels.