Persona MCP

# GitOps & Platform Engineering 2025: Enterprise Guide **Last Updated**: 2025-11-23 **Based on**: CNCF GitOps, ArgoCD, Flux, Enterprise implementations --- ## Executive Summary GitOps in 2025 is the foundation of modern platform engineering. It's not just automation—it's about building self-healing, compliant, scalable platform ecosystems that enable engineering teams to operate at unprecedented velocity. **Key Stats (2025)**: - 7,600+ new Solana developers joined (outpacing Ethereum) - GitOps adoption: Critical infrastructure standard - Platform engineering: Fastest-growing DevOps evolution --- ## GitOps Core Principles ### The Four Pillars 1. **Declarative**: Desired state in Git 2. **Versioned**: Git as source of truth 3. **Automated**: Continuous reconciliation 4. **Self-healing**: Automatic drift correction ### Architecture Pattern ```yaml # Git Repository (Source of Truth) infrastructure-repo/ ├── clusters/ │ ├── production/ │ │ ├── apps/ │ │ ├── infrastructure/ │ │ └── config/ │ └── staging/ ├── base/ │ ├── networking/ │ ├── security/ │ └── monitoring/ └── overlays/ ├── dev/ ├── staging/ └── prod/ # Kubernetes Cluster (Desired State) → ArgoCD/Flux continuously syncs → Detects drift → Auto-corrects → All changes via Pull Requests ``` --- ## Multi-Tenancy & Hierarchical Structure ### App-of-Apps Pattern (ArgoCD) ```yaml # root-app.yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: root-app namespace: argocd spec: project: default source: repoURL: https://github.com/company/gitops-apps targetRevision: main path: apps/production destination: server: https://kubernetes.default.svc namespace: argocd syncPolicy: automated: prune: true selfHeal: true syncOptions: - CreateNamespace=true ``` **Benefits**: - Single app creates all child apps - Hierarchical structure - Centralized management ### Multi-Cluster Fleet Architecture ```yaml # Fleet structure management-cluster/ # ArgoCD hub ├── argocd/ │ ├── app-of-apps/ │ └── cluster-configs/ └── applicationsets/ └── cluster-generator.yaml workload-clusters/ # Spokes ├── us-east-1-prod/ ├── us-west-2-prod/ ├── eu-west-1-prod/ └── ap-southeast-1-prod/ ``` **ApplicationSet for Multi-Cluster**: ```yaml apiVersion: argoproj.io/v1alpha1 kind: ApplicationSet metadata: name: guestbook-clusters spec: generators: - clusters: selector: matchLabels: environment: production template: metadata: name: '{{name}}-guestbook' spec: project: default source: repoURL: https://github.com/company/apps path: guestbook destination: server: '{{server}}' namespace: guestbook ``` --- ## Advanced Deployment Patterns ### Blue-Green Deployment ```yaml # blue-green with Argo Rollouts apiVersion: argoproj.io/v1alpha1 kind: Rollout metadata: name: my-app spec: replicas: 3 strategy: blueGreen: activeService: my-app-active previewService: my-app-preview autoPromotionEnabled: false # Manual approval scaleDownDelaySeconds: 30 template: spec: containers: - name: app image: myapp:v2.0 ``` **Process**: 1. Deploy "green" version alongside "blue" 2. Run smoke tests on green 3. Manual approval (or automated after tests) 4. Switch traffic from blue to green 5. Keep blue for 30s (rollback window) 6. Scale down blue ### Canary with Analysis ```yaml apiVersion: argoproj.io/v1alpha1 kind: Rollout metadata: name: my-app-canary spec: strategy: canary: steps: - setWeight: 10 # 10% traffic to new version - pause: {duration: 5m} - analysis: templates: - templateName: success-rate args: - name: service-name value: my-app - setWeight: 50 - pause: {duration: 10m} - analysis: templates: - templateName: error-rate - setWeight: 100 --- apiVersion: argoproj.io/v1alpha1 kind: AnalysisTemplate metadata: name: success-rate spec: args: - name: service-name metrics: - name: success-rate interval: 1m successCondition: result[0] >= 0.95 provider: prometheus: address: http://prometheus:9090 query: | sum(rate( http_requests_total{ service="{{args.service-name}}", status=~"2.." }[5m] )) / sum(rate( http_requests_total{ service="{{args.service-name}}" }[5m] )) ``` --- ## Security & Compliance ### Secret Management with External Secrets Operator ```yaml # ExternalSecret resource apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: app-secrets namespace: production spec: refreshInterval: 1h secretStoreRef: name: aws-secretsmanager kind: SecretStore target: name: app-secrets-k8s creationPolicy: Owner data: - secretKey: database-password remoteRef: key: prod/app/db-password - secretKey: api-key remoteRef: key: prod/app/api-key --- # SecretStore (AWS Secrets Manager) apiVersion: external-secrets.io/v1beta1 kind: SecretStore metadata: name: aws-secretsmanager spec: provider: aws: service: SecretsManager region: us-east-1 auth: jwt: serviceAccountRef: name: external-secrets-sa ``` **Benefits**: - Secrets never in Git - Automatic rotation - Centralized management - Audit trail ### Policy as Code (Open Policy Agent) ```rego # policy/require-labels.rego package kubernetes.admission deny[msg] { input.request.kind.kind == "Deployment" not input.request.object.metadata.labels["app"] msg := "Deployments must have 'app' label" } deny[msg] { input.request.kind.kind == "Deployment" not input.request.object.metadata.labels["owner"] msg := "Deployments must have 'owner' label" } deny[msg] { input.request.kind.kind == "Deployment" container := input.request.object.spec.template.spec.containers[_] not container.resources.limits.memory msg := sprintf("Container %v must have memory limit", [container.name]) } ``` **Enforcement**: ```yaml # OPA Gatekeeper ConstraintTemplate apiVersion: templates.gatekeeper.sh/v1 kind: ConstraintTemplate metadata: name: k8srequiredlabels spec: crd: spec: names: kind: K8sRequiredLabels validation: openAPIV3Schema: properties: labels: type: array items: type: string targets: - target: admission.k8s.gatekeeper.sh rego: | package k8srequiredlabels violation[{"msg": msg, "details": {"missing_labels": missing}}] { provided := {label | input.review.object.metadata.labels[label]} required := {label | label := input.parameters.labels[_]} missing := required - provided count(missing) > 0 msg := sprintf("Missing required labels: %v", [missing]) } ``` --- ## Troubleshooting Framework ### Common Issues & Solutions #### Issue 1: Application Won't Sync **Symptoms**: ```bash $ argocd app get my-app Health Status: Degraded Sync Status: OutOfSync ``` **Diagnosis**: ```bash # Check sync error $ argocd app get my-app --show-operation # Check resource diff $ argocd app diff my-app # Check events $ kubectl get events -n argocd ``` **Common Causes**: 1. **Invalid YAML**: Syntax error in manifests ```bash # Validate locally $ kubectl apply --dry-run=client -f manifest.yaml ``` 2. **Missing CRDs**: Custom Resource Definition not installed ```bash # Check if CRD exists $ kubectl get crd | grep myresource # Install CRD first $ kubectl apply -f crd.yaml ``` 3. **RBAC Issues**: ArgoCD lacks permissions ```yaml # Grant permissions apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: argocd-application-controller rules: - apiGroups: [""] resources: ["*"] verbs: ["*"] ``` #### Issue 2: Drift Detection **Scenario**: Manual `kubectl edit` changed resource **Detection**: ```bash $ argocd app get my-app Sync Status: OutOfSync (Manual change detected) ``` **Fix (Self-Healing)**: ```yaml syncPolicy: automated: selfHeal: true # Automatically revert drift prune: true # Delete resources not in Git ``` **Manual Fix**: ```bash # Force sync $ argocd app sync my-app --force # Or update Git to match cluster $ kubectl get deployment my-app -o yaml > git-repo/deployment.yaml $ git commit && git push ``` --- ## Platform Engineering Best Practices ### Internal Developer Platform (IDP) Components ```yaml # Stack ├── Infrastructure (Terraform/Pulumi) │ ├── VPC, Subnets, Security Groups │ ├── EKS/GKE/AKS Clusters │ └── Databases, Cache, Storage │ ├── GitOps (ArgoCD/Flux) │ ├── Application Deployment │ ├── Config Management │ └── Multi-Cluster Sync │ ├── CI/CD (GitHub Actions/GitLab CI) │ ├── Build & Test │ ├── Security Scanning │ └── GitOps Commit │ ├── Observability (Prometheus/Grafana/Loki) │ ├── Metrics Collection │ ├── Log Aggregation │ └── Alerting │ ├── Developer Portal (Backstage) │ ├── Service Catalog │ ├── Self-Service Provisioning │ └── Documentation Hub │ └── Policy & Security (OPA/Kyverno) ├── Admission Control ├── Compliance Enforcement └── Audit Logging ``` ### Golden Paths **Concept**: Paved road for common tasks **Example - Deploy New Service**: ```bash # Developer runs: $ idp-cli create service \ --name my-api \ --type nodejs \ --database postgres \ --owner team-backend # Behind the scenes: 1. ✓ Creates Git repository from template 2. ✓ Sets up CI/CD pipeline 3. ✓ Provisions database in Terraform 4. ✓ Creates ArgoCD Application 5. ✓ Configures monitoring dashboards 6. ✓ Sets up alerts 7. ✓ Registers in service catalog # Result: Production-ready in 5 minutes ``` --- ## Career Development ### Skills Roadmap **Level 1 - Foundation (0-1 year)**: - ✓ Git fundamentals - ✓ Kubernetes basics - ✓ ArgoCD installation & usage - ✓ YAML/Helm basics **Level 2 - Intermediate (1-2 years)**: - ✓ Multi-cluster management - ✓ Advanced deployment strategies (canary, blue-green) - ✓ Secret management - ✓ Observability integration **Level 3 - Advanced (2-3 years)**: - ✓ Platform engineering design - ✓ Policy as code - ✓ Disaster recovery - ✓ Cost optimization **Level 4 - Expert (3+ years)**: - ✓ Enterprise architecture - ✓ Multi-tenancy at scale - ✓ Compliance automation - ✓ Team leadership ### Certifications 1. **Certified Kubernetes Administrator (CKA)** - Foundation 2. **GitOps Certified Associate** - CNCF 3. **Certified Kubernetes Security Specialist (CKS)** - Security focus 4. **Platform Engineering Certification** - Linux Foundation (2025) --- ## Future Trends (2025-2026) ### 1. AI-Powered GitOps ```yaml # AI suggests optimizations apiVersion: ai.gitops.io/v1 kind: AIOptimizer metadata: name: cost-optimizer spec: target: production-cluster objectives: - cost_reduction - performance_improvement constraints: - sla: 99.9% - max_cost_increase: 0% ``` **Use cases**: - Auto-scaling predictions - Cost optimization suggestions - Security vulnerability detection ### 2. Progressive Delivery at Scale - Automated canary analysis - ML-powered rollback decisions - Multi-region progressive rollouts ### 3. Cloud-Native Platform Evolution - Serverless GitOps (Knative integration) - Edge computing deployments - Hybrid cloud orchestration --- ## Key Takeaways 1. **GitOps = Source of Truth**: Git is single source for infrastructure & applications 2. **Automation**: Self-healing, drift correction, compliance enforcement 3. **Security**: Secrets external, policy as code, audit trails 4. **Scale**: Multi-cluster, multi-tenant, global deployments 5. **Developer Experience**: Self-service, golden paths, fast feedback --- ## References - CNCF GitOps Working Group - ArgoCD Documentation (2025) - Flux Documentation (2025) - "GitOps 2025: Enterprise Implementation" - Support Tools - Kubernetes Security Best Practices **Related**: See `kubernetes-advanced.md`, `ci-cd-pipelines.md`, `terraform-infrastructure.md`