Skip to main content
Glama
Sign In
KubeGuard-MCP

KubeGuard-MCP

by Lokesh-Guttikonda
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python
Local

🛡️ KubeGuard-MCP

Zero-Trust Kubernetes & Container Triage — MCP Server + CLI

Diagnose OOMKilled, CrashLoopBackOff, ImagePullBackOff, CreateContainerConfigError, Unschedulable and Service port mismatches deterministically, get the exact YAML / Dockerfile diff that fixes them, and never leak a secret to your terminal, your incident report or an LLM.

CI Python Dependencies Tests MCP Cost Platforms License: MIT

$0 to run, offline by design. The core engine is the Python standard library — no PyYAML, no SDKs, no API keys, no cloud, no network calls. httpx is an optional extra used only by the opt-in LLM reviewer, which only ever receives redacted data.

Contents

Related MCP server: K8gentS

✨ Why KubeGuard

The 3 a.m. problem

What KubeGuard does

Pod JSON, events, logs and YAML are scattered; the root cause hides in the combination

One-shot correlator cross-references every signal and scores the evidence

"Just bump the memory" / "just restart it" guesses

Deterministic root cause + the exact unified diff for your file — comments and formatting preserved

Logs contain DB passwords, AWS keys, JWTs, kubeconfig tokens

A redaction firewall masks them before analysis and on every output path

AI agents happily run kubectl delete ns

A zero-trust policy engine classifies every command SAFE_READ / APPROVAL_REQUIRED / BLOCKED_DESTRUCTIVE

LLM triage costs money and ships your data out

Fully offline. LLMs are an optional second opinion on redacted data only

Tooling that works in one terminal only

Pure stdlib, Python 3.9 – 3.14, Linux / macOS / Windows (incl. cp1252 consoles)

🚀 60-second quick start

No pip install, no virtualenv, no API key:

git clone https://github.com/Lokesh-Guttikonda/KubeGuard-MCP.git
cd KubeGuard-MCP
python scripts/demo_offline.py

On Windows, enable full Unicode glyphs first (without it the output cleanly degrades to ASCII):

set PYTHONUTF8=1

Install the kubeguard command (still zero runtime dependencies):

pip install .
kubeguard diagnose examples/crashloop_secret_leak

Or run it straight from GitHub with uv:

uvx --from git+https://github.com/Lokesh-Guttikonda/KubeGuard-MCP kubeguard demo

🖥️ Real demo output

The blocks below are the unedited output of the commands shown, regenerated by scripts/update_readme.py — the test-suite fails if they drift from what the code prints.

The secret-leak scenario: the firewall masks the leaked database URL and AWS keys before the correlator sees them, the root cause is pinned to the stale plaintext credentials in the Deployment, the fix moves them to secretKeyRef, and the agent's "quick fix" with shell chaining is blocked.

$ python scripts/demo_offline.py --scenario crashloop_secret_leak --no-report
╔════════════════════════════════════════════════════════════════════════════════════╗
║   KubeGuard-MCP v1.0.0 · Zero-Trust Kubernetes & Container Triage · OFFLINE DEMO   ║
╚════════════════════════════════════════════════════════════════════════════════════╝
  engine: deterministic correlator · network calls: 0 · API keys: 0 · LLM reviewer: mock

━━━ [1/1] crashloop_secret_leak ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
orders-api crash-loops after the database password rotation
incident KG-8E8F07F5 · pod shop/orders-api-6c8d9b7f45-tq9zp · deployment/orders-api

▶ 1. Secret firewall    13 secrets blocked before analysis (k8s_secret_data ×4, aws_access_key_id ×3, db_connection_string ×3, k8s_env_secret ×2, aws_secret_access_key ×1)
     logs.txt:3         … DEBUG DATABASE_URL=postgresql://orders_svc:[REDACTED:db_connection_string]@orders-db.sh…
     logs.txt:4         2026-11-04T14:02:10.875Z DEBUG AWS_ACCESS_KEY_ID=[REDACTED:aws_access_key_id]
     logs.txt:5         2026-11-04T14:02:10.875Z DEBUG AWS_SECRET_ACCESS_KEY=[REDACTED:aws_secret_access_key]
     deployment.yaml:26 value: postgresql://orders_svc:[REDACTED:db_connection_string]@orders-db.shop.svc.cluste…
     … and 9 more redacted lines
▶ 2. Diagnosis          CrashLoopBackOff · confidence 99% · SEV-1
     root cause: The application cannot authenticate to its backing service (password authentication
     failed) and exits with code 1 on startup, so the kubelet restarts it with exponential back-off.
     The credentials are hard-coded in plaintext in the Deployment (DATABASE_URL, AWS_ACCESS_KEY_ID,
     AWS_SECRET_ACCESS_KEY) instead of being read from a Secret, so they went stale when the Secret
     was rotated - and they leak into the startup logs.
     evidence:
       • [pod.status] container "orders" waiting: CrashLoopBackOff, restarts=9, last exit 1
         (application error)
       • [events] Warning/BackOff x41: Back-off restarting failed container orders in pod
         orders-api-6c8d9b7f45-tq9zp_shop(2c7e9f1a-4d3b-4b8e-9f6a-1e2d3c4b5a69)
       • [logs] line 17: asyncpg.exceptions.InvalidPasswordError: password authentication failed for
         user "orders_svc"
       • [manifest] credentials hard-coded as plaintext env values in deployment.yaml: DATABASE_URL,
         AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
       • [secrets] Secret orders-db-credentials holds key 'database-url' matching DATABASE_URL
         (rotated 2026-11-04T13:45:00Z)
       • [secrets] Secret orders-aws-credentials holds key 'aws-access-key-id' matching
         AWS_ACCESS_KEY_ID
       • [secrets] Secret orders-aws-credentials holds key 'aws-secret-access-key' matching
         AWS_SECRET_ACCESS_KEY
▶ 3. Exact fix          1 patch → deployment.yaml
     --- a/deployment.yaml
     +++ b/deployment.yaml
     @@ -23,11 +23,20 @@
                containerPort: 8000
              env:
              - name: DATABASE_URL
     -          value: [REDACTED:k8s_env_secret]
     +          valueFrom:
     +            secretKeyRef:
     +              name: orders-db-credentials
     +              key: database-url
              - name: AWS_ACCESS_KEY_ID
     -          value: [REDACTED:aws_access_key_id]
     +          valueFrom:
     +            secretKeyRef:
     +              name: orders-aws-credentials
     +              key: aws-access-key-id
              - name: AWS_SECRET_ACCESS_KEY
     -          value: [REDACTED:k8s_env_secret]
     +          valueFrom:
     +            secretKeyRef:
     +              name: orders-aws-credentials
     +              key: aws-secret-access-key
              - name: S3_INVOICE_BUCKET
                value: acme-shop-invoices
              - name: LOG_LEVEL
     1. Replace the plaintext env values with secretKeyRef references (patch below) so the Pod
        always reads the current, rotated credential.
     2. Rotate every credential that appeared in the logs (database password, cloud keys) - they
        must be treated as compromised.
     3. Stop logging configuration values at startup; log only key names.
▶ 4. Zero-trust policy  agent-proposed actions
     ✔ SAFE_READ           kubectl logs orders-api-6c8d9b7f45-tq9zp -n shop --previous
     ⚠ APPROVAL_REQUIRED   kubectl get secret orders-db-credentials -n shop -o jsonpath='{.data.password}'
                           └ reads Secret values (-o yaml/json/jsonpath decodes credentials)
     ✖ BLOCKED_DESTRUCTIVE kubectl delete secret orders-db-credentials -n shop; kubectl apply -f old-secret.yaml
                           └ shell operator ';' (pipes, chaining, redirection) is not allowed; submit one command
▶ 5. AI review (mock)   AGREE - redacted payload only
     3 independent signals corroborate CrashLoopBackOff (db auth failure); the patch is minimal and
     reversible.
     risk: 3 credentials leaked into runtime logs - rotate before rolling out the fix.
     risk: Restarting pods will not help until the underlying startup error is fixed.

━━━ Summary ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

  #  scenario                root cause                   conf  sev    secrets  fix
  1  crashloop_secret_leak   ✔ CrashLoopBackOff            99%  SEV-1       13  deployment.yaml

  1/1 incidents diagnosed correctly · 13 secrets blocked · 0 network calls · 0 API keys · no LLM required
$ python scripts/demo_offline.py --no-report
╔════════════════════════════════════════════════════════════════════════════════════╗
║   KubeGuard-MCP v1.0.0 · Zero-Trust Kubernetes & Container Triage · OFFLINE DEMO   ║
╚════════════════════════════════════════════════════════════════════════════════════╝
  engine: deterministic correlator · network calls: 0 · API keys: 0 · LLM reviewer: mock

━━━ [1/4] oom_killed ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
payments-api restarts every few minutes under settlement load
incident KG-D509530D · pod payments/payments-api-7d9f8b6c5d-x2k4q · deployment/payments-api

▶ 1. Secret firewall    2 secrets blocked before analysis (email ×1, jwt ×1)
     logs.txt:6         … : POST /v1/charges headers={Authorization=Bearer [REDACTED:jwt], X-Request-Id=8f14e45f}
     logs.txt:7         ….a.p.service.ChargeService : charge accepted customer=[REDACTED:email] amount=129.00 EUR
▶ 2. Diagnosis          OOMKilled · confidence 99% · SEV-2
     root cause: JVM max heap -Xmx768m (768Mi) is 150% of the container memory limit (512Mi). Heap +
     metaspace + thread stacks + off-heap buffers exceed the cgroup limit, so the kernel OOM killer
     SIGKILLs the process (exit 137) before the runtime can even throw an OutOfMemoryError.
     evidence:
       • [pod.status] container "api" lastState.terminated.reason=OOMKilled (exitCode=137,
         restarts=6)
       • [events] Warning/OOMKilling: Memory cgroup out of memory: Killed process 4127 (java)
         total-vm:3914212kB, anon-rss:521436kB, file-rss:18220kB, shmem-rss:0kB, UID:10001
         pgtables:1444kB oom_s…
       • [logs] line 1: Picked up JAVA_TOOL_OPTIONS: -Xmx768m -XX:+UseG1GC
         -XX:+ExitOnOutOfMemoryError
       • [dockerfile] -Xmx768m set in Dockerfile, but resources.limits.memory=512Mi
     also: CrashLoopBackOff (54%) - symptom of OOMKilled
▶ 3. Exact fix          2 patches → deployment.yaml, Dockerfile
     --- a/deployment.yaml
     +++ b/deployment.yaml
     @@ -33,7 +33,7 @@
                    memory: 256Mi
                  limits:
                    cpu: "1"
     -              memory: 512Mi
     +              memory: 1Gi
                readinessProbe:
                  httpGet:
                    path: /actuator/health/readiness

     --- a/Dockerfile
     +++ b/Dockerfile
     @@ -1,7 +1,7 @@
      FROM eclipse-temurin:21-jre-alpine
      WORKDIR /app
      COPY target/payments-api.jar /app/app.jar
     -ENV JAVA_TOOL_OPTIONS="-Xmx768m -XX:+UseG1GC -XX:+ExitOnOutOfMemoryError"
     +ENV JAVA_TOOL_OPTIONS="-XX:MaxRAMPercentage=75.0 -XX:+UseG1GC -XX:+ExitOnOutOfMemoryError"
      USER 10001
      EXPOSE 8080
      ENTRYPOINT ["java", "-jar", "/app/app.jar"]
     1. Raise resources.limits.memory of container "api" from 512Mi to 1Gi.
     2. Replace -Xmx768m with -XX:MaxRAMPercentage=75.0 in Dockerfile so the heap tracks the
        container limit instead of a hard-coded size.
     3. After the rollout, confirm the working set stays below ~80% of the limit (kubectl top) and
        consider a VerticalPodAutoscaler in recommendation mode.
▶ 4. Zero-trust policy  agent-proposed actions
     ✔ SAFE_READ           kubectl top pod payments-api-7d9f8b6c5d-x2k4q -n payments --containers
     ⚠ APPROVAL_REQUIRED   kubectl set resources deployment/payments-api -n payments -c api --limits=memory=1Gi
                           └ 'set' changes cluster state
     ✖ BLOCKED_DESTRUCTIVE kubectl delete pods --all -n payments
                           └ delete --all removes every matching object
▶ 5. AI review (mock)   AGREE - redacted payload only
     3 independent signals corroborate OOMKilled (heap exceeds limit); the patch is minimal and
     reversible.
     risk: 1 credential leaked into runtime logs - rotate before rolling out the fix.
     risk: A higher memory limit raises node memory pressure; confirm allocatable headroom before
           the rollout.

━━━ [2/4] crashloop_secret_leak ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
orders-api crash-loops after the database password rotation
incident KG-8E8F07F5 · pod shop/orders-api-6c8d9b7f45-tq9zp · deployment/orders-api

▶ 1. Secret firewall    13 secrets blocked before analysis (k8s_secret_data ×4, aws_access_key_id ×3, db_connection_string ×3, k8s_env_secret ×2, aws_secret_access_key ×1)
     logs.txt:3         … DEBUG DATABASE_URL=postgresql://orders_svc:[REDACTED:db_connection_string]@orders-db.sh…
     logs.txt:4         2026-11-04T14:02:10.875Z DEBUG AWS_ACCESS_KEY_ID=[REDACTED:aws_access_key_id]
     logs.txt:5         2026-11-04T14:02:10.875Z DEBUG AWS_SECRET_ACCESS_KEY=[REDACTED:aws_secret_access_key]
     deployment.yaml:26 value: postgresql://orders_svc:[REDACTED:db_connection_string]@orders-db.shop.svc.cluste…
     … and 9 more redacted lines
▶ 2. Diagnosis          CrashLoopBackOff · confidence 99% · SEV-1
     root cause: The application cannot authenticate to its backing service (password authentication
     failed) and exits with code 1 on startup, so the kubelet restarts it with exponential back-off.
     The credentials are hard-coded in plaintext in the Deployment (DATABASE_URL, AWS_ACCESS_KEY_ID,
     AWS_SECRET_ACCESS_KEY) instead of being read from a Secret, so they went stale when the Secret
     was rotated - and they leak into the startup logs.
     evidence:
       • [pod.status] container "orders" waiting: CrashLoopBackOff, restarts=9, last exit 1
         (application error)
       • [events] Warning/BackOff x41: Back-off restarting failed container orders in pod
         orders-api-6c8d9b7f45-tq9zp_shop(2c7e9f1a-4d3b-4b8e-9f6a-1e2d3c4b5a69)
       • [logs] line 17: asyncpg.exceptions.InvalidPasswordError: password authentication failed for
         user "orders_svc"
       • [manifest] credentials hard-coded as plaintext env values in deployment.yaml: DATABASE_URL,
         AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
       • [secrets] Secret orders-db-credentials holds key 'database-url' matching DATABASE_URL
         (rotated 2026-11-04T13:45:00Z)
       • [secrets] Secret orders-aws-credentials holds key 'aws-access-key-id' matching
         AWS_ACCESS_KEY_ID
       • [secrets] Secret orders-aws-credentials holds key 'aws-secret-access-key' matching
         AWS_SECRET_ACCESS_KEY
▶ 3. Exact fix          1 patch → deployment.yaml
     --- a/deployment.yaml
     +++ b/deployment.yaml
     @@ -23,11 +23,20 @@
                containerPort: 8000
              env:
              - name: DATABASE_URL
     -          value: [REDACTED:k8s_env_secret]
     +          valueFrom:
     +            secretKeyRef:
     +              name: orders-db-credentials
     +              key: database-url
              - name: AWS_ACCESS_KEY_ID
     -          value: [REDACTED:aws_access_key_id]
     +          valueFrom:
     +            secretKeyRef:
     +              name: orders-aws-credentials
     +              key: aws-access-key-id
              - name: AWS_SECRET_ACCESS_KEY
     -          value: [REDACTED:k8s_env_secret]
     +          valueFrom:
     +            secretKeyRef:
     +              name: orders-aws-credentials
     +              key: aws-secret-access-key
              - name: S3_INVOICE_BUCKET
                value: acme-shop-invoices
              - name: LOG_LEVEL
     1. Replace the plaintext env values with secretKeyRef references (patch below) so the Pod
        always reads the current, rotated credential.
     2. Rotate every credential that appeared in the logs (database password, cloud keys) - they
        must be treated as compromised.
     3. Stop logging configuration values at startup; log only key names.
▶ 4. Zero-trust policy  agent-proposed actions
     ✔ SAFE_READ           kubectl logs orders-api-6c8d9b7f45-tq9zp -n shop --previous
     ⚠ APPROVAL_REQUIRED   kubectl get secret orders-db-credentials -n shop -o jsonpath='{.data.password}'
                           └ reads Secret values (-o yaml/json/jsonpath decodes credentials)
     ✖ BLOCKED_DESTRUCTIVE kubectl delete secret orders-db-credentials -n shop; kubectl apply -f old-secret.yaml
                           └ shell operator ';' (pipes, chaining, redirection) is not allowed; submit one command
▶ 5. AI review (mock)   AGREE - redacted payload only
     3 independent signals corroborate CrashLoopBackOff (db auth failure); the patch is minimal and
     reversible.
     risk: 3 credentials leaked into runtime logs - rotate before rolling out the fix.
     risk: Restarting pods will not help until the underlying startup error is fixed.

━━━ [3/4] service_port_mismatch ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
checkout-frontend returns 502 for every request although pods are healthy
incident KG-1D8ACCD3 · pod web/checkout-frontend-58f6d7c9b4-hn7wz · deployment/checkout-frontend

▶ 1. Secret firewall    2 secrets blocked before analysis (email ×1, secret_assignment ×1)
     logs.txt:5         [2026-11-04T16:20:00.915Z] debug: SESSION_SECRET=[REDACTED:secret_assignment]
     logs.txt:10        [2026-11-04T16:22:03.004Z] info: password reset e-mail queued for [REDACTED:email]
▶ 2. Diagnosis          ServicePortMismatch · confidence 95% · SEV-1
     root cause: Service checkout-frontend forwards port 80 to targetPort 8080, but the container
     listens on 3000. kube-proxy sends traffic to a closed port, so clients get connection refused /
     502s while the pods themselves look healthy.
     evidence:
       • [service] service.yaml: port 80 -> targetPort 8080, but containers expose http=3000
       • [logs] application reports it is listening on port 3000
       • [pod.status] pods are Running and Ready - traffic fails only through the Service
       • [endpoints] Endpoints checkout-frontend route to port(s) [8080]
▶ 3. Exact fix          1 patch → service.yaml
     --- a/service.yaml
     +++ b/service.yaml
     @@ -12,5 +12,5 @@
        ports:
          - name: http
            port: 80
     -      targetPort: 8080 # legacy Express server port (pre Node 22 migration)
     +      targetPort: http
            protocol: TCP
     1. Set targetPort of Service checkout-frontend port 80 to http (3000) (patch below).
     2. Referencing the named container port keeps Service and Deployment in sync.
▶ 4. Zero-trust policy  agent-proposed actions
     ✔ SAFE_READ           kubectl get endpoints checkout-frontend -n web -o wide
     ⚠ APPROVAL_REQUIRED   kubectl port-forward svc/checkout-frontend 8080:80 -n web
                           └ opens a network tunnel into the cluster
     ✖ BLOCKED_DESTRUCTIVE kubectl replace --force -f service.yaml -n web
                           └ replace --force deletes and recreates objects
▶ 5. AI review (mock)   AGREE - redacted payload only
     3 independent signals corroborate ServicePortMismatch (target port); the patch is minimal and
     reversible.
     risk: 1 credential leaked into runtime logs - rotate before rolling out the fix.
     risk: Ingress/NetworkPolicy objects that reference the old port number must be updated
           together.

━━━ [4/4] image_pull_backoff ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
ingest-worker stuck in ImagePullBackOff after the registry migration
incident KG-A14B00E1 · pod data/ingest-worker-5f7c9d8b6-m4n2x · deployment/ingest-worker

▶ 1. Secret firewall    4 secrets blocked before analysis (k8s_secret_data ×2, kubeconfig_token ×2)
     ci-deploy.log:11   client-key-data: [REDACTED:kubeconfig_token]
     ci-deploy.log:12   token: [REDACTED:kubeconfig_token]
     secrets.json:11    ".dockerconfigjson": "[REDACTED:k8s_secret_data]"
     secrets.json:19    "data": {"api-token": "[REDACTED:k8s_secret_data]"}
▶ 2. Diagnosis          ImagePullBackOff · confidence 99% · SEV-1
     root cause: The kubelet is not authorised to pull
     registry.acme.internal:5000/data/ingest-worker:2.8.0 from the private registry
     registry.acme.internal:5000 (401/denied). The pod spec has no imagePullSecrets, so the pull is
     anonymous.
     evidence:
       • [pod.status] container "worker" waiting: ImagePullBackOff - Back-off pulling image
         "registry.acme.internal:5000/data/ingest-worker:2.8.0"
       • [events] Warning/Failed: Failed to pull image
         "registry.acme.internal:5000/data/ingest-worker:2.8.0": failed to pull and unpack image
         "registry.acme.internal:5000/data/ingest-worker:2.8.0": failed to resolve reference
         "registry.acme.internal:50…
       • [events] Normal/BackOff x23: Back-off pulling image
         "registry.acme.internal:5000/data/ingest-worker:2.8.0"
       • [secrets] dockerconfigjson Secret acme-registry-creds exists for
         registry.acme.internal:5000
▶ 3. Exact fix          1 patch → deployment.yaml
     --- a/deployment.yaml
     +++ b/deployment.yaml
     @@ -26,3 +26,5 @@
                    memory: 512Mi
                  limits:
                    memory: 1Gi
     +      imagePullSecrets:
     +        - name: acme-registry-creds
     1. Attach the existing pull secret acme-registry-creds to the pod template (patch below).
▶ 4. Zero-trust policy  agent-proposed actions
     ✔ SAFE_READ           kubectl describe pod ingest-worker-5f7c9d8b6-m4n2x -n data
     ⚠ APPROVAL_REQUIRED   kubectl patch serviceaccount default -n data -p '{"imagePullSecrets": [{"name": "acme-registry-creds"}]}'
                           └ 'patch' changes cluster state
     ✖ BLOCKED_DESTRUCTIVE kubectl create clusterrolebinding ingest-admin --clusterrole=cluster-admin --serviceaccount=data:default
                           └ grants 'cluster-admin' via clusterrolebinding (privilege escalation)
▶ 5. AI review (mock)   AGREE - redacted payload only
     3 independent signals corroborate ImagePullBackOff (registry auth); the patch is minimal and
     reversible.
     risk: 2 credentials leaked into runtime logs - rotate before rolling out the fix.
     risk: Pull secrets are namespace-scoped; every namespace that runs the image needs its own
           copy.

━━━ Summary ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

  #  scenario                root cause                   conf  sev    secrets  fix
  1  oom_killed              ✔ OOMKilled                   99%  SEV-2        2  deployment.yaml, Dockerfile
  2  crashloop_secret_leak   ✔ CrashLoopBackOff            99%  SEV-1       13  deployment.yaml
  3  service_port_mismatch   ✔ ServicePortMismatch         95%  SEV-1        2  service.yaml
  4  image_pull_backoff      ✔ ImagePullBackOff            99%  SEV-1        4  deployment.yaml

  4/4 incidents diagnosed correctly · 21 secrets blocked · 0 network calls · 0 API keys · no LLM required

🚨 SRE Incident Report — KG-D509530D

payments-api restarts every few minutes under settlement load

A Spring Boot service was upgraded to v3.4.1 (new in-memory settlement batch). Pods restart with exit code 137 roughly every 3 minutes and the on-call sees CrashLoopBackOff.

Field

Value

Incident ID

KG-D509530D

Severity

SEV-2

Status

Root cause identified — fix proposed

Primary failure

OOMKilled · confidence 99%

Workload

deployment/payments-api in namespace payments

Pod

payments-api-7d9f8b6c5d-x2k4q

First signal

2026-11-04T09:12:03Z

Secret firewall

2 values redacted (email ×1, jwt ×1)

Engine

kubeguard-mcp v1.0.0 · deterministic · offline

Summary

JVM max heap -Xmx768m (768Mi) is 150% of the container memory limit (512Mi). Heap + metaspace + thread stacks + off-heap buffers exceed the cgroup limit, so the kernel OOM killer SIGKILLs the process (exit 137) before the runtime can even throw an OutOfMemoryError.

Root cause analysis

Container "api" is being OOMKilled — OOMKilled / heap_exceeds_limit — confidence 99%

Signal source

Evidence

pod.status

container "api" lastState.terminated.reason=OOMKilled (exitCode=137, restarts=6)

events

Warning/OOMKilling: Memory cgroup out of memory: Killed process 4127 (java) total-vm:3914212kB, anon-rss:521436kB, file-rss:18220kB, shmem-rss:0kB, UID:10001 pgtables:1444kB oom_s…

logs

line 1: Picked up JAVA_TOOL_OPTIONS: -Xmx768m -XX:+UseG1GC -XX:+ExitOnOutOfMemoryError

dockerfile

-Xmx768m set in Dockerfile, but resources.limits.memory=512Mi

Remediation

  1. Raise resources.limits.memory of container "api" from 512Mi to 1Gi.

  2. Replace -Xmx768m with -XX:MaxRAMPercentage=75.0 in Dockerfile so the heap tracks the container limit instead of a hard-coded size.

  3. After the rollout, confirm the working set stays below ~80% of the limit (kubectl top) and consider a VerticalPodAutoscaler in recommendation mode.

Proposed patch — deployment.yaml

memory limit 512Mi -> 1Gi

--- a/deployment.yaml
+++ b/deployment.yaml
@@ -33,7 +33,7 @@
               memory: 256Mi
             limits:
               cpu: "1"
-              memory: 512Mi
+              memory: 1Gi
           readinessProbe:
             httpGet:
               path: /actuator/health/readiness
Proposed patch — Dockerfile

replace -Xmx768m with a limit-relative heap

--- a/Dockerfile
+++ b/Dockerfile
@@ -1,7 +1,7 @@
 FROM eclipse-temurin:21-jre-alpine
 WORKDIR /app
 COPY target/payments-api.jar /app/app.jar
-ENV JAVA_TOOL_OPTIONS="-Xmx768m -XX:+UseG1GC -XX:+ExitOnOutOfMemoryError"
+ENV JAVA_TOOL_OPTIONS="-XX:MaxRAMPercentage=75.0 -XX:+UseG1GC -XX:+ExitOnOutOfMemoryError"
 USER 10001
 EXPOSE 8080
 ENTRYPOINT ["java", "-jar", "/app/app.jar"]

Secondary findings

  • CrashLoopBackOff (54%): Container "api" is crash-looping — symptom of OOMKilled

Zero-trust command plan

Every command was classified by the policy engine; KubeGuard never executes anything itself.

#

Command

Verdict

Purpose

1

kubectl describe pod payments-api-7d9f8b6c5d-x2k4q -n payments

🟢 SAFE_READ

confirm Last State: OOMKilled

2

kubectl top pod payments-api-7d9f8b6c5d-x2k4q -n payments --containers

🟢 SAFE_READ

observe live memory usage

3

kubectl logs payments-api-7d9f8b6c5d-x2k4q -n payments -c api --previous

🟢 SAFE_READ

logs from the killed container

4

kubectl apply -f deployment.yaml -n payments

🟡 APPROVAL_REQUIRED

roll out the patched manifest

5

kubectl rollout status deployment/payments-api -n payments

🟢 SAFE_READ

watch the rollout

Agent guardrail check

Actions an autonomous agent proposed for this incident, as judged by the zero-trust policy:

Proposed action

Verdict

Risk

Reason

kubectl top pod payments-api-7d9f8b6c5d-x2k4q -n payments --containers

🟢 SAFE_READ

0

read-only command

kubectl set resources deployment/payments-api -n payments -c api --limits=memory=1Gi

🟡 APPROVAL_REQUIRED

45

'set' changes cluster state

kubectl delete pods --all -n payments

🔴 BLOCKED_DESTRUCTIVE

100

delete --all removes every matching object

Timeline (UTC)

Time

Source

Type

Reason

Message

2026-11-04T09:12:03Z

event

Normal

Scheduled

Successfully assigned payments/payments-api-7d9f8b6c5d-x2k4q to ip-10-0-3-17.eu-west-1.compute.internal

2026-11-04T09:31:09Z

event

Normal

Pulled

Container image "ghcr.io/acme/payments-api:3.4.1" already present on machine (x7)

2026-11-04T09:31:10Z

event

Normal

Started

Started container api (x7)

2026-11-04T09:33:52Z

event

Warning

OOMKilling

Memory cgroup out of memory: Killed process 4127 (java) total-vm:3914212kB, anon-rss:521436kB, file-rss:18220kB, shmem-rss:0kB, UID:10001 pgtables:1444kB oom_score_adj:983 (x6)

2026-11-04T09:33:52Z

container

Warning

OOMKilled

container "api" terminated (exit 137, restarts=6)

2026-11-04T09:34:20Z

event

Warning

BackOff

Back-off restarting failed container api in pod payments-api-7d9f8b6c5d-x2k4q_payments(6f1c2a9e-2b7d-4c1e-9a43-0d8e5b7f21c4) (x14)

Secret firewall audit

File

Redactions

Max severity

Rules

logs.txt

2

high

email ×1, jwt ×1

⚠️ Security impact: 1 credential written to runtime logs (jwt ×1). Anything that reached a log pipeline must be treated as compromised: rotate it, then purge the log streams.

Values were masked before analysis and before any output, report or LLM call.

Generated by kubeguard-mcp v1.0.0 · deterministic correlation · zero network calls · every line passed the secret firewall.

Run python scripts/demo_offline.py without --no-report to also print the consolidated Markdown SRE incident report for all four incidents, or --report out/incident-report.md to write it to a file.

🏗️ Architecture

   MCP clients                                       You / CI / shell
   Claude Desktop, Claude Code, Cursor, ...          kubeguard CLI, scripts/demo_offline.py
             |  JSON-RPC 2.0 over stdio                         |
             v                                                  v
   +---------------------------+                   +---------------------------+
   | mcp_server.py             |                   | cli.py                    |
   | 5 read-only tools,        |                   | demo  diagnose  redact    |
   | resources, prompts        |                   | audit  check  serve-mcp   |
   +-------------+-------------+                   +-------------+-------------+
                 |                                               |
                 +-----------------------+-----------------------+
                                         |
   pod.json                              v
   events.json         +---------------------------+
   logs.txt  ------->  | redactor.py   (ingress)   |  secret & PII firewall:
   *.yaml / *.json     | AWS, JWT, DB URLs, .env,  |  masks before any detector
   Dockerfile          | kubeconfig, Secrets, PII  |  reads the data
   docker inspect      +-------------+-------------+
                                     |
                                     v
                       +---------------------------+      +---------------------------+
                       | correlator.py             | ---> | policy.py                 |
                       | 6 detectors, weighted     |      | SAFE_READ                 |
                       | evidence, confidence      |      | APPROVAL_REQUIRED         |
                       +-------------+-------------+      | BLOCKED_DESTRUCTIVE       |
                                     |                    +---------------------------+
                                     v
                       +---------------------------+      +---------------------------+
                       | patcher.py + yaml_lite.py |      | auditor.py                |
                       | exact, comment-preserving |      | validate_manifest         |
                       | YAML/JSON/Dockerfile diff |      | audit_cluster_risks, RBAC |
                       +-------------+-------------+      +---------------------------+
                                     |
                                     v
                       +---------------------------+      +---------------------------+
                       | report.py                 | <--- | llm_reviewer.py           |
                       | Markdown SRE report,      |      | optional second opinion   |
                       | console, JSON             |      | mock / groq / gemini /    |
                       +-------------+-------------+      | anthropic (redacted only) |
                                     |                    +---------------------------+
                                     v
                       +---------------------------+
                       | redactor.py   (egress)    |
                       +-------------+-------------+
                                     |
                                     v
             terminal  |  report.md  |  MCP response  |  LLM prompt

  1. Ingress firewall — redactor.py masks secrets in logs/events before any detector reads them.

  2. Correlate — correlator.py normalises Pod status, Events (v1 and events.k8s.io/v1), logs, Deployments, Services, Secrets/ConfigMaps inventories, Nodes, Endpoints, docker inspect and docker-compose, then runs six detectors that each produce weighted evidence and a confidence score.

  3. Patch — patcher.py turns path-addressed edits into surgical line edits using the spans recorded by yaml_lite.py, so the diff touches only what must change (comments, key order and quoting survive).

  4. Classify — every suggested command (and every action an agent proposes) goes through policy.py.

  5. Egress firewall — reports, diffs, MCP responses and LLM prompts are redacted again on the way out.

🔌 Connect Claude Desktop, Claude Code or Cursor

The MCP server speaks newline-delimited JSON-RPC 2.0 over stdio (protocol 2025-06-18, also negotiates 2025-03-26 and 2024-11-05). It holds no cluster credentials and never executes commands — your agent collects data with read-only kubectl calls and KubeGuard analyses it.

Claude Desktop — claude_desktop_config.json (after pip install .):

{
  "mcpServers": {
    "kubeguard": {
      "command": "kubeguard",
      "args": ["serve-mcp"],
      "env": { "PYTHONUTF8": "1" }
    }
  }
}

Zero-install variant straight from a clone (allows diagnose_pod to read bundles under --roots):

{
  "mcpServers": {
    "kubeguard": {
      "command": "python",
      "args": ["/absolute/path/to/KubeGuard-MCP/scripts/kubeguard.py", "serve-mcp", "--roots", "/absolute/path/to/incidents"]
    }
  }
}

Cursor — the same mcpServers block in ~/.cursor/mcp.json (global) or .cursor/mcp.json (per project).

Claude Code:

claude mcp add kubeguard -- kubeguard serve-mcp
claude mcp add kubeguard -- uvx --from git+https://github.com/Lokesh-Guttikonda/KubeGuard-MCP kubeguard serve-mcp

Then ask: "Pod payments-api-7d9f8b6c5d-x2k4q in payments keeps restarting — triage it with KubeGuard." The bundled triage_pod prompt tells the agent which SAFE_READ commands to run and to check every command it proposes with classify_command.

🧰 MCP tools

All tools are annotated readOnlyHint: true, destructiveHint: false, openWorldHint: false, and every result passes the egress firewall.

Tool

Input

Output

diagnose_pod

pod, events, logs, manifests, dockerfile, secrets, nodes, endpoints, docker_inspect (objects or JSON/YAML text) — or example, or bundle_path (restricted to KUBEGUARD_MCP_ROOTS)

Markdown SRE report (or JSON) + structuredContent with diagnoses, evidence, patches, command plan, timeline, redaction audit

redact_logs

text, include_pii, redact_ips

redacted text + audit (counts per rule, line numbers — never values)

validate_manifest

manifest, filename

findings with rule IDs, severities and line numbers, 0–100 score, grade

audit_cluster_risks

manifests (text or {file: text}), commands, role

0–100 risk score, top risks, RBAC escalation paths, secret exposure, per-command verdicts

classify_command

command, role, approved

verdict, reasons, risk score, whether the role may run it

Also exposed: resources kubeguard://policy/verdicts and kubeguard://examples/<scenario>, and the triage_pod prompt.

⌨️ CLI

Command

What it does

kubeguard demo [-s NAME] [--report FILE] [--no-report] [--json]

the four offline incidents end-to-end

kubeguard diagnose DIR | --example NAME | --pod F --events F --logs F -m F …

correlate a bundle; --format text|markdown|json, --patch-out fix.diff, --write-fixed out/, --review

kubeguard redact [FILE…]

firewall for files or stdin; --audit, --json, --no-pii, --ips, --fail-on-secrets (CI gate)

kubeguard audit PATH… [-c CMD…]

lint manifests + RBAC + command policy; --fail-on high for CI

kubeguard check <command>

zero-trust verdict; exit 0 safe, 3 needs approval, 4 blocked

kubeguard serve-mcp [--roots DIR…]

start the MCP server on stdio

From a source checkout every command also works as python scripts/kubeguard.py … or python -m src ….

kubectl logs deploy/api --previous | kubeguard redact --audit > safe.log
kubeguard check kubectl delete pods --all -n prod
kubeguard audit k8s/ --fail-on high

🔬 What gets diagnosed — and fixed

Failure

Signals correlated

Sub-causes

Deterministic fix

OOMKilled

lastState.terminated.reason, exit 137, node OOMKilling events, OOM log lines, JVM -Xmx / Node --max-old-space-size (env, args, Dockerfile, logs) vs. the limit, docker inspect State.OOMKilled

heap_exceeds_limit, limit_too_low

raise limits.memory (Guaranteed-QoS aware) or add limits; -Xmx → -XX:MaxRAMPercentage=75.0 in the manifest or Dockerfile; docker-compose mem_limit

CrashLoopBackOff

waiting reason, BackOff events, restart count, exit-code semantics, logs

db_auth_failure, missing_config, liveness_probe, exec_format, command_not_found, permission_denied, application_error, oom (symptom)

plaintext env → secretKeyRef matched to (rotated) Secret keys; inject missing env from a Secret/ConfigMap; fix probe port or initialDelaySeconds from the measured startup time

ImagePullBackOff / ErrImagePull

waiting reason, pull events, dockerconfigjson Secrets (registry hosts decoded)

registry_auth, tag_not_found, invalid_name, rate_limited, registry_unreachable

attach the pull secret for that registry; roll back to the last known-good tag; lowercase the repository

CreateContainerConfigError

waiting message, events, Secret/ConfigMap inventories

missing_key, missing_secret, missing_configmap, run_as_non_root

fix key/object-name typos (env, envFrom, volumes); runAsUser: 10001 + USER in the Dockerfile

Unschedulable

PodScheduled=False, FailedScheduling breakdown, node allocatable & labels

insufficient_resources, node_selector_mismatch, untolerated_taint, unbound_pvc, too_many_pods

right-size requests to fit the largest node; fix nodeSelector value/key typos

ServicePortMismatch

Service selector/targetPort vs. template labels & containerPorts, "listening on" log lines, Endpoints, pod readiness

target_port, selector_mismatch

point targetPort at the named container port; align the selector

Root causes outrank symptoms (a crash loop caused by OOM is reported as a symptom of OOMKilled), results are fully deterministic (same bundle → same incident ID, report and patch), and no wall-clock time appears in output.

🔥 Secret & PII firewall

Precompiled rules with substring pre-filters, a case-folded fast path and a single splice pass (~6 MB/s of realistic logs on one core). Rules run in priority order and a generic rule can never overlap a span claimed by a specific one, so postgres://app:hunter2@db becomes postgres://app:[REDACTED:db_connection_string]@db — the host and user stay visible for debugging. Redaction is idempotent and every finding is audited by rule, category, severity, line and column — never by value.

Rule

Category

Severity

Detects

private_key

secret

critical

PEM / OpenSSH / PGP private key blocks

aws_access_key_id · aws_secret_access_key · aws_session_token

secret

critical

AWS credentials (env, JSON, CLI output)

kubeconfig_token

secret

critical

kubeconfig token, id-token, client-key-data …

jwt · bearer_token · basic_auth

secret

high

JWTs (incl. service-account tokens), Authorization headers

db_connection_string · url_credentials · url_param_secret

secret

critical / high

passwords in Postgres/MySQL/Mongo/Redis/AMQP/… URLs, user:pass@host, JDBC password=

github_token · gitlab_token · slack_token · stripe_key · google_api_key · llm_api_key

secret

high / critical

provider API keys (incl. Anthropic, OpenAI, Groq)

docker_auth

secret

critical

registry auth inside docker config.json

k8s_env_secret

secret

critical

- name: DB_PASSWORD / value: … (YAML, JSON — even across diff hunks)

k8s_secret_data

secret

critical

every value under data/stringData of kind: Secret (structural pass, YAML & JSON)

json_secret_field · secret_assignment

secret

high

"client_secret": "…", .env KEY=value, password: …

credit_card (Luhn) · us_ssn · email

pii

high / medium

personal data (--no-pii to keep)

ipv4

pii

low

opt-in (--ips) — pod IPs are usually useful for triage

False positives are actively suppressed: secretName:, DB_PASSWORD_FILE=, max_tokens=1024, InvalidPasswordError: password authentication failed, dsn=settings.database_url, ${VAR} placeholders, git@github.com: and in-cluster *.svc.cluster.local addresses are left alone.

🚦 Zero-trust policy engine

Verdict

Meaning

Examples

🟢 SAFE_READ

read-only and non-sensitive; may run unattended

kubectl get/describe/logs/top/events, rollout status, auth can-i, docker ps/logs/inspect, helm status/get/template

🟡 APPROVAL_REQUIRED

mutates state, opens a shell/tunnel, or reads secret values

kubectl apply/patch/scale/rollout restart/exec/port-forward, delete pod <name>, get secret -o yaml, config view --raw, docker run, helm upgrade

🔴 BLOCKED_DESTRUCTIVE

irreversible, bulk, privilege-escalating or injected

delete ns/node/pv/pvc/crd, delete --all/-l/-f, --force --grace-period=0, drain, NoExecute taints, replace --force, apply --prune, cluster-admin bindings, impersonating system:masters, node debugging, mutations in kube-system, docker system/volume prune, --privileged, --pid=host, docker.sock mounts, helm uninstall, any ; && | > $(…) or non-allowlisted binary

Roles (--role / KUBEGUARD_ROLE): viewer may only run SAFE_READ; operator may run APPROVAL_REQUIRED once a human approved it; admin may additionally break glass on BLOCKED_DESTRUCTIVE — only with approval and a written justification, which is flagged in the audit trail. analyze_rbac() statically flags wildcard grants, Secret readers, pods/exec, escalate/bind/impersonate, token minting, webhook control, cluster-admin and public-group bindings.

🤖 Optional LLM reviewer

The deterministic diagnosis is always the answer; an LLM can add a second opinion. The payload is built from the already-redacted result and scanned again right before egress.

PROVIDER

Key

Notes

mock (default)

—

offline, deterministic, free

groq

GROQ_API_KEY

OpenAI-compatible chat completions

gemini

GEMINI_API_KEY or GOOGLE_API_KEY

key sent in a header, never in the URL

anthropic

ANTHROPIC_API_KEY

Messages API

KUBEGUARD_LLM_MODEL, KUBEGUARD_LLM_BASE_URL and KUBEGUARD_LLM_TIMEOUT override the defaults. Uses httpx when installed (pip install ".[llm]") and the stdlib urllib otherwise. Failures never break triage: the review is reported as UNAVAILABLE.

PROVIDER=groq GROQ_API_KEY=... kubeguard diagnose examples/oom_killed --review

🐍 Python API

from src import correlate, Redactor, PolicyEngine  # `from kubeguard import ...` once installed
from src.correlator import redacted_diff

result = correlate("examples/oom_killed")
print(result.primary.failure.value, result.primary.confidence)  # OOMKilled 0.99
for patch in result.primary.patches:
    print(redacted_diff(patch))  # exact, secret-free unified diff

print(Redactor().redact("DB_PASSWORD=hunter2").text)  # DB_PASSWORD=[REDACTED:secret_assignment]
print(PolicyEngine(role="viewer").authorize("kubectl delete ns prod").allowed)  # False

IncidentBundle.from_inputs(pod=..., events=..., logs=..., manifests=...) accepts dicts or JSON/YAML text, so it plugs directly into your own tooling.

🧪 Testing & quality

pip install pytest
python -m pytest -q

500+ tests cover every module: the YAML engine (round-trips, spans, error lines), the patcher, every redaction rule plus false-positive suppression and a 1 MB throughput budget, 100+ policy classifications and the RBAC analyser, all six detectors and their sub-causes, reports, the LLM adapters (fake transports + a real local HTTP server), the full MCP JSON-RPC surface (lifecycle, batching, error codes, notifications, resources, prompts, a real stdio subprocess), every CLI command and exit code, and Windows UTF-8 safety (subprocesses under cp1252 and ascii code pages, isolated python -I runs proving zero dependencies, CRLF manifests, byte-exact MCP UTF-8 under any locale). CI runs them on Linux, macOS and Windows for Python 3.9 – 3.14, plus ruff, a zero-dependency demo job, a wheel build + install smoke test and the README freshness check.

🔒 Security model

  • No credentials, no execution. KubeGuard analyses data you give it; it never talks to a cluster, a registry or the internet (except the opt-in LLM reviewer).

  • Two firewalls. Secrets are masked on ingress (before detectors run) and again on egress (every report, diff, MCP result and LLM prompt).

  • Read-only MCP surface. Tools cannot mutate anything; bundle_path is confined to KUBEGUARD_MCP_ROOTS (default: the server's working directory); stdout carries only JSON-RPC, diagnostics go to stderr.

  • Default deny. Unknown binaries are blocked, unknown verbs need approval, shell metacharacters are rejected.

  • Patches stay local. --write-fixed writes the patched files to a directory you choose; displayed diffs are always redacted (the CLI warns if a hunk contained a masked value).

📁 Project layout

src/                     # installed as the `kubeguard` package
  redactor.py            # secret & PII firewall
  policy.py              # zero-trust command policy + RBAC analyser
  correlator.py          # incident bundle loader + 6 deterministic detectors
  patcher.py             # comment-preserving YAML/JSON/Dockerfile edits -> unified diff
  yaml_lite.py           # stdlib YAML subset parser/dumper with line spans
  auditor.py             # validate_manifest / audit_cluster
  report.py              # Markdown SRE report, console, JSON renderers
  llm_reviewer.py        # PROVIDER=mock|groq|gemini|anthropic (redacted-only)
  mcp_server.py          # JSON-RPC 2.0 stdio MCP server
  cli.py                 # `kubeguard` command line
  console.py             # Unicode/ASCII-safe terminal output
  demo.py, scenarios.py  # offline demo engine + scenario discovery
examples/                # 4 realistic incident bundles (pod.json, events.json, logs, YAML, Dockerfile, ...)
scripts/                 # demo_offline.py, kubeguard.py (zero-install launcher), update_readme.py
tests/                   # pytest suite

License

MIT © 2026 Lokesh Guttikonda

This server cannot be deployed

Claude Code, inside the Godot 4 editorFlockbay puts an AI agent in the real Godot 4 editor: it writes GDScript, runs your game and fixes what breaks. Free, or bring your own Claude Code or Codex.flockbay.comAd

Maintenance

ActivityMaintained
ResponsivenessSyncing

Resources

Related MCP Connectors

Related MCP Servers

  • Debugging MCP Server

    luischang07
    A
    license
    A
    quality
    D
    maintenance
    Enables LLMs to automatically diagnose coding errors through codebase search, test execution, and live debugger integration (DAP/V8 CDP). Provides a secure, policy-gated environment for investigating failures while preventing destructive operations.
    9
    ISC

  • K8gentS

    JDoornink
    F
    license
    Not graded
    quality
    D
    maintenance
    A Kubernetes diagnostic agent that provides on-demand root cause analysis and human-in-the-loop remediation via Slack, using LLM reasoning with OPA-bounded security controls.
    1
    -

  • mimiops-mcp

    sergelogvinov
    A
    license
    Not graded
    quality
    A
    maintenance
    Enables AI agents to safely observe and troubleshoot Kubernetes workloads, with limited recovery actions like pod deletion and Helm rollbacks, while preventing dangerous modifications.
    18
    Apache 2.0

  • OpsContext

    FASTPROD
    A
    license
    A
    quality
    B
    maintenance
    Enables AI coding agents to access live operational context like processes, containers, and git state, while providing a tamper-evident audit log and policy-as-code compliance hooks locally.
    22
    301 npm
    Business Source 1.1