Skip to main content
Glama
nvidia-nvsentinel-mcp

nvidia-nvsentinel-mcp

by omerfsen

nvidia-nvsentinel-mcp

An MCP server for testing NVIDIA NVSentinel (GPU fault detection and remediation on Kubernetes) on a lab cluster, written in Go like NVSentinel itself. An MCP client such as Claude Code can read what NVSentinel reports, inject a fault into a simulated GPU, and check that NVSentinel cordons and drains the node.

This repo holds the server and example test results. It does not install NVSentinel: that is omerfsen.aicr, on a cluster from omerfsen.nvke or on kind. It is meant to go upstream one day: see docs/upstream.md.

Test results

Real output from NVSentinel v1.30.0 on kind, through this server over MCP: docs/test-results. These prove NVSentinel's pipeline against simulated DCGM, not real GPU health.

Related MCP server: Azure HPC/AI MCP Server

Standards

This server follows these standards. Each requirement, its status, the test that proves it and the one known gap are in docs/mcp-standards.md, with links to check for newer versions.

Standard

Version

Model Context Protocol

2026-07-28, and back to 2024-11-05

MCP Registry server.json

2025-12-11, valid and published (0.5.1)

AGENTS.md

AGENTS.md, imported by CLAUDE.md

Agent Skills

skills/nvsentinel-lab

It has not been run through the official conformance suite, which only tests HTTP servers. MCP has no standard for written docs, so docs/tools.md is generated from the server, the common practice.

Tools

Tool

Changes the lab

What it does

lab_status

no

Nodes (cordon, NVSentinel state, failing GPU checks), pods, the workload

list_scenarios

no

Scenarios, stages, allowed XIDs, log sources

wait_for_stage

no

Polls until a stage holds, such as cordoned

component_logs

no

The last lines of a NVSentinel component's logs

node_events

no

Recent events on the GPU node

inject_fault

yes

Injects XID 95 into the simulated GPU

clear_fault

yes

Restarts the simulated hostengine, which drops the fault

reset_lab

yes

Undoes a DCGM outage and an injected fault

run_test_scenario

yes

Runs a scenario end to end and reports each step

Scenario

What it checks

xid95

Inject XID 95: detected, cordoned, workload evicted, drain succeeded. Clear it: healthy, uncordoned, workload back

dcgm_outage

Take the hostengine away: the monitor reports it lost DCGM. Restore it: the condition clears

Tools that change anything are off unless NVSENTINEL_MCP_ALLOW_CHANGES=true, and then work only on a kind cluster. Details: docs/security.md.

Every tool's inputs, outputs and annotations: docs/tools.md, generated from what the server returns for tools/list, as is the raw docs/tools.json. A test fails if they no longer match the server; make docs regenerates them.

Install

The server needs a kubeconfig. For the kind lab, make lab-token writes one (see Run the tests on kind). It speaks MCP over stdio, so any MCP client can start it. All the commands below leave changes off; add -e NVSENTINEL_MCP_ALLOW_CHANGES=true only for a client that should change the lab.

1. Install the server. Download the binary for your platform from the latest release:

Platform

File

Linux, x86-64

nvidia-nvsentinel-mcp-linux-amd64

Linux, Arm

nvidia-nvsentinel-mcp-linux-arm64

macOS, Intel

nvidia-nvsentinel-mcp-darwin-amd64

macOS, Apple silicon

nvidia-nvsentinel-mcp-darwin-arm64

Windows, x86-64

nvidia-nvsentinel-mcp-windows-amd64.exe

Windows, Arm

nvidia-nvsentinel-mcp-windows-arm64.exe

On Linux and macOS, set FILE to yours:

FILE=nvidia-nvsentinel-mcp-linux-amd64
BASE=https://github.com/omerfsen/nvidia-nvsentinel-mcp/releases/latest/download
curl -fsSLO "$BASE/$FILE"
curl -fsSL "$BASE/checksums.txt" | grep " $FILE\$" | sha256sum -c -   # macOS: shasum -a 256 -c -
gh attestation verify "$FILE" --repo omerfsen/nvidia-nvsentinel-mcp    # optional: who built it
install -m 755 "$FILE" ~/.local/bin/nvidia-nvsentinel-mcp

On Windows, in PowerShell, keep the kubeconfig where only you can read it, as the server cannot check that there:

$File = "nvidia-nvsentinel-mcp-windows-amd64.exe"
$Base = "https://github.com/omerfsen/nvidia-nvsentinel-mcp/releases/latest/download"
Invoke-WebRequest "$Base/$File" -OutFile "$HOME\bin\nvidia-nvsentinel-mcp.exe"

Each release also has checksums.txt and an SBOM, nvidia-nvsentinel-mcp.sbom.cdx.json. Only Linux on x86-64 has been run; the others are built but untested. Or install with Go 1.27: go install github.com/omerfsen/nvidia-nvsentinel-mcp@latest. Or use the image, docker.io/omerfsen/nvidia-nvsentinel-mcp, which never runs as root and is about 32 MB.

Check it with nvidia-nvsentinel-mcp --version. Run with no flags, it logs one line to stderr and then waits for an MCP client on stdin, so it looks idle: that is normal, and Ctrl-C stops it. --help lists the flags, and --debug logs each tool call and cluster detail to stderr.

2. Add it to your client. For Claude Code, with the binary on your PATH:

claude mcp add nvsentinel-lab -- nvidia-nvsentinel-mcp

Or give the full path to the binary, such as ~/.local/bin/nvidia-nvsentinel-mcp.

With the image:

claude mcp add nvsentinel-lab -- docker run --rm -i --read-only --cap-drop=ALL \
  --security-opt=no-new-privileges --user "$(id -u):$(id -g)" --memory=128m --cpus=1 \
  --pids-limit=64 --network=kind \
  --volume "$HOME/.kube/nvsentinel-lab-mcp-docker.yaml:/kube/config:ro" \
  docker.io/omerfsen/nvidia-nvsentinel-mcp@sha256:cb6fba848870580f77d347d476a2a7003cd336cd6f4807a72928013538d7aed9

Other clients take the same command in their MCP config. Use the full path if the binary is not on the PATH your client sees:

{ "mcpServers": { "nvsentinel-lab": { "command": "nvidia-nvsentinel-mcp" } } }

3. Optional: add the skill, which teaches an agent how to use the tools. It holds only instructions, so step 1 and 2 are still needed. Either command works:

gh skill install omerfsen/nvidia-nvsentinel-mcp nvsentinel-lab
npx skills add omerfsen/nvidia-nvsentinel-mcp --skill nvsentinel-lab

The GitHub CLI has no gh mcp add as of v2.103.2: gh skill installs skills, and MCP servers go in through the client. It is listed on the official MCP Registry as io.github.omerfsen/nvidia-nvsentinel-mcp.

Run the tests on kind

Needs Go, Docker, kubectl, helm, yq, about 10 GiB of free memory, and a checkout of omerfsen/aicr with make dev done (until https://github.com/omerfsen/aicr/pull/8 merges, its dev/osen/nvsentinel-latest branch).

make lab-up           # the kind cluster, with its own kubeconfig
make lab-install      # NVSentinel v1.30.0 through omerfsen.aicr (AICR_DIR=<checkout>)
make lab-fixture      # simulated DCGM, a labelled node, a workload, the server's RBAC
make lab-token        # a kubeconfig for the server: its own identity, valid 8 hours
make e2e              # the tests, through the real binary over MCP; writes docs/test-results
make image e2e-image  # the same, with the server in its container
make lab-down

Tests with no cluster: make lint test.

Use it from Claude Code

From a checkout, for working on the server itself: make build lab-token, then restart Claude Code in this directory and approve the server from .mcp.json. That one allows changes.

Settings

Variable

Default

What it does

NVSENTINEL_MCP_KUBECONFIG

~/.kube/nvsentinel-lab-mcp.yaml (image: /kube/config)

The one kubeconfig it uses. Must be private (chmod 600)

NVSENTINEL_MCP_ALLOW_CHANGES

off

Exactly true allows the change tools, on the lab only

NVSENTINEL_MCP_LAB_CONTEXT

kind-nvsentinel-lab

The context changes need. Must start with kind-

What is not tested

  • The binaries for anything but Linux on x86-64: they are cross-compiled and not run.

  • A real GPU. The RTX 4080 SUPER is not on NVSentinel's validated list, and real DCGM health is still to be checked.

  • AICR's install on NVKE, as opposed to kind.

  • Remediation. The kind install stops at cordon and drain: AICR's remediation mixin needs metadata-collector, which needs NVML.

  • The syslog monitor: it reads kernel-transport journal entries, which a kind node cannot safely produce, so it is off on kind.

  • The image's cosign signature and SBOM attestation by hand: they are attached, and only its build provenance was verified (gh attestation verify).

Every step, failure and dead end is in docs/build-log.md.

License

Apache-2.0. lab/gpu-spec.yaml and lab/fake-dcgm.yaml come from NVSentinel, under the same license.

M8ven Score

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers

    • F
      license
      Not graded
      quality
      D
      maintenance
      Enables real-time Kubernetes cluster observability and debugging through standardized MCP interface. Provides access to pods, services, nodes, events, and includes built-in tools for cluster health analysis and issue identification.
      -
    • F
      license
      Not graded
      quality
      D
      maintenance
      Enables users to query Azure HPC/AI Kubernetes clusters for GPU node information and InfiniBand topology details through kubectl commands. Provides tools to list GPU pool nodes with their status and retrieve network topology labels for high-performance computing workloads.
      -
    • F
      license
      A
      quality
      D
      maintenance
      Enables diagnostics and troubleshooting of OpenShift clusters through storage analysis, resource monitoring, GPU utilization tracking, and pod health checks using Prometheus metrics and the oc CLI.
      11
      -
    • A
      license
      Not graded
      quality
      C
      maintenance
      Enables operators and MCP clients to perform read-only, multi-cluster Kubernetes diagnostics by correlating deployment inventory, approved Prometheus metrics, firing alerts, and reusable troubleshooting skills without executing cluster commands or modifying workloads. It exposes tools, resources, and prompts for deployment health checks, metric queries, alert inspection, and skill discovery.
      MIT