nvidia-nvsentinel-mcp
Operates against a Kubernetes lab cluster, providing tools to inspect nodes, pods, and workloads, query NVSentinel component logs and node events, wait for cluster stages like 'cordoned', and run end-to-end fault scenarios that exercise cordon, drain, and workload eviction.
Integrates with NVIDIA NVSentinel, the GPU fault detection and remediation system, exposing tools to read reported node and GPU health conditions, inject or clear simulated GPU faults such as XID 95, and verify that NVSentinel cordons and drains the affected node.
This server does not currently have a deployable release on Glama. Follow the repository's setup instructions to install and configure it for your MCP client, including any required credentials and local dependencies.
nvidia-nvsentinel-mcp
An MCP server for testing NVIDIA NVSentinel (GPU fault detection and remediation on Kubernetes) on a lab cluster, written in Go like NVSentinel itself. An MCP client such as Claude Code can read what NVSentinel reports, inject a fault into a simulated GPU, and check that NVSentinel cordons and drains the node.
This repo holds the server and example test results. It does not install NVSentinel: that is omerfsen.aicr, on a cluster from omerfsen.nvke or on kind. It is meant to go upstream one day: see docs/upstream.md.
Test results
Real output from NVSentinel v1.30.0 on kind, through this server over MCP: docs/test-results. These prove NVSentinel's pipeline against simulated DCGM, not real GPU health.
Related MCP server: Azure HPC/AI MCP Server
Standards
This server follows these standards. Each requirement, its status, the test that proves it and the one known gap are in docs/mcp-standards.md, with links to check for newer versions.
Standard | Version |
2026-07-28, and back to 2024-11-05 | |
2025-12-11, valid and published (0.5.1) | |
AGENTS.md, imported by | |
It has not been run through the official conformance suite, which only tests HTTP servers. MCP has no standard for written docs, so docs/tools.md is generated from the server, the common practice.
Tools
Tool | Changes the lab | What it does |
| no | Nodes (cordon, NVSentinel state, failing GPU checks), pods, the workload |
| no | Scenarios, stages, allowed XIDs, log sources |
| no | Polls until a stage holds, such as |
| no | The last lines of a NVSentinel component's logs |
| no | Recent events on the GPU node |
| yes | Injects XID 95 into the simulated GPU |
| yes | Restarts the simulated hostengine, which drops the fault |
| yes | Undoes a DCGM outage and an injected fault |
| yes | Runs a scenario end to end and reports each step |
Scenario | What it checks |
| Inject XID 95: detected, cordoned, workload evicted, drain succeeded. Clear it: healthy, uncordoned, workload back |
| Take the hostengine away: the monitor reports it lost DCGM. Restore it: the condition clears |
Tools that change anything are off unless NVSENTINEL_MCP_ALLOW_CHANGES=true, and
then work only on a kind cluster. Details: docs/security.md.
Every tool's inputs, outputs and annotations: docs/tools.md, generated from what
the server returns for tools/list, as is the raw docs/tools.json. A test fails
if they no longer match the server; make docs regenerates them.
Install
The server needs a kubeconfig. For the kind lab, make lab-token writes one (see
Run the tests on kind). It speaks MCP over stdio, so any MCP client
can start it. All the commands below leave changes off; add
-e NVSENTINEL_MCP_ALLOW_CHANGES=true only for a client that should change the lab.
1. Install the server. Download the binary for your platform from the latest release:
Platform | File |
Linux, x86-64 |
|
Linux, Arm |
|
macOS, Intel |
|
macOS, Apple silicon |
|
Windows, x86-64 |
|
Windows, Arm |
|
On Linux and macOS, set FILE to yours:
FILE=nvidia-nvsentinel-mcp-linux-amd64
BASE=https://github.com/omerfsen/nvidia-nvsentinel-mcp/releases/latest/download
curl -fsSLO "$BASE/$FILE"
curl -fsSL "$BASE/checksums.txt" | grep " $FILE\$" | sha256sum -c - # macOS: shasum -a 256 -c -
gh attestation verify "$FILE" --repo omerfsen/nvidia-nvsentinel-mcp # optional: who built it
install -m 755 "$FILE" ~/.local/bin/nvidia-nvsentinel-mcpOn Windows, in PowerShell, keep the kubeconfig where only you can read it, as the server cannot check that there:
$File = "nvidia-nvsentinel-mcp-windows-amd64.exe"
$Base = "https://github.com/omerfsen/nvidia-nvsentinel-mcp/releases/latest/download"
Invoke-WebRequest "$Base/$File" -OutFile "$HOME\bin\nvidia-nvsentinel-mcp.exe"Each release also has checksums.txt and an SBOM, nvidia-nvsentinel-mcp.sbom.cdx.json. Only
Linux on x86-64 has been run; the others are built but untested. Or install with Go 1.27:
go install github.com/omerfsen/nvidia-nvsentinel-mcp@latest. Or use the image,
docker.io/omerfsen/nvidia-nvsentinel-mcp, which never runs as root and is about 32 MB.
Check it with nvidia-nvsentinel-mcp --version. Run with no flags, it logs one line to stderr and then
waits for an MCP client on stdin, so it looks idle: that is normal, and Ctrl-C stops it. --help lists
the flags, and --debug logs each tool call and cluster detail to stderr.
2. Add it to your client. For Claude Code, with the binary on your PATH:
claude mcp add nvsentinel-lab -- nvidia-nvsentinel-mcpOr give the full path to the binary, such as ~/.local/bin/nvidia-nvsentinel-mcp.
With the image:
claude mcp add nvsentinel-lab -- docker run --rm -i --read-only --cap-drop=ALL \
--security-opt=no-new-privileges --user "$(id -u):$(id -g)" --memory=128m --cpus=1 \
--pids-limit=64 --network=kind \
--volume "$HOME/.kube/nvsentinel-lab-mcp-docker.yaml:/kube/config:ro" \
docker.io/omerfsen/nvidia-nvsentinel-mcp@sha256:cb6fba848870580f77d347d476a2a7003cd336cd6f4807a72928013538d7aed9Other clients take the same command in their MCP config. Use the full path if the binary is not on
the PATH your client sees:
{ "mcpServers": { "nvsentinel-lab": { "command": "nvidia-nvsentinel-mcp" } } }3. Optional: add the skill, which teaches an agent how to use the tools. It holds only instructions, so step 1 and 2 are still needed. Either command works:
gh skill install omerfsen/nvidia-nvsentinel-mcp nvsentinel-lab
npx skills add omerfsen/nvidia-nvsentinel-mcp --skill nvsentinel-labThe GitHub CLI has no gh mcp add as of v2.103.2: gh skill installs skills, and MCP servers
go in through the client. It is listed on the official MCP Registry as
io.github.omerfsen/nvidia-nvsentinel-mcp.
Run the tests on kind
Needs Go, Docker, kubectl, helm, yq, about 10 GiB of free memory, and a checkout of
omerfsen/aicr with make dev done (until
https://github.com/omerfsen/aicr/pull/8 merges, its dev/osen/nvsentinel-latest branch).
make lab-up # the kind cluster, with its own kubeconfig
make lab-install # NVSentinel v1.30.0 through omerfsen.aicr (AICR_DIR=<checkout>)
make lab-fixture # simulated DCGM, a labelled node, a workload, the server's RBAC
make lab-token # a kubeconfig for the server: its own identity, valid 8 hours
make e2e # the tests, through the real binary over MCP; writes docs/test-results
make image e2e-image # the same, with the server in its container
make lab-downTests with no cluster: make lint test.
Use it from Claude Code
From a checkout, for working on the server itself: make build lab-token, then restart Claude
Code in this directory and approve the server from .mcp.json. That one allows changes.
Settings
Variable | Default | What it does |
|
| The one kubeconfig it uses. Must be private ( |
| off | Exactly |
|
| The context changes need. Must start with |
What is not tested
The binaries for anything but Linux on x86-64: they are cross-compiled and not run.
A real GPU. The RTX 4080 SUPER is not on NVSentinel's validated list, and real DCGM health is still to be checked.
AICR's install on NVKE, as opposed to kind.
Remediation. The kind install stops at cordon and drain: AICR's remediation mixin needs metadata-collector, which needs NVML.
The syslog monitor: it reads kernel-transport journal entries, which a kind node cannot safely produce, so it is off on kind.
The image's cosign signature and SBOM attestation by hand: they are attached, and only its build provenance was verified (
gh attestation verify).
Every step, failure and dead end is in docs/build-log.md.
License
Apache-2.0. lab/gpu-spec.yaml and lab/fake-dcgm.yaml come from NVSentinel, under
the same license.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
This server cannot be deployed
Maintenance
Related MCP Connectors
Run GPU acceptance tests on your NVIDIA GPU nodes over SSH: DCGM, GEMM, HBM/PCIe bandwidth, NCCL.
GPU fabric tools: NCCL benchmark analysis, degradation cost, cluster compatibility, XportL facts.
Massed Compute MCP — GPU inventory, VM lifecycle, billing, SSH keys, and setup recipes.
Remote MCP tools for 3-vendor AI workflow diagnostics, HAR sandboxes, 50v audits, and rubrics.
Related MCP Servers
- FlicenseNot gradedqualityDmaintenanceEnables real-time Kubernetes cluster observability and debugging through standardized MCP interface. Provides access to pods, services, nodes, events, and includes built-in tools for cluster health analysis and issue identification.-
- FlicenseNot gradedqualityDmaintenanceEnables users to query Azure HPC/AI Kubernetes clusters for GPU node information and InfiniBand topology details through kubectl commands. Provides tools to list GPU pool nodes with their status and retrieve network topology labels for high-performance computing workloads.-
- FlicenseAqualityDmaintenanceEnables diagnostics and troubleshooting of OpenShift clusters through storage analysis, resource monitoring, GPU utilization tracking, and pod health checks using Prometheus metrics and the oc CLI.11-
- AlicenseNot gradedqualityCmaintenanceEnables operators and MCP clients to perform read-only, multi-cluster Kubernetes diagnostics by correlating deployment inventory, approved Prometheus metrics, firing alerts, and reusable troubleshooting skills without executing cluster commands or modifying workloads. It exposes tools, resources, and prompts for deployment health checks, metric queries, alert inspection, and skill discovery.MIT