WorkloadTruth

Install • Quickstart • CLI reference • Comparison • FAQ

Classify a GPU workload as TRAINING, INFERENCE, or IDLE from telemetry alone. No code changes to the workload, no self-reported job labels.

Every GPU scheduler in common use today, including run:ai, Slurm, and Kubernetes GPU operators, asks you to declare whether a job is training or inference at submission time. None of them check. WorkloadTruth reads GPU telemetry (utilization, memory pattern, power draw) and answers the question independently, so a mislabeled or misbehaving job doesn't go unnoticed.

Install

# Real NVIDIA GPU telemetry (requires an NVIDIA driver on the host)
pip install "workloadtruth-cli[nvml]"

# Try it without a GPU, using the synthetic backend
pip install workloadtruth-cli

# npm launcher (thin wrapper around the PyPI package, see "Why two registries")
npx workloadtruth-cli --help

Related MCP server: Ingero

Quickstart

# No GPU required. Classify a synthetic "training" telemetry trace.
$ workloadtruth classify --backend synthetic --profile training --samples 10 --interval 0
workload_type : TRAINING
confidence    : 1.00
gpu_index     : 0
samples       : 10 over 9.0s
reasons:
  - avg GPU utilization 87.8% >= training threshold 65.0%
  - low GPU utilization variance (std=3.4) <= training ceiling 15.0
  - memory growing 120.0 MiB/sample >= training threshold 5.0
  - low power-draw variance (std=9.4W) <= training ceiling 25.0W

# Real hardware
$ workloadtruth classify --backend nvml --samples 10 --interval 1 --json

--json on every command switches to machine-readable output for scripts and agents.

Quick summary

• Use it for: catching cost-misallocated GPU jobs (a job billed as low-priority "inference" that's actually running full training) and unauthorized workload changes (an inference endpoint that starts training on live traffic without sign-off)

• What it's not: a compliance or regulatory-audit tool. No regulation currently requires this kind of monitoring, see What WorkloadTruth is not below

• Prior art: builds on and cites arXiv:2606.19262 (ICML 2026), see Relationship to prior research

How classification works

WorkloadTruth currently ships a rule-based classifier only: a set of documented, inspectable thresholds over four features extracted from a telemetry window (average and variance of GPU utilization, memory-growth slope, average and variance of power draw). Every threshold lives as a named constant in src/workloadtruth/classifier/rules.py with a comment explaining its intuition. Nothing is a black box.

--experimental (an ML-based classifier) is present as a flag but fails loudly with an explanation rather than shipping a fake result. arXiv:2606.19262's trained model and dataset were never published, and this project has no NVIDIA GPU in its build environment to collect real training data. An ML classifier ships here only once it's trained on a real, disclosed dataset and independently shown to beat the rule-based baseline, not before.

Benchmark

workloadtruth benchmark measures the classifier's accuracy against documented synthetic GPU telemetry traces (see synthetic_backend.py for the exact profile parameters), both clean and under a deliberate evasion transform that mimics an operator disguising a training job as inference (capping utilization swings, flattening memory growth, injecting artificial idle gaps).

This is run on synthetic data, not live NVIDIA hardware, so it is not directly comparable to arXiv:2606.19262's real-hardware numbers. Both are reported below, side by side, never blended into one figure.

The overall evasion number hides the real finding. Broken out by workload type, WorkloadTruth's rule-based classifier is currently completely fooled on evasive training workloads (0% accuracy), while staying accurate on evasive inference and idle traces (100% each). Capping GPU utilization alone is enough to make a training job read as inference to this classifier today.

This is the concrete, disclosed gap the benchmark suite exists to surface and track, not a footnote. A rule-based classifier that only looks at utilization variance is trivially evadable. Closing this gap, through multi-signal fusion or a real trained classifier once data exists, is the roadmap, not a solved problem. Reproduce it yourself:

workloadtruth benchmark --trials 300 --window 30 --json

CLI reference

$ workloadtruth --help
Usage: workloadtruth [OPTIONS] COMMAND [ARGS]...

  Classify a GPU workload as INFERENCE, TRAINING, or IDLE from telemetry
  alone.

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  benchmark   Run the evasion-robustness benchmark against synthetic...
  classify    One-shot classification of the current GPU workload.
  mcp         Start an MCP server exposing classify/benchmark/verify-log...
  verify-log  Verify the hash chain of a local audit log has not been...
  watch       Continuously classify and append hash-chained entries to...

Every command supports --json. Full flag reference: workloadtruth <command> --help.

MCP Server

WorkloadTruth ships a Model Context Protocol server so an AI agent (Claude, Cursor, or any MCP-compatible client) can classify GPU workloads, run the evasion-robustness benchmark, and verify the audit log directly, without a human invoking the CLI by hand.

Install the extra:

pip install "workloadtruth-cli[mcp]"

Add it to your MCP client's config (for Claude Desktop, claude_desktop_config.json). The server is started via the workloadtruth mcp subcommand, not a separate console script:

{
  "mcpServers": {
    "workloadtruth": {
      "command": "uvx",
      "args": ["--from", "workloadtruth-cli", "workloadtruth", "mcp"]
    }
  }
}

The server exposes three tools over stdio:

• classify_workload(backend="nvml", profile="training", gpu_index=0, samples=10, interval_seconds=1.0, write_to_audit_log=False): samples GPU telemetry and classifies it as TRAINING, INFERENCE, or IDLE. backend is "nvml" (real hardware) or "synthetic" (documented synthetic traces, no GPU required). Optionally appends the result to the hash-chained audit log.

• run_benchmark(trials=50, window=30): runs the evasion-robustness benchmark against synthetic telemetry and returns per-profile accuracy under clean and evasion-obfuscated conditions.

• verify_audit_log(log_file="workloadtruth.log.jsonl"): re-derives the hash chain of a local audit log and reports whether it has been tampered with.

Example call, classifying a synthetic training trace with no GPU required:

classify_workload(backend="synthetic", profile="training", samples=10, interval_seconds=0)
-> {"workload_type": "TRAINING", "confidence": 1.0, "gpu_index": 0, ...}

Transport is stdio, so there is nothing to host: the MCP client spawns workloadtruth mcp as a local subprocess. A .well-known/agent.json manifest is also shipped at the repo root for A2A-style discovery, listing both the CLI and MCP interfaces and the packages that provide them. Source: src/workloadtruth/mcp_server.py.

Audit log

workloadtruth watch appends a hash-chained entry to workloadtruth.log.jsonl on every classification window. Each entry's hash covers its own content plus the previous entry's hash, so any edit, reorder, or deletion after the fact breaks the chain from that point forward. workloadtruth verify-log re-derives every hash and reports the first broken link, if any.

This proves what was classified, when, and that the local record hasn't been silently altered afterward. It does not prove the classification itself was correct, and it is not evidence of regulatory compliance. See below.

Why two registries

WorkloadTruth's implementation is Python. NVML access (pynvml/nvidia-ml-py) is the mature, official way to read NVIDIA GPU telemetry, and it's also what the closest prior art (arXiv:2606.19262) uses. The npm package (workloadtruth-cli) is a thin launcher, not a reimplementation. It locates and execs the real workloadtruth binary installed from PyPI, so npx workloadtruth-cli works for npm-first agent tooling without duplicating the classifier in two languages. The npm package versions independently of the PyPI package since it only ships a launcher script, not the classifier itself.

Comparison

Checked directly against each project's own documentation: DCGM exporter docs, run:ai inference overview, W&B system metrics docs. None of these classify workload type from telemetry alone. That gap is what WorkloadTruth fills.

What is WorkloadTruth, and why does it exist

WorkloadTruth is an open-source command-line tool and MCP server that classifies a running GPU workload as TRAINING, INFERENCE, or IDLE using only GPU-level telemetry (utilization, memory pattern, power draw), with no changes to the workload's own code and no reliance on a self-reported job label.

It exists because every mainstream GPU scheduler asks the job's owner to declare its type at submission time and never checks that declaration against what the hardware is actually doing. That gap has two real consequences: cost misallocation (a job scheduled at low-priority "inference" pricing that is actually running full training) and unauthorized workload changes (an inference endpoint that quietly starts training on live traffic). WorkloadTruth closes that verification gap today, and doubles as the first open, installable implementation of a real academic research thread on verifying AI training runs from hardware telemetry (see below).

Relationship to prior research

WorkloadTruth's core technique, classifying training vs. non-training GPU activity from telemetry, is not novel. It's the direct application of a real, active research thread:

1. Yonadav Shavit (Harvard), "What does it take to catch a Chinchilla?" (2023): proposed hardware-level "training transcripts" for verifying large training runs.

2. GovAI, "Computing Power and the Governance of AI" (2024): surveyed compute-governance mechanisms, explicitly framed as exploratory, not endorsed policy.

3. "Hardware-Enabled Mechanisms for Verifying Responsible AI Development" (2025): hardware-security researchers proposing on-chip attestation.

4. Rahman & Tajdari, "Detecting Hidden ML Training With Zero-Overhead Telemetry" (ICML 2026 Technical AI Governance workshop): a working NVML-telemetry classifier, 98.2% accurate on unobfuscated workloads, the direct prior art for this project's core classification technique.

What WorkloadTruth adds: as of this project's own research (2026-07-19), no open-source, installable implementation of this research thread existed, only academic prototypes. WorkloadTruth is that packaging: a real CLI, an MCP server, a hash-chained audit log, and a reproducible evasion-robustness benchmark, built in the open. It does not claim to improve on the paper's classification technique. See the benchmark section above: the current rule-based classifier is considerably more evadable than the paper's ML approach on the one axis it measures.

What WorkloadTruth is not

• Not a compliance or regulatory-audit tool. No law currently requires inference/training classification or reporting. Any future claim otherwise will name the specific enacted regulation; none exists as of this writing.

• Not a content inspector. WorkloadTruth reads GPU-level signals only (utilization, memory, power). It never inspects model weights, training data, prompts, or completions.

• Not proof of "compliant" or "safe" operation. The audit log proves what was classified and when, and that the record wasn't altered afterward, not that the classification was correct or that any policy was followed.

FAQ

Does this need an NVIDIA GPU? Only for the nvml backend. --backend synthetic runs the full classifier and CLI against documented synthetic traces, no GPU required. Useful for trying the tool or for CI.

Can it classify AMD or Intel GPU workloads? Not yet. The telemetry layer is a pluggable interface (TelemetryBackend) specifically so a new vendor backend (AMD ROCm, Intel Level Zero) can be added without touching the classifier. See CONTRIBUTING.md.

Is the classifier accurate enough to bill or penalize someone based on its output? Not yet, and the benchmark section above is the honest reason why: 0% accuracy on evasive training workloads today. Treat workload_type as a signal to investigate, not a verdict.

Why not just use the ML classifier from the paper? Its trained weights and dataset were never published. Reimplementing an ML classifier without real training data would produce an unvalidated accuracy claim, not a measured one. See How classification works.

How is this different from run:ai or NVIDIA DCGM? run:ai and DCGM both expose or use GPU telemetry, but neither classifies workload type from that telemetry. run:ai relies entirely on the label the job's owner declares at submission; DCGM just exposes raw utilization and memory metrics for something else to interpret. WorkloadTruth is the layer that actually looks at the telemetry and answers the question. See the comparison table.

Does this work on Windows, macOS, and Linux? The synthetic backend runs anywhere Python 3.10+ runs, including this project's own macOS build environment (which has no NVIDIA GPU). The nvml backend requires an NVIDIA GPU and driver, which in practice means Linux or Windows with NVIDIA hardware; NVML itself is not available on macOS.

What license is this under, and can I use it commercially? Apache 2.0. Commercial use, modification, and redistribution are all permitted under its terms; see LICENSE.

Contributing

See CONTRIBUTING.md. Security issues: see SECURITY.md.

License

Apache 2.0