Provides tools for working with NVIDIA GPUs and tooling: reports the card, driver, memory, power limit and compute capability via nvidia-smi, runs commands under Nsight Systems and Nsight Compute and parses the reports, lists the heaviest kernels with ncu throughput/occupancy, and disassembles the PTX or SASS of a cubin (or the latest Triton-cache kernel) with cuobjdump. Also computes a roofline verdict for a given card against the spec-sheet or measured streaming bandwidth.
Integrates with a local PyTorch interpreter to profile and optimize kernels: traces a script's top CUDA kernels by device time via torch.profiler, runs scripts repeatedly to collect their timing line (mean/std/min), and runs baseline vs. candidate interleaved to report speedup and a verdict (faster, slower, inconclusive, incorrect, invalid). Also scaffolds targets from arbitrary PyTorch functions, using their fp32 result as the correctness reference.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@hotkernel profile my training step, find the hot kernel, and speed it up"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Find the hot kernel, cool it down.
An MCP server that gives your coding agent hands on Nsight and SASS, a verify-and-record contract it cannot fudge, and a kernel-speedrun loop it runs on its own on the GPU on your desk.
English | 中文
Ask Claude to "make this kernel faster" today and it guesses: it cannot see the Nsight report, does not know which kernel eats 60% of the step, has never read the SASS the compiler produced, and cannot benchmark its own rewrite on your card. hotkernel is the missing pair of hands. Ten tools wrap nsys, ncu, cuobjdump, torch.profiler and a strict verify-and-record contract; one skill turns them into a loop: profile, pick the hot kernel, read its SASS, check it against the roofline, rewrite it in Triton, verify against the baseline, record the verdict, keep or revert, repeat.
**Five targets, five autonomous agent runs, one RTX 5090: rmsnorm 3.2× faster than eager PyTorch, softmax 1.2× faster than torch.softmax, gemv level with cuBLAS. Every kernel the agent kept runs at 95–105% of the card's measured streaming bandwidth. Where the library already sat near that ceiling (cuDNN decode attention, tinygemm int4 GEMV), the agent found nothing, stopped on its own, and the board says so.**¹
¹ Each range comes from three independent
verify()calls of the agent's final kernel against the baseline, n=7 interleaved runs each, on one RTX 5090 at 575 W under WSL2. Speedups are relative to the baseline scripts inexamples/: eager PyTorch for rmsnorm,torch.softmaxfor softmax,torch.mv(cuBLAS) for gemv. "Measured streaming bandwidth" is device memcpy on the same card under the same timer, 1501 GB/s (84% of the 1792 GB/s spec), see the calibration. Raw logs for every number are inresults/.
Replay of a real run, rendered from its log: results/softmax/agent-run-20261014-200625. The 1.22× on screen was measured under timing protocol 2; the same kernel re-measured under protocol 3 is 1.17–1.21×.
Claude Code, one command (Claude Code 2.1.275 or later): it adds this repository as a plugin marketplace and installs the plugin, which is the MCP server plus the kernel-speedrun skill.
claude plugin install hotkernel --marketplace xinhuagu/hotkernelInside a session the same is /plugin marketplace add xinhuagu/hotkernel followed by /plugin install hotkernel@hotkernel. The plugin has one option, the Python that has your PyTorch and Triton: set it with /plugin configure hotkernel@hotkernel, or at install time with claude plugin install hotkernel@hotkernel --config hotkernel_python=$HOME/venvs/mytorch/bin/python. Left empty, hotkernel uses the first python on PATH that can import torch, and says so if there is none. The server itself stays torch-free and runs as uvx hotkernel.
Codex and other MCP clients (Cursor, DSH) keep the plain MCP setup:
codex mcp add --env HOTKERNEL_PYTHON=$HOME/venvs/mytorch/bin/python hotkernel -- uvx hotkernelor {"command": "uvx", "args": ["hotkernel"], "env": {"HOTKERNEL_PYTHON": "..."}} in the client's MCP config, plus a copy of skills/kernel-speedrun/ in the agent's skills directory. Claude Code without the plugin works the same way: claude mcp add -e HOTKERNEL_PYTHON=... hotkernel -- uvx hotkernel.
Related MCP server: Flameox
hotkernel new turns any PyTorch function into a target with the same contract as the examples: the function becomes the baseline, its fp32 result the reference, tolerances default by dtype, and the protocol-3 timer is vendored next to the scripts, so they run in your torch interpreter whether or not hotkernel is installed there.
hotkernel new gelu --from torch.nn.functional:gelu --shape 4096x8192 --dtype bf16 --out targetswrites targets/gelu_baseline.py, targets/gelu_candidate.py (starts as the baseline; this is the file the loop edits) and targets/_hotkernel_contract.py. A function of your own works the same way, with extra inputs in call order: --from ./mykernels.py:fused --extra w:8192. Then bench the baseline for the number to beat, and verify the untouched candidate against it, which must come back inconclusive before anything changed. Agents get the same through the new_target tool. On the reference card the untouched gelu candidate benched at 0.0952 ms and verified inconclusive; a deliberately wrong one came back incorrect.
All rows: RTX 5090 @ 575 W, bf16, timing protocol 3, n=7 per verify(); rows with a win are three independent reproductions.
Target | Baseline | Baseline mean | Agent's kernel mean | Speedup | Share of measured bandwidth | Agent time | What the agent wrote |
rmsnorm 4096×8192 | eager PyTorch (5 kernels) | 0.287–0.300 ms | 0.088–0.095 ms | 3.17–3.32× | 95–101% | 10 min, 7 experiments (a second 3-minute run found the same kernel) | one Triton program per row, row in registers, one read and one write |
softmax 4096×8192 |
| 0.107–0.111 ms | 0.090–0.093 ms | 1.17–1.21× | 96–99% | 6 min, 4 experiments | one program per row, max, exp and sum in registers, 8 warps |
gemv 8192×8192 |
| 0.088–0.091 ms | 0.085–0.088 ms | 1.03–1.05×, 2 of 3 runs significant: level with cuBLAS | 101–105%² | 7 min, 3 experiments | autotuned row-block GEMV, 2 rows × 1024 columns per program |
decode attention, 1 token, 32K context, GQA 32/8, d=128 |
| 0.100–0.103 ms | 0.104–0.107 ms | no gain, 5 of 5 runs inconclusive³ | baseline at ~91% | 10 min, 5 experiments (stopped early) | split-KV Triton decode; the kernel itself beat cuDNN, the combine step ate the gain |
int4 GEMV 8192×8192, group 128 |
| 0.032–0.037 ms | 0.036–0.039 ms | no gain, 4 of 4 runs inconclusive³ | baseline at 64–76% | 11 min, 4 experiments (stopped on its own) | Triton GEMV on packed nibbles; about even on medians, behind on means |
² gemv is almost all reads; the copy ceiling is half reads and half writes, so a read-only kernel can sit slightly above it.
³ Rows without a win were not reproduced; the ranges are over the agent's own
verify()calls. Baselines for these two were picked by measuring every library path that runs on this card; the choice is logged inresults/<target>/baseline_choice.json.
Every row on LEADERBOARD.md was written by record(), including the runs that did not win and a deliberately wrong candidate that verify() rejected as incorrect. Each agent run's full transcript is in results/<target>/agent-run-*/.
Tool | What it does | Needs |
| card, driver, memory, power limit, compute capability |
|
| ridge point for this card; memory- or compute-bound verdict and attainable time for a kernel given flops and bytes, against the spec sheet or the measured bandwidth | nothing |
| top CUDA kernels of a script by device time, via | your torch |
| run a command under Nsight Systems or Nsight Compute, parsed |
|
| the n heaviest kernels from a saved report, with ncu throughput and occupancy | nothing |
| PTX or SASS of a cubin, or of the latest Triton-cache kernel by name |
|
| run a script n times, collect its | your torch |
| run baseline and candidate interleaved, n times each; speedup and a verdict: | your torch |
| write the verify result to | nothing |
| scaffold a target from your own function: baseline, blank candidate and the vendored contract, the same as | nothing |
torch_trace, bench, verify and record work on any machine with PyTorch. Nsight is optional and only makes the picture sharper.
The skill in skills/kernel-speedrun/SKILL.md is the whole method:
bench the baseline. That is the number to beat.
torch_trace or profile to find the kernel that actually dominates.
roofline with the kernel's flops and bytes: memory-bound kernels want fusion, compute-bound kernels want tensor cores and bigger tiles.
disasm the Triton kernel after its first compile: register spills, uncoalesced loads, long dependency chains are all visible in SASS.
One hypothesis, one change, then verify(candidate, baseline, n=7). It returns faster only when the gap beats 2 * sqrt(std_b²/n + std_c²/n) and every candidate run passed the correctness check.
record() the result, win or lose. It writes the log and the leaderboard; the agent never edits the board. Keep the change only if record() says it beats the run's best so far, otherwise revert.
Stop after three changes in a row that do not beat the best, or when the kernel is within 10% of the measured bandwidth floor.
It is autoresearch with kernel wall-clock as the metric instead of training loss.
LLMs writing CUDA kernels have a credibility problem: the AI CUDA Engineer claimed 10 to 100× in 2025 and was found to be bypassing its own correctness checks, and KernelBench exists because "the kernel compiles" and "the kernel is correct and faster" are different things. hotkernel's answer is structural:
The contract lives in the baseline. Shape, dtype, seed, reference, tolerances and the timer are in the baseline file the agent cannot edit; the candidate only supplies a function. verify returns invalid if anything in the contract differs.
The timer was wrong twice and fixed in the open. Version 1 reused one input and let the 96 MB L2 pose as DRAM. Version 2 flushed L2 by zeroing a buffer and charged every call ~45 µs of someone else's write-backs; the softmax agent noticed that Nsight and verify disagreed by 2×. Version 3 rotates buffers larger than twice the L2 and was calibrated against five streaming kernels, which all land at 79–84% of the spec sheet, matching ncu's hardware counters. Old rows are kept and marked as not counted.
Wrong is loud. verify reports no speedup for a run whose ok is false, every row carries n and std, and the baseline is in the repo so anyone can rerun it.
NVIDIA GPU required. Reference card: RTX 5090 (sm_120). The roofline table covers 5090, 4090, 3090 and H100, each with its source; add yours in src/hotkernel/gpus.py.
WSL2 is the reference platform. All nine tools are verified on an RTX 5090 under WSL2 Ubuntu 24.04, driver 617.42: nsys 2026.1.3 and ncu 2026.2.1 extracted into user space from NVIDIA's apt repo (no sudo), cuobjdump and nvdisasm straight from the Triton wheel (no CUDA toolkit needed). ncu needs the GPU performance-counter permission once (NVIDIA Control Panel, Developer settings). Versions, commands and raw outputs: docs/platform-matrix.md.
Native Windows: the server runs (pure Python). If your torch lives in WSL2, start the server through wsl.exe; the platform matrix has the exact claude mcp add line.
Triton on RTX 50 series: FlashAttention 2/3/4 do not run on sm_120, which is exactly why hand-tuned Triton kernels matter on these cards.
Apache-2.0.
Run a Python script n times in the user's interpreter and aggregate its timing. Contract: the script prints one JSON line to stdout containing at least {"ms": } and optionally {"ok": } for a correctness check. Returns mean/std/min over the n timed runs plus every raw sample, so a LEADERBOARD row can cite n and seed.
| Name | Required | Description | Default |
|---|---|---|---|
| n | No | ||
| cwd | No | ||
| args | No | ||
| script | Yes | ||
| warmup | No | ||
| timeout | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It does disclose the important trait that the script runs inside the user's own interpreter and states the stdout JSON contract ({"ms": float}, optional {"ok": bool}), plus the aggregate return shape. It does not say what happens on timeout, non-zero exit, missing ms line, or whether process state is reused across runs.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three sentences, front-loaded with the core action, then the script contract, then the return shape. Dense and mostly waste-free, with only the unexplained 'seed' reference costing clarity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema and no annotations, so the description must do more; it covers the return aggregates (mean/std/min plus raw samples) and the script's output contract well, but leaves half the input surface and all failure behavior undocumented. Adequate to attempt a call, not complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0% across 6 parameters. The description explains only the script contract and, indirectly, n; warmup, timeout, cwd, and args are never mentioned, so an agent must guess their semantics from bare titles. It also mentions a 'seed' that is not actually a parameter, which invites confusion.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource: run a Python script n times in the user's interpreter and aggregate timing. That is concrete and distinct from siblings like profile or record, though it never explicitly contrasts itself with them.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is only implied by the trailing clause about a LEADERBOARD row citing n and a seed, which hints at micro-benchmark use. No explicit when-to-use / when-not, and no mention of profile, torch_trace, or roofline as alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Disassemble a kernel. target is a .cubin/.ptx/.so/.exe path, or 'triton-cache' to search
~/.triton/cache (and TRITON_CACHE_DIR) for the most recent kernel whose name contains
kernel. level: 'sass' (via cuobjdump -sass) or 'ptx' (cuobjdump -ptx / the .ptx file).
A real kernel is 500-3000 SASS lines; max_lines<=0 returns all of them.
| Name | Required | Description | Default |
|---|---|---|---|
| level | No | sass | |
| kernel | No | ||
| target | Yes | ||
| max_lines | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations present, the description does real work: it discloses the underlying implementations (cuobjdump -sass vs cuobjdump -ptx or the raw .ptx file), the magic 'triton-cache' lookup that searches ~/.triton/cache and TRITON_CACHE_DIR for the most recent name-matching kernel, and the size/truncation behavior (500-3000 SASS lines typical; max_lines<=0 returns everything). Gaps remain around failure behavior when no kernel matches or the tool is missing, and there is no statement of the return shape.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Four dense sentences, front-loaded with the core action and then walking parameters in schema order; nothing is padding. It reads a bit run-on and the backtick/code-marker mixing makes it slightly harder to scan than separate per-parameter lines would be.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a four-parameter tool with no annotations and no output schema, the description covers inputs thoroughly and sets output-size expectations, which is the main thing an agent needs to avoid blowing context. It stops short of describing the actual response structure (raw text vs path, truncation notice), which matters since no output schema exists to fill that in.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate, and it largely does: target's accepted values and the 'triton-cache' sentinel are spelled out, level's two values are mapped to concrete commands, kernel is explained as a name-substring filter, and max_lines gains meaning via the <=0 convention. What is left implicit is whether `kernel` has any effect for non-'triton-cache' targets and why the default cap is 1500.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource ('Disassemble a kernel') and immediately clarifies that 'kernel' means GPU kernel code, not an OS kernel, by enumerating the artifact types accepted (.cubin/.ptx/.so/.exe). This is clearly distinguishable from siblings like profile, bench, or top_kernels, which do not produce code listings.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The usage context is implied through the accepted target formats and the 'triton-cache' shortcut, which tells an agent when this tool is the right one to reach for. However, there is no explicit when-to-use/when-not-to-use statement, no mention of alternatives (e.g., use top_kernels first to find a kernel name), and no prerequisites such as needing cuobjdump on PATH or a compiled artifact already on disk.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Identify the GPU(s) in this machine via nvidia-smi: name, driver, memory, power limit, compute capability, max SM clock. Also returns the matching roofline table entry if known.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations, so the description carries the full burden. It usefully discloses the data source (nvidia-smi) and the exact fields returned, plus the conditional bonus return of a roofline table entry. It omits meaningful behavior: what happens on a machine without an NVIDIA GPU or without nvidia-smi installed, and whether the discovery is cached or live.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two tight sentences, front-loaded with the action and source, then the return fields. No filler and no repetition of the tool name.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no input schema and no output schema, the description must supply the return contract, and it does so by enumerating the reported attributes plus the conditional roofline entry. The main residual gap is failure behavior on non-NVIDIA or nvidia-smi-less hosts, which an agent would want to know before relying on it.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool takes zero parameters, so there is nothing to disambiguate; baseline 4 applies. The description correctly frames it as a parameterless probe of the local machine.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Specific verb and resource: 'Identify the GPU(s) in this machine via nvidia-smi', followed by an explicit enumeration of the returned attributes (name, driver, memory, power limit, compute capability, max SM clock). It is clear what the tool does, but only implicitly distinguishes itself from siblings like roofline or bench by noting that it merely returns a matching roofline entry rather than computing one.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is implied rather than stated: an agent can infer this is the hardware-discovery step before roofline/bench work, but the description never says 'call this first' or when to prefer it over siblings. No exclusions or prerequisites are given.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Scaffold a target from the user's own PyTorch function so the loop can run on it: writes
<out_dir>/_baseline.py (the function as the baseline plus the Target definition:
inputs, seed, fp32 reference, dtype-default tolerances, rotation, timer),
<out_dir>/_candidate.py (the file to edit; starts as the baseline) and the vendored
hotkernel_contract.py (protocol 3) they import, so they run in HOTKERNEL_PYTHON without
hotkernel installed there. from: 'torch.nn.functional:gelu' or './mykernels.py:fused';
shape: the primary input x, e.g. '4096x8192'; extra: more inputs in call order like
['w:8192', 'b:8192'] (random tensors of the same dtype); rotate: inputs that stream through
memory (default ['x']); flops: nominal flops per call for roofline(); reference: a different
'module:function' to run in fp32 as the reference when the baseline cannot run in fp32
(a backend-forced SDPA call, a bf16-only library kernel). For inputs randn cannot draw (a
quantized weight): derive: a 'module:function' from the dict of raw random tensors to the
dict of tensors the call takes, in call order; inputs: the names it returns, e.g.
['x', 'w_q', 'scales'] or 'x,w_q,scales', which become the candidate's signature; reference
is then required and gets the derived inputs as they are (it must do its own fp32 math);
min_bytes: the bytes one call must read (one weight layout), which sizes the rotation K.
Same as the CLI hotkernel new. Then bench() the baseline and verify() the blank candidate
(expect 'inconclusive') before changing anything.
| Name | Required | Description | Default |
|---|---|---|---|
| atol | No | ||
| rtol | No | ||
| seed | No | ||
| dtype | No | bf16 | |
| extra | No | ||
| flops | No | ||
| force | No | ||
| from_ | Yes | ||
| iters | No | ||
| shape | Yes | ||
| derive | No | ||
| inputs | No | ||
| rotate | No | ||
| target | Yes | ||
| out_dir | No | . | |
| min_bytes | No | ||
| reference | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It discloses a lot: exact output files, vendored `_hotkernel_contract.py`, baseline/candidate contents, and that they run in HOTKERNEL_PYTHON without hotkernel installed. However, it omits overwrite behavior (`force` is never mentioned), return values, and whether scaffolding is idempotent or destructive.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The content is front-loaded with the main action, but it is a dense run-on paragraph mixing file outputs, parameter explanations, and workflow instructions. For a 17-parameter tool, some length is warranted, but the lack of structure makes it harder to scan than necessary.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity, no annotations, no output schema, and zero schema description coverage, the description is fairly complete on file outputs and core parameters. It is still missing key operational details such as `force` overwrite behavior, iteration/timing controls, and what the call returns, leaving meaningful gaps.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate for 17 parameters. It explains the major ones (`from_`, `shape`, `extra`, `rotate`, `flops`, `reference`, `derive`, `inputs`, `min_bytes`) in useful detail, but leaves `force`, `iters`, and the exact roles of `atol`/`rtol` vague; `seed` and `dtype` are only lightly implied.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and resource: scaffold a target from the user's PyTorch function and write baseline/candidate/contract files. It also distinguishes this setup step from bench() and verify() by naming them as follow-up actions, though it does not explicitly say when not to use this tool versus those siblings.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives clear workflow context: use this to scaffold a target so the loop can run, then bench() the baseline and verify() the blank candidate before editing. The CLI equivalent `hotkernel new` is named. It lacks explicit exclusions or a direct comparison to alternative setup paths, but the intended usage is clear.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Run a command under Nsight Systems (tool='nsys') or Nsight Compute (tool='ncu') and return the path of the report plus a parsed kernel summary. nsys gives the per-kernel time breakdown; ncu gives per-kernel metrics (achieved occupancy, DRAM throughput, SM throughput) and is much slower, so use ncu_kernel_regex to target one kernel.
| Name | Required | Description | Default |
|---|---|---|---|
| cmd | Yes | ||
| cwd | No | ||
| tool | No | nsys | |
| out_dir | No | results/profiles | |
| timeout | No | ||
| ncu_kernel_regex | No | ||
| ncu_launch_count | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It does disclose useful traits: outputs are a report path plus parsed summary, ncu mode is slow and worth narrowing, and ncu metrics returned. It omits timeout behavior, what is written to disk (out_dir), permissions/privilege needs for profiling, and failure modes.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three dense sentences with the action and return values front-loaded, then the mode tradeoff and targeting advice. No padding, though the first sentence is long and could be split for scanability.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema and no annotations, so the description must convey enough to call correctly. It does reasonably well on modes and returns, but with 7 parameters and a long-running profiling job it never covers timeout semantics, launch-count behavior, working-directory relevance, or what the parsed kernel summary actually contains structurally.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0% across 7 params, so the description must compensate. It explains the tool enum values (nsys/ncu), the meaning of ncu_kernel_regex, and implies cmd and the report-path/out_dir behavior, but cwd, timeout, and ncu_launch_count get no meaning at all, leaving half the surface undocumented.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific action (run a command under Nsight Systems/Compute) plus concrete outputs (report path, parsed kernel summary), and differentiates the two internal modes nsys vs ncu. It does not distinguish itself from sibling profiling tools like torch_trace, bench, or roofline, so an agent still has to infer which profiling sibling to pick.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives explicit guidance inside the tool's own mode choice: nsys for per-kernel time breakdown, ncu for detailed metrics and 'much slower, so use ncu_kernel_regex to target one kernel'. However, there is no guidance on when to use this tool rather than torch_trace, bench, roofline, or top_kernels, and no stated exclusions or prerequisites.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Write a verify() result to results//.json, then regenerate
LEADERBOARD.md from every log under results/ (so the table is always reproducible from
the raw files). The Speedup cell is filled only when the verdict is 'faster'; otherwise it
says 'no gain', 'incorrect' or 'invalid'. Call it after every verify(), win or lose; never
edit LEADERBOARD.md by hand. note is the one-line "what changed"; who says honestly
who produced the candidate, e.g. 'agent: running kernel-speedrun unattended' or
'hand-written by during tool bring-up'.
| Name | Required | Description | Default |
|---|---|---|---|
| cwd | No | ||
| who | No | ||
| note | No | ||
| target | Yes | ||
| verify_result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full disclosure and does it well: it names the two files touched, explains that the leaderboard is derived deterministically from every log under results/ (reproducibility rationale), and spells out verdict-dependent cell contents ('faster' vs 'no gain'/'incorrect'/'invalid'). This is beyond what any schema could convey.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with the two primary effects (write file, regenerate leaderboard), then the verdict-dependent cell rule, then caller obligations, then parameter hints. Slightly dense but each sentence carries distinct information; nothing is filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 5-parameter, annotation-free mutation tool with no output schema, the description explains the side effects and artifact semantics thoroughly but omits the shape expected for the required `verify_result` payload and the role of `cwd`, which an agent needs to call it correctly the first time.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate; it does so for `who` ('says honestly who produced the candidate', with examples) and `note` ('one-line what changed'), and implies `target` via the output path. However, `verify_result` (the required, polymorphic object-or-string input) and `cwd` are left entirely unexplained, so compensation is only partial.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and two concrete artifacts: writing the verify() result to results/<target>/<timestamp>.json and regenerating LEADERBOARD.md from the raw logs. That is clearly distinct from siblings like verify, bench, or profile, which produce measurements rather than persisting them.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly says when to call it ('after every verify(), win or lose') and gives a hard prohibition ('never edit LEADERBOARD.md by hand'). The agent has no ambiguity about sequencing relative to the verify sibling.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Roofline for a GPU (default: the first GPU in this machine). Pass the kernel's flops and bytes moved (bench()/verify() return them as roofline_inputs) to get the bound, the attainable TFLOPS and attainable_ms from the spec sheet. Pass measured_mem_gbs (the bandwidth a streaming copy reaches on this card under the repo's timer, see docs/platform-matrix.md; 1501 on the reference RTX 5090) to also get attainable_ms_measured, the floor a kernel can realistically get within 10% of. dtype: bf16 | fp16 | fp8 | fp32.
| Name | Required | Description | Default |
|---|---|---|---|
| gpu | No | ||
| bytes | No | ||
| dtype | No | bf16 | |
| flops | No | ||
| measured_mem_gbs | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the behavioral burden but does well: it names the returned metrics, explains the default GPU, clarifies the dtype choices, and describes the optional measured floor. It does not explicitly state side-effect/read-only behavior, though the tool appears to be a pure calculation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is dense but well structured: default GPU behavior first, core inputs and outputs next, optional measured behavior after that, and dtype last. Every sentence adds necessary information for a tool with five undocumented parameters.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given five parameters, zero schema descriptions, and no output schema, the description is nearly complete: it explains inputs, outputs, defaults, and optional measured behavior. It could be slightly stronger by specifying output formatting or units more explicitly, but an agent has enough to call it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must explain all five parameters, and it does. It covers gpu defaulting to the first GPU, flops/bytes from bench()/verify(), measured_mem_gbs with documentation reference and example value, and dtype options.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states that this tool computes a GPU roofline bound and attainable performance metrics from flops and bytes. It is specific about the computation and mentions bench()/verify() as input sources, but it does not explicitly differentiate itself from sibling tools like profile or gpu_info.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives clear usage context by explaining when to pass flops/bytes from bench()/verify() and when to pass measured_mem_gbs for measured attainable performance. It does not state when not to use the tool or name alternative siblings for comparison, so it falls short of full alternative guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Read a saved profile CSV (from profile()) and return the n heaviest kernels. tool='nsys': cuda_gpu_kern_sum CSV -> share of GPU time, calls, avg/med/min/max/std ns. tool='ncu': details CSV -> per-launch duration, achieved occupancy, DRAM / SM throughput %, registers per thread, and a memory/compute/latency hint.
| Name | Required | Description | Default |
|---|---|---|---|
| n | No | ||
| tool | No | nsys | |
| summary_csv | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the burden, and it does establish a read-only, file-consuming operation and spells out exactly what fields each mode returns (GPU time share, occupancy, DRAM/SM %, registers). It omits failure behavior, e.g., a missing file or a CSV that doesn't match the chosen tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with the core action in the first sentence, then a compact per-mode breakdown. Terse arrow notation is efficient, though the second sentence is dense enough to require careful reading.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema exists, and the description adequately documents the return fields per mode, which is the key completeness gap for this tool. Remaining gaps are error/edge-case handling rather than core call correctness.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must compensate, and it does for two of three parameters: 'n' is the count of heaviest kernels, and 'tool' has its two valid values and what each expects as input. 'summary_csv' is only implied as the path to the profile CSV, and no enum constrains 'tool' in the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource: 'Read a saved profile CSV (from profile()) and return the n heaviest kernels.' The link to profile() implicitly differentiates it from siblings like gpu_info, though no sibling is named explicitly.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Makes the usage context clear: this consumes a CSV produced by profile(), and the 'tool' value must match the CSV type (nsys cuda_gpu_kern_sum vs ncu details). It stops short of stating exclusions (e.g., when to use gpu_info or roofline instead).
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Run a Python script under torch.profiler in the user's interpreter. Returns kernels
(real GPU kernels by device time, pct over total kernel time) and ops (aten ops with
input shapes, inclusive time). Works without Nsight installed; use this first.
| Name | Required | Description | Default |
|---|---|---|---|
| cwd | No | ||
| top | No | ||
| args | No | ||
| script | Yes | ||
| timeout | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the burden. It usefully discloses that the script runs in the user's interpreter and describes returned data, but it does not cover execution side effects, permissions, timeout behavior, or what happens when the profiler is unavailable.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is three short sentences and front-loads the primary action and return values. Every sentence adds useful information without repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with five parameters, no annotations, and no output schema, the description does well on purpose and return keys. It is incomplete on parameter meaning and execution semantics, which are important for correct invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0% across five parameters. The description only implies the `script` parameter by saying it runs a Python script; it does not explain `cwd`, `args`, `top`, or `timeout`, so the low-coverage schema is not compensated.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a specific action and tool context: running a Python script under torch.profiler in the user's interpreter. It also states what the tool returns (`kernels` and `ops`) and distinguishes itself from alternatives by saying it works without Nsight and should be used first.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The phrase 'use this first' and 'Works without Nsight installed' gives clear priority and a condition for choosing this tool. However, it does not explicitly name the alternative profiler sibling or state when to switch away from this tool.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Decide whether a candidate script beats the baseline script. Both follow the bench contract (print one JSON line with "ms" and "ok"). Runs them interleaved (baseline, candidate, baseline, candidate, ...) warmup+n times each, back to back on the same card, discards the warmup pairs, and returns mean/std/min for both plus a verdict: 'faster' or 'slower' when |mean_b - mean_c| > 2*sqrt(std_b^2/n + std_c^2/n), 'inconclusive' otherwise, 'incorrect' if the candidate printed ok=false in any run (then no speedup is reported), 'invalid' if the two scripts disagree on shape/dtype/iters/seed or the baseline fails its own check. Also records gpu, driver, power limit, timestamp and the exact commands. Pass the result to record().
| Name | Required | Description | Default |
|---|---|---|---|
| n | No | ||
| cwd | No | ||
| args | No | ||
| warmup | No | ||
| timeout | No | ||
| baseline | Yes | ||
| candidate | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations exist, so the description carries the full behavioral burden, and it does so well: interleaved execution order, warmup+n repetitions with warmup pairs discarded, same-card execution, and the recorded metadata (gpu, driver, power limit, timestamp, exact commands). What it doesn't state is cost, side effects, or failure behavior (for example, what happens if the timeout is hit or the script crashes); those are left unstated. Those are minor relative to what is covered but keep it below a 5.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
It is front-loaded with the core purpose and the verdict rules are essential content, so almost every sentence earns its place. The long run-on clause enumerating verdicts makes it dense to parse, though nothing is genuinely wasted.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a statistical benchmarking tool with no output schema and no annotations, the description supplies the return shape (mean/std/min plus verdict), the significance threshold, and every verdict state including error cases. Only the ancillary run-control parameters (timeout, cwd, args) remain unaddressed.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0% across 7 parameters, so the description must compensate. It gives meaningful semantics for n and warmup through the 'warmup+n times each' narrative, and it clarifies baseline and candidate as bench-contract scripts; however, cwd, args, and timeout are never explained, leaving 3 of 7 parameters undocumented anywhere.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific decision ('Decide whether a candidate script beats the baseline script') plus the resource (two scripts) and the contract both must follow. This clearly separates it from siblings like bench, profile, or record, which cannot make a comparative verdict.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Implies the use case precisely (compare two benchmark scripts) and gives an explicit downstream step: 'Pass the result to record().' It doesn't name when to prefer bench or profile instead of verify, so no explicit exclusion is stated.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.1.1benchdisasmgpu_infonew_targetprofilerecordrooflinetop_kernelstorch_traceverifyScored across 10 tools
Each tool occupies a distinct step in the optimize-measure-record loop, and descriptions explicitly order them (torch_trace 'use this first', profile for nsys/ncu, top_kernels for saved CSVs). The only real overlap is torch_trace vs profile vs top_kernels, all producing kernel-time breakdowns, though the backend/input distinction is spelled out. gpu_info returning a roofline entry also slightly blurs with roofline, but the primary purposes remain clear.
All names are lowercase snake_case with no style mixing. There is a mild convention split between noun-style query tools (gpu_info, roofline, top_kernels) and verb-style action tools (bench, verify, record, disasm, profile, new_target), but this reads naturally and predictably.
Ten tools map cleanly onto the kernel-speedrun pipeline: environment info, scaffolding, tracing, profiling, analysis, benchmarking, verification, and recording. Nothing feels redundant or missing at the count level.
The surface covers the full lifecycle from scaffolding a target (new_target) through measurement (bench/verify/profile) to persistent, reproducible logging (record regenerating LEADERBOARD.md). Minor gaps: no tool to list existing targets/results or inspect history without regenerating the leaderboard, and no way to remove or invalidate a bad record.
GPU compute for agents: run commands on an L4/A100/H100, watch logs, billed per second.
On-demand GPU nodes for agents: create nodes, run commands, and submit jobs, billed by the minute.
GPU and LLM inference benchmarks, hardware evidence, deployment recommendations, and launch configs.
Provides capabilities that let LLM agents perform a range of infrastructure management tasks.