Skip to main content
Glama
ingero-io

Ingero

by ingero-io
OverviewSchemaRelated ServersScoreDiscussions
Local

Ingero - GPU Causal Observability

Go Report Card License GitHub Release CI MCP

Featured in: awesome-ebpf · awesome-observability · awesome-sre-tools · awesome-cloud-native · awesome-profiling · Awesome-GPU · awesome-devops-mcp-servers · MCP Registry · Glama · mcpservers.org

Version: 0.9.0

The only GPU observability tool your AI assistant can talk to.

"What caused the GPU stall?" → "forward() at train.py:142 - cudaMalloc spiking 48ms during CPU contention. 9,829 calls, 847 scheduler preemptions."

Ingero is a production-grade eBPF agent that traces the full chain - from Linux kernel events through CUDA API calls to your Python source lines - with <2% overhead, zero code changes, and one binary.

Quick Start

# Install (Linux amd64 — see below for arm64/Docker)
VERSION=0.9.0
curl -fsSL "https://github.com/ingero-io/ingero/releases/download/v${VERSION}/ingero_${VERSION}_linux_amd64.tar.gz" | tar xz
sudo mv ingero /usr/local/bin/

# Trace your GPU workload
sudo ingero trace

# Diagnose what happened
ingero explain --since 5m

  • The "Why": Correlate a cudaStreamSync spike with sched_switch events - the host kernel preempted your thread.

  • The "Where": Map CUDA calls back to Python source lines in your PyTorch forward() pass.

  • The "Hidden Kernels": Trace the CUDA Driver API to see kernel launches by cuBLAS/cuDNN that bypass standard profilers.

No ClickHouse, no PostgreSQL, no MinIO - just one statically linked Go binary and embedded SQLite.

See a real AI investigation session - an AI assistant diagnosing GPU training issues on A100 and GH200 using only Ingero's MCP tools. No shell access, no manual SQL - just questions and answers.

What It Does

Ingero uses eBPF to trace GPU workloads at three layers, reads system metrics from /proc, and assembles causal chains that explain root causes:

  1. CUDA Runtime uprobes - traces cudaMalloc, cudaFree, cudaLaunchKernel, cudaMemcpy, cudaMemcpyAsync, cudaStreamSync / cudaDeviceSynchronize via uprobes on libcudart.so

  2. CUDA Driver uprobes - traces cuLaunchKernel, cuMemcpy, cuMemcpyAsync, cuCtxSynchronize, cuMemAlloc via uprobes on libcuda.so. Captures kernel launches from cuBLAS/cuDNN that bypass the runtime API.

  3. CUDA Graph lifecycle uprobes - traces cudaStreamBeginCapture, cudaStreamEndCapture, cudaGraphInstantiate, cudaGraphLaunch for graph capture/replay visibility in torch.compile and vLLM workloads

  4. Host tracepoints - traces sched_switch, sched_wakeup, mm_page_alloc, oom_kill, sched_process_exec/exit/fork for CPU scheduling, memory pressure, and process lifecycle

  5. System context - reads CPU utilization, memory usage, load average, and swap from /proc (no eBPF, no root needed)

The causal engine correlates events across layers by timestamp and PID to produce automated root cause analysis with severity ranking and fix recommendations.

$ sudo ingero trace

  Ingero Trace  -  Live CUDA Event Stream
  Target: PID 4821 (python3)
  Library: /usr/lib/x86_64-linux-gnu/libcudart.so.12
  CUDA probes: 14 attached
  Driver probes: 10 attached
  Host probes: 7 attached

  System: CPU [████████░░░░░░░░░░░░] 47% | Mem [██████████████░░░░░░] 72% (11.2 GB free) | Load 3.2 | Swap 0 MB

  CUDA Runtime API                                               Events: 11,028
  ┌──────────────────────┬────────┬──────────┬──────────┬──────────┬─────────┐
  │ Operation            │ Count  │ p50      │ p95      │ p99      │ Flags   │
  ├──────────────────────┼────────┼──────────┼──────────┼──────────┼─────────┤
  │ cudaLaunchKernel     │ 11,009 │ 5.2 µs   │ 12.1 µs  │ 18.4 µs  │         │
  │ cudaMalloc           │     12 │ 125 µs   │ 2.1 ms   │ 8.4 ms   │ ⚠ p99  │
  │ cudaDeviceSynchronize│      7 │ 684 µs   │ 1.2 ms   │ 3.8 ms   │         │
  └──────────────────────┴────────┴──────────┴──────────┴──────────┴─────────┘

  CUDA Driver API                                                Events: 17,525
  ┌──────────────────────┬────────┬──────────┬──────────┬──────────┬─────────┐
  │ Operation            │ Count  │ p50      │ p95      │ p99      │ Flags   │
  ├──────────────────────┼────────┼──────────┼──────────┼──────────┼─────────┤
  │ cuLaunchKernel       │ 17,509 │ 4.8 µs   │ 11.3 µs  │ 16.2 µs  │         │
  │ cuMemAlloc           │     16 │ 98 µs    │ 1.8 ms   │ 7.1 ms   │         │
  └──────────────────────┴────────┴──────────┴──────────┴──────────┴─────────┘

  Host Context                                                   Events: 258
  ┌─────────────────┬────────┬──────────────────────────────────────────┐
  │ Event           │ Count  │ Detail                                   │
  ├─────────────────┼────────┼──────────────────────────────────────────┤
  │ mm_page_alloc   │    251 │ 1.0 MB allocated (order-0: 251)          │
  │ process_exit    │      7 │ 7 processes exited                       │
  └─────────────────┴────────┴──────────────────────────────────────────┘

  ⚠ cudaStreamSync p99 = 142ms  -  correlated with 23 sched_switch events
    (GPU thread preempted during sync wait, avg 2.1ms off-CPU)

What You'll Discover

Things no other GPU tool can show you.

"cuBLAS was launching 17,509 kernels and you couldn't see any of them." Most profilers trace only the CUDA Runtime API - but cuBLAS calls cuLaunchKernel (driver API) directly, bypassing the runtime. Ingero traces both layers: 11,009 runtime + 17,509 driver = complete visibility into every kernel launch.

"Your training slowed because logrotate stole 4 CPU cores." System Context shows CPU at 94%, Load 12.1. The CUDA table shows cudaStreamSync p99 jumping from 16ms to 142ms. The Host Context shows 847 sched_switch events. ingero explain assembles the full causal chain: logrotate preempted the training process → CUDA sync stalled → training throughput dropped 30%. Fix: nice -n 19 logrotate, or pin training to dedicated cores.

"Your model spends 38% of wall-clock time on data movement, not compute." nvidia-smi says "GPU utilization 98%", but the GPU is busy doing cudaMemcpy, not compute. Ingero's time-fraction breakdown makes this obvious. The fix (pinned memory, async transfers, larger batches) saves 30-50% wall-clock time.

"Your host is swapping and your GPU doesn't know it." System Context shows Swap 2.1 GB. cudaMalloc p99 rises from 0.02ms to 8.4ms. No GPU tool shows this - nvidia-smi says GPU memory is fine, but host-side CUDA bookkeeping is hitting swap.

"Your vLLM inference spiked because a new batch size triggered CUDA Graph re-capture." Ingero traces cudaStreamBeginCapture / cudaGraphLaunch via eBPF uprobes - no CUPTI, no Nsight, no code changes. When GraphLaunch rate drops 50%, Ingero flags graph pool exhaustion. When capture overlaps with OOM, the causal chain explains why. Works with torch.compile(mode="reduce-overhead") and vLLM out of the box.

"Ask your AI: what line of my code caused the GPU stall?" Your AI assistant calls Ingero's MCP server and answers in one shot: "The issue is in forward() at train.py:142, calling cudaMalloc through PyTorch. 9,829 calls, avg 3.1ms but spiking to 48.3ms during CPU contention." Resolved Python source lines, native symbols, timing stats - no logs, no manual SQL, no hex addresses. The engineer asks questions in plain English and gets production root causes back.

See It In Action

ingero demo                 # run all 6 scenarios (auto-detects GPU)
ingero demo incident        # full causal chain in 30 seconds
ingero demo --no-gpu        # synthetic mode (no root, no GPU needed)
sudo ingero demo --gpu      # real GPU + eBPF tracing

Scenarios

Scenario

What It Reveals

incident

CPU spike + sched_switch storm → cudaStreamSync 8.5x latency spike → full causal chain with root cause and fix

cold-start

First CUDA calls take 50-200x longer than steady state (CUDA context init)

memcpy-bottleneck

cudaMemcpy dominates wall-clock time (38%), not compute - nvidia-smi lies

periodic-spike

cudaMalloc spikes 50x every ~200 batches (PyTorch caching allocator)

cpu-contention

Host CPU preemption causes CUDA latency spikes

gpu-steal

Multi-process GPU time-slicing quantified via CUDA API timing patterns

Every scenario prints a GPU auto-detect header showing GPU model and driver version, then displays real-time ASCII bar charts for system context.

Install

Binary Release (recommended)

Download a pre-built binary from GitHub Releases.

Archive filenames include the version: ingero_<version>_linux_<arch>.tar.gz. Replace VERSION below with the latest release (e.g., 0.9.0):

# Linux amd64
VERSION=0.9.0
curl -fsSL "https://github.com/ingero-io/ingero/releases/download/v${VERSION}/ingero_${VERSION}_linux_amd64.tar.gz" | tar xz
sudo mv ingero /usr/local/bin/

# Linux arm64 (GH200, Grace Hopper, Graviton)
VERSION=0.9.0
curl -fsSL "https://github.com/ingero-io/ingero/releases/download/v${VERSION}/ingero_${VERSION}_linux_arm64.tar.gz" | tar xz
sudo mv ingero /usr/local/bin/

Docker Image

Multi-arch images (amd64 + arm64) are published to GHCR on every release:

# Pull the latest image
docker pull ghcr.io/ingero-io/ingero:latest

# Or pin to a specific version
docker pull ghcr.io/ingero-io/ingero:v0.9.0

# Quick test (no root, no GPU needed)
docker run --rm ghcr.io/ingero-io/ingero demo --no-gpu

# System readiness check
docker run --rm --privileged --pid=host ghcr.io/ingero-io/ingero check

# Live eBPF tracing (requires privileges + kernel mounts)
docker run --rm --privileged --pid=host \
  -v /sys/kernel/debug:/sys/kernel/debug \
  -v /sys/kernel/btf:/sys/kernel/btf:ro \
  -v /var/lib/ingero:/var/lib/ingero \
  ghcr.io/ingero-io/ingero trace --record

Minimum capabilities (alternative to --privileged): --cap-add=BPF --cap-add=PERFMON --cap-add=SYS_ADMIN.

Note: eBPF tracing (trace, demo --gpu) requires --privileged --pid=host plus the kernel volume mounts shown above. Without these, only unprivileged commands work (demo --no-gpu, check, version, explain, query). The --pid=host flag shares the host's /proc - do not also bind-mount -v /proc:/proc:ro as this causes OCI runtime errors on Docker Desktop and WSL2.

Data persistence: The container stores the SQLite database at /var/lib/ingero/ingero.db by default. Mount -v /var/lib/ingero:/var/lib/ingero to persist data after the container stops. Without this mount, all trace data is lost when the container exits.

Multiple databases: Use --db or the INGERO_DB env var to work with different databases:

# Trace to a named database
docker run --rm --privileged --pid=host \
  -v /var/lib/ingero:/var/lib/ingero \
  -v /sys/kernel/debug:/sys/kernel/debug \
  -v /sys/kernel/btf:/sys/kernel/btf:ro \
  ghcr.io/ingero-io/ingero trace --db /var/lib/ingero/training-run-42.db

# Investigate a specific database
docker run --rm \
  -v /var/lib/ingero:/var/lib/ingero \
  ghcr.io/ingero-io/ingero explain --db /var/lib/ingero/training-run-42.db

# Compare databases from different runs
docker run --rm \
  -v /var/lib/ingero:/var/lib/ingero \
  ghcr.io/ingero-io/ingero query --db /var/lib/ingero/training-run-41.db --since 1h

docker run --rm \
  -v /var/lib/ingero:/var/lib/ingero \
  ghcr.io/ingero-io/ingero query --db /var/lib/ingero/training-run-42.db --since 1h

The image is ~10 MB (Alpine 3.20 + statically linked Go binary). When building the dev Dockerfile locally, pass version info via build args:

docker build -f deploy/docker/Dockerfile \
  --build-arg VERSION=0.9.0 \
  --build-arg COMMIT=$(git rev-parse --short HEAD) \
  --build-arg BUILD_DATE=$(date -u +%Y-%m-%dT%H:%M:%SZ) \
  -t ingero:local .

GHCR images have version info baked in automatically via GoReleaser. See deploy/docker/Dockerfile for details.

Build from Source

# Quick setup: install all build dependencies (Go, clang, llvm) on Ubuntu 22.04/24.04
curl -fsSL https://raw.githubusercontent.com/ingero-io/ingero/main/scripts/install-deps.sh | bash

# Requires clang-14, Linux kernel with BTF
git clone https://github.com/ingero-io/ingero.git
cd ingero
make              # generates eBPF bindings, builds, tests, and lints  -  single command
sudo make install # optional  -  copies binary to /usr/local/bin/ingero
                  # or just use ./bin/ingero directly, or: alias ingero=$PWD/bin/ingero

Requirements

  • Linux kernel 5.15+ with BTF (CONFIG_DEBUG_INFO_BTF=y)

  • NVIDIA driver 550+ with CUDA 11.x, 12.x, or 13.x

  • Root / CAP_BPF + CAP_PERFMON (eBPF requires elevated privileges)

  • Tested on: GH200, H100, A100, A10, RTX 4090, RTX 3090 (x86_64 and aarch64)

Commands

ingero check

Check if your system is ready for eBPF-based GPU tracing.

$ ingero check

Ingero  -  System Readiness Check

  [✓] Kernel version: 5.15.0-144-generic
      need 5.15+
  [✓] BTF support: /sys/kernel/btf/vmlinux
      available (5242880 bytes)
  [✓] NVIDIA driver: 580.126.09
      open kernel modules (550+)
  [✓] GPU model: NVIDIA GeForce RTX 3090 Ti, 24564 MiB
  [✓] CUDA runtime: /usr/lib/x86_64-linux-gnu/libcudart.so.12
      loaded by 1 process(es)
  [✓] CUDA driver (libcuda.so): /usr/lib/x86_64-linux-gnu/libcuda.so.1
      available for driver API tracing
  [✓] CUDA processes: 1 found
      PID 4821 (python3)

All checks passed  -  ready to trace!

ingero trace

Live event stream with rolling stats, system context, and anomaly detection. Events are recorded to SQLite by default (use --record=false to disable). The database is capped at 10 GB rolling storage and auto-purges old events when the limit is reached (see --max-db).

sudo ingero trace                           # auto-detect all CUDA processes for current user
sudo ingero trace --pid 4821               # trace specific process
sudo ingero trace --pid 4821,5032          # trace multiple specific processes
sudo ingero trace --user bob               # trace all CUDA processes owned by bob
sudo ingero trace --record=false           # disable SQLite recording
sudo ingero trace --duration 60s           # stop after 60 seconds
sudo ingero trace --json                   # JSON output (pipe to jq)
sudo ingero trace --verbose                # show individual events
sudo ingero trace --stack=false            # disable stack traces (saves ~0.4-0.6% overhead)
sudo ingero trace --max-db 10g             # limit DB to 10 GB (default), prunes oldest events
sudo ingero trace --max-db 500m            # limit DB to 500 MB (tight disk budget)
sudo ingero trace --max-db 0               # unlimited (no size-based pruning)
sudo ingero trace --deadband 5              # suppress idle snapshots (5% threshold)
sudo ingero trace --deadband 5 --heartbeat 30s  # deadband + force report every 30s
sudo ingero trace --prometheus :9090       # expose Prometheus /metrics endpoint
sudo ingero trace --otlp localhost:4318    # push metrics via OTLP

Only trace needs sudo - it attaches eBPF probes to the kernel. All other commands (check, explain, query, mcp, demo) run unprivileged. When you run sudo ingero trace, the database is written to your home directory (not /root/) and chown'd to your user, so non-sudo commands can read it.

Process targeting:

  • Default (no flags): traces all CUDA processes owned by the invoking user (via SUDO_USER). On single-user boxes, this means all CUDA processes.

  • --pid: target specific process(es), comma-separated (e.g., --pid 1234,5678).

  • --user: target all CUDA processes owned by a specific user (--user bob, --user root).

  • Dynamic child tracking: fork events auto-enroll child PIDs for host correlation.

The trace display shows five sections:

  1. System Context - CPU, memory, load, swap with ASCII bar charts (green/yellow/red)

  2. CUDA Runtime API - per-operation p50/p95/p99 latency with anomaly flags (cudaMalloc, cudaLaunchKernel, graphLaunch, etc.)

  3. CUDA Driver API - driver-level operations (cuLaunchKernel, cuMemAlloc, etc.) that cuBLAS/cuDNN call directly

  4. Host Context - scheduler, memory, OOM, and process lifecycle events

  5. CUDA Graph events - graph capture, instantiate, and launch events (when graph-using workloads are traced)

ingero explain

Analyze recorded events from SQLite and produce an incident report with causal chains, root causes, and fix recommendations. Reads from the database populated by ingero trace - no root needed.

ingero explain                         # analyze last 5 minutes
ingero explain --since 1h             # last hour
ingero explain --since 2d             # last 2 days
ingero explain --since 1h30m          # human-friendly durations (also: 1w, 3d12h)
ingero explain --last 100             # last 100 events
ingero explain --pid 4821             # filter by specific process
ingero explain --pid 4821,5032        # filter by multiple processes
ingero explain --chains               # show stored causal chains (no re-analysis)
ingero explain --json                 # JSON output for pipelines
ingero explain --from "15:40" --to "15:45"  # absolute time range
ingero explain --per-process              # per-process CUDA API breakdown
ingero explain --per-process --json       # JSON output for pipelines

Per-Process Breakdown

For multi-process GPU workloads (RAG pipelines, model serving with workers, multi-tenant GPU sharing), --per-process shows a CUDA API breakdown grouped by process:

$ ingero explain --per-process --since 5m

PER-PROCESS GPU API BREAKDOWN

  PID 4821 (vllm-worker)
    cuLaunchKernel      12,847 calls   p50=4.8µs   p95=11.2µs   p99=16.1µs
    cudaMemcpyAsync        892 calls   p50=38µs    p95=124µs    p99=891µs
    cudaMallocManaged       14 calls   p50=112µs   p95=2.1ms    p99=8.4ms

  PID 5032 (embedding-svc)
    cuLaunchKernel       3,201 calls   p50=5.1µs   p95=12.8µs   p99=19.4µs
    cudaMemcpy             448 calls   p50=42µs    p95=98µs     p99=412µs

  ⚠ Multi-process GPU contention: 2 processes sharing GPU with CUDA/Driver ops

This answers "which process is hogging the GPU?" - essential for diagnosing RAG pipeline contention where embedding, retrieval, and generation compete for GPU time.

INCIDENT REPORT  -  2 causal chains found (1 HIGH, 1 MEDIUM)

[HIGH] cudaStreamSync p99=142ms (8.5x p50)  -  CPU contention
  Timeline:
    15:41:20  [SYSTEM]  CPU 94%, Load 12.1, Swap 2.1GB
    15:41:20  [HOST]    sched_switch: PID 8821 (logrotate) preempted PID 4821
    15:41:22  [CUDA]    cudaStreamSync 142ms (normally 16.7ms)

  Root cause: logrotate cron job preempted training process 847 times
  Fix: Add `nice -n 19` to logrotate cron, or pin training to dedicated cores

ingero query

Query stored events by time range, PID, and operation type.

ingero query --since 1h
ingero query --since 1h --pid 4821
ingero query --since 1h --pid 4821,5032
ingero query --since 30m --op cudaMemcpy --json

Storage uses SQLite with size-based pruning (default 10 GB via --max-db). Data is stored locally at ~/.ingero/ingero.db - nothing leaves your machine.

ingero mcp

Start an MCP (Model Context Protocol) server for AI agent integration.

ingero mcp                        # stdio (for Claude Code / MCP clients)
ingero mcp --http :8080           # HTTPS on port 8080 (TLS 1.3, auto-generated self-signed cert)
ingero mcp --http :8080 --tls-cert cert.pem --tls-key key.pem  # custom TLS certificate

Note: The --http flag enables the Streamable HTTP transport - all connections use TLS 1.3 only (no plain HTTP). When no --tls-cert/--tls-key is provided, ingero auto-generates an ephemeral self-signed ECDSA P-256 certificate. Use curl -k to skip certificate verification for self-signed certs.

AI-first analysis: MCP responses use telegraphic compression (TSC) by default, reducing token count by ~60%. Set {"tsc": false} per request for verbose output.

MCP tools:

Tool

Description

get_check

System diagnostics (kernel, BTF, NVIDIA, CUDA, GPU model)

get_trace_stats

CUDA + host statistics (p50/p95/p99 or aggregate fallback for large DBs)

get_causal_chains

Causal chains with severity ranking and root cause (deduplicated, top 10 by default)

get_stacks

Resolved call stacks for CUDA/driver operations (symbols, source files, timing)

graph_lifecycle

CUDA Graph lifecycle timeline for a PID: capture, instantiate, launch sequences

graph_frequency

Graph launch frequency per executable: hot/cold classification, pool saturation

run_demo

Run synthetic demo scenarios

get_test_report

GPU integration test report (JSON)

run_sql

Execute read-only SQL for ad-hoc analysis

MCP prompts:

Prompt

Description

/investigate

Guided investigation workflow - walks the AI through stats, chains, and SQL to diagnose GPU issues. Works with any MCP client.

Works with any AI, not just Claude. Use local open-source models via ollmcp (Ollama MCP client):

# Install ollmcp and pull a model
pip install mcp-client-for-ollama
ollama pull minimax-m2.7:cloud

# Create a config pointing to Ingero's MCP server
cat > /tmp/ingero-mcp.json << 'EOF'
{"mcpServers":{"ingero":{"command":"ingero","args":["mcp","--db","trace.db"]}}}
EOF

# Start investigating - /investigate triggers the guided workflow
ollmcp -m minimax-m2.7:cloud -j /tmp/ingero-mcp.json

Tested with MiniMax M2.7 and Qwen 3.5 via Ollama on saved investigation databases. Also works with Claude Desktop, Cursor, and any MCP-compatible client.

curl examples (with --http :8080):

# System diagnostics (-k for self-signed cert)
curl -sk https://localhost:8080/mcp \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"get_check","arguments":{}}}' | jq

# Causal chains (TSC-compressed for AI)
curl -sk https://localhost:8080/mcp \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"get_causal_chains","arguments":{}}}' | jq

# Verbose output (TSC off)
curl -sk https://localhost:8080/mcp \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"get_trace_stats","arguments":{"tsc":false}}}' | jq

ingero dashboard

Start a browser-based GPU monitoring dashboard backed by the SQLite event store. Shows live system metrics, CUDA operation latencies, causal chains, and a capability manifest (grayed-out panels for metrics Ingero doesn't yet collect, with tooltips naming the required external tool). Requires ingero trace to be running (or to have run recently).

ingero dashboard                           # HTTPS on :8080 (self-signed TLS 1.3)
ingero dashboard --addr :9090              # custom port
ingero dashboard --db /path/to/ingero.db   # custom database
ingero dashboard --tls-cert cert.pem --tls-key key.pem  # custom TLS certificate

# Remote access via SSH tunnel:
ssh -L 8080:localhost:8080 user@gpu-vm
# Then open https://localhost:8080 in browser

No sudo needed - the dashboard reads from the SQLite database populated by ingero trace.

Security: TLS 1.3 only. Auto-generates an ephemeral self-signed ECDSA P-256 certificate (valid 24h) if no --tls-cert/--tls-key provided. DNS rebinding protection rejects requests from non-localhost Host headers.

API endpoints:

Endpoint

Description

GET /api/v1/overview

Event count, chain count, latest system snapshot, GPU info, top causal chain

GET /api/v1/ops?since=5m

Per-operation latency stats (percentile or aggregate mode)

GET /api/v1/chains?since=1h

Stored causal chains with severity, root cause, timeline

GET /api/v1/snapshots?since=60s

System metric time series (CPU, memory, swap, load)

GET /api/v1/capabilities

Metric availability manifest (available vs. grayed-out with required tool)

GET /api/v1/graph-metrics

CUDA Graph metrics: capture/launch rates, instantiation durations

GET /api/v1/graph-events

Recent CUDA Graph events with handles and durations

ingero demo

ingero demo                  # all 6 scenarios (incident first)
ingero demo incident         # single scenario
ingero demo gpu-steal        # also: gpu-contention, contention
ingero demo --no-gpu         # synthetic mode

ingero version

$ ingero version
ingero v0.9.0 (commit: 676ab87, built: 2026-03-17)

Stack Tracing

Stack tracing is on by default - every CUDA/Driver API event captures the full userspace call chain. Shows who called cudaMalloc - from the CUDA library up through PyTorch, your Python code, and all the way to main(). GPU-measured overhead is 0.4-0.6% (within noise on RTX 3090 through H100). Disable with --stack=false if needed.

sudo ingero trace --json               # JSON with resolved stack traces (stacks on by default)
sudo ingero trace --debug              # debug output shows resolved frames on stderr
sudo ingero demo --gpu --json          # GPU demo with stack traces (needs sudo)
ingero explain                         # post-hoc causal analysis from DB (no sudo)
sudo ingero trace --stack=false        # disable stacks if needed

Maximum depth: 64 native frames (eBPF bpf_get_stack). This covers deep call chains from CUDA → cuBLAS/cuDNN → PyTorch C++ → Python interpreter and up to main() / _start.

Python Stack Attribution

For Python workloads (PyTorch, TensorFlow, etc.), Ingero extracts CPython frame information directly from process memory. When a native frame is inside libpython's eval loop, the corresponding Python source frames are injected into the stack:

[Python] train.py:8 in train_step()
[Python] train.py:13 in main()
[Python] train.py:1 in <module>()
[Native] cublasLtSSSMatmul+0x1d4 (libcublasLt.so.12)
[Native] cublasSgemm_v2+0xa6 (libcublas.so.12)
[Native] (libtorch_cuda.so)

Supported Python versions: 3.10, 3.11, 3.12 (covers Ubuntu 22.04 default, conda default, and most production deployments). Version detection is automatic via /proc/[pid]/maps.

JSON Output with --stack

Real output from a PyTorch ResNet-50 training run on A100 SXM4 - a cuBLAS matmul kernel launch captured via Driver API uprobes, with the full call chain from Python through cuBLAS to the GPU:

{
  "timestamp": "2026-02-25T12:06:24.753983243Z",
  "pid": 11435,
  "tid": 11435,
  "source": "driver",
  "op": "cuLaunchKernel",
  "duration_ns": 10900,
  "duration": "11us",
  "stack": [
    {"ip": "0x0", "py_file": "train.py", "py_func": "train_step", "py_line": 8},
    {"ip": "0x0", "py_file": "train.py", "py_func": "main", "py_line": 13},
    {"ip": "0x0", "py_file": "train.py", "py_func": "<module>", "py_line": 1},
    {"ip": "0x765bb62cfa44", "symbol": "cublasLtSSSMatmul+0x1d4", "file": "libcublasLt.so.12.8.4.1"},
    {"ip": "0x765be7734046", "symbol": "cublasSgemm_v2+0xa6", "file": "libcublas.so.12.8.4.1"},
    {"ip": "0x765c2517fa49", "file": "libtorch_cuda.so"}
  ]
}

This kernel launch is invisible to CUDA Runtime profilers - cuBLAS calls cuLaunchKernel directly. Only Ingero's Driver API uprobes capture it.

Debug Output with --stack --debug

[DEBUG] stack trace for cuLaunchKernel (PID 11435, TID 11435, 6 frames):
[DEBUG]   [0] [Python] train.py:8 in train_step()
[DEBUG]   [1] [Python] train.py:13 in main()
[DEBUG]   [2] [Python] train.py:1 in <module>()
[DEBUG]   [3] cublasLtSSSMatmul+0x1d4 (libcublasLt.so.12)
[DEBUG]   [4] cublasSgemm_v2+0xa6 (libcublas.so.12)
[DEBUG]   [5] (libtorch_cuda.so)

OTEL Integration (Optional)

OTEL export is off by default - enabled only when you pass --otlp or --prometheus.

# Prometheus metrics endpoint (pull)
sudo ingero trace --prometheus :9090
curl localhost:9090/metrics

# OTLP push (HTTP JSON to any OTEL-compatible receiver)
sudo ingero trace --otlp localhost:4318
sudo ingero trace --otlp localhost:4318 --debug  # see OTLP push logs on stderr

OTLP uses the HTTP JSON transport (POST /v1/metrics). Compatible with: OpenTelemetry Collector, Grafana Alloy, Grafana Cloud, Datadog Agent, New Relic, and any OTLP-compatible receiver.

Metrics use OTEL semantic conventions: gpu.cuda.operation.duration, gpu.cuda.operation.count, system.cpu.utilization, system.memory.utilization, ingero.anomaly.count. Per-operation, per-source granularity.

Zero external dependencies - no OTEL SDK import. The JSON payload is constructed directly using Go's standard library.

How It Works

┌────────────────────────────────────────────────────────────────┐
│  User Space                                                    │
│                                                                │
│  ┌─────────┐    ┌─────────────┐  ┌───────┐    ┌─────────────┐  │
│  │  CUDA   │    │   ingero    │  │SQLite │    │MCP Server   │  │
│  │  App    │    │   agent     │─►│  DB   │◄───│(stdio/HTTPS)│  │
│  │(PyTorch)│    │             │  │       │    └─────────────┘  │
│  │         │    │             │  │       │   ┌───────────┐     │
│  │         │    │             │  │       │◄──│ Dashboard │     │
│  │         │    │             │  └───────┘   │  (HTTPS)  │     │
│  └──┬──┬───┘    │ ┌──────────┐│              └───────────┘     │
│     │  │        │ │ causal   ││   ┌───────────┐                │
│     │  │        │ │ engine   ││   │ OTLP /    │                │
│     │  │        │ └──────────┘│──►│ Prometheus│                │
│     │  │        └──┬──┬──┬────┘   └───────────┘                │
│     │  │           │  │  │ ▲                                   │
│     │  │           │  │  │ │ ring buffers                      │
│─────┼──┼───────────┼──┼──┼─┼───────────────────────────────────│
│     │  ▼           │  ▼  ▼ │                                   │
│     │ ┌─────────┐  │ ┌────────────────────┐                    │
│     │ │libcuda  │◄─┤ │  eBPF uprobes      │  (Driver API)      │
│     │ │  .so    │  │ │  cuLaunchKernel    │                    │
│     │ └─────────┘  │ │  cuMemcpy/Alloc    │                    │
│     ▼              │ └────────────────────┘                    │
│  ┌─────────┐       │ ┌────────────────────┐                    │
│  │libcudart│◄──────┘ │  eBPF uprobes      │  (Runtime API)     │
│  │  .so    │◄────────│  cudaLaunchKernel  │                    │
│  └─────────┘         │  cudaMalloc/Memcpy │                    │
│                      │  Graph: Capture,   │                    │
│                      │  Instantiate,Launch│                    │
│                      └────────────────────┘                    │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │  eBPF tracepoints (sched_switch, mm_page_alloc, oom,    │   │
│  │  sched_process_exec/exit/fork)                          │   │
│  └─────────────────────────────────────────────────────────┘   │
│                                                                │
│  Kernel Space        /proc → CPU%, Mem%, Load, Swap            │
└────────────────────────────────────────────────────────────────┘

  1. Discover - scans /proc for processes linked to libcudart.so, finds libcuda.so automatically

  2. Attach - eBPF probes load onto CUDA runtime uprobes, driver uprobes, and host tracepoints

  3. Capture - eBPF programs record PID, TID, timestamps into per-layer ring buffers

  4. System - reads CPU/memory/load/swap from /proc once per second

  5. Stats - computes rolling p50/p95/p99 per operation, flags anomalies

  6. Correlate - assembles causal chains (SYSTEM + HOST + CUDA Runtime + CUDA Driver + CUDA Graph) by timestamp and PID

  7. Store - writes events to SQLite with size-based pruning (--max-db 10g default). Disable recording with --record=false

  8. Export - pushes metrics via OTLP or serves Prometheus /metrics (optional)

  9. Serve - exposes diagnostics to AI agents via MCP (stdio or HTTPS/TLS 1.3)

  10. Dashboard - browser-based HTTPS dashboard reads from SQLite, shows ops/chains/snapshots/capabilities with auto-polling

Integration Testing

Validated on 6 GPU models across 3 cloud providers (TensorDock, Lambda Labs, Azure). Stack tracing is on by default. GPU-measured overhead: 0.4-1.7% (within noise).

GPU

VRAM

Tests

Pass

Fail

Warn

Stack OH

Stack Cov

GH200

480 GB

80

76

0

4

+1.6%

99.8%

A100 SXM4

40 GB

80

76

0

4

+0.9%

99.4%

A10

24 GB

80

76

0

4

-0.1%

99.2%

H100 (PCIe / SXM5)

80 GB

62

62

0

0

+1.7%

99.5%

RTX 4090

24 GB

34

34

0

0

+0.6%

99.9%

RTX 3090

24 GB

34

34

0

0

-

-

76/80 integration tests PASS (0 FAIL, 4 WARN) on GPUs tested with v0.8. Tested architectures: x86_64 and aarch64 (GH200 Grace Hopper).

What Ingero Addresses Today

Ingero addresses 25 documented GPU problems across training, inference, and AI agent workloads:

#

GPU Problem

Severity

How Ingero Detects It

1

NCCL hangs & distributed training deadlocks

CRITICAL

sched_switch shows blocked rank + CUDA sync timing. TCP retransmit tracing identifies network-caused hangs

2

GPU underutilization / data pipeline starvation

CRITICAL

Host scheduler + cudaStreamSync + cudaMemcpy pipeline bubble diagnosis. Block I/O shows DataLoader disk bottleneck

3

CUDA OOM & memory fragmentation

CRITICAL

cudaMalloc/cuMemAlloc allocation pattern tracing. cudaMallocManaged adds managed-memory over-subscription detection

4

Silent data corruption (SDC)

CRITICAL

Anomalous kernel timing as indirect signal (limited)

5

Inference cost explosion (multi-step agents)

CRITICAL

CUDA API burst/idle patterns per agent session

6

KV cache pressure & preemption cascades

CRITICAL

cudaMalloc patterns + cudaStreamSync spikes during preemption. Managed-memory page fault detection

6b

CUDA Graph re-capture latency spikes (vLLM, torch.compile)

HIGH

Graph lifecycle tracing: capture/instantiate/launch rates, pool exhaustion detection, OOM during capture, CPU contention during launch

7

GPU hardware failures at scale

HIGH

cudaMemcpy baseline drift, sched_switch frequency anomalies

8

CPU bottleneck in GPU serving

HIGH

sched_switch on inference process + cudaStreamSync idle gaps

9

GPU idle waste during agent tool execution

HIGH

CUDA API silence periods correlated with host process activity. TCP tracing shows "GPU idle during 2s HTTP tool call"

10

GPU memory leaks in long-running services

HIGH

cudaMalloc/cudaFree imbalance tracking over time, per-container via cgroup

11

Mixed precision (AMP) instability

HIGH

Anomalous kernel timing (skipped updates = fast sync)

12

Goodput loss (training efficiency gap)

HIGH

Scheduler preemption, memcpy latency, pipeline bubbles. Block I/O shows checkpoint write + data read overhead

13

GPU scheduling & orchestration failures (K8s)

HIGH

Per-cgroup sched_switch latency + pod/namespace metadata. Auto-discovers nvidia.com/gpu pods

14

Model swapping latency (multi-model agents)

HIGH

cudaMalloc + cudaMemcpy patterns during model load. Block I/O shows disk→CPU transfer time

15

CUDA device-side asserts & illegal memory access

MEDIUM

CUDA API call sequence + stack traces before crash

16

NVIDIA driver / CUDA version incompatibility

MEDIUM

Uprobe attachment failure = library/driver mismatch signal

17

Thermal throttling & power limit throttling

MEDIUM

Kernel duration trending over time

18

Noisy neighbor / multi-tenant GPU interference

MEDIUM

Per-cgroup sched_switch latency + CUDA API latency correlation. Noisy neighbor detection via cgroup_schedstat

19

Cold start / model loading latency

MEDIUM

Full cold start sequence via CUDA API timing. Block I/O completes disk→CPU→GPU pipeline

20

Multi-GPU tensor parallel communication overhead

MEDIUM

Host-side straggler detection via sched_switch + CUDA sync. TCP retransmit tracing on NCCL ports

21

RAG pipeline GPU contention

MEDIUM

Per-process CUDA API breakdown (explain --per-process) - shows which process is hogging GPU time

22

Checkpoint save/load failures

MEDIUM

Memory spike detection + I/O blocking in cudaStreamSync. Block I/O shows actual write latency + NFS timeouts

23

PCIe bottleneck (KV cache swap, model loading)

MEDIUM

cudaMemcpy per-operation tracing with direction/size/duration. cudaMallocManaged page migration + Block I/O shows NVMe-PCIe contention

24

Loss spikes (non-AMP)

LOW-MED

System event correlation with loss timing

25

Triton Inference Server multi-GPU bugs

LOW-MED

CUDA API tracing on Triton processes

FAQ

Is it safe for production? Yes. eBPF programs are verified by the kernel before loading - they cannot crash the system. Probes add <2% overhead including stack tracing (0.4-0.6% measured across RTX 3090, RTX 4090, A10, A100, H100 with PyTorch workloads).

Does it require code changes? No. Ingero attaches to libcudart.so and kernel tracepoints at the OS level. Your application code is untouched. Traces any language - Python, C++, Java - anything linked against libcudart.so.

What GPUs are supported? Any NVIDIA GPU with driver 550+ and CUDA 11.x/12.x. Tested on GH200 (aarch64), H100, A100, A10, RTX 4090, RTX 3090 (x86_64). Works on AWS Deep Learning AMIs (auto-discovers versioned libcudart.so).

Does it work in containers? Yes. eBPF programs execute in kernel space - the container just loads them via syscalls. Run with --privileged (or --cap-add=BPF,PERFMON,SYS_ADMIN), --pid=host, and mount /proc, /sys/kernel/debug, and /sys/kernel/btf. The host kernel must have BTF enabled. Pre-built images are available at ghcr.io/ingero-io/ingero - see the Docker Image install section. This is the same pattern used by Falco, Tetragon, and other eBPF DaemonSets.

Where is data stored? Locally in ~/.ingero/ingero.db (SQLite). Nothing leaves your machine. Size-based pruning keeps the DB under 10 GB by default. With --record-all, this covers a few hours of heavy GPU load; with selective storage (default), it lasts much longer. Configure with --max-db (e.g., --max-db 500m, --max-db 0 for unlimited). Use --db /path/to/file.db for a custom location.

Does it check for updates? Yes. On interactive commands (trace, demo, explain, check), ingero checks GitHub Releases for newer versions (once per 24 hours, cached in ~/.ingero/update-check). The check runs in the background and never delays your command. Set INGERO_NO_UPDATE_NOTIFIER=1 to disable. Skipped for query, mcp, version, and dev builds.

License

Ingero is 100% free and open source. Use it for anything - personal, commercial, enterprise, embed it in your product, modify it, redistribute it. No usage restrictions, no phone-home, no paid tiers required.

Dual-licensed following the standard eBPF split-licensing model (same as Cilium, Falco, and most eBPF projects):

  • User-Space (Go agent, CLI, causal engine, SQLite, MCP): Apache License 2.0 - maximum enterprise compatibility, no copyleft.

  • Kernel-Space (eBPF C code in bpf/): GPL-2.0 OR BSD-3-Clause - GPL-2.0 is required by the Linux kernel's BPF subsystem; BSD-3-Clause permits embedding in non-GPL toolchains.

Available Tools

11 tools
get_causal_chainsA

Analyze CUDA + host events and return causal chains with severity, root cause, and recommendations. Deduplicates by operation, returns top 10 by default (use top_n to adjust). AI-first: TSC-compressed by default. Works with both live and saved/offline databases. Omit 'since' for saved DBs.

ParametersJSON Schema
NameRequiredDescriptionDefault
pidNofilter by single process ID. 0 = all. Deprecated: use pids.
tscNotelegraphic compression (default: true)
pidsNofilter by process ID(s). Takes precedence over pid.
sinceNotime range relative to NOW, e.g. 1m, 5m. Omit for saved/offline DBs to query ALL events. Only useful during live tracing.
top_nNomax chains to return (default 10). Deduplicates by operation, keeps highest severity. Use 0 for all.

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Discloses key behaviors: deduplicates by operation, TSC-compressed by default, works with live and saved DBs. No annotations exist, so description carries full burden; it implies read-only analysis but doesn't explicitly state no mutations or permissions needed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise, front-loaded with purpose, and each sentence adds meaningful information without redundancy. No wasted words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with no output schema, it sufficiently describes return values. Parameters are all covered with usage notes. Minor omissions like error handling or edge cases do not significantly detract.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so baseline is 3. The description adds value by explaining top_n default, 'since' usage for live vs saved, and TSC compression context ('AI-first'). This goes beyond the schema's descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool analyzes CUDA + host events and returns causal chains with severity, root cause, and recommendations. This specific verb+resource combination distinguishes it from sibling tools like get_stacks or get_trace_stats.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides guidance on when to omit 'since' (for saved DBs) and explains default behavior (top 10). However, it lacks explicit comparison to alternatives or when not to use this tool versus siblings.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_checkA

Run system diagnostics: kernel version, BTF support, NVIDIA driver, CUDA libraries, running GPU processes

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations exist, so the description carries full burden. It lists all the checks performed, making behavior clear. However, it does not mention side effects, permissions, or output format, though diagnostic tools are typically read-only.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, front-loaded sentence with a colon then list of specifics. Every word adds value, no repetition or fluff.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given no parameters, no output schema, and simple read-only diagnostics, the description adequately covers purpose and scope. It could mention output format but is sufficient for agent understanding.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

There are zero parameters, and schema coverage is 100%. The description adds value by explaining the tool's action without needing parameter details. Baseline for 0 params is 4.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb 'Run system diagnostics' and lists specific resources (kernel version, BTF support, NVIDIA driver, CUDA libraries, running GPU processes). This distinguishes it from siblings like 'run_sql' or 'pagerduty_trigger'.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for system diagnostics but offers no explicit guidance on when to use this tool vs alternatives, nor does it state when not to use it.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_stacksA

Get resolved call stacks for CUDA/driver operations. Returns top stacks by frequency with symbol names, source files, and timing stats. One call answers 'what code path caused this operation?' For older DBs without resolved symbols, falls back to raw IPs (hex addresses).

ParametersJSON Schema
NameRequiredDescriptionDefault
opNoOperation name (e.g. cudaMalloc, cuLaunchKernel)
pidNoProcess ID filter
tscNotelegraphic compression (default: true)
limitNoMax stacks returned (default 10)
sinceNoTime window relative to NOW (e.g. 5m, 1h). Omit for saved/offline DBs to query ALL data.
sourceNoSource filter: 1=CUDA, 3=HOST, 4=DRIVER

TDQS

A4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden. It discloses a fallback behavior for older DBs without resolved symbols (raw IPs) and describes return content, but does not mention if the operation is read-only, idempotent, or any other behavioral traits like side effects or rate limits.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is three sentences long, each serving a clear purpose: stating what the tool does, describing return content, and providing a use case with edge case behavior. No redundant information, and the most important information is front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given no output schema, the description covers return format (top stacks, symbol names, etc.) and a fallback scenario. It doesn't explain the tsc compression parameter or source filter values, but these are documented in the schema. For a moderate-complexity tool, it is largely complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all 6 parameters adequately. The description adds no additional semantic value beyond the schema, meeting the baseline expectation but not exceeding it.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool gets resolved call stacks for CUDA/driver operations, returns top stacks by frequency with symbol names, source files, and timing stats, and explicitly answers 'what code path caused this operation?' This specific verb+resource combination effectively distinguishes it from sibling tools like get_trace_stats.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage by stating 'One call answers what code path caused this operation?' providing clear context for when to use. However, it does not explicitly mention when not to use or provide alternatives, missing some guidance for an AI agent.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_test_reportA

Get the GPU integration test report (JSON). Generated by gpu-test.sh after a full test run. Includes per-test status, timing, and system info.

ParametersJSON Schema
NameRequiredDescriptionDefault
tscNotelegraphic compression (default: true)

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It discloses that the output is JSON with per-test status, timing, and system info, which is adequate for a read operation.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences with no wasted words. The main action is front-loaded, and the additional details are concise and relevant.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given a simple optional parameter, no output schema, and no annotations, the description covers the tool's purpose, output format, and content comprehensively.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The single parameter 'tsc' is described in the schema as 'telegraphic compression (default: true)', and the tool description adds no further clarification. With 100% schema coverage, baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states 'Get the GPU integration test report (JSON)' which specifies both the resource and format. It is distinct from sibling tools like get_causal_chains or get_stacks, which focus on different data.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description mentions 'Generated by gpu-test.sh after a full test run', providing context on when the report is available. It lacks explicit when-not-to-use guidance, but for a straightforward retrieval tool this is sufficient.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_trace_statsA

Get CUDA and host operation statistics. Returns p50/p95/p99 for small DBs (≤500K events), count/avg/min/max from aggregates for large DBs. Works with both live and saved/offline databases. Omit 'since' for saved DBs.

ParametersJSON Schema
NameRequiredDescriptionDefault
tscNotelegraphic compression (default: true). Set false for verbose output.
sinceNotime range relative to NOW, e.g. 1m, 5m, 1h. Omit for saved/offline DBs to query ALL events. Only useful during live tracing.

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations, so description carries full burden. Discloses behavior for small vs large DBs and live vs saved databases. No side effects mentioned, but for a read-only tool this is sufficient.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three sentences, tightly written with no redundant information. Front-loaded with purpose, then details on statistics, then usage guidance.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

No output schema, but description explains return values based on DB size. Covers both live and saved databases. No missing critical information for a statistics retrieval tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with descriptions. Description adds value by explaining 'since' usage (omit for saved DBs) and tsc (telegraphic compression). Adds context beyond schema definitions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Clearly states it gets CUDA and host operation statistics with specific return values (p50/p95/p99 for small DBs, count/avg/min/max for large DBs). Distinguishes from sibling tools like get_causal_chains or get_check.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly says to omit 'since' for saved DBs and use during live tracing. Provides clear context for when the tool is applicable. Could be improved by stating when not to use this tool vs alternatives.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

graph_frequencyB

Analyze CUDA Graph launch frequency per executable. Identifies hot graphs (high replay rate), cold graphs (captured but rarely launched), and graph pool saturation. Essential for vLLM batch size tuning.

ParametersJSON Schema
NameRequiredDescriptionDefault
pidYesProcess ID to query graph launch frequency for (required)
tscNotelegraphic compression (default: true)
sinceNoTime range, e.g. 5m, 1h. Omit for saved DBs.
window_secondsNoAnalysis window in seconds (default 60)

TDQS

B3.4/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries full burden. It describes output categories but fails to disclose whether the tool is read-only, any prerequisites, or performance impact. This is minimal disclosure for a tool that likely queries system data.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences with no fluff, front-loading the purpose and then providing specific analysis categories and a use case. Every sentence earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With 4 parameters, no output schema, and no annotations, the description should explain return format and default behaviors. It only describes analysis categories and a use case, leaving gaps about output structure and error handling.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the schema already describes parameters. The description adds no new parameter details beyond the schema, achieving the baseline score.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description explicitly states the tool analyzes CUDA Graph launch frequency per executable and identifies hot graphs, cold graphs, and graph pool saturation. This clearly differentiates it from sibling tools like graph_lifecycle.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description mentions it is 'Essential for vLLM batch size tuning,' which implies a use case but does not specify when not to use it or provide alternatives among the 10 sibling tools.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

graph_lifecycleA

Show CUDA Graph lifecycle timeline for a PID: capture → instantiate → launch sequences with timestamps and durations. Identifies graph activity patterns in torch.compile and vLLM workloads.

ParametersJSON Schema
NameRequiredDescriptionDefault
pidYesProcess ID to query graph events for (required)
tscNotelegraphic compression (default: true)
sinceNoTime range, e.g. 5m, 1h. Omit for saved DBs.

TDQS

A3.7/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, so description carries full burden. It indicates read-only query behavior (showing timeline) but does not explicitly state safety, permissions, or side effects. The description is adequate but lacks definitive behavioral traits.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, each adding value: first defines action and output, second provides context. No fluff, front-loaded with key information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given no output schema and no annotations, description should cover output format. It mentions timestamps and durations but not structure (e.g., list, graph). Also lacks limitations or edge cases. Adequate but incomplete for a tool with no output schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so baseline 3. Description does not add significant meaning beyond schema; it mentions PID implicitly but does not elaborate on tsc or since parameters. The description adds context about output (timestamps, durations) but not parameter details.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description clearly states it shows CUDA Graph lifecycle timeline for a PID, specifying sequences (capture, instantiate, launch) with timestamps and durations. It differentiates from sibling tools like graph_frequency by focusing on timeline rather than frequency.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Description implies usage for analyzing graph activity in PyTorch workloads but does not provide explicit when-to-use or when-not-to-use guidance, nor does it mention alternatives like graph_frequency or other tools.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

pagerduty_triggerA

Create a PagerDuty incident with rich context. For AI-driven escalation during investigations.

ParametersJSON Schema
NameRequiredDescriptionDefault
sourceNoIdentifies origin (default: ingero)
summaryYesOne-line incident summary,required
severityYesPD severity,required,enum=info,enum=warning,enum=error,enum=critical
dedup_keyNoPD dedup key; auto-generated UUID when omitted
custom_detailsNoFree-form context (max 256 KiB serialized)

TDQS

A3.5/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description must cover behavioral traits. It states creation but does not disclose side effects (e.g., triggering alerts), authorization requirements, rate limits, or the fact that dedup_key enables idempotency. The description is too minimal given the absence of annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two concise sentences that front-load the action and then provide usage context. No unnecessary words, though the first sentence could be slightly more specific about 'rich context'.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With five parameters (including a nested object), no output schema, and no annotations, the description is too sparse. It omits return values, error behavior, and usage constraints like the 256 KiB limit (only covered in schema). The completeness is inadequate for reliable tool selection.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has 100% coverage, describing all five parameters clearly. The description adds 'rich context' but does not enhance understanding beyond the schema. Baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('Create a PagerDuty incident') and the resource, with a specific use case ('AI-driven escalation during investigations'). It effectively distinguishes from the sibling tools, which are primarily read or query operations.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides a clear context for use ('during investigations'), but lacks explicit when-not-to-use guidance or alternative tool references. However, given the sibling tools are all read-oriented, the usage context is sufficiently clear.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

query_fleetA

Query multiple Ingero nodes and return merged results. Requires fleet.nodes configured in ingero.yaml. Actions: chains (causal chains sorted by severity), ops (per-op stats), overview (summary per node), sql (raw SQL fan-out with node column).

ParametersJSON Schema
NameRequiredDescriptionDefault
pidNoFilter by PID (optional, used with ops action)
sqlNoAlias for query — SQL query (required when action is sql)
tscNoTelegraphic compression (default: true). Set false for verbose output.
limitNoMax rows per node (default 1000)
queryNoSQL query to execute across fleet nodes (required when action is sql)
sinceNoTime window (e.g. 5m, 1h). Default: 5m.
actionYesQuery type: chains/ops/overview/sql,enum=chains,enum=ops,enum=overview,enum=sql,required

TDQS

A3.7/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It mentions merged results and prerequisites but lacks detail on error handling, authentication, rate limits, or side effects. The behavior of actions like 'sql' is only hinted at ('raw SQL fan-out with node column').

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is short (two sentences plus a list) and front-loaded with the core purpose. Every sentence adds distinct information: purpose, prerequisites, and action descriptions. No redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

While the description covers prerequisites and action types, it omits details like default limit, required permissions, pagination, or what happens on error. With no output schema and no annotations, it could be more comprehensive for a complex tool with 7 parameters.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, but the description adds valuable context beyond the schema by explaining each action value (e.g., 'chains (causal chains sorted by severity)') and noting default compression (tsc). This helps the agent understand parameter semantics.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool queries multiple Ingero nodes and returns merged results, with a specific verb ('Query') and resource ('multiple Ingero nodes'). It also lists distinct actions (chains, ops, overview, sql), distinguishing it from siblings like get_causal_chains or run_sql.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides a prerequisite ('Requires fleet.nodes configured in ingero.yaml') and briefly explains each action's purpose. However, it does not explicitly state when to use this tool versus alternatives like get_causal_chains or run_sql, leaving usage context implicit.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

run_demoC

Run a synthetic demo scenario and return the stats snapshot. No GPU or root needed.

ParametersJSON Schema
NameRequiredDescriptionDefault
scenarioNoscenario name: incident, cold-start, memcpy-bottleneck, periodic-spike, cpu-contention, gpu-steal

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description is the sole source. It states the tool is synthetic and requires no GPU/root, implying safety, but does not disclose side effects, idempotency, or if it modifies state.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single sentence with no wasted words, efficiently conveying the core action. Could be slightly more structured but remains concise.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple tool with one parameter and no output schema, the description is adequate but lacks details about the returned stats snapshot or any behavioral guarantees.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% and the schema includes a description listing valid values. The description adds no further meaning beyond the schema, so baseline score applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb 'run' and resource 'synthetic demo scenario', and mentions the output 'stats snapshot'. However, it does not explicitly differentiate from sibling tools, but given the context, it is distinct enough.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when to use this tool vs alternatives. The only constraint mentioned ('No GPU or root needed') is a requirement, not usage scenario.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

run_sqlA

Execute read-only SQL on the Ingero database. For ad-hoc analysis the fixed tools can't do: temporal bucketing, threshold queries, per-PID breakdowns, throughput calculations. Timeout: 30s.

Schema: events(id, timestamp INT nanos, pid, tid, source, op, duration INT nanos, gpu_id, arg0, arg1, ret_code, stack_hash, cgroup_id INT default 0, comm TEXT default '' — process name from bpf_get_current_comm(), v0.10+, empty for pre-v0.10 rows), system_snapshots(id, timestamp, cpu_pct, mem_pct, mem_avail, swap_mb, load_avg), causal_chains(id TEXT, detected_at, severity, summary, root_cause, explanation, recommendations JSON, cuda_op, cuda_p99_us, cuda_p50_us, tail_ratio, timeline JSON), sessions(id, started_at, stopped_at, gpu_model, gpu_driver, cpu_model, cpu_cores, mem_total, kernel, os_release, cuda_ver, python_ver, ingero_ver, pid_filter, flags), sources(id, name, description), ops(source_id, op_id, name, description), process_names(pid, name, seen_at — LEGACY: lazy /proc-based PID→name table, used as read-side fallback when events.comm is empty), event_aggregates(bucket, source, op, pid, count, stored, sum_dur, min_dur, max_dur, sum_arg0), stack_traces(hash, ips TEXT JSON, frames TEXT JSON resolved symbols), cgroup_metadata(cgroup_id PK, container_id TEXT, cgroup_path TEXT), cgroup_schedstat(cgroup_id PK, p99_off_cpu_ns, total_off_cpu_ns, event_count, window_start, window_end), schema_info(key, value).

JOINs: events.source=sources.id, events.(source,op)=ops.(source_id,op_id), events.stack_hash=stack_traces.hash, events.cgroup_id=cgroup_metadata.cgroup_id (K8s container context), events.pid=process_names.pid (ALWAYS qualify pid as e.pid when joining - pid exists in both tables). For process names prefer events.comm directly (faster, no JOIN); use COALESCE(NULLIF(e.comm,''), NULLIF(pn.name,''), '') only when also reading legacy pre-v0.10 rows. Sources: 1=CUDA, 3=HOST, 4=DRIVER, 5=IO, 6=TCP, 7=NET. CUDA ops: 1=cudaMalloc, 2=cudaFree, 3=cudaLaunchKernel, 4=cudaMemcpy, 5=cudaStreamSync, 6=cudaDeviceSync, 7=cudaMemcpyAsync, 8=cudaMallocManaged. HOST ops: 1=sched_switch, 2=sched_wakeup, 3=mm_page_alloc, 4=oom_kill, 5=process_exec, 6=process_exit, 7=process_fork, 10=pod_restart, 11=pod_eviction, 12=pod_oom_kill. DRIVER ops: 1=cuLaunchKernel, 2=cuMemcpy, 3=cuMemcpyAsync, 4=cuCtxSynchronize, 5=cuMemAlloc, 6=cuMemAllocManaged. IO ops: 1=block_read, 2=block_write, 3=block_discard. TCP ops: 1=tcp_retransmit. NET ops: 1=net_send, 2=net_recv. arg0/arg1 per op: cudaMalloc/cudaMallocManaged arg0=size_bytes, cudaFree arg0=devPtr, cudaLaunchKernel arg0=kernel_func_ptr, cudaMemcpy/cudaMemcpyAsync arg0=bytes arg1=direction(0=H2H,1=H2D,2=D2H,3=D2D,4=default), cudaStreamSync arg0=stream_handle, mm_page_alloc arg0=page_order(size=4KB<<order), cuMemAlloc/cuMemAllocManaged arg0=size_bytes, block_read/block_write arg0=nr_sectors, net_send/net_recv arg0=bytes. sum_arg0 in event_aggregates = sum of arg0 across bucket (skipped for pointer-valued ops: cudaFree, cudaLaunchKernel, cuLaunchKernel). Timestamps: unix nanos. Duration: nanos (÷1e3=µs, ÷1e6=ms).

Performance: events can have millions of rows. For large DBs, query event_aggregates (per-minute stats, always small) or stack_traces (deduplicated, always small) instead of scanning events. Use get_stacks tool for call stack analysis instead of manual SQL JOINs.

ParametersJSON Schema
NameRequiredDescriptionDefault
tscNotelegraphic compression (default: true)
limitNomax rows returned (default 1000, max 10000)
queryYesRead-only SQL (SELECT/WITH/EXPLAIN). See tool description for schema.

TDQS

A4.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, but the description fully discloses read-only behavior, timeout, schema details, performance notes, and hints to use aggregate tables, providing comprehensive behavioral context.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is long but well-organized with clear sections; each sentence adds value, though the detail on per-op arg mapping could be condensed.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Despite lacking an output schema, the description provides all necessary context about what the tool returns, including schema, joins, performance, and usage tips.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, and the description adds extensive detail for the query parameter by providing the entire database schema, while also clarifying tsc and limit defaults.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states 'Execute read-only SQL on the Ingero database' and specifies it is for ad-hoc analysis that fixed tools cannot handle, distinguishing it from siblings.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It explicitly lists use cases (temporal bucketing, threshold queries, etc.) and recommends alternatives like get_stacks for call stack analysis, but does not exhaustively list when not to use.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.

  1. 11 tool updatesv0.19.0
    • Addedget_causal_chains
    • Addedget_check
    • Addedget_stacks
    • Addedget_test_report
    • Addedget_trace_stats
    • Addedgraph_frequency
    • Addedgraph_lifecycle
    • Addedpagerduty_trigger
    • Addedquery_fleet
    • Addedrun_demo
    • Addedrun_sql
  2. 11 tool updatesv0.18.0
    • Removedget_causal_chains
    • Removedget_check
    • Removedget_stacks
    • Removedget_test_report
    • Removedget_trace_stats
    • Removedgraph_frequency
    • Removedgraph_lifecycle
    • Removedpagerduty_trigger
    • Removedquery_fleet
    • Removedrun_demo
    • Removedrun_sql
  3. 1 tool updatev0.14.2
    • Addedpagerduty_trigger
  4. 10 tool updatesv0.8.1
    • Changedget_causal_chains3 fields changed
      • changedInput schema / properties / since / description
        Previous value: -"time range, e.g. 1m, 5m. Default: all data (0 = no time filter)"New value: +"time range relative to NOW, e.g. 1m, 5m. Omit for saved/offline DBs to query ALL events. Only useful during live tracing."
      • addedInput schema / properties / top_n
        Added value: +{
+  "description": "max chains to return (default 10). Deduplicates by operation, keeps highest severity. Use 0 for all.",
+  "type": "integer"
+}
      • changedOutput schema / (root)
        Previous value: -{
-  "additionalProperties": false,
-  "type": "object"
-}New value: +null
    • Changedget_check1 field changed
      • changedOutput schema / (root)
        Previous value: -{
-  "additionalProperties": false,
-  "type": "object"
-}New value: +null
    • Changedget_stacks2 fields changed
      • changedInput schema / properties / since / description
        Previous value: -"Time window (e.g. 5m, 1h). Default: all data"New value: +"Time window relative to NOW (e.g. 5m, 1h). Omit for saved/offline DBs to query ALL data."
      • changedOutput schema / (root)
        Previous value: -{
-  "additionalProperties": false,
-  "type": "object"
-}New value: +null
    • Changedget_test_report1 field changed
      • changedOutput schema / (root)
        Previous value: -{
-  "additionalProperties": false,
-  "type": "object"
-}New value: +null
    • Changedget_trace_stats2 fields changed
      • changedInput schema / properties / since / description
        Previous value: -"time range, e.g. 1m, 5m, 1h. Default: all data (0 = no time filter)"New value: +"time range relative to NOW, e.g. 1m, 5m, 1h. Omit for saved/offline DBs to query ALL events. Only useful during live tracing."
      • changedOutput schema / (root)
        Previous value: -{
-  "additionalProperties": false,
-  "type": "object"
-}New value: +null
    • Addedgraph_frequency
    • Addedgraph_lifecycle
    • Addedquery_fleet
    • Changedrun_demo1 field changed
      • changedOutput schema / (root)
        Previous value: -{
-  "additionalProperties": false,
-  "type": "object"
-}New value: +null
    • Changedrun_sql1 field changed
      • changedOutput schema / (root)
        Previous value: -{
-  "additionalProperties": false,
-  "type": "object"
-}New value: +null
Install Server
RayrunBuild, deploy, and operate MCP servers without becoming an infrastructure team.ray.runAd

TDQS

A3.8/5.0
Disambiguation5/5

Each tool has a distinct purpose: diagnostic checks, causal chain analysis, call stacks, graph analysis, fleet queries, etc. There is no overlap or ambiguity, making selection straightforward for an agent.

Naming Consistency4/5

Most tools follow a 'verb_noun' pattern (get_*, graph_*, run_*, query_*), which is clear and predictable. However, 'pagerduty_trigger' deviates slightly (noun_verb), and the mix of prefixes (get, graph, run, query) slightly reduces consistency.

Tool Count5/5

With 11 tools, the server covers its domain comprehensively without being overwhelming. Each tool serves a well-defined function, and the count feels appropriate for a GPU observability and analysis tool.

Completeness4/5

The core workflows (diagnostics, statistics, call stacks, causal analysis, graph analysis, fleet queries, SQL access, alerting) are covered. Minor gaps like listing sessions or managing configurations are absent, but these are not essential for the main use case.

Maintenance

ActivityNo data
ResponsivenessSyncing

Resources

Related MCP Connectors

Related MCP Servers

  • Prometheus MCP

    etruong42
    A
    license
    B
    quality
    D
    maintenance
    A proof-of-concept Prometheus MCP server, which likely enables Claude AI to interact with Prometheus monitoring systems through the Model Context Protocol.
    2
    1
    MIT

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/ingero-io/ingero'

If you have feedback or need assistance with the MCP directory API, please join our Discord server