Source/assembly annotation: shows per-line or per-instruction sample percentages inside a specific function.

Use this after perf_report to drill into a hot function and see exactly which lines or instructions are consuming time.

Key parameters:

• symbol: function name to annotate (default: hottest symbol).

• dsos: restrict to a specific binary/library.

• source: true (default) to interleave source code with assembly.

• disassembler_style: 'intel' for Intel syntax (default: AT&T).

• percent_type: 'local-period' (default), 'global-period', 'local-hits', 'global-hits'.

• data_type: annotate a specific data type (DWARF data-type profiling).

• code_with_type: show data type annotations on code.

Output: source/assembly listing with % annotations per line. Requires: debuginfo packages for source interleaving. Works on any perf.data from perf record.

List the ELF build-id hashes for binaries referenced in perf.data.

Use this to verify symbol resolution will work: compare build IDs against installed debuginfo packages.

Key parameters:

• with_hits: only show DSOs that have actual samples.

• kernel: show the running kernel's build ID.

• kernel_maps: show kernel build ID with address map.

Output: ' ' per line. Works on any perf.data.

Cache-to-cache false sharing analysis. Identifies cache lines with the most HITM (Hit Modified) events — the primary indicator of cross-core cache contention.

Use this to diagnose multi-threaded performance issues caused by different threads accessing the same cache line.

Key parameters:

• display: 'tot' (total HITMs, default), 'lcl' (local/same-socket), 'rmt' (remote/cross-socket).

• coalesce: group results by 'tid,pid,iaddr,dso' to see per-thread or per-address breakdown.

• stats: true to show only summary statistics.

• call_graph: enable callchain display.

Output: multi-section report — shared data cache line table, per-cacheline detail with offsets and symbols. Requires: perf record -d -a (with memory data recording).

Convert perf.data to JSON or CTF (Common Trace Format).

Use this to export data for processing in external tools.

Key parameters:

• to_json: output JSON file path (e.g. '/tmp/perf.json').

• to_ctf: output CTF directory path.

• all: include all events, not just samples.

• tod: convert timestamps to wall-clock time.

Exactly one of to_json or to_ctf must be specified. Output: returns the output file/directory path.

Compare two perf.data profiles side by side. Shows per-symbol overhead changes between a baseline and a new measurement.

Use this for before/after comparisons — optimization validation, regression detection, or A/B testing.

Key parameters:

• old_input: path to baseline perf.data (required).

• new_input: path to comparison perf.data (required).

• compute: comparison method — 'delta' (default, percentage-point difference), 'delta-abs' (absolute), 'ratio' (new/old), 'wdiff' (weighted), 'cycles' (cycle-level).

• formula: true to show the computation formula.

• symbols/dsos/comms: filter scope.

• baseline_only: show only symbols present in baseline.

Output: differential table with baseline%, new%, and delta columns. Both files must be from perf record with compatible events.

List events in a perf.data file. Always call this first to understand what was recorded before using other tools.

Shows each event's name (e.g. 'cpu/cycles/'), type, config, and sampling settings. Use freq=true to see sample frequencies, group=true to see event grouping, verbose=1 for full perf_event_attr details.

Output: one event per line (e.g. 'cpu/cycles/Pu'). No prerequisites — works on any perf.data.

Transform a perf.data file: inject build IDs, decode hardware traces, merge scheduler events, or process JIT data. Writes a new perf.data.

Use this as a preprocessing step before perf_report or perf_script.

Key parameters:

• input: source perf.data path (required).

• output: destination perf.data path (required).

• build_ids: inject build-id headers for symbol resolution.

• itrace: decode hardware traces. Values: 'i0ns' = synthesize instructions, 'b' = synthesize branches, 'c' = synthesize calls, 'e' = synthesize errors.

• jit: process JIT-compiled code mappings.

• sched_stat: merge sched_stat and sched_switch events.

Output: writes a new perf.data file, returns path and size.

Look up a kernel symbol by name. Returns address, type, and module.

Does NOT require a perf.data file — reads the running kernel's symbol table directly.

Parameters:

• symbol: kernel function/variable name to look up (required).

• verbose: increase detail level (0-2).

Output: ' [module]'.

Kernel memory allocation statistics: slab and page allocator activity with per-callsite breakdown.

Use this to find excessive allocators, fragmentation, or leaks.

Key parameters:

• slab: analyze slab allocator (kmalloc/kmem_cache).

• page: analyze page allocator.

• caller: show per-callsite statistics.

• live: show only allocations not yet freed (leak detection).

• sort: 'ptr', 'callsite', 'bytes_req', 'bytes_alloc', 'hit', 'pingpong', 'frag'.

Output: allocation statistics table. Requires: perf kmem record.

KVM virtual machine exit analysis: shows VM exit reasons, counts, and time per exit type.

Use this to diagnose virtualization overhead — excessive HLT exits indicate idle guests, I/O exits indicate slow device emulation, EPT violations indicate memory mapping churn.

Key parameters:

• key: sort by 'sample' (count), 'time', 'max', 'min'.

• event: filter to specific exit type (e.g. 'HLT').

• vcpu: filter to specific vCPU.

• guest/host: filter to guest or host events.

Output: exit reason table with count, time, and percentage. Requires: perf kvm stat record.

Kernel work item statistics: IRQ, softIRQ, and workqueue handlers with count, total runtime, and max latency.

Use this to find the most expensive interrupt handlers or work items.

Key parameters:

• sort: 'count', 'runtime', 'max', 'avg'.

• name: filter to a specific handler name.

• kwork: filter by type -- 'irq', 'softirq', 'workqueue'.

Output: per-handler statistics table. Requires: perf kwork record.

Kernel work item latency breakdown showing scheduling delay for each handler.

Key parameters:

• sort: 'count', 'max', 'avg'.

• name: filter to specific handler.

• kwork: filter by type.

Output: per-handler latency table. Requires: perf kwork record.

Timestamped kernel work item events showing when each handler ran and for how long.

Key parameters:

• call_graph: 'fp' or 'dwarf' for callchain.

• name: filter to specific handler.

Output: per-event timeline. Requires: perf kwork record.

Top kernel work items ranked by total runtime.

Quick view of the busiest interrupt/softirq/workqueue handlers.

Key parameters:

• sort: ranking metric -- 'runtime' (default), 'count', 'max'.

Output: ranked handler list. Requires: perf kwork record.

Lock statistics: acquired count, contended count, and wait times per lock.

Use this to find the most contended locks in the system.

Key parameters:

• sort: sort by 'acquired', 'contended', 'avg_wait', 'wait_total', 'wait_max', 'wait_min'.

• type_filter: filter by lock type -- 'spinlock', 'mutex', 'rwsem:R', 'rwsem:W'.

• threads: show per-thread breakdown.

• combine_locks: group locks by caller.

Output: per-lock statistics table. Requires: perf lock record or lock tracepoints.

Detailed lock contention analysis showing where threads wait for locks, with optional BPF-based tracking.

Key parameters:

• lock_addr: show lock addresses.

• lock_owner: show which task holds the lock.

• callstack_filter: filter by callstack pattern.

• use_bpf: use BPF for live contention tracing.

• type_filter: filter by lock type.

• stack_skip: skip N stack frames.

Output: contention table with wait times and optional stacks. Requires: perf lock record or lock tracepoints.

Display general information about locks in perf.data.

Shows lock types and configurations found in the recording.

Output: lock type and configuration summary. Requires: perf lock record.

Memory access profiling: data source (L1/L2/L3/DRAM), latency, and load/store breakdown.

Use this to find cache-miss-heavy code or NUMA-unfriendly access patterns.

Key parameters:

• sort: sort key for the histogram.

• type: 'load' or 'store' to filter access type.

• ldlat: load latency threshold in CPU cycles.

• type_profile: show data type profile (DWARF-based).

• phys_data: show physical memory addresses.

Output: memory access histogram with data source breakdown. Requires: perf mem record (or perf record -d).

Histogram profiling: shows which functions consumed the most CPU time (or other events) as a ranked overhead table.

This is the primary analysis tool. Use it to answer 'where is time spent?'

Key parameters:

• sort: columns to group by. Default 'comm,dso,symbol'. Use 'srcline' for source lines, 'pid,tid' for threads, 'dso' for libraries.

• call_graph: enable callchain. Use 'graph,0.5,caller,function,percent' for a standard caller-based call graph with 0.5% threshold.

• percent_limit: hide entries below N% (e.g. 1.0 to show only >1%).

• symbols: filter to specific function(s).

• dsos: filter to specific DSO(s).

• time: restrict to time range 'start,stop' in seconds.

• header_only: show file metadata without the histogram.

• mem_mode: switch to memory access profiling (needs perf record -d).

• branch_stack: switch to branch profiling (needs perf record -b).

• latency: show latency-centric view (needs perf record --latency).

• children: set to false (--no-children) to show self overhead only.

Output: table with columns like '% overhead | command | DSO | symbol'. Works on any perf.data from perf record.

Per-task scheduling latency statistics. Shows max, average, and total scheduling delay per task.

Use this to identify tasks that are being starved or experiencing long scheduling delays.

Key parameters:

• sort: sort key -- 'max' (default), 'switch', 'runtime', 'avg'.

• pid: filter to specific PIDs.

Output: table with task name, max latency, avg latency, switch count. Requires: perf sched record.

Timestamped scheduler timeline showing every context switch with runtime and scheduling delay.

Use this for detailed scheduling analysis -- find when and why tasks were descheduled.

Key parameters:

• summary: true to show only summary stats (no per-event detail).

• wakeups: show wakeup events between switches.

• migrations: show CPU migration events.

• idle_hist: show idle-time analysis.

• state: show task state (R/S/D/T) at each switch.

• call_graph: 'fp' or 'dwarf' for callchain at each switch.

• with_summary: show both detail and summary.

• comms: filter to specific task names.

• time: restrict to time range.

Output: per-event table with timestamp, task, runtime, wait-time, scheduling delay. Requires: perf sched record.

ASCII CPU activity map showing which task ran on which CPU at each time slice.

Use this for a visual overview of scheduling patterns, CPU affinity issues, and load imbalance.

Key parameters:

• compact: one-character-per-task view.

• cpus: restrict to specific CPUs.

• pids: restrict to specific PIDs.

• color_pids/color_cpus: highlight specific PIDs/CPUs.

Output: ASCII grid with CPUs as rows and time as columns. Requires: perf sched record.

Dump raw scheduler tracepoint events from perf.data.

Use this for custom analysis or when the structured views (latency, timehist, map) don't show what you need.

Output: raw tracepoint event lines. Requires: perf sched record.

Replay recorded scheduler events to simulate the original scheduling.

Replays the workload's scheduling decisions and reports statistics about the simulated run.

Output: replay statistics (throughput, latency). Requires: perf sched record.

Dump raw per-sample events from perf.data. Each line is one sample with configurable fields.

Use this when you need the raw data rather than aggregated histograms — for flamegraph input, custom filtering, or inspecting individual events.

Key parameters:

• fields: comma-separated output fields. Common sets: 'comm,pid,tid,time,event,ip,sym,dso' (general), 'ip,sym,dso' (flamegraph input), 'comm,tid,time,ip,sym,srcline' (source mapping). Available: comm,tid,pid,time,cpu,event,trace,ip,sym,dso,addr,symoff,srcline,period,flags,callindent,insn,brstacksym.

• symbols/dsos/comms/pid/tid: filter to specific functions/DSOs/processes.

• time: restrict to time range 'start,stop'.

• max_events: limit number of events returned.

• header_only: show file metadata only.

• show_task_events/show_mmap_events/show_switch_events: include non-sample events.

• call_trace/call_ret_trace/insn_trace: Intel PT trace modes.

Output: one line per sample. Format depends on fields parameter. Works on any perf.data from perf record.

Generate a timechart SVG showing CPU activity and task scheduling over time as a visual timeline.

Key parameters:

• output: SVG file path (default: output.svg).

• process: filter to specific process name(s).

• power_only: show only CPU power state changes.

• tasks_only: show only task scheduling.

• io_only: show only I/O activity.

• topology: include CPU topology.

• width: SVG width in pixels.

Output: returns the SVG file path and size. Requires: perf timechart record.