# PerfLens

> Evidence-driven Linux performance analysis with a CLI, MCP Server, and project Skills for Codex, Claude Code, OpenCode, and local GitHub Copilot clients.
> 基于证据的 Linux 性能分析工具，支持 Codex、Claude Code、OpenCode 与本地 GitHub Copilot 客户端。

[![CI](https://github.com/link0-o/PerfLens/actions/workflows/ci.yml/badge.svg)](https://github.com/link0-o/PerfLens/actions/workflows/ci.yml)
[![Python 3.12+](https://img.shields.io/badge/Python-3.12%2B-blue)](pyproject.toml)
[![License: Apache-2.0](https://img.shields.io/badge/License-Apache--2.0-green)](LICENSE)

[简体中文](README.zh-CN.md) | English

**First installation: read [Installation and first use](INSTALL.md). Do not extract the wheel; install it with pipx or uv.**

After DEB installation, `sudo perflens-admin setup` selects the host Collector mode.
`perflens init` then detects that safely deployed mode per project. See the
[Collector privilege-mode lifecycle](docs/collector-mode-lifecycle.md) for dry-runs,
switching, rollback, and project resynchronization.
See the [Collector and user-space-lock roadmap](docs/collector-capability-roadmap.md) for the
current maturity boundary and phased extension plan. Release v0.3.1 local-Docker single-process
support is specified separately in the [Docker process guide](docs/docker-container-roadmap.md).
The v0.3.2 bounded build-and-optimize contract is tracked in the
[Docker optimization guide](docs/docker-optimization-roadmap.md).
See the [trustworthy perf evidence pipeline](docs/evidence-pipeline.md) for raw-input binding,
conversion provenance, quality gates, and Agent-facing verification.

PerfLens is an evidence-driven performance-analysis toolkit for Linux
applications and coding agents.

Release v0.3.2 supports Milestones 0 through 9:

- streaming FlameGraph-compatible folded stack input;
- deterministic self and inclusive hotspot aggregation;
- root-to-leaf call-path aggregation;
- symbol plus DSO grouping (DSO is explicitly `unknown` for standard folded input);
- bounded parse diagnostics and versioned JSON artifacts;
- a production CLI with path checks, stable error output, resource limits, and
  crash-durable file-and-directory-synced atomic writes;
- streaming parsing of explicitly-fielded `perf script` text;
- `perf.data` conversion through an allowlisted system `perf` process;
- bounded subprocess output, stderr diagnostics, timeouts, and process-group cleanup.
- ELF Build ID/debug capability inspection and verified module-offset symbolization;
- bounded workspace source context and container/build path mapping;
- generic candidate-only classification, evidence bundles, and Markdown reports.
- an official-SDK MCP server with typed, paginated tools and server-side authorization;
- a repository Performance Analysis Skill for evidence-constrained Agent workflows.
- profile and repeated-benchmark comparison with environment comparability checks;
- pyperf, Google Benchmark, and hyperfine JSON normalization;
- default-off, explicitly authorized `stat`/`record` collection. Release v0.3.0 also provides a
  separate target-filtered Trace Helper and verified `sched`/`off_cpu`/`lock` pipeline through
  the opt-in `full_diagnostics` feature profile.
- v0.3.1 project-scoped discovery and authorized collection for one process in a local Linux
  Docker Engine, including cgroup v2 context, bounded container symbol mapping, managed temporary
  test containers, and evidence-matched A/B comparison.
- v0.3.2 opt-in `bounded_optimization_session`: one reviewed confirmation can cover a fixed
  baseline build, evidence-guided collection, build snapshots whose changes are restricted to
  `mutable_paths`, up to three candidate rebuilds, and deterministic matched A/B validation under
  hard budgets. A non-verified final candidate is retained only after a fresh human disposition
  choice; that choice is recorded without upgrading the evidence verdict. The Agent/client
  sandbox, not PerfLens, enforces filesystem write permission.

It does **not** include an AI/LLM API, Web UI, source-code patch tool, general-purpose benchmark
runner, or custom agent framework. In an authorized optimization session, the external Agent edits
only reviewed mutable paths and the fixed workload contract supplies correctness/Benchmark output.

## Install

PerfLens requires Python 3.12 or newer.

For a GitHub release, download the wheel and install it as an isolated tool:

```bash
pipx install ./perflens-0.3.2-py3-none-any.whl
# or
uv tool install ./perflens-0.3.2-py3-none-any.whl
```

Then opt one project in. Other projects do not see the Skill or MCP server:

```bash
cd /absolute/path/to/project
perflens init
```

For a project that intentionally profiles a local Docker workload, initialize the project with:

```bash
perflens init --docker
```

This writes a project-owned `perflens-setup/container-workload.toml` policy. It does not install
or start Docker, join the Docker group, build/pull an image, deploy a Collector, or grant execution.
The Skill uses typed MCP discovery, authorization, collection, and comparison tools only after the
user confirms either one run or a bounded in-memory session.

This activates Codex and Claude Code by default. Select one of them with
`--client codex` or `--client claude-code`; use `--client opencode` for OpenCode,
or `--client copilot` for both Copilot CLI and VS Code Copilot Agent. Use `--read-only` when the
project should analyze existing evidence without automatic collection.
Repeat `--client` to select several clients for one project. To make that selection the default
for future plain `init` calls, run, for example,
`perflens client-defaults --client codex --client claude-code --client copilot`.
This writes strict `~/.config/perflens/config.toml`; if it is absent, the built-in default remains
Codex plus Claude Code. Explicit `init --client ...` values override it for one invocation.
Claude Code and Copilot CLI share `.mcp.json`; onboarding validates one identical ownership copy
and updates the shared entry atomically when both are selected.
Rerun with `perflens init --update` when upgrading managed integration or
changing collection gates. Update mode requires a matching `setup.json`, updates
only recorded client entries and marked blocks, and refuses
to overwrite a modified Skill or unverified client configuration. The managed
`perflens-setup` directory is rebuilt and must not contain user files; unexpected
entries cause refusal, while staged Collector assets are preserved unless
regeneration is explicitly requested. With no explicit `--client`, update preserves the current
project's recorded client set. Detach a no-longer-needed client before
updating with a narrower `--client` selection.

Follow the generated `NEXT_STEPS.md`. See [Installation and first use](INSTALL.md) for the complete beginner flow.
Onboarding safely selects the native `/usr/bin` or wheel `/opt/perflens` layout,
so copy the exact generated deployment command instead of guessing paths.
It installs only project-scoped integration: Codex uses `.codex/config.toml`, Claude Code
uses `.mcp.json` and `.claude/skills`, OpenCode uses `.opencode/opencode.json`, and the
local Copilot suite uses `.mcp.json` plus `.vscode/mcp.json`. Codex, OpenCode, and Copilot
share `.agents/skills`.
Existing unrelated configuration is preserved; user-level global configuration
is not modified.
Before package uninstall, preview `perflens detach --project <project>
--dry-run`, then repeat without `--dry-run`. By default it removes verified
selected-client MCP entries and unchanged project Skills while preserving
onboarding, analysis evidence, and system Collector data. Use `--keep-skills`
to detach MCP only or `--client` to select one client.

Debian 13 users can instead install the native, offline `.deb` packages. See
[Debian packages](docs/debian-packages.md) for the split main/Collector flow.
Version 0.2.0 also adds explicit `paranoid3_helper`: an unprivileged Python Broker passes typed PID
plans to a bounded root Rust Helper while `perf_event_paranoid=3` remains unchanged. It is never
enabled automatically and requires administrator acknowledgement of the bounded root,
`CAP_SYS_ADMIN`, and `CAP_SYS_PTRACE` risk.

Release v0.3.0 adds a separate Trace Helper, in-kernel target filtering, deterministic
sched/off-CPU/lock analysis, and the `full_diagnostics` lifecycle. The existing privileged
stat/record Helper remains limited to stat/record. See the
[Collector and user-space-lock roadmap](docs/collector-capability-roadmap.md). Release v0.3.1
adds the local-Docker single-process target runtime while the four runtime-lock adapters remain
planned for v0.4.0. Docker support is limited to a local Linux Engine, cgroup v2, an explicit
process, and fixed project policy; it excludes arbitrary Docker arguments, remote engines,
Compose/Kubernetes, image build/pull, and whole-container perf aggregation. Checked-in Runtime
Lock public contracts are groundwork, not available adapters. See the
[v0.3.1 Docker process guide](docs/docker-container-roadmap.md).
Release v0.3.2 adds the separately opt-in, benchmark-required bounded Docker optimization session.
It does not build during preview, does not grant arbitrary Docker access, and never authorizes
commit, push, tags, or releases. See the
[v0.3.2 Docker optimization guide](docs/docker-optimization-roadmap.md).

Run a read-only readiness summary at any time:

```bash
perflens status --project /absolute/path/to/project
```

Human-facing command help is Chinese-first as well. Run `perflens --help`,
`perflens setup --help`, or `perflens-admin --help` when needed. Stable English
command and option names are unchanged, and subcommand help documents duration,
resource, archive-selection, and authorization boundaries.

When automatic collection is configured and local access is available, this
also performs a bounded, read-only health handshake. It verifies the Collector
PID/UID with kernel peer credentials and reports stale, unreachable, or
wrong-identity sockets instead of declaring them ready.

Domain failures are Chinese-first for people. Automation should use the global
`perflens --json-errors <command> ...` option or set
`PERFLENS_JSON_ERRORS=1` to preserve the versioned JSON error artifact.
`perflens doctor` follows the same human-first principle: add `--json` for its
versioned capability artifact or `--output <new-file.json>` to save it safely.
Its five-mode output diagnoses local permission prerequisites; it does not prove that the
independent Collector or any requested host mode passed real collection acceptance.

After administrator deployment and a fresh login, verify the Collector without
finding a PID:

```bash
perflens accept-collector --authorize-host-acceptance
```

The default output is a concise Chinese pass summary with hardware-PMU,
software-counting, and `cpu-clock` sampling status plus the evidence path,
hash, metric count, and conclusion boundary. If hardware PMU evidence is not
useful, automatic collection stays within the same PID, duration, and output
bounds and continues with fixed software events. The result explicitly rules
out IPC, hardware cache-miss, and branch-miss claims. Use `--json` for complete
machine-readable output or `--output ./collector-acceptance.json` to preserve a
new versioned evidence file.

With the v0.3.0 `full_diagnostics` profile, the same command also requires substantive
target evidence, deterministic analysis, and replay verification for sched, off-CPU, and lock.
Empty evidence or failed conservation does not pass. The packaged fixed eBPF filters authorized
TGID/TIDs in kernel and never falls back to `perf -a`.

The wheel installation commands provide `perflens`, `perflens-mcp`, the optional
`perflens-collector`, and the explicit administrator entry point
`perflens-admin`. Confirm the release:

```bash
perflens --version
perflens-mcp --version
perflens-collector --version
perflens-admin --version
```

Installing directly from a source checkout is also supported:

```bash
python -m pip install .
```

For development with uv:

```bash
uv sync --all-groups
```

## Analyze folded stacks

```bash
perflens analyze-folded \
  --input tests/fixtures/folded/normal.folded \
  --output build/analysis.json
```

Input follows standard folded syntax:

```text
main;worker;parse;malloc 182
main;worker;compute 271
```

Frames are normalized to `root → leaf`. The final frame receives self weight.
Every unique `(symbol, DSO)` in a sample receives inclusive weight once, so
recursive frames cannot make a function-level inclusive percentage exceed
100%. Frame occurrences are counted separately.

Standard folded text has no DSO, PID/TID, CPU, timestamp, event, or source
metadata. PerfLens records these fields as `unknown`; it never infers them from
symbol names. Each folded line is one weighted stack record, not `weight`
individual samples.

## Analyze perf profiles

For existing text, generate the supported stable field set and analyze it:

```bash
perf script --ns \
  -F comm,pid,tid,cpu,time,event,period,ip,sym,dso,srcline \
  -i perf.data > profile.perf-script

perflens analyze-perf-script \
  --input profile.perf-script \
  --output build/analysis.json
```

Or let PerfLens run the same read-only conversion:

```bash
perflens analyze-perf-data \
  --input perf.data \
  --output build/analysis.json
```

`analyze-perf-data` never records, attaches to a process, or requests root. It
invokes an absolute, allowlisted `perf` executable without a shell. Use
`--perf-path` when several versions are installed and `--timeout-seconds` to
lower the conversion deadline.

## Inspect symbols and build evidence

```bash
perflens inspect-elf --input build/app --output build/elf.json
perflens resolve-source \
  --binary build/app \
  --module-offset 0x1234 \
  --output build/source.json

perflens classify \
  --analysis build/analysis.json \
  --output build/diagnosis.json
perflens report \
  --analysis build/analysis.json \
  --problem "Throughput regression" \
  --metric "requests/second" \
  --output build/report.md
```

Source resolution requires a verified module-relative offset. A runtime IP by
itself is never rebased heuristically. PerfLens prefers a long-lived
`llvm-symbolizer` JSON provider, then falls back to a long-lived `addr2line`
provider. Cache identity includes Build ID, module offset, and resolver version.

Classification rules label investigation candidates only. Generated reports
keep direct observations, missing evidence, forbidden conclusions, and A/B
validation requirements separate.

## Compare profiles and benchmarks

```bash
perflens compare-profiles \
  --baseline build/baseline-analysis.json \
  --candidate build/candidate-analysis.json \
  --output build/profile-comparison.json \
  --markdown-output build/profile-comparison.md

perflens normalize-benchmark \
  --input benchmark-hyperfine.json \
  --output build/benchmark.json
perflens compare-benchmarks \
  --baseline build/baseline-benchmark.json \
  --candidate build/candidate-benchmark.json \
  --output build/benchmark-comparison.json
```

Profile percentage changes describe the selected event distribution, not
absolute elapsed time. Benchmark comparisons require repeated samples, check
environment differences, apply a practical-impact threshold, and emit only
candidate improvement/regression states.

## Explicitly authorized active collection

Active collection is disabled by default. A CLI invocation requires both a
confirmation switch and the exact per-call authorization phrase:

```bash
perflens collect-profile \
  --mode record \
  --executable /absolute/path/to/app \
  --target-arg=--workload \
  --data-output build/profile.data \
  --metadata-output build/collection.json \
  --authorize-target \
  --authorization I_EXPLICITLY_AUTHORIZE_TARGET_PROFILING
```

Release 0.3.0 supports `record` and `stat`; `stat` uses an independent typed
metric adapter and derives IPC when cycles and instructions are available.
The v0.3.0 `full_diagnostics` profile adds `sched`, `off_cpu`, and `lock` through a separate
Trace Helper and produces dedicated verified artifacts. The existing paranoid=3 Rust
Helper remains strictly stat/record-only. Package installation never enables advanced modes;
an administrator selects the profile and runs real acceptance. PID attachment requires `--pid`, a bounded duration,
`--authorize-pid-attach`, and the separate phrase
`I_EXPLICITLY_AUTHORIZE_PID_ATTACH`. PerfLens never invokes sudo or changes
kernel policy. See [MCP server and Skill setup](docs/mcp-and-skill.md) for the
additional MCP startup gates.

For an approved live PID, PerfLens can automatically inspect permissions, create a
short-lived PID-bound plan, execute it once through a separately policy-enforcing
Collector Broker, and analyze the result. See
[automatic collection](docs/automatic-collection.md). The MCP server and Agent remain
unprivileged. The Collector also enforces cumulative spool byte/file quotas and
a filesystem free-space reserve; exhaustion denies new work without deleting
old evidence.

Each Collector instance permits exactly one ordinary UID. Sharing its
`perflens` group and spool across callers would expose group-readable profiles
between users and is rejected.

After deployment, `perflens-admin spool-status` gives a read-only Chinese
summary of spool usage, filesystem reserve, and currently reservable output;
add `--json` for the versioned machine-readable artifact.

Evidence is never age-deleted automatically. Administrators can use the
archive-then-prune workflow to create a bounded stored ZIP with a versioned
manifest and per-file SHA-256, preserve all sources, verify both copies with a
dry run, and only then explicitly authorize removal of exact matching source
inodes. It selects the Broker or private Rust Helper spool from the deployed
privilege mode and binds that mode and path into the manifest. The archive
remains intact and Agents must not schedule pruning.

Administrators can tune the bilingual policy without memorizing a manual
restart sequence: copy it to a separate mode-`0600` candidate, run
`perflens-admin update-policy --config <candidate> --dry-run`, then repeat with
`sudo`. The command atomically applies and health-checks the policy, rolls back
on activation failure, and refuses UID, fixed-spool, or privilege-mode migration.
Switching between `cap_perfmon` and `paranoid3_helper` requires a reviewed
undeploy and redeploy because the managed service topology is different.

Deploy and upgrade require a bounded, read-only Collector health round trip and
verify the responding PID/UID through kernel credentials. A stale, wrong-owner,
or unlistened socket pathname is not readiness. `perflens-admin deploy` prints a
Chinese dry-run or success summary by default; add `--json` for the complete
versioned artifact.

After installing a new release, run `sudo perflens-admin upgrade --dry-run` and
then `sudo perflens-admin upgrade`. The explicit flow preserves policy and spool
data, replaces only a verified managed unit, restarts the service, and attempts
unit rollback on activation failure.

After one administrator-reviewed deployment, users do not need to discover a
PID. They may ask the Skill to optimize the current project, approve one exact
executable and argument list, and let the ordinary-user launcher obtain the new
PID internally. The Collector still receives only a short-lived PID-bound plan;
the project workload never runs with Collector privilege.

See [product deployment](docs/deployment.md) for configurable service assets,
real Collector verification, upgrades, and uninstall behavior.

## Use MCP with the Skill

The normal path is `perflens init`, which activates only the selected project.
For separate, advanced steps, install the bundled Skill for a specific client:

```bash
perflens install-skill --project /absolute/path/to/workspace
perflens install-skill --client claude-code --project /absolute/path/to/workspace
perflens install-skill --client opencode --project /absolute/path/to/workspace
perflens install-skill --client copilot --project /absolute/path/to/workspace
```

The command creates `.claude/skills/perflens` for Claude Code or the shared
`.agents/skills/perflens` for the other clients, and refuses to overwrite an existing Skill.
To print a project-scoped MCP configuration:

```bash
perflens codex-config --workspace /absolute/path/to/workspace
perflens claude-config --workspace /absolute/path/to/workspace
```

Add `--allow-process-execution` only when `perf.data` conversion or source
symbolization is required. Review the printed TOML before adding it to the
project's `.codex/config.toml`.

From a source checkout, the equivalent direct registration is:

```bash
mkdir -p perflens-results
codex mcp add perflens -- \
  "$PWD/.venv/bin/perflens-mcp" \
  --allowed-root "$PWD" \
  --artifact-root "$PWD/perflens-results" \
  --allow-writes
```

Restart Codex, then ask:

```text
$perflens analyze ./profile.folded and report direct evidence, candidates, and missing evidence.
```

For Claude Code, `perflens init` installs the project Skill under
`.claude/skills/perflens` and safely merges `perflens`
into the project `.mcp.json`. Claude Code asks the user to trust a project MCP
server before first use. Invoke it with `/perflens`.

See [MCP server and Skill setup](docs/mcp-and-skill.md) for permissions,
project-scoped configuration, process-execution opt-in, and the full tool flow.

## Resource limits

Defaults are intentionally explicit:

- input file: 1 GiB;
- logical records: 10 million;
- line length: 1 MiB;
- stack depth: 4,096;
- unique frames: 2 million;
- unique call paths: 1 million;
- retained warnings: 100;
- emitted hotspots: 10,000;
- emitted call paths: 1,000.

Limits can be lowered from the CLI. Exceeding structural limits fails with a
structured error rather than silently dropping exact data. Malformed individual
records are skipped and reported with bounded line previews.

## Exit codes

| Code | Meaning |
|---:|---|
| 0 | success |
| 2 | invalid CLI usage or input |
| 3 | unsupported or malformed profile |
| 4 | resource limit exceeded |
| 5 | output/path safety failure |
| 6 | external tool failure or timeout |
| 70 | unexpected internal failure |

## Development checks

```bash
uv run ruff check .
uv run pyright
uv run pytest --cov=perflens
uv build
uv run pip-audit
```

The reproducible performance harness is:

```bash
uv run python tests/performance/benchmark_folded.py \
  --records 1000 100000 1000000 \
  --repetitions 3
```

See `docs/performance-budget.md` for the recorded environment and baseline.

See [release readiness](docs/release-readiness.md),
[release process](docs/releasing.md),
[real-world profile acceptance](docs/real-world-acceptance.md), and
[known issues](docs/known-issues.md), and
[troubleshooting](docs/troubleshooting.md), the
[v0.3.2 Docker optimization regression playbook](docs/v0.3.2-regression-playbook.md), and the
[Collector and user-space-lock roadmap](docs/collector-capability-roadmap.md), the
[v0.3.1 Docker process roadmap](docs/docker-container-roadmap.md), plus the
[perf evidence pipeline](docs/evidence-pipeline.md) for final verification evidence,
published-version workarounds, and operational failure guidance.

Chinese maintainer documentation is available in the
[development guide](docs/development.zh-CN.md),
[architecture guide](docs/architecture.zh-CN.md),
[compatibility matrix](docs/compatibility.zh-CN.md),
[known limitations](docs/limitations.zh-CN.md),
[real-world acceptance record](docs/real-world-acceptance.zh-CN.md),
[security policy](SECURITY.zh-CN.md), and
[release-readiness record](docs/release-readiness.zh-CN.md). Every document under
`docs/` that has an English version now links to a corresponding Simplified
Chinese version.

## Known limitations

- Folded input cannot distinguish identically named functions from different
  DSOs because the format omits DSO metadata.
- Percentages describe selected event weight, not wall-clock duration.
- Call paths are exact up to the configured unique-path limit.
- Symbol names are preserved with only conservative compiler-suffix cleanup.
- A hotspot is an observation, not a confirmed root cause.
- `perf.data` portability remains dependent on the installed `perf` version and
  access to matching DSOs/symbols; preserved unknown frames make gaps explicit.
- Active collection depends on kernel perf permissions. An ordinary process is
  blocked by `perf_event_paranoid=3`; an installed Collector may still pass its
  separate host acceptance, including an explicitly reported software fallback.
- Release v0.3.0 produces scheduler-delay, off-CPU interval, and low-level
  lock/futex-candidate artifacts through the opt-in `full_diagnostics` profile.
  It cannot promote a futex candidate to a language lock, invent owner/hold time without pairs, or
  cover every user-space fast path.
- Release v0.3.1 Docker collection is limited to one explicit process in a local Linux Engine with
  cgroup v2 and explicit project/session authorization. It excludes remote Engines, Docker Desktop
  VMs, Compose/Kubernetes, image build/pull, arbitrary Docker arguments, and whole-container perf
  aggregation; container-wide cgroup deltas are context rather than process-exclusive evidence.