Skip to main content
Glama
agent-network-access-probe

agent-network-access-probe

README.md
# Probe

<!-- <a href="https://github.com/trustabl/agent-network-access-probe/releases"><img src="https://img.shields.io/github/downloads/trustabl/agent-network-access-probe/total?label=downloads&color=2ea043" alt="Total downloads"></a> -->

> Public product: **Trustabl Probe**

Runs your agent tool in an isolated OpenShell sandbox, observes the network destinations it actually reaches, and generates a least-privilege OpenShell `network_policies` draft from that evidence — no hand-written allow list required. Static analysis also flags hardcoded credentials, missing timeouts, and other reliability issues, with AI-generated fixes for each, reviewed one by one. Your source code never leaves your environment.

> **Language support:** Static analysis is supported for Python, Go, JavaScript/TypeScript, Ruby, and Shell. Sandbox execution is available for all languages.

---

<a id="trustabl-partners"></a>
<h2 align="center">Trustabl partners — install from the catalogue you already use</h2>
<p align="center">
  <sub>One network-access probe, four front doors.</sub>
</p>

<table align="center">
  <tr>
    <td align="center" width="300">
      <a href="https://registry.modelcontextprotocol.io/?q=agent-network-access-probe"><b>MCP Registry</b></a> &middot;
      <a href="https://www.npmjs.com/package/@trustabl/agent-network-access-probe"><b>npm</b></a><br>
      <sub>Scan from any<br>MCP-aware client</sub>
    </td>
    <td align="center" width="300">
      <a href="https://github.com/trustabl/agent-network-access-probe/releases/latest"><b>Claude Desktop</b></a><br>
      <sub>One-click install from<br>the .mcpb bundle</sub>
    </td>
  </tr>
  <tr>
    <td align="center">
      <a href="mcp/README.md#install"><b>Claude Code</b></a> &middot;
      <a href="https://cursor.directory/plugins/agent-network-access-probe"><b>Cursor</b></a><br>
      <sub>Your coding agent checks,<br>installs and runs Probe</sub>
    </td>
    <td align="center">
      <a href="#3-install-autofix"><b>CLI</b></a><br>
      <sub>Homebrew, Scoop<br>or a binary</sub>
    </td>
  </tr>
</table>

<p align="center">
  <sub>
    Every one of these installs the same Probe and runs it on your own machine.<br>
    Scanning agent code instead? <a href="https://github.com/trustabl/agent-reliability-analyzer#trustabl-partners">Agent Reliability Analyzer</a> has ten more front doors.
  </sub>
</p>

---

<p align="center">
  <a href="https://www.star-history.com/#trustabl/agent-network-access-probe&Date">
    <img src="https://api.star-history.com/svg?repos=trustabl/agent-network-access-probe&type=Date" alt="GitHub stars over time for trustabl/agent-network-access-probe" width="800">
  </a>
</p>

## How It Works

1. Your tool is uploaded to a sandboxed OpenShell environment and executed
2. Runtime evidence is collected — network destinations reached, denials, crashes, warnings, policy violations
3. Static analysis scans your source for hardcoded credentials, unauthorized network calls, missing timeouts, and more
4. You get back a least-privilege OpenShell policy YAML (unsigned draft) built from what the tool actually reached, findings from both passes, and a fixed version of the code for each issue found

Each session can also be attested: an optional signed manifest of the run, using [Sigstore](https://www.sigstore.dev/) keyless signing — no private key to manage. In CI (e.g. GitHub Actions with `permissions: id-token: write`), it signs with your workflow's identity automatically. Run locally without that, and it falls back to an interactive browser login bound to your personal identity, recorded in Sigstore's public transparency log — pass `--no-sign` to skip signing entirely instead.

Cloud metadata endpoints and link-local addresses (`169.254.169.254`, `metadata.google.internal`, `169.254.0.0/16`, `fe80::/10`) are always excluded from the generated policy, regardless of what the tool actually reaches — pass `--deny-list FILE` to add your own hosts/IPs/CIDRs on top of that built-in set.

By default, everything a tool observably reaches (and isn't denied) goes into the draft. Pass `--allow-list FILE` to make that stricter: once supplied, an observed destination not on the list is flagged instead of auto-added — the allow list becomes the required source of truth, not just an addition to what was seen. `--platform-allow-list FILE` works the same way for platform must-reach endpoints (e.g. NIM, IdP) and tags matching entries `source=platform` in `probe.profile.json`.

If a tool's code contains a call shape with a real-world side effect — a network write, a subprocess call, a filesystem delete, destructive SQL — its sandbox execution is skipped by default (static analysis still runs and still reports on it) so a default run can never actually charge, delete, or modify anything. Pass `--allow-write-tools` to run it for real once you're ready, ideally alongside `--env-fixtures FILE` so the tool picks up staging/test-mode credentials instead of production ones.

Sandbox results are cached per tool under `os.UserCacheDir()/trustabl-probe/results` — keyed by the tool's source content and effective run config (base image, `--sandbox-args`, `--env-fixtures` content), not its filesystem path, so the same tool referenced by two different agent classes reuses one cache entry. `probe.profile.json`'s `cache_hit` field reflects whether each tool's result came from the cache. Pass `--no-cache` to skip the cache entirely (neither read nor write) — always do this for release gates.

## Requirements

- A sandbox runner for `--sandbox` execution — pick one with `--runner`:
  - `docker` (default-friendly, no separate install beyond Docker itself) — [Docker Desktop or Docker Engine](https://docs.docker.com/get-docker/), daemon running. No OpenShell required.
  - `openshell` (default `--runner`) — [OpenShell](https://build.nvidia.com/openshell) installed, with `openshell gateway start` running. Also requires Python 3.8+ on the host machine, for dependency wheel downloads.
- Access to an OpenAI-compatible LLM endpoint (OpenAI, Anthropic, Ollama, LiteLLM, LM Studio, etc.), only for generated fixes. Without `--model`, Probe still writes the policy, profile, findings and attestation.

## Setup

### 1. Install a sandbox runner

**Docker only (no OpenShell needed):** install [Docker Desktop or Docker Engine](https://docs.docker.com/get-docker/) and make sure the daemon is running (`docker info`). Pass `--runner docker` to `fix`/`watch` — that's the whole setup.

**OpenShell:**

Download from [build.nvidia.com/openshell](https://build.nvidia.com/openshell), then:

```bash
openshell gateway start   # keep running in a separate terminal
openshell sandbox list    # verify CLI is on PATH
```

### 2. Configure an LLM (optional, only for generated fixes)

**Local (Ollama):**
```bash
brew install ollama       # macOS
ollama serve
ollama pull qwen2.5-coder:32b
```

**Cloud:** set `OPENAI_API_KEY` or `ANTHROPIC_API_KEY` — the endpoint is auto-detected from the model name.

### 3. Install autofix

**macOS (Homebrew):**
```bash
brew tap trustabl/autofix
brew install autofix
```

**Windows (Scoop):**
```powershell
scoop bucket add trustabl https://github.com/trustabl/scoop-autofix
scoop install autofix
```

**Linux / manual:**
```bash
# Replace v0.1.0 with the latest version from https://github.com/trustabl/agent-network-access-probe/releases
curl -L https://github.com/trustabl/agent-network-access-probe/releases/download/v0.1.0/autofix_0.1.0_linux_amd64.tar.gz | tar xz
sudo mv autofix /usr/local/bin/
autofix version
```

### Or use it from your coding agent (MCP)

Probe also runs as an MCP server, so a coding agent (Cursor, Claude Code, Claude
Desktop) can check what Probe needs, install it, and scan a tool, then apply the
fix hints itself. No model key is needed. See [mcp/README.md](mcp/README.md).

---

## Commands

### `fix` — Generate a least-privilege policy from observed egress, and fix what's found

```bash
autofix fix --source ./my-tool
autofix fix --source ./my-tool --entrypoint tool.py --sandbox --model gpt-4o
autofix fix --source ./my-tool --existing-policy ./policy.yaml --model claude-3-5-sonnet-20241022
```

**Flags:**

| Flag | Default | Description |
|---|---|---|
| `--source` | required | Path to the tool directory |
| `--entrypoint` | auto-detected | Entry point script(s) — can repeat for multi-file scanning |
| `--model` | — | Model name. Cloud: `gpt-4o`, `claude-3-5-sonnet-20241022`. Local: `qwen2.5-coder:32b` (recommended). Omit it to skip fix generation: the policy, profile, findings and attestation are still written, and `--output json` prints `[]`. |
| `--llm-url` | auto-detected | OpenAI-compatible base URL. Auto-detected for `gpt-*` and `claude-*`. Required for all other models. |
| `--api-key` | env var | API key — defaults to `OPENAI_API_KEY` / `ANTHROPIC_API_KEY` / `LLM_API_KEY` |
| `--llm-timeout` | `5m` | Per-request LLM timeout (increase for slow local models) |
| `--sandbox` | off | Run the tool in a sandbox for runtime findings |
| `--runner` | `openshell` | Sandbox runner: `openshell` or `docker` |
| `--base-image` | auto | Sandbox base image (default: auto-detected from entrypoint extension) |
| `--sandbox-args` | — | Arguments passed to the tool inside the sandbox. Value is a raw shell token — avoid shell metacharacters. |
| `--existing-policy` | — | Path to an existing OpenShell policy YAML to diff against the generated least-privilege policy |
| `--deny-list` | — | Path to a deny list file (one host/IP/CIDR per line, `#` comments) — evaluated on top of the built-in metadata/link-local deny |
| `--allow-list` | — | Path to a declared allow list file (one host/CIDR/`*.suffix` per line, `#` comments) — an observed destination not on this list is flagged, not auto-added to the draft |
| `--platform-allow-list` | — | Path to a platform allow list file (jail must-reach endpoints, e.g. NIM/IdP) — same format as `--allow-list`, tagged `source=platform` in `probe.profile.json` |
| `--allow-write-tools` | off | Allow sandbox execution of tools that appear to perform mutating/irreversible operations (default: skipped, static analysis only) |
| `--env-fixtures` | — | Path to a dotenv-style file (`KEY=value` per line, `#` comments) injected into the sandbox environment — lets a tool pick up staging credentials/endpoints instead of production ones |
| `--no-cache` | off | Skip the per-tool result cache entirely — neither read nor write it (use for release gates that must always execute for real) |
| `--auto-accept` | off | Accept all fixes without interactive review (useful for CI) |
| `--no-sign` | off | Skip Sigstore signing — no network call, no interactive browser flow, no public transparency log entry |
| `--output` | `text` | Output format: `text` or `json` |

**Output files** (written to `--source` directory):

| File | Description |
|---|---|
| `network_policies.draft.yaml` | Least-privilege OpenShell network policy (unsigned draft, built from observed egress) |
| `probe.profile.json` | Per-tool egress table, grant-set hash, and scan metadata (unsigned) |
| `probe.findings.json` | Every finding (title, severity, evidence, fix hint) with file attribution — always written, independent of whether interactive fix review ran |
| `autofix-attestation.json` | Optional signed behavioral manifest with source hash |
| `autofix-session.json` | Full session log (findings, diffs, accept/reject decisions) |
| `autofix-session.patch` | Unified diff of all accepted fixes |

---

### `watch` — Continuous improvement loop

Polls source files for changes. On every save, re-runs static analysis, rewrites `network_policies.draft.yaml`, `probe.profile.json`, `probe.findings.json`, and `autofix-attestation.json`, and prints a delta (score change, resolved/new findings, new source hash).

```bash
autofix watch --source ./my-tool
autofix watch --source ./my-tool --entrypoint tool.py --interval 1s
```

**Flags:**

| Flag | Default | Description |
|---|---|---|
| `--source` | required | Path to the tool directory |
| `--entrypoint` | auto-detected | Entry point script(s) to watch |
| `--interval` | `2s` | Poll interval |
| `--no-sign` | off | Skip Sigstore signing — no network call, no interactive browser flow, no public transparency log entry |
| `--deny-list` | — | Path to a deny list file (one host/IP/CIDR per line, `#` comments) — evaluated on top of the built-in metadata/link-local deny |
| `--allow-list` | — | Path to a declared allow list file (one host/CIDR/`*.suffix` per line, `#` comments) — an observed destination not on this list is flagged, not auto-added to the draft |
| `--platform-allow-list` | — | Path to a platform allow list file (jail must-reach endpoints, e.g. NIM/IdP) — same format as `--allow-list`, tagged `source=platform` in `probe.profile.json` |

---

### `verify` — Verify an attestation and gate CI

Checks an attestation's content digest and Sigstore signature, then optionally fails (non-zero exit code) on a low score or critical findings. See [docs/ci-gate.md](docs/ci-gate.md) for a full CI setup.

```bash
autofix verify ./autofix-attestation.json --min-score 70 --fail-on-critical
autofix verify ./autofix-attestation.json --require-signature --identity "https://github.com/acme-corp/.*"
```

**Flags:**

| Flag | Default | Description |
|---|---|---|
| `--min-score` | `0` (no gate) | Fail if `production_readiness_score` is below this value |
| `--fail-on-critical` | off | Fail if any critical findings are present |
| `--require-signature` | off | Fail if no valid Sigstore signature is present |
| `--identity` | — | Regex the signer's certificate SAN must match |
| `--issuer` | — | Regex the signer's OIDC issuer must match |
| `--output` | `text` | Output format: `text` or `json` |

---

### `version` — Print version

```bash
autofix version
```

### `license` — Print license and copyright information

```bash
autofix license
```

---

## License

Apache License 2.0. See [LICENSE](LICENSE), or run `autofix license` to
read it in the terminal.