# toolfence

Security audit for MCP servers and agent tool-chains. Read-only: it never
executes the code it scans and never calls `tools/call`.

```bash
toolfence https://github.com/owner/repo     # scan a repo (extracts tools from source)
toolfence ./path/to/server                  # scan a local checkout
toolfence tools.json                        # scan a tools/list export
toolfence --endpoint https://host/mcp       # live server — calls tools/list only
```

Or in CI, where it runs on every pull request:

```yaml
- uses: dhb520cat/toolfence@v0
  with:
    fail-on: critical
```

Findings land in the job summary, and `findings` / `critical` / `tools` are
exposed as step outputs.

It is also an MCP server, so an agent can audit a server before trusting it:

```json
{ "mcpServers": { "toolfence": {
    "command": "python3", "args": ["-m", "toolfence.server"] } } }
```

Three tools — `scan_repository`, `scan_tool_manifest`, `explain_rule`. All three
are read-only, declare `readOnlyHint: true` and `destructiveHint: false`, take no
credentials, and **pass toolfence's own audit with zero findings**. A test asserts
that they keep doing so.

## Why

An agent with an inbox can be written to by anyone. An agent with a wallet can
move money. The tools in between decide how bad that gets, and right now almost
nobody audits them.

Every rule here came out of an actual audit, not a threat-modelling session:

| Rule | Where it came from |
|---|---|
| `DESTRUCTIVE_NO_CONFIRM` | Mermail gates destructive tools behind a single-use `confirmationToken`. Most servers gate nothing. |
| `ANNOTATION_MISMATCH` | MCP ships `readOnlyHint` / `destructiveHint`. A wrong hint is a *false* safety guarantee — callers auto-approve on it. |
| `INJECTION_SURFACE` | Tool descriptions land verbatim in the model's context. An imperative in a description is an instruction anyone who can edit it gets to give your agent. |
| `HONEYPOT` | 73.2% of agent-bounty repos hide instructions from humans (HTML comments) that tell automated systems to paste their system prompt into a public PR. |
| `PII_EXPOSURE` | Ported from a TEE contract's PII guard — the check that stops personal data reaching an enclave that shouldn't hold it. |
| `CREDENTIAL_IN_PARAM` | Tool arguments end up in conversation history, logs and telemetry. Secrets belong server-side. |
| `UNBOUNDED_SCOPE` | Arbitrary path / URL / command / SQL — traversal, SSRF, injection. |
| `COMMAND_INJECTION` | A tool argument reaching a shell. Declarations are design choices; this is a bug. |
| `PATH_TRAVERSAL` | A tool argument reaching a filesystem call without a resolved-root check. |
| `SSRF` | A tool argument reaching an HTTP client without a host allowlist. |

The last three read implementations rather than declarations, and only fire when
a model-controlled value reaches the sink with no validation nearby. Validation
in the neighbourhood downgrades the finding to `low` instead of silencing it, so
you can check whether the guard actually covers that path.

## What it found

A survey of 30 repositories drawn from `punkpeye/awesome-mcp-servers`
(3,877 entries, filtered to non-archived TS/JS/Python/Go projects by stars).
21 of them yielded extractable tools — **1,666 tools in total**.

```
critical   17    high   96    medium  135    low   20
```

7 of 21 repositories come back completely clean.

Every remaining critical is an irreversible delete against something that
matters, with no confirmation affordance in the tool itself:

```
awslabs/mcp                   delete_db_cluster, delete_db_instance
                              delete_fhir_resource        (health records)
                              delete_instance_in_study    (medical imaging)
                              mcp_delete_ecs_infrastructure
containers/kubernetes-mcp     resources_delete
cloudflare/mcp-server         container_file_delete
txn2/kubefwd                  remove_namespace, remove_service
```

Two `INJECTION_SURFACE` findings are real text shipped by github-mcp-server:
*"**always call this tool** when the user asks for details about…"*. Not a
vulnerability — a pattern worth naming, since a description that issues orders
goes verbatim into the model's context.

**This is a count of patterns, not a list of vulnerabilities.** Most of these
projects delegate confirmation to the client, which is where MCP's design puts
it. What the number says is that the tool layer itself carries no guardrail —
swap in a client that doesn't prompt, and the delete goes through.

## Precision over recall

A scanner that flags everything is a scanner nobody reads, and the cost of a
false positive is asymmetric: one wrong accusation against a careful team and the
tool is never trusted again.

This one was wrong seven times against real code. **I came within one commit of
publishing that AWS ships unguarded deletion tools.** `awslabs/mcp` returned
criticals on `delete_resource`, `delete_fhir_resource` and
`mcp_delete_ecs_infrastructure`. Reading their source first: `delete_resource`
takes a `confirmed` parameter, `create_resource` takes `security_scan_token` and
`skip_security_check`. Their design is careful. The scanner had never parsed
Python function signatures at all, so every Python tool looked unguarded — a
systematic false positive across an entire language, exposed only by a real
codebase.

Each of the seven is now a regression test:

- `list_emails` is not "send an email" — nouns that double as verbs only count in
  the verb slot.
- `get_create_fields` is a **read** — a read verb in the head position settles the
  whole name.
- `search_files` says *"Only searches within allowed directories"* — a declared
  boundary is not an unbounded scope.
- `"paste this in your configuration file"` in an install guide is not a honeypot.
  Only demands for the agent's **own** system prompt count.
- Test files, fixtures and examples are skipped entirely — `main_test.go` had a
  tool literally named `delete`.
- Python signatures are parsed for parameter names. Without them every Python
  tool looked unguarded; `awslabs/mcp`'s `delete_resource` actually takes a
  `confirmed` parameter, and the scanner was calling it unprotected.
- `credentials_token` described as coming *"from get_aws_session_info()"* is a
  handle, not a secret in transit.
- ZWNJ (U+200C) is Persian and Arabic orthography, not a hidden-instruction
  marker. Flagging it fired on every project shipping RTL translations.
- **Severity follows the object, not the verb.** `folder_remove_motion_blur`
  is not `delete_db_cluster`. Grading destructive tools by what they act on cut
  false criticals by 76% (72 → 17) across the survey.
- **It could not see itself.** This README once claimed the server passed its own
  audit with zero findings. It had not: the tools are declared as dict literals,
  which the Python extractor did not understand, so it extracted none of them.
  Fixed, it now finds five tools here and still reports nothing — and a test
  asserts that the server's own manifest stays clean.

Explicit annotations beat name inference: `create_directory` declares
`destructiveHint: false` and is believed.

## Limits

- **Lexical, not semantic.** It reads manifests and source text; it does not
  execute, type-check, or trace data flow.
- **TypeScript, JavaScript, Python and Go.** Other languages extract nothing —
  a clean report on a Rust server means the scanner found no tools, not that the
  server is clean. Tool count is printed so you can tell the difference.
- **`inputSchema: SomeZodSchema.shape`** hides parameter names; those tools are
  scanned on name and description alone.
- A clean report means *these seven rules did not fire*. It is not a safety
  certificate.

## Install

```bash
pip install git+https://github.com/dhb520cat/toolfence
toolfence --help
```

No dependencies outside the standard library. Python 3.10+.

`--json` for machine output. `--fail-on {critical,high,medium,low,never}` sets the
exit code, so it drops into CI as-is.

## Tests

```bash
python3 test_toolfence.py      # 80 assertions, 30 true negatives, plus a self-audit
```

MIT.