agent-firewall

A prompt-injection firewall for AI agents. It sits between an agent (Claude Code, Claude Desktop, any MCP client) and its tools (MCP servers), and:

• scans every tool output (web pages, emails, files) for injected instructions, including hidden HTML, invisible Unicode, base64, leetspeak and 7 languages, and strips them before the model sees them

• tracks session taint: once the agent has read untrusted or malicious content, risky actions (sending email, HTTP posts, deleting files) are blocked or sent to a human for approval

• blocks steered and exfiltrating calls: destinations that were introduced by injected content, and credentials headed to non-allowlisted hosts (DLP)

• quarantines poisoned tools: MCP tool descriptions that carry hidden instructions, and tools whose definitions change after first use ("rug pulls")

• logs every decision to SQLite, with a live approval dashboard

Live demo: agent-firewall-lwzj.onrender.com. Pick an attack (poisoned web page, phishing email, invisible Unicode) and watch the real detector and policy engine stop it. It's on a free plan, so the first visit after a quiet spell takes about a minute to wake up.

How it works, in detail: docs/HOW_IT_WORKS.md walks through one request end to end, explains every detector signal and the scoring, the policy rules, and includes playground exercises with expected results.

 agent (Claude Code / Desktop / any MCP client)
   │ MCP
   ▼
┌──────────────────────── agent-firewall ────────────────────────┐
│ tools/list  → scan descriptions, pin hashes → quarantine        │
│ tools/call  → policy: risk × session taint → allow/approve/block│
│                 + tainted-destination check + DLP               │
│ result      → detector → sanitize + raise taint                 │
│ everything  → SQLite audit log ──► dashboard (approve / deny)   │
└────────────────────────────────────────────────────────────────┘
   │ MCP
   ▼
 upstream MCP servers (filesystem, email, browser, ...)

Results

Demo (python demo/run_demo.py): a scripted agent that obeys every injected instruction it reads, run against a workspace with poisoned web pages and emails.

Evaluation (agent-firewall eval): 70 attacks across 12 categories plus 54 benign hard negatives (security articles quoting attacks, normal "please send X to Y" emails, docs full of rm -rf, and so on). The "agent" is fully hijacked, so attack success with no firewall is 100% by construction.

What the held-out set shows: the heuristic detector was overfit to the cases it was tuned on, and its recall drops from 96% to 36% on new phrasings (paraphrased overrides, Portuguese/Russian/Korean, <<SYS>> tokens, text-speak). The attacks still don't succeed, because the information-flow policy doesn't depend on recognising the attack. Once the session has read untrusted content, egress and destructive actions need a human. Detection decides whether an attack is auto-blocked or goes to a human; the policy is what keeps attack success at zero. The cost is approval friction, which is the main thing to improve next (see the roadmap).

The held-out set was written after tuning. One real regex bug it exposed (CSS opacity:0 hidden text) was fixed; no rules were tuned on it. Full reports: reports/eval_report.md, reports/holdout_report.md.

Related MCP server: MCP Security Gateway

Quick start

python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

python demo/run_demo.py          # hijacked agent, with vs without firewall
pytest                           # 33 tests
agent-firewall playground        # web playground -> http://127.0.0.1:8000
agent-firewall eval              # tuning-set report  -> reports/eval_report.md
agent-firewall eval --dataset evals/holdout.jsonl --out reports/holdout_report.md
agent-firewall scan suspicious_page.html --sanitize

Use it with a real agent (Claude Code)

The firewall is itself an MCP server. Point your agent at it, and list the real servers under upstreams in firewall.yaml:

upstreams:
  - name: files
    command: npx
    args: ["-y", "@modelcontextprotocol/server-filesystem", "/Users/you/Documents"]
  - name: fetch
    command: uvx
    args: ["mcp-server-fetch"]

# terminal 1: the approval dashboard
agent-firewall dashboard            # http://127.0.0.1:8787

# register the firewall with Claude Code
claude mcp add agent-firewall -- "$(pwd)/.venv/bin/agent-firewall" run --config "$(pwd)/firewall.yaml"

For Claude Desktop, add the same command to claude_desktop_config.json under mcpServers. To see it in action, try the demo workspace (the default firewall.yaml). Ask Claude to "summarize https://recipes.example/lasagna" or "check my inbox", then watch the dashboard.

Playground

agent_firewall/playground.py is a small, stateless web app for showing the firewall to people who won't install it. Pick a scenario (poisoned recipe page, phishing email, invisible-Unicode newsletter, white-on-white travel page, or a benign one) or paste your own content, then choose the call a hijacked agent would make next. The page shows the detector's signals, what the model sees after sanitising, and the policy decision. It never runs a real tool and keeps no state between requests, so it's safe to host publicly.

To host your own copy for free: (or create a Blueprint from render.yaml). The free plan sleeps after 15 idle minutes, so the first visit after that takes about a minute.

How it works

Short version below. The full guide is docs/HOW_IT_WORKS.md.

Detector (agent_firewall/detector.py). Each tool output is scanned in layers. Text is first normalised (NFKC, zero-width characters removed). Hidden channels are then pulled out: HTML comments, CSS-hidden elements, alt/meta text, Unicode tag characters (ASCII smuggling) and base64 blobs. Every channel is checked against ~20 weighted signals (instruction override, role/delimiter spoofing, addressing the AI, secrecy, authority claims, tool steering, exfil phrasing, markdown-image exfil, templated URLs, and others), plus leetspeak and spaced-letter de-obfuscation. Signals combine with a noisy-OR. Each verdict lists the signals that fired, which is what makes the audit log readable. Quoted examples inside text about prompt injection are dampened, so security articles don't trip it.

Optional LLM judge. Set detector.llm_judge: true and ANTHROPIC_API_KEY. Grey-zone outputs then go to a small Claude model, and its score is blended with the heuristic score. If the judge errors, the firewall falls back to heuristics and doesn't crash.

Policy engine (agent_firewall/policy.py), loosely in the spirit of CaMeL / FIDES-style information-flow control:

* Egress to allowlisted destinations (for example @acme.com) is allowed when untrusted and needs approval when compromised. On top of the matrix:

• Tainted destinations: an email address or host that first appeared inside flagged content is blocked as a destination.

• DLP: credential-looking values (sk-…, AKIA…, ghp_…, password=…) headed to any non-allowlisted destination are blocked, whatever the taint.

Tool-definition defences (agent_firewall/proxy.py). Tool descriptions and schema text are scanned at tools/list, and poisoned tools are hidden from the agent. Each tool definition's hash is pinned on first sight, and a tool that changes later is quarantined.

Project layout

agent_firewall/   detector.py  policy.py  proxy.py  audit.py  dashboard.py  playground.py  evaluate.py  cli.py
demo/             victim_server.py (simulated email/web/files), poisoned_server.py, run_demo.py, fixtures/
evals/            build_dataset.py, build_holdout.py, dataset.jsonl, holdout.jsonl
reports/          eval + held-out reports (markdown + json)
tests/            detector, policy, proxy (end-to-end, poisoning, rug pull, approvals), dashboard, playground
firewall.yaml     config: upstreams, thresholds, risk rules, policy matrix, allowlist, approvals
render.yaml       free Render deployment of the playground

Limitations and roadmap

• Heuristics generalise poorly (see held-out recall). Next steps: run the LLM judge on the held-out set, and add a small fine-tuned classifier (for example a DeBERTa prompt-injection model) as a middle tier.

• Approval friction: 64% of held-out attacks and some benign actions need a human. Ideas: per-task intent (allow egress the user explicitly asked for), plan-then-execute (CaMeL-style), and taint that decays or is scoped to specific data.

• Evaluate on public benchmarks (AgentDojo, InjecAgent, BIPIA) with real models instead of a scripted, always-hijacked agent.

• Taint is per session and coarse. Track it per value (which output fed which argument).

• Images and PDFs returned by tools aren't scanned yet.

Resume bullets

• Built agent-firewall, an MCP proxy that defends AI agents against prompt injection with a layered detector (hidden-HTML, Unicode-smuggling, base64 and multilingual detection), information-flow taint tracking, DLP, and human-in-the-loop approvals with a live audit dashboard. Python, MCP SDK, FastAPI, SQLite.

• Cut attack success for a fully hijacked agent from 100% to 0% across 70 injection attacks in 12 categories, with 0% false positives on 54 hard-negative benign samples and ~0.2 ms detection latency.

• Built an evaluation harness with a held-out test set. It showed heuristic detection recall falling from 96% to 36% on unseen attacks, while taint-based policy enforcement still held attack success at 0%, which shaped the design toward defense in depth.

• Detects MCP-specific supply-chain attacks: tool-description poisoning and post-install tool redefinition ("rug pulls").

License

MIT