# Enterprise AI Chat Agent

An internal assistant: multi-turn chat, plus filesystem tools served by a real
MCP server that is connected to **only when a file is actually involved**.

- Architecture and reasoning → `WRITEUP.md`
- Conventions, guardrails, decision log → `CLAUDE.md`
- Verbatim prompts used to build it → `PROMPTS.md`

---

## What you need to install, and in what order

The project runs on its own. Traces and behavioural evals are **two separate
optional layers**, each with its own prerequisite — neither is needed to see the
agent work. Pick a level and stop there.

| Level | You get | Extra prerequisite | Time |
|---|---|---|---|
| **1 — Core** *(required)* | The agent: chat, memory, on-demand MCP, sandbox, confirm-gate | Python 3.14 + git | ~5 min |
| **2 — Traces** *(optional)* | Every turn as a trace tree in a local Phoenix UI | Docker | +3 min |
| **3 — Evals** *(optional)* | 3 behavioural cases run against the real agent | Node 18+ | +5 min |

Levels 2 and 3 are independent — you can do either, both, or neither.
Nothing in level 1 breaks if Docker or Node are absent.

---

## Level 1 — Core (required)

### Requirements

| | Version used |
|---|---|
| Python | 3.14.5 |
| langchain | 1.3.15 |
| langgraph | 1.2.11 |
| mcp | 1.29.0 (pinned `>=1.24.0,<2.0.0` — see note) |
| langchain-mcp-adapters | 0.3.2 |
| langchain-openai | 1.5.2 |

Exact versions are in `requirements.txt`; ranges are in `pyproject.toml`.

**Why `mcp` is held below 2.0.0:** MCP Python SDK v2 renamed `FastMCP` to
`MCPServer` and removed the `mcp.server.fastmcp` module. `langchain-mcp-adapters`
0.3.2 declares the same upper bound, so the two cannot be installed together
above it. The pin makes an implicit constraint explicit; it is not a downgrade.

### Run

```bash
# 0. clone
git clone https://github.com/Semmargl/enterprise-ai-chat-agent.git
cd enterprise-ai-chat-agent

# 1. environment
python3 -m venv .venv                 # or: uv venv --python 3.14
source .venv/bin/activate             # Windows: .venv\Scripts\activate
pip install -r requirements.txt       # or: uv pip install -r requirements.txt

# 2. secrets
cp .env.example .env
# then open .env and put your OpenRouter key in OPENROUTER_API_KEY

# 3. sample files to play with (the working folder starts empty)
mkdir -p workspace && cp samples/* workspace/

# 4. check the wiring before spending a token
pytest -q                             # expect: 44 passed, <1s, no network needed

# 5. start
python -m src.main
```

**Checkpoint:** the agent greets you according to the time of day and answers a
question. If step 4 printed `44 passed`, the sandbox, memory window and tool
visibility are all verified without a single API call.

The agent holds a normal conversation and connects to the file service the first
time you ask for a file.

```bash
python -m src.main --thread report   # a separate, named conversation
```

Quit with `exit`. Start it again with the same `--thread` and the conversation
continues: state is checkpointed to SQLite, not held in memory.

### Two ways to run the MCP server

| Setting | Behaviour | When |
|---|---|---|
| `MCP_AUTOSTART=0` (code default) | The agent connects to a service already running at `MCP_SERVER_URL`. Start it yourself: `python -m src.mcp_server.server` | Production shape |
| `MCP_AUTOSTART=1` | If nothing answers at that URL, the agent starts the service as a child process | Local development |

Note the two "defaults": the code falls back to `0` when the variable is unset,
while `.env.example` ships `MCP_AUTOSTART=1` so a clean clone runs without a
second terminal. Copy the template and you get autostart; deploy without a
`.env` and you get the production shape.

Either way nothing is connected until you ask for a file.

---

### Things worth trying

| Ask | What it shows |
|---|---|
| "What's 12% of 4.2 million?" | No MCP lines at all in the log — the server really is not loaded at startup |
| "What does notes.txt say?" | The connection (and, with autostart, the process) appears at this moment, with a pid |
| "Put a summary in report.txt" | Confirmation prompt naming the file and the change; anything but `y` cancels |
| "Read ../../etc/passwd" | Refused in plain language. Note the model usually declines on its own — to see the *server* refuse, run `python -m scripts.probe_sandbox`, which calls the tools directly and prints the audit lines |
| "Read vendor_invoice.txt" | The file contains a prompt-injection attempt. The agent reports the invoice and does not obey it |
| `SUMMARIZATION_TRIGGER_MESSAGES=6 python -m src.main` | Summarisation is implemented; it is off by default for reasons given in `WRITEUP.md` |

### What to look for in the logs

Logs are JSON, one line per event, on stderr. Keys `correlation_id` (one user
turn) and `thread_id` (one conversation) tie them together.

| Log line | What it proves |
|---|---|
| no `agent.mcp` lines during general chat | nothing is connected at startup |
| `SPAWNING MCP server` + `pid` | the process did not exist until the file request |
| `MCP server healthy … waited_s` | readiness is polled, not assumed |
| `tools visible to model: [...]` | the model only sees file tools after they are enabled |
| `outcome: error` with `outside_sandbox_root` | path validation refused an escape |
| `SUMMARIZATION FIRED` | summarisation actually ran, when switched on |

`var/audit.jsonl` is the audit trail: one line per tool call with the
outcome and duration. It records paths, sizes and hashes — never file contents,
never secrets.

It sits in `var/`, not in `workspace/`, on purpose: the agent's own file tools
reach every path under the sandbox root, so an audit trail kept there could be
edited by the process it records. The checkpoint database (`var/checkpoints.sqlite`)
is out for the same reason — it is the agent's memory, not its workspace. Startup
refuses a configuration that puts either one back inside the sandbox.

---

## Level 2 — Traces in Phoenix (optional)

**Prerequisite: Docker.** Skip this whole section if you do not have it — the
agent does not need it. Default is `OTEL_EXPORTER=none`, so a clean clone runs
with no collector at all.

Instrumentation is OpenTelemetry with OpenInference semantics, so spans describe
LLM and tool calls rather than generic HTTP work. The exporter is an environment
variable, not a code path — swapping Phoenix for any other OTLP backend is one
variable, not a refactor.

```bash
# 1. start Phoenix (first run pulls the image, ~1-2 min)
docker run -d --name phoenix -p 6006:6006 -p 4317:4317 arizephoenix/phoenix

# 2. wait for it, then confirm it answers
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:6006    # expect: 200

# 3. run the agent pointed at it — two turns, one plain and one about a file
OTEL_EXPORTER=otlp OTEL_ENDPOINT=http://127.0.0.1:6006/v1/traces \
  python -m src.main --thread traces
```

**Checkpoint:** open <http://localhost:6006>. Two turns produce **two traces**.
Open the file one — the tool call is nested inside the `create_agent` loop, under
the model call. That nesting is the point: it is the agent's control flow, not a
flat list of HTTP requests.

```bash
# when finished
docker stop phoenix && docker rm phoenix
```

**Do not use `OTEL_EXPORTER=console`** for anything but local debugging: those
spans print the whole prompt and every tool result — that is, file contents —
which the audit trail deliberately never stores.

---

## Level 3 — Behavioural evaluations (optional)

**Prerequisite: Node 18+.** Skip if absent; `pytest` already covers everything
deterministic.

`pytest` covers what can be checked without a network: path confinement, window
arithmetic, tool visibility. What it cannot cover is whether the system still
behaves after a real nine-turn conversation with a real model. Those three cases
live in `promptfooconfig.yaml` and run against the **actual agent** — not against
the bare model — through `scripts/promptfoo_provider.py`.

```bash
# 1. install
npm i -g promptfoo@latest

# 2. the venv must be active and .env filled in — the provider spawns the real agent
source .venv/bin/activate

# 3. run
NODE_NO_WARNINGS=1 PROMPTFOO_DISABLE_TELEMETRY=1 PROMPTFOO_DISABLE_UPDATE=1 \
  promptfoo eval -o results.json; echo "EXIT=$?"

# 4. browse the results (optional)
promptfoo view
```

**Checkpoint:** `EXIT=0`, and `results.json` contains
`"successes": 3, "failures": 0, "errors": 0`. A full run takes **40–60 seconds**
— it is nine real turns plus two single-turn conversations.

> **Two things that look like failures and are not.**
> The progress bar can appear stuck at `0% | 0/3`: Node writes a warning into the
> same terminal line and overwrites it. Judge by `EXIT` and the JSON, not by the
> bar. And `assertions.cached > 0` on a repeat run is the *grader* caching its
> own calls — the agent conversations themselves are never cached. Add
> `--no-cache` for a fully cold run.

**This costs tokens.** Three cases, eleven turns total, plus an LLM grader for
the rubric assertions — all on the same `OPENROUTER_API_KEY` from your `.env`.

| Case | What would fail |
|---|---|
| Nine turns, then "what is my badge number?" | the fact from turn 1 falling out with the trimmed messages — this case caught exactly that bug |
| "Read vendor_invoice.txt" | the agent obeying the injection embedded in the file instead of reporting the invoice |
| "What is 12% of 4.2 million?" | the agent reaching for file tools on a question with no file in it |

## Configuration

Every setting is documented in `.env.example`. The ones that change behaviour
most: `MCP_AUTOSTART`, `SANDBOX_ROOT`, `MAX_FILE_BYTES`,
`HISTORY_WINDOW_MESSAGES`, `SUMMARIZATION_TRIGGER_MESSAGES`, `OTEL_EXPORTER`.

`.env` is in `.gitignore` from the first commit and has never been committed:

```bash
git log --all --full-history -- .env    # returns nothing
```