# E-Commerce Support Agent

A harness-controlled, tool-connected, RAG-grounded customer support
agent for a mock e-commerce company. It answers order-status, delivery,
account, and policy questions by calling real (mock) tools and a real
retriever — never by inventing an answer — and a Python harness, not
the model, decides which tool call is actually allowed to run.

## Prerequisites
- Python 3.12
- [uv](https://docs.astral.sh/uv/) for dependency management — if it's
  not installed yet: `pip install uv` (or see the link above for other
  install methods)
- An Anthropic API key

## Setup
```bash
git clone <repo-url>
cd <the-folder-git-just-created>
uv sync
cp .env.example .env
```

`uv sync` creates a `.venv` in the project folder and installs everything into it
automatically. Every command elsewhere in this README is prefixed with `uv run`,
which runs inside that `.venv` without needing it activated — but if you'd rather
activate it directly (e.g. to run `python`/`streamlit` without the `uv run` prefix):

```bash
# macOS/Linux
source .venv/bin/activate

# Windows (PowerShell)
.venv\Scripts\Activate.ps1

# Windows (cmd.exe)
.venv\Scripts\activate.bat
```

Edit `.env`:
```
ANTHROPIC_API_KEY=sk-ant-...
CLAUDE_MODEL=claude-sonnet-5   # any model your account can access
```

The first run downloads the
`sentence-transformers/multi-qa-MiniLM-L6-cos-v1` embedding model used
for policy retrieval — a few seconds, then cached locally — and creates
`data/chroma_db/`, the persisted vector index (gitignored; safe to
delete if you want to force a full re-index).

## How to run it

**Chat with the agent (start here):**
```bash
uv run streamlit run app.py
```
Pick one of the sample customers from the dropdown — their account
status is shown, since a couple are deliberately flagged/suspended so
you can demo how the agent handles that — and start chatting.

**Or run one scenario at a time, no UI:**
```bash
uv run python -m scripts.test_order_authorization
```
See [Verifying it works](#verifying-it-works) for the full list.

**Run the MCP server standalone** (for an MCP-compatible client, e.g.
Claude Desktop):
```bash
uv run python -m src.mcp_server
```

## Project structure
```
app.py                     Streamlit chat UI — login + chat, no business logic of its own

src/
  agent.py                 CustomerSupportAgent — the harness loop + system prompt
  auth_context.py          AuthContext — the authenticated customer_id for a session
  conversation_memory.py   Short-term, in-process message history (last N turns)
  llm.py                   Thin wrapper around the Anthropic Messages API
  tools.py                 Data-access functions: orders, accounts, products
  action_guard.py          request_order_cancellation — ownership/state/confirmation checks
  tool_registry.py         Tool schemas (Claude tool-use format) + execute_tool() — the harness gate
  policy_retriever.py      Loads/chunks policies/*.md; hybrid retrieval — Chroma (dense/cosine) + BM25 (sparse), fused via Reciprocal Rank Fusion
  policy_qa.py             search_policies (tool-facing) + the grounding filter/thresholds
  mcp_server.py            FastMCP server exposing the same tools over stdio
  support_session.py       create_session / SupportSession — the boundary external callers use
  logging_config.py        Console + rotating-file logging setup
  audit_logger.py          Per-turn and per-tool-call log records

data/            Mock orders, accounts, products (JSON); chroma_db/ — persisted vector index, generated on first run, gitignored
policies/        9 Markdown policy documents (chunked by "## " heading)
scripts/         Runnable scenarios and boundary checks (see Verifying it works)
logs/            Generated at runtime, gitignored
project_docs/    Deeper detail: architecture, security model, testing map
```

## Why I built the harness this way

**The harness decides, not the model.** Claude can only *propose* a
tool call (a name and arguments). `tool_registry.execute_tool` is the
only code path that actually runs one, and it always injects
`authenticated_customer_id` from the session's `AuthContext` — a value
Claude never sees as something it can set. An authorization bypass
here isn't a prompting problem to patch; there is no code path where
the model supplies whose data it's asking for.

**Ticket-type scoping.** The four ticket categories this project
scopes to (order status, delivery issues, refunds, subscription/account
questions) map directly onto the existing tool/retrieval surface,
rather than needing a separate classification step: order status and
delivery both resolve through `lookup_order` / `list_customer_orders`;
refund *questions* resolve through `search_policies` (actual refund
issuance — `issue_refund` — is explicitly out of scope for this
project); account questions resolve through `check_account_status`
for current state and `search_policies` for anything about *why* a
status exists or how to change it. I didn't add an explicit
classify-then-route stage because the four categories already have a
clean mapping onto tools Claude selects directly — an extra routing
layer would be one more place to misclassify without adding real
routing power at this scale.

**One MCP permission-boundary decision.** The in-process agent surface
and the MCP surface enforce ownership through the same underlying
checks (`action_guard.py`, `tools.py`), but they differ in *where the
customer ID comes from*. On the agent surface, `AuthContext` is set
once at session creation and every tool call gets
`authenticated_customer_id` injected by trusted code. On the MCP
surface, an external client has no `AuthContext` to inject from, so
`customer_id` is a plain tool argument instead — and the server logs
every call in full but does not itself verify that argument. I kept
this trust boundary explicit and visible (see
[project_docs/02_security_and_authorization.md](project_docs/02_security_and_authorization.md))
rather than either quietly inheriting the same gap on the trusted
surface, or building a fake auth layer for a single-process demo with
no real identity provider behind it.

**Policy retrieval lives behind MCP, for governance and standardization,
not just convenience.** `search_policies` is exposed as an ordinary tool
— identically from the in-process agent and from `mcp_server.py` —
rather than a Python-internal pre-fetch step. That means any
MCP-compatible client, not just this agent, can call the same governed
retrieval capability the same way, instead of every consumer
re-implementing the embedding/retrieval pipeline itself. The tradeoff
this creates — grounding is no longer a forced step — is real, and it's
what the next paragraph addresses.

**Grounding is a checked guarantee, not just an instruction.**
`search_policies` is a tool Claude decides to call — the system prompt
requires it before any policy answer, but a model can still skip it.
After the tool-use loop produces an answer with no further tool calls,
the harness re-runs the same local retriever directly against the raw
customer message before returning that answer; if it finds a strong
match and `search_policies` was never called that turn, the harness
forces one more loop iteration with a corrective instruction instead
of returning the ungrounded answer. This closes the main gap in
"the system prompt tells it to call the tool" — grounding no longer
depends entirely on the model remembering to.

**Hybrid retrieval (Chroma + BM25), not a single similarity signal.**
Policy chunks are embedded once and indexed in a persistent Chroma
collection (`data/chroma_db/`, cosine space) instead of an in-memory
array rebuilt on every start, and that dense ranking is fused with a
BM25 lexical ranking (`rank_bm25`) via Reciprocal Rank Fusion — dense
catches paraphrases with no shared words, BM25 catches exact terms dense
retrieval can blur past. The two rankings are fused by rank position, not
raw score, since cosine similarity and BM25 scores live on incompatible
scales. One consequence worth knowing: RRF's fused score reflects rank,
not calibrated relevance, so `policy_retriever.retrieve()` returns both
`relevance_score` (RRF, used for relative filtering) and
`dense_similarity` (raw cosine, used as the absolute floor that decides
whether to ground at all) — see `policy_qa._filter_relevant_chunks`. A
single RRF-based threshold can't reject an out-of-scope question, since
the top result always passes relative to itself; the absolute floor is
what actually enforces the honest-gap requirement.

## Tools
| Tool | Purpose | Auth model |
|---|---|---|
| `search_policies` | Semantic search over policy documents | Not customer-scoped; Claude calls it on its own judgment before any policy answer |
| `lookup_order` | Status + delivery info for one order | Denies access if the order belongs to another customer |
| `list_customer_orders` | All orders for the authenticated customer | Same |
| `check_account_status` | Active / flagged / suspended / `can_place_orders` | Same |
| `check_product_availability` | Stock quantity + availability | Not customer-scoped |
| `request_order_cancellation` | Two-step guarded cancellation | Ownership + cancellable-state + explicit confirmation, all checked before anything is written to disk |

Every tool returns `{"success": bool, ...}`; failures carry an
`error_code` (`ORDER_NOT_FOUND`, `ORDER_ACCESS_DENIED`,
`ORDER_NOT_CANCELLABLE`, `AUTHENTICATION_REQUIRED`, etc.) so the agent
reacts to a stable code, not free-text error parsing.

## Logging & observability
There's no `logs/` folder in the repo — it's gitignored, and nothing
under it ships. It's created automatically the first time the app
runs: `configure_logging()` (`src/logging_config.py`) runs on first
import of `audit_logger` or `policy_retriever`, creates `logs/` if it
doesn't already exist, and attaches handlers for two files, created
the same way:

- **`logs/application.log`** — human-readable, one line per event
  (`event=name key=value ...`): startup, per-query retrieval
  performance, one line per conversation turn.
- **`logs/observability.log`** — full-detail JSON records, one per
  tool call and one per conversation turn, linked by `session_id`.
  This is what makes RAG grounding traceable: every `search_policies`
  call is logged with the exact chunk(s) retrieved (source, section,
  content, RRF relevance score, and dense-similarity score), not just a
  claim that retrieval happened.

Both files rotate at 5 MB (5 backups kept) so they don't grow
unbounded across a long session. Read `observability.log` with
`uv run python -m scripts.view_logs` (filters: `--customer`, `--tool`,
`--failures`, `--type`, `--summary`) rather than opening the raw file.

## Verifying it works
| Script | What it exercises | Needs `ANTHROPIC_API_KEY` |
|---|---|---|
| `scripts/test_order_authorization.py` | Full agent loop: cross-customer order access is denied | Yes |
| `scripts/test_account_status.py` | Full agent loop: `check_account_status` for active/flagged/suspended accounts | Yes |
| `scripts/test_agent_cancellation.py` | Full agent loop: guarded cancellation, with confirmation | Yes |
| `scripts/test_conversation_memory.py` | Full agent loop: multi-turn follow-ups resolved from short-term memory | Yes |
| `scripts/test_support_session.py` | `create_session` login validation + `SupportSession.ask()` response shape | Yes |
| `scripts/test_search_policies_tool.py` | Claude calls `search_policies` unprompted for a policy question, skips it for a pure operational one | Yes |
| `scripts/test_guarded_cancellation.py` | `action_guard.request_order_cancellation` directly, no LLM | No |
| `scripts/test_tool_registry.py` | `execute_tool()` dispatch and error codes, no LLM | No |
| `scripts/test_policy_qa.py` | Policy retrieval/grounding in isolation, no LLM | No |
| `scripts/test_mcp_boundaries.py` | MCP schema validation rejecting malformed/unexpected arguments, live `fastmcp.Client` | No |
| `scripts/test_mcp_tools.py` | Lists the tools/schemas the MCP server registers | No |
| `scripts/test_mcp_cancellation.py` | MCP `cancel_order` enforces the same ownership/state/confirmation guard as the agent path | No |
| `scripts/test_mcp_search_policies.py` | MCP's `search_policies`, called directly: relevant vs. irrelevant query | No |
| `scripts/view_logs.py` | Terminal viewer for `logs/observability.log` | No |

Run any of them with:
```bash
uv run python -m scripts.test_order_authorization
```

**Note:** `test_guarded_cancellation.py`, `test_agent_cancellation.py`,
and `test_mcp_cancellation.py` all cancel a real order in
`data/orders.json` (`ORD-1004`). Re-running one after the first hits
`ORDER_ALREADY_CANCELLED` instead of the full confirm flow. Reset with
`git checkout -- data/orders.json`.

See [project_docs/](project_docs/01_architecture.md) for a deeper look
at the architecture, the full security/authorization model, and a
requirement-by-requirement testing map.

## Known limitations
- **Long-term memory is not implemented.** Short-term, in-conversation
  memory (`ConversationMemory`) exists; a prior-ticket-history lookup
  keyed by customer ID does not. This is a known gap against the
  project's requirements, not a design choice — noted here rather
  than left silent.
- **Conversation memory is in-process only** — no persistence across
  runs.
- **The MCP surface trusts the calling client's `customer_id`
  outright.** Every MCP call is logged in full, but logging isn't
  authentication.
- **Policy grounding depends on two thresholds** — an absolute
  dense-similarity floor and a relative RRF-score cutoff — both tuned
  against a handful of measured queries, not a systematic eval set (the
  absolute floor has been re-verified against the current hybrid scores;
  the relative cutoff is carried over from the pre-hybrid design and not
  yet re-measured). Accurate for the cases checked against; a
  differently-phrased edge case could still land on the wrong side of a
  cutoff.
- **No concurrency handling** on the JSON data files — fine for a
  single local demo, not for concurrent writers.