# Marketing Agent Eval Harness

A Python portfolio application for testing SQL-grounded agent workflows over a synthetic marketing dataset. It exposes two tools through an official stdio Model Context Protocol (MCP) server and keeps the supporting planner, SQL guard, and regression evaluation logic inspectable.

## What it does

- Seeds a local SQLite warehouse with synthetic campaign, spend, conversion, and revenue data.
- Maps a small set of analytics questions to deterministic, read-only SQL and returns the answer with the selected intent, SQL, and result rows.
- Rejects write operations and multi-statement SQL before a query reaches SQLite.
- Runs JSON evaluation cases, produces Markdown and JSON reports, and applies a configurable quality gate.
- Exposes `query_marketing_analytics` and `run_marketing_agent_evals` through the official MCP Python SDK over stdio.
- Retains a small JSON-RPC-shaped adapter so the same shared tool service can be tested independently of an MCP client.

## Architecture

```mermaid
flowchart LR
  A["MCP client"] --> B["Official stdio MCP server"]
  B --> C["Shared tool service"]
  C --> D["Deterministic planner"]
  D --> E["Read-only SQL guard"]
  E --> F["SQLite marketing warehouse"]
  C --> G["Eval runner"]
  G --> H["JSON and Markdown reports"]
```

The MCP server is an interface layer only. It calls the same service functions as the existing JSON-RPC-shaped adapter, so tool behavior is tested in one place rather than duplicated.

## Run locally

Use Python 3.10 or newer:

```bash
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e .
python -m unittest discover -s tests -v
marketing-agent-eval seed-db
marketing-agent-eval ask "Which campaign has the best ROAS?"
marketing-agent-eval eval --cases evals/marketing_eval_cases.json --report-dir reports
```

The supplied evaluation cases currently report:

```text
cases: 4
passed: 4
score: 1.00
quality_gate: pass
```

## Run the MCP server

```bash
marketing-agent-eval mcp
```

The command starts a stdio server. Configure a compatible MCP client to run the installed `marketing-agent-eval` executable with the `mcp` argument. The server exposes:

- `query_marketing_analytics(question, database_path)` - returns the answer, intent, read-only SQL, result rows, and row count.
- `run_marketing_agent_evals(cases_path, database_path)` - returns case counts, score, quality gate, and per-case checks.

The test suite launches the server through stdio, completes the MCP initialization handshake, lists both tools, and calls the analytics tool.

## Commands

```bash
marketing-agent-eval seed-db
marketing-agent-eval ask "Show spend by channel"
marketing-agent-eval eval --cases evals/marketing_eval_cases.json --report-dir reports
marketing-agent-eval mcp-sample
marketing-agent-eval mcp
```

## Evidence boundaries

This is a portfolio application, not a production marketing platform. It uses synthetic data and deterministic planner rules; it does not call a live LLM, operate an autonomous agent, connect to customer data, or claim Claude Code or enterprise MCP-client usage. The official MCP server is tested locally through stdio, not against a hosted gateway.