Marketing Agent Eval Harness

A Python portfolio application for testing SQL-grounded agent workflows over a synthetic marketing dataset. It exposes two tools through an official stdio Model Context Protocol (MCP) server and keeps the supporting planner, SQL guard, and regression evaluation logic inspectable.

What it does

• Seeds a local SQLite warehouse with synthetic campaign, spend, conversion, and revenue data.

• Maps a small set of analytics questions to deterministic, read-only SQL and returns the answer with the selected intent, SQL, and result rows.

• Rejects write operations and multi-statement SQL before a query reaches SQLite.

• Runs JSON evaluation cases, produces Markdown and JSON reports, and applies a configurable quality gate.

• Exposes query_marketing_analytics and run_marketing_agent_evals through the official MCP Python SDK over stdio.

• Retains a small JSON-RPC-shaped adapter so the same shared tool service can be tested independently of an MCP client.

Architecture

flowchart LR
  A["MCP client"] --> B["Official stdio MCP server"]
  B --> C["Shared tool service"]
  C --> D["Deterministic planner"]
  D --> E["Read-only SQL guard"]
  E --> F["SQLite marketing warehouse"]
  C --> G["Eval runner"]
  G --> H["JSON and Markdown reports"]

The MCP server is an interface layer only. It calls the same service functions as the existing JSON-RPC-shaped adapter, so tool behavior is tested in one place rather than duplicated.

Run locally

Use Python 3.10 or newer:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e .
python -m unittest discover -s tests -v
marketing-agent-eval seed-db
marketing-agent-eval ask "Which campaign has the best ROAS?"
marketing-agent-eval eval --cases evals/marketing_eval_cases.json --report-dir reports

The supplied evaluation cases currently report:

cases: 4
passed: 4
score: 1.00
quality_gate: pass

Run the MCP server

marketing-agent-eval mcp

The command starts a stdio server. Configure a compatible MCP client to run the installed marketing-agent-eval executable with the mcp argument. The server exposes:

• query_marketing_analytics(question, database_path) - returns the answer, intent, read-only SQL, result rows, and row count.

• run_marketing_agent_evals(cases_path, database_path) - returns case counts, score, quality gate, and per-case checks.

The test suite launches the server through stdio, completes the MCP initialization handshake, lists both tools, and calls the analytics tool.

Commands

marketing-agent-eval seed-db
marketing-agent-eval ask "Show spend by channel"
marketing-agent-eval eval --cases evals/marketing_eval_cases.json --report-dir reports
marketing-agent-eval mcp-sample
marketing-agent-eval mcp

Evidence boundaries

This is a portfolio application, not a production marketing platform. It uses synthetic data and deterministic planner rules; it does not call a live LLM, operate an autonomous agent, connect to customer data, or claim Claude Code or enterprise MCP-client usage. The official MCP server is tested locally through stdio, not against a hosted gateway.