# G2 Product Analytics MCP

A portfolio project exploring a practical question:

> **How do you let an AI agent answer product-analytics questions autonomously without giving it unrestricted SQL access or letting it invent business definitions?**

This project builds a small governed analytics layer, exposes it through **Model Context Protocol (MCP)**, and lets an MCP-capable LLM investigate product questions in natural language.

The synthetic dataset intentionally contains a hidden July 2026 deterioration in mobile review completion. The agent is not told the answer. It has to discover the metric, compare periods, segment the change, and stop where the evidence stops.

## What the demo proved

In a cold-start test, the agent was asked:

> “One of our product metrics changed materially in July 2026. Using only the G2 Product Analytics MCP tools, investigate what happened.”

The agent independently found:

- `review_completion_rate` was the material outlier.
- Overall completion fell from **66.7% in June to 60.5% in July**.
- The decline was concentrated in **mobile: 62.7% → 47.9% (-14.8pp)**.
- Desktop was nearly flat, which made device type a much cleaner signal than country or product category.
- Daily volume was too thin to claim an exact “cliff day.”
- The evidence established **association, not causation**.

That distinction is the core design goal: **LLM reasoning on top of governed evidence, not free-form guessing against raw tables.**

---

## Architecture

```mermaid
flowchart TD
    U["User<br/>Natural-language business question"]
    A["LLM / Agent<br/>Reasoning + tool selection"]
    MCP["MCP Server<br/>Tools + Resource + Prompt"]
    SEM["Governed metric layer<br/>definitions + dimensions + ownership + freshness"]
    EXE["Analytics execution<br/>Python + DuckDB"]
    DATA["Synthetic product data<br/>CSV tables"]

    U --> A
    A --> MCP
    MCP --> SEM
    MCP --> EXE
    EXE --> DATA
    SEM --> EXE
    EXE --> MCP
    MCP --> A
    A --> U
```

### Mental model

**LLM decides. MCP connects. Tools execute. Semantic definitions govern. Data provides evidence.**

MCP is **not** the model, semantic layer, or database. It is the standardized interface through which the agent discovers and invokes capabilities.

---

## Why I built it this way

A naive analytics agent might expose:

```python
run_sql(sql: str)
```

That is convenient, but it also lets the model:

- silently redefine metrics,
- choose the wrong denominator,
- double-count joins,
- query deprecated or sensitive fields,
- create inconsistent answers across users,
- and sound confident even when the underlying question is ambiguous.

This project takes the opposite approach.

The agent receives a deliberately small set of **governed analytical primitives**:

| Tool | Purpose |
|---|---|
| `list_metrics` | Discover approved business metrics. |
| `get_metric_definition` | Retrieve the governed definition before interpretation. |
| `get_metric` | Calculate one approved metric for an inclusive date range. |
| `compare_review_completion_by_dimension` | Slice review completion by an approved dimension. |
| `compare_periods` | Quantify absolute and relative change across two periods. |
| `diagnose_review_completion_change` | Compare device segments across two periods and explicitly warn that association is not causation. |

The server also exposes:

- an MCP **resource**: `metrics://catalog`
- an MCP **prompt**: `product_analytics_investigation`

---

## Repository map

```text
g2-product-analytics-mcp/
├── README.md
├── LICENSE
├── pyproject.toml
├── .gitignore
│
├── data/
│   ├── users.csv
│   ├── software_products.csv
│   ├── buyer_sessions.csv
│   ├── search_events.csv
│   ├── comparison_events.csv
│   ├── reviews.csv
│   ├── review_events.csv
│   └── metric_catalog.csv
│
├── scripts/
│   └── generate_dummy_data.py
│
├── src/
│   └── g2_product_analytics_mcp/
│       ├── __init__.py
│       └── server.py
│
├── tests/
│   └── test_server.py
│
├── docs/
│   ├── ARCHITECTURE.md
│   ├── DATA_MODEL.md
│   ├── CODE_WALKTHROUGH.md
│   ├── AGENT_EVALUATION.md
│   └── INTERVIEW_WALKTHROUGH.md
│
└── .github/
    └── workflows/
        └── test.yml
```

See [`docs/CODE_WALKTHROUGH.md`](docs/CODE_WALKTHROUGH.md) for a detailed explanation of the Python code and [`docs/DATA_MODEL.md`](docs/DATA_MODEL.md) for every synthetic table.

---

## Quick start

### 1. Install prerequisites

- Python 3.10+
- [`uv`](https://docs.astral.sh/uv/)
- Node/npm only if you want MCP Inspector
- An MCP-capable client such as Claude Code if you want the agent demo

### 2. Clone and install

```bash
git clone <YOUR-REPO-URL>
cd g2-product-analytics-mcp
uv sync --extra dev
```

### 3. Run tests

```bash
uv run pytest -q
```

### 4. Inspect the MCP server manually

```bash
uv run mcp dev src/g2_product_analytics_mcp/server.py
```

MCP Inspector should discover the tools, prompt, and metric-catalog resource.

### 5. Connect Claude Code

From this repository:

```bash
claude mcp add g2-product-analytics -- \
  uv run --with mcp==2.0.0 mcp run src/g2_product_analytics_mcp/server.py
```

Check health:

```bash
claude mcp list
```

Then launch Claude:

```bash
claude
```

A useful cold-start evaluation prompt is:

```text
One of our product metrics changed materially in July 2026.
Using only the G2 Product Analytics MCP tools, investigate what happened.
Identify the metric, quantify the change, determine when the change emerged,
find where it is most concentrated, and explain what the evidence does and
does not support. Do not inspect source code or CSV files.
```

---

## Reproduce the synthetic data

The committed CSVs make the demo immediately runnable.

They can also be regenerated deterministically:

```bash
uv run python scripts/generate_dummy_data.py
```

The generator uses a fixed random seed and intentionally plants one anomaly:
**mobile review completion deteriorates in July 2026**.

This is synthetic portfolio data. It contains no G2 proprietary information.

---

## Tests and guardrails

The current unit tests validate two foundational properties:

1. `review_completion_rate` is a valid ratio with a non-zero denominator.
2. mobile completion in July is lower than mobile completion in June.

The more important evaluation layer is behavioral. See [`docs/AGENT_EVALUATION.md`](docs/AGENT_EVALUATION.md) for prompts covering:

- definition governance,
- unsupported metrics,
- unsupported dimensions,
- ambiguous questions,
- causality traps,
- leading questions,
- cold-start anomaly investigation.

---

## What I would productionize next

The portfolio version is intentionally compact:

```text
CSV → DuckDB → governed Python metrics → MCP → LLM agent
```

A production version would likely add:

- warehouse/dbt models instead of CSVs,
- a formal semantic/metrics layer,
- identity propagation and SSO/OAuth,
- row- and column-level authorization,
- audit logs and tool-call observability,
- metric lineage/versioning,
- freshness/schema monitoring,
- query budgets and timeouts,
- PII policies,
- golden-question eval suites,
- and a self-service UI/Slack surface.

---

## Design tradeoff I learned

The project intentionally contains both **atomic tools** (`get_metric`) and a more opinionated workflow tool (`diagnose_review_completion_change`).

During testing, the agent showed it could compose atomic tools into new workflows, including weekly trend analysis, without a dedicated `weekly_trend()` function.

That suggests the scalable direction is **not** to hard-code every analyst question as a new MCP tool. It is to expose a small number of trusted primitives with strong semantics and let the agent compose them.

---

## Documentation

- [Repository guide](docs/REPOSITORY_GUIDE.md)
- [Architecture](docs/ARCHITECTURE.md)
- [Data model](docs/DATA_MODEL.md)
- [Code walkthrough](docs/CODE_WALKTHROUGH.md)
- [Agent evaluation](docs/AGENT_EVALUATION.md)
- [Interview walkthrough](docs/INTERVIEW_WALKTHROUGH.md)

## License

MIT. Synthetic data only.