# commitment-tracker-mcp

An MCP server that tracks **commitments** — promises people make in
conversation ("I'll send the report by Friday") — so they can be listed, closed
out, and flagged when they go overdue.

The repo currently covers **Phase 1** (core data model + standalone, reusable
MCP server over SQLite) and **Phase 2** (the agent orchestrator that does LLM
classification, extraction, and drafting, and talks to the server over MCP).

## Layout

**Phase 1 — storage (MCP server side):**

| File             | Responsibility                                             |
|------------------|------------------------------------------------------------|
| `schema.sql`     | The `commitments` table definition + indexes.              |
| `models.py`      | Typed models (`Commitment`, `Status`) and UTC time helpers.|
| `db.py`          | SQLite data-access layer. **Knows nothing about MCP.**     |
| `server.py`      | FastMCP server exposing the storage/query tools.           |
| `test_client.py` | Manual end-to-end test that drives the server over stdio.  |

**Phase 2 — agent (MCP client side):**

| File                 | Responsibility                                                  |
|----------------------|-----------------------------------------------------------------|
| `llm.py`             | All Gemini API calls: flash-lite classification gate, flash extraction/drafting/digest (structured outputs). Includes a `StubLLM` for offline testing. |
| `mcp_client.py`      | Async client for the commitment-tracker server. The orchestrator's *only* path to storage. |
| `memory.py`          | Short-term thread memory (`ConversationMemory`). Long-term memory is the commitments DB — no vector store, by design. |
| `orchestrator.py`    | Reactive loop: thread context → fulfillment check → classify → extract → store → draft → **confirm**. |
| `digest.py`          | Proactive daily job: overdue sweep + due-soon summary + nudge drafts. |
| `sample_emails.json` | Demo inbox (includes a 2-message thread that demonstrates fulfillment). |
| `sources.py`         | Email sources: `JsonInboxSource` (offline default) and `GmailSource` (read-only OAuth), normalized to one dict shape. |
| `web/api.py`         | FastAPI backend for the dashboard (reuses `db.py` + `llm.py`). |
| `web/static/index.html` | Single-file React dashboard (no build step). |
| `eval.py`            | Phase 4 eval harness: metrics over a labeled set, resumable disk cache. |
| `eval_dataset.json`  | 32 labeled synthetic emails (gold commitments, deadlines, confidence). |

## Setup

```bash
pip install -r requirements.txt
```

Requires Python 3.10+.

## Run the server

```bash
python server.py          # stdio transport, for local/agent use
```

It creates `commitments.db` next to the code on first run. Point it elsewhere
with the `COMMITMENT_DB_PATH` environment variable.

## Prove it works

```bash
python test_client.py
```

This spins up the server over stdio against a throwaway database, discovers the
tools, and runs a full lifecycle (add → list → fulfill → overdue), printing
PASS/FAIL for each assertion. It touches your real `commitments.db` never.

## Run the agent (Phase 2)

The live path uses **Google Gemini's free tier** (no billing). Get a key at
<https://aistudio.google.com/apikey>, then put it in a `.env` file:

```bash
cp .env.example .env      # then edit .env and paste your key
```

`.env` is **gitignored** — the key never gets committed. The entrypoints call
`load_dotenv()`, so it's picked up automatically:

```bash
python orchestrator.py                 # process sample_emails.json
python digest.py                       # daily digest + nudge drafts

# Offline — full pipeline & MCP wiring with a deterministic stub LLM (no key):
python orchestrator.py --dry-run
python digest.py --dry-run             # digest without the LLM summary
```

(You can also just `export GEMINI_API_KEY=...` in your shell instead of `.env`.)
If a model isn't enabled on your key, override the IDs:
`GEMINI_CLASSIFIER_MODEL` / `GEMINI_EXTRACTOR_MODEL`.

### Reading from real Gmail

Emails come from a **source** (`sources.py`) so the pipeline is ignorant of
where mail lives. Default is the JSON fixture; `--source gmail` reads real mail
over **read-only OAuth** (`gmail.readonly` — this app can read your mail and
nothing else). Gmail's own `threadId` maps onto `thread_id`, so thread memory
and fulfillment detection work on real conversations.

One-time setup:
1. In [Google Cloud Console](https://console.cloud.google.com/): create a
   project → enable the **Gmail API** → **OAuth consent screen** (External, add
   yourself as a test user) → **Credentials → Create OAuth client → Desktop app**.
2. Download it as `credentials.json` into this folder (gitignored).
3. Run it — a browser opens for consent once; the token is cached in `token.json`
   (gitignored):
   ```bash
   python orchestrator.py --source gmail --query "in:inbox newer_than:7d" --max 10
   python orchestrator.py --source gmail --dry-run    # fetch real mail, stub LLM (no LLM cost)
   ```

Gmail API usage is free. `--query` takes any Gmail search string; `--max` caps
how many messages to pull.

The reactive loop per email: **classify** (Haiku — cheap gate) → **extract**
commitments (Opus, structured outputs, relative deadlines resolved against an
injected UTC date) → **store** via the MCP `add_commitment` tool → **draft** a
grounded reply → **confirm** with the user. Nothing is ever auto-sent — every
draft requires explicit approval (`--yes` exists for demos/CI only).

The digest job is the proactive side, meant for cron/Task Scheduler: it runs
`check_overdue`, lists what's due this week, and drafts nudges for review.

Model choices: `gemini-2.5-flash-lite` for classification (runs on every email,
so cheap/fast, thinking disabled), `gemini-2.5-flash` for extraction, drafting,
and the digest (runs only on emails that pass the gate). All LLM outputs use
structured outputs (Gemini `response_schema` + pydantic → `response.parsed`), so
no free-form JSON parsing anywhere. The provider is isolated to `llm.py` — the
server, MCP client, orchestrator, and digest are provider-agnostic.

## Tool surface

The server exposes storage and query tools only:

| Tool                              | Purpose                                          |
|-----------------------------------|--------------------------------------------------|
| `add_commitment(...)`             | Record a new, already-structured commitment.     |
| `list_open_commitments(before_date=None)` | List outstanding commitments.            |
| `mark_fulfilled(commitment_id)`   | Close a commitment (stamps `fulfilled_at`).      |
| `check_overdue(as_of=None)`       | Flag & return open commitments past their deadline. |
| `get_commitment(commitment_id)`   | Fetch one commitment by id.                      |

## Design boundaries

- **Extraction lives in the agent, not the server.** Turning raw messages into
  structured commitments is an LLM job that lives in `llm.py` on the
  orchestrator side, which then calls `add_commitment` with the resolved fields.
  Keeping the LLM call out of the server keeps this a pure, client-agnostic
  interop layer that a CLI, a Slack bot, or the agent can all reuse.
- **The orchestrator never imports `db.py`.** All storage goes through
  `mcp_client.py` over stdio — proving the MCP boundary is real, not decorative.
- **The user confirms every outward action.** Drafts (replies, nudges) are
  always presented for approval; the agent proposes, the human disposes.
- **LLM calls are isolated in `llm.py`.** `orchestrator.py` takes any object
  with the same methods, so the Phase 4 eval harness can call
  `extract_commitments()` directly on a labeled set — and `StubLLM` runs the
  whole pipeline offline.

## Memory (Phase 3)

Two kinds of memory, split by shape:

- **Short-term** (`memory.py`, `ConversationMemory`): a rolling window of recent
  messages per email thread, injected into every LLM call so references resolve
  ("the report we discussed") and already-captured promises aren't re-extracted.
- **Long-term**: the commitments SQLite DB itself — already structured,
  queryable, and persistent, reached via the MCP server. There is **no vector
  store / RAG**: v1 has no unstructured-retrieval need, so adding one would be
  dead weight. `ConversationMemory.search()` is a documented extension point if
  a semantic need ("did I ever discuss X with this person") ever appears.

**Fulfillment detection** ties the two together. On each email, the agent pulls
the sender's open commitments from long-term memory and asks the model whether
the message completes any of them ("I've just sent the Q3 report" → close it).
This runs *regardless of classification* — a "done" email doesn't look
actionable but still closes a commitment. Marking fulfilled is an internal,
reversible state change, so it's applied automatically and logged; only
*outward* actions (drafts) require confirmation.
- **`db.py` has no MCP dependency.** It is plain functions over `sqlite3`, so it
  is directly unit-testable and reusable — the seam that makes the Phase 4 eval
  harness cheap.

## Web UI (FastAPI + React)

A dashboard for the tracker, deployable to a free public link. It's **another
client of the same `db.py` and `llm.py`** — the HTTP API reuses those layers
rather than duplicating logic (the reason `db.py` was kept transport-agnostic).

- `web/api.py` — FastAPI JSON API (`/api/stats`, `/api/commitments`,
  `/api/process`, `/api/check-overdue`, `.../fulfill`). Seeds a few demo
  commitments on first start; falls back to the stub extractor if no key is set.
- `web/static/index.html` — single-file React 18 dashboard (no build step):
  stat cards, filterable commitment list, mark-fulfilled, overdue sweep, and a
  **paste-an-email** panel that extracts commitments live.

Run locally:

```bash
uvicorn web.api:app --reload      # then open http://127.0.0.1:8000
```

### Deploy to a public link (free, ~5 min)

Demo mode only — Gmail stays a local feature (its desktop OAuth can't run on a
headless host, and `gmail.readonly` is a Google-verified scope). The deployed
app works for anyone with the link via the sample inbox + paste-email box.

1. Push this repo to GitHub (the `render.yaml` and `Procfile` are included).
2. On [render.com](https://render.com): **New → Web Service → connect the repo.**
   Render reads `render.yaml` (build `pip install -r requirements.txt`, start
   `uvicorn web.api:app --host 0.0.0.0 --port $PORT`).
3. Add an environment variable **`GROQ_API_KEY`** = your key.
4. Deploy → you get `https://<name>.onrender.com`.

Notes: the free tier sleeps after inactivity (~40s cold start on first hit);
SQLite is ephemeral so processed commitments reset on restart (the startup seed
keeps the demo populated — point `COMMITMENT_DB_PATH` at a hosted Postgres/Turso
DB for real persistence). The public app uses your Groq key, so anyone with the
link can spend from your free daily quota. Railway, Fly.io, and HF Spaces work
equally well.

### Optional: live Gmail on the deployed app (just you)

The deployed app can *also* let **you** connect Gmail — via a server-side web
OAuth flow (`web/gmail_web.py` + `/api/gmail/*` routes). Because `gmail.readonly`
is a Google restricted scope, an unverified app only admits **test users**, so
this works for you (and anyone you add as a test user), not the public. This is a
**separate OAuth client** from the desktop one the CLI uses.

1. **Create a Web OAuth client**: Google Cloud → Credentials → Create OAuth
   client → **Web application**. Authorized redirect URIs:
   - `https://<your-app>.onrender.com/api/gmail/callback`
   - `http://localhost:8000/api/gmail/callback` (for local testing)
2. Make sure your Gmail is a **test user** on the OAuth consent screen.
3. **On Render**, set env vars:
   - `GMAIL_OAUTH_CLIENT_JSON` = the downloaded Web-client JSON (as one line)
   - `OAUTH_REDIRECT_URI` = `https://<your-app>.onrender.com/api/gmail/callback`
4. Redeploy → open the app → **Connect Gmail** → consent → **Fetch inbox**.

Locally: save the Web-client JSON as `gmail_web_credentials.json` (gitignored) —
or set `GMAIL_OAUTH_CLIENT_JSON` — then `uvicorn web.api:app --reload --port 8000`
and click Connect Gmail. Token caches to `gmail_token.json` (gitignored); on the
ephemeral free host it resets on restart, so you re-consent occasionally.

## Evaluation (Phase 4)

The extractor is a plain function with a pinned `today_utc` (no MCP, no server,
no wall clock), which makes it directly measurable and reproducible. `eval.py`
scores the classifier and extractor against `eval_dataset.json` — 32 labeled
emails spanning clear/explicit/relative deadlines, multi-commitment, reported
speech, the vague "should this count?" cases, and negatives (requests-to-you,
FYIs, fulfillments, declines).

```bash
python eval.py               # live (Groq by default), resumable
python eval.py --dry-run     # validate the harness with the stub, no key
python eval.py --no-classify # extractor only (halves the request count)
```

Predictions are cached to `eval_cache.json` (keyed by model + email id), so a
run is **resumable and never re-spends quota** — the fix for free-tier daily
caps. Re-running is instant and free once every email is scored.

**Headline results** (Groq `llama-3.3-70b-versatile` extractor / `llama-3.1-8b-instant` gate):

| Metric | Result |
|---|---|
| Extraction recall / precision / F1 | 100% / 100% / 100% |
| Deadline resolution (exact ISO)    | 100% (21/21) |
| `contains_commitments` accuracy    | 90.6% |
| Confidence-label accuracy          | 92% |

The harness drove a concrete iteration. The first prompt (`v1`) scored 80.6%
precision: all 6 false positives were negatives the extractor over-read — a
request aimed at *you*, a past/completed action, a decline. Adding explicit
"the reader is never a committer / no past actions / no negations" rules
(`PROMPT_VERSION = "v2-negatives"`) took precision to **100% with no recall
loss**. The prediction cache keys on the prompt version, so that edit
auto-invalidated the old scores — reproducible A/B, no stale numbers.

**The finding worth discussing in an interview:** the two-tier classifier gate
is a **precision/recall lever**. With the tightened extractor it now costs only
recall — gating drops recall 100% → 88% (it suppresses 3 of the 4 vague "I'll
get to it soon" commitments) while precision stays 100% either way. So the
product question sharpens to: *do you track soft intentions at all?* If yes,
extract on everything actionable and let the `confidence` field drive downstream
filtering, rather than gating them out before they're ever seen. The harness
quantifies that tradeoff instead of hand-waving it. See `eval_results.txt`.

## Data-model decisions (Step 2 & Step 7)

- **Deadlines stored twice.** `deadline` holds the *resolved absolute* ISO-8601
  value (source of truth for queries and overdue logic). `deadline_raw` keeps
  the original phrase ("by Friday") for debugging/eval only.
- **UTC internally, always.** Every timestamp (`created_at`, `fulfilled_at`) and
  every resolved `deadline` is UTC ISO-8601. Convert to local time only at
  display. Overdue comparison relies on lexicographic ISO ordering.
- **NULL deadlines are never overdue.** A commitment with no resolvable deadline
  is excluded from `check_overdue` and from `before_date` filtering.
- **Overdue is strict (`deadline < as_of`).** A date-only deadline becomes
  overdue the day *after* it, not on the day itself.
- **Idempotent inserts.** Re-processing the same message won't create
  duplicates: `add_commitment` dedups on `(source_msg_id, text, who_promised)`
  and returns the existing record. There is intentionally *no* hard UNIQUE
  constraint on `source_msg_id` alone, because one message can carry several
  distinct commitments.

## Status lifecycle

```
open ──mark_fulfilled──▶ fulfilled
 │
 └──check_overdue (deadline passed)──▶ overdue
```