commitment-tracker-mcp

An MCP server that tracks commitments — promises people make in conversation ("I'll send the report by Friday") — so they can be listed, closed out, and flagged when they go overdue.

The repo currently covers Phase 1 (core data model + standalone, reusable MCP server over SQLite) and Phase 2 (the agent orchestrator that does LLM classification, extraction, and drafting, and talks to the server over MCP).

Layout

Phase 1 — storage (MCP server side):

Phase 2 — agent (MCP client side):

Related MCP server: Task Manager MCP Server

Setup

pip install -r requirements.txt

Requires Python 3.10+.

Run the server

python server.py          # stdio transport, for local/agent use

It creates commitments.db next to the code on first run. Point it elsewhere with the COMMITMENT_DB_PATH environment variable.

Prove it works

python test_client.py

This spins up the server over stdio against a throwaway database, discovers the tools, and runs a full lifecycle (add → list → fulfill → overdue), printing PASS/FAIL for each assertion. It touches your real commitments.db never.

Run the agent (Phase 2)

The live path uses Google Gemini's free tier (no billing). Get a key at https://aistudio.google.com/apikey, then put it in a .env file:

cp .env.example .env      # then edit .env and paste your key

.env is gitignored — the key never gets committed. The entrypoints call load_dotenv(), so it's picked up automatically:

python orchestrator.py                 # process sample_emails.json
python digest.py                       # daily digest + nudge drafts

# Offline — full pipeline & MCP wiring with a deterministic stub LLM (no key):
python orchestrator.py --dry-run
python digest.py --dry-run             # digest without the LLM summary

(You can also just export GEMINI_API_KEY=... in your shell instead of .env.) If a model isn't enabled on your key, override the IDs: GEMINI_CLASSIFIER_MODEL / GEMINI_EXTRACTOR_MODEL.

Reading from real Gmail

Emails come from a source (sources.py) so the pipeline is ignorant of where mail lives. Default is the JSON fixture; --source gmail reads real mail over read-only OAuth (gmail.readonly — this app can read your mail and nothing else). Gmail's own threadId maps onto thread_id, so thread memory and fulfillment detection work on real conversations.

One-time setup:

1. In Google Cloud Console: create a project → enable the Gmail API → OAuth consent screen (External, add yourself as a test user) → Credentials → Create OAuth client → Desktop app.

2. Download it as credentials.json into this folder (gitignored).

3. Run it — a browser opens for consent once; the token is cached in token.json (gitignored):

   python orchestrator.py --source gmail --query "in:inbox newer_than:7d" --max 10
   python orchestrator.py --source gmail --dry-run    # fetch real mail, stub LLM (no LLM cost)

Gmail API usage is free. --query takes any Gmail search string; --max caps how many messages to pull.

The reactive loop per email: classify (Haiku — cheap gate) → extract commitments (Opus, structured outputs, relative deadlines resolved against an injected UTC date) → store via the MCP add_commitment tool → draft a grounded reply → confirm with the user. Nothing is ever auto-sent — every draft requires explicit approval (--yes exists for demos/CI only).

The digest job is the proactive side, meant for cron/Task Scheduler: it runs check_overdue, lists what's due this week, and drafts nudges for review.

Model choices: gemini-2.5-flash-lite for classification (runs on every email, so cheap/fast, thinking disabled), gemini-2.5-flash for extraction, drafting, and the digest (runs only on emails that pass the gate). All LLM outputs use structured outputs (Gemini response_schema + pydantic → response.parsed), so no free-form JSON parsing anywhere. The provider is isolated to llm.py — the server, MCP client, orchestrator, and digest are provider-agnostic.

Tool surface

The server exposes storage and query tools only:

Design boundaries

• Extraction lives in the agent, not the server. Turning raw messages into structured commitments is an LLM job that lives in llm.py on the orchestrator side, which then calls add_commitment with the resolved fields. Keeping the LLM call out of the server keeps this a pure, client-agnostic interop layer that a CLI, a Slack bot, or the agent can all reuse.

• The orchestrator never imports db.py. All storage goes through mcp_client.py over stdio — proving the MCP boundary is real, not decorative.

• The user confirms every outward action. Drafts (replies, nudges) are always presented for approval; the agent proposes, the human disposes.

• LLM calls are isolated in llm.py. orchestrator.py takes any object with the same methods, so the Phase 4 eval harness can call extract_commitments() directly on a labeled set — and StubLLM runs the whole pipeline offline.

Memory (Phase 3)

Two kinds of memory, split by shape:

• Short-term (memory.py, ConversationMemory): a rolling window of recent messages per email thread, injected into every LLM call so references resolve ("the report we discussed") and already-captured promises aren't re-extracted.

• Long-term: the commitments SQLite DB itself — already structured, queryable, and persistent, reached via the MCP server. There is no vector store / RAG: v1 has no unstructured-retrieval need, so adding one would be dead weight. ConversationMemory.search() is a documented extension point if a semantic need ("did I ever discuss X with this person") ever appears.

Fulfillment detection ties the two together. On each email, the agent pulls the sender's open commitments from long-term memory and asks the model whether the message completes any of them ("I've just sent the Q3 report" → close it). This runs regardless of classification — a "done" email doesn't look actionable but still closes a commitment. Marking fulfilled is an internal, reversible state change, so it's applied automatically and logged; only outward actions (drafts) require confirmation.

• db.py has no MCP dependency. It is plain functions over sqlite3, so it is directly unit-testable and reusable — the seam that makes the Phase 4 eval harness cheap.

Web UI (FastAPI + React)

A dashboard for the tracker, deployable to a free public link. It's another client of the same db.py and llm.py — the HTTP API reuses those layers rather than duplicating logic (the reason db.py was kept transport-agnostic).

• web/api.py — FastAPI JSON API (/api/stats, /api/commitments, /api/process, /api/check-overdue, .../fulfill). Seeds a few demo commitments on first start; falls back to the stub extractor if no key is set.

• web/static/index.html — single-file React 18 dashboard (no build step): stat cards, filterable commitment list, mark-fulfilled, overdue sweep, and a paste-an-email panel that extracts commitments live.

Run locally:

uvicorn web.api:app --reload      # then open http://127.0.0.1:8000

Deploy to a public link (free, ~5 min)

Demo mode only — Gmail stays a local feature (its desktop OAuth can't run on a headless host, and gmail.readonly is a Google-verified scope). The deployed app works for anyone with the link via the sample inbox + paste-email box.

1. Push this repo to GitHub (the render.yaml and Procfile are included).

2. On render.com: New → Web Service → connect the repo. Render reads render.yaml (build pip install -r requirements.txt, start uvicorn web.api:app --host 0.0.0.0 --port $PORT).

3. Add an environment variable GROQ_API_KEY = your key.

4. Deploy → you get https://<name>.onrender.com.

Notes: the free tier sleeps after inactivity (~40s cold start on first hit); SQLite is ephemeral so processed commitments reset on restart (the startup seed keeps the demo populated — point COMMITMENT_DB_PATH at a hosted Postgres/Turso DB for real persistence). The public app uses your Groq key, so anyone with the link can spend from your free daily quota. Railway, Fly.io, and HF Spaces work equally well.

Optional: live Gmail on the deployed app (just you)

The deployed app can also let you connect Gmail — via a server-side web OAuth flow (web/gmail_web.py + /api/gmail/* routes). Because gmail.readonly is a Google restricted scope, an unverified app only admits test users, so this works for you (and anyone you add as a test user), not the public. This is a separate OAuth client from the desktop one the CLI uses.

1. Create a Web OAuth client: Google Cloud → Credentials → Create OAuth client → Web application. Authorized redirect URIs:

   • https://<your-app>.onrender.com/api/gmail/callback

   • http://localhost:8000/api/gmail/callback (for local testing)

2. Make sure your Gmail is a test user on the OAuth consent screen.

3. On Render, set env vars:

   • GMAIL_OAUTH_CLIENT_JSON = the downloaded Web-client JSON (as one line)

   • OAUTH_REDIRECT_URI = https://<your-app>.onrender.com/api/gmail/callback

4. Redeploy → open the app → Connect Gmail → consent → Fetch inbox.

Locally: save the Web-client JSON as gmail_web_credentials.json (gitignored) — or set GMAIL_OAUTH_CLIENT_JSON — then uvicorn web.api:app --reload --port 8000 and click Connect Gmail. Token caches to gmail_token.json (gitignored); on the ephemeral free host it resets on restart, so you re-consent occasionally.

Evaluation (Phase 4)

The extractor is a plain function with a pinned today_utc (no MCP, no server, no wall clock), which makes it directly measurable and reproducible. eval.py scores the classifier and extractor against eval_dataset.json — 32 labeled emails spanning clear/explicit/relative deadlines, multi-commitment, reported speech, the vague "should this count?" cases, and negatives (requests-to-you, FYIs, fulfillments, declines).

python eval.py               # live (Groq by default), resumable
python eval.py --dry-run     # validate the harness with the stub, no key
python eval.py --no-classify # extractor only (halves the request count)

Predictions are cached to eval_cache.json (keyed by model + email id), so a run is resumable and never re-spends quota — the fix for free-tier daily caps. Re-running is instant and free once every email is scored.

Headline results (Groq llama-3.3-70b-versatile extractor / llama-3.1-8b-instant gate):

The harness drove a concrete iteration. The first prompt (v1) scored 80.6% precision: all 6 false positives were negatives the extractor over-read — a request aimed at you, a past/completed action, a decline. Adding explicit "the reader is never a committer / no past actions / no negations" rules (PROMPT_VERSION = "v2-negatives") took precision to 100% with no recall loss. The prediction cache keys on the prompt version, so that edit auto-invalidated the old scores — reproducible A/B, no stale numbers.

The finding worth discussing in an interview: the two-tier classifier gate is a precision/recall lever. With the tightened extractor it now costs only recall — gating drops recall 100% → 88% (it suppresses 3 of the 4 vague "I'll get to it soon" commitments) while precision stays 100% either way. So the product question sharpens to: do you track soft intentions at all? If yes, extract on everything actionable and let the confidence field drive downstream filtering, rather than gating them out before they're ever seen. The harness quantifies that tradeoff instead of hand-waving it. See eval_results.txt.

Data-model decisions (Step 2 & Step 7)

• Deadlines stored twice. deadline holds the resolved absolute ISO-8601 value (source of truth for queries and overdue logic). deadline_raw keeps the original phrase ("by Friday") for debugging/eval only.

• UTC internally, always. Every timestamp (created_at, fulfilled_at) and every resolved deadline is UTC ISO-8601. Convert to local time only at display. Overdue comparison relies on lexicographic ISO ordering.

• NULL deadlines are never overdue. A commitment with no resolvable deadline is excluded from check_overdue and from before_date filtering.

• Overdue is strict (deadline < as_of). A date-only deadline becomes overdue the day after it, not on the day itself.

• Idempotent inserts. Re-processing the same message won't create duplicates: add_commitment dedups on (source_msg_id, text, who_promised) and returns the existing record. There is intentionally no hard UNIQUE constraint on source_msg_id alone, because one message can carry several distinct commitments.

Status lifecycle

open ──mark_fulfilled──▶ fulfilled
 │
 └──check_overdue (deadline passed)──▶ overdue