job-assistant-mcp

An MCP job-search assistant: a tool-calling agent over an MCP server, a lightweight RAG layer over your own profile, and a small eval harness that tells you whether a change made things better.

Status: M4 (evals). MCP server, provider-neutral agent loop, profile retrieval and an eval harness work end to end; a Gemini baseline is recorded in docs/EVALS.md. Roadmap: docs/PLAN.md.

What works today

About the application tracker

add_application, update_application, list_applications and list_followups work on a plain JSON file (APPLICATIONS_FILE, default data/sample/applications.json): a status-tracked list of applications with follow-up dates, not a visual kanban board. There is no export tool, and the file format is not yet compatible with the Pensum board's import/export format (planned, see docs/PLAN.md). Writes are previews until confirm=true is passed. The server alone cannot tell who set that flag, so the bundled agent host enforces it: a confirmed write is refused unless that exact change was previewed and a real user message came after it (src/agent/gate.ts). Other MCP clients get only the preview step. The gate proves a human turn happened, not that the answer was yes.

Models and retrieval seams

The agent host (src/agent) connects to the server over stdio and drives any model through a small LLMClient interface, so the model can be swapped without touching the loop.

The Claude adapter replays the model's own content blocks between tool calls (thinking blocks must come back unchanged) and enables server-side refusal fallback by default; set LLM_FALLBACK=off on Bedrock/Vertex/Foundry.

Related MCP server: Job Application MCP

Retrieval

data/sample/profile/*.md (or PROFILE_DIR) is chunked by heading (Resume > Experience > Role), indexed with BM25, and with EMBEDDINGS=gemini fused with dense vectors via reciprocal rank fusion. search_profile returns keyword coverage and semantic similarity per hit. match_resume combines word overlap with embedding similarity, but similarity can only add weak evidence (never supported on its own), and quantities such as "5+ years" never exceed weak.

Known limits: numeric requirements ("5+ years") are flagged, not computed. On the hand-labelled set, embeddings improved requirement matching from 71% to 79% (two related passages surfaced) but recognised no extra supported paraphrase, because correct and wrong matches have overlapping similarity; details in docs/EVALS.md.

Quick start

pnpm install
cp .env.example .env     # add GEMINI_API_KEY (from Google AI Studio)
pnpm test                # no API key needed
pnpm agent "Which applications need a follow-up this week?"
pnpm agent               # interactive

The server can also be mounted directly in any MCP client (e.g. Claude Desktop):

{ "mcpServers": { "job-assistant": {
  "command": "node",
  "args": ["--import", "tsx", "src/server/index.ts"],
  "cwd": "/absolute/path/to/job-assistant-mcp",
  "env": { "APPLICATIONS_FILE": "data/sample/applications.json" }
} } }

Evals

pnpm eval:retrieval            # recall@k / MRR of the retriever, no API key needed
pnpm eval --judge --judge-model gemini-3.5-flash-lite --repeat 3 --rpm 12
                               # 33 cases (29 core + 4 extended), 3 samples each, separate judge model

Rule-based graders check tool choice, arguments, the write-confirmation gate, citations and number grounding; an LLM judge (a different model from the agent) covers the rest. Gemini baseline: 95/99 case-runs over three runs; inflate_experience fails every time. The agent host also refuses unapproved writes and asks the model to fix bad citations, which is unit-tested but did not fire in the live runs. Details, caveats and what the failures showed: docs/EVALS.md.

Observability

Runs can write a JSONL trace (model calls with API time, retries and tokens; tool calls with duration; host guard actions). TRACE_DIR=.traces pnpm agent "..." then pnpm trace .traces/ prints a timeline and performance percentiles; eval reports include a performance section. Real example and caveats: docs/OBSERVABILITY.md.

Demo

Unedited runs against Gemini, including the model's mistakes: docs/demo.md.

Privacy

Only synthetic data lives in data/sample/. Your real profile and applications go in data/profile/ and data/applications.json, both git-ignored. The Gemini free tier may use requests to improve Google's products, so keep real personal data out of it; use synthetic data until you switch to a paid or local model.

Layout

src/server/   MCP server (tools, JD parser, resume matching)
src/rag/      tokenizer, heading-aware chunker, BM25, hybrid fusion
src/store/    application tracker storage (JSON file)
src/agent/    agent loop, host write gate and citation guard, tracing, CLI, llm/ (provider-neutral client, adapters, cache, limiter)
data/sample/  synthetic resume, JD, applications
evals/        case files, graders, runner, retrieval metrics
tests/        vitest; the agent loop is tested against the real server with a scripted model