job-assistant-mcp

An MCP job-search assistant: a tool-calling agent over an MCP server, a lightweight RAG layer over your own profile, and a small eval harness that tells you whether a change made things better.

Status: 01–08 implemented: local JD-to-board app, structured evidence reports, bounded claim repair, exact human approval, transactional/idempotent storage and correlated redacted traces. Run the app · Implementation and checks · Next steps. Live scores in docs/EVALS.md are historical; no fresh live improvement claim is made for this upgrade.

What works today

About the application tracker

pnpm app opens a local visual kanban with JD analysis, source passages and resume suggestions. The default mode is an offline source-extract demo; APP_MODE=live pnpm app enables model drafting and verification. See docs/APP.md for the full walkthrough.

The tracker uses Node 24's built-in SQLite. APPLICATIONS_FILE ending in .json is imported once into a sibling .json.sqlite; the source JSON is preserved. Application changes and operation outcomes commit together, so a retried confirmed write returns the original result. Updates detect changes since preview using integer versions.

Write tools preview first and require the same operation id and exact payload on confirmation. The bundled host requires explicit approval after the preview (CLI: yes for one pending change, or /approve <operationId>; web: a dedicated confirm button). Cancellation, ambiguous replies, expired previews and changed payloads are rejected. Other MCP clients must obtain human approval themselves; the server verifies operation integrity, not human intent. Pensum import/export compatibility remains future work. Storage details.

Models and retrieval seams

The agent host (src/agent) connects to the server over stdio and drives any model through a small LLMClient interface, so the model can be swapped without touching the loop.

The Claude adapter replays the model's own content blocks between tool calls (thinking blocks must come back unchanged) and enables server-side refusal fallback by default; set LLM_FALLBACK=off on Bedrock/Vertex/Foundry.

Related MCP server: Job Assistant MCP

Retrieval

data/sample/profile/*.md (or PROFILE_DIR) is chunked by heading (Resume > Experience > Role), indexed with BM25, and with EMBEDDINGS=gemini fused with dense vectors via reciprocal rank fusion. search_profile returns keyword coverage and semantic similarity per hit. match_resume combines word overlap with embedding similarity, but similarity can only add weak evidence (never supported on its own), and quantities such as "5+ years" never exceed weak.

Known limits: numeric requirements ("5+ years") are flagged, not computed. On the hand-labelled set, embeddings improved requirement matching from 71% to 79% (two related passages surfaced) but recognised no extra supported paraphrase, because correct and wrong matches have overlapping similarity; details in docs/EVALS.md.

Personal Markdown organization, source/correction records and separate offline audits are described in Personal profile corpus. Custom eval paths keep personal labels separate from the synthetic baseline.

Quick start

node --version           # Node 24+ required (built-in SQLite)
pnpm install
pnpm app                 # offline local app, no key needed
cp .env.example .env     # add GEMINI_API_KEY (from Google AI Studio)
pnpm test                # no API key needed
pnpm agent "Which applications need a follow-up this week?"
pnpm agent               # interactive

The server can also be mounted directly in any MCP client (e.g. Claude Desktop):

{ "mcpServers": { "job-assistant": {
  "command": "node",
  "args": ["--import", "tsx", "src/server/index.ts"],
  "cwd": "/absolute/path/to/job-assistant-mcp",
  "env": { "APPLICATIONS_FILE": "data/sample/applications.json" }
} } }

Evals

pnpm eval:retrieval            # recall@k / MRR of the retriever, no API key needed
pnpm eval --judge --judge-model gemini-3.5-flash-lite --repeat 3 --rpm 12
                               # 33 cases (29 core + 4 extended), 3 samples each, separate judge model

Rule-based graders check tool choice, arguments, the write-confirmation gate, citations and number grounding; an LLM judge (a different model from the agent) covers the rest. Historical Gemini baseline: 95/99 case-runs over three runs; inflate_experience failed every time before the current runtime checks. The historical live runs preceded the 01–08 upgrade. Current host guards are covered by deterministic regression and fault-injection tests; no new live suite has been run for this upgrade. Details, caveats and what the failures showed: docs/EVALS.md.

Observability

Runs can write a JSONL trace (model calls with API time, retries and tokens; tool calls with duration; host guard actions). TRACE_DIR=.traces pnpm agent "..." then pnpm trace .traces/ prints a timeline and performance percentiles; eval reports include a performance section. Real example and caveats: docs/OBSERVABILITY.md.

Demo

Unedited runs against Gemini, including the model's mistakes: docs/demo.md.

Privacy

Only synthetic data lives in data/sample/. Your real profile and applications go in data/profile/ and data/applications.json, both git-ignored. The Gemini free tier may use requests to improve Google's products, so keep real personal data out of it; use synthetic data until you switch to a paid or local model.

Layout

src/server/   MCP server (tools, JD parser, resume matching)
src/rag/      tokenizer, heading-aware chunker, BM25, hybrid fusion
src/store/    transactional SQLite tracker, durable preview and idempotency ledger
src/app/      local HTTP host and visual JD-to-board workbench
src/agent/    agent loop, host write gate and citation guard, tracing, CLI, llm/ (provider-neutral client, adapters, cache, limiter)
data/sample/  synthetic resume, JD, applications
evals/        case files, graders, runner, retrieval metrics
tests/        vitest; the agent loop is tested against the real server with a scripted model

JD parsing

parse_jd and match_resume share a Gemini structured-output parser and validated cache. Only JD text is sent for extraction; personal Markdown chunking and source citations are unchanged. JD_PARSER_MODE=auto (default) uses GEMINI_API_KEY; a missing key, API error, invalid output or 10s timeout returns the existing rule-based result. JD_PARSER_MODE=rules forces local parsing. The workbench's APP_MODE=extractive always uses local rules. Use APP_MODE=live for Gemini parsing and model drafting. Parser model and budget are independent of the agent: JD_PARSER_MODEL and JD_PARSER_TIMEOUT_MS (1–12000ms). The stdio server also loads .env without replacing explicit environment variables.

No automatic model switch or provider retry is performed. Free access depends on the configured Google project's quota/billing tier. JD_PARSER_CACHE_DIR=off disables disk caching; otherwise validated JD extracts are stored under .cache/parsing/jd. Clear that directory to remove stored extracts; restart to clear memory. Rule fallbacks are shared in memory for 30s, then API extraction can recover. Tool results include parsing metadata; the workbench shows the method and fallback reason. Traces record jd_parse events with API counts/tokens separately from agent calls. Exact extracts are checked against source text, but source presence alone cannot prove correct interpretation of negation or mandatory/preferred wording.

Run pnpm eval:jd for the labelled synthetic rule baseline and pnpm eval:jd --live for a live comparison. The latter explicitly calls the configured Gemini API; automatic tests use fake SDK responses only. See the implementation plan for scope and validation details.

Requirement-level matching: APP_MODE=live MATCH_MODE=agent pnpm app uses the configured Gemini/Anthropic model to assess evidence, optionally search for missing evidence, and show per-requirement verdicts with source quotes. MATCH_MODE=assessment disables supplementary search; MATCH_MODE=retrieval preserves the retrieval baseline. Offline remains model-free. See matching configuration and evaluation; pnpm eval:matching runs the synthetic offline baseline without API calls.