Job-Search Copilot

A human-in-the-loop Python tool that finds the right ~40 analyst roles instead of blasting 400 applications, then tailors a truthful one-page resume for each one.

It pulls live postings from 85 employers' public hiring APIs, ranks them with an LLM, and builds a resume PDF using only facts from a verified experience bank. It never submits anything: a person reviews every resume and clicks submit.

Stack: Python 3.12 · REST/JSON APIs · Claude API (Haiku + Sonnet) · MCP server · LaTeX (Tectonic) · httpx · ThreadPoolExecutor

How it works

flowchart LR
    A["companies.json<br/>85 employers, 6 ATS types"] --> B["discover_jobs.py<br/>fetch · filter · dedupe"]
    B --> C[("job_feed.csv<br/>job_details.json")]
    C --> D["score_jobs.py<br/>Claude Haiku · cached"]
    D --> E["Ranked shortlist<br/>+ skill-gap report"]
    E --> F["Tailoring<br/>Claude selects content"]
    G["experience_bank.json<br/>only source of facts"] --> F
    H["resume_rules.py<br/>honesty guardrails"] --> F
    F --> I["render_resume.py<br/>JSON → LaTeX → 1-page PDF"]
    I --> J["tracker.py<br/>applications.csv"]
    M{{"MCP server · 13 tools<br/>runs it all from chat"}} -.-> B & D & F & I & J

All of it runs from a chat window through an MCP server in the Claude desktop app: "refresh my feed" → "show my top 10" → "tailor my resume to row 3" → "log that I applied".

Related MCP server: JobHound

Key results

• 85 employers across 6 applicant-tracking systems, searched nationwide, using only their public job-board APIs.

• Full refresh cut from 20+ minutes to about 1 minute after the employer list doubled: companies are fetched in parallel (8 workers), paging stops once relevance-sorted results stop matching, and only server errors (5xx) are retried. A mock benchmark went from 142 s to 7 s with identical results.

• An AI ranking measured against a hand-labeled answer key: agreement with my labels rose from 33% to 63% on 30 tuning jobs and from 50% to 70% on 10 held-out jobs; 7 of the AI's top 10 are now roles I'd actually apply to. See Evaluation.

• Cheap at scale: the full feed of 560 jobs was rescored for $2.73, scores are cached, and clearly overseas jobs are filtered in code and never sent to the API.

• 13-tool MCP server covering the feed, job descriptions, tailoring, rendering, skill gaps, screening answers and application tracking.

Design decisions

1. Human in the loop. The tool never submits applications. It speeds up finding and tailoring; a person makes every decision.

2. No invented facts. The experience bank is the single source of truth. The model may select, reorder and lightly rephrase, but never add tools, numbers or experiences. The same rules file (resume_rules.py) guards both tailoring paths, so they can't drift apart.

3. Content separate from layout. The LLM only produces JSON content; deterministic code owns the layout, LaTeX escaping, file names and the one-page limit.

4. Let the LLM judge fuzzy things and extract facts; let code decide rules. Skill fit is a judgment call, so the model scores it. Location and experience thresholds are rules, so code applies them after scoring: the model reports "3 years required", and code enforces the cap. (The model was inconsistent about both, which is how this split came about.)

5. Sanctioned APIs only. Public ATS JSON endpoints, polite parallelism (one request stream per employer), no LinkedIn or Indeed scraping.

6. Cheap model for bulk work, stronger model for writing. Haiku scores hundreds of jobs; Sonnet (or Claude in the app) writes resumes.

Evaluation: measuring the AI fit scorer

The copilot ranks every job with an LLM (Claude Haiku) that scores fit from 0 to 100. To find out whether those scores are right, and to improve them with evidence instead of guesswork, I built an eval: a fixed set of real postings, a hand-made answer key, and a grader that compares the two.

Results

Method

1. Frozen test set. 40 real postings from the live feed, full text saved so the test never changes as postings expire. Over-sampled high and borderline scores plus known-tricky types (SAP roles, finance roles, "3+ years required", senior-sounding titles), split 30 tune / 10 holdout with a fixed seed.

2. Blind labels. I labeled every posting Strong / Maybe / Skip against a written rubric, without seeing the AI's scores. When the AI disagreed, I checked the posting text before changing anything; one disagreement turned out to be my labeling error.

3. Grader. eval_scorer.py calls the production scoring function directly (no cache, writes nothing), three times per job, and reports agreement, a confusion table, the two costly errors, precision@10, consistency and cost. Every run is logged to eval/results.csv.

4. One change per run, each kept or rejected on the numbers:

What I learned

• The biggest gain was a data fix, not a prompt fix. No prompt can score requirements the model never receives.

• LLMs read rules better than they apply them. The model wrote "lacks required 3+ years" and still scored the job as a Maybe. Moving the rule into code (the model extracts the facts, code applies the threshold) made it deterministic and unit-testable, the same pattern the project already uses for location.

• Conflicting instructions produce compromises. When the score bands and an experience rule disagreed, five jobs landed on exactly 52: the bottom of the band, between the two instructions.

• Stricter isn't always better. A narrower definition raised precision@10 to 8/10 but found only 1 of 11 good jobs, so it was rejected.

Limits: 30 tune cases gives roughly ±17 points of margin, so differences of one or two cases are noise; all labels come from one labeler; the holdout set has now been used, so the next round needs freshly labeled postings; the eval covers the fit scorer, not resume tailoring. The test postings and my labels are kept out of the repo (.gitignore); the scripts, the rubric and the summary results are included.

Project structure

Setup

Tested on macOS with Python 3.12.

git clone https://github.com/miguel-jaimes/job-search-copilot.git
cd job-search-copilot

pip install -r requirements.txt
conda install -c conda-forge tectonic        # LaTeX engine for the PDFs

cp .env.example .env                          # then add your Anthropic API key
cp experience_bank.example.json experience_bank.json   # then replace with your own facts

.env and experience_bank.json are in .gitignore, so your key and personal data stay on your machine.

Usage

python discover_jobs.py --verify      # check every employer's endpoint (1 page each)
python discover_jobs.py --score       # full refresh + AI scores
python score_jobs.py --gaps           # most common missing skills in near-fit jobs
python tailor_resume.py job.txt --company "Acme"   # tailor to a pasted posting
python render_resume.py resumes/X.json             # re-render an edited resume

python eval_scorer.py                 # regression test for the scorer (~$0.40 per run)
python eval_scorer.py --rules eval/prompt_v5.txt --note "what changed"   # test a prompt change

To use it from chat, add the server to the Claude desktop app's config (~/Library/Application Support/Claude/claude_desktop_config.json):

{
  "mcpServers": {
    "job-copilot": {
      "command": "/path/to/python",
      "args": ["/path/to/job-search-copilot/job_copilot_server.py"]
    }
  }
}

What I learned

• Status codes tell you whose problem it is. A 4xx means my request is wrong, so fail fast. A 5xx means their server hiccupped, so retry with backoff. Treating a 502 as "bad URL" was an early bug.

• Normalize early. Six different response formats become one record at the edge, which kept the filters, scoring and MCP tools simple.

• Verify assumptions against real responses. Workday only reports its result total on the first page, and Oracle returns non-JSON without a specific header. Neither showed up in documentation; both showed up in testing.

• Scaling exposes design limits. A sequential loop that was fine for 40 employers took 20+ minutes at 85. Parallelism plus stopping early fixed it.

Limitations and next steps

• Employers on iCIMS, Eightfold, Taleo, SuccessFactors and Avature aren't supported yet; their email job alerts cover the gap.

• companies.json reflects one person's search (entry-level analyst roles, DFW-focused). Edit it for your own targets.

• The scorer reads candidate facts from experience_bank.json, but its prompt in score_jobs.py describes the candidate as a new-grad analytics student. Adjust that wording for other profiles.

• The scorer reads only the first 6,000 characters of a posting, so requirements at the end of very long postings can be missed.

• The model sometimes reads general experience ("2 years with Tableau") as professional experience and caps the score too early. It also leans conservative now: most remaining misses are jobs scored lower than my label.

• No pytest suite yet. Next: pytest with httpx.MockTransport for the fetchers, filters, LaTeX escaping, auto-fit and the experience cap.

License

MIT. See LICENSE.

Built by Miguel Jaimes, B.S. Business Analytics, UT Dallas (Dec 2026) · LinkedIn