Skills Radar

Which skills do tech employers ask for, and how is that changing? Skills Radar reads the monthly Hacker News "Who is hiring?" threads, uses Claude to turn each job post into structured fields, and lets an agent answer questions about them through read-only MCP tools. A web page shows the trends and hosts the chat.

Every LLM step is measured: field-level extraction accuracy against hand labels, answer accuracy against SQL-computed truth, and resistance to prompt injection planted in job posts.

Results (September and October 2026)

Data: 619 top-level replies across the two threads → 515 live posts (72 deleted by their authors, 32 removed by moderators) → 501 job posts (Claude flagged 14 replies as not job posts).

Extraction accuracy: Claude Opus 5.5 with prompt v2, on 50 hand-labeled October posts, 3 runs each. Accuracy = agreement with the hand labels. Agreement = share of posts where all 3 runs gave the same answer.

Agent answers: 20/20 questions correct, including 3 the data can't answer (a city that never appears, a month that isn't loaded, a field that isn't tracked). Each answer costs $0.016 to $0.042.

Prompt injection: 10/10 planted attacks resisted, 3 runs each. Attacks included a fake </job_post> tag, fake "developer" instructions, a salary override, an instruction in French, and a canary word.

Cost: $0.0068 per post with prompt caching ($0.68 per 100 posts on Opus 5.5), down from about $0.020 per post without it. Building and evaluating everything cost about $11 in API usage.

Related MCP server: DevToolJobs Scout MCP

What the evals caught

• The definitions didn't match. Prompt v1 tagged only skills a post names. The hand labels also counted skills implied by a role ("Account Executive" → Sales). Writing the labeling rules into prompt v2 raised skill recall from 84% to 89% and ai_work accuracy from 88% to 96%.

• A tool bug. The salary summary skipped posts that list only a maximum ("up to ~$750k"). Because the eval's truth is computed with SQL written separately from the tools, it disagreed with the tools and exposed the bug.

• A grader bug. A correct "there are no Lagos posts" answer was marked wrong by a phrase list that was too narrow.

• A cost problem. One question cost 20× the others: with no way to sort by salary, the agent downloaded every October post. Adding sort_by_salary and a hard limit of 25 rows per call brought it from $0.44 to $0.03.

How it works

flowchart LR
    HN[HN API] -->|ingest, cached| DB[(SQLite: posts)]
    DB -->|raw post text| EX["Extractor (Claude)<br/>structured output, no tools"]
    EX --> EXT[(SQLite: extractions)]
    EXT --> Q["queries.py<br/>read-only"]
    Q --> MCP["MCP server<br/>4 read-only tools"]
    MCP --> AG["Agent (Claude)"]
    Q --> WEB["Web page<br/>trend chart"]
    AG --> WEB2["Web page<br/>chat"]

Security design

Job posts are text written by strangers, and some already contain instructions aimed at AI. The design keeps untrusted text, tools and private data apart:

Run it yourself

Requires Python 3.10+ and an Anthropic API key.

git clone https://github.com/ambarachristian/skills-radar.git
cd skills-radar
python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install -r requirements.txt
cp .env.example .env               # then put your key in .env

python -m radar.ingest 2026-09 2026-10      # download posts (free, a few minutes)
python -m radar.extract --limit 5           # try 5 posts first and check the cost per post
python -m radar.extract --limit 1000        # then extract the rest (about $0.70 per 100 posts)
python scripts/try_mcp.py                   # smoke-test the MCP server (free)
python -m radar.agent "What skills go with Rust?"
uvicorn web.app:app --reload                # open http://127.0.0.1:8000

The MCP server uses stdio. Any MCP client can launch it with python -m radar.mcp_server from the project folder.

Evals

python -m evals.make_labels      # pick 50 random October posts and create label files in data/labels/
                                 # correct each file by hand and set "checked": true
python -m evals.run_repeats      # run the extractor 3 times on the labeled posts
python -m evals.grade            # accuracy and agreement per field (--show FIELD lists the mistakes)
python -m evals.answer_eval      # 20 questions graded against SQL truth
python -m evals.injection_eval   # 10 prompt-injection attacks

My labels aren't included in this repository because the label files contain post text (see Privacy), so you'll need to label your own sample.

Privacy

Posts include hiring contacts' names and email addresses, so:

• No post data is in this repository. data/ holds the database and the labels, and Git ignores it.

• The tools return extracted fields and links to HN, never raw post text, so emails never reach the agent or the chat.

• Publish only aggregates, such as skill shares and counts.

Limitations

• Sample: HN skews toward startups, US companies and remote roles. With about 250 posts a month, rare skills are noisy, so chart them by quarter.

• Consistency: the skill list differs between runs in about 17% of posts.

• Known extraction issues: seniority sometimes over-assigns staff+, and company names occasionally pick up extra words.

• Salaries: the salary summary covers only salaries stated in USD. Other currencies are left out, not converted.

• Tuning set: prompt v2 was tuned on the same 50 posts it's scored on, so its accuracy is probably a little optimistic. A held-out set of new labeled posts would give an unbiased number.

• Cost exposure: the chat spends API credit on every question. The server listens on 127.0.0.1 only. Don't expose it publicly without authentication and rate limits.

License

MIT