mcp-hr-knowledge-evals

Enterprise-ready Model Context Protocol (MCP) server in TypeScript exposing structured HR policy tools, RAG-style context retrieval with citations, and an automated evaluation harness that benchmarks accuracy against 20 synthetic HR scenarios.

Docs · Live tool demo · npm

Stack

• Runtime: Node.js >= 18 (ESM), TypeScript strict mode (NodeNext, ES2022)

• MCP: @modelcontextprotocol/sdk — tools, prompts, resources over stdio

• Evals: vitest harness with deterministic lexical scoring; optional Claude-as-judge scoring via @anthropic-ai/sdk

• Validation: zod input schemas on every tool

Related MCP server: HR Assist MCP Server

Quickstart

npm install
npm run build      # compile TypeScript -> dist/
npm start          # run server on stdio
npm run dev        # watch mode via tsx
npm run inspector  # MCP Inspector: npx @modelcontextprotocol/inspector node dist/index.js
npm run eval       # vitest run evals/ — 20 scenario benchmarks
npm run web        # local browser demo on http://localhost:8000

Or install from npm and register with an MCP client:

npm install -g mcp-hr-knowledge-evals

{ "mcpServers": { "hr-knowledge": { "command": "mcp-hr-knowledge-evals" } } }

Optional: copy .env.example to .env and set ANTHROPIC_API_KEY to enable the Claude-as-judge scorer (EVAL_MODEL overrides the default claude-3-5-sonnet-20241022). Without a key, evals use the deterministic scorer — all 20 cases still run and pass.

Tools

query_hr_policy

Keyword/semantic retrieval over the HR handbook. Returns the top matching policy documents with full section text and citation ids.

get_onboarding_milestones

Role-specific onboarding expectations and completion criteria. Roles map to Engineering / Sales / HR / General families.

fetch_manager_coaching_rubric

Synthesizes a 1-on-1 agenda, coaching prompts, and SBI feedback guidance keyed to the scenario topic (deadlines, growth, conflict, workload) and rating.

Prompts & resources

• Prompt: conduct_manager_1on1 — formats employee name, rating, and scenario into a policy-grounded system prompt.

• Resources: every policy exposed at hr://policies/<id> (e.g. hr://policies/parental-leave) as markdown with its citation id.

Eval harness

npm run eval executes 20 synthetic scenarios covering all three tools and reports:

----------------------------------------------------
  HR KNOWLEDGE EVAL BENCHMARK
----------------------------------------------------
  Cases evaluated .......... 20
  Judge .................... deterministic
  Tool selection accuracy .. 100.0%
  Policy retrieval precision 100.0%
  Hallucination rate ....... 0.0%  (goal 0.0%)
----------------------------------------------------

Metrics

• Tool selection accuracy — heuristic router picks the expected tool

• Policy retrieval precision — fraction of ground-truth facts present in output

• Hallucination rate — forbidden content, fabricated citation ids, or missing required citations (goal: 0.0%)

With ANTHROPIC_API_KEY set, each case additionally receives a Claude-as-judge PASS/FAIL faithfulness verdict.

Layout

src/
  index.ts            entry point — connects StdioServerTransport
  server.ts           MCP server: tools, prompts, resources
  tools/hrTools.ts    tool implementations (pure functions)
  data/hrPolicies.ts  mock HR handbook with citation ids
  types/index.ts      interfaces + zod schemas
evals/
  scenarios.ts        20 eval cases with ground truth
  scorer.ts           deterministic scorer + Claude judge + prompt router
  hrScenarioEval.test.ts
scripts/
  smoke-stdio.mjs     JSON-RPC handshake smoke test
web/
  server.mjs          zero-dep HTTP wrapper around dispatchTool (live demo)
  public/index.html   browser playground for the three tools
docs/
  index.html          GitHub Pages docs site

Live demo

web/server.mjs exposes POST /api/tools/<name> and a small static playground, calling the same dispatchTool() the MCP tools/call handler uses — no LLM in the loop. render.yaml deploys it as a free Render web service (health check at /api/health); the UI is served same-origin so no extra config is needed.