Naukri.com Domain Support Agent Capstone

Completed track: Naukri.com (Recruitment & HR).

This repository contains one complete, locally runnable capstone. It will use MOCK_LLM by default, require no API keys, and include reproducible evidence for all 16 rubric tasks.

Task 1 dataset design

dataset.py uses Python's seeded pseudo-random generator with seed 20260827 to produce exactly 50 job applications. Category weights are Software Engineer: 0.32, Data Analyst: 0.22, Product Manager: 0.15, HR Executive: 0.13, and Sales Associate: 0.18; status weights are Applied: 0.30, Screening: 0.25, Interview Scheduled: 0.20, Offered: 0.12, and Rejected: 0.13; priority-review probability is 0.20.

Expected annual salary is generated from ₹350,000 to ₹2,500,000 because this covers plausible entry-level HR/sales roles through experienced software, data, and product candidates without pretending the values are real applicant data. days_since_created is an integer from 0 to 30. Run python dataset.py to generate, validate, and print the required distribution report.

With the stated seed and weights, the validated report is:

Related MCP server: JobScannerMCP

Repository layout

src/            Application code: RAG, graph, guardrails, API, MCP, resilience
data/           Generated deterministic dataset and evaluation inputs
knowledge_base/ Original policy documents
tests/          Automated checks
scripts/        Repeatable demonstrations and evaluation runners
transcripts/    Text evidence for the 16 required tasks
logs/           Local JSONL request logs (not committed)

Implementation phases

1. Foundation: dataset generator, validator, knowledge base.

2. RAG: two chunking strategies, embeddings, Chroma, grounded MOCK_LLM answers, retrieval evaluation.

3. Agent: LangGraph routing, record lookup, JSON memory, response schema, guardrails.

4. Service and evaluation: FastAPI, safe JSONL logging, 15-query RAG-triad evaluation.

5. Reliability and interoperability: FastMCP, SQLite checkpoints, retries, node/global timeouts.

6. Submission evidence: transcripts, acceptance checks, final README.

Phase 1 status

• Task 1 — deterministic 50-record dataset and validation

• Task 2 — twelve original Naukri policy documents

• Task 3 — two ChromaDB chunking/indexing pipelines

• Task 4 — calibrated grounded generation with MOCK_LLM

• Task 5 — chunking comparison with Precision@3 and Recall@3

• Tasks 6–10 — LangGraph agent, tools, memory, schema, and guardrails

• Tasks 11–13 — FastAPI, safe logging, and RAG-triad evaluation

• Tasks 14–16 — MCP, SQLite checkpoints, retries, and timeouts

Task 4 similarity calibration

Using the sentence-based collection, measured top-1 cosine similarities were 0.5757, 0.6645, and 0.4158 for three in-scope queries, versus 0.0759 and 0.0899 for two deliberately out-of-scope queries. The selected threshold is 0.2529, the midpoint between the observed in-scope minimum (0.4158) and out-of-scope maximum (0.0899); therefore the MOCK_LLM returns its deterministic “I don't know” fallback below this evidence-based value.

Task 5 chunking recommendation

The fixed-overlap collection achieved mean Precision@3 of 0.33 and mean Recall@3 of 1.00. The sentence-based collection achieved mean Precision@3 of 0.43 and mean Recall@3 of 1.00. I would deploy sentence-based chunking because it preserved the same complete recall while returning fewer duplicate chunks from the same source document, improving document-level precision; the full per-query arithmetic is in transcripts/task5_chunking_evaluation.md.

Task 6 escalation score

The lookup tool uses:

escalation_score = 0.65 * flagged_priority_review + 0.35 * (days_since_created / 30)

with an escalation recommendation at 0.75 or above. In the seeded 50-record dataset, the highest score among all unflagged records is 0.35, while 8 of the 9 flagged records score at least 0.75; this makes 0.75 a data-driven boundary that escalates older priority-flagged records without escalating any unflagged record.

Tasks 7–10 agent evidence

The LangGraph agent has four nodes (guardrail, route, status_tool, rag_tool) and a conditional edge from route to the status or RAG path. transcripts/task7_to_10_agent_demo.json demonstrates both routes, phone masking, prompt-injection blocking, groundedness fallback, and persisted versus fresh memory. AgentResponse is a Pydantic schema and every response is constructed through that schema.

Task 11–12 API and logging

src/api.py exposes POST /ask and POST /add-document. Each completed endpoint request writes one JSONL entry containing a UUID trace ID, masked request text, route, and duration; the phone masker is applied before the request is written to disk.

Task 14 MCP

Run python -m src.mcp_server in one VS Code terminal. The private HTTP MCP endpoint is http://127.0.0.1:8001/mcp. In a second terminal, run python -m scripts.mcp_client; it calls the lookup tool for JOB-1001 and JOB-1002.

Task 15–16 resilience

scripts/demo_resilience.py creates a clean checkpoints.sqlite, uses a four-node LangGraph with SQLite checkpointing, interrupts before tool, and resumes the same thread without re-executing guardrail or route. The same script also demonstrates bounded exponential backoff (max_attempts=4, initial delay 0.05 s, maximum delay 0.2 s, 0.01 s jitter), a clean per-node timeout, and a clean global timeout. Evidence is stored in transcripts/task15_to_16_resilience.json.

Task 13 RAG-triad evaluation

The deterministic MOCK_LLM judge prompt is implemented in src/mock_llm.py; it evaluated 15 queries: one per required policy topic and three out-of-scope questions. Average context relevance was 0.80, groundedness was 1.00, and answer relevance was 1.00. The full per-query scores and answers are in transcripts/task13_rag_triad.json.