# Naukri.com Domain Support Agent Capstone

> Completed track: **Naukri.com (Recruitment & HR)**.

This repository contains one complete, locally runnable capstone. It will use `MOCK_LLM` by default, require no API keys, and include reproducible evidence for all 16 rubric tasks.

## Task 1 dataset design

`dataset.py` uses Python's seeded pseudo-random generator with **seed `20260827`** to produce exactly **50** job applications. Category weights are `Software Engineer: 0.32`, `Data Analyst: 0.22`, `Product Manager: 0.15`, `HR Executive: 0.13`, and `Sales Associate: 0.18`; status weights are `Applied: 0.30`, `Screening: 0.25`, `Interview Scheduled: 0.20`, `Offered: 0.12`, and `Rejected: 0.13`; priority-review probability is `0.20`.

Expected annual salary is generated from **₹350,000 to ₹2,500,000** because this covers plausible entry-level HR/sales roles through experienced software, data, and product candidates without pretending the values are real applicant data. `days_since_created` is an integer from 0 to 30. Run `python dataset.py` to generate, validate, and print the required distribution report.

With the stated seed and weights, the validated report is:

| Measure | Result |
|---|---:|
| Records | 50 |
| Software Engineer | 18 |
| Data Analyst | 11 |
| Product Manager | 9 |
| HR Executive | 6 |
| Sales Associate | 6 |
| Applied | 18 |
| Screening | 6 |
| Interview Scheduled | 10 |
| Offered | 6 |
| Rejected | 10 |
| Flagged for priority review | 9/50 (18.0%) |

## Repository layout

```text
src/            Application code: RAG, graph, guardrails, API, MCP, resilience
data/           Generated deterministic dataset and evaluation inputs
knowledge_base/ Original policy documents
tests/          Automated checks
scripts/        Repeatable demonstrations and evaluation runners
transcripts/    Text evidence for the 16 required tasks
logs/           Local JSONL request logs (not committed)
```

## Implementation phases

1. Foundation: dataset generator, validator, knowledge base.
2. RAG: two chunking strategies, embeddings, Chroma, grounded `MOCK_LLM` answers, retrieval evaluation.
3. Agent: LangGraph routing, record lookup, JSON memory, response schema, guardrails.
4. Service and evaluation: FastAPI, safe JSONL logging, 15-query RAG-triad evaluation.
5. Reliability and interoperability: FastMCP, SQLite checkpoints, retries, node/global timeouts.
6. Submission evidence: transcripts, acceptance checks, final README.

## Phase 1 status

- [x] Task 1 — deterministic 50-record dataset and validation
- [x] Task 2 — twelve original Naukri policy documents
- [x] Task 3 — two ChromaDB chunking/indexing pipelines
- [x] Task 4 — calibrated grounded generation with `MOCK_LLM`
- [x] Task 5 — chunking comparison with Precision@3 and Recall@3
- [x] Tasks 6–10 — LangGraph agent, tools, memory, schema, and guardrails
- [x] Tasks 11–13 — FastAPI, safe logging, and RAG-triad evaluation
- [x] Tasks 14–16 — MCP, SQLite checkpoints, retries, and timeouts

## Task 4 similarity calibration

Using the sentence-based collection, measured top-1 cosine similarities were `0.5757`, `0.6645`, and `0.4158` for three in-scope queries, versus `0.0759` and `0.0899` for two deliberately out-of-scope queries. The selected threshold is **0.2529**, the midpoint between the observed in-scope minimum (`0.4158`) and out-of-scope maximum (`0.0899`); therefore the `MOCK_LLM` returns its deterministic “I don't know” fallback below this evidence-based value.

## Task 5 chunking recommendation

The fixed-overlap collection achieved mean Precision@3 of **0.33** and mean Recall@3 of **1.00**. The sentence-based collection achieved mean Precision@3 of **0.43** and mean Recall@3 of **1.00**. I would deploy sentence-based chunking because it preserved the same complete recall while returning fewer duplicate chunks from the same source document, improving document-level precision; the full per-query arithmetic is in `transcripts/task5_chunking_evaluation.md`.



## Task 6 escalation score

The lookup tool uses:

`escalation_score = 0.65 * flagged_priority_review + 0.35 * (days_since_created / 30)`

with an escalation recommendation at **0.75 or above**. In the seeded 50-record dataset, the highest score among all unflagged records is **0.35**, while **8 of the 9 flagged records** score at least 0.75; this makes 0.75 a data-driven boundary that escalates older priority-flagged records without escalating any unflagged record.

## Tasks 7–10 agent evidence

The LangGraph agent has four nodes (`guardrail`, `route`, `status_tool`, `rag_tool`) and a conditional edge from `route` to the status or RAG path. `transcripts/task7_to_10_agent_demo.json` demonstrates both routes, phone masking, prompt-injection blocking, groundedness fallback, and persisted versus fresh memory. `AgentResponse` is a Pydantic schema and every response is constructed through that schema.

## Task 11–12 API and logging

`src/api.py` exposes `POST /ask` and `POST /add-document`. Each completed endpoint request writes one JSONL entry containing a UUID trace ID, masked request text, route, and duration; the phone masker is applied before the request is written to disk.

## Task 14 MCP

Run `python -m src.mcp_server` in one VS Code terminal. The private HTTP MCP endpoint is `http://127.0.0.1:8001/mcp`. In a second terminal, run `python -m scripts.mcp_client`; it calls the lookup tool for `JOB-1001` and `JOB-1002`.

## Task 15–16 resilience

`scripts/demo_resilience.py` creates a clean `checkpoints.sqlite`, uses a four-node LangGraph with SQLite checkpointing, interrupts before `tool`, and resumes the same thread without re-executing `guardrail` or `route`. The same script also demonstrates bounded exponential backoff (`max_attempts=4`, initial delay 0.05 s, maximum delay 0.2 s, 0.01 s jitter), a clean per-node timeout, and a clean global timeout. Evidence is stored in `transcripts/task15_to_16_resilience.json`.

## Task 13 RAG-triad evaluation

The deterministic `MOCK_LLM` judge prompt is implemented in `src/mock_llm.py`; it evaluated 15 queries: one per required policy topic and three out-of-scope questions. Average context relevance was **0.80**, groundedness was **1.00**, and answer relevance was **1.00**. The full per-query scores and answers are in `transcripts/task13_rag_triad.json`.