# GitHub MCP Server (Human-in-the-Loop)

A Model Context Protocol (MCP) server that securely connects Large Language Models (LLMs) to the GitHub API.

Instead of giving AI unchecked access to modify repositories, this server implements a strict **Human-in-the-Loop (HITL)** architecture. It uses Zod validation and Discriminated Unions to ensure the AI can safely fetch data, but forces it to "stage" any destructive write actions (like creating or closing issues) for explicit human approval before execution.

## Key Features

* **Safe Data Retrieval:** Securely fetch recent repositories, open issues, and pull requests.
* **Staged Write Actions:** The AI prepares issues and closures, returning a staging ID instead of directly mutating the repository.
* **Strict Validation:** Built with Zod to enforce required parameters, preventing the AI from hallucinating missing data.

---

## Evaluation & Testing

To ensure the system's routing logic was safe and wouldn't hallucinate destructive commands, I built a custom evaluation script to test the model (Gemini 2.0 Flash, Temperature 0.0) against 50 different prompts ranging from simple commands to ambiguous edge cases.

### Final Accuracy: 90.0% (45 / 50)

The initial raw score of the automated test was 80% (40/50). However, after auditing the error logs, I found that the AI was actually making the correct, safe choice in 5 of the "failed" edge cases.

My test labels assumed the AI should call a tool even if the user didn't provide enough information, or asked for something a tool wasn't built to do. Instead, my strict Zod schemas and tool definitions forced the AI to safely decline the request rather than inventing fake data or overstepping its own capabilities:

| ID | Prompt | Why refusing was the correct behavior |
|---|---|---|
| 36 | "Get the issues for just 'test'." | `'test'` is not a valid `owner/repo` string. |
| 38 | "Close issue 'fifty' in devanshu23bcs030/test." | The schema strictly requires a number. Refusing to guess that "fifty" meant `50` prevented a potential type error. |
| 39 | "Get the closed issues for facebook/react." | `get_issues` is hardcoded to fetch only `state=open` issues by design. The AI correctly recognized it had no tool capable of fulfilling this request. |
| 42 | "Stage an issue in fakeUser/fakeRepo." | Missing `title` and `body`. Declining is safer than fabricating placeholder content. |
| 45 | "Create an issue in devanshu23bcs030/test." | Same as above — mandatory fields were missing. |

Excluding these 5 intentional schema defenses, the true routing accuracy of the system is **45/50 (90%)**.

### Category Breakdown

| Category | Passed | Accuracy |
|---|---|---|
| simple | 15 / 15 | 100% |
| ambiguous | 9 / 10 | 90% |
| write_action | 8 / 10 | 80% |
| edge_case | 8 / 10 | 80% |
| unrelated | 5 / 5 | 100% *(Zero hallucinations on out-of-scope prompts)* |