# GitHub Model Context Protocol (MCP) Server with Tool-Calling Evaluation

[![Python 3.10+](https://img.shields.io/badge/Python-3.10%2B-blue.svg)](https://www.python.org/)
[![Model Context Protocol](https://img.shields.io/badge/Protocol-MCP-purple.svg)](https://modelcontextprotocol.io/)
[![Llama 3.1 70B](https://img.shields.io/badge/LLM-Llama--3.1--70B-orange.svg)](https://build.nvidia.com/)
[![FastMCP](https://img.shields.io/badge/Framework-FastMCP-blueviolet.svg)](https://github.com/jlowin/fastmcp)

> **A production-grade FastMCP GitHub server exposing read and confirm-gated write operations, paired with a 30-case evaluation harness scoring tool selection, argument accuracy, and destructive confirmation safety.**

---

## Executive Summary & Evaluation Matrix

LLM agent tool-calling requires strict safety boundaries. This MCP server enforces **server-level confirmation gating** on all mutating actions: unconfirmed requests return dry-run previews rather than executing mutations on GitHub.

### Benchmark Performance (30 Labeled Scenarios)

Evaluated against a 30-case benchmark testing standard tool calls, underspecified/ambiguous queries, and destructive confirm-gated actions:

| Metric | Pass 1 (Zero-Shot) | **Pass 3 (Optimized)** |
| :--- | :---: | :---: |
| **Tool Selection Accuracy** | 76.7% (23/30) | **90.0% (27/30)** |
| **Argument Accuracy (given correct tool)** | 52.6% (10/19) | **100.0% (19/19)** |
| **Confirmation Flag Accuracy (destructive)** | 50.0% (3/6) | **100.0% (7/7)** |
| — Standard Tool Selection Category | 100.0% (8/8) | **100.0% (8/8)** |
| — Error / Destructive Action Category | 90.9% (10/11) | **90.9% (10/11)** |
| — Ambiguous / Missing-Arg Category | 25.0% (2/8) | **75.0% (6/8)** |

---

## Tool Architecture & Operations

```mermaid
flowchart TD
    A[User Request] --> B[LLM Tool-Calling Engine]
    B --> C{Tool Category}
    
    C -- Read Tool --> D[Execute GitHub API Call Directly]
    D --> E[Return API Payload]
    
    C -- Write / Destructive --> F{confirm == true ?}
    F -- Yes --> G[Execute State Mutation on GitHub]
    G --> H[Return Success / Number / URL]
    
    F -- No --> I[Return Dry-Run Preview of Mutation]
```

### Available Tools

#### Read Operations (Immediate Execution)
- `list_issues(owner, repo, state)`: List open/closed issues in a repository.
- `get_issue(owner, repo, issue_number)`: Retrieve detailed metadata for a single issue.
- `list_pull_requests(owner, repo, state)`: List pull requests in a repository.
- `get_pull_request(owner, repo, pr_number)`: Retrieve details and mergeability for a PR.
- `search_code(query, owner, repo)`: Search GitHub codebase globally or scoped to a repository.

#### Destructive Operations (Confirm-Gated)
- `create_issue(owner, repo, title, body, confirm=False)`: Creates an issue only when confirmed; returns preview otherwise.
- `close_issue(owner, repo, issue_number, confirm=False)`: Closes an issue only when confirmed; returns preview otherwise.
- `add_comment(owner, repo, issue_number, body, confirm=False)`: Comments on an issue/PR only when confirmed; returns preview otherwise.
- `merge_pull_request(owner, repo, pr_number, confirm=False)`: Merges a PR only when confirmed; returns preview otherwise.

---

## Project Structure

```
├── app/
│   ├── __init__.py          # Public MCP server & client exports
│   ├── server.py            # FastMCP server definition & tool routing
│   └── github_client.py     # GitHub REST API client with dry-run previews
├── eval/
│   ├── __init__.py          # Evaluation package
│   ├── tool_schemas.py      # OpenAI-compatible function definitions & prompt
│   ├── test_set.json        # 30 labeled evaluation test cases
│   └── run_eval.py          # Benchmark harness scoring tool, args, and confirm flags
├── requirements.txt         # Dependencies (mcp, requests, openai)
├── WRITEUP.md               # Detailed evaluation report & iteration analysis
└── README.md
```

---

## Quickstart & Usage

### 1. Installation

```bash
git clone https://github.com/vishalmurugan1986/github-mcp-server.git
cd github-mcp-server

pip install -r requirements.txt
```

### 2. Environment Configuration

```bash
export NVIDIA_API_KEY="nvapi-your-key-here"
# Optional: real GitHub write access
export GITHUB_TOKEN="ghp_your_personal_access_token"
```

### 3. Running the MCP Server

```bash
python -m app.server
```

Connect an MCP client (such as Claude Desktop or custom agents) over standard I/O.

### 4. Running the Tool-Calling Evaluation Suite

```bash
python -m eval.run_eval
```

Results are saved to `eval/results.json` and printed with full metric breakdowns.

---

## Safety Design Principles

1. **Defense-in-Depth Confirmation**: Rather than relying purely on prompt instructions, confirmation guards are hardcoded directly in `app/github_client.py`. An unconfirmed request cannot cause an accidental mutation.
2. **Ambiguity Rejection**: The model is instructed to refuse execution and ask clarifying questions whenever repositories, issue numbers, or parameters are unspecified, avoiding placeholder hallucinations.