mcp-github-tool-eval
GitHub Model Context Protocol (MCP) 服务器与工具调用评估
一个生产级 FastMCP GitHub 服务器,提供读取和确认门控的写入操作,并配有 30 个案例的评估框架,用于评估工具选择、参数准确性和破坏性确认安全性。
执行摘要与评估矩阵
LLM 代理工具调用需要严格的安全边界。此 MCP 服务器对所有变更操作强制执行服务器级确认门控:未确认的请求返回干运行预览,而不是在 GitHub 上执行变更。
基准性能(30 个带标签场景)
针对 30 个案例基准进行评估,涵盖标准工具调用、未指定/模糊查询以及破坏性确认门控操作:
指标 | 第 1 轮(零样本) | 第 3 轮(优化后) |
工具选择准确率 | 76.7% (23/30) | 90.0% (27/30) |
参数准确率(给定正确工具) | 52.6% (10/19) | 100.0% (19/19) |
确认标志准确率(破坏性操作) | 50.0% (3/6) | 100.0% (7/7) |
— 标准工具选择类别 | 100.0% (8/8) | 100.0% (8/8) |
— 错误/破坏性操作类别 | 90.9% (10/11) | 90.9% (10/11) |
— 模糊/缺失参数类别 | 25.0% (2/8) | 75.0% (6/8) |
Related MCP server: GitHub MCP Server
工具架构与操作
flowchart TD
A[User Request] --> B[LLM Tool-Calling Engine]
B --> C{Tool Category}
C -- Read Tool --> D[Execute GitHub API Call Directly]
D --> E[Return API Payload]
C -- Write / Destructive --> F{confirm == true ?}
F -- Yes --> G[Execute State Mutation on GitHub]
G --> H[Return Success / Number / URL]
F -- No --> I[Return Dry-Run Preview of Mutation]可用工具
读取操作(立即执行)
list_issues(owner, repo, state):列出仓库中已打开/已关闭的问题。get_issue(owner, repo, issue_number):检索单个问题的详细元数据。list_pull_requests(owner, repo, state):列出仓库中的拉取请求。get_pull_request(owner, repo, pr_number):检索 PR 的详细信息和可合并性。search_code(query, owner, repo):全局或限定在仓库范围内搜索 GitHub 代码库。
破坏性操作(确认门控)
create_issue(owner, repo, title, body, confirm=False):仅在确认后创建问题;否则返回预览。close_issue(owner, repo, issue_number, confirm=False):仅在确认后关闭问题;否则返回预览。add_comment(owner, repo, issue_number, body, confirm=False):仅在确认后对问题/PR 发表评论;否则返回预览。merge_pull_request(owner, repo, pr_number, confirm=False):仅在确认后合并 PR;否则返回预览。
项目结构
├── app/
│ ├── __init__.py # Public MCP server & client exports
│ ├── server.py # FastMCP server definition & tool routing
│ └── github_client.py # GitHub REST API client with dry-run previews
├── eval/
│ ├── __init__.py # Evaluation package
│ ├── tool_schemas.py # OpenAI-compatible function definitions & prompt
│ ├── test_set.json # 30 labeled evaluation test cases
│ └── run_eval.py # Benchmark harness scoring tool, args, and confirm flags
├── requirements.txt # Dependencies (mcp, requests, openai)
├── WRITEUP.md # Detailed evaluation report & iteration analysis
└── README.md快速开始与用法
1. 安装
git clone https://github.com/vishalmurugan1986/github-mcp-server.git
cd github-mcp-server
pip install -r requirements.txt2. 环境配置
export NVIDIA_API_KEY="nvapi-your-key-here"
# Optional: real GitHub write access
export GITHUB_TOKEN="ghp_your_personal_access_token"3. 运行 MCP 服务器
python -m app.server通过标准 I/O 连接 MCP 客户端(如 Claude Desktop 或自定义代理)。
4. 运行工具调用评估套件
python -m eval.run_eval结果保存到 eval/results.json,并打印完整的指标明细。
安全设计原则
纵深防御确认:与其仅依赖提示指令,确认守卫直接硬编码在
app/github_client.py中。未确认的请求不可能导致意外变更。歧义拒绝:模型被指示在仓库、问题编号或参数未指定时拒绝执行并要求澄清问题,避免占位符幻觉。
This server cannot be installed
Maintenance
Resources
Unclaimed servers have limited discoverability.
Looking for Admin?
If you are the server author, to access and configure the admin panel.
Related MCP Servers
- FlicenseNot gradedqualityDmaintenanceEnables interaction with GitHub repositories through the GitHub API using fastMCP. Supports repository information retrieval and management with secure token-based authentication.
- AlicenseNot gradedqualityDmaintenanceEnables GitHub API integration via MCP, allowing search, repository details, file contents, and issue management.452ISC
- AlicenseNot gradedqualityDmaintenanceEnables management of GitHub issues, milestones, projects, and labels for a single repository via MCP tools.4521ISC
- FlicenseNot gradedqualityDmaintenanceEnables managing GitHub repositories, files, and user information through MCP, with support for creating, updating, and deleting repository contents, as well as fetching user profiles.1
Related MCP Connectors
Turn a GitHub repo or docs site into agent-ready context: pack it or search it, over MCP.
GitHub Private MCP Pack — access private repos, org data via OAuth.
Connect AI assistants to GitHub - manage repos, issues, PRs, and workflows through natural language.
Latest Blog Posts
- Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)By Om-Shree-0709 on .mcpAgent IdentityOAuth 2.1
- Your AI Chatbot Just Exposed Your CEO's Salary to an InternBy Om-Shree-0709 on .Agent IdentityMCP SecurityOAuth Delegation
- Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)By Om-Shree-0709 on .Agentic AiPrompt InjectionWebAssembly
MCP directory API
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/vishalmurugan1986/mcp-github-tool-eval'
If you have feedback or need assistance with the MCP directory API, please join our Discord server