<?xml version="1.0" encoding="UTF-8"?>
<!--
GitHub MCP Server Evaluation Tests
==================================
These questions test the MCP server's ability to help LLMs
accomplish real-world GitHub tasks.
Requirements:
- Each question is independent
- Only read-only operations required
- Answers are stable and verifiable
-->
<evaluation>
<qa_pair>
<question>Using the GitHub MCP server, find the most starred repository that contains "kubernetes" in its name and is written primarily in Go. What is the exact number of stars it has? (Use search_repositories with appropriate filters)</question>
<answer>The answer will vary based on current star counts, but should return kubernetes/kubernetes with 100k+ stars</answer>
</qa_pair>
<qa_pair>
<question>Search for repositories matching "terraform aws provider" sorted by stars. What is the full name (owner/repo) of the top result?</question>
<answer>hashicorp/terraform-provider-aws</answer>
</qa_pair>
<qa_pair>
<question>Get the repository information for "microsoft/vscode". What is the primary programming language listed for this repository?</question>
<answer>TypeScript</answer>
</qa_pair>
<qa_pair>
<question>List the contents of the root directory of the "kubernetes/kubernetes" repository. How many items (files and directories combined) are in the root?</question>
<answer>The answer will be approximately 20-30 items including directories like api, build, cmd, etc.</answer>
</qa_pair>
<qa_pair>
<question>Get the file contents of "README.md" from the "docker/compose" repository. What is the first heading (H1) in the README?</question>
<answer>Docker Compose</answer>
</qa_pair>
<qa_pair>
<question>Get the user profile for "torvalds" (Linus Torvalds). In what year was this GitHub account created?</question>
<answer>2011</answer>
</qa_pair>
<qa_pair>
<question>List the repositories for user "hashicorp" sorted by stars in descending order. What is the name of their most starred repository?</question>
<answer>terraform</answer>
</qa_pair>
<qa_pair>
<question>Search for code containing "def main" in the language Python within the repository "python/cpython". What is the path of the first result returned?</question>
<answer>Results will vary but should return Python files from the cpython repository</answer>
</qa_pair>
<qa_pair>
<question>Get the commit history for the "facebook/react" repository, limited to 5 commits. What is the 7-character SHA of the most recent commit?</question>
<answer>The answer will be a 7-character SHA that changes with new commits</answer>
</qa_pair>
<qa_pair>
<question>List the branches in the "golang/go" repository. Is the "master" branch protected according to the API response?</question>
<answer>Yes (the master branch is protected)</answer>
</qa_pair>
</evaluation>
