Skip to main content
Glama
Aravindargutus

MCP Gatekeeper

by Aravindargutus
OverviewSchemaRelated ServersScoreDiscussions
TypeScript
Hybrid

MCP has 200+ server implementations but zero quality gates before publishing. MCP Gatekeeper fills that gap.

Run it before you publish to the MCP Registry, npm, or Smithery. It catches schema violations, security vulnerabilities, functional bugs, and quality issues that other tools miss — including an LLM-powered semantic evaluation that compares what your tools claim to do against what they actually do.

npx mcpqa run --server-url https://your-mcp-server.com/mcp

What It Catches

Real findings from running against a production Zoho MCP server (65 tools):

Finding

Gate

What It Means

125 tools accept invalid inputs silently

Functional

Server returns isError: false for empty/wrong-type args

107 parameters have no description

Schema

LLMs can't use tools with undocumented params

26 write tools lack rate limit info

Security

Consumers don't know if there's a throttle

SQL injection payloads reflected in output

Security

Dynamic injection testing caught real vulnerability

Tool descriptions don't match actual behavior

AI Eval

Claude compared descriptions vs invocation results

Related MCP server: APIMatic Validator MCP Server

Quick Start

# Validate an MCP server (Gates 1-3: schema + security + functional)
npx mcpqa run --server-url https://your-server.com/mcp

# Add AI semantic evaluation (Gate 4 — requires Anthropic API key)
ANTHROPIC_API_KEY=... npx mcpqa run --server-url https://your-server.com/mcp --gates 1,2,3,4

# Validate a Claude Code skill
npx mcpqa run --skill-path ./.claude/skills/my-skill --gates 6

# Validate an extension
npx mcpqa run --extension-path ./my-extension --gates 7

# Validate your npm package before publishing
npx mcpqa run --package-path . --gates 8

# Everything at once
npx mcpqa run --server-url https://server.com/mcp --package-path . --gates 1,2,3,4,8 --mode lenient

Gates

Gate

What It Validates

Validators

Speed

1. Schema

Protocol conformance, tool names, JSON Schema, descriptions, parameter types

6

< 1s

2. Security

Prompt injection, SSRF, secrets, permissions, dangerous patterns, dynamic injection with real payloads

9

2-10s

3. Functional

Handshake, invocation, boundary testing, error format, timeouts, idempotency

9

2-5min

4. AI Semantic

Description accuracy vs behavior, param clarity, response completeness, integration readiness, tool chains

7

2-5min

5. Human Review

Dashboard with approve/reject/escalate, auto-approve for high scores, audit trail

1

On-demand

6. Skills

SKILL.md structure, frontmatter, description quality, content length, references, scripts

6

< 1s

7. Extensions

Manifest, permissions, bundled MCP configs, security scanning

4

< 1s

8. Package

package.json, server.json, LICENSE, dependency audit, secrets in source

5

< 1s

AI Evaluation

Gate 4 uses an LLM-as-judge (Claude or OpenAI-compatible) to evaluate things code can't:

  • Description accuracy: Does the description match what the tool actually does? (compares against Gate 3 invocation results)

  • Parameter clarity: Could a developer use this tool correctly on the first try?

  • Integration readiness: Can someone build an integration from metadata alone?

  • Tool chain analysis: Do the tools form coherent workflows, or are there gaps?

Applies patterns from Anthropic's harness design and eval framework:

  • Skeptical evaluator — system prompt forces adversarial QA, not praise

  • pass@k / pass^k — multi-trial evaluation for reliability metrics

  • Actionable fixes — every finding includes specific developer guidance

  • Transcript recording — full LLM conversations saved for debugging

Use as MCP Server

MCP Gatekeeper is itself an MCP server. Add it to Claude Desktop:

{
  "mcpServers": {
    "mcpqa": {
      "command": "npx",
      "args": ["-y", "mcpqa-server"]
    }
  }
}

Then ask Claude: "Validate the MCP server at https://example.com/mcp"

6 tools exposed: validate_mcp_server, validate_skill, validate_extension, validate_package, get_report, list_validators

CLI Reference

mcpqa run [options]          # Run validation pipeline
mcpqa validate-config <path> # Validate YAML config
mcpqa diff <before> <after>  # Compare two reports
mcpqa dashboard [--port N]   # Launch human review UI

Options:
  --server-url <url>         MCP server URL (http/sse)
  --server-cmd <cmd>         MCP server command (stdio)
  --skill-path <path>        Claude Code skill directory
  --extension-path <path>    Extension directory
  --package-path <path>      npm package directory
  --gates <nums>             Gate numbers (e.g., 1,2,3,4,6,7,8)
  --mode <strict|lenient>    Stop on first fail vs run all
  --trials <number>          LLM evaluation trials for pass@k
  --save-baseline            Save results for regression tracking
  --check-regression         Compare against saved baseline
  --verbose / --debug        Control log verbosity
  --dry-run                  Show what would run without executing

Reports

Format

File

Use Case

Console

stdout

Real-time progress with colored badges and progress bars

JSON

reports/latest.json

CI/CD integration, programmatic access

HTML

reports/latest.html

Dark-themed visual report for stakeholders

SARIF

reports/latest.sarif

GitHub Code Scanning, VS Code SARIF Viewer

Transcripts

reports/transcripts/

Gate 4 LLM conversation debugging

CI/CD Integration

# GitHub Actions
- name: Validate MCP Server
  run: npx mcpqa run --server-cmd "node dist/server.js" --gates 1,2,3,8
  # Exit code 0 = pass, 1 = fail
# Docker
docker build -t mcpqa .
docker run mcpqa run --server-url https://server.com/mcp

Configuration

# configs/default.yaml
pipeline:
  mode: strict
  enabledGates: [1, 2, 3]

server:
  transport: http
  url: https://your-server.com/mcp
  headers:
    Authorization: "Bearer your-token"

semantic:
  trials: 3                    # pass@k evaluation
  autoApproveThreshold: 4.5   # Gate 5 auto-approve

reporting:
  formats: [console, json, sarif]
  outputDir: ./reports

Development

npm install
npm test              # 172 tests
npm run lint          # type-check
npm run build         # compile to dist/

Architecture

CLI (mcpqa) ─┐                    ┌─ MCP Server (mcpqa-server)
              │                    │
              ▼                    ▼
         PipelineOrchestrator (shared core)
              │
    ┌─────────┼──────────┬──────────┐
    ▼         ▼          ▼          ▼
 Gate 1-3  Gate 4     Gate 5     Gate 6-8
 Code      LLM Judge  Dashboard  File
 (24 val)  (7 val)    (htmx)    (15 val)

Contributing

PRs welcome. Each validator is a single file implementing IValidator. To add a new check:

  1. Create src/gates/gateN-name/validators/your-check.ts

  2. Implement IValidator interface (name, description, validate)

  3. Register in the gate's index.ts

  4. Add tests in tests/unit/gateN/

License

MIT

Links

This server cannot be installed

F
license - not found
-
quality - not tested
D
maintenance

How are these scores calculated?

Maintenance

–Maintainers
–Response time
–Release cycle
–Releases (12mo)
Commit activity

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/Aravindargutus/mcp-gatekeeper'

If you have feedback or need assistance with the MCP directory API, please join our Discord server