judge-audit-mcp

by asif786ka

Overview Schema Related Servers Score Discussions

Python

Local

Server Quality Checklist

Profile completionA complete profile improves this server's visibility in search results.

Latest release: v0.1.1
Disambiguation5/5
Each tool serves a clearly distinct purpose: comprehensive audit, specific bias testing, calibration against humans, drift detection, data generation, and metric explanation. No two tools overlap in functionality.
Naming Consistency5/5
All tool names follow a consistent verb_noun pattern in snake_case (e.g., audit_judge, bias_probe, explain_metric). The convention is uniform and predictable.
Tool Count5/5
6 tools cover the essential aspects of LLM judge auditing without redundancy or bloat. Each tool is justified and serves a specific role in the workflow.
Completeness4/5
The set covers comprehensive auditing, bias probes, calibration, drift detection, data generation, and explanation. Minor gap: no tool for running a judge on raw inputs, but that is likely external to this server's scope.
Average 4.2/5 across 6 of 6 tools scored.
See the Tool Scores section below for per-tool breakdowns.
- No community issues in the last 6 months
- 3 commits in the last 12 weeks
- No stable releases found
- No critical vulnerability alerts
- No high-severity vulnerability alerts
- No code scanning findings
- CI status not available
This repository is licensed under MIT License.
This repository includes a README.md file.
No tool usage detected in the last 30 days. Usage tracking helps demonstrate server value.
Tip: use the "Try in Browser" feature on the server page to seed initial usage.
Add a glama.json file to provide metadata about your server.
If you are the author, simply .
If the server belongs to an organization, first add glama.json to the root of your repository:
```
{
  "$schema": "https://glama.ai/mcp/schemas/server.json",
  "maintainers": [
    "your-github-username"
  ]
}
```
Then . Browse examples.
Add related servers to improve discoverability.

How to sync the server with GitHub?

Servers are automatically synced at least once per day, but you can also sync manually at any time to instantly update the server profile.

To manually sync the server, click the "Sync Server" button in the MCP server admin interface.

How is the quality score calculated?

The overall quality score combines two components: Tool Definition Quality (70%) and Server Coherence (30%).

Tool Definition Quality measures how well each tool describes itself to AI agents. Every tool is scored 1–5 across six dimensions: Purpose Clarity (25%), Usage Guidelines (20%), Behavioral Transparency (20%), Parameter Semantics (15%), Conciseness & Structure (10%), and Contextual Completeness (10%). The server-level definition quality score is calculated as 60% mean TDQS + 40% minimum TDQS, so a single poorly described tool pulls the score down.

Server Coherence evaluates how well the tools work together as a set, scoring four dimensions equally: Disambiguation (can agents tell tools apart?), Naming Consistency, Tool Count Appropriateness, and Completeness (are there gaps in the tool surface?).

Tiers are derived from the overall score: A (≥3.5), B (≥3.0), C (≥2.0), D (≥1.0), F (<1.0). B and above is considered passing.

Tool Scores

Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description must disclose behavioral traits. It states 'Explain' but does not confirm it is read-only, nor mentions any side effects, authentication needs, or output format. The presence of an output schema mitigates the lack of return description, but behavioral safety is unaddressed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise, front-loading the purpose in the first sentence. The args list is clearly formatted, and every element serves a purpose. No redundant or irrelevant information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a single-parameter explanation tool with an output schema, the description adequately covers the core purpose and valid inputs. It does not explicitly state that the output is a textual explanation, but the output schema likely handles that. One could argue for completeness given the tool's simplicity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters5/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Although schema description coverage is 0%, the description enumerates all valid values for the 'metric' parameter (kappa, anchors, fingerprint, etc.) within the args section, providing crucial context that the schema (which only defines a string) lacks. This fully compensates for the missing schema descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it explains 'what a judge-audit metric means and why it's the right question', specifying the resource (judge-audit metric) and distinguishing it from sibling tools that perform actions like auditing, probing, or calibrating.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives. The description implies it's for understanding metrics but does not provide explicit when-to-use or when-not-to-use instructions, nor does it mention alternative tools for related tasks.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description discloses that free probes run unconditionally and calibration probes are added with gold labels, but it does not explicitly state whether the tool has side effects, requires specific permissions, or is idempotent. Since no annotations are provided, the description carries the full burden, and this lack of detail leaves some gaps.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured with a summary sentence, context paragraph, and Args list. It is clear and to the point, though slightly verbose for the context.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (6 parameters, conditional behavior) and the presence of an output schema, the description covers the overall workflow, including what happens under different conditions and what the output contains (reports missing probes with generation commands). It is sufficiently complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Despite 0% schema description coverage, the description's Args section adds meaning to all six parameters, explaining the purpose of each (e.g., gold_path enables half the checks, verbose provides full evidence). This compensates for the bare schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states that the tool runs every check supported by the data and reports what's missing, positioning it as the entry point for auditing a judge. It distinguishes itself from siblings like bias_probe and calibrate_judge by specifying its comprehensive check-running role.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description specifies when to use (when you have an eval log and need to check if the judge is okay) and provides conditional behavior (gold labels trigger additional checks). However, it does not explicitly state when not to use or compare with alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations exist, so the description carries full burden. It transparently describes outputs (kappa, bootstrap CI, baseline, ceiling) and warns about the accuracy trap. However, it does not explicitly disclose read-only or safety properties, though the statistical nature implies no side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with a clear introduction and then structured parameter documentation. While slightly verbose, every sentence adds value, and the format is easy to parse.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given that an output schema exists (not shown but noted), the description adequately covers the tool's purpose, parameters, and key behavioral notes. No gaps remain for a calibration tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters5/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, but the description includes a detailed Args section that explains each parameter's purpose, default behavior (e.g., auto weights), and optional nature. This adds meaning far beyond the input schema alone.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool measures agreement between an LLM judge and human labels beyond chance, using Cohen's weighted kappa. This specific verb+resource distinguishes it from siblings like audit_judge or bias_probe.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implicitly conveys usage (when you have judge outputs and human gold labels), but does not explicitly state when to use this tool versus alternatives like audit_judge or detect_judge_drift. No when-not-to-use guidance is provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description covers the algorithm's logic, fingerprint comparison, anchor usage, and conditional output. It lacks explicit safety or permission info but is thorough for a read-only analysis tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with the purpose, uses efficient sentences, and avoids redundancy. It includes both a high-level answer and necessary caveats without unnecessary detail.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers the tool's logic, parameter behavior, and output expectations. An output schema exists, so return format details are not required. It is complete for a technical tool, though some edge cases (e.g., file format errors) are omitted.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters5/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, but the description provides clear semantics for all 4 parameters: run paths with format, anchor_ids with optionality and auto-detection, verbose with content list. This fully compensates for schema gaps.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool compares two eval runs and attributes score change to system or judge, answering a specific question ('is that real?'). It provides a detailed mechanism (diffing judge fingerprint, using anchor items) that distinguishes it from sibling tools like audit_judge or bias_probe.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explains the anchor requirement and behavior without anchors, giving context on when anchors are needed. However, it does not explicitly contrast with sibling tools or state when not to use this tool.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries full burden. It discloses that the tool measures specific biases, explains what each probe detects, and mentions that inferring the scale from the judge's own output would define the pathology out of existence. It also notes the 'verbose' flag includes evidence, fixes, and citations. The description is transparent about requirements and caveats.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is relatively long but well-structured with a clear opening sentence and bullet-pointed probes. The front-loading of the main purpose and parameter details helps an agent quickly grasp the tool. Some redundancy might be trimmed, but overall it is efficient for the complexity of the tool.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool has 8 parameters (2 required) and an output schema. The description covers all probes, required parameters, and optional ones with explanations. It does not explicitly describe the return value format, but the presence of an output schema likely covers that. The description is complete enough for an agent to select and invoke the tool effectively.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters5/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate, and it does. It explains the 'probe' parameter by detailing all six possible values with their meanings and data needs. It clarifies 'gold_path' for self_preference and length_confound, 'scale_min/max' for distribution, and 'claimed_delta' for interpreting distribution results. This adds substantial meaning beyond the bare schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: 'Test an LLM judge for a specific bias.' It then names and explains six distinct probes, each with a specific use case. This distinguishes it from sibling tools like audit_judge or calibrate_judge, which have different focuses.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides explicit context for each probe, including required data variants and conditions (e.g., needs variant=''/''swapped'' records for position bias). It also notes that the 'distribution' probe runs on any existing log. However, it does not explicitly contrast the tool with siblings or state when not to use it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full burden. It discloses that the tool writes a JSONL file to a configurable output path (defaulting to <path>.<probe>.jsonl), describes the output format, and explains that the server requires no API key. However, it does not explicitly state whether existing output files are overwritten or if there are any side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured with a clear first sentence defining purpose, followed by output details, workflow, cost note, and argument list. It is somewhat verbose but each sentence adds value; a slight trim could improve conciseness.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the presence of an output schema and the complexity of the tool, the description adequately covers output format and workflow. It explains the round-trip and cost benefit. It could briefly mention the input format expected for path, but overall it is sufficiently complete for effective use.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters5/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate, which it does thoroughly. It explains each parameter: path (existing judge run), probe (enumeration with context on which probes need variants), out_path (default derivation), and hint (examples for sycophancy). This adds significant meaning beyond the bare schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool generates variant files for probe scoring, specifically writing a JSONL of items to judge with variant tags and original item_id. It distinguishes itself from siblings by noting which probes need variants and which don't (distribution, length_confound).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines5/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly explains the workflow: run emit_probe_set, then run a judge, then pass results to bias_probe. It specifies when to use (for position, verbosity, sycophancy probes) and when not (for distribution, length_confound), and references sibling bias_probe for the next step.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

GitHub Badge

Glama performs regular codebase and documentation scans to:

Confirm that the MCP server is working as expected.
Confirm that there are no obvious security issues.
Evaluate tool definition quality.

Our badge communicates server capabilities, safety, and installation instructions.

Card Badge

Copy to your README.md:

[![judge-audit-mcp MCP server](https://glama.ai/mcp/servers/asif786ka/judge-audit-mcp/badges/card.svg)](https://glama.ai/mcp/servers/asif786ka/judge-audit-mcp)

Score Badge

Copy to your README.md:

[![judge-audit-mcp MCP server](https://glama.ai/mcp/servers/asif786ka/judge-audit-mcp/badges/score.svg)](https://glama.ai/mcp/servers/asif786ka/judge-audit-mcp)

Latest Blog Posts

Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)
By Om-Shree-0709 on July 25, 2026.
mcp
Agent Identity
OAuth 2.1
Your AI Chatbot Just Exposed Your CEO's Salary to an Intern
By Om-Shree-0709 on July 2, 2026.
Agent Identity
MCP Security
OAuth Delegation
Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)
By Om-Shree-0709 on June 30, 2026.
Agentic Ai
Prompt Injection
WebAssembly

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/asif786ka/judge-audit-mcp'

If you have feedback or need assistance with the MCP directory API, please join our Discord server