Skip to main content
Glama

Server Configuration

Describes the environment variables required to run the server.

NameRequiredDescriptionDefault

No arguments

Instructions

Guidance the server publishes about itself, which clients place ahead of the tool catalog so the model reads it before choosing anything.

This server publishes no instructions, or was last inspected before Glama recorded them.

Capabilities

Features and capabilities supported by this server

Protocol revision2025-11-25

CapabilityDetails
tools
{
  "listChanged": false
}

Tools

Functions exposed to the LLM to take actions

NameDescription
jevlab_benchB

Benchmark recorded judges on synthetic gold labels; default is an offline demo.

jevlab_calibrateB

Fit Platt, temperature and histogram scaling; return holdout reliability and thresholds as JSON.

jevlab_bias_auditB

Measure position, length, format and provenance-based self-preference; default is synthetic.

jevlab_list_rubricsA

Return the ten public Apache-2.0 rubric templates.

Prompts

Interactive templates invoked by user choice

NameDescription

No prompts

Resources

Contextual data attached and managed by the client

NameDescription

No resources

TDQS

B3.4/5.0

Scored across 4 tools

Disambiguation4/5

Each tool targets a distinct evaluation activity: bias auditing, rubric listing, benchmarking, and calibration. The bias audit and benchmarking tools could be loosely confused as both assess judges, but the descriptions clearly distinguish bias measurement from gold-label benchmarking.

Naming Consistency3/5

All tools share the jevlab_ prefix and snake_case, but the naming pattern is mixed: jevlab_list_rubrics is verb_noun, jevlab_calibrate is a bare verb, jevlab_bench is a short form, and jevlab_bias_audit is a noun phrase. This is readable but not predictably consistent.

Tool Count5/5

Four tools is a well-scoped set for a focused judge-evaluation toolkit. Each tool covers a distinct core capability without excess or apparent redundancy.

Completeness4/5

The surface covers the key lifecycle steps for judge auditing: listing rubrics, auditing bias, benchmarking, and calibrating. Minor gaps exist around custom rubric management and fetching individual rubrics, but agents can work around them.

Maintenance

ActivityMaintained
ResponsivenessNo recent community issues