Skip to main content
Glama

Inter-rater agreement (kappa)

revisia_interrater_agreement
Read-onlyIdempotent

Use this when the user wants agreement statistics between reviewers, e.g. for dual screening or data extraction: Cohen's kappa (unweighted, linear or quadratic weights) for two raters, Fleiss' kappa for three or more, with Gwet's AC1/AC2, PABAK, 95% confidence interval, p-value, the Landis and Koch band, a sentence for the methods section and a figure. Accepts a k×k contingency table or one row of labels per item (one column per rater). Deterministic and checked against R (irr, psych, irrCAC); do not compute kappa yourself when this tool is available. It does not resolve disagreements or judge which rater is right. No account needed.

Tool call template

This tool declares no required parameters. Add optional parameters from the input schema as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {},
    "name": "revisia_interrater_agreement"
  }
}

Input Schema

NameRequiredDescriptionDefault
langNoOutput language (default en).en
tableNoTwo raters: k×k counts, rater A rows, rater B columns. Give this or ratings.
ratingsNoOne row per item, one label per rater, e.g. [['Include','Exclude']]. 2 raters: Cohen; 3+: Fleiss. Give this or table.
weightingNoOrdered categories, Cohen, 3+ categories. Default none.none
categoriesNoCategory names in table order. Only with table.

Output Schema

NameRequiredDescriptionDefault
nYes
pYes
zYes
seYes
altYes
bandYes
gwetYes
langYes
kappaYes
pabakYes
ci_lowYes
methodYes
ratersYes
ci_highYes
formulaYes
citationYes
headlineYes
sentenceYes
warningsYes
ci_methodYes
kappa_maxYes
watermarkYesTrue when the figure or document carries the revisia mark (no verified account).
weightingYes
band_labelYes
bias_indexYes
categoriesYes
next_actionsYes
per_categoryYes
open_in_revisiaYes
prevalence_indexYes
expected_agreementYes
observed_agreementYes

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observed

TDQS

A4.6/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare read-only, idempotent, non-destructive, closed-world behavior, and the description reinforces this with real added context: deterministic results, validation against R packages (irr, psych, irrCAC), and no account needed. It also states the scope boundary that it will not adjudicate disagreements, which prevents misused expectations. The remaining gap is minor — it does not discuss input-size limits or error behavior, but the annotations carry the safety profile.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

One dense paragraph, front-loaded with the use case before the statistical detail and the exclusions. It is long but nearly every clause carries information; the list of output artifacts (CI, p-value, band, methods sentence, figure) is the only part that partly duplicates what the output schema already provides.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given an output schema exists, the description need not explain return values, and it still names them for decision-making. It covers inputs, variant selection, provenance, and scope limits, so an agent has everything required to call it correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3, but the description adds the critical semantic that 'table' and 'ratings' are alternative input modes — a k×k contingency table versus one row of labels per item with one column per rater — plus the mapping from rater count to Cohen vs Fleiss. That mutual-exclusivity framing helps an agent choose correctly even if the schema descriptions already gesture at 'Give this or ratings'.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific computation (agreement statistics / kappa) with named variants (Cohen unweighted/linear/quadratic, Fleiss, Gwet AC1/AC2, PABAK) and expected outputs. An agent can immediately distinguish it from sibling tools like revisia_screening_act or revisia_rob, which act on reviews rather than computing statistics.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives an explicit trigger ('when the user wants agreement statistics between reviewers, e.g. for dual screening or data extraction'), an explicit exclusion ('does not resolve disagreements or judge which rater is right'), and an explicit instruction not to compute kappa by hand while this tool exists. It also routes to the correct statistical variant via the 2-rater vs 3+-rater rule.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources