Skip to main content
Glama
TrialRun MCP

TrialRun MCP

by kostadindev

trialrun-mcp

CI

An MCP gateway that assesses data stewardship and AI-literacy habits while any MCP agent works on your files. The agent reads files only through the gateway, so you get evidence of what actually reached the AI, not what someone says happened.

  • Every file the agent opens is logged with a timestamp.

  • Personal data, credentials and your own sensitive terms are flagged, or redacted before the agent sees them.

  • A live data stewardship score: prohibited data reached the AI → Zero; personal or internal data → Partial; else Full.

  • A habit log and AI literacy report that a coaching skill can write to (trialrun-skill).

  • The session log, reports, the audit log and tool diagnostics hold labels and redacted snippets only (for example "national insurance number: JR***"), never raw values. File content is different: in monitor mode read_file returns the file as written, so the agent does see the original values (that is what gets scored); in enforce mode restricted values are redacted before the agent sees them, and the finding is logged as withheld, which does not lower the score.

It also ships a built-in demo scenario: the TrialRun accounting assessment, with restricted files planted among the ones the task needs.

Quickstart

Needs Node 20+ and git. Nothing to clone: npx installs straight from GitHub. The first run takes about a minute while npm builds the package; later runs are instant.

1. Run the demo (no agent or API key needed)

npx -y -p github:kostadindev/trialrun-mcp trialrun-demo ./trialrun-demo

This writes a small fictional client folder (budget.csv, payroll.csv, client-email.txt) with a policy that blocks payroll.csv and marks "Project Kestrel" as confidential. It then starts the gateway and plays the tool calls an agent would make, printing what the gateway does at each step:

  1. lists the files (payroll is hidden by policy);

  2. reads budget.csv (allowed, nothing restricted);

  3. tries payroll.csv (refused);

  4. reads client-email.txt (allowed; the bank details and the confidential term are flagged and logged);

  5. checks a draft answer and catches the account number before it is sent;

  6. records a habit moment, as the coaching skill would;

  7. prints the AI literacy report.

It ends with Demo complete. and the command for step 2. The folder is kept; re-running the demo resets its log.

2. Connect a real agent to the demo folder

TRIALRUN_WORKSPACE="$PWD/trialrun-demo" npx -y -p github:kostadindev/trialrun-mcp trialrun-policy reset   # clear the demo's log, keep the policy
claude mcp add trialrun -e TRIALRUN_WORKSPACE="$PWD/trialrun-demo" -- npx -y github:kostadindev/trialrun-mcp
claude

In Claude Code, /mcp should list trialrun as connected. Then try:

  • "What's the average salary in payroll.csv?" (the gateway refuses the file; the agent should say so, not guess)

  • "Summarise client-email.txt" (the read is logged and the bank details are flagged)

  • "Show me my AI literacy report" (calls get_report)

For short coaching tips while you work, add the skill inside Claude Code: /plugin marketplace add kostadindev/trialrun-skill, then /plugin install trialrun@trialrun (see trialrun-skill).

Remove it later with claude mcp remove trialrun.

3. Use it on your own folder

claude mcp add trialrun -e TRIALRUN_WORKSPACE=/path/to/folder -- npx -y github:kostadindev/trialrun-mcp

Block anything the AI must not read first (see Owner controls).

Claude Desktop (claude_desktop_config.json):

{
  "mcpServers": {
    "trialrun": {
      "command": "npx",
      "args": ["-y", "github:kostadindev/trialrun-mcp"],
      "env": { "TRIALRUN_WORKSPACE": "/path/to/folder" }
    }
  }
}

Leave out TRIALRUN_WORKSPACE to serve the demo scenario.

Related MCP server: llm-guard-gateway

Two modes

Mode

Turn on

Serves

Restricted data found by

Workspace

TRIALRUN_WORKSPACE=/abs/path

Your real folder (text files)

Pattern detectors and your own terms

Scenario

default

The TrialRun demo files

Unique marker strings planted in restricted sections

Owner controls

The agent can't change the policy or wipe the log. The folder's owner can, with the policy file or the CLI:

export TRIALRUN_WORKSPACE=/path/to/folder
npx -y -p github:kostadindev/trialrun-mcp trialrun-policy block "secrets/**" "*.env" credentials.txt
npx -y -p github:kostadindev/trialrun-mcp trialrun-policy term "Project Kestrel" cat1
npx -y -p github:kostadindev/trialrun-mcp trialrun-policy mode enforce
npx -y -p github:kostadindev/trialrun-mcp trialrun-policy show
npx -y -p github:kostadindev/trialrun-mcp trialrun-policy reset     # clears the log, keeps the policy

<workspace>/.trialrun.json:

{
  "block": ["credentials.txt", "secrets/**", "*.env"],
  "sensitive_terms": [{ "term": "Project Kestrel", "category": "cat1" }],
  "mode": "monitor"
}

Globs support *, ** and ?; a pattern with no / matches the file name at any depth.

The same exclusion rule decides what is listed and what can be read, and it is checked on both the requested path and the real path behind any symlink. Never listed or read: any path with a segment starting with . (so .env, .git/, the policy file and the default session log), node_modules/, the configured session log wherever TRIALRUN_STATE puts it, files matching a block pattern, and symlinks that leave the workspace.

If the policy file exists but is not valid JSON or has a malformed field, the gateway fails closed: it refuses to list or read anything, and trialrun-policy refuses to edit the file, until the owner fixes it. Policy edits are written atomically. The session log is append-only JSON Lines, one line per tool call, and writes take a lock file, so the server and the policy CLI can write to it at the same time.

Tools

Tool

Purpose

list_files

Files the agent may see (blocked ones are hidden)

read_file

Read a file, a line range, or a scenario section; logged; restricted data flagged or redacted

get_access_policy

Blocked patterns and mode

check_text

Scan an answer, draft or pasted text for restricted data

record_moment

Log a habit moment: check / decide / protect × did / prompted / missed

get_report

AI literacy report: stewardship score, leaks by type and channel, files read, habits

get_audit_log

Timestamped events and leak summary

set_access, reset_session

Assessor only (TRIALRUN_ROLE=assessor)

Resources: trialrun://report, trialrun://audit-log, and in scenario mode trialrun://briefing.

Detectors

Category

What

1, prohibited

Labelled credentials, AWS / sk- / GitHub keys, bearer tokens and JWTs, private keys, Luhn-valid card numbers, passport numbers next to "passport", your cat1 terms

2, not for AI

Emails, phone numbers, UK NI numbers, UK home addresses, IBANs, sort code + account, dates of birth, your cat2 terms

What counts as restricted:

  • Emails: personal addresses are flagged. Role mailboxes (accounts@, info@, finance@, noreply@ and similar) belong to a function, not a person, so they are not.

  • Phones: mobile and geographic numbers (07, 01, 02, +44, other + country codes) are flagged. Non-geographic business numbers (03, 08, 09) are not.

  • Addresses: home addresses are flagged; the postcode is the span. Business addresses (lines with Ltd, plc, Unit, a business park or estate, a registered office) are not.

  • Bank details: sort code + account number and IBANs are always flagged.

Measured accuracy

npm run accuracy scores the detectors against labelled corpora of fictional accounting documents (payroll, bank exports, invoices, loan agreements, emails, configs, audit working papers). A hit counts when its label matches and its span overlaps a labelled span; a private key must be covered from its BEGIN line to its END line.

Corpus

Documents

Labelled items

Precision

Recall

Tuning (accuracy/corpus), used while developing

40

77

100%

100%

Held-out v1 (accuracy/corpus-heldout-v1), misses fixed, now a regression set

10

28

100%

100%

Held-out v2 (accuracy/corpus-heldout-v2), misses fixed, now a regression set

12

22

100%

100%

Held-out v3 (accuracy/corpus-heldout-v3), written after those fixes, never tuned on

12

30

96.0%

80.0%

The tuning corpus gates CI (precision and recall at least 95%, and at least 90% recall for every label with five or more items). Held-out v3 is the honest estimate. Each held-out set was scored once before its misses were fixed: v1 at 100% precision and 85.7% recall, v2 at 90.0% and 81.8%. One v2 label was corrected after its first run: a sort code and account in a JSON expense export were restricted under the policy but had been left unlabelled. v3 was labelled before any detector ran on it.

Known misses on held-out v3:

  • Home addresses written entirely in capitals, as in bank statements ("9 ABBEY WALK").

  • A home postcode on its own in prose, with no street ("Home (CB4 2QW) to client site").

  • Tab-separated files: date-of-birth and postcode columns are only recognised in comma-separated files.

  • Role mailboxes outside the built-in list (for example onboarding@) are flagged as personal.

Also not detected: names on their own (add them as terms), non-UK identity numbers, and addresses without a UK postcode.

Limitations

  • Text files only. PDFs and Office files are refused; export them to text or CSV.

  • The gateway sees only what goes through it. Files the agent reads with its own tools, or text pasted straight into chat, are seen only if the agent calls check_text.

  • Habit moments are logged by the AI. The stewardship score comes only from what the gateway observed directly.

Development

npm install
npm test           # unit, in-process MCP and stdio end-to-end tests (vitest)
npm run check      # typecheck + lint + tests with coverage (thresholds: 90% lines, 80% branches)
npm run build      # compile src/ to dist/
npm run accuracy   # detector precision and recall on the labelled corpora
npm run test:package  # after build: npm pack, install the tarball in a temp project, drive the bins over stdio
npm run test:online   # install from GitHub with an empty npm cache, as the Quickstart does, and run the demo
node dist/bin/demo.js ./trialrun-demo   # the demo, from a local build
npm run corpus     # regenerate the corpora from accuracy/*-src.ts

Path

What

src/gateway.ts

Builds the MCP server for a config

src/tools/

Tool registration: workspace.ts, scenario.ts, and shared.ts for both modes

src/detectors.ts

Restricted-data detectors and redaction

src/workspace.ts

Policy file, globs, safe path resolution, listing and reading

src/state.ts, src/report.ts, src/scan.ts

Session log, stewardship score and report, text scanning

src/policy-cli.ts

Owner / assessor commands

src/bin/

trialrun-mcp, trialrun-policy and trialrun-demo entry points

src/scenario/

The built-in TrialRun demo files and markers

test/

Vitest suites

accuracy/

Labelled corpora and the precision/recall scorer

Variable

Default

Meaning

TRIALRUN_WORKSPACE

unset

Real folder; switches to workspace mode

TRIALRUN_STATE

<workspace>/.trialrun-session.jsonl or ./trialrun-session.jsonl

Session log (JSON Lines)

TRIALRUN_MODE

monitor

monitor or enforce (the policy file's mode wins)

TRIALRUN_ROLE

student

student or assessor

Part of TrialRun, built for Cambridge EduX 2026.

Licence

MIT

Available Tools

7 tools
check_textScan text for restricted dataA

Scan an outgoing AI answer, a draft, or text the user pasted, for restricted data (credentials, cards, personal details, sensitive terms) before it is shown or saved. Hits are logged as leaks into the AI answer. Returns labels and redacted snippets only.

Inputs: text (required), record.

get_access_policyGet access policyA

Show the student's access checklist: which files the AI may and may not open.

Inputs: no named parameters.

get_audit_logAudit logB

Timestamped gateway events, restricted-data leak summary and a preliminary Data Stewardship score (Full / Partial / Zero). Contains labels and redacted snippets only.

Inputs: no named parameters.

get_reportAI literacy reportA

A short markdown report for the user: data stewardship score, restricted data by type and channel (counts only, no values), files read, and the three habits with recent examples.

Inputs: no named parameters.

list_filesList workspace filesB

List the files on the Fernvale Kitchens Ltd assessment desktop that the AI is allowed to see. Files the student has blocked are hidden.

Inputs: folder.

read_fileRead a workspace fileA

Read a file (or one section of it) from the assessment desktop. Every read is logged by the gateway. Blocked files are refused.

Inputs: file_id (required), section_id.

record_momentRecord an AI-literacy habit momentA

For a coaching skill: log a moment for one of three habits. habit: "check" (verifying AI output), "decide" (keeping a judgement call), "protect" (keeping restricted data out). outcome: "did" (the user did it unprompted), "prompted" (after a coaching note), "missed". Restricted values in the note are redacted before storage.

Inputs: note (required), habit (required), outcome (required).

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.8/5.0

    Scored across 7 tools

    Disambiguation4/5

    Each tool has a fairly distinct purpose: policy viewing, file listing, file reading, text scanning, moment logging, reporting, and audit. The main overlap is get_report vs get_audit_log, which both surface restricted-data information, though the descriptions differentiate user-facing scoring from raw gateway events.

    Naming Consistency5/5

    All seven tools follow a consistent snake_case verb_noun pattern (get_access_policy, list_files, read_file, check_text, record_moment, get_report, get_audit_log). No camelCase mixing or vague bare verbs.

    Tool Count5/5

    Seven tools is well within the ideal 3-15 range and each maps to a distinct step of the stewardship workflow. Nothing feels padded or missing for the stated scope.

    Completeness4/5

    The read-and-monitor surface is coherent: policy, list, read, check, record, report, audit covers the assessment lifecycle. The only gaps are the absence of any file-search/write operation, which appears intentional for a read-only assessment sandbox.

    Maintenance

    ActivityMaintained
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      D
      maintenance
      Governed MCP gateway that lets AI agents call tools with policy enforcement, prompt-injection screening, a kill-switch, and tamper-evident signed audit logs.
      Apache 2.0
    • A
      license
      Not graded
      quality
      D
      maintenance
      Provides a governance gateway for LLM and agent traffic via MCP, blocking prompt injection, redacting PII, enforcing per-tenant token budgets, and caching repeat questions semantically. Enables AI agents to route model calls through the same guardrails instead of calling providers directly.
      MIT
    • A
      license
      Not graded
      quality
      C
      maintenance
      Enables AI agents to reach upstream MCP tool servers through a central policy gateway that authenticates each agent, enforces least-privilege and default-deny rules, and applies response obligations such as redaction, inference stripping, PII masking, and truncation. It also records every decision in a tamper-evident, hash-chained audit log that can be verified later.
      Apache 2.0
    • A
      license
      Not graded
      quality
      D
      maintenance
      Acts as an MCP gateway between AI agents and upstream tool servers, evaluating YAML policies on every tool call, injecting real credentials at the last mile, scanning requests and responses for secrets, PII, and prompt injection, and recording each action in a tamper-evident audit log. Also supports HTTP forward-proxy governance with rate limiting, cost tracking, and policy replay against historical audit data.
      4
      Apache 2.0