memtrust

Overview Schema Source Related Servers Discussions Admin

run

Benchmark agent-memory backends with a reproducible JSON report. Run MemPalace, Mem0, Zep/Graphiti, or OpenViking against evals like LongMemEval and LoCoMo. Get per-backend scores for quality and performance, bypassing vendor claims.

Instructions

Runs memtrust's agent-memory benchmark suite (LongMemEval, LoCoMo, contradiction-detection, and a dozen other evals) against one or more configured memory backends -- MemPalace, Mem0, Zep/Graphiti, OpenViking -- and returns the resulting JSON report. Call this when an agent needs a fresh, reproducible comparison of memory-backend quality or performance rather than trusting a vendor's self-reported numbers; do not call it just to read documentation (pass ["--help"] for that instead) or to inspect a report that already exists on disk (use the CLI's report subcommand directly for that). A backend is only evaluated if its credential env var is set in the server's environment (e.g. MEM0_SELFHOSTED_API_KEY, GRAPHITI_GEMINI_API_KEY); backends missing credentials are reported as skipped rather than causing an error, so no prerequisite check is required before calling.

Behavior: this shells out to the real memtrust run CLI as a subprocess (network calls happen for whichever backends are configured; evals also read/write a private temp file for the JSON report, cleaned up automatically). It is read-only with respect to the backends themselves -- it queries and scores them, it does not modify backend state -- and each call is idempotent: running the same args twice re-runs the full suite and produces a fresh, independent report rather than mutating prior results. Runs can take several minutes (subprocess timeout is 600s) depending on which evals and backends are selected. On a non-zero CLI exit, the tool does not raise; it returns {"error": ..., "returncode": ..., "stderr": ...} so the calling agent can inspect what failed.

Parameter: args is a list[str] of argv tokens appended to memtrust run (NOT a shell string, and do not include "run" itself or an --output flag -- the wrapper supplies its own temp output path). Real examples pulled from memtrust run --help:

[] -- run every eval against every configured backend (both flags default to "all")
["--backends", "mempalace", "--eval", "contradiction"] -- one backend, one eval
["--backends", "mem0,zep", "--eval", "longmemeval,locomo", "--scale-stress-n-records", "10000"] -- multiple backends and evals, plus a numeric tuning flag
["--help"] -- exits before writing a report, so it surfaces as an error dict whose stdout field holds the CLI's own live help text; useful for discovering flags this description doesn't enumerate, such as --sign or --locomo-dataset-path

Return shape: on success, the parsed contents of memtrust's JSON report file (per-backend, per-eval results and scores -- see memtrust report --help for how a human would render the same file). On failure, a dict with an error key plus whichever of returncode / stderr / stdout is available.

Input Schema

TableJSON Schema

Name	Required	Description	Default
`args`	Yes

Output Schema

TableJSON Schema

Name	Required	Description	Default
No arguments

Schema Changelog

Changes observed during successful MCP inspections.

First observedv0.1.0Aug 10, 2026

TDQS

A4.9/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden and does so thoroughly. It discloses that it shells out to a subprocess, makes network calls, reads/writes a temp file cleaned up automatically, is read-only with respect to backends, is idempotent, can take minutes with a 600s timeout, and returns an error dict on non-zero exit instead of raising. This is exemplary behavioral disclosure.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is long but every section earns its place: purpose, when-to-use, behavior, parameter semantics, and return shape are all clearly separated. It is front-loaded with the core purpose and usage guidance. It loses one point only because the length is substantial, though the density of useful information justifies most of it.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description covers purpose, usage, prerequisites, failure behavior, return shape, parameter semantics, and even points to `--help` for discovering additional flags. Given the tool's complexity (subprocess, network, long runtime, multiple backends/evals), nothing an agent needs to call it correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must fully compensate. It explains that `args` is a list of argv tokens, not a shell string, warns not to include 'run' or an `--output` flag, and provides four realistic examples with explanations of what each does. This is far beyond what the bare schema provides.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb ('runs'), a specific resource ('memtrust's agent-memory benchmark suite'), and enumerates the concrete evals and backends involved. It also distinguishes itself from reading docs and from the `report` subcommand, so an agent can tell exactly what this tool does.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly says when to call it ('when an agent needs a fresh, reproducible comparison'), when not to call it ('do not call it just to read documentation' or 'to inspect a report that already exists on disk'), and names the alternative (`report` subcommand). It also explains the credential prerequisite and that missing credentials are skipped, so no pre-check is needed.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools

runA