A
licenseNot graded
qualityC
maintenanceAn MCP server that enables AI agents to observe and interact with trackio experiment tracking, providing tools for managing ML experiments through natural language.
3
MIT
pdml-agent
pdml-agent
Un servidor MCP y agente de llamada a herramientas sobre el pipeline de experimentos property-driven-ml, con una puerta de control humano en el bucle para todo lo que consuma cómputo y un rastro estructurado de cada llamada.
Property-driven ML entrena clasificadores contra restricciones de lógica formal, por lo que una ejecución se define por una restricción, un conjunto de datos, una lógica diferenciable y una semilla, y produce métricas por época tanto para el rendimiento predictivo como para la seguridad de las restricciones. Esto lo convierte en un dominio genuinamente moldeado para herramientas, no en uno de demostración: los experimentos se pueden listar, las configuraciones recuperar, los resultados leer, las ejecuciones comparar y nuevas ejecuciones planificar, aprobar y ejecutar.
Estado: completo según lo previsto. Servidor, agente, puerta de control, rastreo. Ejecución real demostrada en CPU.
Arquitectura
┌──────────────────────────────────────────────────────────────┐
│ agent.py (Anthropic SDK tool runner) │
│ │
│ claude-opus-5 ──► pending tool_use ──► ToolLedger.wrap │
│ ▲ │ memoise (RO) │
│ │ │ gate (compute)│
│ │ tool_result │ trace (JSONL) │
│ └───────────────────────────────────┘ │ │
└─────────────────────────────┬───────────────────────┼────────┘
MCP over stdio ▼
┌─────────────────────────────┴────────────────┐ traces/*.jsonl
│ server.py (mcp MCPServer, thin) │
│ list_experiments get_experiment_config │
│ get_results compare_runs │
│ search_logic_definitions │
│ run_experiment ──► PDML_ALLOW_EXECUTE=1 ? │
└────┬───────────┬──────────────┬──────────────┘
▼ ▼ ▼
experiments.py logic_defs.py runner.py ──► subprocess: main.py
(read CSVs) (parse source) (plan/execute) in property-driven-mlagent.py no sabe nada del dominio. Se conecta al servidor a través de stdio como cualquier otro cliente MCP y trabaja solo con las herramientas que el servidor expone. Los módulos del dominio no tienen dependencia de MCP y se pueden probar mediante importación. server.py solo registra herramientas y delega.
Related MCP server: MLflow MCP Server
Diseño
pdml_agent/
experiments.py reading and comparing runs
logic_defs.py searching the logic implementations
runner.py validating, planning and executing runs
server.py the MCP layer, deliberately thin
agent.py the agent: runner, gate, memoisation, tracing
scripts/
make_fixtures.py generate sample runs
smoke_test.py start the server, exercise every tool, check refusals
demo.py run the agent on five tasks
fixtures/results/ sample runs, so nothing needs a GPU to demo
demo_output/ what the agent said and did, one JSON per task
traces/ one JSONL per run, every turn and every callHerramientas
Herramienta | Devuelve |
| ejecuciones, filtrables por restricción, conjunto de datos o lógica |
| la configuración con la que realmente se entrenó una ejecución |
| métricas para una época, por defecto la última |
| diferencia de configuración y métricas entre dos ejecuciones |
| clases de lógica, sus operadores y cadenas de documentación |
| con |
La puerta de control
run_experiment es la única herramienta que consume cómputo, y dos cosas independientes se interponen ante ella.
El servidor no ejecutará a menos que se haya iniciado con PDML_ALLOW_EXECUTE=1. Esa es una decisión tomada por quien ejecuta el servidor, y ninguna solicitud puede cambiarla. Sin ella, dry_run=false devuelve status: refused con el plan adjunto, y no es un error.
El agente no enviará una solicitud de ejecución sin que un operador apruebe la llamada exacta. El mensaje de aprobación muestra el nombre de la herramienta y los argumentos completos como JSON, no un resumen. Una denegación devuelve un resultado normal que lee declined_by_operator, y se instruye al modelo para que lo informe y se detenga, en lugar de reintentar.
Cualquiera de las dos capas por sí sola detendría una ejecución no deseada. Ambas juntas significan que ninguna tiene que ser perfecta. La política que decide qué necesita aprobación es una función, needs_approval, lo suficientemente pequeña como para leerla de un vistazo.
El rastro
Cada ejecución añade a traces/<timestamp>-<question>.jsonl. Una línea por evento, nunca se reescribe.
Los registros de turn llevan el número de paso, la razón de parada del modelo, su texto y resumen de pensamiento, las llamadas que está a punto de hacer y el uso de tokens de ese turno. Los registros de tool_call llevan la herramienta, sus argumentos, si la llamada tuvo éxito, vino de la caché o fue controlada, su latencia, un resumen del resultado y la razón declarada por el propio modelo, tomada de la frase que escribió junto a la llamada. Los registros de gate llevan la decisión. run_start y run_end lo enmarcan con totales.
El mensaje del sistema pide al modelo que indique en una frase por qué está haciendo cada llamada, y lo hace. Del rastro de la ruta de denegación:
turn 1 "I'll start by finding the existing YG runs to confirm identifiers."
turn 2 "No results with those filters; let me broaden."
turn 3 "The constraint is named `standard-robustness`. Let me get the seed-0 run's config and results."
turn 4 "Now the dry-run plan for the requested run (matching epsilon 0.3 from the seed-0 baseline)."
turn 5 "Plan validated. Now executing it." ← gate: declined
turn 6 "The training run was not executed: the operator declined ..."Ese rastro también detectó un defecto en las herramientas de este propio repositorio. El turno 1 obtuvo un resultado vacío porque list_experiments filtraba por el nombre de la carpeta de resultados mientras que run_experiment tomaba el nombre de la clase, dos vocabularios para un mismo concepto. El modelo se recuperó por sí solo, a costa de un turno, y la razón de su turno 3 dice exactamente lo que dedujo. list_experiments ahora acepta cualquiera de las dos grafías.
Lo que mostraron las demostraciones
Cinco tareas, ninguna respondible en una sola llamada. Transcripciones completas en demo_output/, rastros completos en traces/.
A. Mejor lógica dentro de un presupuesto de precisión. Tres turnos. Listó las ejecuciones, obtuvo los cuatro resultados en un turno paralelo, respondió YG con 0.9981 de seguridad por 0.76 puntos de precisión, y dijo que no se ejecutó nada.
B. Planificar una variante de una ejecución existente. Cuatro turnos. Obtuvo configuración, comparación y definición de lógica en un turno paralelo, llamó a run_experiment con dry_run=true, informó el plan y el comando exacto, y, como existía una ejecución coincidente, las comparó.
C. Comparar contra una ejecución que no existe. Tres turnos. Listó primero en lugar de adivinar, confirmó que STL es una lógica real que simplemente no tiene ejecución, y lo dijo.
D. Entrenar, el operador deniega. Seis turnos. Planeó primero con una ejecución en seco, como pide la descripción de la herramienta, luego solicitó la ejecución. El aprobador denegó. El modelo informó que no se ejecutó y no reintentó, dio el plan y respondió con lo que existía.
E. Entrenar, el operador aprueba. Seis turnos, y una ejecución de entrenamiento real. Misma secuencia de planificar y luego ejecutar; el aprobador aceptó; el servidor, iniciado con la ejecución habilitada, ejecutó main.py durante una época en CPU en 28.6 segundos y escribió fixtures/results/standard-robustness/mnist/1/YG.csv. El agente luego llamó a get_results y compare_runs en la nueva ejecución e informó Test-P-Metric final 0.9160 y Test-C-Sec-self 0.5482. Ambos coinciden con el CSV. Sin que se le pidiera, enumeró los factores de confusión contra la comparación de semilla 0 (una época frente a diez, retardo, un presupuesto de ataque deliberadamente debilitado) y observó a partir de la fila de la época 0 que la seguridad de la restricción es trivialmente 1.0 en un modelo no entrenado y solo significa algo junto a una precisión convergente. Esa es una lectura correcta de la métrica.
Ese CSV de semilla 1 es una ejecución real y se mantiene junto a los fixtures sintéticos a propósito. Su primera línea es el argv con el que se entrenó, como cualquier otra ejecución.
Dos cosas que vale la pena saber sobre los datos
La época 0 es una evaluación previa al entrenamiento. Una ejecución configurada con --epochs 10 escribe once filas numeradas del 0 al 10. El número de filas y la época final se informan por separado, porque llamar al número de filas "épocas" sobreestima el entrenamiento en uno.
El script de entrenamiento escribe -1 para las métricas que no evaluó. get_results normaliza esos valores a nulo, por lo que un centinela no puede leerse como una medición. Una ejecución de referencia no tiene métricas de restricción en absoluto, y debería decirlo en lugar de informar menos uno.
Límites, declarados para no exagerar
El modelo nunca encontró un resultado de herramienta is_error en vivo en las cinco tareas, porque siguió la instrucción de listar antes de confiar en un identificador. La ruta de error se prueba a nivel de protocolo en smoke_test.py y a nivel de envoltorio, pero no se demostró la recuperación en vivo de un error de herramienta a mitad de tarea.
La memoización nunca se activó en vivo. El modelo no repitió una llamada idéntica en ninguna ejecución. Está probada unitariamente e inactiva en todos los rastros.
El almacenamiento en caché de mensajes no está configurado. cache_read_input_tokens es cero en todos los rastros, y los recuentos de tokens de entrada (11k a 46k por tarea) son en su mayoría contexto reenviado. Los puntos de interrupción de caché en las definiciones de herramientas y el mensaje del sistema lo reducirían sustancialmente y son la mejora obvia siguiente.
Ejecutar una ejecución requirió un checkout cuyo main.py analiza. En el main ascendente no lo hace: --epsilon y --delta están definidos dos veces cada uno y argparse rechaza el duplicado antes de que se lea cualquier argumento, por lo que python main.py --help falla. Eso está corregido en la rama fix/duplicate-argparse-flags del fork, con una prueba de regresión, y la demostración apuntó PDML_REPO_DIR a ese checkout.
Pruébalo
uv sync
uv run python scripts/make_fixtures.py
uv run python scripts/smoke_test.pyLa prueba de humo inicia el servidor a través de stdio, enumera las herramientas, llama a cada una, comprueba que la ejecución sin PDML_ALLOW_EXECUTE es rechazada y comprueba que un id de experimento desconocido da error en lugar de tener éxito silenciosamente. No cuesta nada.
Para preguntarle algo al agente, con ANTHROPIC_API_KEY configurada:
uv run python -m pdml_agent.agent "Which mnist run has the best constraint security?"
uv run python scripts/demo.py A B C DPara permitirle entrenar realmente, apúntalo a un checkout de property-driven-ml cuyo main.py analice y a un intérprete con torch, luego pasa el indicador que habilita la ejecución:
export PDML_REPO_DIR=~/property-driven-ml
export PDML_PYTHON=~/property-driven-ml/.venv/bin/python
uv run python -m pdml_agent.agent --allow-execute "Train a one-epoch YG run on mnist at seed 2 ..."
uv run python scripts/demo.py ESe te mostrará la llamada exacta y se te pedirá que la apruebes.
Variables de entorno que el servidor lee: PDML_RESULTS_DIR (dónde viven las ejecuciones, por defecto fixtures/results), PDML_REPO_DIR (el checkout de property-driven-ml), PDML_PYTHON (intérprete para main.py, si no el .venv del repositorio), PDML_ALLOW_EXECUTE (1 para permitir la ejecución), PDML_EXECUTE_TIMEOUT (segundos, por defecto 3600).
Available Tools
6 toolscompare_runsA
Diff two runs on both configuration and final headline metrics.
Returns only the config keys that differ, plus deltas on predictive
performance and constraint security. Prefer this over calling get_results
twice when the question is what changed between two runs.
ParametersJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| experiment_id_a | Yes | ||
| experiment_id_b | Yes |
TDQS
A4.3/5.0
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It discloses that the tool returns only differing config keys and deltas on predictive performance and constraint security, which is informative. However, it does not explicitly state that the tool is read-only (no destructive side effects), which would be a minor improvement for transparency.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is three sentences, each serving a distinct purpose: stating the action, detailing the output, and providing usage guidance. No extraneous words; every sentence earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's low complexity (two string parameters, no output schema, no annotations), the description covers the core functionality and output format. It mentions what is returned (differing config keys, deltas on metrics). However, it could be more complete by describing the return structure (e.g., a JSON object with keys for config and metrics). The lack of output schema makes this gap more noticeable, but the description is still sufficient for an agent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters2/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, meaning the input schema provides no property descriptions. The description does not mention the parameter names 'experiment_id_a' and 'experiment_id_b' or explain their format, source, or constraints. While the names are self-explanatory, the description should add detail (e.g., 'unique identifiers for runs') to compensate for the lack of schema descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'diff' and the resource 'two runs', specifying that it compares configuration and headline metrics. It distinguishes itself from the sibling tool 'get_results' by mentioning it returns only differing config keys and deltas, making its purpose unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines5/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly advises to prefer this tool over calling 'get_results' twice when the question is about what changed between two runs. This provides a clear usage guideline and directly contrasts with an alternative, helping the agent decide when to invoke this tool.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_experiment_configA
Recover the exact configuration a run was trained with.
Reads the command line the training script recorded in the run's own output, so this is what actually ran rather than what was intended. Use it before proposing a new run based on an existing one.
ParametersJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| experiment_id | Yes |
TDQS
A4.4/5.0
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It discloses that the tool reads the command line recorded in the run's own output, and emphasizes that it recovers 'what actually ran rather than what was intended'. This is valuable behavioral context for a read-only introspection tool, though it does not mention auth requirements or rate limits.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is three concise sentences, each adding value. The first sentence states the purpose, the second explains the mechanism, and the third provides usage guidance. No wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has no output schema, the description appropriately skips explaining return values. It covers what the tool does, how it works, and when to use it. However, it does not mention the format of the returned configuration (e.g., as a JSON object or command-line string), which would be helpful for the agent to know how to use the output.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0% (the parameter names are not described in the schema), so the description must compensate. The description does not directly describe the 'experiment_id' parameter, but the context of 'recover the exact configuration a run was trained with' strongly implies that the parameter identifies the run. With only one required parameter and no enums, this is sufficiently clear.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'recover' and the resource 'exact configuration a run was trained with', distinguishing it from siblings like 'list_experiments' (which lists experiments) and 'run_experiment' (which creates a new run). It explains that the configuration is read from the run's own output, adding specificity.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly says 'Use it before proposing a new run based on an existing one', providing clear guidance on when to use this tool. However, it does not explicitly state when not to use it or mention alternatives among the siblings, though the context implies it's for inspection before modification.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_resultsA
Metrics for one epoch of a run, defaulting to the final epoch.
Constraint security is reported as Test-C-Sec-self and Test-C-Sec-common; predictive performance is Test-P-Metric. Metrics the run did not evaluate are returned as null rather than as the -1 sentinel the training script writes, so a missing measurement cannot be mistaken for a real one.
ParametersJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| epoch | No | ||
| experiment_id | Yes |
TDQS
A3.8/5.0
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description fully carries the burden of behavioral disclosure. It reveals key behaviors: the default epoch is the final one, the naming convention for constraint security and predictive performance metrics, and crucially that missing metrics are returned as null instead of the -1 sentinel used by the training script. This eliminates ambiguity and prevents misinterpretation of sentinel values.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is composed of three sentences that are concise and front-loaded with the primary purpose. It wastes no words, though it could be slightly more structured by separating the parameter default from the metric naming conventions. Overall, it is efficient and well-organized.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has 2 parameters, no output schema, and no annotations, the description covers the default behavior, metric naming, and null handling. However, it fails to describe the return format (e.g., dictionary or list), possible error conditions (e.g., invalid experiment_id), or the full set of metrics. It is adequate but leaves gaps for an agent to infer.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It does so by explaining the default behavior of the epoch parameter (defaulting to final epoch) and implicitly linking experiment_id to the run. However, it does not explicitly describe the parameters' types, constraints, or the experiment_id parameter's purpose beyond the tool context. The added value is moderate but not fully compensatory.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool retrieves metrics for a specific epoch of a run, defaulting to the final epoch. This is a specific verb-resource combination that distinguishes it from siblings like list_experiments (listing all experiments) or get_experiment_config (getting configuration). The mention of defaulting to the final epoch adds precision.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no explicit guidance on when to use this tool versus alternatives. It does not mention when not to use it, nor does it reference sibling tools like compare_runs or search_logic_definitions. The usage context is only implied by the tool's purpose (getting metrics), but no exclusions or prerequisites are stated.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_experimentsA
List completed experiment runs, with optional filters.
Returns one entry per run with its experiment_id, the constraint, dataset,
seed and logic it used, and how many epochs it recorded. Call this first:
every other experiment tool takes an experiment_id that comes from here.
The constraint filter accepts either the class name (StandardRobustness)
or the folder name that appears in ids (standard-robustness).
ParametersJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| logic | No | ||
| dataset | No | ||
| constraint | No |
Output Schema
ParametersJSON Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
A4.3/5.0
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries full behavioral burden. It describes the return structure (fields: experiment_id, constraint, dataset, seed, logic, epochs) and explains constraint filter behavior (accepts class name or folder name). This adds meaningful transparency beyond just 'list runs'.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is three concise sentences, each serving a purpose: stating the action, detailing the output, and providing usage guidance. No redundancy or unnecessary words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity and the presence of an output schema, the description covers all essential aspects: what is returned, the filter format, and the inter-tool dependency. It is complete for effective use.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It provides specific detail for the constraint parameter (accepts class name or folder name) but does not describe the logic or dataset parameters beyond being optional filters. Some added value, but incomplete.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly specifies the verb 'List', the resource 'completed experiment runs', and notes optional filters. It differentiates from siblings by stating that this tool returns experiment_ids used by other experiment tools, making its role distinct.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly instructs to call this tool first because every other experiment tool takes an experiment_id from here. This provides clear context for usage, though it does not mention when not to use it or name specific alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
run_experimentA
Plan a training run, or execute one. The only tool that consumes compute.
With dry_run=true (the default) it validates every argument, returns the
exact command, and touches nothing. Do that first and show the plan.
With dry_run=false it executes, subject to two independent gates: the
operator must approve the exact call, and the server must have been
started with execution enabled. If either refuses, the result says
status refused or declined_by_operator. Do not retry a refused or
declined call; report it. A completed run returns the experiment_id to
pass to get_results. Training takes minutes even for one epoch;
oracle_steps and oracle_restarts control the adversarial attack cost.
ParametersJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| lr | Yes | ||
| seed | No | ||
| delay | No | ||
| logic | Yes | ||
| epochs | Yes | ||
| dataset | Yes | ||
| dry_run | No | ||
| epsilon | No | ||
| batch_size | Yes | ||
| constraint | Yes | ||
| results_dir | No | ||
| oracle_steps | No | ||
| oracle_restarts | No |
TDQS
A4.4/5.0
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description fully discloses behavior: dry_run validates without side effects, execution requires operator approval and server enablement, possible statuses are named, and training time is estimated ('minutes even for one epoch'). Also mentions oracle_steps/restarts control adversarial attack cost.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured and front-loaded with the key distinction. Each sentence provides essential operational guidance—no filler or repetition. The three paragraphs flow logically from purpose to validation to execution.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a complex compute-consuming tool with no output schema, the description covers safety gates, refusal handling, time cost, and next steps (get_results). It falls short only in not explaining parameter meanings, but overall it is remarkably complete for an agent to use safely.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters2/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It explains dry_run and oracle_steps/oracle_restarts, but 10 of 13 parameters (dataset, constraint, logic, epochs, batch_size, lr, seed, delay, epsilon, results_dir) have no semantic explanation beyond their names. This is a notable gap for critical parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it 'Plan a training run, or execute one. The only tool that consumes compute.' This specifies the verb (plan/execute), resource (training run), and distinguishes from sibling read-only tools like get_results and list_experiments.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines5/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides explicit workflow guidance: 'Do that first and show the plan' (dry-run), 'Do not retry a refused or declined call; report it', and 'A completed run returns the experiment_id to pass to get_results'. It also notes it is the only compute-consuming tool, indicating when this tool is appropriate.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
search_logic_definitionsA
Find differentiable logic implementations in the source.
Matches on class name, docstring and filename, returning the operators each logic implements and where it is defined. An empty query returns all of them. Use this to understand what a logic does before interpreting a result or proposing a run that uses it.
ParametersJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| query | No |
TDQS
A3.8/5.0
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description must carry the full burden of behavioral disclosure. It tells the user about search scope and return type (operators and location) but does not disclose whether the operation is read-only, its performance characteristics, or any side effects like network calls. For a search tool, this is adequate but not rich in detail.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is three sentences and efficiently front-loads the core purpose in the first sentence, followed by scope details and usage guidance. Each sentence contributes distinct information without redundancy, though a slightly more condensed phrasing could improve it further.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has a single parameter with no enums, no output schema, and no nested objects, the description adequately covers the search behavior, parameter semantics, and usage context. It explains what fields are searched and how the empty query works, which is sufficient for an agent to use the tool effectively in the context of understanding logic implementations.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has 1 parameter (query) with 0% description coverage and no schema-level descriptions, so the description must compensate. The description explains the query parameter well: it matches on class name, docstring, and filename, and states that an empty query returns all entries. This adds meaningful semantics beyond the schema's default field.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool searches for differentiable logic implementations, specifying the search scope (class name, docstring, filename) and what it returns (operators and location). This is specific and distinguishable from siblings like get_results or run_experiment, though it does not explicitly name a sibling for differentiation.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear context for when to use the tool, explicitly stating 'Use this to understand what a logic does before interpreting a result or proposing a run that uses it.' This gives actionable guidance on the tool's role in the workflow, though it does not mention when not to use it or list alternatives explicitly.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
6 tool updates
v0.1.0- First observed
compare_runs - First observed
get_experiment_config - First observed
get_results - First observed
list_experiments - First observed
run_experiment - First observed
search_logic_definitions
TDQS
A4.2/5.0
Scored across 6 tools
Disambiguation5/5
Each tool has a clearly distinct responsibility: listing experiments, retrieving configs, fetching metrics, comparing runs, searching logic definitions, and executing runs. There is no overlap in purpose, and run_experiment is the only side-effecting tool.
Naming Consistency5/5
All six tool names follow a consistent snake_case verb_noun pattern (list_experiments, get_results, compare_runs, etc.). The verbs are semantically appropriate and predictable.
Tool Count5/5
Six tools is well-scoped for an experiment management agent: discovery, configuration, results, comparison, domain search, and execution. Each tool earns its place without redundancy.
Completeness4/5
The tool surface covers the full core workflow: plan/execute a run, list runs, inspect configuration and metrics, and compare runs. Missing operations like deleting or cancelling runs are not obvious gaps for immutable experiment records.
Maintenance
ActivitySlowing
ResponsivenessNo issues
Related MCP Connectors
MCP server for building and testing AI agents with multi-model experimentation and insights.
MCP server providing access to the Scorecard API to evaluate and optimize LLM systems.
MCP server for generating rough-draft project plans from natural-language prompts.
MCP server for progressive tool usage at any scale (see https://klavis.ai)
Related MCP Servers
- AlicenseAqualityBmaintenanceA Model Context Protocol server that enables LLMs to interact with MLflow tracking servers, allowing users to query experiments, analyze runs, compare metrics, manage the model registry, and promote models through natural language.40148 PyPI15MIT
- AlicenseAqualityAmaintenanceMCP server for Coalesce that manages nodes, pipelines, environments, jobs, and runs, and enables project validation, DDL/DML preview, deployment planning, and cloud environment application.100195 npm2MIT
- AlicenseBqualityDmaintenanceA standalone MCP server that brings complete data science capabilities to AI assistants, enabling them to load data, train models, and track experiments through natural language.301MIT
