Skip to main content
Glama

eschaton_dbms_eval

Grades SQL database queries across SQLite, PostgreSQL, MySQL, and Oracle by seeding an ephemeral database per testcase and comparing query output against expected results.

Instructions

Automated SQL database grading across SQLite, PostgreSQL, MySQL, and Oracle: seeds an ephemeral database per testcase with schema_sql DDL/DML and evaluates queries.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "source_code": "<source_code>",
      "schema_sql": "<schema_sql>"
    },
    "name": "eschaton_dbms_eval"
  }
}

Input Schema

NameRequiredDescriptionDefault
stdinNoOptional standard input
databaseNoDatabase engine backendsqlite
schema_sqlYesDDL/DML SQL statements to seed the test database
source_codeYesPython database query/app code
expected_outputNoExpected query stdout

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv1.0.6

TDQS

B3.2/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are supplied, so the description carries the full burden. It usefully discloses that a database is ephemeral and created per testcase, which tells the agent runs are isolated and state is disposable. It says nothing about execution limits, timeouts, sandboxing of arbitrary source_code, cost, or what the grading result contains.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single front-loaded sentence that puts the engines and the per-testcase seeding mechanism first with no filler. It is efficient, though it compresses so much that the evaluation semantics are left implicit.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a five-parameter tool with no annotations and no output schema, the description should say more about what grading produces (pass/fail, stdout comparison against expected_output) and about safety when running arbitrary Python code. The seeding mechanism is well covered, but the result and execution semantics are gaps.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so all five parameters, including the database enum, are already documented in the schema and the baseline is 3. The description adds only that schema_sql seeds the database and that queries are evaluated, which slightly reinforces the required parameters but adds no format or interaction details beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names a specific verb+resource ('Automated SQL database grading') and scopes it to four named engines, plus it explains the core mechanism of seeding a testcase database. It stops short of differentiating itself from siblings such as eschaton_execute_suite or eschaton_fast_batch, so an agent cannot fully rule out overlap from the text alone.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no explicit when-to-use or when-not-to-use guidance and no named alternative among the many eschaton_* and mnemon_* siblings. The grading context is only implied by the word 'grading', leaving the agent to infer that this is for evaluating submitted SQL against a seeded database rather than for general execution or profiling.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools