Skip to main content
Glama
Sign In
llmledger

llmledger

by pouriaetab
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python
Local

Log Run

log_run

Record model call timings or promptfoo results in a local file to improve time estimates from stored tokens, seconds, and call method.

Instructions

Record a timed run (model, tokens, seconds, how it was called), or every result of a promptfoo results file, in a file on this computer so estimate_time can learn from it. Use right after a call or a promptfoo run whose time is known. Not a query: it stores and returns what it stored. Nothing is sent anywhere.

Tool call template

This tool declares no required parameters. Add optional parameters from the input schema as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {},
    "name": "log_run"
  }
}

Input Schema

NameRequiredDescriptionDefault
wayNoHow the model was called (default api; for a promptfoo file, read from each provider)
taskNoThe kind of work, if known
modelNoThe model that ran
secondsNoHow long the call took
thinkingNoThinking was on (a promptfoo result that counts thinking tokens is marked on by itself)
input_tokensNoPrompt tokens of the call
output_tokensNoAnswer tokens of the call, hidden thinking included
promptfoo_resultsNoPath of a promptfoo results file (promptfoo eval -o results.json): every result becomes a run

Output Schema

NameRequiredDescriptionDefault

No arguments

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv0.1.7

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With annotations already declaring non-destructive, local (openWorldHint=false) and non-idempotent, the description adds real value: writes to a file 'on this computer', 'Nothing is sent anywhere' (privacy/data-locality guarantee an agent may need to relay), and 'returns what it stored'. It does not warn that repeated calls append duplicate records despite idempotentHint=false, which is the one consequence an agent would most want spelled out.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three tight sentences: what it records first, when to use it second, and the non-query/locality caveats last. No filler and no repetition of schema detail.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Output schema exists and annotations cover safety, so the description only needs purpose, trigger and behavioral caveats — all present. The main residual gap is what happens on repeated logging (duplicate records vs. update) and where the file lives, which matters for a stateful writer.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so every parameter is already documented in the schema, including the promptfoo_results path semantics and the thinking auto-detection. The description only re-lists 'model, tokens, seconds, how it was called' without adding format, precedence, or mutual-exclusion rules between a single run and a results file. Baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Record a timed run') plus the two supported input modes (single run with model/tokens/seconds/way, or a whole promptfoo results file) and the downstream consumer (estimate_time). An agent can distinguish it from estimate_time or forecast_job without inspecting any schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

'Use right after a call or a promptfoo run whose time is known' gives a clear timing condition, and 'Not a query: it stores and returns what it stored' explicitly rules out the wrong usage pattern. It stops short of naming which sibling to call instead when the agent does want a query, so it's a strong 4 rather than a 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools