Tuning Engines
The Tuning Engines server lets you fine-tune open-source LLMs/SLMs on your GitHub repos and manage the full ML lifecycle via CLI or MCP tools.
Training Jobs
Create fine-tuning jobs using agents like Cody (code autocomplete) or SIERA (bug-fix specialist) on GitHub repos
Estimate costs before submission; monitor live status including GPU usage and estimated charges
List, view, cancel, and retry failed jobs from checkpoints
Validate S3 credentials before submitting jobs with S3 export
Models
List, view, and delete trained/imported models
Import from or export to S3; check import/export status
Browse supported base models (Qwen, Llama, DeepSeek, Mistral, CodeLlama, StarCoder, Gemma, Phi — 1B to 72B parameters)
Use LoRA, QLoRA, or full fine-tuning; iteratively fine-tune previously trained models
Datasets
Create, list, view, and delete datasets sourced from S3 for training or evaluation
Evaluations
Create, list, cancel, and monitor evaluations against datasets using evaluators like code execution, similarity, or LLM judge
Estimate evaluation costs and view detailed scores/metrics
Inference
List available inference models, view API usage stats, and retrieve JWT tokens for direct API access
Agents
List and view details of available specialized training agents
Account & Billing
Check balance, view transaction history, add credits, and manage account details
Configuration
Set API key, override API URL, and view current config
Allows the server to use GitHub repositories as the data source for fine-tuning LLMs and SLMs, enabling the creation of specialized agents like 'Cody' for code autocomplete and 'SIERA' for bug-fix specialization.
Tuning Engines CLI & MCP Server
Govern every AI workflow through one API.
Tuning Engines is a governed AI runtime for model, agent, skill, and MCP workflows. Route inference through one OpenAI-compatible API, apply RBAC and traffic policies, request approvals for high-risk actions, inspect traces and usage, and connect durable orchestration frameworks such as LangGraph and Temporal. The same CLI and MCP server also manage domain-specific fine-tuning of open-source models.
Training Agents
Tuning Engines uses specialized agents that control how your data is analyzed and converted into training data. Each agent produces a different kind of domain-specific fine-tuned model optimized for its use case. Current agents focus on code, with more coming for customer support, data extraction, security review, ops, and other domains.
Cody (code_repo) — Code Autocomplete Agent
Cody fine-tunes on your GitHub repo using QLoRA (4-bit quantized LoRA) via the Axolotl framework (HuggingFace Transformers + PEFT). It learns your codebase's patterns, naming conventions, and project structure to produce a fast, lightweight adapter optimized for real-time completions.
Best for: code autocomplete, inline suggestions, tab-complete, code style matching, pattern completion.
te jobs create --agent code_repo \
--base-model Qwen/Qwen2.5-Coder-7B-Instruct \
--repo-url https://github.com/your-org/your-repo \
--output-name my-cody-modelSIERA (sera_code_repo) — Bug-Fix Specialist
SIERA (Synthetic Intelligent Error Resolution Agent) uses the Open Coding Agents approach from AllenAI to generate targeted bug-fix training data from your repository. It synthesizes realistic error scenarios and their resolutions, then fine-tunes a model that learns your team's debugging style, error handling conventions, and fix patterns.
Best for: debugging, error resolution, patch generation, root cause analysis, fix suggestions.
te jobs create --agent sera_code_repo \
--quality-tier high \
--base-model Qwen/Qwen2.5-Coder-7B-Instruct \
--repo-url https://github.com/your-org/your-repo \
--output-name my-siera-modelQuality tiers (SIERA only):
low— Faster, fewer synthetic pairs (default)high— Deeper analysis, more training data, better results
Coming Soon
Agent | Persona | What it does |
Resolve | Mira | Fine-tunes on support tickets, macros, and KB articles for automated ticket resolution |
Extractor | Flux | Trains for strict schema extraction from docs, PDFs, and business text |
Guard | Aegis | Security-focused code reviewer that catches risky patterns and proposes safer fixes |
OpsPilot | Atlas | Incident response agent trained on runbooks, postmortems, and on-call notes |
Related MCP server: ML Lab MCP
Supported Base Models
Size | Models |
3B |
|
7B |
|
13-15B |
|
32-34B |
|
70-72B |
|
Quick Start
npm install -g tuningengines-cli
# Or run without installing
npx -y --package tuningengines-cli@latest te auth status
# Sign up or log in (opens browser — works for new accounts too)
te auth login
# Add credits (opens browser to billing page)
te billing add-credits
# Estimate cost before training
te jobs estimate --base-model Qwen/Qwen2.5-Coder-7B-Instruct
# Train Cody on your repo
te jobs create --agent code_repo \
--base-model Qwen/Qwen2.5-Coder-7B-Instruct \
--repo-url https://github.com/your-org/your-repo \
--output-name my-model
# Monitor training
te jobs status <job-id> --watch
# View your trained models
te models list
# Create a governed orchestration starter
te orchestration init langgraph
te orchestration init temporal
te orchestration init inngest
te orchestration init triggerdev
te orchestration init hatchet
te orchestration init restate
te orchestration init dbos
te orchestration init dapr
te orchestration init prefect
te orchestration init dagster
te orchestration init airflowMCP Server Setup
The CLI includes a built-in MCP server with 60+ tools. Any AI assistant that supports MCP can fine-tune models, manage training jobs, run evaluations, check inference usage, inspect traces, review approvals, and manage non-secret tenant registry metadata through natural language.
For security, the MCP server intentionally does not expose internal proxy routes. It also refuses MCP-side inference-key creation and raw secret-bearing mutation fields. Use the CLI or web UI for workflows that intentionally create one-time keys, submit raw provider secrets, validate S3 credentials, or import/export S3 assets with raw credentials.
Claude Desktop
Add to ~/Library/Application Support/Claude/claude_desktop_config.json:
{
"mcpServers": {
"tuning-engines": {
"command": "npx",
"args": ["-y", "--package", "tuningengines-cli@latest", "te", "mcp", "serve"],
"env": {
"TE_API_KEY": "te_your_key_here"
}
}
}
}Claude Code
claude mcp add tuning-engines -- npx -y --package tuningengines-cli@latest te mcp serveWork Sessions and outcomes
Label the desired outcome for a project without interrupting your coding workflow:
te goal start "Fix flaky checkout retries"
te goal show
te goal complete --result succeededInstall optional native telemetry hooks for Claude Code or Codex:
te guard claude-code install --mode observe --project .
te guard claude-code doctor
te guard claude-code doctor --probe
te guard codex install
te guard codex doctorClaude Code writes project-local hooks into .claude/settings.local.json. On
Windows, verify with dir .\.claude, type .\.claude\settings.local.json,
then restart Claude Code from the same project root and review claude /hooks.
doctor --probe is available in tuningengines-cli 0.4.20 and later; it runs
synthetic hook events through the installed commands and checks that the trace is
visible to Tuning Engines. Hook invocations also write a local redacted status
log at .claude/tuning-engines-hook-status.jsonl.
Codex project hooks require review and trust from /hooks. The installer pins
hooks to the exact CLI executable that created them; te guard codex doctor
reports that path, CLI version, native-event contract, required lifecycle
hooks, and any conflicting te command found on PATH. Each Codex or Claude
Code user prompt starts a new trace while retaining one stable Work Session for
the native conversation. Tuning Engines sends pseudonymous session and
transcript references by default, not transcript contents or local absolute
paths.
Claude Code Plugin
The repository also ships a Claude Code plugin wrapper around the same MCP
server. It keeps installation discoverable while preserving the same
TE_API_KEY environment-variable boundary:
claude plugin marketplace add cerebrixos-org/tuning-engines-cli
claude plugin install tuning-engines@tuning-enginesThe same plugin directory includes a Codex plugin manifest, marketplace entry, MCP server, and tenant-operations skill. It uses the Tuning Engines web app as the shared control-plane UI for traces, approvals, policies, cost, and review.
DeepSeek Harness Plugin
Install the native Harness bundle to capture turn, model-step, and tool lifecycle telemetry and optionally enforce TE governance before tool execution:
export TE_API_KEY="your-tenant-token-or-inference-key"
dsh plugin --profile default add tuningengines-cliThe adapter uses a disk-backed local spool, background batching, bounded retries, stable tool-call correlation, and metadata-only capture by default.
VS Code / Cursor / Windsurf
Add to your MCP settings (.vscode/mcp.json or equivalent):
{
"servers": {
"tuning-engines": {
"command": "npx",
"args": ["-y", "--package", "tuningengines-cli@latest", "te", "mcp", "serve"],
"env": {
"TE_API_KEY": "te_your_key_here"
}
}
}
}What the AI assistant can do
When connected, your AI assistant can:
"Fine-tune Qwen 7B on my-org/my-repo using the SIERA agent with high quality"
"How much would it cost to train a 32B model for 3 epochs on this repo?"
"Check the status of my latest training job"
"List my trained models"
"Export my model to s3://my-bucket/models/"
"Show my account balance"
"Train a bug-fix specialist on this repo" (auto-selects SIERA)
"Create an autocomplete model for this codebase" (auto-selects Cody)
The create_job tool description includes full agent details and model lists, so AI assistants automatically select the right agent and model based on what you ask for.
Unified API Endpoint
Tuning Engines can be used anywhere a tool accepts an OpenAI-compatible API base URL. Point the client at:
https://api.tuningengines.com/v1Use an inference key that starts with sk-te-... for live model calls, and use
the model IDs shown by:
te inference modelsThis lets OpenCode, Temporal activities, LangGraph apps, OpenAI SDK clients, and other custom-provider clients route through the same Tuning Engines control plane for model RBAC, routing, fallbacks, guardrails, AGT policy, traces, usage metering, and cost attribution.
See docs/unified-api-endpoint.md for copy-paste examples for OpenCode, Temporal, Python, JavaScript, and other OpenAI-compatible clients.
Agent Runtime SDK and Orchestration Starters
Use the CLI/MCP package when you want npx tools for assistants. Use the
Python SDK when you want your own app to run durable agent workflows while
Tuning Engines remains the governed control plane for models, agents, skills,
MCP tools, RBAC, AGT policy, audit, usage, and token economics.
OpenAI Agents SDK users can install the native model and trace integration:
pip install "tuning-engines[openai-agents]"See packages/tuning-agents/README.md for setup. The OpenAI Agents runtime owns the loop; TE supplies the governed endpoint and receives metadata-only SDK traces through its background processor.
Install the published Python SDK:
pip install "tuning-engines[langgraph]"
pip install "tuning-engines[temporal]"The package installs the tuning_agents Python module.
LangGraph example:
from langgraph.checkpoint.memory import InMemorySaver
from tuning_agents import TuningClient
from tuning_agents.langgraph import create_tuning_langgraph_agent, invoke_with_trace
client = TuningClient(api_key="te_your_key_here")
agent = create_tuning_langgraph_agent(
client,
model="llama-3.3-70b-fp8",
agent_names=["billing-escalation"],
checkpointer=InMemorySaver(),
interrupt_before=["tools"],
)
result = invoke_with_trace(
client,
agent,
[{"role": "user", "content": "Triage this ticket and escalate if needed."}],
thread_id="ticket-123",
)
client.flush_trace(name="ticket-triage", runtime="langgraph", status="succeeded")Temporal example:
from tuning_agents.temporal import (
TuningEnginesTemporalFeatures,
create_tuning_engines_plugin,
define_temporal_workflow,
)
plugin = create_tuning_engines_plugin(
features=TuningEnginesTemporalFeatures(
built_in_workflow=False,
model_calls=True,
skill_tools=True,
mcp_tools=True,
agents=True,
approvals=True,
traces=True,
state_references=True,
interventions=True,
model_catalog=True,
usage=True,
)
)
TuningAgentWorkflow = define_temporal_workflow()
# Pass plugin to Client.connect(..., plugins=[plugin]) and register the workflow.The SDK captures runtime events from LangGraph/Temporal and posts them to
POST /api/v1/traces. Each event carries a run_id, request_id, and a
normalized event type such as model.call, mcp.tool_call, agent.message,
workflow.step, human.edit, action.finalized, outcome.recorded, or
state.reference. The app pairs that with inference usage, request capture,
policy decisions, approval requests, external state references, audit, and
billing logs.
JavaScript/TypeScript users can also import lightweight tracing helpers from the npm package:
import { createOpenAIAgentsTraceAdapter } from "tuningengines-cli/adapters/openai-agents";
import { createClaudeAgentSdkTraceAdapter } from "tuningengines-cli/adapters/claude-agent-sdk";Both helpers send redacted run, model, tool, handoff, error, goal, and outcome
events to the existing trace API. goal_key, goal_status, and goal_score
are normalized into the same success-signal analytics as outcome_key.
For decision traces, store redacted signals in metadata.decision, for example
proposal_summary, changed_fields, change_summary, final_action,
outcome_label, and reason_summary. Do not place raw prompts, provider keys,
tenant secrets, or full customer data in trace metadata.
Generate a starter kit:
te orchestration init langgraph --dir ./lg-te-demo
te orchestration init temporal --dir ./temporal-te-demo
te orchestration init inngest --dir ./inngest-te-demo
te orchestration init triggerdev --dir ./trigger-te-demo
te orchestration init hatchet --dir ./hatchet-te-demo
te orchestration init restate --dir ./restate-te-demo
te orchestration init dbos --dir ./dbos-te-demo
te orchestration init dapr --dir ./dapr-te-demo
te orchestration init prefect --dir ./prefect-te-demo
te orchestration init dagster --dir ./dagster-te-demo
te orchestration init airflow --dir ./airflow-te-demoLangGraph and Temporal starters use the Python runtime SDK. Inngest, Trigger.dev, and Hatchet starters generate TypeScript projects with a small self-contained Tuning Engines helper. Restate, DBOS, and Dapr starters use the same TypeScript helper. Prefect, Dagster, and Airflow starters generate Python workflow examples with a small helper module. All generated examples include governed model calls, trace flushing, registry manifests, policy context metadata, decision metadata, runtime state references, and approval retry patterns.
CLI Commands
Authentication
Command | Description |
| Sign up or log in via browser |
| Clear saved credentials |
| Show current auth status (email, balance) |
Training Jobs
Command | Description |
| List all training jobs |
| Show job details |
| Submit a training job ( |
| Live status ( |
| Cancel a running job |
| Retry from last checkpoint |
| Cost estimate before submitting |
| Pre-validate S3 credentials |
Models
Command | Description |
| List your trained models |
| Show model details |
| List supported base models |
| Import a model from S3 |
| Export a model to S3 |
| Delete a model |
| Check import/export status |
Datasets
Command | Description |
| List all datasets |
| Show dataset details |
| Create a dataset from S3 ( |
| Delete a dataset |
| Check import/processing status |
Evaluations
Command | Description |
| List all evaluations |
| Show evaluation details and scores |
| Run an evaluation ( |
| Cancel a running evaluation |
| Live evaluation progress |
| List available evaluators |
| Cost estimate for an evaluation |
Inference
Command | Description |
| List available inference models |
| Show inference API usage stats |
| Get a JWT for direct API access |
| Exchange an inference key ( |
| Run an OpenAI-compatible chat completion |
| Run an OpenAI Responses request |
| Create embeddings |
| Run an Anthropic-compatible Messages request |
Runtime Traces and Approvals
Command | Description |
| List LangGraph, Temporal, and custom runtime traces |
| Show one trace, including events, policy decisions, and approvals when linked |
| Ingest or update a trace using a user API token or inference key |
| List observed outcomes, goals, evals, and workflow success signals |
| Record a success signal for a run |
| Map unmapped events to an outcome key |
| List Insight Loop recommendations |
| Accept an insight as valid; does not change production |
| Apply or queue the approved action for an accepted insight |
| Simulate inference access, role, endpoint, policy, and resource checks |
| List AGT YAML policy decisions |
| Show one policy decision with redacted context |
| List curated AGT YAML policy templates |
| Render disabled/shadow policy YAML from safe structured parameters |
| Generate an AI-assisted disabled/shadow draft for review and testing |
| List policy approval requests |
| Show approval detail and retry metadata |
| Approve a pending request |
| Deny a pending request |
| Manage runtime pause, resume, cancel, and replay requests |
| Manage safe external workflow-state and memory references |
| Diff or apply agent, skill, and MCP registry manifests |
| Inspect an applied registry sync |
| Inspect and update Work Sessions |
| Manage strategic initiative groupings |
Orchestration Starters
Command | Description |
| Create a LangGraph starter wired to Tuning Engines governance and traces |
| Create a Temporal worker starter wired to Tuning Engines governance and traces |
| Create an Inngest function starter wired to Tuning Engines governance and traces |
| Create a Trigger.dev task starter wired to Tuning Engines governance and traces |
| Create a Hatchet workflow starter wired to Tuning Engines governance and traces |
| Create a Restate service starter wired to Tuning Engines governance and traces |
| Create a DBOS workflow starter wired to Tuning Engines governance and traces |
| Create a Dapr Workflow starter wired to Tuning Engines governance and traces |
| Create a Prefect flow starter wired to Tuning Engines governance and traces |
| Create a Dagster asset starter wired to Tuning Engines governance and traces |
| Create an Airflow DAG starter wired to Tuning Engines governance and traces |
Agents
Command | Description |
| List available agents |
| Show agent details and capabilities |
| Send a governed A2A agent message |
Skills and MCP Execution
Command | Description |
| List skills visible to the inference identity |
| Prepare a governed skill |
| Invoke a governed skill |
| Call an enabled governed MCP tool |
| Refresh MCP tool discovery |
| Administer discovered MCP tools |
Compliance Automation
Command | Description |
| Validate bounded content against adopted rulepacks |
| Produce and revalidate a safe rewrite |
| Inspect compliance evidence |
| Manage the tenant risk register |
| Operate the risk-to-control lifecycle |
| Ingest normalized scanner or webhook results |
| Run and inspect compliance certification jobs |
Tenant Admin Automation
These commands require an API token for a tenant owner or tenant admin. They are designed for CI smoke tests and end-to-end product checks. Secret fields can be sent on create/update where the server supports them, but responses never print stored provider keys, AWS secrets, or invitation tokens.
Command | Description |
| List supported tenant resource names |
| List resources such as |
| Show one tenant resource |
| Create a tenant resource from JSON |
| Update a tenant resource from JSON |
| Delete a tenant resource; inference keys are revoked |
| Validate/test an unsaved simple guardrail without creating records |
| Validate/test an unsaved Governance Rule without creating records |
| Dry-run a Governance Rule |
| Compatibility alias for governance policy dry-runs |
| List tenant members, pending invitations, and allowed domains |
| Invite a user by email; the invite token is emailed and never printed |
| Assign an inference role to a member |
| Disable a member |
| Re-enable a member |
| Remove a member |
| Cancel a pending invitation |
| Replace allowed email domains |
| Show inference capture settings |
| Update inference capture settings |
Billing & Account
Command | Description |
| Balance and transaction history |
| Open browser to add credits |
| Account info |
Configuration
Command | Description |
| Set API key manually |
| Override API URL |
| Show current config |
All commands support --json for machine-readable output.
MCP Tools Reference
Training Jobs
Tool | Description |
| Fine-tune an LLM on a GitHub repo. Supports agent selection (Cody, SIERA), quality tier, base model, epochs, S3 export. |
| Cost estimate before training. Returns cost range, balance, sufficiency check. |
| List training jobs with status filter |
| Full job details including agent, model, GPU usage, cost, retry info |
| Live status with GPU minutes, charges, delivery progress |
| Cancel a running/queued job |
| Retry a failed job from its last checkpoint |
Models
Tool | Description |
| List trained and imported models |
| Model details (status, size, base model, training job) |
| Delete a model from cloud storage |
| Import/export progress |
| Available base models with GPU hours per epoch |
Marketplace
Tool | Description |
| Browse pre-built models and datasets |
| Details of a marketplace item |
| Check marketplace export progress |
Datasets
Tool | Description |
| List datasets for training and evaluation |
| Dataset details and status |
| Create a dataset from S3 |
| Delete a dataset |
| Check dataset import/processing status |
Evaluations
Tool | Description |
| List model evaluations |
| Evaluation details, scores, and metrics |
| Run an evaluation against a dataset |
| Cancel a running evaluation |
| Live evaluation progress |
| Available evaluators (code_execution, similarity, llm_judge, etc.) |
| Cost estimate for an evaluation |
Inference
Tool | Description |
| Models available for inference |
| Inference API usage statistics |
| Get JWT token for direct API access |
| Exchange an inference key for a short-lived inference JWT |
| Call chat, Responses, embeddings, or Messages using the configured credential |
| Send a governed A2A agent message |
| Discover, prepare, or invoke governed skills |
Runtime, Policy, and Approvals
Tool | Description |
| List runtime traces |
| Show a trace with linked events, policy decisions, and approvals |
| Ingest a trace payload without secrets |
| List observed outcomes/goals normalized as success signals |
| List Insight Loop recommendations |
| Show one Insight Loop recommendation |
| Simulate inference access, role, endpoint, policy, and resource checks |
| Record an outcome/goal signal; requires |
| Create an outcome mapping rule; requires |
| Accept an insight for review; requires |
| Apply or queue an accepted insight; requires |
| List AGT YAML policy decisions |
| Show one decision with redacted context |
| List curated AGT YAML policy templates |
| Render disabled/shadow policy YAML from safe structured parameters |
| Generate an AI-assisted disabled/shadow draft; secret-looking prompts are refused |
| List policy approval requests |
| Show one approval request |
| Approve a pending request |
| Deny a pending request |
| Inspect runtime control requests |
| Operate intervention lifecycle; requires |
| Manage safe external state and memory pointers |
| Diff, apply, and inspect registry manifests |
| Inspect and complete Work Sessions |
| Inspect initiative groupings |
| Inspect the tenant risk register |
| Validate content and inspect evidence |
| Ingest normalized external test results; writes require |
| Inspect a certification run |
Tenant Admin MCP Tools
These tools require a tenant owner/admin API token. The MCP server refuses internal proxy routes, inference-key creation, and raw secret-bearing mutation fields.
Tool | Description |
| List allowlisted tenant resource names |
| List models, roles, policies, MCP servers, agents, skills, credential sources, and related metadata |
| Show one resource without returning stored secrets |
| Create non-secret tenant registry/config metadata |
| Update non-secret tenant registry/config metadata |
| Delete or revoke a tenant resource |
| Validate/test unsaved guardrail or AGT policy payloads without creating records |
| Dry-run an AGT YAML governance policy |
| List members, invitations, and allowed domains |
| Invite a user without returning invitation tokens |
| Assign or clear an inference role |
| Disable or re-enable a member |
| Remove a tenant member |
| Cancel a pending invitation |
| Replace allowed email domains |
| Manage request-capture settings using credential-source references |
Agents
Tool | Description |
| List available agents |
| Agent details and capabilities |
Account
Tool | Description |
| Account balance and recent transactions |
| Account details |
Environment Variables
Variable | Description |
| API key (overrides config file) |
| API URL (default: |
| Optional |
| Inference base URL (default: |
Tenant management commands keep the configured te_* API token local and
exchange it for a short-lived management JWT before calling the API. Inference
keys (sk-te-*) are for inference-only flows such as te inference token and
proxy calls; they are not accepted for tenant registry management commands.
Inference Smoke Testing
Use te-inference-smoke to exercise inference behavior as a tenant admin and, optionally, real tenant users. The default run is read-only. Set TE_SMOKE_MUTATE=1 to create temporary inference roles, keys, policies, guardrails, MCP servers, agents, and skills, then test permission permutations and clean them up.
If you only have an sk-te-* inference key, set TE_INFERENCE_KEY for
proxy-only checks. Full role/user/policy permutations require a tenant-admin
app API key that starts with te_.
TE_API_URL=https://app.tuningengines.com \
TE_ADMIN_API_KEY=te_admin_key_here \
TE_USER_API_KEY=te_user_key_here \
npx -y --package tuningengines-cli@latest te-inference-smokeFor actual proxy model calls, enable live calls explicitly:
TE_API_URL=https://app.tuningengines.com \
TE_INFERENCE_BASE=https://api.tuningengines.com/v1 \
TE_ADMIN_API_KEY=te_admin_key_here \
TE_SMOKE_MUTATE=1 \
TE_SMOKE_LIVE_CALLS=1 \
TE_SMOKE_CREATE_MODEL_DEPLOYMENT=1 \
TE_SMOKE_ALLOWED_MODEL=llama-3.1-8b-fast \
TE_SMOKE_DENIED_MODEL=llama-3.3-70b-fp8 \
TE_SMOKE_AGENT_URL=https://httpbin.org/post \
npx -y --package tuningengines-cli@latest te-inference-smokeTE_SMOKE_CREATE_MODEL_DEPLOYMENT=1 is useful for disposable tenants that do
not already have an enabled model. By default the runner treats a provider
authentication failure on an allowed model as proof that Tuning Engines RBAC
allowed the request through to the provider. Set
TE_SMOKE_ALLOW_PROVIDER_AUTH_FAILURE=0 when the tenant has real provider
credentials and the allowed call must return 200.
To test multiple tenant users, provide their API tokens:
TE_SMOKE_USERS_JSON='[
{"email":"member1@example.com","api_key":"te_user_key_1"},
{"email":"member2@example.com","api_key":"te_user_key_2"}
]' \
TE_ADMIN_API_KEY=te_admin_key_here \
TE_SMOKE_MUTATE=1 \
npx -y --package tuningengines-cli@latest te-inference-smokePreview coverage:
npx -y --package tuningengines-cli@latest te-inference-smoke --listEach run writes a masked JSON report under te-smoke-results/, or to
TE_SMOKE_REPORT when that env var is set.
Authentication
te auth login uses a secure device authorization flow (same pattern as gh auth login):
CLI generates a device code and opens your browser
Sign up or log in (email/password, Google, or GitHub)
Click "Authorize" to grant CLI access
Token flows back automatically — no copy-paste
Works for both new sign-ups and existing accounts. Token saved to ~/.tuningengines/config.json with 0600 permissions.
Links
License
MIT
Available Tools
38 toolscancel_evaluationB
Cancel a running or queued evaluation.
| Name | Required | Description | Default |
|---|---|---|---|
| evaluation_id | Yes | Evaluation ID to cancel |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries full burden for behavioral disclosure. It only states 'Cancel', which implies mutation, but does not specify if the action is irreversible, any side effects, or what happens to the evaluation's data and associated resources.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, front-loaded sentence with no extraneous words. It efficiently communicates the core function.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple cancel action with one parameter and no output schema, the description is minimally adequate. However, it lacks information about error conditions, typical use cases, or confirmation of state change, which would be useful for an AI agent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema already provides a description for the single parameter 'evaluation_id' ('Evaluation ID to cancel'). The tool description adds no additional meaning beyond what is in the schema, so baseline score of 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action 'Cancel' and the resource 'evaluation', and specifies the applicable states 'running or queued', which prevents misuse on completed evaluations. Although a sibling tool 'cancel_job' exists, the resource differentiation is clear from the tool name itself.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No explicit guidance on when to use this tool versus alternatives like 'cancel_job' or 'delete_evaluation'. There is no mention of prerequisites, such as checking the evaluation status first, or that cancellation may not be possible beyond a certain point.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
cancel_jobA
Cancel a running or queued fine-tuning job. The job will be charged for any GPU time already used.
| Name | Required | Description | Default |
|---|---|---|---|
| job_id | Yes | Job ID to cancel |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description discloses a key behavioral trait beyond annotations: the job will be charged for GPU time already used. This adds significant value and helps the agent understand the financial impact.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, front-loaded with the core action, and no wasted words. Every sentence serves a purpose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the simplicity of the tool (1 parameter, no output schema), the description covers the purpose, parameter, and a key consequence (charging). It is fully adequate for this context.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and description of parameter (job_id) is already provided in the schema. The description adds no further semantic information beyond what the schema indicates.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('Cancel') and the specific resource ('running or queued fine-tuning job'), making it distinct from siblings like cancel_evaluation or retry_job.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage when canceling a fine-tuning job but does not explicitly state when not to use it or provide alternatives. The context of sibling tools offers implicit differentiation, but explicit guidance is missing.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
catalog_export_statusA
Check the status of a Marketplace export operation. Returns status, charge info, and any error messages.
| Name | Required | Description | Default |
|---|---|---|---|
| model_id | Yes | Catalog model ID (UUID) | |
| export_id | Yes | Export operation ID (UUID) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It discloses that the tool returns 'status, charge info, and any error messages', which gives some behavioral insight. However, it does not specify whether the operation is read-only, idempotent, or requires authentication, leaving gaps in transparency.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise with two short sentences: the first states the purpose, the second lists what is returned. Every word is necessary, and no fluff is present.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity (two required UUID parameters, no output schema), the description covers the basic return values. However, it lacks context such as the need to call export_catalog_model first, whether polling is safe, or how to interpret errors. It is adequate but not fully comprehensive.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the base score is 3. The description adds no additional semantic information about the parameters (model_id and export_id) beyond what the schema already provides. It does not clarify their format or relationship beyond UUIDs.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'Check the status of a Marketplace export operation', specifying the verb (check), resource (status of export operation), and scope (Marketplace). It distinguishes this tool from sibling status tools like job_status or evaluation_status by being specific to Marketplace export operations.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description does not explicitly state when to use this tool versus alternatives or mention prerequisites. While it's implied that it should be used after initiating an export via export_catalog_model, there is no direct guidance. The name helps, but explicit usage context would improve clarity.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
create_datasetA
Create a new dataset by importing from S3. Datasets can be used for fine-tuning or model evaluation.
| Name | Required | Description | Default |
|---|---|---|---|
| name | Yes | Name for the dataset | |
| description | No | Description of the dataset contents | |
| source_type | Yes | Source type (e.g. 's3') | |
| s3_url | No | S3 URL of the dataset (e.g. s3://bucket/path/data.jsonl) | |
| s3_access_key_id | No | AWS access key ID | |
| s3_secret_access_key | No | AWS secret access key | |
| s3_region | No | AWS region (e.g. us-east-1) | |
| for_evaluation | No | Whether this dataset is for evaluation (default: false) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations are absent, so the description must disclose all behavioral traits. It mentions the S3 import mechanism but fails to detail failure modes, error handling, idempotency, or permission requirements (e.g., AWS credentials, region). The impact of existing datasets with the same name is not addressed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, no wasted words. Action is front-loaded ('Create a new dataset'), and the purpose is clearly stated. Ideal conciseness.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With 8 parameters, no output schema, and no annotations, the description provides acceptable context (S3 import, usage for fine-tuning/evaluation) but lacks critical details like error handling, duplicate handling, and output expectations for a creation tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so baseline is 3. The description adds no additional meaning beyond the schema; it merely restates the S3 import aspect. No further enrichment of parameter semantics.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('Create a new dataset'), the source ('importing from S3'), and the purpose ('fine-tuning or model evaluation'). It effectively distinguishes from sibling tools like 'delete_dataset' or 'list_datasets'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives. The description does not mention prerequisites, when not to use it, or suggest other tools like 'validate_s3' for pre-checking S3 access.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
create_evaluationA
Create a new model evaluation. Run your trained model or a base model against a dataset using selected evaluators. Use list_evaluators to see available evaluators (e.g. code_execution, similarity, llm_judge).
| Name | Required | Description | Default |
|---|---|---|---|
| name | No | Name for this evaluation run | |
| user_model_id | No | ID of your trained model to evaluate. Either this or base_model is required. | |
| base_model | No | HuggingFace model ID to evaluate (e.g. 'Qwen/Qwen2.5-Coder-7B-Instruct'). Either this or user_model_id is required. | |
| dataset_id | Yes | ID of the evaluation dataset to use. Must be a dataset marked for_evaluation. | |
| evaluator_ids | Yes | List of evaluator IDs to run (use list_evaluators to see options) | |
| max_samples | No | Maximum samples to evaluate (default: all) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must cover behavioral traits. It implies a run action but doesn't disclose side effects like cost, asynchronous behavior, or resource usage. The description is too brief for a tool with no annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences long, with the purpose front-loaded. Every sentence adds meaningful information without redundancy or fluff.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the schema covers all parameters and there is no output schema, the description provides an adequate but minimal overview. It could be more complete by noting return values or asynchronous behavior, but it suffices for a straightforward creation tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema already describes all parameters with 100% coverage. The description adds value by referencing list_evaluators for available evaluators and clarifying that either user_model_id or base_model is required (though this contradicts the schema's optionality). This extra context helps the agent.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool creates a new model evaluation and explains the process (running a model against a dataset with evaluators). It distinguishes itself from sibling tools like list_evaluators and show_evaluation.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description mentions using list_evaluators to find available evaluators, which provides some guidance. However, it doesn't explicitly state when not to use this tool, such as when to use estimate_evaluation instead, or clarify the choice between user_model_id and base_model.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
create_jobA
Fine-tune an LLM on a GitHub repository using Tuning Engines. This trains a custom model that learns from the code patterns, style, and conventions in the repo. Choose an agent to control the training approach:
AVAILABLE AGENTS:
agent='code_repo' (Cody) — LoRA-based code fine-tuning using QLoRA (4-bit quantized LoRA) via the Axolotl framework. Trains on your repo's code patterns, naming conventions, and project structure to produce a fast, lightweight adapter. Best for: code autocomplete, inline suggestions, tab-complete, code style matching.
agent='sera_code_repo' (SIERA) — Bug-fix specialist using the Open Coding Agents approach from AllenAI. Generates synthetic error-resolution training pairs from your repo, producing a model that understands your codebase's failure patterns and fix conventions. Best for: debugging, error resolution, patch generation, root cause analysis. Supports quality_tier='low' (faster) or quality_tier='high' (deeper analysis, more training data).
SUPPORTED BASE MODELS (by size):
3B: Qwen/Qwen2.5-Coder-3B-Instruct
7-8B: codellama/CodeLlama-7b-hf, deepseek-ai/deepseek-coder-7b-instruct-v1.5, Qwen/Qwen2.5-Coder-7B-Instruct, Qwen/Qwen3-8B
13-15B: codellama/CodeLlama-13b-Instruct-hf, bigcode/starcoder2-15b, Qwen/Qwen2.5-Coder-14B-Instruct, Qwen/Qwen3-14B
22-27B: mistralai/Codestral-22B-v0.1, google/gemma-2-27b
30-34B: deepseek-ai/deepseek-coder-33b-instruct, codellama/CodeLlama-34b-Instruct-hf, Qwen/Qwen2.5-Coder-32B-Instruct, Qwen/Qwen3-Coder-30B-A3B, Qwen/Qwen3-32B
70-72B: codellama/CodeLlama-70b-Instruct-hf, meta-llama/Llama-3.1-70B-Instruct, Qwen/Qwen2.5-72B-Instruct
TYPICAL WORKFLOW: estimate_job first to check cost, then create_job, then job_status to monitor progress.
| Name | Required | Description | Default |
|---|---|---|---|
| base_model | No | HuggingFace model ID to fine-tune (e.g. 'Qwen/Qwen2.5-Coder-7B-Instruct'). Required unless base_user_model_id is provided. Use list_supported_models to see all options. | |
| base_user_model_id | No | ID of a previously trained model to fine-tune further (iterative training). The base model is resolved automatically. Use list_models to find IDs. | |
| output_name | Yes | Name for the resulting fine-tuned model (e.g. 'my-project-cody-7b') | |
| repo_url | Yes | GitHub repository URL to train on (e.g. 'https://github.com/org/repo') | |
| branch | No | Git branch to use (default: main) | |
| num_epochs | No | Number of training epochs (more = better quality but higher cost) | |
| max_examples | No | Maximum training examples to extract from the repo (minimum: 2) | |
| agent | No | Training agent to use. 'code_repo' (Cody) = QLoRA-based fine-tuning for code autocomplete and inline suggestions. 'sera_code_repo' (SIERA) = bug-fix specialist using AllenAI's Open Coding Agents approach. Default: 'code_repo'. | |
| quality_tier | No | Quality tier (SIERA agent only). 'low' = faster, fewer synthetic pairs. 'high' = deeper analysis, more training data, better results. Default: 'low'. | |
| s3_output_bucket | No | S3 bucket to export the trained model to. If omitted, model is stored in Tuning Engines cloud storage. | |
| s3_access_key_id | No | AWS access key ID for S3 export | |
| s3_secret_access_key | No | AWS secret access key for S3 export | |
| s3_region | No | AWS region for S3 export (e.g. us-east-1) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden of behavioral disclosure. It explains that training produces a custom model using QLoRA (code_repo) or synthetic error-resolution pairs (sera_code_repo), and mentions that the model can be stored in cloud storage or exported to S3. It does not detail potential side effects, rate limits, or the exact output format, but the description is fairly transparent about the training process and output destinations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured with sections for agents, base models, and workflow, but it is quite lengthy, particularly the exhaustive list of supported base models which could be omitted since users can use list_supported_models. While the structure is clear, conciseness is slightly compromised by the inclusion of redundant details.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the complexity of fine-tuning an LLM, the description is fairly complete. It explains the tool's purpose, agents, supported base models, and workflow. It references sibling tools for cost estimation and monitoring. However, it does not describe the return value or response structure (e.g., job ID or status), which would be helpful. Since there is no output schema, a brief note on what the tool returns would improve completeness.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has 100% description coverage, so the schema already documents all parameters. The description adds value by providing extra context for the agent and quality_tier parameters, including best-use recommendations ('Best for: code autocomplete, inline suggestions...') and agent-specific behavior. This goes beyond the schema's enum descriptions, justifying a score above the baseline of 3.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: 'Fine-tune an LLM on a GitHub repository using Tuning Engines.' It specifies the action (fine-tune), the resource (LLM), and the data source (GitHub repo). The description distinguishes the tool from siblings like estimate_job and job_status by outlining the typical workflow, and it further differentiates the two agents (code_repo and sera_code_repo) with explicit use cases.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides explicit usage guidelines, including a 'TYPICAL WORKFLOW' section that advises users to run estimate_job first for cost estimation, then create_job, and finally job_status to monitor progress. It also clarifies when to use each agent and the quality_tier parameter for the SIERA agent, offering clear decision criteria.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
dataset_statusC
Check the status of a dataset import or processing operation.
| Name | Required | Description | Default |
|---|---|---|---|
| dataset_id | Yes | Dataset ID (UUID) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, and the description does not disclose any behavioral traits such as side effects, authorization requirements, or rate limits. As a read-only status check, more detail on safety and repeatability would be helpful.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, clear sentence with no wasted words. It is appropriately concise for a simple tool.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a low-complexity tool with one parameter and no output schema, the description is adequate but could be improved by indicating possible return values or statuses.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% (one parameter with description). The description adds no extra meaning beyond the schema; baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'check' and the resource 'status of a dataset import or processing operation'. It is specific to datasets, distinguishing it from other status tools like job_status, but does not explicitly differentiate from show_dataset.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives like show_dataset or job_status. There is no mention of prerequisites or context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
delete_datasetC
Delete a dataset from the platform.
| Name | Required | Description | Default |
|---|---|---|---|
| dataset_id | Yes | Dataset ID to delete |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description bears full responsibility for behavioral disclosure. It only states 'delete' without indicating that the operation is destructive, irreversible, or whether it affects dependent resources. Important behavioral traits are missing.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, concise sentence with no wasted words. It is front-loaded with the verb and resource. However, it could include a few more critical details without becoming verbose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Although the tool is simple with one parameter, the description fails to mention essential context like whether the deletion is permanent, any prerequisites (e.g., ownership), or side effects. This lack of completeness could lead to misuse.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has 100% coverage with a description for dataset_id. The tool description adds no additional meaning beyond what the schema already provides. Baseline 3 is appropriate as the schema carries the full burden.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action (delete a dataset) and the resource (dataset from the platform). It effectively distinguishes from sibling tools like create_dataset (create) and show_dataset (read). However, it could be more specific by noting the deletion is permanent.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided about when to use this tool versus alternatives. It does not mention prerequisites, irreversibility, or when deletion should be avoided. This lack of context makes it harder for an agent to decide appropriately.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
delete_modelB
Delete a trained model from cloud storage.
| Name | Required | Description | Default |
|---|---|---|---|
| model_id | Yes | Model ID to delete |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description must carry full burden. It indicates a destructive action but lacks details on side effects (e.g., cascading deletes, reversibility, permission requirements). The 'delete' verb implies destruction, but more context is needed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence that efficiently conveys the tool's purpose without extraneous words. It is front-loaded with the key action.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple delete operation with one required parameter and no output schema, the description adequately covers the basic action. It could mention permanence or prerequisites, but overall it is sufficient given the tool's simplicity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% with one parameter 'model_id' described as 'Model ID to delete'. The description adds no extra meaning beyond the schema, so baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'delete' and resource 'trained model' with location 'from cloud storage'. It distinguishes itself from sibling tools like 'delete_dataset' by specifying the resource type, providing clear purpose.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives like 'cancel_evaluation' or 'delete_dataset'. There are no usage conditions, prerequisites, or notes about when deletion is appropriate.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
estimate_evaluationB
Get a cost estimate for an evaluation before running it.
| Name | Required | Description | Default |
|---|---|---|---|
| user_model_id | No | ID of your trained model | |
| base_model | No | Or a HuggingFace model ID | |
| dataset_id | Yes | Evaluation dataset ID | |
| evaluator_ids | Yes | List of evaluator IDs | |
| max_samples | No | Max samples to evaluate |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description bears full responsibility for behavioral disclosure. The description only states the tool returns a cost estimate, but fails to mention any behavioral traits: it does not indicate whether the operation is read-only, whether it affects state, what the response format is, or if authentication or quota are needed. This is a significant gap.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence with no redundant words. It front-loads the core purpose, making it immediately clear what the tool does. Every part of the description earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (5 parameters, no output schema, no annotations), the description is too sparse. It does not explain what the cost estimate depends on (e.g., model, dataset size, evaluators), how to interpret the result, or any constraints (e.g., balance requirement). The description lacks completeness for a meaningful cost estimation tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema describes all 5 parameters with descriptions (100% coverage), so the baseline is 3. The description adds no additional parameter semantics beyond what the schema already provides. It does not explain how each parameter influences the cost estimate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: 'Get a cost estimate for an evaluation before running it.' It specifies a concrete verb ('Get') and resource ('cost estimate for evaluation'), effectively distinguishing it from sibling tools such as create_evaluation (which runs the evaluation) and estimate_job (which estimates jobs, not evaluations).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage before running an evaluation ('before running it'), providing some context. However, it does not explicitly state when to use this tool over alternatives, nor does it mention prerequisites or when not to use it. The guidance is implicit rather than explicit.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
estimate_jobA
Get a cost estimate for a fine-tuning job before submitting it. Returns estimated cost, cost range, current balance, and whether balance is sufficient. Always estimate before creating a job.
| Name | Required | Description | Default |
|---|---|---|---|
| base_model | No | HuggingFace model ID (e.g. 'Qwen/Qwen2.5-Coder-7B-Instruct'). Required unless base_user_model_id is provided. | |
| base_user_model_id | No | ID of a previously trained model. The base model is resolved automatically. | |
| num_epochs | No | Training epochs | |
| max_examples | No | Maximum examples | |
| repo_size_mb | No | Approximate repository size in MB (helps refine the estimate) | |
| use_case | No | Agent to use for the estimate (e.g. 'code_repo' for Cody, 'sera_code_repo' for SIERA). Defaults to code_repo. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are present, so the description fully bears responsibility. It discloses return information (estimated cost, cost range, current balance, sufficiency). It does not mention side effects (likely none) or authorization, but the read-only nature is implied.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description consists of two concise sentences with no redundant information. It front-loads the core purpose and includes a directive.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With 6 parameters and no output schema, the description could be more complete by specifying the return format or units. It lists return items but lacks detail like how errors are handled.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the baseline is 3. The description adds no additional parameter meaning beyond what the schema already provides (e.g., 'HuggingFace model ID').
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('Get a cost estimate'), the resource ('a fine-tuning job'), and the purpose ('before submitting it'). It distinguishes from sibling tools like estimate_evaluation by specifying the job type.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly advises 'Always estimate before creating a job,' providing clear when-to-use guidance. It does not specify when not to use, but the context implies it's a prerequisite step.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
evaluation_statusB
Get live status of an evaluation including progress and current metrics.
| Name | Required | Description | Default |
|---|---|---|---|
| evaluation_id | Yes | Evaluation ID (UUID) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries full burden. It mentions 'live' status but does not describe side effects, authorization needs, latency, or consistency. Read-only is implied but not explicit.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, clear sentence. It is efficient without being overly terse, and it front-loads the key information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description gives a high-level idea of the output (progress and metrics) but no detail on the structure or data fields. Given no output schema, more context would be helpful for an agent to interpret results.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The single parameter evaluation_id is fully documented in the schema (100% coverage). The tool description adds no additional meaning to the parameter beyond what the schema already provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('Get'), the resource ('live status of an evaluation'), and what it includes ('progress and current metrics'). It distinguishes from sibling tools like list_evaluations by focusing on live, individual status.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives like show_evaluation or job_status. Only the purpose is stated, without contextual usage advice.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
export_catalog_modelA
Export a pre-built model or dataset from the Marketplace to your S3 bucket. Credits will be charged based on the export price upon successful completion.
| Name | Required | Description | Default |
|---|---|---|---|
| model_id | Yes | Catalog model ID (UUID) to export | |
| s3_bucket | Yes | Destination S3 bucket name | |
| s3_prefix | No | Optional S3 key prefix for the exported model | |
| s3_access_key_id | Yes | AWS access key ID | |
| s3_secret_access_key | Yes | AWS secret access key | |
| s3_region | Yes | AWS region (e.g. us-east-1) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided; description only adds credits charge. Lacks disclosure of async nature, side effects on S3, or failure behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two efficient sentences conveying purpose and cost without unnecessary detail.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Missing important context: async operation (indicated by sibling catalog_export_status), no mention of return value, no permissions guidance for S3 credentials.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema covers 100% of parameters with descriptions. The tool description adds no additional parameter context beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states the action (export) and resource (pre-built model or dataset from Marketplace to S3), distinguishing it from sibling tools like export_model.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Implies it's for marketplace exports and mentions cost, but does not explicitly compare with alternatives or state when not to use it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
export_modelA
Export a trained model from Tuning Engines cloud storage to your S3 bucket.
| Name | Required | Description | Default |
|---|---|---|---|
| model_id | Yes | Model ID (UUID) to export | |
| s3_bucket | Yes | Destination S3 bucket name | |
| s3_prefix | No | Optional S3 key prefix for the exported model | |
| s3_access_key_id | Yes | AWS access key ID | |
| s3_secret_access_key | Yes | AWS secret access key | |
| s3_region | Yes | AWS region (e.g. us-east-1) | |
| delete_after | No | Delete the model from Tuning Engines storage after export (default: false) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries full burden. It only states the action but does not disclose side effects (e.g., whether the original is deleted by default, if it's synchronous or async, or required permissions beyond schema). This is minimal for a mutation tool with no annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single concise sentence (12 words) that front-loads the purpose. Every word is necessary, with no redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite a rich schema, the description lacks important context: no mention of typical workflow (e.g., verifying export status), no explanation of the 'delete_after' parameter behavior, and no guidance on prerequisites. Incomplete for a 7-parameter tool with no output schema.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
All 7 parameters have descriptions in the input schema (100% coverage). The description adds no additional parameter-specific information beyond the schema, so baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('Export') and the resource ('trained model from Tuning Engines cloud storage to your S3 bucket'). It distinguishes itself from siblings like 'export_catalog_model' by specifying the source and destination.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear context on when to use the tool (export trained model to S3). It does not explicitly state when not to use it or mention alternatives, but the context is sufficient for an AI agent to infer appropriate usage.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_accountA
Get your Tuning Engines account details and settings.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided. The description only says 'Get', implying no side effects, but does not disclose what exactly is returned, authentication requirements, or any limitations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Single sentence with no unnecessary words. Front-loaded with key information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a zero-parameter tool with no output schema and no annotations, the description is minimal but adequate. It lacks detail on what fields or settings are returned.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
There are zero parameters, and schema coverage is 100% (empty object). Per guidelines, baseline for 0 params is 4, and description adds no additional meaning but is not needed.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Get' and the resource 'your Tuning Engines account details and settings.' It is distinct from sibling tools like get_balance and get_catalog_model.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use or not use this tool. It does not mention alternatives or contexts where it is appropriate.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_balanceB
Check your Tuning Engines account balance and recent transactions.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so the description carries the full burden. It indicates a read operation ('Check') but does not disclose behavioral traits such as authentication needs, rate limits, or what 'recent' means. Basic safety is implied but not detailed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, clear sentence with no unnecessary words. Extremely concise and front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no parameters, no output schema, and no annotations, the description covers the basic purpose. However, it lacks specifics about the output format, transaction count, or time range for 'recent'. Somewhat incomplete for a well-rounded understanding.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
No parameters exist, so schema coverage is 100%. The description adds no parameter info, which is acceptable since there are none. Baseline 4 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool checks 'account balance and recent transactions', which is a specific verb+resource. However, it does not differentiate from the sibling tool 'get_account', which might also provide account details.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool vs. alternatives like 'get_account'. The description implies it's for checking balance and transactions, but does not explicitly state context or exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_catalog_modelA
Get detailed information about a specific pre-built model or dataset from the Marketplace including description, pricing, and export options.
| Name | Required | Description | Default |
|---|---|---|---|
| model_id | Yes | Catalog model ID (UUID) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must fully disclose behavioral traits. While 'Get' implies a read operation, it does not explicitly state that the tool is read-only, safe to call without side effects, or any prerequisites like authentication or quota implications. This leaves ambiguity for the agent.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, well-formed sentence that conveys the purpose and key details without redundancy or unnecessary words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple lookup tool with one parameter and no output schema, the description covers the return value (description, pricing, export options) adequately. It could mention that the response includes all available metadata, but it is sufficiently complete for most use cases.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has 100% coverage describing the single parameter 'model_id' as a UUID. The description adds no extra meaning beyond what the schema already provides, so a baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses the verb 'Get' and specifies the resource as 'a specific pre-built model or dataset from the Marketplace', clearly indicating a singular retrieval operation. This distinguishes it from sibling tools like 'list_catalog_models' (listing) and 'export_catalog_model' (exporting).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No explicit guidance on when to use this tool versus alternatives such as 'list_catalog_models' for browsing or 'catalog_export_status' for export tracking. The description implies usage when needing detailed info on a single item, but lacks exclusions or when-not scenarios.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_inference_jwtA
Get a JWT token for authenticating with the Tuning Engines inference API. Use this to make direct API calls to the inference endpoint.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so description carries burden. States it returns a JWT token, implying a read operation, but doesn't disclose expiration, permissions, or side effects. Adequate but not detailed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, directly front-loaded with core action and purpose. No wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a zero-parameter tool with no output schema, description explains what it does and when to use it. Could note token expiration or security, but overall sufficient.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
No parameters exist, so baseline 4. Description adds no parameter info, but schema coverage is 100%.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states it retrieves a JWT token for the Tuning Engines inference API, distinct from sibling tools like get_account or inference_usage.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly says to use for direct API calls to inference endpoint, providing clear context. No exclusions or alternatives mentioned, but purpose is straightforward.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
import_modelA
Import a model from S3 into Tuning Engines cloud storage so it can be used as a base for future fine-tuning jobs.
| Name | Required | Description | Default |
|---|---|---|---|
| name | Yes | Name for the imported model | |
| source_s3_url | Yes | S3 URL of the model to import (e.g. s3://bucket/path/to/model) | |
| base_model | Yes | HuggingFace model ID that this model was fine-tuned from | |
| s3_access_key_id | Yes | AWS access key ID | |
| s3_secret_access_key | Yes | AWS secret access key | |
| s3_region | Yes | AWS region (e.g. us-east-1) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description reveals that the operation is mutative and requires S3 credentials, but fails to disclose important behaviors like whether it is synchronous, what happens on duplicate names, or any permissions needed. Given no annotations, this leaves gaps.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, well-structured sentence that immediately conveys the core purpose. No redundant or extraneous information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a mutative tool with 6 required parameters and no annotations, the description is minimal. It explains the 'what' and 'why' but not the expected outcome, error states, or side effects. Adequate but not complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
All six parameters are documented in the input schema with adequate descriptions. The tool description adds no extra parameter-level context beyond the schema, so score is baseline 3 due to full schema coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the specific action ('import a model from S3'), the destination ('into Tuning Engines cloud storage'), and the purpose ('so it can be used as a base for future fine-tuning jobs'). This distinguishes it from siblings like export_model, delete_model, etc.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool over alternatives, nor are there any prerequisites or exclusions mentioned. The agent has no help deciding between import and other tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
inference_usageB
Get inference API usage statistics including request counts, token usage, and costs.
| Name | Required | Description | Default |
|---|---|---|---|
| start_date | No | Start date (YYYY-MM-DD) | |
| end_date | No | End date (YYYY-MM-DD) | |
| model | No | Filter by model name |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, and the description does not disclose behavioral traits such as read-only nature, authentication requirements, rate limits, or side effects. It only describes the basic function.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single concise sentence that gets straight to the point without any extraneous information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description partially compensates for the lack of output schema by listing included metrics. However, it omits details like whether dates are required, response format, or aggregation behavior.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has 100% coverage, so baseline is 3. The description does not add meaning beyond the schema; it mentions output contents but not parameter details.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it retrieves inference API usage statistics and lists included metrics (request counts, token usage, costs). However, it does not differentiate from sibling tools like get_catalog_model or get_inference_jwt, but the purpose is specific enough.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives. The description merely implies usage without stating prerequisites, limitations, or exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
job_statusA
Get live status of a fine-tuning job including current status, GPU minutes used, estimated charges, remaining balance, and delivery progress. Use this to monitor a running job.
| Name | Required | Description | Default |
|---|---|---|---|
| job_id | Yes | Job ID |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries full burden. It discloses the tool returns live status and specific fields, but lacks details on read-only nature, error behavior, or rate limits. The disclosure is adequate but not comprehensive.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences with no wasted words. The first sentence states purpose and output, the second provides usage context. Well front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema, the description partially describes return fields (GPU minutes, charges, etc.). For a one-parameter tool, this is fairly complete, though a full output description would be better.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema has 100% coverage with a single parameter 'job_id' described as 'Job ID'. The description adds no additional meaning beyond this, so it meets the baseline but does not elevate understanding.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool retrieves live status of a fine-tuning job, listing specific fields like GPU minutes, charges, balance, and delivery progress. It uses a specific verb and resource, distinguishing it from sibling status tools like evaluation_status or dataset_status.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly says 'Use this to monitor a running job', providing clear when-to-use guidance. However, it does not mention when not to use it or list alternative tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_agentsA
List available agents configured for your organization. Agents are AI assistants with specific capabilities and tool access.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of disclosing behavioral traits. It only states that it lists agents, but does not mention any potential side effects, permission requirements, pagination, ordering, or limits. Since this is a list operation, the agent might expect complete results, but there is no confirmation or caveat.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences long, with the first sentence stating the core function and the second adding helpful context about agents. Every word earns its place; there is no redundancy or filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a zero-parameter, no-output-schema tool, the description is reasonably complete: it states what the tool does and defines agents. However, it lacks any mention of what information is returned for each agent (e.g., name, capabilities), which could be relevant for an agent deciding whether to use this tool. A bit more detail would improve completeness.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
There are no parameters, so the input schema coverage is trivially 100%. Per guidelines, 0 parameters yields a baseline of 4. The description does not add parameter information because none exist, so it is adequate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'List' and resource 'available agents', and distinguishes the tool's scope by specifying 'configured for your organization'. It also provides a brief definition of agents, which helps the agent understand what it is listing. This clearly differentiates it from sibling tools like list_models or list_datasets.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description does not provide any guidance on when to use this tool versus alternatives. Sibling tools include show_agent, which likely retrieves details of a single agent, but no comparison or usage context is given. The agent is left to infer when to use list_agents versus other list or show tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_catalog_modelsA
List available pre-built models and datasets from the Tuning Engines Marketplace. These are platform-owned, ready-to-use assets that can be exported to your S3 bucket. Returns name, description, base model, size, export price, and category.
| Name | Required | Description | Default |
|---|---|---|---|
| category | No | Filter by category (e.g. 'code', 'bug-fix', 'general'). Omit to see all. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, but description adequately explains the read-only list operation and return fields. Lacks mention of pagination or empty results, but acceptable for a simple list.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences with no unnecessary words; clear and front-loaded with key information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Covers purpose, parameter, and return fields; no output schema needed. Missing pagination info but adequate for a straightforward list tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, and the description adds examples ('code', 'bug-fix', 'general') for the category parameter, providing extra meaning beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states it lists pre-built models/datasets from Marketplace, distinguishes from sibling list_models (user models) and list_inference_models by specifying platform-owned, ready-to-use assets.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Describes the tool's purpose but does not explicitly state when to use vs alternatives; however, mentioning 'pre-built models from Tuning Engines Marketplace' implies context for selecting this tool over others.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_datasetsB
List datasets available for training and evaluation. Datasets can be uploaded from S3 and used for fine-tuning or model evaluation.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | Max results (default 20) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
As a read-only listing tool, behavioral transparency is adequate. However, with no annotations, the description could mention side effects or authorization, but the simple nature of listing mitigates this gap.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise at two sentences, but the second sentence about S3 uploads is off-topic for a listing tool, slightly reducing relevance.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple list tool with one parameter, the description is mostly complete. However, it doesn't explain the return format or pagination, which is a gap without an output schema.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The only parameter 'limit' is fully described in the schema (100% coverage). The description adds no extra meaning beyond that, so baseline score is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states that the tool lists datasets for training and evaluation. While it doesn't explicitly differentiate from siblings like 'show_dataset', the name and description make the purpose obvious.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives like 'show_dataset' or 'create_dataset'. The second sentence about uploading S3 data is irrelevant to listing.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_evaluationsB
List model evaluations. Evaluations run your trained models against benchmark datasets using various evaluators to measure quality.
| Name | Required | Description | Default |
|---|---|---|---|
| status | No | Filter by status: queued, running, succeeded, failed, canceled | |
| limit | No | Max results (default 20) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It only states the basic purpose and does not disclose any behavioral traits such as pagination, default ordering, or side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences: first sentence states the action, second provides context. It is concise but the second sentence is explanatory rather than directly about the tool's usage, which is acceptable.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers what the tool does and the context of evaluations. However, it lacks information about the output structure (e.g., what fields are returned) since there is no output schema. For a simple list tool, it is minimally adequate.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Input schema has 2 parameters with 100% description coverage. The description does not add any meaning beyond the schema; it doesn't mention the parameters. Baseline score applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'List model evaluations' with a specific verb and resource, and the additional sentence explains what evaluations are, distinguishing it from related tools like create_evaluation or evaluation_status.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies the tool is for listing evaluations but provides no explicit guidance on when to use it versus alternatives such as list_datasets or list_models. No exclusions or context are given.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_evaluatorsA
List available evaluators for model evaluation. Evaluators measure different aspects of model quality like code execution, similarity, or LLM-based judgment.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It only describes the purpose of evaluators, not the tool's behavior (e.g., read-only, auth requirements, response format). This is insufficient for a mutation-free listing tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two concise sentences front-load the purpose and provide context about evaluator types. Every sentence adds value with no redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema, the description could explain what the output represents (e.g., list of evaluator IDs/names). It adequately covers the basic purpose but lacks completeness regarding output structure and differentiation from similar list tools.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has zero properties, so the description cannot add parameter meaning. Baseline score of 4 is appropriate since no parameters require explanation.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'List' and the resource 'available evaluators for model evaluation', distinguishing it from siblings like 'list_evaluations' which likely lists evaluation jobs. It also provides examples of evaluator types, adding specificity.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No explicit guidance on when to use this tool versus alternatives like list_evaluations or list_models. The description implies usage when selecting evaluators for a model evaluation, but lacks when-not conditions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_inference_modelsA
List models available for inference through the Tuning Engines inference API. Includes both platform models and your deployed trained models.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided (no readOnlyHint). The description accurately states it lists models, which implies read-only behavior. However, it lacks details on performance, pagination, or any potential side effects, which is acceptable for a simple 0-param tool but could be more transparent.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two concise sentences that front-load the purpose and include key details about what is listed. No redundant phrases.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple 0-param tool with no output schema, the description is sufficient. It explains the scope (platform + trained models) and the API context. Could mention that output is a list of model IDs or similar, but not necessary.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Input schema has 0 parameters, so description does not need to elaborate on params. Baseline 4 is appropriate as the description adds no param info but schema coverage is 100% and no params exist.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool lists models for inference via Tuning Engines API, distinguishing it from generic model listing tools like `list_models` by specifying the inference context and included model types (platform + deployed trained).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies use when needing inference-available models, but does not explicitly state when not to use or provide comparisons to siblings like `list_models` or `list_catalog_models`. The context is clear enough for a simple tool.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_jobsA
List fine-tuning training jobs on Tuning Engines. Returns recent jobs with status, base model, agent type, GPU usage, and cost. Use this to check on existing training runs or find a job ID.
| Name | Required | Description | Default |
|---|---|---|---|
| status | No | Filter by status: queued, running, succeeded, failed, canceled | |
| limit | No | Max results (default 20) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations exist, so description must disclose behavior. It indicates a read-only listing operation with no destructive hints, which is accurate. It does not address potential rate limits or pagination, but the basic behavior is clear.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences: one states the action and resource, the other states return fields and use case. Every sentence is informative with no wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers purpose, return fields, and usage context. Missing details like ordering or permissions are minor for this simple list tool, but could be slightly more complete by referencing sibling tools for specific job details.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Parameter schema coverage is 100% with clear descriptions for status and limit. The description does not add new meaning beyond the schema, so baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool lists fine-tuning training jobs and specifies the returned fields (status, model, agent, GPU, cost). It differentiates from sibling listing tools like list_agents or list_datasets by focusing on jobs.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description tells when to use it (check existing runs, find job ID). However, it does not explicitly contrast with related tools like job_status, though the context implies a browsing use case.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_modelsB
List your trained and imported models on Tuning Engines.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description should disclose behavioral traits. It only says 'list' and does not mention whether it's read-only, response format, pagination, or any side effects. The read-only nature is implied but not confirmed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence, front-loaded, with no wasted words. It is optimally concise for the information provided.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema and no parameters, the description is incomplete. It does not explain what information is returned (e.g., model IDs, names, statuses) or any filtering capabilities, which is necessary for an agent to use it effectively.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
There are no parameters, so by rule baseline is 4. The description does not add parameter details because none exist.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool lists trained and imported models on Tuning Engines. It distinguishes from sibling listing tools like list_catalog_models by specifying 'your' models, but could be more explicit about the scope.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives (e.g., list_catalog_models, list_supported_models). It only states the basic function without context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_supported_modelsA
List the supported base HuggingFace models available for fine-tuning on Tuning Engines. Optionally filter by agent to see only compatible models.
| Name | Required | Description | Default |
|---|---|---|---|
| agent | No | Filter models compatible with this agent (e.g. 'code_repo', 'sera_code_repo'). Omit to see all models. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description correctly implies a read-only operation via 'List', but does not elaborate on any other behavior such as pagination or rate limits. Adequate for a simple list tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, no wasted words, front-loaded with purpose. Excellent conciseness.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers the core functionality and optional filter, but does not mention return format or any output details. Still sufficient for selection given no output schema.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, and the description adds the detail 'to see only compatible models', which provides more context than the schema description alone.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'List', the resource 'supported base HuggingFace models', and the context 'for fine-tuning on Tuning Engines', distinguishing it from siblings like list_models or list_catalog_models.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description mentions optional filtering by agent, providing context on when to use the parameter, but does not specify when not to use this tool or compare it to alternatives like list_models.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
model_statusB
Check the status of a model import or export operation.
| Name | Required | Description | Default |
|---|---|---|---|
| model_id | Yes | Model ID (UUID) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Description indicates a read-only operation ('check the status'), which is straightforward. No annotations are present, so the description does not contradict any. However, it lacks details on behavior like idempotency, error cases, or side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence, front-loaded with the action and resource. It is concise, though more details could be added without excessive length.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has no output schema and simple parameters, the description does not explain return values (e.g., status types, format). For a single-parameter tool, it could be more complete regarding what the status check entails.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Input schema includes one parameter 'model_id' with description 'Model ID (UUID)'. The description does not add extra semantic meaning beyond the schema, and schema coverage is 100%, so baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states 'Check the status of a model import or export operation.' It uses a specific verb and resource, but does not differentiate from similar sibling tools like 'catalog_export_status' or 'show_model', which may also relate to import/export status.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives such as 'catalog_export_status' or 'show_model'. No context on prerequisites or typical usage scenarios.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
retry_jobA
Retry a failed fine-tuning job from its last checkpoint. Creates a new job that resumes training where the failed one stopped, saving GPU time. Each retry is billed separately.
IMPORTANT: This tool fetches a cost estimate and includes it in the response. You MUST show the estimate to the user and get their explicit approval before considering the retry confirmed. The retry is submitted automatically (the server validates balance), but always present the cost to the user.
| Name | Required | Description | Default |
|---|---|---|---|
| job_id | Yes | ID of the failed job to retry | |
| github_token | No | GitHub Personal Access Token (required if original job used a private repo). Not stored — only sent to the training backend. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Discloses key behaviors: creates a new job, bills separately, fetches a cost estimate, and requires user approval. Without annotations, the description carries full burden and covers the most critical behavioral aspects, though it omits details about whether the original job remains unchanged.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two concise paragraphs: first paragraph states purpose and key benefit (saving GPU time); second paragraph has critical usage guidance. Every sentence adds value, no redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (billing, user approval, retry logic), the description covers the essential flow: retry mechanism, cost estimate, and required approval. No output schema is present, but the response structure (estimate inclusion) is mentioned. Lacks details on error handling or edge cases, but adequate for the main use case.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so baseline is 3. The description adds minimal new parameter-specific meaning beyond what the schema provides (e.g., job_id and github_token are already well-described in the schema). The user approval context is related but does not enhance parameter semantics directly.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'Retry a failed fine-tuning job from its last checkpoint,' specifying the action (retry), resource (failed fine-tuning job), and distinctive behavior (resume from checkpoint). It distinguishes itself from sibling tools like create_job or cancel_job.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly instructs the agent to show the cost estimate to the user and get explicit approval before proceeding, with 'MUST' emphasis. Provides clear when-to-use context, though it does not explicitly mention when not to use this tool (e.g., if the job is not failed).
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
show_agentA
Get details of a specific agent including capabilities, tools, and configuration.
| Name | Required | Description | Default |
|---|---|---|---|
| agent_id | Yes | Agent ID |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must disclose behavioral traits. It only states it gets details, but does not mention whether it is read-only, requires authentication, or has any side effects. For a read operation, this is insufficient.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence that is front-loaded with the core action and resource, containing no unnecessary words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple read tool with one parameter, the description is mostly adequate. It lacks details on return format but is otherwise complete given the tool's simplicity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% for the single parameter 'agent_id', which is described in the schema. The description adds no additional meaning beyond what the schema provides, so baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Get' and the resource 'details of a specific agent' with specific content (capabilities, tools, configuration). It distinguishes itself from sibling tool 'list_agents' by focusing on a single agent.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies when to use it (to get details of a specific agent), and the sibling 'list_agents' provides an alternative for listing all agents. However, no explicit when-not or prerequisites are stated.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
show_datasetA
Get details of a specific dataset including status, source, and metadata.
| Name | Required | Description | Default |
|---|---|---|---|
| dataset_id | Yes | Dataset ID (UUID) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries full burden for behavioral transparency. 'Get details' implies a read-only operation, but it does not explicitly state that no modifications occur, nor does it disclose permissions, rate limits, or data retention policies. The description is minimally adequate.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, front-loaded sentence of 12 words with no filler. Every word is meaningful, making it highly scannable and efficient.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity (1 parameter, no output schema, no annotations), the description is sufficient for basic understanding. However, it lacks context on response format (e.g., shape of metadata) and any caveats about data freshness or accessibility, leaving room for confusion.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% for the single parameter (dataset_id as UUID string). The description adds no additional context or constraints beyond what the schema already provides, so it does not improve parameter understanding.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description starts with 'Get details of a specific dataset', clearly stating the verb 'Get' and resource 'dataset'. It lists specific included fields (status, source, metadata) which distinguishes it from sibling tools like list_datasets (list all) or create_dataset (create new).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for retrieving details of a single dataset, but does not provide explicit guidance on when to use this tool versus alternatives (e.g., list_datasets for overview, show_evaluation for evaluation details). No exclusions or prerequisites are mentioned.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
show_evaluationB
Get full details of a specific evaluation including status, scores, metrics, and comparison data.
| Name | Required | Description | Default |
|---|---|---|---|
| evaluation_id | Yes | Evaluation ID (UUID) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must carry the full burden. It does not disclose that this is a read-only operation, nor any authentication requirements, rate limits, or potential side effects. The description only restates the purpose without behavioral context.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is one sentence of 15 words, which is efficient for a simple retrieval tool. However, it could be slightly expanded to include context like what the response contains, but it is not wasteful.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity (1 param, no output schema, no nested objects), the description lists some return fields (status, scores, metrics, comparison data) but does not mention error cases, access limitations, or the response format. It is adequate but incomplete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has 100% coverage with one parameter 'evaluation_id' described as 'Evaluation ID (UUID)'. The description does not add any additional meaning or format beyond what the schema already provides. Baseline is 3.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Get' and the resource 'specific evaluation', and lists what details are included: 'status, scores, metrics, and comparison data'. It distinguishes from sibling tools like 'list_evaluations' (which lists all evaluations) and 'evaluation_status' (which likely only returns status).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives such as 'evaluation_status' for just status, or 'list_evaluations' for an overview. No 'when-to-use' or 'when-not-to-use' information is given.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
show_jobA
Get full details of a specific fine-tuning job including status, base model, agent type, GPU minutes, cost, error messages, and whether it can be retried from checkpoint.
| Name | Required | Description | Default |
|---|---|---|---|
| job_id | Yes | Job ID (UUID) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, but the description discloses key behavioral aspects such as returning specific fields (status, base model, cost, error messages, checkpoint retryability). It could mention read-only nature explicitly.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Single sentence, front-loaded with purpose, efficiently lists details without redundancy. Every word adds value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given one parameter and no output schema, the description covers the return fields well. Could mention error conditions (e.g., job not found) but is adequate for a retrieval tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with job_id described as 'Job ID (UUID)'. The description adds no further parameter meaning beyond identifying the job, so baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('Get full details') and resource ('specific fine-tuning job'), and lists multiple fields, clearly distinguishing from siblings like 'job_status' or 'list_jobs'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Implies usage when full details of a single job are needed, but no explicit when-not or alternatives mentioned. Siblings like 'job_status' could serve a simpler purpose, and this is not addressed.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
show_modelB
Get details of a specific trained model.
| Name | Required | Description | Default |
|---|---|---|---|
| model_id | Yes | Model ID (UUID) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries full burden but only says 'Get details'. It does not disclose read-only nature, required permissions, or what 'details' entails. Lacks transparency beyond basic purpose.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Single sentence, zero waste, perfectly concise. Every word earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite low complexity (1 param, no annotations), description fails to specify what 'details' are returned or any usage constraints. Incomplete for an agent to use effectively.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with a well-described parameter. Description adds no additional meaning about the parameter beyond what the schema already provides. Baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states verb 'get' and resource 'details of a specific trained model'. It distinguishes from siblings like 'list_models' (list all) and other 'show_*' tools for different entities (agents, datasets).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives such as 'list_models' or 'show_evaluation'. Agent receives no context about appropriate usage scenarios.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
validate_s3A
Validate S3 credentials by testing read/write access to the specified bucket. Use before submitting a job with S3 export.
| Name | Required | Description | Default |
|---|---|---|---|
| s3_bucket | Yes | S3 bucket name | |
| s3_access_key_id | Yes | AWS access key ID | |
| s3_secret_access_key | Yes | AWS secret access key | |
| s3_region | Yes | AWS region (e.g. us-east-1) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so description is sole source. It mentions testing read/write access, but does not disclose potential side effects like creating temporary test objects or the exact nature of the validation. Adequate but incomplete.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two concise sentences. First sentence describes action, second provides usage context. No wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema, so description should include return value hints. It does not mention what the tool returns on success/failure. Adequate for purpose and usage but missing behavioral details.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema covers all parameters with descriptions (100% coverage). The description adds no new information beyond schema. Baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states the verb 'validate', resource 'S3 credentials', and context 'bucket'. It distinguishes from sibling tools which deal with jobs, evaluations, models, etc., as it's a validation tool.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly states 'Use before submitting a job with S3 export', giving clear context. No explicit exclusions or alternatives, but the usage is straightforward for a validation tool.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.
23 tool updates
v0.3.5- Added
cancel_evaluation - Added
catalog_export_status - Added
create_dataset - Added
create_evaluation - Added
dataset_status - Added
delete_dataset - Added
estimate_evaluation - Changed
estimate_job1 field changed- added
Input schema / properties / use_caseAdded value: +{ + "description": "Agent to use for the estimate (e.g. 'code_repo' for Cody, 'sera_code_repo' for SIERA). Defaults to code_repo.", + "type": "string" +}
- Added
evaluation_status - Added
export_catalog_model - Added
get_catalog_model - Added
get_inference_jwt - Added
inference_usage - Added
list_agents - Added
list_catalog_models - Added
list_datasets - Added
list_evaluations - Added
list_evaluators - Added
list_inference_models - Changed
list_supported_models1 field changed- added
Input schema / properties / agentAdded value: +{ + "description": "Filter models compatible with this agent (e.g. 'code_repo', 'sera_code_repo'). Omit to see all models.", + "type": "string" +}
- Added
show_agent - Added
show_dataset - Added
show_evaluation
17 tool updates
v1.0.0- First observed
cancel_job - First observed
create_job - First observed
delete_model - First observed
estimate_job - First observed
export_model - First observed
get_account - First observed
get_balance - First observed
import_model - First observed
job_status - First observed
list_jobs - First observed
list_models - First observed
list_supported_models - First observed
model_status - First observed
retry_job - First observed
show_job - First observed
show_model - First observed
validate_s3
TDQS
Each tool targets a distinct resource and action (e.g., create_job vs. cancel_job, list_datasets vs. show_dataset). There is no overlap, and descriptions clearly differentiate related tools like estimate_job and create_job.
All tools follow a consistent verb_noun pattern in snake_case (e.g., list_models, export_model, cancel_evaluation). The naming is uniform and predictable, aiding agent selection.
38 tools is on the higher side but appropriate for a comprehensive fine-tuning platform covering jobs, evaluations, datasets, models, marketplace, inference, and account management. The scope justifies the count.
The tool surface covers the full lifecycle: job creation, estimation, cancellation, retry, and monitoring; dataset and model CRUD; evaluation workflows; account management; marketplace; and inference. No obvious gaps exist.
Maintenance
Resources
Unclaimed servers have limited discoverability.
Looking for Admin?
If you are the server author, to access and configure the admin panel.
Related MCP Connectors
Provision private AI model endpoints on dedicated GPUs (Llama, Qwen, Mistral). Pay per minute.
Your AI Agent's Infrastructure Layer. Connect Claude, Copilot, Codex, or ChatGPT to 200+ managed open source services. Start databases, pipelines, and applications through natural language.
Deterministic compliance and vertical knowledge bases for autonomous agents. Free 24hr trial.
Sovereign Agent OS — Persistent Memory, Governance & Compliance for AI Agents.
Related MCP Servers
- FlicenseAqualityBmaintenanceProvides tools for optimizing, fine-tuning, and deploying large language models with Unsloth, enabling 2x faster training with 80% less memory through model loading, fine-tuning, text generation, and model export capabilities.67-
- AlicenseNot gradedqualityCmaintenanceTransforms AI assistants into a full ML engineering environment for training and fine-tuning models across multiple backends (local GPU, Mistral, Together AI, OpenAI) and cloud providers (Lambda Labs, RunPod, SSH-accessible VPS), with dataset management, experiment tracking, cost estimation, and deployment to Ollama/Open WebUI.3PolyForm Noncommercial 1.0.0
- AlicenseBqualityFmaintenanceExposes vLLM capabilities to AI assistants, enabling chat completions, model management, and platform-aware container control with automatic detection of Docker/Podman and GPU availability across Linux, macOS, and Windows.128Apache 2.0
- AlicenseNot gradedqualityAmaintenancePersistent memory layer for AI agents with entity resolution, PII detection, AES-256-GCM encryption at rest, and hybrid search. Self-hosted. 100% on LoCoMo benchmark.15MIT
Appeared in Searches
Latest Blog Posts
- Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)By Om-Shree-0709 on .mcpAgent IdentityOAuth 2.1
- Your AI Chatbot Just Exposed Your CEO's Salary to an InternBy Om-Shree-0709 on .Agent IdentityMCP SecurityOAuth Delegation
- Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)By Om-Shree-0709 on .Agentic AiPrompt InjectionWebAssembly
MCP directory API
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/cerebrixos-org/tuning-engines-cli'
If you have feedback or need assistance with the MCP directory API, please join our Discord server