Skip to main content
Glama

Tuner MCP

CI Python 3.11+ MCP tools Docker License

Tuner is an independent, agent-native control plane over the public Tinker SDK and Tinker Cookbook. It exposes safe JSON contracts for dataset validation, sampling, supervised fine-tuning, preference learning, RL, distillation, evaluation, checkpoints, and experiment comparison.

Tuner is an independent open-source project built using the public Tinker SDK and Tinker Cookbook. It is not affiliated with or endorsed by Thinking Machines Lab.

Prefer a visual workflow? Tuner UI is the first-party browser interface and delegates live training to this control plane.

Agent skill

SKILL.md gives Codex and other skill-aware agents the complete operating guide for Tuner. It routes agents across all 48 MCP tools and 34 reviewed Cookbook recipes, including dataset preparation, bounded training, monitoring, evaluation, checkpoint management, usage reporting, recovery, and the rule that live Tuner work must use native MCP tools rather than shell or direct API substitutes. Detailed tool, recipe, contract, workflow, and runtime references live under references/.

To make the repository-owned skill available to a local Codex installation on Windows without creating a second copy, link this repository as the skill directory and restart Codex:

New-Item -ItemType Junction `
  -Path "$env:USERPROFILE\.codex\skills\tuner-mcp" `
  -Target (Resolve-Path .)

Related MCP server: ml-party

Use cases

These workflows progress from basic model and data operations to advanced, multi-stage experimentation.

Basic

  1. Discover available Tinker models. List supported models and compare size, context length, modality, and availability.

  2. Find Hugging Face datasets. Search by language, domain, task, author, popularity, or tags.

  3. Inspect and validate datasets. Check splits, columns, message structure, preference pairs, samples, and formatting problems.

  4. Prepare reproducible training data. Convert data into conversation, instruction, preference, prompt, or tool-call formats with deduplication, seeded shuffling, and validation splits.

  5. Preview model-ready examples. Inspect the rendered prompt, token count, truncation, and assistant loss mask before training.

Intermediate

  1. Run bounded supervised fine-tuning. Adapt a model for a language, domain, writing style, instruction-following behavior, or company-specific task.

  2. Train a tool-calling or CLI assistant. Teach command generation and structured tool calls using demonstrations and validation data.

  3. Preference-align a model with DPO. Improve response quality, style, safety, or concision using chosen and rejected answers.

  4. Evaluate and compare models. Measure held-out loss, exact or normalized answers, tool-call correctness, and Cookbook benchmarks.

  5. Manage training and checkpoints. Monitor metrics, inspect failures, stop or resume supported runs, manage retention, and export PEFT or merged Hugging Face models.

Advanced

  1. Train autonomous coding and terminal agents. Combine SFT with Code RL or Harbor/TerminalBench and sandboxed execution feedback.

  2. Distill expert models into smaller models. Use one or multiple teachers with on-policy, off-policy, reasoning, prompt, or self-distillation.

  3. Build search and retrieval agents. Train models to decide when to search, construct queries, interpret evidence, and return grounded answers.

  4. Run multi-stage RLHF or continual specialization. Chain supervised, preference, RL, and continual-learning stages while retaining dataset and checkpoint lineage.

  5. Build a closed-loop improvement workflow. Discover data, run a bounded pilot, evaluate it, inspect failures, create corrective data, retrain, compare checkpoints, and export the selected model. Today these steps are composed from MCP tools; fully automatic multi-stage orchestration remains an implementation target.

Quick start

For this Windows workspace, use the Docker setup below. The configured Codex endpoint is http://127.0.0.1:8765/mcp; run ./scripts/doctor.ps1 to check it. See the tool workflow and limits and current implementation status. The following direct-install commands require Linux or WSL for Cookbook support.

uv sync
$env:TINKER_API_KEY = "..."
uv run tuner --transport stdio

The Cookbook runtime uses CPU PyTorch for model-specific rendering, local tensor preparation, and recipe-specific losses. Tinker still executes all GPU training remotely. The upstream tml-renderers package does not publish a Windows distribution, so run live Cookbook workflows in Docker, Linux, or WSL.

Local Streamable HTTP:

$env:TUNER_AUTH_TOKEN = "<generate-a-long-random-token>"
uv run tuner --transport http --host 127.0.0.1 --port 8000

The HTTP MCP endpoint is http://127.0.0.1:8000/mcp and requires Authorization: Bearer <TUNER_AUTH_TOKEN>. HTTP mode fails closed when the token is absent. Stdio does not require this transport token.

Docker

Run the authenticated HTTP server with Docker Compose:

$env:TINKER_API_KEY = "..."
$env:HF_TOKEN = "..." # optional; increases Hugging Face download limits
$env:TUNER_AUTH_TOKEN = "<a-random-token-at-least-32-characters-long>"
docker compose -f compose.standalone.yaml up --build -d

The service binds only to 127.0.0.1:8765, persists Tuner state in the tuner-state volume, and exposes files under ./data as read-only /data. The standalone Compose file starts its own Redis service and persistent cache.

The workspace-specific compose.yaml can instead use an existing tuner-test-redis container through the external tuner-mcp-network. Set that network up once before using that file:

docker network create tuner-mcp-network
docker network connect tuner-mcp-network tuner-test-redis

The workspace-specific service connects to Redis as redis://tuner-test-redis:6379/0, uses the external tuner-harbor-cache volume, and keeps Hugging Face model downloads in tuner-model-cache. For an update to the existing installation, build docker build -t tuner-mcp:candidate ., run ./scripts/deploy.ps1, then ./scripts/doctor.ps1. Deployment checks for active runs and retains the old image under a timestamped rollback tag. It reads the existing Windows user credentials without printing them. Never run both servers against the same queue/state at once. A client MCP restart may be needed to attach updated tools.

The image installs the pinned official tinker-cookbook[all] distribution and the CPU-only PyTorch wheel. Some recipes still require their own configured services: Modal for code/Harbor execution, Gemini plus Chroma for Search Tool, and external environment credentials for Verifiers.

Dataset format

Conversation JSONL contains one object per line:

{"messages":[{"role":"user","content":"What is 2+2?"},{"role":"assistant","content":"4"}]}

Paths are restricted to TUNER_ALLOWED_ROOTS (semicolon-separated on Windows, colon-separated elsewhere). The current directory is the default allowed root.

Finding Hugging Face datasets

dataset_search_hf searches public Hub datasets by popularity without leaving Tuner:

{"request": {"query": "tool calling function calling", "sort": "likes", "limit": 5}}

Single-term queries (function-calling, terminal-bench, shell) rank best. Each hit returns hf_repo, pinned sha, likes, downloads, and the gated flag — pass hf_repo + sha to dataset_probe_hf before dataset_fetch_hf. Search is read-only and free; Hub outages surface as retryable DATASET_ERROR, never fabricated entries.

Fetching Hugging Face datasets

dataset_probe_hf first inspects configuration names and samples a pinned split to report compatible mappings. dataset_fetch_hf then streams that pinned public dataset split into a staged, validated dataset_id usable by every training tool:

{
  "request": {
    "hf_repo": "teknium/OpenHermes-2.5",
    "hf_revision": "<40-character commit SHA>",
    "hf_split": "train",
    "output_type": "conversation_jsonl",
    "max_records": 1000,
    "selection_strategy": "reservoir",
    "scan_records": 10000,
    "shuffle_seed": 30
  }
}

Raw xLAM datasets are also supported without generating replacement records. Use the mapping returned by dataset_probe_hf; for the original Salesforce dataset the request has this shape:

{
  "request": {
    "hf_repo": "Salesforce/xlam-function-calling-60k",
    "hf_revision": "<40-character commit SHA from dataset_search_hf>",
    "hf_split": "train",
    "output_type": "conversation_jsonl",
    "user_field": "query",
    "tool_calls_field": "answers",
    "tools_field": "tools",
    "max_records": 1000,
    "validation_records": 100,
    "shuffle_seed": 30
  }
}

Rules: the revision must be the full 40-character commit SHA (no branch names, so a re-resolve can never silently change the data); trust_remote_code is always off; rows are mapped to messages (ShareGPT conversations convert automatically, JSON-encoded message strings are parsed), preference triples, or prompt text depending on output_type; instruction-style rows map via user_field/assistant_field (e.g. instruction→user, cmd→assistant); unmapped rows fail fast naming the record index and available fields. The original Salesforce xLAM repository is gated: accept its Hub terms and configure HF_TOKEN in the server environment. When Hub data is requested, Tuner reports access or schema blockers instead of silently replacing it with synthetic inline records. dataset_prepare accepts the same hf_repo/hf_revision pair for one-step local-or-HF staging.

Use selection_strategy: "reservoir" with a fixed shuffle_seed to select a deterministic representative sample from scan_records; the default head strategy reads only max_records. Prepared manifests report prompt diversity without discarding Albanian or other non-Latin scripts, plus distributions for available intent, category, domain, and language metadata.

dataset_prepare also accepts up to 1,000 inline_records, so an MCP client can persist synthetic or hand-authored tool-use demonstrations without creating a server-local file:

{
  "request": {
    "inline_records": [
      {
        "messages": [
          {"role": "user", "content": "Find the latest run."},
          {
            "role": "assistant",
            "content": null,
            "tool_calls": [
              {
                "type": "function",
                "function": {"name": "training_list", "arguments": "{\"limit\":1}"}
              }
            ]
          }
        ],
        "tools": [
          {
            "type": "function",
            "function": {
              "name": "training_list",
              "description": "List runs",
              "parameters": {"type": "object", "properties": {"limit": {"type": "integer"}}}
            }
          }
        ]
      }
    ],
    "output_type": "conversation_jsonl",
    "validation_records": 0
  }
}

Stored provenance contains the inline count and content hash rather than a second copy of the full inline payload.

Preference datasets can map common Hub field names explicitly with preference_prompt_field, chosen_field, and rejected_field. The probe returns these selectors when it recognizes fields such as instruction/chosen_response/rejected_response; staged output always uses canonical prompt or messages, chosen, and rejected keys. Full-conversation chosen/rejected rows that repeat the shared prompt are normalized to assistant-only completions. Set invalid_record_policy to skip to omit invalid rows and record removal counts in the transform manifest; the default error policy fails preparation.

Conversation rows may carry a top-level tools list. Tuner preserves tool schemas, tool call IDs, assistant calls with null content, tool results, and Cookbook message fields such as name and unparsed_tool_calls.

Selecting an experiment automatically

experiment_autoplan combines live Tinker capabilities, Cookbook renderer metadata, and Hugging Face candidates that pass schema probing. It selects a trainable model and returns an exact dataset_fetch_request, a staged recipe sequence, and blockers. Multi-configuration datasets include hf_config:

{
  "request": {
    "objective": "Train a CLI assistant that makes reliable tool calls",
    "task": "terminal_agent",
    "constraints": {"max_model_params_billions": 10}
  }
}

Use recipes_list and recipe_get to discover every reviewed official Cookbook recipe. recipe_plan validates exact upstream config fields for a selected recipe and recipe_start runs that immutable plan through the same Redis-backed execution lifecycle as the typed SFT/DPO/RL/distillation tools.

Safety and cost

Live sampling, evaluation, and training require TINKER_API_KEY and may incur charges. Normal tests never make live Tinker calls. Training limits are controlled by TUNER_MAX_TRAINING_STEPS, TUNER_MAX_DATASET_BYTES, TUNER_MAX_SAMPLES, and TUNER_MAX_GENERATION_TOKENS. TUNER_MAX_TOTAL_GENERATION_TOKENS caps aggregate generated tokens per operation, and TUNER_MAX_PROMPT_BYTES bounds direct sampling/logprob request payloads.

TUNER_TASK_URL defaults to memory:// for the local MVP. A remote deployment can point it at a supported Redis/Valkey URL and run the corresponding worker model.

Run metadata is persisted in SQLite under the state directory. Existing JSON run records are imported without deleting the originals. Idempotency keys are scoped to the operation: the same request replays its run, and changed content returns IDEMPOTENCY_CONFLICT. Admission and execution claims are transactional across processes sharing that database.

Training and evaluation use the shared controller for concurrency limits, cancellation records and heartbeats. Startup marks stale active runs as needing reconciliation; it does not replay training. Cancellation stops local orchestration; already submitted remote work may continue. training_resume starts a new SFT attempt from saved optimizer/epoch/batch state; train_dpo, train_rl and train_distill are exposed. Typed SFT and DPO have completed bounded live verification; typed RL and distillation have also completed bounded live verification. The current public-release evidence and remaining clean-machine gate are tracked in docs/RELEASE_TEST_MATRIX.md.

training_metrics returns recent metrics before completion. Pass cursor=0 to page complete JSONL records and use the returned byte cursor for the next page. training_logs lists nested artifacts and inlines only metrics/checkpoint JSONL by default; pass artifact_path and cursor to read bounded chunks. TUNER_MAX_ARTIFACT_BYTES caps each read. Training admission also checks TUNER_MAX_BATCH_SIZE, TUNER_MAX_INPUT_TOKENS and TUNER_MAX_CONCURRENT_RUNS.

Typed plans report method-specific natural steps after holdouts, whether each important hyperparameter was explicit or resolved from Cookbook defaults, the Cookbook learning-rate recommendation, and dataset concentration warnings. Distillation currently accepts exactly one epoch because its typed adapters make one dataset pass.

Capabilities distinguish credential presence from verified connectivity. live_available is unknown (null) until a live capability check succeeds. Evaluation is marked as mutating because it creates run records and artifacts. Custom evaluation supports exact, normalized exact, structured tool-call, and domain-rubric scoring. Rubric scoring requires a separate Tinker judge model and stores bounded judge evidence. compare_runs returns inconclusive instead of a winner for truncated, incompatible, undersized, below-threshold, or statistically weak results. train_sft/dpo/rl/distill and evaluate accept background=true to submit through Docket and return a run_id promptly; native task clients get progress the same way. checkpoint_export queues work by default and promptly returns an export_id; poll it with training_get and stop it with training_stop. Pass background=false only when the archive or conversion is known to fit the client deadline. It returns a signed archive URL (tinker_archive) or builds a PEFT adapter / merged HF model (peft/hf_merged). PEFT preserves the adapter produced by Tinker's remote checkpoint exporter and does not download the base model. Merged-HF runs through Cookbook on Linux/WSL and needs the full base model; the persistent model cache avoids downloading that model again on later merges.

MCP client configuration

{
  "mcpServers": {
    "tuner": {
      "command": "uv",
      "args": ["--directory", "C:/path/to/tuner", "run", "tuner"]
    }
  }
}

Run uv run tuner-discover to refresh generated/tinker_api_manifest.json after updating either upstream repository.

Available Tools

48 tools
benchmarks_listBenchmarks ListA
Read-only

List benchmark names discovered from installed Cookbook source.

Inputs: no named parameters. Returns: object

capabilities_getCapabilities GetB
Read-onlyIdempotent

Describe Tuner's tool surface and optionally fetch live Tinker server capabilities.

Inputs: live. Returns: object

checkpoint_deleteCheckpoint DeleteB
Destructive

Permanently delete the checkpoint at this exact Tinker path.

Inputs: tinker_path (required). Returns: object

checkpoint_exportCheckpoint ExportB
Idempotent

Queue a checkpoint export, or wait when background=false.

Inputs: format (required), tinker_path (required), background, base_model, +1 more. Returns: object

checkpoint_getCheckpoint GetA
Read-only

Get checkpoint weight metadata using an exact tinker:// path.

Inputs: tinker_path (required). Returns: object

checkpoint_listCheckpoint ListB
Read-only

List training and sampler checkpoints for a Tinker training run.

Inputs: training_run_id (required). Returns: object

checkpoint_publishCheckpoint PublishB
Idempotent

Make this exact checkpoint publicly accessible on Tinker.

Inputs: tinker_path (required). Returns: object

checkpoint_set_ttlCheckpoint Set TtlB
Idempotent

Set checkpoint retention; null requests indefinite retention.

Inputs: tinker_path (required), ttl_seconds (required). Returns: object

checkpoint_unpublishCheckpoint UnpublishB
Idempotent

Remove public access to this exact checkpoint on Tinker.

Inputs: tinker_path (required). Returns: object

compare_runsCompare RunsC
Read-onlyIdempotent

Compare compatible evaluations using an optional explicit metric policy.

Inputs: baseline_run_id (required), candidate_run_id (required), metric, direction. Returns: object

compute_logprobsCompute LogprobsB
Read-only

Compute prompt token log probabilities for a base model or checkpoint.

Inputs: request (required). Returns: object

dataset_fetch_hfDataset Fetch HfC

Fetch a pinned Hugging Face split, map rows to a Tuner schema, stage a dataset ID.

Inputs: request (required). Returns: object

dataset_inspectDataset InspectC
Read-onlyIdempotent

Validate and return a small preview of a local dataset.

Inputs: dataset (required), sample_size. Returns: object

dataset_prepareDataset PrepareB

Stage validated local or pinned Hugging Face data as a persistent dataset ID.

Inputs: request (required). Returns: object

dataset_probe_hfDataset Probe HfC
Read-only

Probe configs and row mappings for a pinned Hugging Face dataset split.

Inputs: request (required). Returns: object

dataset_render_previewDataset Render PreviewB
Read-only

Preview Cookbook rendering and loss masks without submitting training.

Inputs: model, dataset, plan_id, renderer, +1 more. Returns: object

dataset_search_hfDataset Search HfC
Read-only

Search public Hugging Face datasets by popularity; returns repo IDs and SHAs.

Inputs: request (required). Returns: object

dataset_validateDataset ValidateB
Read-onlyIdempotent

Validate a local JSON/JSONL dataset, reporting exact malformed record indexes.

Inputs: dataset (required). Returns: object

evaluateEvaluateC

Run a Cookbook benchmark and persist evaluation artifacts. This spends credits.

Inputs: request (required), background. Returns: object

evaluation_failuresEvaluation FailuresB
Read-only

Read evaluation trajectories with errors or non-positive rewards.

Inputs: evaluation_id (required), limit, offset. Returns: object

evaluation_getEvaluation GetB
Read-only

Inspect an evaluation's persistent status, benchmark scores and artifacts.

Inputs: evaluation_id (required). Returns: object

experiment_artifactsExperiment ArtifactsB
Read-only

List a run's files or read one artifact using a bounded byte cursor.

Inputs: run_id (required), path, limit, cursor, +1 more. Returns: object

experiment_autoplanExperiment AutoplanC

Select live models, Cookbook recipes, and pinned HF candidates without training.

Inputs: request (required). Returns: object

experiment_rolloutsExperiment RolloutsC
Read-only

Read a bounded page of persisted rollout or evaluation trajectories.

Inputs: run_id (required), limit, offset. Returns: object

models_listModels ListA
Read-onlyIdempotent

List Cookbook-known models; live mode returns authoritative server-supported models.

Inputs: live. Returns: object

object_getObject GetB
Read-only

Retrieve a saved dataset or resolved plan by its persistent ID.

Inputs: object_id (required). Returns: object

objects_listObjects ListC
Read-only

Recover saved datasets and plans after reconnecting; returns bounded summaries.

Inputs: kind (required), limit, offset. Returns: object

recipe_getRecipe GetB
Read-only

Inspect a recipe's typed configuration and execution requirements.

Inputs: recipe (required). Returns: object

recipe_planRecipe PlanB

Validate and freeze an exact config for one allowlisted Cookbook recipe.

Inputs: request (required). Returns: object

recipes_listRecipes ListC
Read-only

Discover recipe requirements, availability and verification status.

Inputs: no named parameters. Returns: object

recipe_startRecipe StartA
Idempotent

Submit a reviewed official Cookbook recipe. This may spend Tinker credits.

Inputs: plan_id (required), idempotency_key (required). Returns: object

sampleSampleC
Read-only

Sample a base model or checkpoint through the model-recommended Cookbook renderer.

Inputs: request (required). Returns: object

session_getSession GetA
Read-only

Inspect one owned Tinker session and its remote run identifiers.

Inputs: session_id (required). Returns: object

sessions_listSessions ListC
Read-only

List sessions owned by the server's Tinker account.

Inputs: limit, offset. Returns: object

session_trace_exportSession Trace ExportB

Save an owned session trace as a local artifact for bounded inspection.

Inputs: session_id (required). Returns: object

train_distillTrain DistillC

Run on-policy or off-policy teacher/student distillation. Spends credits.

Inputs: request (required), background. Returns: object

train_dpoTrain DpoC

Train on chosen/rejected pairs using Cookbook DPO. Spends credits.

Inputs: request (required), background. Returns: object

training_getTraining GetC
Read-only

Get a Tuner workflow record or a Tinker training run by ID.

Inputs: run_id (required), source. Returns: object

training_listTraining ListB
Read-only

List remote Tinker runs or persistent local Tuner workflow records.

Inputs: limit, offset, source. Returns: object

training_logsTraining LogsC
Read-onlyIdempotent

Read bounded recursive logs, or an artifact path with a byte cursor.

Inputs: run_id (required), cursor, offset, max_lines, +1 more. Returns: object

training_metricsTraining MetricsA
Read-onlyIdempotent

Read latest metrics, or page from a byte cursor (start at zero) while training runs.

Inputs: run_id (required), limit, cursor, artifact_path. Returns: object

training_planTraining PlanA

Prepare an immutable training plan and report blockers; does not launch training.

Inputs: request (required), objective. Returns: object

training_resumeTraining ResumeB
Idempotent

Resume SFT with total max_steps or additional_steps after checkpoint. Spends credits.

Inputs: run_id (required), idempotency_key (required), max_steps, num_epochs, +1 more. Returns: object

training_startTraining StartA
Idempotent

Submit a validated plan to Docket and return its run ID promptly. Spends credits.

Inputs: plan_id (required), idempotency_key (required). Returns: object

training_stopTraining StopB
Idempotent

Stop local orchestration. Already submitted remote work may continue.

Inputs: run_id (required). Returns: object

train_rlTrain RlC

Run an allowlisted arithmetic/math group-rollout RL recipe. Spends credits.

Inputs: request (required), background. Returns: object

train_sftTrain SftC

Run Cookbook supervised fine-tuning. This operation spends credits.

Inputs: request (required), background. Returns: object

usage_getUsage GetA
Read-only

Get account usage for a half-open YYYY-MM-DD date range; preserve upstream units.

Inputs: starting_on (required), ending_before (required). Returns: object

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    C2.8/5.0

    Scored across 48 tools

    Disambiguation3/5

    Most tools are grouped by resource and action, but the large set has several close planning/staging tools (experiment_autoplan/training_plan/recipe_plan, dataset_fetch_hf/dataset_probe_hf/dataset_prepare, dataset_validate/dataset_inspect) that could cause misselection. Descriptions help, but boundaries are not always obvious.

    Naming Consistency3/5

    Names consistently use snake_case and resource prefixes, but action position varies: list/get tools are noun_verb (recipes_list, checkpoint_get) while training actions are verb_noun (train_sft, compute_logprobs) and some are bare verbs (evaluate, sample). The prefix grouping keeps it readable, but the mixed conventions are not fully predictable.

    Tool Count2/5

    48 tools is far beyond the 25+ threshold and would overwhelm an agent even though the domain spans training, datasets, checkpoints, and evals. Several narrow operations (dataset_validate vs dataset_inspect, checkpoint_* variants) could be consolidated.

    Completeness4/5

    The surface covers the full training lifecycle: datasets, recipes, plan/start/resume/stop training, multiple RL/DPO/SFT/distill methods, checkpoint management, evaluation, sessions, and usage. Minor gaps exist (no evaluation_list, no training/cancel/delete for remote runs, no dataset deletion), but agents can complete core workflows.

    Maintenance

    ActivityStale
    ResponsivenessOutdated data

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      A
      quality
      C
      maintenance
      Enables AI agents to browse and filter experiment runs, retrieve metrics, logs, tables, and artifacts, plot comparisons as PNG images, and download full-fidelity training history for local analysis against a self-hosted Open Train server. It also provides diagnostic tooling that surfaces metric evidence and investigation leads without asserting causality.
      16
      MIT
    • A
      license
      Not graded
      quality
      B
      maintenance
      Enables agents to track ML experiments with a reproducible write contract, internal git snapshots, and a queryable knowledge graph, plus a live web UI for humans.
      Apache 2.0
    • A
      license
      Not graded
      quality
      B
      maintenance
      Enables agents to turn a real model failure into a deterministic benchmark task by forging it with exact assertions, grading the task itself across six weighted factors, and inspecting a tamper-evident SHA-384 audit trail. Exposes eleven typed JSON-RPC 2.0 tools that call the same idempotent service layer as the UI, so mutations and decisions stay sealed and reproducible.
      MIT