Skip to main content
Glama

A/B test titles, thumbnails or descriptions

create_ab_test

Runs a balanced A/B test of up to three challenger title, thumbnail, or description versions against a live YouTube video, compares analytics, and optionally applies the significant winner.

Instructions

Tests 1-3 challenger versions of a live video against its current version (A). The server rotates which version is live one full day at a time (Pacific days, matching YouTube Analytics), in balanced random order so weekdays even out, then compares daily views, watch time, average view percentage, subscribers or engagement, adjusting for the video's natural decline in daily views, with a significance test. At the end it restores A, waits ~3 days for analytics to land, and (if auto_apply_winner) puts a significant winner live. YouTube Studio's own Test & Compare (with impressions click-through rate) has no API; this works on any channel through the official APIs. Each variant can change title, thumbnail and/or description. Changes are public, so the user confirms first.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "video_id": "<video_id>",
      "variants": "<variants>"
    },
    "name": "create_ab_test"
  }
}

Input Schema

NameRequiredDescriptionDefault
daysNoTest length; at least 3 days per version.
startNo"tomorrow" (recommended: full clean days), "today", or YYYY-MM-DD (Pacific).tomorrow
metricNoWhat decides the winner.views
channelNoWhich connected channel: channel ID (UC...), @handle or name. Omit to use the default channel.
variantsYesChallengers B, C, D. Fields left out stay as in A.
video_idYes
auto_apply_winnerNo
confirmation_tokenNoOnly after the user approved: the confirmation_token returned by the previous call with identical arguments.
original_thumbnail_pathNoThe current thumbnail file (needed for thumbnail tests when YouTube has no 1280x720 copy of it).

Schema Changelog

Changes observed during successful MCP inspections.

  1. Addedv1.1.0

TDQS

A4.5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the annotations, the description discloses substantial behavior: balanced random day-by-day rotation in Pacific days, natural-decline adjustment, a significance test, restoration of version A, a ~3 day analytics lag, and the auto_apply_winner end state. It also exposes the two-step confirmation flow and the fact that changes go public, which is exactly the safety context an agent needs for a non-idempotent mutation.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The purpose and preconditions are front-loaded and every sentence carries information — rotation schedule, comparison metrics, restoration, confirmation. It is a single dense paragraph, though, and parenthetical detail like 'Pacific days, matching YouTube Analytics' and the weekday balancing could be compressed or split for scanability.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 9-parameter, no-output-schema, long-running mutation, the description covers the lifecycle (start, rotate, compare, restore, apply) and the confirmation requirement thoroughly. The main gap is that it never says how to monitor the running test — the sibling tools list_ab_tests and get_ab_test would need to be discovered independently — and it does not hint at what the initial call returns besides the schema-documented confirmation_token.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

At 78% schema coverage the baseline is 3, but the description adds real meaning for the least self-explanatory parameters: it explains what auto_apply_winner actually does (publishes a significant winner after restoration), how variants are interpreted (each may change title, thumbnail and/or description), and the confirmation_token gating. It does not add much for days, start or metric beyond what the schema already documents.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb and resource ('Tests 1-3 challenger versions of a live video against its current version'), and the rotation/significance-test mechanics make it unmistakable against siblings such as stop_ab_test, list_ab_tests and get_ab_test. It also names the thing it replaces (YouTube Studio's Test & Compare) so an agent can place it precisely.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives clear context for when to use it (testing title/thumbnail/description variants on a live video, on any channel, without Studio API access) and states the precondition that the user must confirm because changes are public. It stops short of explicit routing rules such as checking for an existing test with list_ab_tests first or cancelling via stop_ab_test, leaving those to inference from sibling names.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.