Skip to main content
Glama
Sign In

Testsy

by dev.testsy
OverviewDiscussionsAdmin

Create a test

create_test

Create a test of a website (url) or of an app (platform android or ios, and appId; device phone or tablet and osVersion, default a phone on the latest OS). Pass plan (from plan_test, after the requester confirmed it): it is stored as the approved checklist and the tester follows it as written. Or pass requirement, plain-language text the tester plans itself at run time. Never put passwords in the text: name a sign-in account in credentials.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "projectId": "<projectId>",
      "title": "<title>"
    },
    "name": "create_test"
  }
}

Input Schema

NameRequiredDescriptionDefault
urlNoStart address (https://…); omit for app tests
planNoA plan from plan_test: {title, startUrl, steps[], checks[], record[], assumptions[], credentials[]}. Stored as the approved checklist; the tester follows it as written.
appIdNoApps: package name or bundle id, for example com.example.app (see list_builds)
titleYesShort title (up to 200 characters)
deviceNoWebsites: screen, desktop (default), tablet or mobile. Apps: phone (default) or tablet — an iPhone or iPad, an Android phone or tablet (see list_devices)
buildIdNoApps: pin this build (ab_…); default: the app's newest build
optionsNoTest options, all optional: device (desktop | tablet | mobile; websites), timeoutMinutes (2-20; 20 when left out, the app sets 15), recordVideo (default true), trackers (block | allow; websites, default block), jsErrors (report | fail; websites, default report: errors from the site's own scripts are reported and fail the run only when one breaks a step; fail = any of them fails it), credentials (names of the project's sign-in accounts), services (email = its own mailbox, sms and call = its own phone number, http = the project's endpoints; sms, call and http need a paid plan), devices (1 or 2 phones; apps), platform (web | android | ios), appId and buildId (apps), buildUrl (apps: an https link to the build, downloaded every run), orientation (apps: default | portrait | landscape — the screen's shape when the run starts; default is the device's own).
platformNoweb (default), android or ios
osVersionNoApps: the OS version, latest (default) or a version — iOS: iOS 27, iOS 26, iOS 18, iOS 17; Android: Android 16, Android 15, Android 14, Android 13, Android 12, Android 11, Android 10 (see list_devices)
projectIdYesProject id (p_…)
credentialsNoNames of the project's sign-in accounts the tester may use (see list_credentials)
orientationNoApps: the screen's shape when the run starts — default (the device's own: phones and iPads upright, Android tablets on their side), portrait or landscape. The test may still rotate the device during the run.
requirementNoWhat to do and check, in plain language (optional when plan is given)

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observed

TDQS

A4.1/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare this is a non-readonly, non-destructive, non-idempotent create. The description adds meaningful behavior beyond that: a `plan` is persisted as the approved checklist and the tester follows it literally, whereas a `requirement` is planned by the tester at run time — a substantive difference in how the run behaves. The security caveat about passwords and credential names adds more. It never states that each call produces a distinct new test (relevant to idempotentHint=false) or what identifier comes back, so a 4 rather than 5.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three sentences, front-loaded with the core action and the plan/requirement decision. The first sentence, however, is overloaded with parenthetical parameter defaults ("device phone or tablet and osVersion, default a phone on the latest OS") that the schema states more cleanly, which is wasted space in the highest-attention position.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 13-parameter create with a nested `options` object and no output schema, the description covers the critical decisions — target type, planning mode, and credential handling — plus the plan-from-plan_test dependency. It omits what the call returns (a test id, presumably needed before run_test) and doesn't note that credential names must already exist, but nothing needed to invoke it correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% across all 13 parameters, so the schema already documents types, defaults, and enum-like values. The description mostly restates schema content (device defaults, platform choices) rather than adding new semantics. The one genuine addition is that `plan` is stored as the approved checklist and honored as written, which is a 3-level contribution given how much the schema already carries.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb and resource ("Create a test") and immediately splits the two supported targets: a website via `url`, or an app via `platform` plus `appId`. It goes further than the title by naming the two authoring modes (`plan` vs `requirement`), so an agent can tell exactly what this tool produces.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives explicit conditional guidance: pass `plan` from plan_test "after the requester confirmed it", or pass `requirement` for runtime planning, which is a real decision rule naming a sibling. It also states a hard exclusion ("Never put passwords in the text") and points to `credentials` instead. It does not cover when to prefer update_test/create_schedule over a fresh create, so it falls short of a full 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All Testsy toolsInspect
Try in Browser
Claude Code, inside the Godot 4 editorFlockbay puts an AI agent in the real Godot 4 editor: it writes GDScript, runs your game and fixes what breaks. Free, or bring your own Claude Code or Codex.flockbay.comAd

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources