This server is a read-only MCP bridge to Squally CI/test data. It lets an agent inspect projects, CI runs, per-test results, failure details, test stability metrics, flaky/broken test lists, and grouped error signatures — all over HTTPS GETs, with no write access.
List projects (squally-list-projects): get project IDs, names, stable branches, and creation dates — the required starting point for every other tool.
Find CI runs (squally-find-run): search runs by branch, commit SHA prefix, or status (passed/failed) within 7/30/90-day windows; returns summary counters and pagination cursors.
Get one run (squally-get-run): fetch a full run with per-test rows across all shards — test names, file paths, statuses, attempt counts, durations, error messages, and shard info — to see which test is red.
Debug a failing test (squally-debug-failure): get every attempt of one test in one run, including error messages, stacks, code snippets, and the Copy-for-AI prompt (which may include an ARIA snapshot of the app at failure).
Get a test's status (squally-get-test-status): read the stored verdict (healthy/flaky/broken/insufficient_data), active signals, and labels for a single test.
List flaky/broken tests (squally-list-flaky-tests): rank the project's flaky and broken tests by time lost, flake rate, or stable-branch impact, with scores, failure streaks, and last-run details.
List error signatures (squally-list-errors): see failures grouped by fingerprint across runs/branches — occurrence counts, affected files/tests, first/last seen, and whether the signature is new.
Get per-test metrics (mentioned in README as squally-get-test-metrics): stability, flaky rate, failure rate, and time lost for one test over 7/14/30/90 days.
List tests (mentioned in README as squally-list-tests): rank all tests by stability, flaky rate, or failures over a period.
Read-only by design: every tool is annotated readOnlyHint: true; the server only issues HTTP GETs and cannot write, mutate, or access artifacts like screenshots/videos/traces.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@squally-mcp why is checkout.spec.ts failing on main?"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
A local MCP server that lets a coding agent read your Squally data: which CI runs happened, how stable each test is, and why a particular test failed.
It runs on your machine over stdio and talks to Squally's read API over HTTPS.
Point Claude Code, Codex, Cursor or Claude Desktop at it and ask "why is
checkout.spec.ts failing on main?".
The server issues HTTP GETs and nothing else. There is no code path that
writes, and every tool is annotated readOnlyHint: true, so a client can run
them without asking you to confirm each one.
It also cannot reach past your own organization: the key is scoped to one organization, and scoping, plan checks and secret exclusion are all enforced by the API, not by this client.
Related MCP server: Tesults MCP
Worth knowing before you hand a key to an assistant. Through this server an agent can read:
your projects, their names and stable branches;
your CI runs — commit SHA and message, branch, author, pull request, timing, pass/fail counts;
per-test results for a run, including which shard ran what;
for a failing attempt: the error message and stack, the code snippet — verbatim source code from your test file — and the Copy-for-AI prompt, which carries the same snippet plus an ARIA snapshot of your application at the moment of failure (whatever was on screen, truncated to 3000 characters);
per-test metrics: how often each test passed on the first try, passed only after a retry, or failed, and how much time retries and failures cost;
error signatures: failures grouped across runs and branches.
It cannot see screenshots, videos or traces — artifacts are deliberately out of scope — and it cannot see any key, secret or webhook URL.
The key reaches every project in the organization. Create one per machine so a single revocation does not lock out everything, and revoke it in Settings → API keys when the machine is retired.
You need an organization read key (sqly_ro_…): Squally → Settings → API
keys → Create read key. It is shown once. It is not the same thing as the
project ingest key your Playwright reporter uses — this server refuses that one
by name.
Requires Node 22 or newer. Read keys are available from the Standard plan.
claude mcp add squally --scope user \
--env SQUALLY_API_KEY=sqly_ro_your_key_here \
-- npx -y squally-mcp~/.codex/config.toml:
[mcp_servers.squally]
command = "npx"
args = ["-y", "squally-mcp"]
env = { SQUALLY_API_KEY = "sqly_ro_your_key_here" }~/.cursor/mcp.json (or .cursor/mcp.json in a project):
{
"mcpServers": {
"squally": {
"command": "npx",
"args": ["-y", "squally-mcp"],
"env": { "SQUALLY_API_KEY": "sqly_ro_your_key_here" }
}
}
}claude_desktop_config.json — macOS:
~/Library/Application Support/Claude/, Windows: %APPDATA%\Claude\ — then
restart the app:
{
"mcpServers": {
"squally": {
"command": "npx",
"args": ["-y", "squally-mcp"],
"env": { "SQUALLY_API_KEY": "sqly_ro_your_key_here" }
}
}
}Variable | Required | Meaning |
| yes | Organization read key, |
| no | Base URL, default |
| no |
|
squally-mcp trusts your operating system's certificate store, like your browser does; set SQUALLY_USE_SYSTEM_CA=0 to use Node's bundled certificates only.
The server does not start in time. The first start through npx
downloads the package, which can take longer than Claude Desktop's 60-second
start-up limit. Install it once beforehand with npm install -g squally-mcp;
npx -y squally-mcp then runs that copy without a download. Update it with
the same command.
Where the log is. The Microsoft Store build of Claude Desktop writes it to
%LOCALAPPDATA%\Claude\logs\mcp-server-squally.log. Its first lines say which
Node runs the server and whether it trusts the system's certificate store;
a request that never arrives is logged there with its cause.
Edit the config only while the app is closed. Quit Claude Desktop
completely, including from the system tray, before changing
claude_desktop_config.json.
Tool | What it answers | Cost |
| Which projects exist. Start here — every other tool needs a | cheap |
| Which runs happened — latest, or by branch, commit SHA or status. Counters only. | cheap |
| One run with its per-test rows, across all shards. Which test is red. | cheap |
| Every attempt of one test in one run: error, stack, Copy-for-AI prompt. | cheap |
| One test's stability, flaky rate and failure rate over a period. | cheap — one test |
| Every test with a CI run in the period, ranked by stability, flaky rate or failures. | cheap — two queries, paged |
| Error signatures in a period: what keeps failing, grouped. | cheap |
For a single test use squally-get-test-metrics, not squally-list-tests —
the tool descriptions say so, and the server repeats it in its instructions.
The two test tools count completed CI runs in the period (7, 14, 30 or 90 days; 14 by default), on every branch unless you name one — without runs where most of the suite failed at once. Local runs never count. Per test:
stability — runs that passed on the first try ÷ runs;
flakyRate — runs that passed only after a retry ÷ runs;
failureRate — failed runs ÷ runs.
Each is a fraction from 0 to 1, and null when the test has no run in the
period. There is no verdict: nothing calls a test flaky, broken or
healthy. The tools return the numbers; the agent — or you — judges them.
squally-list-tests{ "projectId": "3f7c2a91-5b8e-4d0a-9c61-2e4b7f0d8a15", "perPage": 3 }The three least stable tests of the last 14 days (abridged):
{
"population": "ci_completed_not_excluded",
"days": 14,
"sort": "stability",
"total": 409,
"items": [
{
"testName": "chromium > checkout.spec.ts > Checkout > applies a coupon code",
"filePath": "tests/checkout.spec.ts",
"runs": 8,
"stability": 0.5,
"flakyRate": 0.5,
"failureRate": 0,
"timeLostMs": 7361,
"topBranch": { "branch": "main", "kind": "retries", "count": 3 }
}
]
}squally-get-test-metrics{
"projectId": "3f7c2a91-5b8e-4d0a-9c61-2e4b7f0d8a15",
"testName": "chromium > checkout.spec.ts > Checkout > applies a coupon code"
}The same numbers as that test's row, for one test (abridged):
{
"days": 14,
"branch": null,
"runs": 8,
"stableRuns": 4,
"flakyRuns": 4,
"failedRuns": 0,
"stability": 0.5,
"flakyRate": 0.5,
"failureRate": 0,
"branches": ["feature/saved-carts", "main"]
}Pass the name exactly as Squally shows it. When it exists in several spec
files, the tool answers with the filePath values to retry with; a test with
results but no completed CI run in the period answers runs: 0 and null
rates — an answer, not an error.
npm install
npm run build # tsc, then copy the vendored OpenAPI document into dist/
npm test # builds, then runs the suite
npm run vendor:openapi # re-fetch the API document; review the diff, then re-pinThe tool schemas are derived from Squally's published OpenAPI document,
vendored at src/openapi/v1.json — an input schema is the operation's
parameters, an output schema is its 200 response. Nothing is transcribed by
hand, so the two cannot drift. test/drift.test.js compares the vendored copy
against the live document and fails when an operation or parameter has moved;
it skips loudly when offline, and while the live API is still an older version
than the vendored one — a release that has not been deployed yet.
Publishing is staged: CI uploads the tarball, a maintainer approves it, and
only then is the version installable. npm stage publish never asks for 2FA,
which is what makes it usable from a workflow; the 2FA prompt moves to the
approval step, so a compromised workflow can stage a version but cannot put one
in front of users. Direct npm publish is not permitted for this package's
trusted publisher, so this is the only route.
0.1.0 is the exception. It was published by hand with
npm publishbefore the trusted publisher existed, so it carries no provenance attestation. Every version from 0.1.1 on goes through the steps below.
Bump and tag. The lockfile belongs to the release commit, so let npm version write both and commit them together:
npm install # only if package.json changed by hand
npm version patch # or minor / major - writes package.json + lockfile, makes the tag
git push --follow-tagsThe tag starts the workflow. .github/workflows/publish.yml runs on
v*: it tests, checks that the tag and package.json agree, and runs
npm stage publish --provenance --access publicthrough npm's trusted publishing (OIDC) — no npm token in GitHub secrets.
--provenance attaches a signed statement linking the tarball to that
workflow run and commit.
Approve the staged version. On npmjs.com/package/squally-mcp → Staged Packages → Approve, which asks for 2FA. Or from a terminal:
npm stage list squally-mcp
npm stage view <stage-id> # what is in the tarball
npm stage approve <stage-id> # also asks for 2FA
npm stage reject <stage-id> # if something is wrongUntil this step the version exists in the registry but installs nothing.
Requirements for step 3 on your machine: npm 11.15.0 or later and Node
22.14.0 or higher (npm docs).
Node 22 still bundles npm 10.9.9, so check npm --version rather than assuming
your Node version brought a new enough npm. The workflow uses Node 24 for the
same reason and fails with that requirement if a runner ever ships an older npm.
MIT
Every attempt of one test in one run, oldest first, each with the error, the stack and the Copy-for-AI prompt (the same text the Squally UI puts on the clipboard). This is the tool for 'why did this test fail'. If the test name is ambiguous, pass filePath.
| Name | Required | Description | Default |
|---|---|---|---|
| runId | Yes | A run id from GET /projects/{projectId}/runs - or any shard's id of a sharded run. | |
| filePath | No | The spec file of the test, as stored (e.g. tests/checkout.spec.ts); an empty value means a test with no file. Pass it exactly as shown in Squally; do not URL-encode, the server does. Needed only when the name exists in several files - the ambiguous_test error lists them. | |
| testName | Yes | The full test name as Squally stores it (the Playwright title path), for example "checkout flow > completes payment". Pass the name exactly as shown in Squally; do not URL-encode, the server does. A name used in several spec files also needs filePath. | |
| projectId | Yes | A project id from GET /projects. |
| Name | Required | Description |
|---|---|---|
| run | Yes | |
| test | Yes | |
| project | Yes | |
| attempts | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already establish readOnly, idempotent, non-destructive behavior. The description adds genuinely useful behavioral detail: results are grouped by test/run, ordered oldest attempt first, and include the exact Copy-for-AI text a user would see in Squally.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three sentences, all earning their place: output detail, use case, and disambiguation hint. It is front-loaded with the most decision-relevant information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With a full input schema, output schema, and strong annotations, this is complete. The description covers what the tool returns, the ordering, and the ambiguity handling; nothing needed to call it successfully is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents all four parameters. The description reinforces the ambiguous testName/filePath relationship, but doesn't add syntax or format details beyond what the schema provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a concrete, specific contract: every attempt of one test in one run, oldest first, with error, stack, and Copy-for-AI prompt. The phrase "This is the tool for 'why did this test fail'" nails the purpose and differentiates it from run-level or status siblings.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It clearly states when to use it: when the agent needs to answer why a test failed. It adds a disambiguation rule (pass filePath if the test name is ambiguous), but doesn't explicitly name alternatives or conditions when another sibling would be more appropriate.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Finds CI runs of a project - the latest, or filtered by branch, commit SHA or status. One row per logical run: a sharded run is collapsed into a single entry. Returns summary counters only; use squally-get-run for the per-test rows.
| Name | Required | Description | Default |
|---|---|---|---|
| sha | No | A commit SHA or a prefix of one (4-40 hex characters). Matches the run's commit or its tested revision (a pull request's head behind a merge commit). | |
| days | No | The period, in days, ending now. Must not exceed the plan's retention (Standard 60, Pro 90 days); a longer period is refused with 400 invalid_period rather than answered with less data. | |
| branch | No | Exact branch name. | |
| cursor | No | Opaque. Pass nextCursor for older runs, or prevCursor together with direction=prev for newer runs. | |
| status | No | Only runs with this outcome. | |
| perPage | No | Rows per page. | |
| direction | No | Which way the cursor pages. | next |
| projectId | Yes | A project id from GET /projects. |
| Name | Required | Description |
|---|---|---|
| runs | Yes | |
| total | Yes | |
| window | Yes | |
| project | Yes | |
| nextCursor | Yes | Pass as ?cursor= for the next page of older runs; null on the last page. |
| prevCursor | Yes | Pass as ?cursor= with direction=prev for newer runs; null on the first page. |
| stableBranch | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already establish read-only, idempotent, non-destructive behavior. The description adds useful behavioral context beyond that: sharded runs are collapsed into one row and only summary counters are returned, which is important for interpreting results correctly.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three sentences, each earning its place: the core purpose, the logical-run behavior, and the tool-selection pointer. It is front-loaded and has no filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With annotations covering safety and an output schema explaining return shape, the description covers the remaining non-obvious context: run collapsing, summary-only results, and when to escalate to squally-get-run. Nothing needed for correct invocation is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with detailed parameter descriptions, so the baseline is 3. The description adds value by defining the default query as 'the latest' when no filters are supplied, and by naming the three filter dimensions (branch, SHA, status) that map to schema parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific action ('Finds CI runs of a project') and immediately clarifies scope: latest, or filtered by branch, SHA, or status. It also distinguishes itself from squally-get-run by noting it returns summary counters only, so an agent can tell which tool fits.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly tells the agent to use squally-get-run when per-test rows are needed, and clarifies that this tool returns only summary counters. It does not enumerate exclusions for every sibling, but the one meaningful routing decision is covered.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
One run with its per-test result rows, across every shard. Use this to find out which test is red in a run; the run list carries only the counters.
| Name | Required | Description | Default |
|---|---|---|---|
| runId | Yes | A run id from GET /projects/{projectId}/runs - or any shard's id of a sharded run. | |
| projectId | Yes | A project id from GET /projects. |
| Name | Required | Description |
|---|---|---|
| run | Yes | |
| tests | Yes | |
| counts | Yes | |
| project | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint, destructiveHint, idempotentHint, so the safety profile is known. The description adds that results span every shard and includes per-test rows, which is behavioral context beyond annotations. No contradiction. It doesn't discuss errors or edge cases, but with annotations and output schema, this is sufficient.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, front-loaded with the core function, then a usage tip. No wasted words, clear and concise.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple read operation with fully described schema and annotations, the description covers the purpose, the key differentiator (shards, per-test rows), and the use case. Output schema handles return values, so nothing critical is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with detailed descriptions for both parameters. The tool description adds no additional parameter details, so it relies on the schema. Baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool returns a single run with per-test result rows across all shards, and distinguishes it from the run list which only carries counters. It names a specific use case (find which test is red), making it distinct from siblings.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly tells when to use this tool: to find out which test is red, because the run list only carries counters. It implies this is the go-to for per-test details, but does not mention when to use alternatives like find-run or get-test-status, so it's not fully comprehensive.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
The stored flakiness status and active signals for ONE test. Cheap - one findUnique. Use this when you are asking about a single test; do not reach for squally-list-flaky-tests, which is expensive.
| Name | Required | Description | Default |
|---|---|---|---|
| filePath | No | The spec file of the test, as stored (e.g. tests/checkout.spec.ts); an empty value means a test with no file. Pass it exactly as shown in Squally; do not URL-encode, the server does. Needed only when the name exists in several files - the ambiguous_test error lists them. | |
| testName | Yes | The full test name as Squally stores it (the Playwright title path), for example "checkout flow > completes payment". Pass the name exactly as shown in Squally; do not URL-encode, the server does. A name used in several spec files also needs filePath. | |
| projectId | Yes | A project id from GET /projects. |
| Name | Required | Description |
|---|---|---|
| test | Yes | |
| labels | Yes | |
| status | Yes | |
| project | Yes | |
| activeSignals | Yes | |
| awaitingFirstWrite | Yes | True when the test is known but no verdict has been stored for it yet; status then reads insufficient_data. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare the operation read-only, idempotent, and non-destructive. The description adds useful behavioral context beyond that by disclosing cost and implementation footprint: 'Cheap - one findUnique.' It also clarifies that the returned status is the stored one, which helps set expectations about freshness.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences with no filler. The core purpose and scope are front-loaded, and the alternative-avoidance guidance is compactly included in the second sentence.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description is complete for a read-only single-record lookup. It states what is returned, how cheap it is, and when to use it. The output schema exists and covers return shape, so the description does not need to explain return values in detail.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the parameters are already fully documented in the schema. The description reinforces the single-test scope but does not add significant new meaning about how to fill projectId, testName, or filePath beyond what the schema already provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states exactly what the tool does: it returns 'the stored flakiness status and active signals for ONE test.' It uses a specific verb and resource, and explicitly contrasts itself with squally-list-flaky-tests, making the purpose easy to distinguish from siblings.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives direct when-to-use guidance: 'Use this when you are asking about a single test.' It also names the alternative to avoid and why: 'do not reach for squally-list-flaky-tests, which is expensive.' This is explicit and actionable.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
The project's error signatures in a period: failures grouped by fingerprint, with how often and how recently each occurred. Answers 'is this failure one instance of something that keeps happening' - a grouping no agent can compute from the repository alone.
| Name | Required | Description | Default |
|---|---|---|---|
| days | No | The period, in days, ending now. Must not exceed the plan's retention (Standard 60, Pro 90 days); a longer period is refused with 400 invalid_period rather than answered with less data. | |
| page | No | 1-based page number. A page past the end answers the last page. | |
| perPage | No | Rows per page. | |
| projectId | Yes | A project id from GET /projects. |
| Name | Required | Description |
|---|---|---|
| page | Yes | |
| errors | Yes | |
| window | Yes | |
| project | Yes | |
| unfingerprintedCount | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already cover read-only, idempotent, non-destructive behavior. The description adds useful behavioral details beyond annotations: the 400 invalid_period refusal for periods exceeding plan retention, and the pagination behavior (page past end returns last page). This goes beyond the schema and helps the agent anticipate edge cases.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences with no filler. The first sentence states the function and output, the second explains the purpose. Information is front-loaded and every word earns its place. Excellent structure.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has 4 parameters, full schema coverage, annotations covering safety, and an output schema, the description is complete enough. It explains the grouping concept, the period constraint, and the error behavior. The only minor gap is that it does not describe the return format, but the output schema covers that. A 4 reflects strong completeness without being perfect.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so all parameters are already well-documented. The description does not add significant parameter-level meaning beyond what the schema provides; it mentions 'period' conceptually but does not elaborate on parameter syntax or constraints. Baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb ('list') and resource ('error signatures') and explains the grouping by fingerprint with frequency and recency. It also gives the core question it answers, which clearly distinguishes it from siblings like list-flaky-tests or get-run. No ambiguity about what this tool does.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides a clear context for when to use it: to determine if a failure is a recurring instance. It implies a use case but does not explicitly name alternatives or state when not to use it. Still, the guidance is concrete and not misleading, so it earns a 4.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
The project's flaky and broken tests, ranked, with the time each has cost (timeLostMs). Expensive - one engine pass over the project's recent runs. For a single test use squally-get-test-status instead.
| Name | Required | Description | Default |
|---|---|---|---|
| page | No | 1-based page number. A page past the end answers the last page. | |
| sort | No | Ranking: time lost (default), flake rate, or impact on the stable branch. | timeLost |
| search | No | Case-insensitive substring of the test name or its file. | |
| status | No | Only tests with this verdict. | |
| perPage | No | Rows per page. | |
| projectId | Yes | A project id from GET /projects. |
| Name | Required | Description |
|---|---|---|
| page | Yes | |
| sort | Yes | |
| tests | Yes | |
| project | Yes | |
| windowDays | Yes | The engine's fixed window in days. The flaky list takes no other period. |
| gatheringDataCount | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint, idempotentHint, and destructiveHint, so the safety profile is covered. The description adds a non-obvious behavioral trait: 'Expensive - one engine pass over the project's recent runs', which is valuable cost/performance context beyond the annotations. It also mentions the output field timeLostMs, but that's partially covered by the output schema. No contradiction.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences with zero fluff. The primary purpose is front-loaded, followed by a cost warning and an alternative tool reference. Every sentence earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with 6 parameters, an output schema, and rich annotations, the description is complete. It covers purpose, cost, and routing to the right sibling. Pagination, filtering, and sorting are documented in the schema; return format is in the output schema. Nothing an agent needs to call it correctly is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so all parameters are already documented with descriptions. The description adds minimal parameter-specific meaning—it mentions timeLostMs as an output field, but doesn't clarify any parameter semantics beyond the schema. Baseline 3 is appropriate since the schema carries the load.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb ('list') and resource ('the project's flaky and broken tests'), and clarifies it is ranked with time cost. It explicitly distinguishes from the sibling squally-get-test-status by saying 'For a single test use squally-get-test-status instead', so an agent can tell them apart without opening schemas.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives explicit context: this is for listing flaky/broken tests across a project, and warns it is expensive. It names the alternative for single-test queries and implies when not to use it. Clear exclusions and context are provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Lists the projects in your Squally organization. Start here: every other tool needs a projectId, and this is the only tool that produces one.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
| Name | Required | Description |
|---|---|---|
| projects | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly/idempotent/non-destructive, so the bar is lower. The description adds the organizational scope and its unique role as the sole projectId source, which is valuable behavioral context beyond the annotations. No contradictions.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, zero filler. The action is front-loaded, and the crucial 'start here' guidance is placed immediately after the verb phrase, making the purpose instantly actionable.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 0-parameter query tool with a safety profile covered by annotations and an output schema present, the description fully covers what an agent needs: what it does, why it matters, and when to call it. Nothing essential is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters and 100% schema description coverage, so the baseline is 4. The description adds no parameter detail, but none is needed—there is nothing to document.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Lists the projects in your Squally organization') and differentiates from siblings by noting it is the only tool that produces a projectId. An agent can immediately tell this from the run/error-focused sibling tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly instructs 'Start here' and justifies why: every other tool needs a projectId and only this tool produces one. This gives clear when-to-use guidance and implicitly rules out alternatives until a projectId is obtained.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.1.0squally-debug-failuresqually-find-runsqually-get-runsqually-get-test-statussqually-list-errorssqually-list-flaky-testssqually-list-projectsScored across 7 tools
Each tool targets a distinct resource and action: project discovery, run search, run detail, failure debugging, single-test status, flaky-test listing, and error-signature aggregation. The descriptions explicitly cross-reference when one tool should be preferred over another, so there is no real ambiguity.
All tools share a consistent 'squally-' prefix and lowercase hyphenated action-object pattern such as list-projects, find-run, get-run, and debug-failure. The verbs are semantically meaningful and consistently used throughout.
Seven tools is a well-scoped size for a CI/test debugging server. Each tool covers a distinct step in the investigation workflow without redundant or bloated additions.
The surface covers the full read-only debugging journey: find a project, locate runs, inspect individual run results, drill into a failing test's attempts, check single-test flakiness, list flaky tests, and aggregate error signatures. There are no obvious dead ends or missing operations for the stated purpose.
Direct access to Cypress tests results and accessibility reports in your AI workflow.
BuildPulse CI test analytics for AI agents — flaky tests, CI failures, flakiness, and code coverage.
Change-aware CI validation and affected-test guidance for coding agents.
Change-aware CI validation and affected-test guidance for coding agents.
