Skip to main content
Glama
jgt87

Copilot Studio MCP

by jgt87
OverviewSchemaSourceRelated ServersDiscussionsAdmin
TypeScript
Local

Run an evaluation

cs_run_evaluation

Start an evaluation run for a Copilot Studio test set, target the draft or published agent, and optionally wait for the summary. Requires confirm to execute.

Instructions

Start an evaluation run for a test set (draft agent by default, or the published one). Optionally wait for completion and return the summary. mcsConnectionId runs it signed in as a user profile (the id of that user's Copilot Studio connection) so authenticated knowledge and tools answer; toolsConnections picks the connection each connector tool uses. Counts against the 20 runs per agent per 24h limit: requires confirm: true.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
waitNo
botIdNoAgent (bot) id. Defaults to workspace sync metadata or CPS_AGENT_ID.
confirmNoRequired to actually perform a change in a live environment. Without it the tool returns a dry run.
runNameNo
clientIdNoEntra app (client) id for MSAL. Defaults to CPS_CLIENT_ID, then the first-party VS Code id.
tenantIdNoEntra tenant id. Defaults to the workspace sync metadata, then CPS_TENANT_ID.
testSetIdYes
workspaceNoPath to (or inside) the agent workspace. Defaults to CPS_WORKSPACE or the current directory.
environmentIdNoPower Platform environment id (GUID). Defaults to workspace sync metadata or CPS_ENVIRONMENT_ID.
timeoutSecondsNo
mcsConnectionIdNoCopilot Studio connection id of the user profile to run as; omit for an anonymous run. Power Automate > Connections > Microsoft Copilot Studio, the GUID in the URL
toolsConnectionsNoWhich connection each connector tool should use during the run
runOnPublishedBotNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed3 schema fields changedv0.1.8
    • removedInput schema / additionalProperties
      Removed value: -false
    • changedInput schema / properties / mcsConnectionId / description
      Previous value: -"User profile connection id for authenticated knowledge/tools"New value: +"Copilot Studio connection id of the user profile to run as; omit for an anonymous run. Power Automate > Connections > Microsoft Copilot Studio, the GUID in the URL"
    • addedInput schema / properties / toolsConnections
      Added value: +{
+  "description": "Which connection each connector tool should use during the run",
+  "items": {
+    "properties": {
+      "botId": {
+        "description": "Dataverse bot id; default: the evaluated agent",
+        "type": "string"
+      },
+      "botSchemaName": {
+        "type": "string"
+      },
+      "connections": {
+        "items": {
+          "properties": {
+            "connectionId": {
+              "description": "The user connection to run it with (cs_list_connections)",
+              "type": "string"
+            },
+            "connectionReferenceName": {
+              "description": "Connection reference name in the agent, when the tool is bound through one",
+              "type": "string"
+            },
+            "connectorId": {
+              "description": "Connector id, e.g. shared_office365 (cs_list_connectors)",
+              "type": "string"
+            }
+          },
+          "required": [
+            "connectorId",
+            "connectionId"
+          ],
+          "type": "object"
+        },
+        "type": "array"
+      }
+    },
+    "required": [
+      "connections"
+    ],
+    "type": "object"
+  },
+  "type": "array"
+}
  2. First observedv0.1.5

TDQS

A3.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are supplied, so the description carries the full burden and does disclose important behavior: draft-vs-published default, optional wait/summary, authenticated run via mcsConnectionId, connector connection selection, the 20-runs/24h quota, and the confirm:true requirement. It omits failure/timeout behavior and what is returned when wait is false.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

One front-loaded paragraph with no filler; the primary action and default target come first, followed by optional behavior and constraints. The auth sentence is dense but each clause carries operational information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 13-parameter tool with no annotations and no output schema, the description covers the main call contract, authentication, quota, and confirm gate. It leaves the non-wait return value and timeout semantics implicit, which an agent may need clarified.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 62%, so the description needs to add value and does: it explains wait's effect, the published-bot switch, the confirm requirement, and the rationale for mcsConnectionId. Several parameters (runName, timeoutSeconds) remain undocumented beyond their names, keeping it from a 5.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Start an evaluation run for a test set') and clarifies the default target (draft agent) versus the published one. It does not explicitly name sibling alternatives like cs_get_evaluation_run or cs_run_conversation_tests, so it stops short of full sibling differentiation.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides operating context (draft default, optional wait, confirm requirement, run quota) but never says when to choose this tool over cs_get_evaluation_run, cs_list_evaluation_runs, or cs_run_conversation_tests. Usage is implied by the name and defaults rather than explicitly routed.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools