act
Control virtual Ubuntu desktops to automate web browsing, run code, and execute bash commands through mouse/keyboard actions for data extraction and task automation.
Instructions
Take action on a Scrapybara instance through an agent. The agent can control the instance with mouse/keyboard and bash commands.
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| instance_id | Yes | The ID of the instance to act on. | |
| prompt | Yes | The prompt to act on. <EXAMPLES> - Go to https://ycombinator.com/companies, set batch filter to W25, and extract all company names. - Find the best way to contact Scrapybara. - Order a Big Mac from McDonald's on Doordash. </EXAMPLES> | |
| schema | No | Optional schema if you want to extract structured output. |
Implementation Reference
- src/index.ts:206-242 (handler)Handler for the 'act' tool: parses input with ActSchema, retrieves the Scrapybara instance, sets up tools (computer, bash, edit), calls client.act with model, system prompt, prompt, and schema, then returns the response.
case "act": { const args = ActSchema.parse(request.params.arguments); const instance = await client.get(args.instance_id, { abortSignal: currentController.signal, }); const tools: Scrapybara.Tool[] = [computerTool(instance)]; if (instance instanceof UbuntuInstance) { tools.push(bashTool(instance)); tools.push(editTool(instance)); } const actResponse = await client.act({ model: actModel, tools, system: actSystem, prompt: args.prompt, schema: args.schema, requestOptions: { abortSignal: currentController.signal, }, }); return { content: [ { type: "text", text: JSON.stringify( { text: actResponse.text, output: actResponse.output }, null, 2 ), } as TextContent, ], }; } - src/schemas.ts:25-38 (schema)Zod schema for 'act' tool inputs: instance_id (string), prompt (string with examples), schema (optional any).
export const ActSchema = z.object({ instance_id: z.string().describe("The ID of the instance to act on."), prompt: z.string().describe(`The prompt to act on. <EXAMPLES> - Go to https://ycombinator.com/companies, set batch filter to W25, and extract all company names. - Find the best way to contact Scrapybara. - Order a Big Mac from McDonald's on Doordash. </EXAMPLES> `), schema: z .any() .optional() .describe("Optional schema if you want to extract structured output."), }); - src/index.ts:93-98 (registration)Tool registration for 'act' in the ListToolsRequestHandler response, including name, description, and inputSchema from ActSchema.
{ name: "act", description: "Take action on a Scrapybara instance through an agent. The agent can control the instance with mouse/keyboard and bash commands.", inputSchema: zodToJsonSchema(ActSchema), }, - src/index.ts:33-45 (helper)Helper variables configuring the model and system prompt for the 'act' tool based on ACT_MODEL environment variable.
let actModel = process.env.ACT_MODEL === "anthropic" ? anthropic() : process.env.ACT_MODEL === "openai" ? openai() : anthropic(); // Default to Anthropic let actSystem = process.env.ACT_MODEL === "anthropic" ? ANTHROPIC_UBUNTU_SYSTEM_PROMPT : process.env.ACT_MODEL === "openai" ? OPENAI_UBUNTU_SYSTEM_PROMPT : ANTHROPIC_UBUNTU_SYSTEM_PROMPT; // Default to Anthropic's prompt