analyze_screenshot
Capture a screenshot of a web page and use AI to analyze and describe its visible content, with options for full-page capture, specific instructions, and detailed structural analysis.
Instructions
Take a screenshot and analyze it with AI (Gemma3) to describe what is visible on the page
Tool call template
This tool declares no required parameters. Add optional parameters from the input schema as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {},
"name": "analyze_screenshot"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| fullPage | No | Capture full scrollable page | |
| path | No | Path to save screenshot (optional) | |
| pretext | No | Optional context or specific instructions for what to look for in the analysis | |
| model | No | AI model to use for analysis (default: gemma3:4b) | gemma3:4b |
| detailed | No | Provide detailed structural analysis of the page |
Implementation Reference
- src/index.ts:64-70 (schema)Zod schema defining the input parameters for analyze_screenshot (fullPage, path, pretext, model, detailed)
const AnalyzeScreenshotSchema = z.object({ fullPage: z.boolean().default(false), path: z.string().optional(), pretext: z.string().optional(), model: z.string().default('gemma3:4b'), detailed: z.boolean().default(false) }); - src/index.ts:330-361 (registration)Tool registration in the ListToolsRequestSchema handler, defining the tool's name, description, and input JSON schema
{ name: 'analyze_screenshot', description: 'Take a screenshot and analyze it with AI (Gemma3) to describe what is visible on the page', inputSchema: { type: 'object', properties: { fullPage: { type: 'boolean', default: false, description: 'Capture full scrollable page' }, path: { type: 'string', description: 'Path to save screenshot (optional)' }, pretext: { type: 'string', description: 'Optional context or specific instructions for what to look for in the analysis' }, model: { type: 'string', default: 'gemma3:4b', description: 'AI model to use for analysis (default: gemma3:4b)' }, detailed: { type: 'boolean', default: false, description: 'Provide detailed structural analysis of the page' } } } }, - src/index.ts:681-759 (handler)Main handler logic for the analyze_screenshot tool: takes a screenshot using Playwright, sends it to Ollama (Gemma3) for AI analysis, and returns the result
case 'analyze_screenshot': { if (!currentPage) { throw new Error('No browser page available. Launch a browser first.'); } const params = AnalyzeScreenshotSchema.parse(args); // Take screenshot const screenshotPath = params.path || `screenshot-${Date.now()}.png`; const screenshotBuffer = await currentPage.screenshot({ fullPage: params.fullPage, path: screenshotPath }); try { // Initialize Ollama client const ollama = new Ollama({ host: 'http://localhost:11434' }); // Convert screenshot to base64 const base64Image = screenshotBuffer.toString('base64'); // Prepare the prompt let prompt = 'Analyze this website screenshot and describe exactly what you see. '; if (params.detailed) { prompt += 'Provide a detailed structural analysis including layout, navigation elements, content sections, forms, buttons, and any interactive elements. '; } else { prompt += 'Focus on the main content and key elements visible on the page. '; } if (params.pretext) { prompt += `Additional context/instructions: ${params.pretext}. `; } prompt += 'Be specific about colors, text content, images, and the overall design and functionality of the page.'; // Make AI request const response = await ollama.generate({ model: params.model, prompt: prompt, images: [base64Image], stream: false }); return { content: [ { type: 'text', text: `AI Analysis of Screenshot (${screenshotPath}): ${response.response} Screenshot saved to: ${screenshotPath} Model used: ${params.model} Analysis type: ${params.detailed ? 'Detailed structural analysis' : 'General description'}` } ] }; } catch (aiError) { // If AI analysis fails, still return screenshot info const fallbackMessage = aiError instanceof Error ? aiError.message : String(aiError); return { content: [ { type: 'text', text: `Screenshot taken and saved to: ${screenshotPath} AI Analysis Error: ${fallbackMessage} Note: Make sure Ollama is running locally with the ${params.model} model installed. You can install it with: ollama pull ${params.model} And start Ollama with: ollama serve` } ] }; } }