Vision MCP

Instructions

Implementation Reference

• vision_mcp/server.py:34-84 (handler)

  MCP tool handler for 'analyze_image': decorated with @mcp.tool(), processes image_source to data URL, calls OpenAI client for analysis, handles errors, returns TextContent.
  @mcp.tool( description=""" A powerful LLM that can analyze and understand image content from files or URLs, follow your instruction. Use this tool to analyze images by LLM. Only support jpeg, png, webp formats. Other formats like pdf/gif/psd/svg and so on are not supported. Args: prompt (str): The text prompt describing what you want to analyze or extract from the image. image_source (str): The source location of the image to analyze. Accepts: - HTTP/HTTPS URL: "https://example.com/image.jpg" - Local file path: - Relative path: "images/photo.png" - Absolute path: "/Users/username/Documents/image.jpg" IMPORTANT: If the file path starts with @ symbol, you MUST remove the @ prefix before passing to this function. For example: - If you see "@Documents/photo.jpg", use "Documents/photo.jpg" - If you see "@/Users/username/image.png", use "/Users/username/image.png" Supported formats: JPEG, PNG, WebP Returns: Text content with the image analysis result. """ ) def analyze_image( prompt: str, image_source: str, ) -> TextContent: try: if not prompt: raise VisionRequestError("Prompt is required") if not image_source: raise VisionRequestError("Image source is required") processed_image_url = process_image_url(image_source) content = openai_client.analyze_image(prompt, processed_image_url) if not content: raise VisionRequestError("No content returned from VLM API") return TextContent( type="text", text=content ) except VisionAPIError as e: return TextContent( type="text", text=f"Failed to analyze image: {str(e)}" )
• vision_mcp/utils.py:7-74 (helper)

  Helper function to process image_source (URL, file path, data URL) into base64 data URL format used by the handler.
  def process_image_url(image_url: str) -> str: """ Process image URL and convert to base64 data URL format. This function handles three types of image inputs: 1. HTTP/HTTPS URLs: Downloads the image and converts to base64 2. Base64 data URLs: Passes through as-is 3. Local file paths: Reads the file and converts to base64 Args: image_url (str): The image URL, data URL, or local file path Returns: str: Base64 data URL in format "data:image/{format};base64,{data}" Raises: VisionRequestError: If image cannot be downloaded, read, or processed """ if image_url.startswith("@"): image_url = image_url[1:] if image_url.startswith("data:"): return image_url if image_url.startswith(("http://", "https://")): try: image_response = requests.get(image_url) image_response.raise_for_status() image_data = image_response.content content_type = image_response.headers.get('content-type', '').lower() if 'jpeg' in content_type or 'jpg' in content_type: image_format = 'jpeg' elif 'png' in content_type: image_format = 'png' elif 'webp' in content_type: image_format = 'webp' else: image_format = 'jpeg' base64_data = base64.b64encode(image_data).decode('utf-8') return f"data:image/{image_format};base64,{base64_data}" except requests.RequestException as e: raise VisionRequestError(f"Failed to download image from URL: {str(e)}") else: if not os.path.exists(image_url): raise VisionRequestError(f"Local image file does not exist: {image_url}") try: with open(image_url, "rb") as f: image_data = f.read() image_format = 'jpeg' if image_url.lower().endswith('.png'): image_format = 'png' elif image_url.lower().endswith('.webp'): image_format = 'webp' elif image_url.lower().endswith(('.jpg', '.jpeg')): image_format = 'jpeg' base64_data = base64.b64encode(image_data).decode('utf-8') return f"data:image/{image_format};base64,{base64_data}" except IOError as e: raise VisionRequestError(f"Failed to read local image file: {str(e)}")
• vision_mcp/client.py:27-61 (helper)

  Helper method in OpenAICompatibleClient that performs the actual API call to /v1/chat/completions with image_url content, called by the handler.
  def analyze_image(self, prompt: str, image_data_url: str) -> str: """Analyze an image using the Vision API. Args: prompt: The text prompt describing what to analyze image_data_url: The image as a data URL (data:image/...;base64,...) Returns: The model's response text Raises: VisionRequestError: If the request fails """ url = f"{self.api_base}/v1/chat/completions" payload = { "model": self.model, "messages": [{ "role": "user", "content": [ {"type": "text", "text": prompt}, {"type": "image_url", "image_url": {"url": image_data_url}} ] }] } try: response = self.session.post(url, json=payload) response.raise_for_status() data = response.json() return data["choices"][0]["message"]["content"] except requests.exceptions.RequestException as e: raise VisionRequestError(f"Request failed: {str(e)}") except (KeyError, IndexError) as e: raise VisionRequestError(f"Invalid response format: {str(e)}")