zero_shot_image_classification
Classify images using custom labels without prior training. Upload an image and specify categories to identify its content with AI-powered visual recognition.
Instructions
Classify an image with zero-shot labels using DeepInfra OpenAI-compatible API (CLIP).
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| image_url | Yes | ||
| candidate_labels | Yes |
Output Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| result | Yes |
Implementation Reference
- src/mcp_deepinfra/server.py:121-150 (handler)The handler function that executes the zero_shot_image_classification tool. It constructs a vision-enabled chat completion request to classify the image from the given candidate labels using the configured model.
async def zero_shot_image_classification(image_url: str, candidate_labels: list[str]) -> str: """Classify an image with zero-shot labels using DeepInfra OpenAI-compatible API (CLIP).""" model = DEFAULT_MODELS["zero_shot_image_classification"] try: # Use chat/completions with vision capability to get classification response = await client.chat.completions.create( model=model, messages=[ { "role": "user", "content": [ { "type": "text", "text": f"Classify this image into one of these categories: {', '.join(candidate_labels)}. Return a JSON with 'label' and 'score' fields." }, { "type": "image_url", "image_url": {"url": image_url} } ] } ], max_tokens=200, ) if response.choices: return response.choices[0].message.content else: return "Unable to classify image" except Exception as e: return f"Error classifying image: {type(e).__name__}: {str(e)}" - src/mcp_deepinfra/server.py:119-120 (registration)Conditional registration of the zero_shot_image_classification tool using the FastMCP @app.tool() decorator.
if "all" in ENABLED_TOOLS or "zero_shot_image_classification" in ENABLED_TOOLS: @app.tool() - src/mcp_deepinfra/server.py:121-122 (schema)Function signature providing input schema (image_url: str, candidate_labels: list[str]) and output str, along with docstring description used for tool schema in MCP.
async def zero_shot_image_classification(image_url: str, candidate_labels: list[str]) -> str: """Classify an image with zero-shot labels using DeepInfra OpenAI-compatible API (CLIP).""" - src/mcp_deepinfra/server.py:36-36 (helper)Helper configuration defining the default model for the zero_shot_image_classification tool.
"zero_shot_image_classification": os.getenv("MODEL_ZERO_SHOT_IMAGE_CLASSIFICATION", "openai/gpt-4o-mini"),