Multimodal MCP - Week 6 Lab

Overview

This project demonstrates the integration of multimodal artificial intelligence techniques using CLIP embeddings, Visual Question Answering (VQA), and the Model Context Protocol (MCP).

The objective is to explore how images and text can be represented in a shared embedding space, enabling semantic search and natural language interaction with visual content.

Related MCP server: Vision Toolkit

Project Structure

.
├── 6.lab.1-multimodal-mcp.ipynb
├── client.py
├── image_embeddings.npy
├── README.md
├── requirements.txt
└── server.py

Components

MCP Server

The server exposes multimodal capabilities through MCP tools.

Available tools:

• search_images(query)

  • Performs semantic image retrieval using CLIP embeddings.

• ask_about_image(image_id, question)

  • Uses a Visual Question Answering (VQA) pipeline to answer questions about a selected image.

MCP Client

The client connects to the MCP server and demonstrates:

1. Tool discovery

2. Semantic image search

3. Visual question answering

4. End-to-end multimodal interaction

Example Workflow

User Query
      ↓
search_images("dog running")
      ↓
Most Relevant Image
      ↓
ask_about_image(image_id, question)
      ↓
Textual Answer

Technologies

• Python

• MCP (Model Context Protocol)

• Hugging Face Transformers

• OpenAI CLIP

• BLIP / VQA Models

• PyTorch

Laboratory Objectives

• Explore multimodal embeddings using CLIP.

• Evaluate zero-shot image classification.

• Build a Visual Question Answering system.

• Expose multimodal capabilities through MCP.

• Demonstrate interaction between retrieval and reasoning components.

Author

Erick Lima

Jala University

Software Engineering