Defect Vision MCP Agent

Real-time steel surface defect classification system combining a fine-tuned PyTorch computer vision model, a genuine MCP (Model Context Protocol) server, and a LangGraph multi-agent monitoring layer with human-in-the-loop guardrails.

Live Demo: https://defect-vision-mcp-agent-j3eq5p8huviz4hbz495imi.streamlit.app/

Built to close three specific gaps: hands-on PyTorch model training, real MCP protocol integration, and MLOps-style model monitoring — all wired together as one coherent system rather than three disconnected pieces.

Results

Fine-tuned a ResNet18-based classifier (transfer learning, frozen backbone, trained classification head) on the NEU-DET surface defect dataset — 6 classes, 1,800 images (1,440 train / 360 validation, balanced).

Per-class performance:

Related MCP server: machine-maintenance-agent

Architecture

1. PyTorch Model (model/) — ResNet18-based transfer-learning classifier. Backbone frozen; only the classification head is fine-tuned, converging in 6 epochs on CPU.

2. MCP Server (mcp_server/) — a real, protocol-compliant MCP server (built on MCPServer from the official mcp SDK, verified against the MCP Inspector) exposing two tools:

   • classify_image(image_path) — runs inference, logs the prediction

   • get_model_metrics() — returns aggregate stats from the prediction log

3. Multi-Agent Layer (agents/) — LangGraph-orchestrated pipeline with three nodes in sequence:

   • Inference agent — calls the MCP tool to classify an image

   • Guardrail agent — auto-accepts high-confidence predictions; low-confidence predictions are flagged for human review instead of being silently accepted, mirroring a human-in-the-loop pattern rather than blind automation

   • Drift-monitor agent — tracks the rolling ratio of low-confidence predictions and flags DRIFT_DETECTED if it crosses a threshold, signaling the model may need review or retraining

4. Dashboard (dashboard/) — Streamlit UI showing live prediction counts, average confidence, low-confidence count, a full prediction log, and a confidence-over-time chart.

Setup

pip install -r requirements.txt
cp .env.example .env   # fill in your own GROQ_API_KEY

Train the model

Requires the NEU-DET dataset arranged as:

data/processed/
├── train/<class_name>/*.jpg
└── val/<class_name>/*.jpg

python model/train.py
python model/evaluate.py

Run the MCP server

python mcp_server/server.py

Test it interactively with the MCP Inspector:

npx @modelcontextprotocol/inspector python mcp_server/server.py

Run the agent pipeline

python agents/graph.py

Run the dashboard

streamlit run dashboard/app.py

Run tests

pytest tests/ -v

Tech Stack

Python · PyTorch · torchvision · MCP SDK · LangGraph · LangChain · Streamlit · scikit-learn · GitHub Actions (CI)

Notes

• data/processed/, model/checkpoints/*.pt, and the live prediction log (logs/predictions.jsonl) are gitignored — not included in this repo. logs/predictions_sample.jsonl is committed separately so the deployed dashboard has sample data to display.

• Deployed on Streamlit Community Cloud (free tier) — expect a brief cold start if the app has been idle.