Visual QA MCP

Visual QA MCP is an early project workspace for building tools that help AI agents verify educational, scientific, medical-education, and engineering visuals before they are used in serious instructional or technical contexts.

The core idea is simple: AI-generated images should be checked like code, and their visual claims should be checked against theory, references, and extracted evidence. A generated diagram, chart, mechanical illustration, anatomy teaching image, or technical visual should have a machine-readable spec, automated checks, visible error evidence, and a human review path for high-risk domains.

Project Goal

Create a toolchain that helps agents:

• Turn a lesson objective into a structured visual spec.

• Extract text, shapes, arrows, chart data, objects, and geometry from an image.

• Run domain-specific checks against the spec.

• Produce grounded findings with coordinates and evidence.

• Generate an annotated overlay and repair guidance.

Related MCP server: agent-vision-mcp

Initial Scope And Long-Term Direction

The first practical scope is educational media where correctness can be measured without model fine-tuning:

• Physics diagrams: force arrows, torque, free-body diagrams, light rays, circuits.

• Charts and infographics: axes, labels, scales, bar heights, pie totals, trend/data consistency.

• Mechanical illustrations: holes, callouts, arrows, geometry, missing or extra parts.

The long-term direction is broader and stricter: medical education, open-ended anatomy, complex chemistry and biology, and full CAD reconstruction are target high-assurance tracks. They require stronger references, theory-aligned rule modules, validation datasets, and expert review before the project can claim readiness for those domains.

See docs/high-assurance-roadmap.md for the roadmap toward these harder domains.

Workspace Layout

docs/
  problem-map.md
  product-brief.md
  mvp-scope.md
  validation-plan.md

specs/
  visual-spec.schema.json
  findings.schema.json
  examples/

skills/
  educational-visual-qa/
    SKILL.md

mcp-server/
  README.md
  tools.md

datasets/
  README.md

experiments/
  README.md

Design Principle

Do not ask a vision model, "Is this correct?" as the only check.

Instead, ask tools to extract evidence, then run checks grounded in specs, theory, source references, and tolerances:

image -> evidence graph -> claim graph -> domain rules -> findings + overlay

The agent can still use a vision-language model, but only as one part of an evidence-backed QA loop.

Current Executable MVP

The current runtime has six bounded executable verticals:

• chart-v2: template-backed bar charts with image-read Y-axis scale evidence

• arrow-v1: controlled free-body diagrams with arrow identity, direction, anchor, and opt-in translational force balance

• geometry-v1: controlled mechanical plates with circular-hole count, relative diameter, linear alignment/spacing, and fixed-catalog dimension labels

• coordinate-graph-v1: controlled dual-axis scatter/polyline diagrams

• flowchart-v1: controlled rectangle/diamond flowcharts with directed topology

• circuit-v1: a two-gate controlled structural-netlist verifier (v1a series loops; v1b explicit-junction simple-parallel and bounded series-parallel branches)

The first five can also project into an additive PrimitiveEvidenceGraph audit layer containing basic shapes, arrows, text regions, spatial relationships, provenance, and links back to domain evidence. Domain rules still consume the established domain graphs.

Implemented pieces:

• EvidenceGraph schema with tick detections, axis mapping, and bar geometry.

• ClaimGraph schema and chart-v2 claim generator so rule execution consumes explicit claims instead of ad hoc spec parsing.

• claim-generation gaps and claim_graph.json audit artifacts so unsupported checks degrade to needs_review instead of disappearing silently.

• Local callable Python tool surface in mcp-server/src/visual_qa_mcp/ for claim generation, evidence extraction, verification, and artifact writing.

• MCP server wrapper over chart, arrow, and geometry claim/extraction/verification surfaces.

• Spec-blind parse_primitives MCP and extract-primitives CLI surfaces for five bounded profiles; circuit evidence remains on its typed domain graph.

• Audit-oriented provenance and confidence separation:

  • extractor provenance in EvidenceGraph

  • stable rule_id values in claims and findings

  • separate extraction versus rule confidence in VisualQaReport

• Dual tick-reader path:

  • default template backend

  • optional OCR backend scaffold

• Validation dataset with 24 cases:

  • 8 golden

  • 16 mutated

• Separate noisy chart-v2 validation dataset for Phase 2 evidence expansion.

• A separate 24-case chart-v2-realworld-pilot track with Pillow/Matplotlib renderer diversity, World Bank reference-backed source snapshots, provenance/license metadata, and frozen checksums.

• Generic chart source records using category / value, while retaining compatibility with the original month / rainfall_mm controlled fixtures.

• Overlay generation for flagged findings.

• Verification tests, validation summary artifacts, and advisor-gate evidence packs.

Geometry-v1 has a 14-case controlled Pillow-rendered family (7/7 typed, 2/2 ambiguity) and a separate checksum-frozen 20-case noisy family (5/5 typed, 5/5 ambiguity, 10/10 golden). This does not cover arbitrary mechanical drawings, independently authored images, general OCR, calibrated units, or native CAD.

Circuit-v1a has 11 controlled cases (4/4 typed, 5/5 ambiguity, 2/2 golden, terminal-netlist evidence 6/6). Circuit-v1b has 14 controlled cases (7/7 typed, 3/3 ambiguity, 4/4 golden, terminal-netlist and junction-count evidence 11/11). These claims cover only controlled colored Pillow symbols, orthogonal non-crossing wires, explicit junction dots, simple parallel, and one bounded series-parallel family. They do not cover arbitrary schematics, crossings, OCR, electrical values/laws, or engineering certification.

The bounded readiness claim remains narrow: the validated default is the template backend on the controlled chart-v2 family and the configured noisy transform family. The real-world pilot is an evidence-expansion track, not proof of general real-world chart readiness. Its public-reference cases are locally rendered from a frozen World Bank data snapshot; they do not establish robustness across arbitrary publishers, fonts, palettes, or chart images. OCR remains a separate unvalidated backend.

See docs/chart-mvp-workflow.md for the operational workflow and advisor gates.