Skip to main content
Glama

Foreman

Agentic vision AI for warehouse safety review, built on NVIDIA NIM.

Point it at warehouse footage and it returns a short queue of verified safety alerts, each one carrying the evidence it was confirmed on and the standard it maps to, plus a searchable index of everything else it saw. It runs on NVIDIA-hosted NIM endpoints, so it reproduces on a laptop with an API key and no GPU.

Live demo · analysis of a real run, no setup required. Evidence clips are local-only there, for the licensing reason in DATA.md.

Foreman review console: alert queue, evidence chain, search, and audit

The review console: verified alerts with the evidence each was confirmed on, natural language search across every window, and an audit view of everything suppressed.


The problem this is actually solving

Ask a vision language model "is there a safety hazard in this footage" and it will almost always say yes. On the labelled set in this repo, a single-pass VLM flags a hazard in 77% of windows that contain none. That is not a system a safety supervisor will use twice, and no amount of prompt rewriting fixes it, because the prompt is not the problem. Showing a model a safety camera and asking about hazards hands it an overwhelming prior that hazards are present.

Foreman treats that as an architecture problem rather than a wording problem. The perception pass is allowed to over-report, and a second model that can only remove things re-opens the same frames and decides whether the evidence actually meets the bar for the class that was claimed.

Measured on 49 hand-labelled windows:

arm

precision

recall

F1

windows falsely alerting

vlm_only — every candidate becomes an alert

0.15

0.80

0.25

30/39 (77%)

vlm_conf_80 — keep candidates above 0.80 confidence

0.18

0.80

0.30

28/39 (72%)

llm_verifier — reasoning LLM reviews the perception text

0.14

0.30

0.19

12/39 (31%)

vlm_verifier — reasoning VLM re-opens the frames

0.35

0.60

0.44

10/39 (26%)

Over three repeats the verifier arm runs P=0.34 [0.31–0.38], R=0.50 [0.40–0.60], F1=0.40 [0.36–0.46]. Precision roughly doubles; recall pays for it. Full numbers, including a per-class false-positive breakdown, are in evals/RESULTS.md. Reproduce with python evals/run_eval.py.

Three findings worth more than the headline number

1. Confidence thresholds barely work. Going from every candidate to only those the model rated 0.80+ moved precision 0.15 → 0.18. A VLM's self-reported confidence is close to useless as a filter here: it is confident about the things it is wrong about.

2. Text-only verification is worse than no verification. The llm_verifier arm reads the perception pass's written evidence and adjudicates on that. It collapses to F1 0.19 — below the naive baseline. The written evidence is too thin to clear the bar the verifier is applying, so it rejects nearly everything, real hazards included. The verifier has to look at the pixels. This one surprised me and it is the reason the architecture has a VLM in the second position rather than a cheaper LLM.

3. VLMs read on-screen text as though it were the world. The first window of the first video is a title card: black screen, white words, "PASSING IN FRONT OF A FORKLIFT". The perception model reported a pedestrian walking in front of a moving forklift at 0.95 confidence. The verifier confirmed it as high severity. Both models had turned printed words into an observed event.

Real deployments are full of this — burned-in timestamps, camera labels, training overlays. The fix is a scene gate: the perception pass declares whether it is looking at real camera footage at all, in the same call, at no extra cost. It removes an entire class of confident nonsense and is worth +0.07 precision on top of the verifier (vlm_verifier 0.35 vs vlm_verifier_nogate 0.28).

What was filtered

Every suppressed detection stays visible and auditable. A safety tool that cannot show what it threw away is one nobody should trust.


Related MCP server: ewm-ops-mcp

Architecture

flowchart LR
    A[video] --> B[chunker<br/>8s windows, 4 frames]
    B --> C[perception<br/>Nemotron Nano VL 12B]
    C -->|is_camera_footage=false| X[gated: slides,<br/>title cards, presenters]
    C -->|hazard candidates| D[verifier<br/>Nemotron Nano Omni 30B<br/>re-opens the frames]
    D -->|rejected + reason| X
    D -->|confirmed| E[alert queue<br/>+ evidence clip]
    C --> F[embeddings<br/>Llama Nemotron Embed VL]
    F --> G[semantic search<br/>over every window]
    E --> H[MCP server]
    G --> H
    H --> I[Claude Code /<br/>any agent]

stage

model

why this one

perception

nvidia/nemotron-nano-12b-v2-vl

smallest model that reliably holds the JSON contract across every chunk; ~2s per window

verification

nvidia/nemotron-3-nano-omni-30b-a3b-reasoning

reasoning VLM, different family and size from the proposer, so a confirmation is a genuine second look rather than a model agreeing with itself

search

nvidia/llama-nemotron-embed-vl-1b-v2

vision-language retriever, so queries land in the space the perception pass described

On Cosmos. nvidia/cosmos-reason2-8b is the natural perception model here — Cosmos Reason is post-trained for physical-world spatial and temporal reasoning rather than generic captioning, which is exactly the axis a hazard call turns on. It is not provisioned on the free build.nvidia.com tier (account-scoped 404 as of Aug 2026), so this runs on Nemotron VL. VLM_PRIMARY in src/foreman/nim.py is one line; swap it on a self-hosted NIM and the eval harness will report the comparison honestly instead of my asserting it.

Design decisions I would defend in review

8-second windows, 4 sampled frames. Shorter and the model cannot see motion, so a pedestrian standing still and a pedestrian walking into a travel path look identical. Longer and an alert cannot be localised to a moment a reviewer can act on.

The taxonomy is upstream of the model. hazards.py defines five classes from OSHA 29 CFR 1910.178 and the struck-by categories that dominate warehouse injury data — chosen from what an inspector needs, not from what a VLM happens to describe well. The perception prompt, the verifier's reject criteria, the label schema and the UI filters all read from that one definition, so a class cannot drift between the model proposing it and the harness grading it.

The proposer and the adjudicator get different information. The verifier sees each class's known false positive; the perception pass does not. Telling the proposer what not to say suppresses real detections along with phantom ones. Telling the adjudicator what to watch for raises precision without touching recall.

Tools are shaped like questions, not like the call graph. mcp_server.py exposes search_timeline, list_alerts, explain_alert, list_rejected_detections, shift_summary, hazard_taxonomy — the questions a supervisor actually asks. Agent surfaces that mirror internal module boundaries are how they become unusable.


Quickstart

git clone https://github.com/YashNirwan/foreman && cd foreman
python3 -m venv .venv && ./.venv/bin/pip install -e .
cp .env.example .env      # add a free key from https://build.nvidia.com

python scripts/fetch_data.py                          # pull sample footage locally
./.venv/bin/python -m foreman.pipeline data/raw_yt/MqvOjo62BHQ.mp4
./.venv/bin/streamlit run app.py                      # review console

Requires ffmpeg and yt-dlp on PATH (brew install ffmpeg yt-dlp).

A 172-second video: 21 windows, 26 candidates, 11 confirmed alerts, 98 seconds wall clock on a laptop against the free tier, 47 NIM calls.

Drive it from an agent

claude mcp add foreman -- /abs/path/to/foreman/.venv/bin/python -m foreman.mcp_server

Then ask in plain language: "what were the high-severity alerts in that shift, and show me what the verifier threw out."

Reproduce the eval

python evals/run_eval.py --perceive   # perception + all 7 arms
python evals/variance.py --repeats 3  # run-to-run spread on the headline arms

What this is not

Honest limits, because a work sample that oversells is worse than one that is small:

  • The eval is small. 49 windows, 10 positive events, one annotator (me). Class balance is skewed — 7 of 10 positives are pedestrian_in_path, and blocked_egress has no positive examples at all, so its numbers mean nothing yet. Treat the precision figures as a directional result on one labelled set, not a benchmark.

  • Recall drops meaningfully. 0.80 → 0.50. On a real floor you would tune the verifier's bar per class, and you would almost certainly accept lower precision on pedestrian_in_path than on missing_ppe, because the cost of missing them differs.

  • Labels are judgement calls. The guideline is written at the top of ground_truth.json's generator and ambiguous windows are labelled negative. A second annotator would move these numbers.

  • Sampled frames are not video. Four frames across eight seconds miss fast events. The production answer is a CV pipeline (DeepStream, or the tracking stage in the Metropolis VSS blueprint) triggering VLM review on clips it has already localised, rather than a VLM scanning uniformly.

  • No post-training. Everything here is prompted and orchestrated, not fine-tuned. The obvious next step is distilling the verifier's confirmed/rejected decisions into a small VLM and checking whether it holds precision at a fraction of the cost.

  • Not a substitute for a trained safety professional, and the eval footage is training material rather than live operational CCTV.

Where this goes at scale

The shape here is deliberately the shape of NVIDIA's VSS blueprint: ingest, VLM perception, retrieval over descriptions, verified alerts. Foreman is the laptop-scale version, and each stage has a real replacement — flat numpy index → a vector DB, uniform sampling → DeepStream-triggered clips, hosted NIM → self-hosted NIM containers, single video → multi-stream. What survives the swap is the part I would argue matters most and that the blueprint leaves to you: the verification stage and the eval harness that proves it earns its place.

Data and licensing

Code is MIT. No video is redistributed in this repo. The eval ships as labels and derived model output keyed on chunk ids; scripts/fetch_data.py rebuilds the local footage from public sources for analysis. See DATA.md.


Built by Yash Nirwan · github.com/YashNirwan

Available Tools

7 tools
explain_alertA

Return the full evidence chain behind one alert.

Gives the proposing model's claim, its confidence, the verifier's reasoning and the standard cited, so a supervisor can judge the alert rather than take it on trust.

Inputs: alert_id (required), video_id. Returns: object

hazard_taxonomyA

The hazard classes this system detects, with evidence bars and known failure modes.

Read this before interpreting alerts: each class has a specific evidence standard, and knowing it is the difference between reading an alert and trusting it.

Inputs: no named parameters. Returns: result

list_alertsA

List verified safety alerts for a processed video.

Args: video_id: which processed video; omit if only one has been processed. hazard_type: optional filter, one of the taxonomy keys. min_severity: low, medium or high. Defaults to low, meaning everything.

Returns confirmed alerts only. Rejected detections are available via list_rejected_detections if you want to audit what the verifier removed.

Inputs: video_id, hazard_type, min_severity. Returns: result

list_processed_videosA

List videos that have been processed and are available to query.

Inputs: no named parameters. Returns: result

list_rejected_detectionsA

List detections the verifier rejected, with the evidence it found missing.

Useful for auditing whether the system is suppressing real hazards. A rejection that looks wrong here is the fastest signal that the taxonomy's evidence bar for that class is set incorrectly.

Inputs: video_id. Returns: result

search_timelineA

Search the footage in natural language.

Searches every analysed window, not just the ones that produced an alert, so it answers questions the hazard taxonomy does not cover: "someone carrying a long load past the racking", "the aisle by the loading door", "anyone on a phone".

Args: query: what to look for, in plain language. video_id: which processed video; omit if only one has been processed. k: how many windows to return.

Inputs: query (required), k, video_id. Returns: result

shift_summaryB

Summarise a processed shift: volumes, alert mix, and what was filtered out.

Inputs: video_id. Returns: object

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.9/5.0

    Scored across 7 tools

    Disambiguation5/5

    Each tool has a clear, non-overlapping purpose: listing videos, listing alerts, summarizing shifts, listing rejected detections, semantic search, explaining alert evidence, and exposing the taxonomy. There is no ambiguity about which tool to use for a given task.

    Naming Consistency4/5

    Most tools follow a verb_noun pattern (list_processed_videos, list_alerts, list_rejected_detections, search_timeline, explain_alert), but shift_summary and hazard_taxonomy break the pattern by leading with a noun. The deviation is minor and the naming remains readable.

    Tool Count5/5

    Seven tools is well-scoped for a video safety analysis and auditing system. Each tool covers a distinct query or audit function without redundancy, and the count is neither too thin nor too heavy.

    Completeness4/5

    The surface covers the core workflows: enumerating videos, querying alerts, auditing rejections, searching footage, and understanding alerts. A minor gap is the lack of a dedicated get_video detail tool, but list_processed_videos provides sufficient access for most use cases.

    Maintenance

    ActivitySlowing
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • F
      license
      Not graded
      quality
      C
      maintenance
      Enables querying PPE compliance analytics from precomputed video inference results, providing tools for site summaries, worker status, violations, and trends without re-running inference.
      1
      -
    • A
      license
      Not graded
      quality
      D
      maintenance
      Enables warehouse operations agents to inspect tasks, deliveries, stock, and KPIs, and to attempt write operations in shadow mode with human approval and segregation of duties enforced as code.
      MIT
    • F
      license
      Not graded
      quality
      B
      maintenance
      Enables warehouse operations teams to investigate delayed inventory stages and safely create deduplicated human-review escalations using read-only evidence and signed continuation tokens.
      -
    • A
      license
      Not graded
      quality
      C
      maintenance
      Enables analytics agents to run validated read-only SQL against a warehouse API with enforced row limits, required JSON responses, and secret-safe audit metadata.
      MIT