HALLUMARK

LLM hallucination & grounding auditor for RAG systems

AI Security & Governance — securing LLMs, agents, and the MCP supply chain.

pip install cognis-hallumark
hallumark scan .            # → prioritized findings in seconds

🔎 Example output

Real, reproducible output from the tool — runs offline:

$ hallumark-emit --version
hallumark 0.1.0

$ hallumark-emit --help
usage: hallumark [-h] [--version] <command> ...

HALLUMARK - audit LLM/RAG answers for hallucinations by checking whether each
claim is grounded in the retrieved context.

positional arguments:
  <command>
    audit     Audit a file of RAG records for ungrounded / hallucinated
              claims.

options:
  -h, --help  show this help message and exit
  --version   show program's version number and exit

Input is JSON or JSONL where each record has: question, answer, and contexts
(a list of retrieved chunks). Returns non-zero exit when unsupported claims
are found.

Blocks above are real hallumark output — reproduce them from a clone.

Sample result format (illustrative values — run on your own data for real findings):

{
"feed": {
"type": "STIX",
"value": "{\"indicator\":{\"id\":\"1234567890\",\"name\":\"Example Indicator\"},\"observed-data\":[{\"id\":\"1\",\"timestamp\":1643723400,\"data\":\"example data\"}]}"
},
"status": 200,
"message": "Findings successfully forwarded to STIX platform"
}

{"indicator":{"id":"1234567890","name":"Example Indicator"},"observed-data":[{"id":"1","timestamp":1643723400,"data":"example data"}]}

Related MCP server: policy-gate-mcp

Usage — step by step

1. Install:

   pip install hallumark

2. Audit RAG records — each record is JSON/JSONL with question, answer, and contexts (the retrieved chunks). HALLUMARK checks whether each claim is grounded:

   hallumark audit records.jsonl

   You get per-record PASS/FAIL plus faithfulness, context-utilization, and answer-relevance scores.

3. Read from stdin with -:

   cat records.jsonl | hallumark audit -

4. Tune the strictness — per-claim support threshold and the minimum record faithfulness to PASS:

   hallumark audit records.json --threshold 0.35 --min-faithfulness 0.9 --show-grounded

5. CI gate — emit JSON and rely on the exit code (1 when unsupported/hallucinated claims are found):

   hallumark audit records.jsonl --format json | jq '.total_unsupported'

Contents

• Why hallumark? · Features · Quick start · Example · Architecture · AI stack · How it compares · Integrations · Install anywhere · Related · Contributing

Why hallumark?

LLM hallucination & grounding auditor for RAG systems — without standing up heavyweight infrastructure.

hallumark is single-purpose, scriptable, and self-hostable: point it at a target, get prioritized results in the format your workflow already speaks (table · JSON · SARIF), gate CI on it, and let agents drive it over MCP.

Features

• ✅ Split Claims

• ✅ Audit Record

• ✅ Audit Records

• ✅ Load Records

• ✅ Parse Records

• ✅ Runs on Linux/macOS/Windows · Docker · devcontainer

• ✅ Ports in Python, JavaScript, Go, and Rust (ports/)

Quick start

pip install cognis-hallumark
hallumark --version
hallumark scan .                       # scan current project
hallumark scan . --format json         # machine-readable
hallumark scan . --fail-on high        # CI gate (non-zero exit)

Example

$ hallumark scan .
  [HIGH    ] HAL-001  example finding             (./src/app.py)
  [MEDIUM  ] HAL-002  another signal              (./config.yaml)

  2 findings · risk score 5 · 38ms

Architecture

flowchart LR
  IN[target / manifest] --> P[hallumark<br/>checks + rules]
  P --> OUT[findings (JSON / SARIF)]

Use it from any AI stack

hallumark is interoperable with every popular way of using AI:

• MCP server — hallumark mcp (Claude Desktop, Cursor, Cognis.Studio, uncensored-fleet)

• OpenAI-compatible / JSON — pipe hallumark scan . --format json into any agent or LLM

• LangChain · CrewAI · AutoGen · LlamaIndex — wrap the CLI/JSON as a tool in one line

• CI / scripts — exit codes + SARIF for non-AI pipelines

How it compares

Built in the spirit of explodinggradients/ragas, re-framed the Cognis way. Missing a credit? Open a PR.

Integrations

Pipes into your stack: SARIF for code-scanning, JSON for anything, an MCP server (hallumark mcp) for AI agents, and a webhook forwarder for SIEM/Slack/Jira. See docs/INTEGRATIONS.md.

Install — every way, every platform

pip install "git+https://github.com/cognis-digital/hallumark.git"    # pip (works today)
pipx install "git+https://github.com/cognis-digital/hallumark.git"   # isolated CLI
uv tool install "git+https://github.com/cognis-digital/hallumark.git" # uv
pip install cognis-hallumark                                          # PyPI (when published)
docker run --rm ghcr.io/cognis-digital/hallumark:latest --help        # Docker
brew install cognis-digital/tap/hallumark                             # Homebrew tap
curl -fsSL https://raw.githubusercontent.com/cognis-digital/hallumark/main/install.sh | sh

Related Cognis tools

• aegis — AI Agent Permission & Access Auditor — surfaces the lethal trifecta of credentials + injection + reach

• promptmirror — Prompt-injection & indirect-injection scanner for any LLM context input

• ledgermind — Local LLM cost & token forensics proxy with anomaly detection

• adversa — LLM red-team harness — OWASP LLM Top 10 + MITRE ATLAS attack packs

• guardpost — Runtime agent firewall — PII redaction, rate limits, policy enforcement

• aicard — Auto-generated NIST AI RMF / EU AI Act Annex IV model & system cards

Explore the suite → 🗂️ all 170+ tools · ⭐ awesome-cognis · 🔗 cognis-sources · 🤖 uncensored-fleet · 🧠 engram

Contributing

PRs, new rules, and demo scenarios are welcome under the collaboration-pull model — see CONTRIBUTING.md and SECURITY.md.

⭐ If hallumark saved you time, star it — it genuinely helps others find it.

Interoperability

{} composes with the 300+ tool Cognis suite — JSON in/out and a shared OpenAI-compatible /v1 backbone. See INTEROP.md for the suite map, composition patterns, and reference stacks.

License

Source-available under the Cognis Open Collaboration License (COCL) v1.0 — free for personal, internal-evaluation, research, and educational use; commercial / production use requires a license (licensing@cognis.digital). See LICENSE.