LLMFeeder

Checks whether AI-generated text is actually supported by your source documents, one claim at a time.

What it looks like

$ llmfeeder check answer.md --sources ./reports

claims needing attention
CONTRADICTED   0.45  Free cash flow was 510 million dollars for the quarter.
                     q3-earnings.txt chars 728-772
                     claim states 510 million dollars; the cited passage states 410 million dollars
CONTRADICTED   0.28  Headcount at the end of the quarter was 12,400 employees.
                     q3-earnings.txt chars 903-948
                     claim states 12,400; the cited passage states 11,400, 10,900
UNSUPPORTED    0.14  The company announced a quarterly dividend of 12 cents per share.
                     no supporting span found

9 claims  6 supported  0 partial  1 unsupported  2 contradicted
faithfulness 0.73  (lexical judge, 1 source)

Add --report out.html and you get a page where clicking a claim lights up the exact characters it rests on.

Try the live example report

That page is generated by llmfeeder demo and committed to the repo, so it is the real output and not a mockup. Nothing to install.

Related MCP server: CiteGuard

How it works

text -> segment -> retrieve -> align -> judge -> verdicts + citations

Segment

Split the text into individual claims. This was fiddlier than I expected. You cannot split on full stops, because Oct. 2025 and 3.5 percent break immediately. I keep a list of abbreviations, treat a digit either side of a dot as a decimal, and skip headings, questions, code blocks and short fragments, since none of those assert anything and scoring them only adds noise to the final number.

Retrieve

Index the sources by sentence, then build candidate windows of one to three sentences so a claim spanning a sentence break can still match. Windows are ranked by IDF-weighted word overlap, so rare words count more than common ones. I tuned this for recall rather than precision, because a passage that never gets retrieved means the claim is marked unsupported no matter how good the judging is.

Align

Narrow the winning window down to the smallest character range that still covers the matched words. This is the step that turns "somewhere on page 4" into offsets you can highlight.

Judge

Combine word overlap with the two checks similarity cannot do: reconcile numbers by value, and compare polarity so a flipped negation gets caught.

The four verdicts

I started with supported and not supported, and that turned out to be too coarse, because the two failure cases need completely different responses from whoever is reading.

UNSUPPORTED means go find a citation. CONTRADICTED means you wrote something false. Collapsing those into one bucket makes the output much less useful.

How I checked whether it works

llmfeeder eval        # reproduces every number below

68 labelled claims, offline judge, no API key needed.

Collapsed down to the question a user actually has, which is "should I go and check this claim":

What those numbers do not prove. The set is 68 claims I wrote, so it measures the failure modes I thought of, and there are certainly ones I did not. The partial row rests on 8 examples, so its 0.57 moves a long way if a single claim flips, and I would not read anything into that figure on its own. The one result I do lean on is the false alarm count: 1 in 32 means the tool is not crying wolf, which is the thing that would make me stop using it.

I put these thresholds into the test suite, so if I change the scoring later and it gets worse, CI goes red instead of quietly letting it slide.

Where it still gets things wrong

The tool disagreed with my labels on 10 of the 68. I find these more useful than the headline number, so here they all are.

Swapped names, places or facts, 3 cases. "The rover is powered by solar panels" against a source saying it is powered by a radioisotope thermoelectric generator comes back supported. Every content word in the claim appears in the passage, and the offline judge counts words, so it has no idea the important one got swapped. Same story for data centres in "Dublin and Tokyo" against a source saying "Dublin and Singapore".

Opposites, 2 cases. "Background tasks run before the response has been sent" against "Background tasks run after the response has been sent". No number disagrees and no negation word appears, so nothing trips.

Relationships that are never actually stated, 2 cases. If a claim says A happened because of B, and the source mentions A and mentions B but never connects them, the word overlap is satisfied anyway.

Synonyms, 1 case. "Okafor scored" against "Okafor took the lead with a header". My stemmer does not bridge that.

The remaining 2 are partial borderline calls that I find hard to label consistently myself, which is its own kind of finding about the class.

The first two groups are the whole reason the optional LLM judge exists. They also point at the real boundary of this design: the offline judge is word overlap plus arithmetic, it is English only because the stemmer and stopword list are English, and it judges each claim against a single best passage, so a claim that is only true once you combine two documents shows up as partial.

The optional LLM judge

Off by default. It only activates if it finds an API key on your machine, and it prints a notice before sending anything so you know when your source text is about to leave the machine.

The part I was careful about is that the model does not get to invent citations. It only sees the same candidate passages the offline judge saw, and whatever it quotes gets looked up in that passage afterwards. If the quote is not there, the citation is thrown away and the claim is downgraded. A made up citation cannot reach the output, which felt like the minimum bar for a tool whose entire job is checking things.

cp .env.example .env        # add OPENAI_API_KEY or ANTHROPIC_API_KEY
llmfeeder check answer.md -s ./docs --judge llm --model gpt-4o-mini

No SDK required. It is about a hundred lines over urllib covering OpenAI-compatible endpoints (OpenAI, Groq, Together, OpenRouter, local Ollama) and Anthropic, which keeps the base install at two dependencies.

MCP server

This lets an AI agent check its own output before showing it to you. The case I had in mind is self-correction partway through a task: the model writes a summary, calls the tool, sees one sentence is unsupported, and rewrites that sentence instead of shipping it.

pip install -e ".[mcp]"
llmfeeder mcp --sources ./docs

// claude_desktop_config.json
{
  "mcpServers": {
    "llmfeeder": { "command": "llmfeeder", "args": ["mcp", "--sources", "/path/to/docs"] }
  }
}

Two tools. verify_against_sources reads a folder on disk, verify_against_text takes passages already in the conversation. Both return the score, the per-claim verdicts, and a needs_attention list sorted worst first, so the model can go and fix the specific sentence that failed.

Using it from Python

from llmfeeder import check, write_report

result = check("Revenue grew 34% to $2.1B.", ["./sources"])

print(result.faithfulness)
print(result.counts())              # {'supported': 4, 'partial': 0, ...}

for claim in result.problems():     # worst first
    print(claim.verdict, claim.claim.text)
    if claim.best:
        print("  ", claim.best.span.locator())   # report.pdf p.4 chars 1180-1223
        print("  ", claim.best.span.text)        # the actual supporting text

write_report(result, "out.html")

One rule I held to everywhere: a SourceSpan is always a real character range into the loaded document, so document.text[span.start:span.end] == span.text always holds. That invariant is what makes the highlighting trustworthy, and there is a test for it.

Installing and running it

pip install git+https://github.com/adwitiyashukla/LLMFeeder.git

Or clone it, which is easier if you want to read the code:

git clone https://github.com/adwitiyashukla/LLMFeeder.git
cd LLMFeeder
pip install -e ".[all]"

Python 3.11 or newer. The command is llmfeeder. Optional extras are pdf for PDF sources, mcp for the server, and all for both. Without them it still reads txt, markdown, HTML and JSON, and the only dependencies are typer and rich.

llmfeeder demo --open                     # runs the built-in example and opens the report

llmfeeder check answer.md --sources ./docs
llmfeeder check answer.md -s ./docs -s ./notes.pdf --report out.html --open
echo "Revenue grew 34%." | llmfeeder check - --sources ./docs
llmfeeder check answer.md -s ./docs --json results.json --quiet

It also works as a build gate:

llmfeeder check generated-summary.md --sources ./source-of-truth --fail-under 0.9

That exits non-zero if the score is too low, so a docs build can refuse to publish a page whose claims have drifted from the source material.

What is in the repo

src/llmfeeder/
  cli.py            the llmfeeder command
  corpus.py         loading and normalising source files
  segment.py        splitting text into individual claims
  retrieve.py       sentence index and candidate windows
  textutil.py       tokenising, stemming, number parsing
  judge/
    lexical.py      the offline judge
    llm.py          the optional model judge
  report.py         the self-contained HTML report
  evaluation.py     the scoring harness and metrics
  models.py         the data types
  mcp_server.py     the two MCP tools
  data/eval.json    68 labelled claims across 8 corpora
tests/              108 tests
docs/               the committed example report, served by GitHub Pages

Commands

Flags for check: --report out.html, --json out.json, --open, --judge lexical|llm|auto, --threshold, --fail-under, --top-k, --verbose, --quiet.

File types it reads

With no extra dependencies: .txt, .md, .html, .json, .jsonl, .csv, .yaml and most plain source files. HTML is stripped to readable text with the standard library, and JSON is flattened into path: value lines so text buried inside it is still findable. PDFs need the pdf extra and carry page numbers through into the citations.

Author

Adwitiya Shukla

License

MIT