NotesToReport

An open-source MCP developer tool for SuperDocs — turns a folder of rough notes into a citation-grounded report, then hands it off to SuperDocs to create a live document. Every claim in the final report is either backed by an exact source citation (file + line range) or flagged as unverified — never silently invented.

Built as part of a job application task for SuperDocs.

Why this exists

AI-written summaries can quietly add details that were never in the source material. That's fine for casual use, but risky if the output is going to a client or a boss. NotesToReport adds a verification layer on top of an AI-generated report: every sentence is checked against the original notes before the document is ever created.

How it works

1. Read notes — point the tool at a folder of rough notes (e.g. standup notes).

2. Draft a report — an LLM reads all the notes and writes a clean, structured report.

3. Ground every claim — the report is split into individual claims. Each one is checked against the source notes; if supported, it gets a citation pointing to the exact file and line range. If not, it's flagged as unverified instead of silently passing through.

4. Score faithfulness — the percentage of claims that are actually grounded becomes the report's faithfulness score.

5. Circuit breaker — if the score is below 0.85, the tool refuses to hand off to SuperDocs at all.

6. Hand off to SuperDocs — if it passes, the report is sent to SuperDocs via the chat_async tool, which creates a live document with citations as footnotes and unverified claims as inline comments.

Project structure

connector/    MCP connector — connects to SuperDocs' real MCP server
citation/     Citation grounding engine (chunking, retrieval, entailment)
handoff/      Faithfulness circuit breaker + SuperDocs hand-off logic
engine/       Report synthesis + end-to-end test scripts
sample-notes/ Example notes for testing

Setup

npm install
cp .env.example .env
# then fill in GROQ_API_KEY and SUPERDOCS_API_KEY in .env

Running the dry-run pipeline

Runs the full pipeline against sample notes without contacting SuperDocs — prints what would be sent.

npx ts-node engine/src/testPipeline.ts ./sample-notes

Running the adversarial test

Injects a false claim directly into the AI-generated report (not the source notes) and confirms the grounding engine correctly rejects it. See FINDINGS.md for full test results and methodology.

npx ts-node engine/src/testFakeClaimInReport.ts ./sample-notes

Current status

• Citation grounding engine — built and tested, including an adversarial test confirming it correctly rejects hallucinated claims not present in the source notes.

• Faithfulness circuit breaker — built and tested.

• SuperDocs MCP connector — built, uses only SuperDocs' confirmed tools (chat_async, get_job, upload_attachment_base64, get_attachment_status).

• Report synthesis — built and tested end-to-end.

Known limitations

• The embedding function used in tests (simpleEmbed) is a lightweight local word-hash stand-in, not a real semantic embedding model. A real embeddings API is the natural next step for grounding quality on larger, real-world note sets.

• The connector has not yet been run against a live SuperDocs connection end-to-end (dry-run only so far).

• No CLI wrapper yet (bin/cli.ts referenced in package.json is not yet built) — the pipeline is currently run directly via ts-node.

See FINDINGS.md for detailed test results.

License

MIT