Skip to main content
Glama

Groundlens:RAG 答案的校对器

Groundlens

PyPI Python License Runtime dependencies groundlens MCP server

CI OpenSSF Best Practices OpenSSF Scorecard Determinism

groundlens.dev

工作原理 · 安装 · 快速开始 · MCP 服务器 · 局限性 · 可复现性

Groundlens 是您模型所写内容的校对器。它会标出您的来源不支持的词语——并告诉您每个词本应怎么说。它检查 RAG 答案相对于其检索来源的依据性和忠实性,这是人们为进行幻觉检测、引文检查或 RAG 评估而做的工作——不同之处在于,它返回的是供审阅者查看的标记和证据,而不是一个判定或一个需要设定阈值的分数。

QUESTION    What is the invoice total?
SOURCE      ...the total amount due is 10,000 dollars, payable within 30 days...
ANSWER      The invoice total is 1,000 dollars, due in 30 days.

GROUNDLENS  1,000   nothing supports this.   Closest in invoice.pdf#p1: '10,000'

它从不告诉您答案错了。它告诉您该看哪个词,以及该打开哪份文档。三十秒的人工关注,而不是五分钟。

工作原理

How Groundlens checks words and numbers

Groundlens 以两种不同的方式处理词语和数字的比较:

词语

数字

词语以意义为锚。一个词的支持度是它与来源中任何词达到的最高余弦相似度,使用一个冻结的现成编码器——与您的检索已经使用的同类编码器。

数字以算术为锚。数字被解析为格式规范化的值——10,000、10000、$10,000、10 000 以及(在声明了区域设置的情况下)10.000 是同一个数字——然后与来源中的每个值进行核对。支持度恰好是 1.0 或恰好是 0.0。相似度不允许投票。

Groundlens 输出最低分,而不是平均值。每个 token 相似度指标都按均值聚合,而均值正是单个 token 错误消亡的地方。

一个实际例子:十不是一百

检索到的文档说应付总额是 10,000 美元。答案说 1,000 美元。人类一眼就能发现,不需要金融学位。

嵌入相似度做不到。正确答案和错误答案之间的余弦约为 0.99——错误像一滴墨水溶入水池一样溶入向量中。LLM 评判者也做不到:它阅读是为了合理性,而“总额是 1,000 美元”是一句关于发票的完全合理的句子。经过训练的跨度检测器也做不到,因为单位数字替换在其训练标签中很少见。

句子编码器按词汇、主题和结构组织文本。从不按事实。一个正确句子中的错误数字,对于将释义折叠的编码器来说,几乎就是一句释义。

在那张发票上,错误答案的平均支持度是 0.79——看起来不错。最弱的锚点是 0.00——这是页边的一个标记。

操作阈值

这个库没有默认阈值。阈值是部署的属性,而不是方法的属性。它取决于编码器、您的数据,以及假阳性与假阴性相比对您造成的代价。这些在这里都是未知的。

规则背后有一个测量。在我们运行的操作点网格中,对于我们测试的每个单遍检测器(包括这个),在 95% 召回率下的最佳假阳性率为 0.65。在受监管审查实际需要的召回率下,该网格中没有固定的截断值可用。发布一个这样的值意味着发布一个我们已经知道不成立的数字。

Support scores and the weakest anchor

groundlens 提供的是:

  • 每个词的支持度分数,分数越低表示来源支持越少。

  • 带凭证的标记:词、其跨度、其支持度以及最近的证据句子,以便审阅者能在几秒钟内检查任何调用。

  • 一个 calibrate() 函数,它根据您自己的标注数据拟合一个截断值。它拒绝在少于 200 个标注示例上运行,因为低于该数量,截断值就是噪声。

如果您的流程中需要阈值,请在您的标注数据上运行 calibrate():

from groundlens import calibrate

point = calibrate(labelled, target_recall=0.95)
print(point.threshold, point.fpr, point.fpr_ci95)   # read the fpr first

calibrate() 至少需要 200 个标注示例,因为低于该数量,95% 召回率阈值是从少数几个点估计出来的。

Related MCP server: Sentry MCP

安装

pip install groundlens              # zero runtime dependencies. Not numpy, not torch
pip install "groundlens[encoder]"   # + the reference sentence encoder
pip install "groundlens[encoder,mcp]"   # + the MCP server, for Claude Desktop and friends

核心安装完全不引入任何包,如果这一点发生变化,CI 作业将使构建失败。之前的版本在您做任何事情之前就安装了大约两 GB 的深度学习堆栈。

快速开始

from groundlens import proofread, SentenceTransformerEncoder

answer = "The invoice total is 4.75% payable within 45 days."
sources = [("policy.pdf#p3", "The rate stated in the policy is 3.90% and the term is 30 days.")]

marks = proofread(answer, sources, encoder=SentenceTransformerEncoder(), k=2)

print(marks.report())
#  4.75%   support 0.00    nearest in policy.pdf#p3: '3.90%'
#  45      support 0.00    nearest in policy.pdf#p3: '30'

每个标记都带有其凭证:

for anchor in marks.weakest:
    anchor.text            # '4.75%'          the word in the answer
    anchor.span            # (21, 26)         where it sits
    anchor.kind            # 'numeral'        checked by arithmetic, not meaning
    anchor.support         # 0.0              absent from the sources
    anchor.evidence_id     # 'policy.pdf#p3'  which document to open
    anchor.evidence_text   # '3.90%'          what it should have matched

从 shell 中:

groundlens read --answer answer.txt --context policy.pdf#p3=policy.txt

MCP 服务器

同样的校对器,就在您的助手中。Groundlens 附带一个 MCP 服务器,因此 Claude Desktop、Claude Code、Cursor、VS Code 或任何其他 MCP 客户端都可以在不离开对话的情况下检查答案与其来源。它通过 stdio 在本地运行。任何文本都不会外传。

pip install "groundlens[encoder,mcp]"
python -m groundlens.mcp

然后将您的客户端指向它。在 claude_desktop_config.json 中——或 Cursor 和 VS Code 中对应的 mcp.json:

{
  "mcpServers": {
    "groundlens": {
      "command": "python",
      "args": ["-m", "groundlens.mcp"]
    }
  }
}

如果安装了 Groundlens 的 Python 不在您的 PATH 中,请使用其绝对路径:/path/to/venv/bin/python。

唯一的工具

find_unsupported_words(answer, sources, k=4, locale="und")

answer

要检查的模型输出

sources

[{"id": "policy.pdf#p3", "text": "..."}]。id 会出现在结果中,因此读者知道要打开哪份文档

k

要返回的最弱锚点数量

locale

这些文档如何书写数字。es 将 1.234 读作 1234,en 将其读作 1.234,und 保留两种读法

它返回最弱的锚点及其凭证、下限、编码器 id 以及结果的 sha256:

{
  "weakest_anchors": [
    {
      "word": "4.75%",
      "support": 0.0,
      "checked_by": "arithmetic",
      "closest_in_sources": "3.90%",
      "source_id": "policy.pdf#p3",
      "notes": []
    }
  ],
  "floor": 0.0,
  "n_marked": 12,
  "encoder_id": "all-mpnet-base-v2@<revision-sha>",
  "sha256": "..."
}

故意只提供一个工具。之前的服务器宣传了三个,这就是一个产品在任何人安装之前就变成三个故事的方式。

这里和这个库的其他地方一样,没有判定,也没有阈值。数字上的 support 为 0.00 意味着该值在来源中不存在。对于词语,它意味着没有找到词汇锚点,这在忠实的释义中是常见的。服务器报告标记;读者决定。

编码器在第一次调用时加载,而不是在启动时,模型在第一次使用时下载一次(约 420 MB)。

局限性

  • 它无法验证计算值——例如,来源说“收入从 5M 增长到 15M”,而答案说“收入翻了三倍”。

  • 词语通道检查一个词是否得到来源的支持。它不检查该词是否附着在正确的事物上。如果答案关于发票 A 说“30 天内付款”,而 30 天属于同一上下文中的发票 B,那么该词得到支持,不会出现标记。

  • 它无法检查推理。这属于蕴含模型的范畴。

  • 它继承了您的检索。如果段落是错误的,那么答案的依据也是错误的。

  • 分词假设使用空格分隔的脚本,当文本主要是 CJK 或泰语时,它会发出警告而不是假装。

可复现性

  • 数字通道是精确的。 十进制比较、固定的算术上下文、区域设置来自参数而从不来自 LC_ALL。在任何机器上逐字节相同——CI 在 PYTHONHASHSEED=random 和土耳其区域设置下的十种操作系统 × Python 组合上证明了这一点。

  • 词汇通道是来自固定编码器修订版的 float32 余弦——不是模型名称,因为静默重新上传会改变您发布过的每个数字。它在各平台上可复现到 1e-6,并且最弱锚点的顺序是稳定的。它在 x86 和 Apple Silicon 之间不是逐位相同的,我们也不声称如此。

  • marks.sha256 精确覆盖结构和数字支持度,并将词汇支持度四舍五入到六位小数。重现哈希即重现结果,而不是算术的最后几位。

groundlens.dev · PyPI · 撤回 · 贡献 · Apache-2.0

Available Tools

3 tools
verify_answerB

Verify an answer against its sources under a policy and return the sealed record.

    sources: (id, text) pairs, {"id","text"} dicts, or bare strings.
    policy: a built-in name (e.g. "eu_ai_act_high_risk_v1"), a path, or YAML.
    Returns the decision (PASS/REVIEW/FAIL), the evidence, the regulatory
    mapping and the record with its content hash.
    
ParametersJSON Schema
NameRequiredDescriptionDefault
answerYes
localeNound
policyNo
sourcesYes
questionNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

B3.4/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of disclosing behavior. It does describe the return value (decision, evidence, regulatory mapping, record with content hash), which is helpful. However, it does not state whether the operation is read-only, whether it stores or modifies any data, or what side effects might occur. For a verification tool, this is a notable gap, especially since the action of returning a 'sealed record' implies some immutability but not explicitly a non-destructive operation.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise and front-loaded, stating the core action in the first sentence. It then efficiently lists input format variants and the return contents. The multi-line formatting with indentation is slightly unconventional but does not harm readability. There is minimal redundancy, and every sentence adds value.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool has 5 parameters (2 required) and an output schema exists, the description is moderately complete. It covers the key inputs (sources, policy) and mentions the return structure. However, it omits explanation of 'locale' and 'question', and does not provide usage context relative to sibling tools or error scenarios. The presence of an output schema lightens the need to detail return fields, but the missing parameter semantics and lack of sibling differentiation reduce completeness.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate. It explains the semantics of 'sources' (formats) and 'policy' (built-in, path, YAML). The 'answer' parameter is implicitly clear from the first sentence. However, 'locale' and 'question' are not described at all. Thus, the description covers only a portion of the parameters, leaving two parameters with no guidance beyond their names.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description begins with a clear, specific verb and resource: 'Verify an answer against its sources under a policy and return the sealed record.' This distinguishes it from siblings (verify_run, verify_records) by focusing on answer verification, which is a distinct operation.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides usage details such as acceptable formats for sources (id/text pairs, dicts, strings) and policy (built-in name, path, YAML), which implicitly guides the caller. However, it does not explicitly state when to use this tool versus the sibling tools verify_run or verify_records, nor does it mention any exclusions or alternative conditions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

verify_recordsA

Verify a log of records offline: every hash, every link, every signature.

    records: the JSON Lines text of an answer-record or run-record log.
    Returns {"ok", "verified", "kind"}; fails if any record or link was altered.
    
ParametersJSON Schema
NameRequiredDescriptionDefault
recordsYes

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full behavioral burden and does meaningful work: it discloses the return shape ('Returns {"ok", "verified", "kind"}'), the failure mode ('fails if any record or link was altered'), and that the operation happens offline. It stops short of explicitly stating verification is non-destructive, a minor gap given 'verify' implies it.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is compact — purpose is front-loaded in the first sentence, followed by the parameter and then the return/failure behavior. Every clause carries information an agent needs; there is no filler or restatement of the tool name.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a one-parameter verification tool with an output schema present, the description covers purpose, input format, return shape, and failure behavior — nearly everything needed to call it correctly. Minor gaps like the possible values of 'kind' are left to the output schema, which is acceptable per the rubric.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate, and it does: it documents 'records' as 'the JSON Lines text of an answer-record or run-record log,' adding format and content meaning the schema lacks. It doesn't specify the exact structure of a valid record, but for a single string parameter the added semantics are substantial.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb and resource ('Verify a log of records offline') with concrete scope ('every hash, every link, every signature'), so an agent can tell exactly what operation this performs. It also distinguishes this from the siblings verify_run and verify_answer by clarifying that it accepts both answer-record and run-record logs.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage by noting the tool accepts 'an answer-record or run-record log,' which hints it covers the domains of both siblings. However, it never names verify_run or verify_answer or gives an explicit when-to-use vs. when-not-to-use rule, leaving the routing decision to inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

verify_runA

Verify an MCP execution trace under an execution policy and return the run record.

    trace: the MCP session as JSON-RPC messages (JSON Lines).
    policy: the execution policy, as YAML/JSON text or a path.
    Returns the gate (ALLOW/REVIEW/DENY), any breaches, and the signed run record.
    
ParametersJSON Schema
NameRequiredDescriptionDefault
traceYes
policyYes
run_idYes
systemYes
started_atNo
system_versionNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A3.7/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It discloses the return values (gate, breaches, signed run record) but does not mention potential side effects (e.g., whether it writes or stores anything), permission requirements, or error behavior. This is some behavioral context but incomplete for a tool with no annotation safety net.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is reasonably concise, with the purpose front-loaded and parameters broken into clear lines. It avoids redundant wording and communicates the key return values efficiently, though it could be tightened slightly.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has an output schema, so return format details are not strictly required, and the description already provides a high-level return summary. However, given the six-parameter complexity and lack of annotations, the description should explain all parameters and ideally differentiate usage from siblings. It covers the core purpose but leaves several parameters and usage guidance gaps.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate. It explains trace (format: JSON-RPC messages as JSON Lines) and policy (format: YAML/JSON text or path), which is useful. However, it does not explain run_id, system, started_at, or system_version, leaving 4 of 6 parameters undocumented in both schema and description. This is a significant gap.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool verifies an MCP execution trace against an execution policy and returns the run record with gate, breaches, and signed record. This specific verb+resource distinguishes it from sibling tools verify_answer and verify_records, which target different resources.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage by specifying it is for verifying execution traces, which gives clear context. However, it does not explicitly mention when not to use it or point to alternatives like verify_answer or verify_records, so it lacks explicit exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 4 tool updatesv3.0.6
    • Removedfind_unsupported_words
    • Addedverify_answer
    • Addedverify_records
    • Addedverify_run
  2. 1 tool updatev0.1.0
    • First observedfind_unsupported_words

TDQS

A4/5.0

Scored across 3 tools

Disambiguation5/5

The three tools address clearly different verification targets: execution traces, answer-source pairs, and record logs. No two tools accept the same kind of input or produce the same kind of output, so an agent can select among them without ambiguity.

Naming Consistency5/5

All tool names follow the same verify_<noun> pattern with snake_case, matching the verb-object convention. The naming makes the input type immediately predictable from the tool name.

Tool Count5/5

At three tools, the surface is tightly scoped to the verification domain: run traces, answers, and record-chain integrity. Each tool covers a distinct workflow and none feels redundant.

Completeness5/5

The toolkit covers the full observed verification lifecycle: generating verified run records, generating answer records, and validating logs of those records. Policies are provided as parameters rather than requiring separate management tools, so there are no obvious dead ends.

Maintenance

ActivityActive
ResponsivenessNo issues

Related MCP Connectors

Related MCP Servers