Semantix-verify
针对明确的意图验证每个 LLM 输出。获取分数、判定结果和防篡改凭证。在本地进行。耗时约 15-50 毫秒。无需 API 密钥。
pip install semantix-aifrom semantix import Intent, validate_intent
class ResolutionPolite(Intent):
"""The response must acknowledge the customer's issue and propose a concrete next step, in a polite tone."""
@validate_intent(ResolutionPolite, audit=True)
def handle_complaint(message: str) -> str:
return call_my_llm(message)
reply = handle_complaint(incoming)
# Returns the validated reply — or raises SemanticIntentError.
# The audit engine has already written a hash-chained receipt to disk.为什么存在这个项目
LLM 应用程序通常会悄悄跳过证明输出是否符合目的这一步骤。常见的解决方法——调用更大的 LLM 作为评判者——存在三个问题:
它会漂移。 相同的输入,在不同的运行中得分不同。当监管机构要求“重新运行此验证”时,会得到不同的答案,这与系统损坏的证据无法区分。
它会将个人信息发送到您的网络之外。 每次评判者调用都会将输出发送到第三方 API。根据 POPIA §72(或 GDPR 第 44 条,或欧盟 AI 法案的高风险系统义务),这是一个需要记录的问题,而不是默认行为。
它不产生凭证。 验证发生了,分数返回了,但没有任何记录能经受住审计。
semantix 用本地、确定性的验证器和防篡改日志取代了这种反射式操作。每次验证都会生成一个签名过的 JSON-LD 证书,并与前一个证书进行哈希链连接。修改任何条目,后续的每个哈希都会失效。监管机构不需要信任您的数据库——数学证明了链条是完整的。
Related MCP server: agentvet-mcp
您将获得什么
1. 作为装饰器的验证
from semantix import Intent, validate_intent
class MedicalAdvice(Intent):
"""The text provides a medical diagnosis or treatment recommendation."""
@validate_intent(~MedicalAdvice) # Must NOT give medical advice
def chatbot(msg: str) -> str:
return call_my_llm(msg)使用 &(必须全部通过)和 |(必须通过其中任意一个)进行组合:
SafeAndPolite = Polite & ~MedicalAdvice & ~LegalAdvice2. 防篡改审计追踪
from semantix.audit.engine import AuditEngine
engine = AuditEngine()
# Every @validate_intent call with audit=True writes a hash-chained certificate.
engine.verify_chain() # True if no tampering每个证书记录已验证文本的哈希、意图、评判者身份和配置、判定结果、时间戳以及前一个证书的哈希。兼容 JSON-LD 工具和标准审计流水线。
3. 自愈式重试
失败时,semantix 会注入结构化反馈,以便 LLM 知道哪里出了问题:
from typing import Optional
@validate_intent(ResolutionPolite, retries=2)
def reply(msg: str, semantix_feedback: Optional[str] = None) -> str:
prompt = f"Reply to: {msg}"
if semantix_feedback:
prompt += f"\n\n{semantix_feedback}"
return call_llm(prompt)首次调用:semantix_feedback 为 None。重试时:它会收到一份包含分数、原因和被拒绝输出的 Markdown 报告。在三个意图类别中,测量到的可靠性从 21% 提高到了 70%。
4. 取证级 Token 归因
from semantix import ForensicJudge, QuantizedNLIJudge
judge = ForensicJudge(QuantizedNLIJudge())
# Verdict.reason: "Suspect tokens: [indemnify, forfeit, waive]"5. pytest 集成
from semantix.testing import assert_semantic
def test_chatbot_is_polite():
response = my_chatbot("handle angry customer")
assert_semantic(response, "polite and professional")失败时:
AssertionError: Semantic check failed (score=0.12)
Intent: polite and professional
Output: "You're an idiot for asking that."
Reason: Text contains aggressive language一流的 pytest 插件,带有 fixture、标记和 CI 报告:pytest-semantix。
框架集成
放入您现有的技术栈中——重试由每个框架原生处理。
DSPy
import dspy
from semantix.integrations.dspy import semantic_reward
qa = dspy.ChainOfThought("question -> answer")
refined = dspy.Refine(module=qa, N=3, reward_fn=semantic_reward(Polite))semantic_reward / semantic_metric 也可插入 dspy.BestOfN、dspy.Evaluate 和 MIPROv2 中——本地化,无 API 调用,每次评估约 15 毫秒。查看 benchmarks/ 以获取针对 LLM 评判者奖励函数的可复现比较。
from semantix.integrations.langchain import SemanticValidator
validator = SemanticValidator(Polite)
chain = prompt | llm | StrOutputParser() | validatorfrom pydantic_ai import Agent
from semantix.integrations.pydantic_ai import semantix_validator
agent = Agent("openai:gpt-4o", output_type=str)
agent.output_validator(semantix_validator(Polite))from guardrails import Guard
from semantix.integrations.guardrails import SemanticIntent
guard = Guard().use(SemanticIntent("must be polite and professional"))from semantix.integrations.instructor import SemanticStr
from pydantic import BaseModel
class Response(BaseModel):
reply: SemanticStr["must be polite and professional", 0.85]pip install "semantix-ai[mcp,nli]"
mcp run semantix/mcp/server.py任何支持 MCP 的代理(Claude Desktop、Cursor 等)都可以将意图验证作为工具使用。
- uses: labrat-akhona/semantic-test-action@v1
with:
test-path: tests/将语义测试报告发布为 PR 评论。
安装额外功能:pip install "semantix-ai[dspy]", "[langchain]", "[pydantic-ai]", "[guardrails]", "[instructor]", "[mcp]", "[all]"。
可插拔评判者
选择速度/准确性/推理的权衡:
from semantix import NLIJudge, EmbeddingJudge, LLMJudge, CachingJudge
@validate_intent(judge=NLIJudge()) # local, ~15 ms, deterministic
@validate_intent(judge=EmbeddingJudge()) # local, ~5 ms, similarity-based
@validate_intent(judge=LLMJudge(model="gpt-4o-mini")) # reasoning, ~500 ms, API
@validate_intent(judge=CachingJudge(NLIJudge(), maxsize=256)) # LRU-wrapped量化模式 (INT8 ONNX, ~25 MB, 无 PyTorch):
pip install "semantix-ai[turbo]"何时使用此工具
您正在运行一个由 LLM 支持的系统,该系统处理个人信息,并且需要一个可审计的验证步骤。
您正在优化 DSPy 程序,而 LLM 评判者奖励循环太慢、太昂贵或太不确定。
您需要在 pytest / CI 中进行语义测试断言,且不调用付费 API。
您处于受监管的行业(金融服务、保险、医疗保健),而“模型说没问题”不是一个可辩护的答案。
何时不使用此工具
您的验证意图需要多跳推理或世界知识(“这是否符合 2026 年税法第 4(b) 节”)。NLI 无法做到这一点;推理型 LLM 可以。
您需要评判者用散文解释“为什么”,而不仅仅是给出一个分数。
您每月评估的输出少于 100 个,且 LLM 作为评判者的延迟/成本无关紧要。
查看 semantix 的适用场景 以获取与 TruLens、DeepEval、Vectara HHEM、Guardrails、RAGAS 和 NeMo 的比较。
关键特性
本地推理 — NLI 模型在 CPU 上运行,数据不会离开您的机器。
确定性 — 相同的输入,相同的分数,每次运行,在每台机器上都一样。可设置种子。
快速 — 使用量化评判者,每次检查约 15-50 毫秒。
零 API 成本 — 验证不消耗 Token。
可审计 — 每次检查都有哈希链连接的 JSON-LD 证书。
经过充分测试 — 249 个测试,MIT 许可。
安装
pip install semantix-ai # Core (default NLI judge)
pip install "semantix-ai[turbo]" # Quantized ONNX (smallest footprint)
pip install "semantix-ai[openai]" # LLM judge (GPT-4o-mini)
pip install "semantix-ai[all]" # EverythingPyPI 上的包名称为
semantix-ai。导入方式为from semantix import ...。
贡献
查看 CONTRIBUTING.md 以获取开发设置、测试和提交指南。
许可
MIT — 查看 LICENSE。
Available Tools
1 toolverify_text_intentA
Check whether text satisfies a semantic intent using NLI.
Args:
text: The text to verify.
intent_description: What the text should convey.
threshold: Minimum entailment score to pass (0-1, default 0.5).
Returns:
JSON with score, passed, reason, and correction_suggestion on failure.
| Name | Required | Description | Default |
|---|---|---|---|
| text | Yes | ||
| threshold | No | ||
| intent_description | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided. Description reveals it returns JSON with score, passed, reason, and correction_suggestion, which implies inference without side effects. However, it does not explicitly state whether it's read-only or if it has any side effects, leaving some ambiguity.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Description is relatively concise, with parameter list and return format. Could be slightly more compact, but no wasted sentences.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Has output schema (though not shown), and description covers return fields adequately. Missing usage guidelines reduces completeness. For a simple 3-parameter tool, it's sufficient but not thorough.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema documentation coverage is 0%, so description must explain parameters. It does so: text is 'the text to verify', intent_description is 'what the text should convey', threshold is 'minimum entailment score' with default. This adds meaningful context beyond schema structure.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states 'Check whether text satisfies a semantic intent using NLI.' Specifies verb, resource, and method (NLI). No sibling tools, so differentiation is not needed.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives. No mention of prerequisites, limitations, or when not to use it. The description solely explains what it does without usage context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
1 tool update
- First observed
verify_text_intent
TDQS
Scored across 1 tool
With only one tool, there is no potential for confusion between tools. The single tool has a clearly distinct purpose.
Only one tool exists, so naming consistency is not an issue. The tool name 'verify_text_intent' follows a clear verb_noun pattern.
One tool is borderline appropriate for a focused verification utility. While minimal, it covers the core functionality without being trivial.
The tool effectively covers the primary task of text intent verification, including correction suggestions. Minor gaps exist (e.g., no batch verification or intent listing), but the set is complete for the stated purpose.
Maintenance
Related MCP Connectors
MCP server providing access to the Scorecard API to evaluate and optimize LLM systems.
MCP server for progressive tool usage at any scale (see https://klavis.ai)
MCP server for AI agents to plan, verify, and deploy Cloudflare-native apps.
Deterministic validation for AI-generated artifacts: JSON Schema, OpenAPI response, SQL syntax.
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceA lightweight MCP server that enables intelligent tool management and semantic search for APIs using sentence-transformers. It supports both REST and MCP interfaces across dual transport modes, allowing users to upload, manage, and query API tools with natural language.1MIT
- AlicenseAqualityDmaintenanceMCP server that validates LLM-generated tool-call arguments, lints tool definitions, and produces retry messages for AI assistants.325 npm1MIT
- AlicenseAqualityBmaintenanceMCP server for verifying AI agent claims vs reality — single-transcript inline grounding-check that flags when an agent's response states facts not in the input context, when its code silently swallows exceptions and substitutes mock data, or when its multi-turn transcript contains contradictions or unverified completion claims. Sub-second, local, free, no API calls.41MIT
- AlicenseAqualityCmaintenanceA local MCP server that packages LLM evaluation gates as reusable CI/CD primitives, enabling AI agents to run datasets against models, score responses, and enforce quality thresholds.10MIT