需求 → 数据资产 智能匹配系统

FastAPI · BM25 + vector hybrid retrieval · cross-encoder re-ranking · agentic closed loop with a calibratable adjudicator · MCP

The system accepts natural-language requirement descriptions, requirement screenshots, or Excel files of any arbitrary layout, matches usable tables in the data asset catalog, and outputs recommended solutions with field-level justification.

The matching process is an agentic closed loop, not a one-way call. After each retrieval round, the adjudicator evaluates every candidate asset's relevance to the requirement (relevant / partially relevant / irrelevant); when the threshold is not met, the system autonomously picks a corrective action based on diagnostic conclusions drawn from the intermediate retrieval state and re-retrieves. If the catalog truly contains no asset that can support the requirement, it outputs a cannot be supported verdict with evidence rather than filling in near-matches. Users may after the returned results, and the system continues to iterate while retaining all prior adjudication context.

智能体闭环 (Agent Loop)

自我审阅 (Self-Review)

After each retrieval round, the adjudicator determines the relevance of each candidate to the requirement (relevant / partial / none), the code aggregates these verdicts into a score and compares it against the threshold; if the threshold isn't met, the system enters corrective mode.

match（检索）→ judge（逐条判相关性）→ 达标？
                                      ├ 是 → 交付
                                      └ 否 → 选纠偏动作 → 重新检索 → 再判

自主纠偏：

Correction actions come from a fixed list; the permission boundary is enforced by the code, not by "prompt self-discipline":

Opening the adjudico threshold is equivalent to letting the model lower the acceptance bar so that results "pass" and nullifying the cannot be supported verdict. It is therefore not whitelisted and is instead constrained by tests.

归因有据 (Evidence-Based Attribution)

Before correction, the code computes an evidence-backed diagnostic conclusion from the intermediate retrieval state and files it to the adjudicator. The same symptom can often have two causes requiring opposing remedies; the role of the diagnosis is to distinguish those two:

多轮协作 (Multi-Round Collaboration)

• 解析后核对 (Post-parse confirm): The extracted results are but returned, then confirmed by the user before retrieval starts — the cheapest correction point sits at the very front of the chain.

• 结果后追问 (Post-result follow-up): When the user raises correction comments, the agent continues with all previous context, without resetting the conversation.

• 会话持久化 (Session state): Session state is persisted to disk; session_id remains valid after a service restart.

可审计、可中断 (Auditable and Interuptible)

Every step logs action / reason / verdict / effect (the action taken, the rationale, the literal verdict text, and the before/after delta) and returns them with the result. The first action is always a full retrieval, so any interruption still yields a deliverable; the worst case is equivalent to pure retrieval. Long-running tasks support cooperative cancellation.

判决器须先标定 (The Adjudicator Must First Be Calibrated)

Before entering the automated decision chain, the adjudicator must be measured against a manual gold standard for its agreement with the true ordering of candidate quality (Cohen's κ), and κ > 0.6 is required before it any automated use. After changing the model, the prompt, or the scoring rules, calibration must be redone.

Related MCP server: MCP RAG Agent Server

自主优先级推荐

Relevance and asset priority (ordered status, asset type, data-warehouse tier are on different dimensions; directly blending them needs manual grade, and weights become stale as the data distribution shifts. The system instead uses adaptive natural-break bins + within-bin lexicographic order; bin width is data-determined:

1. Estimate the just-noticeable difference (JND): Take the median ρ of the positive gaps in the relevance list sorted descending for this query, answering "how large is the typical score gap for this query's candidates?".

2. Cut at natural breaks: From the current highest score extend downward until a significant gap of gap > gap_factor × ρ appears and only then cut. Tight clusters (near-ties) are not split — within such a cluster the order falls to system priority; a marked gap gives relevance control.

3. Hard order within a bin: Select by lexicographic comparison over [priority worst-first vector, relevance, coverage, fewer tables]. For a combination plane, the bin is the lowest bin among its members, so a single high-tier table cannot inflate lower-tier members.

Bin width adapts to the data, not to a preset vocabulary of query shapes — “what the user asks” assigned to hard-coded if-else is the same jitter as manual hand weights.

Priority levels are synthesized lexicographically from "is already ordered + asset type + warehouse tier"; every mapping is from the config:

# config/config.yaml → matching.ranking.priority
ordered_rank: 6         # 已订购（由订购清单 vlookup 标记，最高档）
asset_type_rank: {...}  # 资产类型 → 档位
layer_only_types: [...] # 仅这些类型内部再按数仓层级细分
layer_rank: {DWA: 5, DIM: 5, DWV: 4, DWI: 3, ODS: 2, STG: 1}

# config/config.yaml → matching.layering
gap_factor: 2.0         # 间隙达 JND 几倍算断层：越大同档越宽，越偏向优先级
min_jnd: 0.02           # JND 绝对下限，仅兜底分数全等 / 单点等退化情形

其它能力

快速开始

python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -r requirements.txt

# 可选：安装后启用语义召回 / 精排 / 本地大模型；未安装则自动降级为词法检索 + 规则解析
pip install sentence-transformers
pip install torch --index-url https://download.pytorch.org/whl/cpu

python -m uvicorn src.api.app:app --host 127.0.0.1 --port 8001

Web UI at http://127.0.0.1:8001. API call usage:

# 一步式：解析 → 检索 → 自我审阅 → 交付
curl -X POST http://127.0.0.1:8001/api/agent/run \
  -F "text=分析会员复购行为，需要用户ID、订单数、支付金额"

# 多轮：对上一轮结果提出修正意见，携带历史继续纠偏
curl -X POST http://127.0.0.1:8001/api/agent/resume \
  -F "session_id=<上一轮返回的 session_id>" \
  -F "feedback=当前结果不符合预期，需要履约时效相关的资产"

The example catalog ships with the repo (in data/samples/; 3000 tables covering transaction, user, product, fulfillment, market, service, risk, risk, content, location, finance — ten business domains), with no extra data prep; it also runs okay on a CPU-only environment.

The first call builds retrieval indices and loads the models — expecting a minute or two on CPU for 3000 tables; the index persists to data/.cache/, and afterwards hits content-digest cache so a restart doesn't rebuild.

换成自有数据

The system does not hard-code assumptions about what type of object it can "recommend". As long as the data can be organized into "entry + attribute list + description", it works — data tables, doc repos, products, APIs, knowledge entries all fit. No change is needed for retrieval, ranking, or the agentic bone.

1. 必换：目录数据

Get a CSV in the same structure (sample: data/samples/tassets.sample.csv):

| 列 (Column) | 作用 (Role) | 等效于 (Equivalent object) | |---|---| | <code> | 唯一标识 | 文档 ID / 商品 / API 名 | | 表名 | Display name + matches | 文档标题 / 商品名 | | 字段中文名 | attribute list (|-分割), the basis of field-level matching | 文档章节 / 商品 attribute 平台 / API 参数 | | 业务分类名称 | used for segment-encoding | own category scheme (一级/二级) | | 模型概述 | description text, primary basis of semantic recall | summary/简介 | | 技术口径 | supplementary note | rule-of-thumb / update freqency | | 周期 / 层级 / 数据域 | the three hard filters | any tri enum dimensions; leave empty if don't filters | | 是否上架 / 资产类型 | business priority + filter | fixed value when no direct analog | | 系统名称 / 库名 / barangon-line 是否UCX | ownership and flags | fixed value acceptable |

# config/config.yaml
data:
  assets_file: data/自有目录.csv      # 默认按 gb18030 读取

Default source encoding is assumed gb18030. For UTF-8 files either transcode or edit the read encoding in src/data_loader.py — that is the only encoding-coupled point in the codebase.

Enum values must agree with the ranking logic. The 资产类型 / 层级 / 周期 columns join priority ranking and hard filter; each new value must be in the mapping list or the row hits the lowest rank or is never filtered in:

After you replace with your own catalog, either add your values to the mapping tables above or normalize the data to existing values — pick one. 数据域 must keep a consistent granularity; cofferels and smaller intermingle cause domain filtering to miss semantics. The repo's sample catalog already abides by this.

ordered_rank (is-ordered) depends on manifest by data.ordered_file; this sample config does not ship the manifest so the order rank can't trigger; it will take effect once you supply your own ordered list.

2. 建议更换：四份领域词表 (Recommended: four domain vocabularies, directly change quality)

Only the synonyms file's location is via data.synonyms_file; the other three routes are hard-coded in pipeline.py and normalize.py — just replace contents in place. The four in this repo are bootstrap-size only (8 synonym imaples, 64 categories list, 67 terms), expand for your own business.

The synonym list is its best value. Format is {"规范词": ["别名1", "别名2"]} — group the same concept under the catalog spelling used back, there are dozens of high-frequency concepts to quickly matter.

3. 可选：替换解析提示词中的示例

src/matching/extractor.py and/or so-called src/matching/llm_parser.py, the prompt sample legs define "what is a need, what a field". These examples are domain samples no decision branch; but with a very different field, two-to-three use your domain to raise largely parse quality.

4. 更换后必须重新标定 (Re-calibrate)

• 语义阈值： field_match.retrieve_min_sim / min_ce are model dependent and shift with the embedding/domain lexicon. Pick about a dozen "truly synonymous" and "uncorrelated" fields from your domain, and split the two distributions at the score mean.

• 判决器 κ：著 aversion uses domain semantics — oracle numbers from another domain don't transfer. Measure Cohen's κ on a domain-labeled set again still > 0.6 before automated use.

• 评测集 (eval set) the repo doesn't include a gold test; for your data it's a must to build this baseline, cross-domain numbers are not comparable.

三种接入 (三种接口)

• Web (UI): single-page at /: describe → confirm draft → generate plan → the exporter of excel; repeat our edits and table cancellation.

• REST: /api/agent/run (one-shot), /api/agent/resu (multi-round follow-up), /api/atch (pure retrieval), /api/parse + confirm (two-step); file-containing requests get automatically converted to async tasks with cancellation support.

• MCP: The /cp API exposes six tools (parse_queries / revise_draft / match_assets / refine_results / submit_feedback / list_ast_facets) + async tasks + collaborative cancellation, even in a conversation platform auto-detect.

Full API contract: docs/API接口文档.md.

文档 (Docs)

工程原则 (Engineering Principles)

• Let the semantic decisions live in the model, mechanical segmentation in the code; do not hard-code assumptions about various requirement doc layout; make no doc-form assumption.

• Do not degrade silently: truncators, dis, dis missed, and framing are all reported to /api/health / the teletry.

• An adjudicator must be calibratied before it enters an auto decisions-chain. Any enab ing line is a config change only, code unchanged.

• Every bug fixed must be paired with a regression test and its original root cause recorded.

说明 (Note)

This repository provides reusable methods and an engineering backbone. The bundled catalog is a synthetic, general-purpose sample (3000 tables, aligned in structure and statistical distribution with actual production catalogs); phrase/vocabulary word lists and evaluation sets are shipped fully after desensitization. No business-specific catalog is included — swap in your own and run.