dify-eval-mcp
Server Configuration
Describes the environment variables required to run the server.
| Name | Required | Description | Default |
|---|---|---|---|
| DIFY_API_KEY | No | Your Dify API key (app-xxx) for live mode. If not set, the server runs in mock offline mode. | |
| DIFY_BASE_URL | No | The base URL of your Dify instance, used for live mode. If not set, the server runs in mock offline mode. |
Instructions
Guidance the server publishes about itself, which clients place ahead of the tool catalog so the model reads it before choosing anything.
This server publishes no instructions, or was last inspected before Glama recorded them.
Capabilities
Features and capabilities supported by this server
Protocol revision2025-11-25
| Capability | Details |
|---|---|
| tools | {
"listChanged": false
} |
| prompts | {
"listChanged": false
} |
| resources | {
"subscribe": false,
"listChanged": false
} |
| experimental | {} |
Tools
Functions exposed to the LLM to take actions
| Name | Description |
|---|---|
| run_evalB | 跑一批评测用例。mock=True(默认)离线运行,不需要 Dify 实例;live 模式用 DIFY_BASE_URL/DIFY_API_KEY 环境变量 |
| compare_runsB | 对比两次运行结果(竞品对比 / 版本回归)。输入是 run_eval 的 json_out 产物 |
| export_reviewA | 从运行结果导出人工盲评 CSV(自动判定留空,避免锚定效应) |
| import_reviewA | 把人工评审 CSV 合并回运行结果;传多份 CSV 即众包(多数投票 + 一致率) |
| list_eval_setsA | 扫描目录里的评测集(.yaml/.yml),返回名称、用例数、是否可 mock 离线跑 |
Prompts
Interactive templates invoked by user choice
| Name | Description |
|---|---|
No prompts | |
Resources
Contextual data attached and managed by the client
| Name | Description |
|---|---|
No resources | |
TDQS
Scored across 5 tools
Each tool has a distinct, non-overlapping purpose: running evaluations, comparing runs, exporting/importing human reviews, and listing evaluation sets. The paired export/import tools are clearly separated by their direction of data flow, and run_eval vs list_eval_sets are unambiguously different (execution vs discovery).
All tool names follow a consistent verb_noun pattern in snake_case: run_eval, compare_runs, export_review, import_review, list_eval_sets. The verbs are all lowercase and descriptive, and the nouns are appropriate. The slight singular/plural variation (eval vs eval_sets, runs) is semantically justified and does not break the pattern.
Five tools is well-scoped for the domain of Dify evaluation workflows. Each tool covers a distinct step in the lifecycle (discover, run, compare, review), without unnecessary overlap or bloat. This is within the ideal 3-15 range and feels purpose-built rather than padded.
The tool surface covers the core evaluation workflow: listing eval sets, running them, comparing results, and handling human reviews. The only noticeable gap is the lack of a way to list or inspect existing run outputs, which an agent would need to know prior runs for comparison. This is a minor dead end that can be worked around if run artifacts are known or passed explicitly.