Leave Copilot — From MCP to a Dedicated Agentic Model

A runnable reference implementation: use MCP to standardize a set of leave/attendance tools that are deliberately made hard, use Google ADK to build an Agent that can operate them, use dual evaluation to measure the base model's shortcomings, then fine-tune a dedicated model that natively excels at operating this tool set.

This is the companion code for the 2026 iThome Ironman 30-day series.

Why the tools are "deliberately made hard"

API design usually pursues intuitiveness and ease of use, but this project needs the opposite.

The acceptance criterion is comparing performance before and after fine-tuning — if the tools are too intuitive, the base model can already call them correctly, accuracy starts near perfect from the beginning, and fine-tuning naturally shows no improvement. That's not because fine-tuning has no effect, but because there's simply no room for improvement.

So there's only one selection criterion: the base model almost certainly gets it wrong, and fine-tuning can teach it.

Four deliberately implanted difficulties

The common trait of these four difficulties: they are all rules that JSON Schema cannot express. Schema can constrain status to be one of four strings, but it cannot constrain "where this ID comes from."

Related MCP server: MCP Leave Management

Quick Start

Environment

The first item is especially easy to get wrong, because the MCP Python SDK official site shows documentation for a different API by default (MCPServer), which is completely different from the FastMCP style used here.

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

Start the MCP Server

python -m mcp_server.server        # streamable-http on 127.0.0.1:8090

Verify the four difficulties

python eval/verify_difficulties.py

It will actually connect to the Server, trigger each of the four difficulties one by one, and check the error messages and return semantics:

難點 ① 跨呼叫依賴
  ✓ 捏造的編號被擋下
  ✓ 錯誤訊息指出正確取得方式
  ✓ search_leaves 查得到真實編號
…
難點 ④ 參數陷阱
  ✓ 餘額欄位以小時命名
  ✓ 傳姓名而非員工編號被擋下
難點 ② Elicitation 三態
  ✓ accept            → cancelled
  ✓ decline           → aborted
  ✓ cancel            → aborted
  ✓ accept_but_false  → aborted
  ✓ 撤銷後時數退回餘額
  ✓ decline 的訊息明確禁止繞道
  ✓ cancel 的訊息與 decline 語意不同

──────────────────────────────────────────────
19/19 通過

Reset test data

update_leave_status and cancel_approved_leave will actually change data. You must reset before every evaluation round; otherwise the second round's preconditions differ from the first round's, and the results are not comparable.

python eval/reset.py

Tool Set

Nine tools, plus one admin endpoint for the evaluation script.

readOnlyHint is not just documentation — the evaluation tool uses it to compute "read-only compliance": whether the Agent used write tools during read-only tasks.

_reset_fixtures must be excluded on the Agent side using tool_filter. A tool named "reset" has an inexplicable appeal to LLMs.

Why error messages are written so clearly

The tool's error messages are returned verbatim to the model, becoming the basis for its next step.

# ✗ 模型只知道錯了，得猜哪裡錯
raise ValueError("Invalid status transition")

# ✓ 模型知道錯在哪、也知道該改成什麼
raise ValueError(
    f"狀態不可從 {current} 跳至 {target}，下一個合法狀態為 {next_valid}"
)

This is a typical technique of using tool design to compensate for model capability, and the cost is just a few extra words.

Project Structure

.
├── mcp_server/          # ✅ MCP Server：九個工具 + 四個難點
│   ├── server.py
│   ├── store.py         #    模擬資料層
│   └── fixtures.py      #    初始資料與 reset
├── eval/                # ✅ 驗證與重置腳本
│   ├── verify_difficulties.py   # 19/19
│   ├── verify_agent.py          # 架構驗證 10/10
│   └── reset.py
├── agents/leave_copilot/ # ✅ Google ADK Agent（含 elicitation callback）
├── plugins/             # ⏳ 軌跡記錄與生產防禦 Plugin
├── data/                # ⏳ 軌跡萃取與資料擴增
├── training/            # ⏳ SFT 訓練腳本
└── deploy/              # ⏳ 權重合併、量化、vLLM 部署

✅ Completed and tested　⏳ Under construction

Port Allocation

⚠️ FastMCP and Google ADK api_server both default to port 8000; one of them must be changed. This project moves the MCP Server to 8090.

Related Projects

• ADEval — Google ADK Agent evaluation tool (Apache-2.0)

• Twinkle Eval — Standard Benchmark evaluation (MIT)

License

Apache-2.0

Verified Parts

eval/verify_difficulties.py and eval/verify_agent.py have both been actually run, not just "written that way in the docs."

MCP Server 層（eval/verify_difficulties.py）              19/19
  四個難點的錯誤訊息、Elicitation 四條路徑

Google ADK 層（eval/verify_agent.py，A 段架構驗證）        10/10
  McpToolset 載入、tool_filter 排除管理端點
  accept / decline / cancel / accept-but-false 四條路徑
  都確認走到 Client callback，且語意正確回報

Environment: mcp 1.29.1 + google-adk 2.7.1 + gemini-3.7-flash.

Base model behavior observations

Section B of verify_agent.py makes no assertions, only records — the model answering incorrectly doesn't mean the test failed; that's exactly what's being measured. The most notable failure mode observed in actual runs is this:

The model responds with text instead of calling tools. Faced with a destructive operation, gemini-3.7-flash tends to ask "are you sure?" in the conversation itself, rather than calling cancel_approved_leave to let the Server issue an Elicitation. The result: the confirmation flow drops from the protocol layer back to the conversation layer, and confirmation at the conversation layer has no enforcement power.

A more severe variant is hallucinated success — the model replies "I've submitted the leave request for approval," but the tool sequence contains no update_leave_status at all, and the leave request status hasn't changed. The user thinks it's done.

This kind of failure cannot be eliminated with prompts, because it stems from the model's built-in tendency toward "safety." This is exactly what fine-tuning will address later.