Skip to main content
Glama

MCP Eval デモ

評価を使って、LLM エージェントが MCP サーバーを実際に効果的に使えるかどうかを検証する方法の実例です。サーバーのコードが正しいかどうかだけを確認するのではありません。

ユニットテストは「delete_note はノートを削除するか?」に答えます。しかし、MCP サーバーが実践で役立つかどうかを左右する以下の問いには答えられません。

  • ユーザーが ID ではなく言葉でノートを説明したとき、エージェントは正しいノートを見つけられるか?

  • 一覧のプレビューが途中で切れていることに気づくか、それともノートの半分だけをもとに回答するか?

  • update_note が上書きすると理解しているか、それとも「買い物リストに行を追加して」と頼まれてユーザーの内容を黙って破壊してしまうか?

  • エラーメッセージから回復できるか、それとも諦めてしまうか?

これらはツールサーフェス——名前、説明、スキーマ、結果の形、エラーテキスト——の性質であり、確認する唯一の方法は、実際のエージェントをサーバーに対して実行し、その行動を採点することです。それがこのリポジトリの目的です。

ステータス

MCP サーバー、そのインフラストラクチャ、そして eval ハーネスはすべて配置済みです。

Related MCP server: notes-mcp-server

テスト対象のサーバー: Notes MCP

インメモリのノートブックです。状態はサーバープロセス内に保持され、終了時に破棄されるため、各 eval 実行は同じ既知のコーパスから開始されます(seed.py を参照)。

ツール

動作のヒント

機能

create_note

書き込み

ノートを作成します。タイトルは大文字小文字を区別せず一意である必要があります。

get_note

読み取り専用

id 指定で 1 つのノートの完全な内容を返します。

list_notes

読み取り専用

ノートを更新が新しい順に、省略されたプレビューとして一覧表示します。任意の部分文字列 query に対応します。

update_note

破壊的

ノートのタイトルや内容を上書きします。

delete_note

破壊的操作

ノートを完全に削除します。

eval が検出できるようにするために意図された設計上の決定がいくつかあります。

  • ID であり、タイトルではない。 すべての変更系ツールは note_id を受け取るため、「私の買い物リスト」を変更するように依頼されたエージェントは、まず ID を調べる必要があります。ここでエージェントは推測をすることがよくあります。

  • 切り詰められたプレビュー。 list_notes は各ノートの最初の 120 文字のみを返し、content_truncated と content_length でフラグを立てます。一覧からそのまま内容の質問に答えるエージェントは間違えます。優れたエージェントは get_note を呼び出します。

  • 追加ではなく置き換え。 update_note は上書きします。したがって、「私の買い物リストに卵を追加して」は読み取り、変更、書き込みの操作であり、読み取りをスキップするエージェントはデータを破壊します。

  • 教えるエラー値。 すべての失敗は、問題の値を示し、それを解決するツールを指し示すため、エージェントは行き詰まるのではなく、前に進む道筋を得られます。

構成

src/notes_mcp/
  models.py    Pydantic models — also the tool input/output schemas the agent sees
  store.py     In-memory storage and its error types
  seed.py      Fixed corpus: stable ids and timestamps, so evals are reproducible
  server.py    MCP tool definitions, descriptions, and annotations
  cli.py       `notes-mcp` entry point
evals/
  agent.py       Builds the pydantic-ai agent under test + local trace capture
  task.py        One agent turn against a freshly seeded server — the thing evaluated
  evaluators.py  Custom pydantic-evals evaluators (tool-not-called, argument-contains)
  cases.yaml     The dataset itself: cases that probe specific MCP misuse patterns
  cases.py       Loads cases.yaml — registers the custom evaluators, picks the judge model
  __main__.py    `python -m evals` — runs the dataset against a live model
tests/
  test_store.py    Unit tests for the storage layer
  test_server.py   Protocol-level tests through a real MCP client session
scripts/
  lint.sh    Ruff + pyright + format check
  test.sh    Unit + protocol tests (fast, free)
  evals.sh   Agent-behaviour evals against a live model (slow, costs money)

ツールの説明は、インラインの docstring ではなく server.py 内のモジュールレベルの定数です。説明の文言は、失敗した eval に対応して調整する主な要素であり、これを 1 か所にまとめることで diff が読みやすくなります。

はじめに

uv と Python 3.12 が必要です(.python-version で固定)。

uv sync                       # create .venv and install everything
uv run scripts/test.sh        # unit + protocol tests
uv run scripts/lint.sh        # ruff check, pyright (strict), format check
uv run pre-commit install     # optional: run the same checks on commit

サーバーの実行

uv run notes-mcp                        # stdio, seeded with the sample notes
uv run notes-mcp --empty                # stdio, no notes
uv run notes-mcp --transport streamable-http

.mcp.json は、このプロジェクトの stdio サーバーを登録します。このディレクトリから起動された MCP ホストグレード——たとえば Claude Code——は、notes サーバーを自動的に認識し、手動で操作できます。

エージェントが表示するツールサーフェス(これらの eval が実際に検証しているもの)をエージェントを起動せずに確認するには、次のようにします。

uv run fastmcp list .mcp.json                   # names, signatures, descriptions
uv run fastmcp list .mcp.json --input-schema    # ...with the full JSON schemas
npx @modelcontextprotocol/inspector uv run notes-mcp   # MCP Inspector, for clicking around

mcp バージョンについて: サーバーは、mcp SDK に mcp.server.fastmcp として含まれていたコピーではなく、独立した FastMCP ライブラリ上に構築されています。mcp 2.0 でそのモジュールは削除されました。FastMCP が依存する mcp のバージョンを決定するため(3.x は mcp 1.x を解決します)、pyproject.toml には手書きの mcp バージョン指定はありません。eval ハーネスは反対側から同じライブラリに到達します:pydantic-ai の MCP クライアントは FastMCP の Client 上に構築されています。したがって、このリポジトリの両方の半分は、誰かがメンテナンスしなければならない固定ではなく、構造的にバージョンが一致します。FastMCP 4 は、両方を mcp 2.x に移行するステップであり、これが依存関係がその下に制限されている理由です。

テストアプローチ

scripts/test.sh によって実行される 2 つの pytest レイヤー:

  • test_store.py は、ストレージのセマンティクス(一意性、並べ替え順、制限、タイムスタンプ)をカバーします。高速で網羅的、プロトコルの関与はありません。

  • test_server.py は、インプロセスの MCP クライアントセッション(fastmcp.Client 経由、FastMCP のインメモリトランスポート上)でサーバーを駆動するため、エージェントが実際に受信する内容、つまりツールリスト、 JSON スキーマ、動作アノテーション、構造化結果、エラーテキストを検証します。プロトコルは実物です。サブプロセスとソケットだけが仮想的です。

非同期テストは、pytest-asyncio ではなく anyio の pytest プラグインを使用します。これは、MCP クライアントがセッションの存続期間中キャンセルスコープを開いたままにし、anyio がフィクスチャのセットアップとティアダウンを同じタスクで実行するためです。

2 番目のタイプのチェック、つまりエージェント動作 eval は、実在のモデルを呼び出し、コストがかかるため、pytest スイートには含まれません。専用のランナーとスクリプトがあります。以下で説明します。

eval ハーネス

evals/ は、最小限の pydantic-ai エージェント(汎用的な 1 行のシステムプロンプト、 数ショットの例なし、特殊化された指示なし)を構築し、その唯一のツールを pydantic_ai.mcp.MCPToolset 経由でインプロセスの Notes MCP サーバーに接続します(agent.py)。システムプロンプトは意図的に最小限です。これらの eval は、サーバー自身のツール名、説明、スキーマが正しい動作を導くのに十分かを確認するためのものであり、プロンプトエンジニアリングが弱いものを隠せるかどうかを確認するためではありません。

evals/ は src/ の下ではなく最上位レベルにあります。これは、このリポジトリの開発ツールであり、誰もインストールしない notes-mcp パッケージの一部ではありません。

pydantic_evals は、このエージェントを Case の Dataset で実行します。各 Case は、このファイルの先頭にある 4 つの動作のうちの 1 つをターゲットにしています。

Case

チェック内容

delete_by_description_looks_up_the_id_first

「私の買い物リストのノートを削除して」という要求されたとき、エージェントは delete_note の前に list_notes を呼び出し、正しい ID を削除します。

answers_past_the_list_notes_preview_cutoff

答えが list_notes プレビューから切り詰められる質問は、エージェントが get_note を呼び出した場合のみ正しく回答されます。

appending_to_a_note_preserves_its_truncated_tail

「私の買い物リストにクラッカーを追加して」は、最初に完全なノートを読む必要があります。update_note 呼び出しは、プレビュー上限を超えてのみ存在するテキストを呼び許されるかをチェックされます。

title_conflict_on_create_is_not_silently_lost

既に存在するタイトルのノートを作成する場合、新しい内容を黙って失ったり、重複が作成されたと主張してはなりません。

deleting_a_nonexistent_note_does_not_fabricate_success

存在しないノートの削除要求は、推測した ID での delete_note 呼び出しや、成功を主張する応答をしてはなりません。

simple_lookup_answers_from_the_right_note

正気を確認するためのハッピーパス。

ケースは Python ではなく cases.yaml に存在します。ケースはデータであるため、ケースを追加したりルーブリックを言い換えたりしてもコードは変更されません。cases.py はローダーにすぎません。カスタム評価を Dataset.from_file に渡します(YAML ファイルはローダーが登録した評価器のみを指定できます)。また、判定モデルを設定します。YAML の yaml-language-server ヘッダーは cases_schema.json をファイルを指しているため、エディタは評価器名とその引数を補完して検証できます。カスタム評価器を追加または変更した後に再生成します:

uv run python -c "from evals.cases import write_json_schema; print(write_json_schema())"

評価器は、pydantic-evals の組み込み保存される (ToolCorrectness、 Contains、 MaxToolCalls、 LLMJudge は複数の有効な回復がある 2 つのケース用) と evaluators.py の 2 つの小さなカスタム評価器を組み合わせます: ToolNotCalled (ツールが一度も呼び出されていないことを低信憑 — 組み込みの否定チェックはありません)と ArgumentContains(ツールの引数に対する部分文字列チェックで、「古いコンテンツが残っている必要があります」ケース用。LLM の正確な表現は等価性やサブセット辞書のマッチングでは固定できないたため)。両方とも組み込みと同様に、Agent.instrument_all() とローカルの (send_to_logfire=False) 構成し、logfire.configure() がキャプセスするツールコールスパンを読み取ります。agent.py の configure_instrumentation() を参照。

__main__.py はデータセットを実行し、完全なレポートを出力します。タスクエラー、クラッシュした評価器、または失敗したアサーションがあれば、非ゼロで終了します。次のコマンドで実行します:

uv run scripts/evals.sh

プロバイダーの設定

NOTES_MCP_EVAL_MODEL は、プロバイダーとモデルの両方を、pydantic-ai の provider:model 文字列として選択し、デフォルトは anthropic:claude-haiku-4-5-2025100 です。.env.example を .env にコピーし、この 3 つのうちいずれかを使用しているセクションに入力してください。scripts/evals.sh は .env を自動的に読み込みます(python-dotenv 経由なので、シェルで既に設定されている変数はオーバーライドされません)。また、.env は gitignore されます。

  • Anthropic API(デフォルト)— ANTHROPIC_API_KEY が必要です。

  • OpenAI — NOTES_MCP_EVAL_MODEL=openai:gpt-5 および OPENAI_API_KEY が必要です。

  • Amazon Bedrock — NOTES_MCP_EVAL_MODEL=bedrock:<bedrock-model-id>。boto3 の通常の認証情報チェーンを通じて認証されるため、標準の AWS SDK 変数以外に eval 固有の設定はありません。名前付きプロファイルを使用するには AWS_PROFILE を設定します(AWS_DEFAULT_REGION も同様、そのプロファイルがまだリージョンを設定していない場合。boto3 のリージョン解決がチェックしない AWS_REGION ではなく、AWS_DEFAULT_REGION である必要があります)。または、両方を未設定のままにして、デフォルトのプロファイル/リージョンを使用します。

プロバイダに応じてコードは分岐しません。eval_model() の文字列はエージェントと LLMJudge の両方に直接渡され、pydantic-ai の infer_model は、指定されたプロバイダープレフィックスの適切なクライアントと認証情報を決済します。

Available Tools

5 tools
create_noteA

Create a new note and return it, including the generated id to use in later calls.

Titles must be unique case-insensitively; creating a note whose title already exists is an error, so if the user wants to change an existing note, use update_note.

NameRequiredDescriptionDefault
titleYesShort, distinctive title. Must be unique.
contentNoBody of the note. May be empty.

Output Schema

NameRequiredDescription
idYesStable identifier for the note.
titleYesShort human-readable title.
contentYesFull note body.
created_atYesWhen the note was created (UTC, ISO 8601).
updated_atYesWhen the note was last modified (UTC, ISO 8601); equals `created_at` for a note that has never been updated.

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the annotations (readOnlyHint=false, destructiveHint=false), the description reveals the result shape (returns note with generated id), a critical error condition (case-insensitive title collision), and a routing fallback. This adds real behavioral context without contradiction.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, the most important return behavior up front, followed by the duplicate-title warning and alternative. Each sentence adds unique operational value; there is no padding.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With only two parameters, one required, and a full output schema, the description covers everything an agent needs: what the tool does, what it returns, key constraints, and the correct sibling for modification. Nothing essential is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents both parameters well. The description adds only the case-insensitive uniqueness nuance beyond the schema's existing 'Must be unique' note. That is small extra value, consistent with the baseline 3 for high coverage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb-resource pair ('Create a new note') and clearly states the distinguishing behavior: returning the generated id for later calls. It also contrasts with update_note by pointing out that duplicate titles are errors, making the tool's role unmistakable among its siblings.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly names when not to use this tool ('if the user wants to change an existing note, use update_note') and clearly flags the uniqueness constraint that should silence spurious create attempts. This gives the agent direct decision support.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

delete_noteA
Destructive

Permanently delete the note with the given id and return what was removed.

This cannot be undone and the note's content is not recoverable, so confirm the id with list_notes or get_note before calling when the user identified the note by description rather than by id.

NameRequiredDescriptionDefault
note_idYesId of the note to delete.

Output Schema

NameRequiredDescription
idYesIdentifier of the note that was deleted.
titleYesTitle of the note that was deleted.
deletedYesAlways true; a missing note is an error, not a false here.

TDQS

A4.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

While annotations already set destructiveHint=true, the description adds critical context: the deletion is permanent, content is not recoverable, and it returns what was removed. This warns about irreversibility and the need for confirmation beyond the annotation's binary hint.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences, front-loaded with the core action, and every clause earns its place. The caution and verification guidance are placed logically without waste.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool is simple (one parameter, no nested objects, has output schema). The description covers what the tool does, the consequence, and the necessary verification pre-step. Nothing an agent needs to call it correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema covers the only parameter note_id with a clear description 'Id of the note to delete'. Since schema description coverage is 100%, the baseline is 3. The description refers to the parameter as `id` but adds no new format or semantic detail beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb 'delete' with a specific resource 'note' and the return behavior 'return what was removed'. It distinguishes from siblings like create, get, list, update by naming the destructive action unambiguously.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives explicit guidance on when to use this tool: confirm the id with list_notes or get_note before calling when the note was identified by description. This directly addresses potential misuse and surfaces the appropriate alternative behavior.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_noteA
Read-onlyIdempotent

Read one note in full by id.

Use this whenever you need a note's complete body — list_notes previews are cut off after 120 characters. Ids come from list_notes or create_note; this tool does not accept titles.

NameRequiredDescriptionDefault
note_idYesId of the note, as returned by `list_notes`.

Output Schema

NameRequiredDescription
idYesStable identifier for the note.
titleYesShort human-readable title.
contentYesFull note body.
created_atYesWhen the note was created (UTC, ISO 8601).
updated_atYesWhen the note was last modified (UTC, ISO 8601); equals `created_at` for a note that has never been updated.

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already establish read-only, idempotent, non-destructive behavior. The description adds useful context about return behavior (full note body) and explicitly documents a limitation ('does not accept titles'), which goes beyond the annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is compact and front-loaded. The first sentence conveys the core purpose, and the second sentence gives the exact context needed to avoid common mistakes without unnecessary elaboration.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

A single-parameter read operation with annotations, a full output schema, and clear sibling differentiation. The description covers all functional details needed to invoke the tool correctly; nothing important is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% and the parameter description already says 'Id of the note, as returned by list_notes.' The tool description reinforces that note_id comes from list_notes or create_note and that titles won't work, but does not add significant meaning beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific action: 'Read one note in full by `id`.' It names the resource and the mechanism, and the first sentence identifies the tool as the way to get a note's complete body, distinguishing it from list previews.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly says to use this tool whenever a note's complete body is needed, notes that list_notes previews are truncated at 120 characters, and clarifies that IDs come from list_notes or create_note and that titles are not accepted. This provides clear when-to-use and when-not-to-use guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_notesA
Read-onlyIdempotent

List notes, most-recently-updated first, as previews.

Pass query to filter to notes whose title or body contains that text (case-insensitive substring, not fuzzy or semantic — prefer one distinctive word over a whole phrase, and drop the filter entirely to browse everything).

Each entry carries id, title, preview, content_truncated, content_length, created_at, and updated_at. total_count is the number of matches, which exceeds the returned notes when limit cut the page short.

NameRequiredDescriptionDefault
limitNoMaximum notes to return.
queryNoCase-insensitive substring filter on title and body. Omit to list every note.

Output Schema

NameRequiredDescription
notesYesMatching notes, newest-updated first. Previews only — use `get_note` for full content.
total_countYesNumber of notes matching the query, which may exceed `len(notes)` when `limit` truncated the page.

TDQS

A4.3/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

In addition to the annotations (read-only, idempotent, non-destructive), the description adds meaningful behavior: ordering of results, preview semantics, an explicit explanation of the substring behavior, and clarity about total_count exceeding returned results when limit truncates the page. This is exactly the kind of behavioral context the description should provide.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is three concise paragraphs with a front-loaded summary, a filter guidance section, and a brief output-shape section. Every sentence contributes necessary behavioral or usage detail, and the structure makes it easy for an agent to parse.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description covers the core operation, ordering, filtering behavior, returned entry fields, total_count semantics, and limit-pagination behavior. Combined with a rich output schema and the read-only annotations, virtually everything an agent needs to invoke this tool correctly and interpret the default behavior is present.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema already covers both parameters thoroughly: limit defines maximum notes with defaults and bounds, query explains the case-insensitive substring filter and omission. The description restates some of this guidance but does not add significant new meaning for the parameters; its advice about using one distinctive word is minor beyond the schema's coverage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states a specific action ('List notes') and a distinctive resource, and also specifies the ordering (most-recently-updated first) and output shape (previews), which precisely distinguishes it from the single-record sibling get_note. It doesn't explicitly call out the sibling, but listing versus retrieving a single note is semantically self-evident.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives concrete usage guidance for the query filter: it explains that the filter matches case-insensitive substrings, warns that it is not fuzzy or semantic, and counsels preferring one distinctive word over a phrase. It also says to omit the filter to browse everything, but it doesn't explicitly contrast list_notes with get_note for retrieving a single full note.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

update_noteA
Destructive

Overwrite the title and/or content of an existing note and return the updated note.

This REPLACES whatever you pass; it does not merge. Omit a field (or pass null) to leave it alone. To add to a note's body without losing the current text, call get_note first and pass the full combined text as content.

NameRequiredDescriptionDefault
titleNoReplacement title, or null to keep the current one.
contentNoReplacement body — overwrites, does not append. Null keeps the current body.
note_idYesId of the note to change.

Output Schema

NameRequiredDescription
idYesStable identifier for the note.
titleYesShort human-readable title.
contentYesFull note body.
created_atYesWhen the note was created (UTC, ISO 8601).
updated_atYesWhen the note was last modified (UTC, ISO 8601); equals `created_at` for a note that has never been updated.

TDQS

A4.5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the annotations (readOnlyHint=false, destructiveHint=true), the description clearly discloses the destructive overwrite behavior: it replaces, does not merge, and explains the semantics of omitting or passing null. It also flags the risk of losing current text and instructs the agent on the safe pattern (get_note first). No contradiction with annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is compact and front-loaded with the core function. Every sentence serves a distinct purpose: state the operation and return value, warn about overwrite semantics, explain null/omit behavior, and provide the append workflow. No redundancy or filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the output schema is present, return values are already documented. The description covers the essential behavioral nuance (no merge, omission semantics) and the critical alternative workflow for append. An agent has everything necessary to call the tool correctly without external context.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% and the schema already explains that null keeps the current value for both fields. The description adds the clarification that omitting a field also leaves it alone, which is a small extra, but overall it does not materially extend the schema's parameter documentation.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (Overwrite), resource (title and/or content of an existing note), and outcome (return the updated note). This clearly distinguishes the tool from its siblings list_notes, get_note, create_note, and delete_note.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides clear context for use: modifying an existing note by overwriting fields. It also gives explicit guidance for the alternative of appending content, telling the agent to call get_note first and pass the full combined text. However, it does not explicitly state exclusions such as 'use create_note for new notes,' though this is implied by 'existing note.'

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 5 tool updatesv0.1.0
    • First observedcreate_note
    • First observeddelete_note
    • First observedget_note
    • First observedlist_notes
    • First observedupdate_note

TDQS

A4.6/5.0

Scored across 5 tools

Disambiguation5/5

Each tool maps to one distinct resource action (create, get, list, update, delete), with no overlapping operations. The descriptions reinforce boundaries by explaining when to use get_note over list_notes and create_note over update_note.

Naming Consistency5/5

All tool names follow the same verb_note pattern: create_note, get_note, list_notes, update_note, delete_note. list_notes is pluralized because it returns a collection, but the convention is otherwise uniform and predictable.

Tool Count5/5

Five tools form a tightly scoped set for a notes server: one create tool, one read tool, one list tool, one update tool, and one delete tool. Each tool earns its place and the count is ideal for the domain.

Completeness5/5

The tool surface covers the full note lifecycle with no dead ends: create, list, read by id, update, and delete. The update tool's replace-only behavior is mitigated by explicit guidance to call get_note first, so agents have a complete workflow.

Maintenance

ActivityStale
ResponsivenessNo recent community issues

Related MCP Connectors

Related MCP Servers

  • -
    license
    Not graded
    quality
    Not graded
    maintenance
    A simple notes system that allows creating, storing, and accessing text notes through MCP resources and tools, with built-in prompt support for generating summaries of stored notes.
    -
  • A
    license
    A
    quality
    B
    maintenance
    Provides cross-session persistent memory for coding agents via MCP tools to store, retrieve, and manage notes with hybrid keyword/semantic search and automatic deduplication.
    8
    340 npm
    MIT
  • A
    license
    Not graded
    quality
    C
    maintenance
    Enables AI assistants to manage in-memory notes by listing, fetching, and creating notes through MCP tools over stdio.
    104 npm
    MIT