Skip to main content
Glama

LangSmith MCP Server (TypeScript)

License: MIT Node.js 18+

A TypeScript implementation of the Model Context Protocol (MCP) server for LangSmith. This is a full port of the official Python LangSmith MCP Server with 100% functional parity.

Example Use Cases

The server enables powerful capabilities including:

  • Conversation History: "Fetch the history of my conversation from thread 'thread-123' in project 'my-chatbot'" (paginated by character budget)

  • Prompt Management: "Get all public prompts in my workspace" / "Pull the template for the 'legal-case-summarizer' prompt"

  • Traces & Runs: "Fetch the latest 10 root runs from project 'alpha'" / "Get all runs for trace <uuid> (page 2 of 5)"

  • Datasets: "List datasets of type chat" / "Read examples from dataset 'customer-support-qa'"

  • Experiments: "List experiments for dataset 'my-eval-set' with latency and cost metrics"

  • Billing: "Get billing usage for September 2025"

Related MCP server: Orchestration MCP

Quick Start

LANGSMITH_API_KEY=your-key npx langsmith-mcp-server

Available Tools

The LangSmith MCP Server provides the following tools for integration with LangSmith.

Conversation & Threads

Tool Name

Description

get_thread_history

Retrieve message history for a conversation thread. Uses char-based pagination: pass page_number (1-based), and use returned total_pages to request more pages. Optional max_chars_per_page and preview_chars control page size and long-string truncation.

Prompt Management

Tool Name

Description

list_prompts

Fetch prompts from LangSmith with optional filtering by visibility (public/private) and limit.

get_prompt_by_name

Get a specific prompt by its exact name, returning the prompt details and template.

push_prompt

Documentation-only: how to create and push prompts to LangSmith.

Traces & Runs

Tool Name

Description

fetch_runs

Fetch LangSmith runs (traces, tools, chains, etc.) from one or more projects. Supports filters (run_type, error, is_root), FQL (filter, trace_filter, tree_filter), and ordering. All results are automatically paginated by character budget. Always pass limit and page_number.

list_projects

List LangSmith projects with optional filtering by name, dataset, and detail level (simplified vs full).

Datasets & Examples

Tool Name

Description

list_datasets

Fetch datasets with filtering by ID, type, name, name substring, or metadata.

list_examples

Fetch examples from a dataset by dataset ID/name or example IDs, with filter, metadata, splits, and optional as_of version.

read_dataset

Read a single dataset by ID or name.

read_example

Read a single example by ID, with optional as_of version.

create_dataset

Documentation-only: how to create datasets in LangSmith.

update_examples

Documentation-only: how to update dataset examples in LangSmith.

Experiments & Evaluations

Tool Name

Description

list_experiments

List experiment projects (reference projects) for a dataset. Requires reference_dataset_id or reference_dataset_name. Returns key metrics (latency, cost, feedback stats).

run_experiment

Documentation-only: how to run experiments and evaluations in LangSmith.

Usage & Billing

Tool Name

Description

get_billing_usage

Fetch organization billing usage (e.g. trace counts) for a date range. Optional workspace filter; returns metrics with workspace names inline.

Pagination (char-based)

Several tools use stateless, character-budget pagination so responses stay within a size limit and work well with LLM clients:

  • Where it's used: get_thread_history and fetch_runs.

  • Parameters: You send page_number (1-based) on every request. Optional: max_chars_per_page (default 25000, cap 30000) and preview_chars (truncate long strings with "... (+N chars)").

  • Response: Each response includes page_number, total_pages, and the page payload (result for messages, runs for runs). To get more, call again with page_number = 2, then 3, up to total_pages.

  • Why it's useful: Pages are built by JSON character count, not item count, so each page fits within a fixed size. No cursor or server-side state -- just integer page numbers.

Installation

From npm

npx langsmith-mcp-server

MCP Client Integration

Cursor / Claude Code

Add to your MCP settings:

{
  "mcpServers": {
    "langsmith": {
      "command": "npx",
      "args": ["langsmith-mcp-server"],
      "env": {
        "LANGSMITH_API_KEY": "your-key"
      }
    }
  }
}

Environment Variables

Variable

Required

Description

Example

LANGSMITH_API_KEY

Yes

Your LangSmith API key for authentication

lsv2_pt_1234567890

LANGSMITH_WORKSPACE_ID

No

Workspace ID for API keys scoped to multiple workspaces

your_workspace_id

LANGSMITH_ENDPOINT

No

Custom API endpoint URL (for self-hosted or EU region)

https://eu.api.smith.langchain.com

Notes:

  • Only LANGSMITH_API_KEY is required for basic functionality

  • LANGSMITH_WORKSPACE_ID is useful when your API key has access to multiple workspaces

  • LANGSMITH_ENDPOINT allows you to use custom endpoints for self-hosted LangSmith installations or the EU region

Development and Contributing

Setup

# Clone the repository
git clone https://github.com/langchain-ai/langsmith-mcp-server-js.git
cd langsmith-mcp-server-js

# Install dependencies
npm install

# Build
npm run build

# Run in development mode
LANGSMITH_API_KEY=your-key npm run dev

# Run production build
LANGSMITH_API_KEY=your-key npm start

Testing

# Run unit tests
npm test

MCP Inspector

For interactive development and debugging, use the MCP Inspector:

LANGSMITH_API_KEY=your-key npx @modelcontextprotocol/inspector npx .

This opens a browser UI where you can browse all tools, inspect their schemas, and invoke them interactively.

Verify the server responds

echo '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"test","version":"0.1.0"}}}' | LANGSMITH_API_KEY=test npx . 2>/dev/null

Architecture

This is a direct port of the Python LangSmith MCP Server with the same module structure:

src/
  index.ts                    # Entry point (stdio transport)
  server.ts                   # McpServer setup
  common/
    helpers.ts                # Client creation, data conversion utilities
    pagination.ts             # Char-based stateless pagination
    formatters.ts             # Message extraction and formatting
  services/
    register-tools.ts         # MCP tool registration with Zod schemas
    tools/
      prompts.ts              # Prompt management tools
      traces.ts               # Trace/run/project tools
      datasets.ts             # Dataset and example tools
      experiments.ts          # Experiment listing tools
      usage.ts                # Billing/usage REST API tools

Contributing

This TypeScript implementation is a community port of the official Python LangSmith MCP Server by LangChain.

Contributions are welcome! Please open an issue or pull request on GitHub.

License

This project is distributed under the MIT License. For detailed terms and conditions, please refer to the LICENSE file.

Available Tools

15 tools
create_datasetA

Call this tool when you need to understand how to create datasets in LangSmith. This is a documentation-only tool.

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the burden of behavioral disclosure. It states 'documentation-only tool,' which clearly indicates the tool does not perform mutations and likely returns instructional content. It does not describe return format, but for a zero-parameter docs tool this is sufficient.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is extremely concise at two sentences, front-loads the usage context, and contains no filler. Every word earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple documentation tool with no parameters and no output schema, the description is complete enough. It tells the agent exactly when to use it and what to expect, though it could briefly mention what topics the documentation covers.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool has zero parameters, so the schema fully covers everything. The description needs no parameter information, and the baseline for zero-parameter tools is 4.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose as a documentation resource for understanding dataset creation. It explicitly uses 'documentation-only' to distinguish itself from actual dataset creation tools, aligning with the resource and verb requirements.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It explicitly says 'Call this tool when you need to understand how to create datasets,' providing clear when-to-use context. However, it does not name specific alternatives or state when not to use it, but the 'documentation-only' hint implies exclusion of actual creation tasks.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

fetch_runsA

Fetch LangSmith runs from one or more projects with flexible filters and automatic pagination.

All results are paginated by character budget to keep responses manageable. Use page_number and total_pages from the response to iterate through multiple pages.

Args: project_name (str): The project name. For multiple projects, use JSON array string. limit (int): Max runs to fetch from LangSmith API (capped at 100). These runs are then paginated by character budget into pages. page_number (int): 1-based page index. Use with total_pages from response to iterate through pages. trace_id (str, optional): Return only runs that belong to this trace. run_type (str, optional): Filter by type: "llm", "chain", "tool", "retriever". error (str, optional): "true" for errored runs, "false" for successful. is_root (str, optional): "true" for only top-level traces. filter (str, optional): Filter Query Language (FQL) expression. trace_filter (str, optional): Filter applied to the root run. tree_filter (str, optional): Filter applied to any run in the trace tree. order_by (str, optional): Sort field; prefix with "-" for descending. Default "-start_time". reference_example_id (str, optional): Filter runs by reference example ID. max_chars_per_page (int): Max chars per page, capped at 30000. Default 25000. preview_chars (int): Truncate long strings to this length. Default 150.

ParametersJSON Schema
NameRequiredDescriptionDefault
errorNo"true" for errored runs, "false" for successful
limitYesMaximum number of runs to fetch from LangSmith API (capped at 100)
filterNoFilter Query Language (FQL) expression
is_rootNo"true" for only top-level traces
order_byNoSort field; prefix with '-' for descending-start_time
run_typeNoFilter by type: "llm", "chain", "tool", "retriever"
trace_idNoReturn only runs belonging to this trace UUID
page_numberNo1-based page index
tree_filterNoFilter applied to any run in the trace tree
project_nameYesThe project name to fetch runs from
trace_filterNoFilter applied to the root run in each trace tree
preview_charsNoTruncate long strings to this length
max_chars_per_pageNoMax character count per page, capped at 30000
reference_example_idNoFilter runs by reference example ID

TDQS

A4.5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It thoroughly explains automatic pagination by character budget, caps on 'limit' (100) and 'max_chars_per_page' (30000), defaults, and the format for multiple projects (JSON array). This is rich contextual detail beyond a simple 'fetch runs' statement.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is front-loaded with a concise summary and pagination note, followed by an organized Args list. It is lengthy due to 14 parameters but every sentence adds context (e.g., caps, defaults, usage hints). It does repeat schema descriptions somewhat, but it's structured efficiently for the complexity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given 14 parameters, no output schema, and no annotations, the description is quite complete. It covers all parameters, filters, pagination, and defaults. The only gap is the lack of detail about the structure of returned runs, but since the focus is on invocation, this is acceptable and still highly informative.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema already covers all 14 parameters with descriptions, so the baseline is 3. The description adds extra value by clarifying the multi-project JSON array format for 'project_name', how 'page_number' interacts with 'total_pages', and how 'limit' relates to pagination. This goes beyond schema information.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('Fetch LangSmith runs'), the resource ('runs'), and scope ('one or more projects'), with additional features ('flexible filters and automatic pagination'). This distinguishes it from sibling tools such as list_projects or list_experiments, which target different resources.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear context on how to use the tool, including pagination mechanics ('Use page_number and total_pages from the response to iterate through multiple pages'). However, it does not explicitly mention when to use this tool over alternatives or exclusions, so it falls short of a 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_billing_usageA

Fetch organization billing usage (trace counts) with workspace names inline.

Args: starting_on (str): Start of date range (ISO 8601) ending_before (str): End of date range (ISO 8601) workspace (str, optional): Optional workspace UUID or display name to filter on_current_plan (str): "true" to include only usage on current plan (default: "true")

ParametersJSON Schema
NameRequiredDescriptionDefault
workspaceNoOptional workspace UUID or display name to filter
starting_onYesStart of date range (ISO 8601)
ending_beforeYesEnd of date range (ISO 8601)
on_current_planNo"true" to include only usage on current plantrue

TDQS

A3.9/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description uses the verb 'Fetch' to imply a safe read-only operation, and mentions 'workspace names inline' as output format. It does not explicitly state non-destructive behavior, permissions, or pagination/limits, leaving some behavioral aspects undisclosed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise: a one-sentence purpose followed by a clean Args list. Every element is necessary and adds value, with no redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description provides the purpose and parameter details, but with no output schema, it only hints at the return format ('trace counts with workspace names inline') without specifying structure. It also does not cover pagination or edge cases, making it slightly incomplete for a complex fetch.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema covers all parameters with descriptions that exactly match the description's Args section (100% coverage). The description adds no additional meaning beyond the schema, so it meets the baseline of 3.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb 'Fetch' and a specific resource 'organization billing usage (trace counts)' with output detail 'workspace names inline'. This clearly differentiates it from sibling tools that deal with prompts, runs, datasets, and experiments.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description clearly implies this tool is for retrieving billing usage data, and the optional workspace parameter indicates filtering. However, it does not explicitly state when to use this over alternatives or mention any exclusions. Since no sibling tool handles billing, the context is clear, but explicit guidance is missing.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_prompt_by_nameA

Get a specific prompt by its exact name.

Args: prompt_name (str): The exact name of the prompt to retrieve

Returns: Dict containing the prompt details and template, or an error message

ParametersJSON Schema
NameRequiredDescriptionDefault
prompt_nameYesThe exact name of the prompt to retrieve

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the burden of conveying behavioral details. It explicitly states the return format ('Dict containing the prompt details and template') and the error behavior ('or an error message'). This discloses the key outcome and failure mode for a simple read operation, though it does not mention side effects (likely none) or authentication requirements.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences long, front-loaded with the main action, and uses a clear Args/Returns structure. It contains no fluff or redundant commentary; every line serves a purpose, making it highly concise and well-organized.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's simplicity — one parameter, no nested objects, no output schema — the description covers the essential aspects: what it does, the parameter, and the return/error behavior. It could be more explicit about when to use it relative to list_prompts (e.g., 'when you know the exact name'), but the context signals and description are sufficient for a straightforward retrieval tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% — both the schema and the description describe prompt_name identically ('The exact name of the prompt to retrieve'). The description's 'Args' section adds no new meaning beyond the schema, so it meets the baseline of 3 for high coverage without providing extra detail like examples, formatting, or edge-case behavior.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description starts with 'Get a specific prompt by its exact name', which uses a clear verb ('Get') and resource ('prompt') and emphasizes 'specific' and 'exact name'. This clearly distinguishes it from sibling tools like list_prompts (which lists prompts) and push_prompt (which creates/updates), and gives immediate clarity about its scope.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The phrase 'by its exact name' conveys the usage context: the user must know the precise prompt name to retrieve. It implicitly contrasts with list_prompts for enumeration, but it does not explicitly state 'use this when you need one prompt by name, use list_prompts to see all prompts'. Since the context is clear and there are no exclusions, this earns a 4 rather than a 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_thread_historyA

Retrieve one page of message history for a specific conversation thread.

Uses char-based pagination: pages are built by character budget (max_chars_per_page). Long strings are truncated to preview_chars. Supply page_number (1-based) on every call; use the returned total_pages to request further pages.

Args: thread_id (str): The unique ID of the thread to fetch history for project_name (str): The name of the project containing the thread page_number (int): 1-based page index (required) max_chars_per_page (int): Max character count per page, capped at 30000 (default: 25000) preview_chars (int): Truncate long strings to this length (default: 150)

Returns: Dict with result (list of messages), page_number, total_pages, etc.

ParametersJSON Schema
NameRequiredDescriptionDefault
thread_idYesThe unique ID of the thread to fetch history for
page_numberYes1-based page index (required)
project_nameYesThe name of the project containing the thread
preview_charsNoTruncate long strings to this length
max_chars_per_pageNoMax character count per page, capped at 30000

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden of behavioral disclosure. It explains the char-based pagination algorithm, the truncation of long strings via preview_chars, and the cap on max_chars_per_page at 30000. These are meaningful details beyond a simple 'returns messages' and give the agent a clear picture of how the tool behaves, though it does not explicitly state read-only status or error conditions.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured with a one-sentence summary, a focused explanation of pagination, and clearly labeled Args/Returns sections. It is appropriately sized for the tool's complexity, front-loads the main purpose, and avoids unnecessary verbiage.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given no output schema, the description includes a Returns section detailing the response shape (result, page_number, total_pages). It also covers the pagination contract and parameter constraints, making it sufficient for an agent to call the tool correctly. Minor gaps remain for error cases, but the essentials are present.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3. The description adds value by explaining the 'character budget' concept that ties max_chars_per_page to page construction, and clarifies the truncation behavior of preview_chars. This gives the agent a deeper understanding than the schema's basic field descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a clear, specific statement of what the tool does: 'Retrieve one page of message history for a specific conversation thread.' This uses a specific verb ('retrieve'), defines the resource ('message history'), and notes the single-page scope, which distinguishes it from sibling tools like list_prompts or fetch_runs.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides explicit pagination guidance—'Supply page_number (1-based) on every call; use the returned total_pages to request further pages'—which communicates the intended usage pattern. However, it does not explicitly mention when to avoid this tool or suggest alternatives, though the context implies it is the appropriate tool for fetching thread history.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_datasetsA

Fetch LangSmith datasets.

If no arguments are provided, all datasets will be returned.

Args: dataset_ids (str, optional): Dataset IDs as JSON array string or single ID data_type (str, optional): Filter by dataset data type (e.g., 'chat', 'kv') dataset_name (str, optional): Filter by exact dataset name dataset_name_contains (str, optional): Filter by substring in dataset name metadata (str, optional): Filter by metadata as JSON object string limit (int): Max number of datasets to return (default: 20)

ParametersJSON Schema
NameRequiredDescriptionDefault
limitNoMax number of datasets to return
metadataNoFilter by metadata as JSON object string
data_typeNoFilter by dataset data type
dataset_idsNoDataset IDs as JSON array string or single ID
dataset_nameNoFilter by exact dataset name
dataset_name_containsNoFilter by substring in dataset name

TDQS

A3.6/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description bears the burden. It discloses the default limit of 20 and the filtering capabilities, but lacks details on pagination, sorting, or output structure. For a read-only list tool, this is acceptable but not rich in behavioral disclosure.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured with a one-sentence purpose followed by a clear Args list. It is detailed but not verbose, with every line providing useful information. The front-loaded purpose makes it easy to scan.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool has 6 optional parameters, no output schema, and is a straightforward listing operation, the description is fairly complete. It covers the default behavior, filtering options, and limit, which is sufficient for an agent to invoke it correctly, though it could mention pagination if applicable.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema already covers all parameter descriptions (100% coverage), but the description adds meaningful detail beyond the schema: 'dataset_ids' as JSON array string, 'dataset_name' as exact match, 'dataset_name_contains' as substring, and an example for 'data_type'. This enriches understanding of how to use the parameters.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool 'Fetch LangSmith datasets' and explains the default behavior of returning all datasets when no arguments are provided. This distinguishes it from sibling tools like list_prompts or list_experiments, though it doesn't explicitly name alternatives.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives context about when no arguments returns all datasets, implying a listing use case. However, it does not explicitly state when to prefer this tool over siblings like read_dataset or list_examples, nor does it mention any exclusions or prerequisites.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_examplesA

Fetch examples from a LangSmith dataset with advanced filtering options.

Either dataset_id, dataset_name, or example_ids must be provided.

Args: dataset_id (str, optional): Dataset ID to retrieve examples from dataset_name (str, optional): Dataset name to retrieve examples from example_ids (str, optional): Example IDs as JSON array string or single ID filter (str, optional): Filter string using LangSmith query syntax metadata (str, optional): Metadata filter as JSON object string splits (str, optional): Dataset splits as JSON array string or single split inline_s3_urls (str, optional): "true" or "false" include_attachments (str, optional): "true" or "false" as_of (str, optional): Dataset version tag or ISO timestamp limit (int): Max examples to return (default: 10) offset (str, optional): Number of examples to skip

ParametersJSON Schema
NameRequiredDescriptionDefault
as_ofNoDataset version tag or ISO timestamp
limitNoMaximum number of examples to return
filterNoFilter string using LangSmith query syntax
offsetNoNumber of examples to skip
splitsNoDataset splits as JSON array string or single split
metadataNoMetadata filter as JSON object string
dataset_idNoDataset ID to retrieve examples from
example_idsNoExample IDs as JSON array string or single ID
dataset_nameNoDataset name to retrieve examples from
inline_s3_urlsNo"true" or "false"
include_attachmentsNo"true" or "false"

TDQS

A3.8/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the burden. It states that one of three parameters must be provided, which is a behavioral constraint, but it does not disclose read-only status, auth requirements, or pagination behavior. The word 'Fetch' implies read-only, though not explicitly.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is structured with a summary line, a requirement line, and a parameter list. It is clear and front-loaded, but somewhat redundant with the schema as every parameter description is duplicated, adding length without much new information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With 11 parameters and no output schema, the description covers all inputs but does not explain the return format, pagination, or error behavior. The 'advanced filtering options' are merely listed without further explanation, leaving some gaps for a complex tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The description repeats the parameter list, which matches the schema's 100% coverage. It adds the requirement that one of dataset_id, dataset_name, or example_ids must be provided, which is useful beyond the schema. No additional syntax or format details are given beyond the schema descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states 'Fetch examples from a LangSmith dataset with advanced filtering options' with a specific verb and resource. It also clarifies that one of dataset_id, dataset_name, or example_ids must be provided. This distinguishes it from sibling tools like list_datasets or read_example by focusing on examples and filtering.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for retrieving multiple examples with filtering and requires at least one identifier, but it does not explicitly compare itself to alternatives like read_example for single-example retrieval or mention when not to use it. It provides context but no clear exclusions or alternatives.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_experimentsA

List LangSmith experiment projects (reference projects) with mandatory dataset filtering.

Requires either reference_dataset_id or reference_dataset_name.

Args: reference_dataset_id (str, optional): Dataset ID to filter experiments by reference_dataset_name (str, optional): Dataset name to filter experiments by limit (int): Maximum number of experiments to return (default: 5) project_name (str, optional): Filter by name (partial match)

ParametersJSON Schema
NameRequiredDescriptionDefault
limitNoMaximum number of experiments to return
project_nameNoFilter projects by name using partial matching
reference_dataset_idNoThe ID of the reference dataset to filter experiments by
reference_dataset_nameNoThe name of the reference dataset to filter experiments by

TDQS

A3.6/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description must convey all behavioral traits. It mentions the mandatory dataset filter and partial matching, but it does not describe return format, error behavior when no filter is supplied, or whether this is a read-only safe operation. This is a significant gap for a tool that may filter results and return potentially large data.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is efficiently structured: a one-sentence purpose, a constraint line, and an Args list. It is not bloated, though the Args list duplicates the schema's parameter descriptions, which is slightly redundant in an MCP context.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given that there is no output schema and no annotations, the description is only partially complete. It explains the purpose and parameters well but omits return value behavior, possible errors (e.g., what if no dataset filter is provided), and how this relates to sibling tools. For a simple list tool, it is adequate but not thorough.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema covers all parameters at 100% coverage, establishing a baseline of 3. The description adds the 'mandatory filter' constraint that is not captured in the schema properties, which is useful. However, it also repeats the parameter descriptions verbatim, adding little new meaning beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('List') and the resource ('LangSmith experiment projects (reference projects)'). It also distinguishes itself from sibling tools like list_projects by specifying 'experiment projects' and the mandatory dataset filtering, making its purpose specific and unique.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It provides clear contextual usage by stating 'Requires either reference_dataset_id or reference_dataset_name', which tells the agent when this tool is applicable. However, it does not explicitly compare against alternatives or state when not to use it, stopping short of a full usage guide.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_projectsB

List LangSmith projects with optional filtering and detail level control.

Args: limit (int): Maximum number of projects to return (default: 5) project_name (str, optional): Filter projects by name (partial match) more_info (str): "true" for full details, "false" for simplified (default: "false") reference_dataset_id (str, optional): Filter by reference dataset ID reference_dataset_name (str, optional): Filter by reference dataset name

ParametersJSON Schema
NameRequiredDescriptionDefault
limitNoMaximum number of projects to return
more_infoNo"true" for full details, "false" for simplifiedfalse
project_nameNoFilter projects by name using partial matching
reference_dataset_idNoFilter by reference dataset ID
reference_dataset_nameNoFilter by reference dataset name

TDQS

B3.3/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It notes optional filtering and detail control but does not disclose read-only nature, pagination, sort order, response shape, or any side effects. The description mostly restates parameter purposes without adding behavioral context.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Description is compact and front-loaded with the action. The Args list is efficient, each line provides necessary parameter information, and there is no redundant prose.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

All parameters are covered, but the description omits details about the return value, pagination, or when to prefer sibling tools. Given the absence of annotations and output schema, this is adequate but incomplete for a list operation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% and the description's argument list mirrors the schema descriptions almost verbatim. It adds no new meaning beyond the schema but does present parameters in a readable format. Baseline of 3 is appropriate since the schema documents everything.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description clearly states the tool's function: "List LangSmith projects" with a specific verb and resource. It also mentions optional filtering and detail control, distinguishing it from sibling list tools like list_prompts or list_datasets by resource name.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when to use this tool versus alternatives such as list_experiments or list_datasets. There is no mention of prerequisites, exclusions, or comparison with sibling tools. Usage is only implied by the tool name and basic description.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_promptsB

Fetch prompts from LangSmith with optional filtering.

Args: is_public (str): Filter by prompt visibility - "true" for public prompts, "false" for private prompts (default: "false") limit (int): Maximum number of prompts to return (default: 20)

Returns: Dict with prompts and metadata

ParametersJSON Schema
NameRequiredDescriptionDefault
limitNoMaximum number of prompts to return
is_publicNoFilter by prompt visibility - "true" for public prompts, "false" for private promptsfalse

TDQS

B3.1/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden of behavioral disclosure, but it only states 'Fetch prompts' and returns 'Dict with prompts and metadata.' It does not disclose pagination behavior, ordering, auth requirements, or any side effects. For a read operation, this minimal detail is a significant gap.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single sentence followed by Args and Returns sections. It is concise and front-loaded, though the Args section duplicates schema info, which is slightly redundant. Still, it avoids waste and maintains a clear structure.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has a low-complexity schema but no output schema. The description only gives a vague return type ('Dict with prompts and metadata') without explaining the structure, pagination, or filtering semantics. Given the lack of annotations and output schema, more contextual detail is needed for correct invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema provides 100% coverage of both parameters with descriptions and defaults. The description's Args section simply reiterates the schema information without adding new meaning, so it does not compensate beyond the existing schema coverage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states 'Fetch prompts from LangSmith with optional filtering' which clearly identifies the operation (fetch/list) and resource (prompts). It distinguishes from siblings like get_prompt_by_name (which fetches a single prompt) by implying a list operation with optional filters.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives such as get_prompt_by_name or other list tools. It mentions optional filtering but does not state exclusions or alternative references, leaving the agent without decision-making context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

push_promptA

Call this tool when you need to understand how to create and push prompts to LangSmith.

This is a documentation-only tool that explains how to:

  • Create prompts using LangChain's prompt templates

  • Push prompts to LangSmith for version control and management

  • Handle prompt creation vs. version updates

Use the LangSmith Client's push_prompt() method. See LangSmith documentation for details.

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of disclosure. It transparently states that it is documentation-only, which is a critical behavioral trait since the name 'push_prompt' could imply an actual push. It also lists the topics it explains. It does not contradict any annotations because none exist, and it communicates the safe, non-mutating nature of the tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is reasonably concise, using a short introductory sentence, a clear statement about being documentation-only, and a bulleted list of topics. The final instruction to see LangSmith documentation is useful context. It is well-structured and front-loaded with the most important information, though the last sentence is slightly redundant.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a zero-parameter, no-output-schema documentation tool, the description is complete. It fully explains the tool's purpose, when to use it, and what specific topics it covers, even pointing to external documentation for further details. There is no ambiguity about what the agent will get from invoking this tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool has zero parameters, so the schema is trivially complete. The description adds value by outlining what content the documentation covers, which is more useful than parameter explanations. Since the baseline for zero parameters is 4, and the description enhances understanding of the tool's functionality, a 4 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool is a documentation-only resource for understanding how to create and push prompts to LangSmith. It uses specific verbs like 'understand' and 'explains,' and distinguishes itself from sibling tools that actually perform operations like listing or fetching prompts.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly says 'Call this tool when you need to understand how to create and push prompts,' providing a clear usage condition. It also outlines the topics covered, which helps the agent decide if this tool matches its need. However, it does not explicitly mention when NOT to use it or name alternative sibling tools, but the context is clear for a documentation tool.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

read_datasetA

Read a specific dataset from LangSmith.

Either dataset_id or dataset_name must be provided.

Args: dataset_id (str, optional): Dataset ID to retrieve dataset_name (str, optional): Dataset name to retrieve

ParametersJSON Schema
NameRequiredDescriptionDefault
dataset_idNoDataset ID to retrieve
dataset_nameNoDataset name to retrieve

TDQS

A3.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It discloses only the read action and the either/or requirement, but does not mention return format, error handling, or behavior when both parameters are provided. This is a significant gap for a tool with zero annotation coverage.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise, with a clear first sentence and an Args section. Every sentence/line serves a purpose, with no redundant information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple read tool with no output schema and no annotations, the description adequately covers inputs and the either/or condition, but it omits any mention of what the tool returns. This is a moderate gap, though the simplicity of the tool partially compensates.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% for both parameters. The description adds value beyond the schema by explicitly stating the either/or constraint, which is not present in the schema itself.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states 'Read a specific dataset from LangSmith' with a specific verb and resource. It distinguishes from siblings like list_datasets (which lists all datasets) and read_example (which reads an example).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage by specifying 'Either dataset_id or dataset_name must be provided.' This gives clear context on how to invoke the tool but does not explicitly mention alternatives or when-not-to-use compared to sibling tools.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

read_exampleA

Read a specific example from LangSmith.

Args: example_id (str): Example ID to retrieve as_of (str, optional): Dataset version tag or ISO timestamp

ParametersJSON Schema
NameRequiredDescriptionDefault
as_ofNoDataset version tag or ISO timestamp
example_idYesExample ID to retrieve

TDQS

A3.8/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description must convey safety and side effects. It states 'Read', implying a non-mutating operation, but does not disclose error behavior, return format, or permissions. The as_of parameter is described but not its behavioral implications. For a simple read, this is borderline adequate.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is appropriately brief and front-loads the core purpose in the first sentence. The Args list is redundant with the schema but is not verbose and maintains a clean docstring structure without waste.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple retrieval tool, the description is largely complete: it states what it does and documents both parameters. The lack of an output schema is mitigated by the straightforward 'Read' semantics, though it does not explicitly describe the return payload. Overall, the context is sufficient for an agent to invoke it correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema covers both parameters fully (100% coverage), and the description's Args section replicates the same descriptions without adding new meaning. As a result, the description adds no semantic value beyond the schema, warranting the baseline score.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb ('Read') and specifies the resource ('a specific example from LangSmith'), clearly distinguishing it from sibling tools like list_examples which enumerate examples. The scope is unambiguous and immediately actionable.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The intended use is implied: you need an example_id to retrieve a specific example. However, it does not explicitly state when to prefer this over list_examples or mention any alternatives or exclusions. The guidance is present but not explicit.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

run_experimentA

Call this tool when you need to understand how to run experiments and evaluations in LangSmith. This is a documentation-only tool.

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

TDQS

A4.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries full responsibility and does an excellent job by explicitly stating 'documentation-only tool.' This discloses that the tool will not execute experiments or produce side effects, which is a critical behavioral trait for an agent to know.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two concise sentences that front-load the when-to-use instruction and immediately clarify the documentation-only nature. Every word earns its place with no fluff or repetition.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a zero-parameter, no-output-schema documentation tool, the description is completely sufficient. It fully communicates the purpose and scope, and the sibling tools provide enough context for the agent to distinguish this from action-oriented tools.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool has 0 parameters, so per guidelines the baseline is 4. The description appropriately says nothing about parameters because there are none to explain.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly identifies the tool as a documentation resource for understanding how to run experiments and evaluations in LangSmith. It explicitly says 'documentation-only tool,' which distinguishes it from sibling tools that perform actual actions like listing or creating resources.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides an explicit instruction: 'Call this tool when you need to understand how to run experiments and evaluations.' This states exactly when to use the tool, making the usage condition unambiguous.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

update_examplesA

Call this tool when you need to understand how to update dataset examples in LangSmith. This is a documentation-only tool.

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the transparency burden. It clearly discloses that the tool is 'documentation-only', indicating it has no side effects and does not modify data. However, it does not elaborate on what the documentation will contain or how it is delivered.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences with no redundant words. The first sentence states the trigger, the second clarifies the tool's nature. Every word contributes to understanding.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple documentation-only tool with no parameters and no output schema, the description adequately covers purpose and behavioral nature. It could specify more about the documentation content, but it is sufficient for an agent to decide when to invoke it.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool has zero parameters, so the baseline for this dimension is 4. The description adds no parameter information, but none is needed since the input schema is empty.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: to help the agent understand how to update dataset examples in LangSmith. It uses specific language ('understand how to update dataset examples') and distinguishes this documentation tool from sibling tools that actually perform operations like list_examples or read_example.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It explicitly specifies when to use the tool ('Call this tool when you need to understand how to update dataset examples'). The phrase 'documentation-only' provides an implicit exclusion (do not use for actual updates), but it does not name alternative tools or provide explicit when-not guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 15 tool updatesv0.1.7
    • First observedcreate_dataset
    • First observedfetch_runs
    • First observedget_billing_usage
    • First observedget_prompt_by_name
    • First observedget_thread_history
    • First observedlist_datasets
    • First observedlist_examples
    • First observedlist_experiments
    • First observedlist_projects
    • First observedlist_prompts
    • First observedpush_prompt
    • First observedread_dataset
    • First observedread_example
    • First observedrun_experiment
    • First observedupdate_examples

TDQS

A3.5/5.0

Scored across 15 tools

Disambiguation4/5

Most tools have distinct purposes (list vs. get vs. read vs. fetch), but some pairs like list_prompts/get_prompt_by_name and list_datasets/read_dataset could be confused without carefully reading descriptions. The documentation-only tools (push_prompt, create_dataset, etc.) are clearly different in role but still add a bit of ambiguity.

Naming Consistency3/5

Tool names use a consistent snake_case verb_noun pattern, but the verbs are not uniform: list_*, get_*, fetch_*, read_* are used interchangeably (e.g., list_projects vs. fetch_runs, get_prompt_by_name vs. read_dataset). This mixed style is readable but not fully consistent.

Tool Count4/5

15 tools is at the upper boundary of the recommended range. The count is reasonable for a platform like LangSmith, but 4 of the tools are documentation-only, which reduces the effective functional tool count. Still, the scope is not excessive.

Completeness2/5

The tool surface is heavily read-oriented: prompts, runs, projects, datasets, examples are mostly list/get/fetch. Write operations like creating datasets or running experiments are only documentation tools, not actual operations. Missing update/delete for prompts and datasets, and no way to create examples. Significant gaps for workflow completion.

Maintenance

ActivityInactive
ResponsivenessWithin a week

Related MCP Connectors

Related MCP Servers

  • A
    license
    C
    quality
    A
    maintenance
    MCP server for Langfuse observability. Query traces, debug exceptions, analyze sessions, and manage prompts and datasets for your LLM applications.
    48
    3,403 PyPI
    105
    MIT
  • F
    license
    A
    quality
    D
    maintenance
    A TypeScript MCP server for launching, tracking, and managing external coding-agent runs across local and remote backends like Codex and Claude Code. It allows top-level agents to orchestrate subagents through tools for spawning tasks, polling events, and handling interactive sessions.
    7
    2
    -
  • A
    license
    A
    quality
    C
    maintenance
    Comprehensive MCP server for Langfuse, enabling AI assistants to access and manage traces, observations, scores, datasets, and sessions for observability.
    24
    13 npm
    1
    MIT