Tracing a ChatGPT app into LangSmith

A ChatGPT app is an MCP server. ChatGPT runs the conversation and the reasoning, your server runs the tools. On each tool call ChatGPT sends your server the tool name, the arguments, and a conversation id. It does not send the user's message, its own reasoning, or the answer it writes afterwards.

This example traces every tool call into LangSmith as one run, with the arguments and the result, and groups the calls from one conversation into a thread. The tools themselves are not changed.

How it works

ChatGPT sends _meta with every tool call. Two fields matter, per the Apps SDK reference.

The MCP Python SDK opens an OpenTelemetry span for every request. That span has the tool name and nothing else. langsmith_tracing.py adds a middleware that runs inside the span for each tools/call and sets the attributes LangSmith reads, from its OpenTelemetry attribute table.

A standard OTLP exporter sends the spans to LangSmith. Same route as any agent that already emits OpenTelemetry, so it fits an existing collector, and with it any masking or export to a second backend.

Two details the middleware handles.

The SDK also spans initialize, tools/list, and server/discover. ChatGPT sends those constantly and they carry no conversation id, so the exporter drops everything except tools/call.

A tool that fails returns a normal result with isError: true. LangSmith marks a run as failed from an exception event, so the middleware adds one carrying the error text ChatGPT received. Without it a failed tool call looks successful.

That text is all there is. SDK 2.x masks a raised exception as Error executing tool <name> before it leaves the server, so the reason is on your own stderr and not on the trace. 1.x appends the exception message. Neither is something the middleware can recover, which is the second reason a tool should return a miss rather than raise on one.

Inputs and outputs are chat messages rather than the raw argument and result JSON, which is a requirement of thread-level evaluation, covered below. Two readers need different things from those messages, so the user turn carries the customer's own words followed by the arguments that were actually sent, and the assistant turn carries the tool's own return value rather than the MCP envelope around it. A judge gets the intent, and anyone debugging still sees the parameters. The untouched argument and result JSON is on the run as tool_arguments and tool_result metadata either way.

Related MCP server: LangSmith MCP Server

Run it

cp .env.example .env        # add your LangSmith API key
uv sync
uv run server.py            # terminal 1, the MCP server on http://127.0.0.1:8765/mcp
uv run simulate_chatgpt.py  # terminal 2, calls the server the way ChatGPT does
uv run verify_langsmith.py  # reads the runs back, grouped by thread

To drive the same server with a real model instead of the simulator, which is how to check what a model writes into customer_request, add the harness extra and give it model credentials.

uv sync --extra harness
uv run model_harness.py

simulate_chatgpt.py plays two conversations from two users, several tool calls each, with the _meta ChatGPT would send. The second conversation asks for an id that does not exist, which comes back as a normal result, and then sends a malformed date, which is the one run that shows as an error.

What you see in LangSmith

One run per tool call, named after the tool, carrying latency and status. Its inputs are the customer's request followed by the arguments that were sent, and its outputs are the tool's return value, both as chat messages for the reason given above. The untouched argument and result JSON is on the run as tool_arguments and tool_result metadata. Open the Threads view and each conversation is one thread with its tool calls in order.

On MCP Python SDK 1.x

OpenAI's Apps SDK examples use 1.x (from mcp.server.fastmcp import FastMCP), which 2.x removed, so an app started from them is pinned to mcp<2.

1.x has no middleware and no built-in OpenTelemetry. The v1/ folder does the same job by wrapping the server's CallToolRequest handler. One span per tool call, the same attributes, the same threads, the same error marking.

cd v1 && uv sync && uv run server_v1.py

Run the simulator from the repo root, not from v1/, in a second terminal.

uv run simulate_chatgpt.py

v1/ pins mcp<2, where from mcp import Client does not exist, so the simulator only runs against the root environment. The server it talks to can be either version.

Do not use the Mcp-Session-Id HTTP header as the conversation key. It was removed in the 2026-07-28 MCP specification. openai/session is the field ChatGPT provides for this.

Tool responses shape what the agent can say

None of the three tools raises on a miss. search_holidays does not return a bare empty list when nothing matches, and get_holiday and check_availability do not raise on an id that does not exist. Each returns why it missed and what it does have.

{
  "results": [],
  "count": 0,
  "no_match_reason": "No Malaga package matched because the cheapest is \u00a31105, above the \u00a3900 ceiling.",
  "closest": {"id": "HOL-3305", "destination": "Malaga", "nights": 7, "price_gbp": 1105, "month": "October"}
}

{
  "found": false,
  "requested_id": "HOL-9999",
  "no_match_reason": "No holiday has the id HOL-9999. Ids look like HOL-1041.",
  "available_ids": ["HOL-1041", "HOL-2210", "HOL-3305", "HOL-4120"]
}

An empty list gives the model nothing, so the customer gets a dead end. With a reason and a near miss the model can explain the gap and offer the alternative. This matters more for a ChatGPT app than for an agent you run yourself, because the tool response is the only thing you control once the conversation is on OpenAI's side. It is also visible on the trace, so the quality of these responses is something you can evaluate rather than guess at.

A malformed argument is the case that should still raise. check_availability rejects a departure_date that is not an ISO date, because that is a broken call rather than an answer, and the trace is more useful when a failed run means something went wrong instead of meaning a customer asked for something we do not sell.

Evaluating a whole conversation

A run-level evaluator scores one tool call. It cannot answer whether the customer got what they asked for, because a conversation spans several calls and each one may be individually fine. That needs a thread-level evaluator, which reads the assembled conversation instead.

create_thread_evaluator.py creates one from code.

uv run create_thread_evaluator.py --project chatgpt-app-tracing

It posts a rule with group_by: "thread_id", which is the field that makes it a thread evaluator, and an inline LLM-as-judge whose prompt receives the assembled conversation as all_messages. The UI creates the judge as a Prompt Hub reference instead, so a rule made here reads slightly differently from one made by hand, but it behaves the same.

Three things to know.

Inputs and outputs must carry a top-level messages key, in LangChain, OpenAI, or Anthropic format. Raw MCP arguments do not qualify, and a thread evaluator given traces it cannot assemble produces nothing at all rather than an error. That is why the middleware emits messages.

It fires when a thread goes idle, ten minutes by default and two minutes at the least. The setting lives on the project rather than on the rule, so --idle-seconds sets it on the way through and you never need the project id.

uv run create_thread_evaluator.py --idle-seconds 120

The same change by hand, for a project that already has a rule on it. A PATCH replaces extra outright, so send back any other keys the project holds. The script merges them for you.

curl -X PATCH -H "x-api-key: $LANGSMITH_API_KEY" -H "Content-Type: application/json" \
  "$LANGSMITH_ENDPOINT/api/v1/sessions/<project_id>" \
  -d '{"extra": {"thread_idle_seconds": "120"}}'

To see a result without waiting at all, trigger the rule by hand.

curl -X POST -H "x-api-key: $LANGSMITH_API_KEY" \
  "$LANGSMITH_ENDPOINT/api/v1/runs/rules/<rule_id>/trigger"

Feedback attaches to one representative trace in the thread, not to every call in it, so in the project view switch the Threads/Traces/Runs toggle to Threads (?runview=threads) to see the score against the conversation.

What it does not show, and how to narrow it

The user's question, ChatGPT's reasoning between calls, and the final answer stay on OpenAI's side and never reach your server. Two ways to narrow that.

Ask the schema for it. A model fills in whatever a tool's schema asks for, so a field that your tool logic ignores still arrives with the arguments and lands on the trace. search_holidays here takes a customer_request string described as one short sentence covering what the customer asked for. model_harness.py drives the server with a real model whose instructions say nothing about that field, and the model populates it from the schema alone.

search_holidays  {"destination": "Lisbon", "month": "October", "max_price_gbp": 900}
  customer_request -> 'week in Lisbon under 900 pounds'

search_holidays  {"destination": "Reykjavik", "month": "November"}
  customer_request -> 'short break to see the northern lights'

The second one is the point. Northern lights appears in none of the structured arguments, so that intent would otherwise be invisible, and on the trace it becomes something an evaluator can judge the returned holiday against.

It is the model's paraphrase rather than a transcript, so treat it as evidence and not a record. It also means you are capturing conversation content your tool does not need, which is worth a look from whoever owns data handling before it goes near production.

Move the reasoning onto your own runtime. Trace that runtime and the tool call becomes one step inside a full trace rather than the whole of it.

Files