tracewise-mcp
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@tracewise-mcppropose trace links for requirement UC22S1 and show coverage gaps"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
tracewise: AI Agents for Requirements-to-Code Traceability
Software in regulated products such as medical devices has to show, for every requirement, which code implements it (IEC 62304 traceability). Keeping that trace matrix by hand is slow, so this project asks: how well can AI agents propose trace links, and which design works best? It also builds the workflow around them:
Baselines: keyword search (TF-IDF) and embeddings.
An LLM verifier: it picks implementing classes from search results in one call.
A tool-calling agent: it searches the code, reads classes and follows the call graph from an action class to the database classes it uses.
A review queue: a person accepts or rejects every proposal before it enters the trace matrix.
An MCP server: Claude Code or any other MCP client can do the tracing.
All models run locally through Ollama (Gemma 4, 8B, and nomic-embed-text), so evaluation costs nothing.
Data
The iTrust benchmark from CoEST (Center of Excellence for Software & Systems Traceability). iTrust is an open-source electronic health records system from NC State. The benchmark has:
131 requirements (use-case flows such as "the HCP enters a patient's office visit");
226 Java classes (23.6K lines);
286 trace links made by hand, which serve as the gold set.
tracewise.corpus.fetch downloads it (GPL-3.0) from
tobhey/finegrained-traceability; it is not committed.
Honest splits. Prompts and the trace convention were chosen on a dev split (use cases 4, 8, 12, …: 26 flows).
Results below are on the other 105 flows, which were not used for tuning. The LLM methods ran on a fixed random
sample of 40 of them, to keep the run short on a laptop (--sample 40, seed 0).
Related MCP server: arch-viewer
Results
Test sample: 40 requirements, 85 gold links (results/eval-test-sample40.json):
Method | Precision | Recall | F1 | Time per requirement | Prompt tokens |
Keyword search (TF-IDF), top 3 | 0.18 | 0.25 | 0.21 | <0.1 s | - |
Embeddings, top 3 | 0.14 | 0.20 | 0.17 | <0.1 s | - |
Hybrid (both, rank fusion), top 3 | 0.23 | 0.32 | 0.26 | <0.1 s | - |
LLM verifier (hybrid top 12 → model picks) | 0.54 | 0.38 | 0.44 | 6.3 s | 2.3K |
Tool-calling agent (search, read, call graph) | 0.42 | 0.40 | 0.41 | 12.0 s | 12.2K |
Ranking quality on all 105 test requirements (results/rankers-test-full.json):
Ranker | MAP | Recall@5 | Recall@10 | Recall@20 |
TF-IDF | 0.292 | 0.356 | 0.479 | 0.593 |
Embeddings | 0.237 | 0.258 | 0.335 | 0.470 |
Hybrid | 0.370 | 0.411 | 0.487 | 0.593 |
Both LLM methods roughly double the precision of search alone. At about the same recall as the hybrid ranker's top 3, the verifier is right on 54% of its links against 23%.
The simpler design won. One verifier call matched the agent's recall within 2 points, with higher precision, half the time and a fifth of the tokens. The agent's advantage is reach: through the call graph it can find a DAO that search never ranked. That helped on some flows and also produced plausible but unlinked DAOs on others.
Neither method invented a class name (0 of 140 proposed links), because every proposal is checked against the code and tools return only real names.
Search sets the ceiling. Even the top 20 search results contain only 59% of the gold links, so a verifier can't find the rest. That is the argument for an agent that can explore, and it is where the agent should improve.
The gold set is incomplete, so these scores understate both methods. 16 of the agent's 47 "wrong" links are on requirements the gold set links to nothing, including flows with an obvious implementation:
UC22S1, "the administrator can add a new appointment type or modify its duration":
EditApptTypeActionhasaddApptTypeandeditApptTypemethods, yet the gold set has no link.Use case 40 (the appointment calendar, dev split): its seven flows have no gold links although
GenerateCalendarActionimplements them. These features look newer than the trace set.
This is exactly why proposals go to a reviewer instead of straight into the matrix.
What changed during development (dev split only).
A parsing bug, not the prompt, was the biggest problem. When the agent hit its step limit, it wrote its answer as
{"action": "SetSecurityQuestionAction"}instead of{"class": ...}. The parser read that as no links, so investigations that had found the right classes scored zero. The parser now accepts either key, and the fallback prompt spells out the format.Prompt revisions mostly traded precision for recall. Telling the agent to follow the call graph to the DAOs, and to link
EmailUtilwhen a flow sends email, moved dev F1 between 0.42 and 0.44.
The workflow
uv sync
uv run python -c "from tracewise.corpus import fetch; fetch()" # data/itrust
ollama pull gemma4 && ollama pull nomic-embed-text
uv run tracewise propose UC22S1 UC22S3 # the agent proposes links with a reason for each
uv run tracewise import results/runs/test-agent.jsonl # or load a saved evaluation run
uv run tracewise queue
# #22 UC22S1 -> EditApptTypeAction This action class provides methods to add and edit appointment types, ...
# #23 UC22S1 -> ApptTypeDAO This DAO is used by EditApptTypeAction to read and write appointment ...
uv run tracewise accept 22 --by "A. Pallath" --note "addApptType and editApptType implement both parts"
uv run tracewise accept 22 --by "someone else" # -> link 22 was already accepted by A. Pallath
uv run tracewise matrix --out trace_matrix.csv # accepted links with who reviewed them and when
uv run tracewise gaps # requirements nothing has been traced to yet
uv run python -m tracewise.evaluate --split test --sample 40 # the evaluation aboveMCP server. claude mcp add tracewise -- uv --directory /path/to/tracewise run tracewise-mcp gives any MCP
client these tools: get_requirement, search_code, read_class, related_classes, propose_link,
trace_matrix and coverage_gaps. A client can only propose; acceptance stays with a person. Proposals are
validated: no unknown requirements, no invented classes, no empty reasons.
Design notes
A rejected link stays rejected. The next agent run cannot re-propose it, and a link can't be reviewed twice. The database enforces both, and records the reviewer, time and note for an audit.
The agent starts with a search already run. Before the first model call it already has the hybrid search results for the requirement. In an earlier project (rulebook-ai), agents that had to decide to search on their own often skipped it.
LLM runs are saved per requirement, so an interrupted evaluation resumes where it stopped.
Limitations
One benchmark and one 8B local model. The 40-requirement sample makes the verifier-vs-agent gap (0.44 vs 0.41 F1) too small to call a winner; the gap to search alone is not.
The trace convention given to the models ("link the action and the DAOs it uses, not data beans") was read off the dev split's gold links. Another project's convention would need its own.
Classes are the unit of tracing, as in the benchmark; a real trace matrix often goes down to methods.
Tests
uv run pytest: a synthetic three-class codebase checks Java parsing, the gold and call-graph loading, ranking and
the metrics against hand-computed values, and link parsing with odd keys and invented names. A scripted fake model
drives the agent through tool calls and the step-limit fallback. The review queue, the CLI and the MCP tools are
tested too. CI runs lint and tests on every push; no model is needed.
Built with AI assistance (Claude Code). I reviewed the code and checked the results; the parsing bug and the gold set gaps above came from reading the agent's runs one by one.
Available Tools
7 toolscoverage_gapsC
Requirements with no accepted link yet.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full behavioral burden, yet it says nothing beyond the definition of the result set — not that it is a read-only query, whether it is exhaustive or paginated, or how 'accepted' vs. proposed links are treated. This is the main weakness for a tool an agent may call blindly.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single short sentence with no filler, which is proportionate to a zero-parameter tool. It is a noun phrase rather than a front-loaded verb clause, which slightly weakens the opening.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The presence of an output schema removes the need to document return values, and no parameters need explaining, so the remaining gap is behavioral and usage context. For a simple read tool that gap is tolerable but still leaves the agent guessing about scope and semantics of 'accepted'.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool takes zero parameters, so there is nothing for the description to disambiguate; baseline 4 applies. An empty schema leaves no ambiguity for the agent to resolve.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The fragment 'Requirements with no accepted link yet' identifies the resource set returned but supplies no verb, so it is unclear whether the tool lists gaps, counts them, or computes a coverage metric. It is distinguishable from siblings like get_requirement and propose_link by subject matter, but the operation itself is only implied.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no statement of when to call this versus alternatives such as trace_matrix or get_requirement, and no preconditions are given. Usage can only be inferred from the phrase 'no accepted link yet'.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_requirementC
A requirement (use-case flow, e.g. UC10S1) with the other flows of its use case.
| Name | Required | Description | Default |
|---|---|---|---|
| requirement_id | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations exist, so the description carries the full disclosure burden and does little. It does convey one useful trait — that the response includes the other flows of the parent use case, not just the requested item — but says nothing about behavior for a missing/invalid ID, permissions, or output shape.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single short sentence with no wasted words, but it is a fragment that omits the action verb; brevity here reads as under-specification rather than discipline.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple single-parameter lookup with no annotations and no output schema, the description leaves too much implicit: it doesn't state retrieval semantics, error behavior, or how it relates to the other knowledge-graph tools. Only the return-scope hint is genuinely useful.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Only one parameter with 0% schema description coverage, so the description must compensate. The 'UC10S1' example does clarify the expected ID format, which the schema does not, but the description never states what the parameter means or whether it accepts other ID schemes.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description is a noun phrase rather than a verb+resource statement, but it does identify the resource (a requirement / use-case flow) and its scope (the other flows of the same use case). The 'UC10S1' example anchors the ID style. It fails to say the tool retrieves by ID and does not differentiate itself from siblings like related_classes or trace_matrix.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No indication of when to call this versus the sibling tools (search_code, read_class, related_classes, trace_matrix, coverage_gaps). Usage has to be inferred entirely from the name.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
propose_linkA
Propose that a class implements a requirement. It goes to the review queue, not the trace matrix.
| Name | Required | Description | Default |
|---|---|---|---|
| reason | Yes | ||
| class_name | Yes | ||
| requirement_id | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description must carry the full behavioral burden; it does disclose the key trait that the result is a pending proposal routed for review rather than an immediate write to the trace matrix. It says nothing about permissions, duplicate/pending-link handling, idempotency, or what the call returns, so the write path is only partly characterized.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two tight sentences with no filler; the core action comes first and the queue-vs-matrix distinction follows immediately. Every clause earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 3-required-param mutation tool with no annotations, no output schema, and zero schema descriptions, the description captures the most important workflow semantics (pending review) but leaves parameter intent and edge-case behavior undocumented. Adequate to route the call, not sufficient to call it confidently.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description is responsible for parameter meaning, yet it explains none of the three required fields explicitly. 'class implements a requirement' lets an agent infer class_name and requirement_id, but 'reason' is completely unaddressed and no format or constraint hints are given.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (propose) and resource (a link between a class and a requirement), and explicitly contrasts the outcome with the sibling trace_matrix. An agent can distinguish it from trace_matrix and related_classes without opening any schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The clause 'goes to the review queue, not the trace matrix' tells the agent when this tool is the right choice versus trace_matrix. It stops short of stating preconditions (e.g., use only when the link is unverified or does not yet exist) or exclusions for other siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
read_classC
A class's package, comment and method signatures.
| Name | Required | Description | Default |
|---|---|---|---|
| name | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden of behavioral disclosure. It states what data is returned but omits any information about read-only nature, permissions, error handling, or other behavioral traits.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise and front-loaded, containing no wasted words. However, it is a noun phrase rather than a structured explanation, which slightly limits its helpfulness.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so the description's enumeration of return contents is largely redundant. Meanwhile, it fails to address parameter semantics or usage alternatives, leaving clear gaps for an agent trying to invoke the tool correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The single required parameter 'name' has 0% schema description coverage, and the description does not explain its format or meaning beyond implying a class name. It adds minimal value over the bare schema field.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly identifies the resource (a class) and the specific information returned: package, comment, and method signatures. It is distinguishable from most siblings by content, but it does not explicitly differentiate itself from related_classes or search_code.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description offers no explicit guidance on when to use this tool versus alternatives such as related_classes or search_code. The intended use (retrieving class details) is only implied by the listed return contents.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
search_codeB
Find classes by what they do. Returns class names, packages and purposes.
| Name | Required | Description | Default |
|---|---|---|---|
| query | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations at all, the description carries the full behavioral burden and does little: it never says whether matching is keyword or semantic, whether results are ranked or capped, or what happens with zero matches. It only discloses the shape of each result (names, packages, purposes), which the output schema already covers.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two short sentences with zero filler, and the capability statement is front-loaded ahead of the return-value note. Nothing here could be cut without losing information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The output schema means return values needn't be re-explained, which the description correctly avoids over-elaborating. However, for a search tool with one entirely undocumented parameter and several overlapping siblings, the description leaves search semantics and sibling selection under-specified.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0% for the single required `query` parameter, so the description must compensate. The phrase "by what they do" usefully implies the query is a natural-language description of behavior rather than a class name, but it gives no guidance on phrasing, expected length, or matching strictness.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
"Find classes by what they do" states a specific verb (find) and resource (classes) and adds a scoping qualifier — behavior-based rather than name-based lookup. It never names the obvious alternative, read_class, so an agent must infer the split from the sibling list rather than from the description.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no explicit when-to-use statement and no alternative is named. A reader can infer that this tool is for discovery when the class name is unknown, but that inference is not stated, and nothing says when to prefer read_class or related_classes instead.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
trace_matrixC
Reviewed and accepted links: requirement id -> classes.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full disclosure burden, yet it never states that this is a read-only, side-effect-free operation, nor what 'reviewed and accepted' status filtering entails. It adds only a thin filter semantic (accepted links only) and says nothing about scope or freshness of the matrix.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single compact sentence with no filler, front-loading the key qualifier 'Reviewed and accepted'. It is terse to the point of being cryptic, but nothing is wasted.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return values need not be explained, and zero parameters keep the call surface trivial. However, the description leaves the action and scope of the matrix unstated, which is a real gap for an agent deciding between this and coverage_gaps.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool takes zero parameters, so the baseline is 4; there is nothing for the description to clarify beyond what an empty schema already implies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The phrase 'Reviewed and accepted links: requirement id -> classes' conveys the data the tool exposes (a requirement-to-class mapping) but contains no verb or explicit action, so an agent must infer that this is a lookup/retrieval. The 'reviewed and accepted' qualifier does distinguish it from propose_link, but only indirectly.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no explicit when-to-use guidance, no mention of alternatives like coverage_gaps or get_requirement, and no stated prerequisites. The only implied usage cue is that it returns accepted rather than proposed links.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
7 tool updates
v0.1.0- First observed
coverage_gaps - First observed
get_requirement - First observed
propose_link - First observed
read_class - First observed
related_classes - First observed
search_code - First observed
trace_matrix
TDQS
Scored across 7 tools
Each tool targets a clearly distinct operation: requirement lookup, code search, class reading, related-class traversal, link proposal, accepted trace inspection, and gap detection. The boundary between proposed links and accepted trace-matrix links is especially clear.
All names use consistent snake_case and are readable. Some are verb-led (get_requirement, search_code, read_class, propose_link) while others are noun phrases (trace_matrix, coverage_gaps), which is a minor deviation from a strict verb_noun pattern.
Seven tools is well-scoped for a traceability-focused server. Each tool covers a meaningful part of the workflow without obvious redundancy.
The surface covers requirement lookup, code exploration, link proposal, trace inspection, and gap analysis. However, there is no way to list or search requirements, and proposed links go to a review queue with no tool to inspect or accept/reject them, leaving notable lifecycle gaps.
Maintenance
Related MCP Connectors
Spec-driven development for AI coding agents: live requirements, stories and traceability over MCP
- HypAwareOAuthai.hypaware
Query your team's AI coding history and act on HypAware recommendations from any MCP client.
Security reviews, threat models over a repo or website, and remediation tracking, in your editor. MCP Endpoints - US - https://mcp.trent.ai/mcp - EU - https://mcp.eu.trent.ai/mcp
Change-impact intelligence for tracing dependencies, stale references and required updates.
Related MCP Servers
- AlicenseNot gradedqualityCmaintenanceEnables AI clients to perform local code search, indexing, and analysis across Java, JavaScript/TypeScript, .NET/C#, and Python projects through the MCP protocol.1Apache 2.0
- AlicenseNot gradedqualityDmaintenanceProvides AI-powered architecture analysis and visualization of codebases, exposing 17 MCP tools for querying components, dependencies, and generating interactive diagrams.16 npm5MIT
- FlicenseNot gradedqualityBmaintenanceMCP server that exposes code tracing capabilities including journey flows, HTTP seams, and findings from indexed projects, allowing AI assistants to query software architecture.-
- AlicenseNot gradedqualityDmaintenanceEnables AI coding assistants to semantically search and retrieve relevant code patterns, documentation, and implementations from a codebase via MCP tools.8MIT