Skip to main content
Glama

tracewise: AI Agents for Requirements-to-Code Traceability

Software in regulated products such as medical devices has to show, for every requirement, which code implements it (IEC 62304 traceability). Keeping that trace matrix by hand is slow, so this project asks: how well can AI agents propose trace links, and which design works best? It also builds the workflow around them:

  • Baselines: keyword search (TF-IDF) and embeddings.

  • An LLM verifier: it picks implementing classes from search results in one call.

  • A tool-calling agent: it searches the code, reads classes and follows the call graph from an action class to the database classes it uses.

  • A review queue: a person accepts or rejects every proposal before it enters the trace matrix.

  • An MCP server: Claude Code or any other MCP client can do the tracing.

All models run locally through Ollama (Gemma 4, 8B, and nomic-embed-text), so evaluation costs nothing.

Data

The iTrust benchmark from CoEST (Center of Excellence for Software & Systems Traceability). iTrust is an open-source electronic health records system from NC State. The benchmark has:

  • 131 requirements (use-case flows such as "the HCP enters a patient's office visit");

  • 226 Java classes (23.6K lines);

  • 286 trace links made by hand, which serve as the gold set.

tracewise.corpus.fetch downloads it (GPL-3.0) from tobhey/finegrained-traceability; it is not committed.

Honest splits. Prompts and the trace convention were chosen on a dev split (use cases 4, 8, 12, …: 26 flows). Results below are on the other 105 flows, which were not used for tuning. The LLM methods ran on a fixed random sample of 40 of them, to keep the run short on a laptop (--sample 40, seed 0).

Related MCP server: arch-viewer

Results

Test sample: 40 requirements, 85 gold links (results/eval-test-sample40.json):

Method

Precision

Recall

F1

Time per requirement

Prompt tokens

Keyword search (TF-IDF), top 3

0.18

0.25

0.21

<0.1 s

-

Embeddings, top 3

0.14

0.20

0.17

<0.1 s

-

Hybrid (both, rank fusion), top 3

0.23

0.32

0.26

<0.1 s

-

LLM verifier (hybrid top 12 → model picks)

0.54

0.38

0.44

6.3 s

2.3K

Tool-calling agent (search, read, call graph)

0.42

0.40

0.41

12.0 s

12.2K

Ranking quality on all 105 test requirements (results/rankers-test-full.json):

Ranker

MAP

Recall@5

Recall@10

Recall@20

TF-IDF

0.292

0.356

0.479

0.593

Embeddings

0.237

0.258

0.335

0.470

Hybrid

0.370

0.411

0.487

0.593

  • Both LLM methods roughly double the precision of search alone. At about the same recall as the hybrid ranker's top 3, the verifier is right on 54% of its links against 23%.

  • The simpler design won. One verifier call matched the agent's recall within 2 points, with higher precision, half the time and a fifth of the tokens. The agent's advantage is reach: through the call graph it can find a DAO that search never ranked. That helped on some flows and also produced plausible but unlinked DAOs on others.

  • Neither method invented a class name (0 of 140 proposed links), because every proposal is checked against the code and tools return only real names.

  • Search sets the ceiling. Even the top 20 search results contain only 59% of the gold links, so a verifier can't find the rest. That is the argument for an agent that can explore, and it is where the agent should improve.

The gold set is incomplete, so these scores understate both methods. 16 of the agent's 47 "wrong" links are on requirements the gold set links to nothing, including flows with an obvious implementation:

  • UC22S1, "the administrator can add a new appointment type or modify its duration": EditApptTypeAction has addApptType and editApptType methods, yet the gold set has no link.

  • Use case 40 (the appointment calendar, dev split): its seven flows have no gold links although GenerateCalendarAction implements them. These features look newer than the trace set.

This is exactly why proposals go to a reviewer instead of straight into the matrix.

What changed during development (dev split only).

  • A parsing bug, not the prompt, was the biggest problem. When the agent hit its step limit, it wrote its answer as {"action": "SetSecurityQuestionAction"} instead of {"class": ...}. The parser read that as no links, so investigations that had found the right classes scored zero. The parser now accepts either key, and the fallback prompt spells out the format.

  • Prompt revisions mostly traded precision for recall. Telling the agent to follow the call graph to the DAOs, and to link EmailUtil when a flow sends email, moved dev F1 between 0.42 and 0.44.

The workflow

uv sync
uv run python -c "from tracewise.corpus import fetch; fetch()"   # data/itrust
ollama pull gemma4 && ollama pull nomic-embed-text

uv run tracewise propose UC22S1 UC22S3       # the agent proposes links with a reason for each
uv run tracewise import results/runs/test-agent.jsonl   # or load a saved evaluation run
uv run tracewise queue
#   #22  UC22S1 -> EditApptTypeAction  This action class provides methods to add and edit appointment types, ...
#   #23  UC22S1 -> ApptTypeDAO         This DAO is used by EditApptTypeAction to read and write appointment ...
uv run tracewise accept 22 --by "A. Pallath" --note "addApptType and editApptType implement both parts"
uv run tracewise accept 22 --by "someone else"   # -> link 22 was already accepted by A. Pallath
uv run tracewise matrix --out trace_matrix.csv   # accepted links with who reviewed them and when
uv run tracewise gaps                            # requirements nothing has been traced to yet

uv run python -m tracewise.evaluate --split test --sample 40   # the evaluation above

MCP server. claude mcp add tracewise -- uv --directory /path/to/tracewise run tracewise-mcp gives any MCP client these tools: get_requirement, search_code, read_class, related_classes, propose_link, trace_matrix and coverage_gaps. A client can only propose; acceptance stays with a person. Proposals are validated: no unknown requirements, no invented classes, no empty reasons.

Design notes

  • A rejected link stays rejected. The next agent run cannot re-propose it, and a link can't be reviewed twice. The database enforces both, and records the reviewer, time and note for an audit.

  • The agent starts with a search already run. Before the first model call it already has the hybrid search results for the requirement. In an earlier project (rulebook-ai), agents that had to decide to search on their own often skipped it.

  • LLM runs are saved per requirement, so an interrupted evaluation resumes where it stopped.

Limitations

  • One benchmark and one 8B local model. The 40-requirement sample makes the verifier-vs-agent gap (0.44 vs 0.41 F1) too small to call a winner; the gap to search alone is not.

  • The trace convention given to the models ("link the action and the DAOs it uses, not data beans") was read off the dev split's gold links. Another project's convention would need its own.

  • Classes are the unit of tracing, as in the benchmark; a real trace matrix often goes down to methods.

Tests

uv run pytest: a synthetic three-class codebase checks Java parsing, the gold and call-graph loading, ranking and the metrics against hand-computed values, and link parsing with odd keys and invented names. A scripted fake model drives the agent through tool calls and the step-limit fallback. The review queue, the CLI and the MCP tools are tested too. CI runs lint and tests on every push; no model is needed.

Built with AI assistance (Claude Code). I reviewed the code and checked the results; the parsing bug and the gold set gaps above came from reading the agent's runs one by one.

Available Tools

7 tools
coverage_gapsC

Requirements with no accepted link yet.

NameRequiredDescriptionDefault

No parameters

Output Schema

NameRequiredDescription
resultYes

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full behavioral burden, yet it says nothing beyond the definition of the result set — not that it is a read-only query, whether it is exhaustive or paginated, or how 'accepted' vs. proposed links are treated. This is the main weakness for a tool an agent may call blindly.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single short sentence with no filler, which is proportionate to a zero-parameter tool. It is a noun phrase rather than a front-loaded verb clause, which slightly weakens the opening.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The presence of an output schema removes the need to document return values, and no parameters need explaining, so the remaining gap is behavioral and usage context. For a simple read tool that gap is tolerable but still leaves the agent guessing about scope and semantics of 'accepted'.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool takes zero parameters, so there is nothing for the description to disambiguate; baseline 4 applies. An empty schema leaves no ambiguity for the agent to resolve.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose3/5

Does the description clearly state what the tool does and how it differs from similar tools?

The fragment 'Requirements with no accepted link yet' identifies the resource set returned but supplies no verb, so it is unclear whether the tool lists gaps, counts them, or computes a coverage metric. It is distinguishable from siblings like get_requirement and propose_link by subject matter, but the operation itself is only implied.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no statement of when to call this versus alternatives such as trace_matrix or get_requirement, and no preconditions are given. Usage can only be inferred from the phrase 'no accepted link yet'.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_requirementC

A requirement (use-case flow, e.g. UC10S1) with the other flows of its use case.

NameRequiredDescriptionDefault
requirement_idYes

TDQS

C2.5/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations exist, so the description carries the full disclosure burden and does little. It does convey one useful trait — that the response includes the other flows of the parent use case, not just the requested item — but says nothing about behavior for a missing/invalid ID, permissions, or output shape.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single short sentence with no wasted words, but it is a fragment that omits the action verb; brevity here reads as under-specification rather than discipline.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple single-parameter lookup with no annotations and no output schema, the description leaves too much implicit: it doesn't state retrieval semantics, error behavior, or how it relates to the other knowledge-graph tools. Only the return-scope hint is genuinely useful.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Only one parameter with 0% schema description coverage, so the description must compensate. The 'UC10S1' example does clarify the expected ID format, which the schema does not, but the description never states what the parameter means or whether it accepts other ID schemes.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose3/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description is a noun phrase rather than a verb+resource statement, but it does identify the resource (a requirement / use-case flow) and its scope (the other flows of the same use case). The 'UC10S1' example anchors the ID style. It fails to say the tool retrieves by ID and does not differentiate itself from siblings like related_classes or trace_matrix.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No indication of when to call this versus the sibling tools (search_code, read_class, related_classes, trace_matrix, coverage_gaps). Usage has to be inferred entirely from the name.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

read_classC

A class's package, comment and method signatures.

NameRequiredDescriptionDefault
nameYes

Output Schema

NameRequiredDescription
resultYes

TDQS

C2.7/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden of behavioral disclosure. It states what data is returned but omits any information about read-only nature, permissions, error handling, or other behavioral traits.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is extremely concise and front-loaded, containing no wasted words. However, it is a noun phrase rather than a structured explanation, which slightly limits its helpfulness.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so the description's enumeration of return contents is largely redundant. Meanwhile, it fails to address parameter semantics or usage alternatives, leaving clear gaps for an agent trying to invoke the tool correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The single required parameter 'name' has 0% schema description coverage, and the description does not explain its format or meaning beyond implying a class name. It adds minimal value over the bare schema field.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly identifies the resource (a class) and the specific information returned: package, comment, and method signatures. It is distinguishable from most siblings by content, but it does not explicitly differentiate itself from related_classes or search_code.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description offers no explicit guidance on when to use this tool versus alternatives such as related_classes or search_code. The intended use (retrieving class details) is only implied by the listed return contents.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

search_codeB

Find classes by what they do. Returns class names, packages and purposes.

NameRequiredDescriptionDefault
queryYes

Output Schema

NameRequiredDescription
resultYes

TDQS

B3.1/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations at all, the description carries the full behavioral burden and does little: it never says whether matching is keyword or semantic, whether results are ranked or capped, or what happens with zero matches. It only discloses the shape of each result (names, packages, purposes), which the output schema already covers.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two short sentences with zero filler, and the capability statement is front-loaded ahead of the return-value note. Nothing here could be cut without losing information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The output schema means return values needn't be re-explained, which the description correctly avoids over-elaborating. However, for a search tool with one entirely undocumented parameter and several overlapping siblings, the description leaves search semantics and sibling selection under-specified.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% for the single required `query` parameter, so the description must compensate. The phrase "by what they do" usefully implies the query is a natural-language description of behavior rather than a class name, but it gives no guidance on phrasing, expected length, or matching strictness.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

"Find classes by what they do" states a specific verb (find) and resource (classes) and adds a scoping qualifier — behavior-based rather than name-based lookup. It never names the obvious alternative, read_class, so an agent must infer the split from the sibling list rather than from the description.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no explicit when-to-use statement and no alternative is named. A reader can infer that this tool is for discovery when the class name is unknown, but that inference is not stated, and nothing says when to prefer read_class or related_classes instead.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

trace_matrixC

Reviewed and accepted links: requirement id -> classes.

NameRequiredDescriptionDefault

No parameters

Output Schema

NameRequiredDescription

No output parameters

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full disclosure burden, yet it never states that this is a read-only, side-effect-free operation, nor what 'reviewed and accepted' status filtering entails. It adds only a thin filter semantic (accepted links only) and says nothing about scope or freshness of the matrix.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single compact sentence with no filler, front-loading the key qualifier 'Reviewed and accepted'. It is terse to the point of being cryptic, but nothing is wasted.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return values need not be explained, and zero parameters keep the call surface trivial. However, the description leaves the action and scope of the matrix unstated, which is a real gap for an agent deciding between this and coverage_gaps.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool takes zero parameters, so the baseline is 4; there is nothing for the description to clarify beyond what an empty schema already implies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose3/5

Does the description clearly state what the tool does and how it differs from similar tools?

The phrase 'Reviewed and accepted links: requirement id -> classes' conveys the data the tool exposes (a requirement-to-class mapping) but contains no verb or explicit action, so an agent must infer that this is a lookup/retrieval. The 'reviewed and accepted' qualifier does distinguish it from propose_link, but only indirectly.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no explicit when-to-use guidance, no mention of alternatives like coverage_gaps or get_requirement, and no stated prerequisites. The only implied usage cue is that it returns accepted rather than proposed links.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 7 tool updatesv0.1.0
    • First observedcoverage_gaps
    • First observedget_requirement
    • First observedpropose_link
    • First observedread_class
    • First observedrelated_classes
    • First observedsearch_code
    • First observedtrace_matrix

TDQS

B3.2/5.0

Scored across 7 tools

Disambiguation5/5

Each tool targets a clearly distinct operation: requirement lookup, code search, class reading, related-class traversal, link proposal, accepted trace inspection, and gap detection. The boundary between proposed links and accepted trace-matrix links is especially clear.

Naming Consistency4/5

All names use consistent snake_case and are readable. Some are verb-led (get_requirement, search_code, read_class, propose_link) while others are noun phrases (trace_matrix, coverage_gaps), which is a minor deviation from a strict verb_noun pattern.

Tool Count5/5

Seven tools is well-scoped for a traceability-focused server. Each tool covers a meaningful part of the workflow without obvious redundancy.

Completeness3/5

The surface covers requirement lookup, code exploration, link proposal, trace inspection, and gap analysis. However, there is no way to list or search requirements, and proposed links go to a review queue with no tool to inspect or accept/reject them, leaving notable lifecycle gaps.

Maintenance

ActivityMaintained
ResponsivenessNo issues

Related MCP Connectors

Related MCP Servers

  • A
    license
    Not graded
    quality
    C
    maintenance
    Enables AI clients to perform local code search, indexing, and analysis across Java, JavaScript/TypeScript, .NET/C#, and Python projects through the MCP protocol.
    1
    Apache 2.0
  • A
    license
    Not graded
    quality
    D
    maintenance
    Provides AI-powered architecture analysis and visualization of codebases, exposing 17 MCP tools for querying components, dependencies, and generating interactive diagrams.
    16 npm
    5
    MIT
  • F
    license
    Not graded
    quality
    B
    maintenance
    MCP server that exposes code tracing capabilities including journey flows, HTTP seams, and findings from indexed projects, allowing AI assistants to query software architecture.
    -
  • A
    license
    Not graded
    quality
    D
    maintenance
    Enables AI coding assistants to semantically search and retrieve relevant code patterns, documentation, and implementations from a codebase via MCP tools.
    8
    MIT