\documentclass[11pt]{article}
\usepackage[utf8]{inputenc}
\usepackage[margin=1in]{geometry}
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{hyperref}
\usepackage{listings}
\usepackage{xcolor}
\title{SE333 Final Project Reflection:\\
AI-Powered Testing Agent with Decision Table Test Generation}
\author{Armin Tavassoli}
\date{\today}
\begin{document}
\maketitle
\section{Abstract}
This project implements a Model Context Protocol (MCP) server that automates testing workflows for Java Maven projects. The agent provides decision table-based test generation, security vulnerability scanning, coverage analysis, and Git automation capabilities. The implementation uses Python with FastMCP, integrates with VS Code through HTTP transport, and demonstrates measurable improvements in code coverage through automated test generation.
\section{Introduction}
The goal of this project was to build an intelligent testing agent that could analyze Java codebases, identify gaps in test coverage, generate comprehensive test cases using decision table methodology, and scan for security vulnerabilities. The agent operates as an MCP server, making it accessible through VS Code's Chat interface, enabling developers to interact with testing tools through natural language.
The project required implementing several key features: coverage analysis using JaCoCo reports, decision table-based test generation that analyzes method logic and decision points, security vulnerability scanning for common Java security issues, and Git automation for tracking improvements. The technical stack includes Python 3.14, FastMCP for the MCP server implementation, Maven for Java project management, and JaCoCo for coverage reporting.
\section{Methodology}
\subsection{Technical Approach}
The MCP server was built using FastMCP, a Python framework for creating MCP-compatible servers. The server implements nine distinct tools, each serving a specific purpose in the testing workflow. The implementation follows a modular design where each tool is a Python function decorated with \texttt{@mcp.tool()}, making them available to MCP clients.
The server uses HTTP transport with streamable-http protocol, running on port 8001. This choice was made after encountering compatibility issues with SSE transport and VS Code's HTTP MCP client. The streamable-http protocol provides better compatibility with VS Code while maintaining the HTTP-based architecture required by the project specifications.
\subsection{Key Features Implementation}
\textbf{Coverage Analysis:} The \texttt{find\_jacoco\_path} and \texttt{missing\_coverage} tools parse JaCoCo XML reports to identify uncovered code segments. The implementation uses Python's XML parsing libraries to extract coverage metrics and identify classes and methods with low coverage.
\textbf{Test Generation:} The \texttt{generate\_decision\_table\_tests} tool analyzes Java source files, identifies decision points (if/else statements, switch cases, ternary operators), and generates JUnit test cases that cover various input scenarios including null inputs, empty inputs, valid inputs, boundary conditions, and exception scenarios.
\textbf{Security Scanning:} The \texttt{scan\_security\_vulnerabilities} tool uses pattern matching and static analysis techniques to identify common security issues such as SQL injection, command injection, path traversal, hardcoded secrets, and insecure deserialization. Findings are classified by severity (high, medium, low).
\textbf{Git Automation:} A suite of Git tools (\texttt{git\_status}, \texttt{git\_add\_all}, \texttt{git\_commit}, \texttt{git\_push}, \texttt{git\_pull\_request}) automates version control workflows, enabling the agent to track coverage improvements through Git commits.
\section{Challenges and Solutions}
\subsection{Environment Setup Issues}
One of the first challenges encountered was port conflicts on macOS. The default port 8000 was already in use by another process, causing the MCP server to fail to start. This was resolved by configuring the server to use port 8001 by default, with the ability to override via the \texttt{MCP\_PORT} environment variable.
Homebrew directory issues also caused problems during initial setup. The Homebrew installation path was not properly configured in the system PATH, making it difficult to install dependencies. This required manual PATH configuration and verification of the Homebrew installation.
\subsection{Maven Installation}
Maven was not initially installed in the project directory, and the system-wide Maven installation was not accessible. This required installing Maven either through Homebrew or manually, and ensuring it was available in the system PATH. Once Maven was properly installed, the JaCoCo plugin configuration in the \texttt{pom.xml} file could be used to generate coverage reports.
\subsection{Transport Protocol Compatibility}
The most significant technical challenge was achieving compatibility between the MCP server and VS Code's HTTP client. Initially, the server was configured to use SSE (Server-Sent Events) transport, but VS Code's HTTP MCP client had compatibility issues with FastMCP's SSE implementation. VS Code was attempting to POST requests to the SSE endpoint, which only accepts GET requests for establishing the SSE connection.
After investigating the FastMCP documentation and testing different transport protocols, the solution was to switch to \texttt{streamable-http} transport, which provides better compatibility with VS Code's HTTP client. The server now runs on the \texttt{/mcp} endpoint using streamable-http protocol, and VS Code successfully connects and can invoke all tools.
\subsection{Tool Integration}
Integrating the MCP tools with VS Code required careful configuration of the \texttt{.vscode/settings.json} file. The initial configuration attempts failed because VS Code was trying to use local process (stdio) mode when HTTP mode was specified, or vice versa. The final working configuration uses HTTP mode with the correct endpoint URL (\texttt{http://localhost:8001/mcp}).
\section{Results and Discussion}
\subsection{Coverage Improvement Analysis}
The agent successfully demonstrates the ability to improve code coverage through automated test generation. By analyzing JaCoCo reports to identify uncovered code segments and then generating decision table-based tests, the agent can systematically increase test coverage. The decision table methodology ensures comprehensive coverage of different code paths, including edge cases and boundary conditions.
The coverage improvement patterns show that AI-generated tests can effectively explore new code paths that might be missed in manual test writing. However, the quality of generated tests depends on the complexity of the method being tested and the accuracy of the decision point analysis.
\subsection{AI-Assisted Development Insights}
Using an LLM-powered agent for test generation revealed both strengths and limitations. The agent excels at generating boilerplate test code and identifying common test scenarios, but sometimes requires human review to ensure tests are meaningful and correctly test the intended behavior. The decision table approach provides structure to the test generation process, making it more reliable than purely generative approaches.
The integration with VS Code's Chat interface makes the tools accessible through natural language, which is more intuitive than writing test code manually. However, the agent sometimes requires multiple iterations or clarifications to generate the desired output, highlighting the importance of clear prompts and tool descriptions.
\subsection{Recommendations for Future Enhancements}
Several improvements could enhance the agent's capabilities:
\textbf{Test Quality Validation:} Add functionality to automatically compile and run generated tests, providing immediate feedback on test validity. This would help catch syntax errors and logical issues before manual review.
\textbf{Incremental Coverage Targeting:} Implement smarter prioritization of which classes and methods to test first, focusing on high-impact areas or frequently used code paths.
\textbf{Test Refinement:} Add the ability to refine generated tests based on execution results, automatically fixing common issues or improving test assertions.
\textbf{Multi-language Support:} Extend the agent to support other programming languages beyond Java, making it more versatile for different project types.
\textbf{Integration with CI/CD:} Add capabilities to integrate the agent into continuous integration pipelines, automatically generating and running tests as part of the build process.
\section{Conclusion}
This project successfully demonstrates the feasibility of using AI-powered agents to automate testing workflows. The MCP server implementation provides a flexible, extensible platform for testing automation that integrates seamlessly with development tools like VS Code. While challenges were encountered with environment setup, transport protocol compatibility, and tool integration, these were resolved through systematic troubleshooting and adaptation.
The decision table-based test generation approach provides structure and comprehensiveness to automated test creation, while the security scanning capabilities add value beyond just coverage improvement. The Git automation tools enable tracking of improvements over time, making the agent useful for iterative development workflows.
Key learnings from this project include the importance of understanding transport protocol requirements when integrating with development tools, the value of modular tool design for extensibility, and the need for human oversight even when using AI-powered automation. The project demonstrates that AI-assisted development can significantly accelerate testing workflows while maintaining quality through structured methodologies like decision tables.
\end{document}
