BugPilot

AI-Powered Engineering Bug Intelligence Agent

Core Architecture: BugPilot runs on a clean, end-to-end decoupled pipeline: React / Vite → FastAPI Backend → ReAct Orchestrator → Specialist Agents → MCP Client → MCP Server → 10 Read-Only Tools → SQLite Database (Synthetic Jira Data).

1. System Architecture

┌─────────────────────────────────────────────────────────────────────────────┐
│                       React / Vite Frontend (TypeScript)                     │
└──────────────────────────────────────┬──────────────────────────────────────┘
                                        │ HTTP REST API (JWT + RBAC + Tenant Isolation)
                                        ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                          FastAPI Backend (Port 8000)                         │
└──────────────────────────────────────┬──────────────────────────────────────┘
                                        │
                                        ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                        ReAct Orchestrator Agent                              │
│         Dynamic reasoning loop: Goal → LLM Decision → Tool Call →            │
│                    Observation → Next Decision → FINISH                     │
│               [Groq Primary API + Local Ollama Fallback]                    │
└───────────┬──────────────────────────┼──────────────────────────┬───────────┘
            │                          │                          │
            ▼                          ▼                          ▼
┌──────────────────────┐   ┌──────────────────────┐   ┌──────────────────────┐
│  Bug Analyst Agent   │   │ Trend Analyst Agent  │   │ Risk Analyst Agent   │
└───────────┬──────────┘   └──────────┬───────────┘   └──────────┬───────────┘
            │                          │                          │
            └──────────────────────────┼──────────────────────────┘
                                        │
                                        ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                                 MCP Client                                   │
│                 Dynamic tool discovery, timeout & sandboxing                 │
└──────────────────────────────────────┬──────────────────────────────────────┘
                                        │ stdio JSON-RPC Transport
                                        ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                          MCP Server (mcp_server)                            │
│                  Exposes 10 Strict READ-ONLY Tools                          │
└──────────────────────────────────────┬──────────────────────────────────────┘
                                        │
                                        ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                              AnalyticsService                                │
│              Deterministic metric calculation & statistical trends          │
└──────────────────────────────────────┬──────────────────────────────────────┘
                                        │
                                        ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                 DataProvider Interface (SQLDataProvider / SQLite)            │
│                 Multi-tenant tenant isolation (`organization_id`)           │
└──────────────────────────────────────┬──────────────────────────────────────┘
                                        │
                                        ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                    SQLite Database (`sqlite:///./bugpilot.db`)              │
│       Realistic Jira-style Defect Catalog, Sprints, Users & Audit Trails    │
└─────────────────────────────────────────────────────────────────────────────┘

Strict Data Access Contract

✅ Agent → MCP Client → MCP Server → AnalyticsService → DataProvider → SQLite Data
❌ Agent → Direct Database Access (FORBIDDEN)
❌ Agent → Direct Data File Reading (FORBIDDEN)
❌ External Vector Database / RAG dependencies (FORBIDDEN)

Agents interact exclusively through dynamically discovered MCP tools, preserving sandboxing and full testability.

Related MCP server: mcp-sqlite-manager

2. 10 MCP Tools Reference

All 10 tools are strictly read-only, tenant-scoped (org_id), and dynamically discovered via the MCP protocol:

3. Dynamic ReAct Orchestration & Comparative Analysis

The Orchestrator Agent operates on a genuine Reasoning + Action (ReAct) loop:

1. Intent & Out-of-Domain Guardrail — Early checks filter non-engineering queries without wasting LLM/tool invocations.

2. Dynamic Tool Selection — LLM decides each action (CALL_TOOL, DELEGATE, or FINISH) based on the query, dynamically discovered tools, and accumulated observations.

3. Iterative Multi-Candidate Inspection — For comparative and ranking queries ("analyze authentication bugs and identify the highest-risk issue"), search_bugs discovers candidates, and the Orchestrator iteratively invokes get_bug on each candidate defect before allowing FINISH, ensuring full technical evidence (root cause, blast radius, reproduction steps) is gathered.

4. Differentiated Evidence-Grounded Risk Scoring — Evaluates severity, priority, status, production environment, security impact (e.g. SOC2/session hijacking), and technical root causes (e.g. race condition, crash). Generates non-saturating scores (0.0–99.5) to avoid artificial 100/100 ties.

5. Reflection Agent Quality Evaluation — Validates generated reports against ground-truth MCP data to prevent hallucinations and confirm accurate reporting.

4. Multi-Tenancy & RBAC Security

• Tenant Isolation — Every database record (issues, sprints, users, audit_logs) is strictly scoped by organization_id (e.g. org-acme). Cross-organization data access is blocked at the repository and MCP layers.

• Role-Based Access Control (RBAC):

  • Admin — Full access, user management, and issue administration.

  • Engineer / Developer — Create, update, transition, and analyze issues.

  • Viewer — Read-only access to issues, analytics, and reports.

• Secrets Management — No secret keys or credentials are hardcoded. JWT secrets, API keys, and environment variables are strictly loaded from .env and excluded from version control.

5. Technology Stack

6. Setup & Execution Guide

Prerequisites

• Python 3.12+

• Node.js 18+ (for frontend)

• A Groq API key (optional — the app runs without one, falling back to local Ollama or deterministic mode)

1. Backend Setup

# Clone and enter project
git clone <repo-url> bugpilot
cd bugpilot

# Create and activate virtual environment
python -m venv .venv
# Windows: .\.venv\Scripts\activate | macOS/Linux: source .venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# Configure environment (defaults to SQLite with zero setup)
cp .env.example .env
# Optional: add your GROQ_API_KEY to .env for live LLM responses

2. Run Standalone MCP Server

# Windows
.\.venv\Scripts\python -m mcp_server.server

# macOS / Linux
.venv/bin/python -m mcp_server.server

3. Run FastAPI Backend

uvicorn backend.main:app --host 127.0.0.1 --port 8000 --reload

The database is created and seeded automatically on first startup — no migration step required. Verify it's healthy at http://127.0.0.1:8000/api/v1/health and http://127.0.0.1:8000/docs.

4. Build & Run Frontend

cd frontend
npm install
npm run dev

The Vite dev server proxies /api requests to the FastAPI backend on port 8000 (see vite.config.ts), so both services need to be running together.

5. Execute Test Suite

# Run all unit and integration tests (329 tests)
pytest tests/unit tests/integration -q

LLM calls are mocked during tests (see tests/conftest.py), so the suite runs deterministically without a Groq/Ollama connection. Because several tests spin up a fresh MCP server subprocess, the full suite takes a few minutes — this is expected, not a hang.

7. Evaluation & Quality Results

BugPilot ships with an automated evaluation harness (evaluation/) that scores the agent against a 23-query golden dataset across 11 dimensions — intent accuracy, tool selection, groundedness, hallucination rate, trajectory validity, instruction following, safety, and latency — without relying on manual grading. Run it yourself with:

python -m evaluation.run_eval

Latest committed results (evaluation_report.json):

The two non-passing queries were intent-routing edge cases (e.g. a query classified as COMPONENT_ANALYSIS instead of METRIC), not hallucinations or failures — the agent never fabricated information in any of the 23 test cases.

A concurrency/load test (1–50 simultaneous users) is also included via evaluation/load_tester.py. At up to 25 concurrent users the system holds a 0% error rate; at 50 concurrent users, error rate rises to ~66%, indicating the current single-instance setup is not yet tuned for high-concurrency production traffic. See Known Limitations below.

Note on cost/token figures: the estimated_total_cost_usd and average_tokens_per_query values in evaluation_report.json are word-count-based estimates, not real Groq API usage data. Treat them as rough indicators, not billing figures.

8. Known Limitations

In the interest of transparency for reviewers:

• Concurrency ceiling — load testing shows a sharp rise in error rate at 50 simultaneous users (see above). Suitable for demo/small-team use as-is; would need connection pooling / async tuning for larger production traffic.

• Estimated (not measured) token/cost tracking — the evaluation report's cost figures are heuristic estimates based on word counts, not real API usage accounting.

• Small golden evaluation set — the automated evaluation covers 23 representative queries; broader coverage (more adversarial/prompt-injection cases, more edge cases) would strengthen confidence further.

• generate_pdf.py is a standalone documentation-export utility with a Windows-specific default output path; pass an explicit filename argument on macOS/Linux.

---.