# 🐝 HiveMind - MCP Subagent Orchestrator

[![TypeScript](https://img.shields.io/badge/TypeScript-5.0+-blue.svg)](https://www.typescriptlang.org/)
[![Node.js](https://img.shields.io/badge/Node.js-20+-green.svg)](https://nodejs.org/)
[![License](https://img.shields.io/badge/License-ISC-yellow.svg)](LICENSE)

A high-concurrency, asynchronous **MCP (Model Context Protocol) Orchestrator** that coordinates multiple AI coding agents using the Copilot CLI as the underlying intelligence engine. Built on an **Actor Model** architecture for non-blocking operations, robust fault tolerance, and strict resource synchronization.

## 📋 Table of Contents

- [Overview](#-overview)
- [Key Features](#-key-features)
- [Architecture](#-architecture)
- [Components](#-components)
- [Installation](#-installation)
- [Usage](#-usage)
- [Configuration](#-configuration)
- [API Reference](#-api-reference)
- [Development](#-development)
- [Project Structure](#-project-structure)
- [Roadmap](#-roadmap)
- [Contributing](#-contributing)

## 🎯 Overview

HiveMind enables a **Main Agent** to spawn and coordinate multiple **Subagents**, each running isolated Copilot CLI instances. This allows for:

- **Parallel task execution** across multiple files/modules
- **Hierarchical planning** with dependency-aware scheduling
- **Safe concurrent file access** via lock management
- **Unified observability** with structured logging and tracing

### Problem Solved

Traditional single-agent coding assistants struggle with:
- Large codebases requiring parallel analysis
- Multi-file refactoring with dependencies
- Long-running tasks that block the main thread
- Resource contention when multiple tools access files

HiveMind solves these by orchestrating a **swarm of specialized agents** that work concurrently while respecting file locks and execution order.

## ✨ Key Features

| Feature                       | Description                                                                        |
| ----------------------------- | ---------------------------------------------------------------------------------- |
| **🚀 Autonomous Execution**  | Submit a task and let the orchestrator plan, schedule, and execute without polling |
| **🔒 File Locking (Warden)** | Pessimistic locking with deadlock detection and wait queues                        |
| **📊 Hierarchical Planning** | Break complex tasks into dependency graphs with parallel execution groups          |
| **🔌 MCP Integration**       | Exposes orchestration capabilities as MCP tools                                    |
| **📈 Observability**         | Structured logging, OpenTelemetry tracing, and real-time metrics                   |
| **💾 Persistence**           | SQLite-based checkpointing with crash recovery                                     |
| **🎛️ Dashboard**           | REST API + WebSocket for real-time monitoring                                      |

## 🏗️ Architecture

HiveMind uses a three-layer architecture:

```
┌─────────────────────────────────────────────────────────────────────────────┐
│                              CONTROL PLANE                                  │
│  ┌─────────────┐  ┌──────────────┐  ┌────────────┐  ┌─────────────────────┐ │
│  │ Orchestrator│  │Task Planner  │  │ Scheduler  │  │ Resource Warden     │ │
│  │    Core     │──│ (Decomposer) │──│            │──│ (Lock Manager)      │ │
│  └─────────────┘  └──────────────┘  └────────────┘  └─────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
                                     │
                                     ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                               DATA PLANE                                    │
│  ┌──────────────────────────┐            ┌──────────────────────────────┐   │
│  │      Event Bus (IPC)     │            │      Trace Aggregator        │   │
│  │  Pub/Sub Message Passing │            │  Logs, Metrics, Spans        │   │
│  └──────────────────────────┘            └──────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────────────┘
                                     │
                                     ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                             EXECUTION PLANE                                 │
│  ┌───────────┐  ┌───────────┐  ┌───────────┐       ┌───────────────────┐   │
│  │ Agent Pod │  │ Agent Pod │  │ Agent Pod │  ...  │  Copilot CLI      │   │
│  │     1     │  │     2     │  │     N     │       │  (Child Process)  │   │
│  └───────────┘  └───────────┘  └───────────┘       └───────────────────┘   │
└─────────────────────────────────────────────────────────────────────────────┘
```

### Request Flow

1. **Main Agent** calls MCP tool (e.g., `submit_task_auto`)
2. **Planner** analyzes task, creates dependency graph
3. **Scheduler** creates execution groups respecting dependencies
4. **Warden** acquires file locks before each group executes
5. **Agent Pods** spawn Copilot CLI processes in parallel
6. Results aggregated and returned to Main Agent

## 🧩 Components

### Core Components

| Component             | Location           | Description                                   |
| --------------------- | ------------------ | --------------------------------------------- |
| **Orchestrator Core** | `src/core/`        | Central coordination, lifecycle management    |
| **Agent Manager**     | `src/agents/`      | Agent pool, pod lifecycle, message routing    |
| **Event Bus**         | `src/ipc/`         | Pub/sub messaging, async communication        |
| **Lock Manager**      | `src/locks/`       | File locking, deadlock detection, wait queues |
| **Planner**           | `src/planner/`     | Task decomposition, dependency graphs         |
| **CLI Adapter**       | `src/cli/`         | Copilot CLI spawning, I/O handling            |
| **MCP Layer**         | `src/mcp/`         | Tool definitions, request handling            |
| **Tracing**           | `src/tracing/`     | Structured logging, OpenTelemetry             |
| **Persistence**       | `src/persistence/` | SQLite, checkpointing, recovery               |
| **Dashboard**         | `src/dashboard/`   | REST API, WebSocket server                    |

### Type System

| Module                | Description                       |
| --------------------- | --------------------------------- |
| `src/types/agent.ts`  | Agent status, config, results     |
| `src/types/job.ts`    | Job lifecycle, subtasks, progress |
| `src/types/lock.ts`   | Lock modes, handles, conflicts    |
| `src/types/events.ts` | Event bus messages, channels      |
| `src/types/mcp.ts`    | MCP tool schemas, responses       |
| `src/types/cli.ts`    | CLI command construction, options |

## 📦 Installation

### Prerequisites

- **Node.js** 20+
- **npm** or **yarn**
- **GitHub Copilot CLI** installed and authenticated
- **Git** for version control

### Install

```bash
# Clone the repository
git clone https://github.com/sureshsankaran/mcp-subagent-orchestrator.git
cd mcp-subagent-orchestrator

# Install dependencies
npm install

# Build the project
npm run build
```

### Verify Installation

```bash
# Run tests
npm test

# Check build output
ls dist/
```

### Quick Demo

Run the end-to-end demo to see all features in action:

```bash
# Run comprehensive demo showcasing all phases (1-7)
npm run demo:e2e

# Run basic component demo
npm run demo
```

The demo showcases:
- **Event Bus** - Pub/sub messaging with wildcard patterns
- **State Machine** - Agent lifecycle transitions
- **Lock Manager** - Resource locking with deadlock detection
- **DAG** - Task dependency graphs with parallel execution groups
- **MCP Tools** - 14+ orchestration tools exposed via MCP
- **Structured Logging** - Winston logger with JSON/pretty formats
- **Distributed Tracing** - OpenTelemetry integration
- **Metrics Registry** - Counters, histograms, and gauges
- **Health Checks** - Component health monitoring
- **Full Orchestration** - End-to-end task execution simulation

## 🚀 Usage

### As an MCP Server

Register HiveMind as an MCP server in your AI tool configuration:

```json
{
  "mcpServers": {
    "hivemind": {
      "command": "node",
      "args": ["path/to/mcp-subagent-orchestrator/dist/index.js"],
      "env": {
        "GH_TOKEN": "your-github-token"
      }
    }
  }
}
```

### Programmatic Usage

```typescript
import { OrchestratorCore, OrchestratorConfig } from 'mcp-subagent-orchestrator';

// Create orchestrator
const config: OrchestratorConfig = {
  maxConcurrentAgents: 5,
  workspacePath: '/path/to/workspace',
  persistence: {
    enabled: true,
    dbPath: '.hivemind/state.db'
  }
};

const orchestrator = new OrchestratorCore(config);
await orchestrator.initialize();

// Spawn a single agent
const agent = await orchestrator.spawnAgent('Fix the authentication bug in auth.ts');
const result = await orchestrator.waitForAgent(agent.id);

// Submit an autonomous job
const job = await orchestrator.submitJob('Refactor the entire API layer to use async/await');
const status = await orchestrator.waitForJob(job.id);
```

### MCP Tool Examples

```typescript
// Spawn a focused subagent
await mcp.callTool('spawn_agent', {
  task: 'Review auth.ts for security vulnerabilities',
  config: {
    system_instruction: 'You are a security expert. Be thorough.',
    allowed_paths: ['src/auth/'],
    read_only: true
  }
});

// Submit autonomous job (orchestrator handles everything)
await mcp.callTool('submit_task_auto', {
  instruction: 'Add comprehensive unit tests to all service files',
  strategy: 'parallel',
  max_agents: 4
});

// Monitor job progress
await mcp.callTool('get_job_status', { job_id: 'job-123' });

// Acquire file lock (for manual coordination)
await mcp.callTool('acquire_lock', {
  resource: 'src/api/users.ts',
  mode: 'exclusive',
  timeout_ms: 30000
});
```

## ⚙️ Configuration

### Environment Variables

| Variable              | Description                           | Default              |
| --------------------- | ------------------------------------- | -------------------- |
| `GH_TOKEN`            | GitHub token with Copilot access      | Required             |
| `HIVEMIND_LOG_LEVEL`  | Logging level (debug/info/warn/error) | `info`               |
| `HIVEMIND_MAX_AGENTS` | Maximum concurrent agents             | `10`                 |
| `HIVEMIND_DB_PATH`    | SQLite database path                  | `.hivemind/state.db` |
| `HIVEMIND_PORT`       | Dashboard server port                 | `3000`               |
| `HIVEMIND_PREFERRED_CLI_TYPE` | Override detected CLI type (`copilot_v2`, `copilot_v1`, `codex`, `gh_copilot`, `claude`) | auto-detect |
| `HIVEMIND_PREFERRED_CLI_PATH` | Absolute path to CLI binary (bypass detection) | auto-detect |
| `OTEL_ENABLED`        | Enable OpenTelemetry                  | `false`              |

### Orchestrator Config (file-first)

Configuration is loaded in this order: `user overrides` → `env` → `hivemind.config.json` → defaults.

Add a `hivemind.config.json` at the repo root to set project-wide defaults (used for both decomposition and spawned agents):

```jsonc
{
  "cli": {
    // Force a specific adapter type instead of auto-preference (copilot_v2 > copilot_v1 > codex > gh_copilot > claude)
    "preferred_cli_type": "codex",
    // Optional explicit binary path (skips detection)
    "preferred_cli_path": "/usr/local/bin/codex",
    // Existing options
    "cli_command": "copilot",
    "default_timeout_ms": 300000,
    "auto_approve_tools": true
  }
}
```

```typescript
interface OrchestratorConfig {
  // Agent management
  maxConcurrentAgents: number;       // Max parallel agents
  agentTimeout: number;              // Agent execution timeout (ms)
  agentPoolSize: number;             // Pre-warmed agent pool size
  
  // Workspace
  workspacePath: string;             // Root workspace path
  trustedFolders: string[];          // Pre-trusted Copilot folders
  
  // Locking
  lockTimeout: number;               // Default lock acquisition timeout
  deadlockDetection: boolean;        // Enable deadlock detection
  
  // Persistence
  persistence: {
    enabled: boolean;
    dbPath: string;
    checkpointInterval: number;      // Checkpoint frequency (ms)
  };
  
  // Observability
  logging: {
    level: 'debug' | 'info' | 'warn' | 'error';
    format: 'json' | 'pretty';
  };
  
  // Security
  toolApproval: {
    allowedShellCommands: string[];
    deniedShellCommands: string[];
  };

  // CLI (decomposition + spawned agents)
  cli: {
    cli_command: string;
    preferred_cli_type?: 'copilot_v2' | 'copilot_v1' | 'codex' | 'gh_copilot' | 'claude' | 'unknown';
    preferred_cli_path?: string;
    default_timeout_ms: number;
    auto_approve_tools: boolean;
  };
}
```

## 📖 API Reference

### MCP Tools

| Tool                  | Description                           |
| --------------------- | ------------------------------------- |
| `spawn_agent`         | Spawn a single subagent with a task   |
| `submit_task_auto`    | Submit job for autonomous execution   |
| `get_job_status`      | Get current job status and progress   |
| `get_job_tree`        | Get hierarchical view of job subtasks |
| `cancel_job`          | Cancel a running job                  |
| `acquire_lock`        | Manually acquire a file lock          |
| `release_lock`        | Release a held lock                   |
| `list_locks`          | List all active locks                 |
| `stream_logs`         | Stream logs for job/agent             |
| `orchestrator_health` | Get orchestrator health status        |

### REST API Endpoints

| Endpoint                  | Method | Description        |
| ------------------------- | ------ | ------------------ |
| `/api/v1/agents`          | GET    | List all agents    |
| `/api/v1/agents`          | POST   | Spawn new agent    |
| `/api/v1/agents/:id`      | GET    | Get agent details  |
| `/api/v1/agents/:id`      | DELETE | Kill agent         |
| `/api/v1/jobs`            | GET    | List all jobs      |
| `/api/v1/jobs`            | POST   | Submit new job     |
| `/api/v1/jobs/:id`        | GET    | Get job details    |
| `/api/v1/jobs/:id/cancel` | POST   | Cancel job         |
| `/api/v1/locks`           | GET    | List active locks  |
| `/api/v1/system/health`   | GET    | Health check       |
| `/api/v1/system/metrics`  | GET    | Prometheus metrics |

## 🛠️ Development

### Scripts

```bash
# Build TypeScript
npm run build

# Watch mode
npm run build:watch

# Run tests
npm test

# Run tests with coverage
npm run test:coverage

# Lint code
npm run lint
npm run lint:fix

# Format code
npm run format

# Clean build artifacts
npm run clean
```

### Testing

```bash
# Run all tests
npm test

# Run specific test file
npm test -- tests/unit/locks/LockManager.test.ts

# Run with coverage
npm run test:coverage
```

### Code Quality

- **ESLint** for linting
- **Prettier** for formatting
- **Jest** for testing
- **TypeScript** strict mode enabled

## 📁 Project Structure

```
mcp-subagent-orchestrator/
├── src/
│   ├── agents/          # Agent pod management
│   ├── cli/             # Copilot CLI adapter
│   ├── core/            # Orchestrator core
│   ├── dashboard/       # REST API & WebSocket
│   ├── errors/          # Custom error classes
│   ├── ipc/             # Event bus & messaging
│   ├── locks/           # Lock manager (Warden)
│   ├── mcp/             # MCP tool definitions
│   ├── persistence/     # SQLite & checkpointing
│   ├── planner/         # Task decomposition
│   ├── tracing/         # Logging & telemetry
│   ├── types/           # TypeScript interfaces
│   ├── utils/           # Shared utilities
│   └── index.ts         # Main entry point
├── tests/
│   ├── unit/            # Unit tests
│   └── integration/     # Integration tests
├── tasks/               # Phase task breakdowns
├── dist/                # Compiled output
├── spec.md              # Full specification
├── TASKS.md             # Development roadmap
└── package.json
```

## 🗺️ Roadmap

The project is developed in 9 phases:

| Phase | Name                          | Status          | Tasks |
| ----- | ----------------------------- | --------------- | ----- |
| 1     | Project Skeleton & Interfaces | ✅ Complete     | 68    |
| 2     | Nervous System (EventBus/IPC) | 🔄 In Progress | 52    |
| 3     | CLI Abstraction Layer         | ⏳ Planned       | 68    |
| 4     | File Warden (Lock Manager)    | ⏳ Planned       | 58    |
| 5     | Hierarchical Planner          | ⏳ Planned       | 72    |
| 6     | MCP Exposure                  | ⏳ Planned       | 58    |
| 7     | Observability                 | ⏳ Planned       | 52    |
| 8     | Persistence & Recovery        | ⏳ Planned       | 62    |
| 9     | Dashboard & REST API          | ⏳ Planned       | 55    |

**Total: ~542 detailed subtasks**

See [TASKS.md](TASKS.md) for detailed task breakdowns.

## 🤝 Contributing

1. Fork the repository
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request

### Development Guidelines

- Follow TypeScript strict mode
- Write tests for new features
- Update documentation as needed
- Use conventional commit messages

## 📄 License

This project is licensed under the ISC License - see the [LICENSE](LICENSE) file for details.

---

<p align="center">
  Built with 🐝 by the HiveMind team
</p>