# MCP Fleet Orchestrator

A [FastMCP](https://gofastmcp.com) server that lets a central agent such as [Hermes](https://github.com/NousResearch/hermes-agent) manage a fleet of edge nodes in natural language:

- **Onboard** new nodes: SSH checks, packages, Docker access, app clone, generated credentials.
- **Install skills** on nodes from the app's `skill-catalog.json`.
- **Deploy** the [Multimodal Weld Defect Detection](https://github.com/open-edge-platform/manufacturing-ai-suite/tree/main/industrial-edge-insights-multimodal) app and its MCP server with `make up_agentic MCP=true`. Deployment follows the shipped [`multimodal-deploy`](https://github.com/open-edge-platform/manufacturing-ai-suite/tree/main/industrial-edge-insights-multimodal/.github/skills/multimodal-deploy) skill: preflight, deploy, status, and smoke check.
- **Work with each node's MCP server** (`describe`, `start_pipeline`, `list_insights_data`, `run_agent`, …), one node at a time or across the fleet.

```text
 "Onboard 192.168.1.23 as weld_node3 and deploy the weld app with its MCP server"
                                   │
┌──────────────── central host ────┼───────────────────────────┐
│ Hermes ── skill: fleet-orchestrator                          │
│   └─ mcp_servers.fleet ─► fleet-orchestrator (127.0.0.1:8765)│
│                             ├ inventory.db (SQLite)          │
│                             ├ skill cache (sparse git clone) │
│                             ├ certs/, secrets.env (mode 600) │
│                             └ ssh / ssh -L tunnels           │
└──────────────────────────────┼───────────────────────────────┘
                 SSH (key-based)│              tunnel 1400N ─► node 127.0.0.1:3000
        ┌───────────────────────▼──────────────────────────────┐
        │ node: ~/.fleet/skills/multimodal-deploy               │
        │       ~/manufacturing-ai-suite/industrial-edge-…      │
        │       make up_agentic MCP=true                         │
        │       nginx (127.0.0.1:3000) ─► /mcp ─► multimodal-mcp-server │
        └───────────────────────────────────────────────────────┘
```

No agent daemon runs on the nodes. The orchestrator uses plain key-based SSH. Each node keeps `HOST_IP=127.0.0.1`, so its UI and `/mcp` endpoint stay on loopback and are reached only through an SSH tunnel owned by the orchestrator.

## Tools

| Tool | What it does |
| --- | --- |
| `register_node` | Add a node (name, host, SSH user/port, tags) and test key-based SSH |
| `list_nodes`, `node_status`, `remove_node` | Inventory, live status (`make status`, MCP reachability), forget a node |
| `onboard_node` *(job)* | Install missing packages (passwordless `sudo` only) and check Docker. Clone the app. Fill **empty** `.env` credentials with random valid values and pin `HOST_IP=127.0.0.1`. Store the node's MCP login centrally |
| `list_skills`, `install_skill` | Read the app's skill catalog; copy a skill to `~/.fleet/skills/<skill>` on the node and verify its SHA-256 |
| `stage_models` *(job)* | Download the vLLM base model and the weld LoRA adapter as in the app's user guide. Requires `accept_model_license=true` |
| `deploy_app` *(job)*, `deploy_fleet` | Run the skill's deploy flow for a variant: `default`, `vllm`, `agentic`, or `agentic_mcp` (= `make up_agentic MCP=true`). With `agentic_mcp`, the job also connects the node's MCP server |
| `teardown_app` *(job)* | `make down MCP=true` (requires `confirm=true`) |
| `get_job`, `list_jobs` | Poll long-running jobs; returns the result and log tail |
| `connect_node_mcp` | (Re)open the tunnel, pin the node's current nginx certificate, wait for `tools/list` |
| `list_node_tools`, `call_node_tool`, `fleet_call_tool` | Use a node's own MCP server. `start_pipeline`/`stop_pipeline` require `confirm=true` |
| `register_node_in_hermes` | Optional: add the node's MCP server directly to Hermes as `mcp__<node>__<tool>` |

Long operations return a `job_id` right away, so tool calls don't hit agent timeouts. The `fleet-orchestrator` skill tells the agent to poll `get_job`.

## Install on the central host

Requirements: Linux, Python 3.11+, `ssh`, `git`, and key-based SSH to every node. Each node needs an `apt`-based OS, a deployment account in the `docker` group (or passwordless `sudo` so onboarding can install packages), and the app's [hardware requirements](https://docs.openedgeplatform.intel.com/dev/edge-ai-suites/ai-suite-manufacturing/industrial-edge-insights-multimodal/get-started/system-requirements.html).

```bash
git clone https://github.com/vkb1/mcp-fleet-orchestrator.git
python3 -m venv ~/.local/share/fleet-orchestrator/venv
~/.local/share/fleet-orchestrator/venv/bin/pip install ./mcp-fleet-orchestrator
```

Run it as a long-lived service, so jobs and tunnels survive between chat sessions:

```bash
install -D -m 0644 mcp-fleet-orchestrator/deploy/fleet-orchestrator.service ~/.config/systemd/user/fleet-orchestrator.service
systemctl --user daemon-reload
systemctl --user enable --now fleet-orchestrator
loginctl enable-linger "$USER"
```

The HTTP endpoint binds to `127.0.0.1:8765` and requires a bearer token. On first start the token is written to `~/.fleet-orchestrator/api-token` (mode `600`); `fleet-orchestrator token` prints its path. The service runs without your SSH agent, so the node SSH key it uses must be readable without a passphrase, or set `SSH_AUTH_SOCK` in `~/.config/fleet-orchestrator.env`.

### Configuration

Set these in the environment or in `~/.config/fleet-orchestrator.env`:

| Variable | Default | Purpose |
| --- | --- | --- |
| `FLEET_REPO_URL` | `https://github.com/open-edge-platform/manufacturing-ai-suite.git` | Repository cloned on nodes and used as the skill source |
| `FLEET_REPO_BRANCH` | `main` | Branch for both |
| `FLEET_SKILLS_REPO` | *(unset)* | Use a local checkout as the skill source instead of cloning |
| `FLEET_REMOTE_ROOT` | `manufacturing-ai-suite` | Clone location under the node user's home |
| `FLEET_HOME` | `~/.fleet-orchestrator` | Inventory, logs, certificates, secrets |
| `FLEET_TUNNEL_BASE_PORT` | `14000` | Node *N* gets local port `14000+N` (kept clear of the `1300N` tunnels from `hermes-setup-v2.md`) |
| `FLEET_DEPLOY_TIMEOUT` | `3600` | Seconds allowed for one deployment |
| `FLEET_STRICT_HOST_KEYS` | `false` | `true`: only connect to nodes whose host key is already in `$FLEET_HOME/known_hosts` (otherwise trust on first use) |
| `FLEET_API_TOKEN` | *(generated)* | Bearer token for HTTP mode (32+ characters) |
| `FLEET_HERMES_BIN`, `HERMES_HOME` | `hermes`, `~/.hermes` | Used by `register_node_in_hermes` |

## Connect Hermes

Store the token in Hermes's `.env` without echoing it, then register the server and install the skill:

```bash
printf 'FLEET_API_TOKEN=%s\n' "$(cat ~/.fleet-orchestrator/api-token)" >> ~/.hermes/.env
chmod 600 ~/.hermes/.env
hermes config set mcp_servers.fleet.url "http://127.0.0.1:8765/mcp"
hermes config set mcp_servers.fleet.headers.Authorization 'Bearer ${FLEET_API_TOKEN}'
install -d ~/.hermes/skills/devops
cp -r mcp-fleet-orchestrator/skills/fleet-orchestrator ~/.hermes/skills/devops/
hermes mcp test fleet
```

Stdio also works (`command: ~/.local/share/fleet-orchestrator/venv/bin/fleet-orchestrator`, `args: ["serve", "--transport", "stdio"]`). In that mode, running jobs and tunnels stop when the Hermes session ends.

## Use it

```bash
hermes -s fleet-orchestrator chat
```

> Register 192.168.1.21 as weld_node1 and 192.168.1.22 as weld_node2, both tagged line1, user deploy. Onboard them.

> Install the multimodal-deploy skill on both and deploy the agentic app with the MCP server on line1.

> What tools does weld_node1's MCP server have? Describe every node.

> Start the weld pipeline on weld_node2 with vision on GPU and time series on CPU. *(The agent asks you to confirm.)*

> Show the latest 10 insights from weld_node2, run the agent over the last 1m and fetch the results.

> Tear down weld_node1. *(The agent asks you to confirm.)*

If key-based SSH is not set up for a node, `register_node` returns the `ssh-copy-id` command to run in your own terminal. Passwords never go through the chat.

## Security model

- **No new network exposure.** Node MCP endpoints stay on loopback and are reached through SSH tunnels. The orchestrator binds to loopback and requires a bearer token. Binding elsewhere needs `--allow-remote` plus `--ssl-certfile`/`--ssl-keyfile`.
- **SSH host keys.** The orchestrator uses its own `$FLEET_HOME/known_hosts`. By default a node's key is trusted on first use. For stricter setups, set `FLEET_STRICT_HOST_KEYS=true` and add verified keys first, for example `ssh-keyscan -H <node> >> ~/.fleet-orchestrator/known_hosts` after checking the fingerprint out of band.
- **Untrusted tool input.** Every argument that reaches SSH, a path, or a config key is validated against a strict pattern. Remote commands are fixed scripts bundled with this package, with each argument shell-quoted. The agent cannot pass free-form shell.
- **Secrets stay out of chats and command lines.** Generated `.env` values are sent over SSH stdin. Node MCP logins are stored in `~/.fleet-orchestrator/secrets.env` (mode `600`) and are never returned by a tool.
- **Pinned TLS.** Each node's self-signed nginx certificate is fetched over the authenticated SSH channel and is the only trust anchor for that node. nginx regenerates the certificate on every deploy, so the pin is refreshed the same way. Only Python 3.13's strict X.509 profile is relaxed, because the app's certificate has no keyUsage extension.
- **Allow-listed node tools.** Only the app's known MCP tools can be called. `start_pipeline` and `stop_pipeline` need `confirm=true`, and so do `teardown_app` and `remove_node`. `stage_models` needs explicit license acceptance. These flags are set by the agent, so they guard against accidental calls, not against a compromised agent. Node tool output is data from the node, and the bundled skill tells the agent to act only on what the user asked.
- **Least privilege on nodes.** Onboarding uses `sudo -n` only, so it never prompts. The deployment user's Docker access is root-equivalent, so use a dedicated account with key-only SSH.
- **One orchestrator per `FLEET_HOME`.** A lock file enforces it, because per-node job locks and tunnels live in the process. Tunnels left behind by a crash are detected through their pidfiles and replaced. Cancelling a job stops the local `ssh` client. A remote `make` that is already running may still finish on the node.

## Development

```bash
python3 -m venv .venv && .venv/bin/pip install -e '.[dev]'
.venv/bin/ruff check . && .venv/bin/ruff format --check .
.venv/bin/pytest
# Run the end-to-end test against the real multimodal-deploy skill:
FLEET_TEST_SKILLS_REPO=/path/to/manufacturing-ai-suite .venv/bin/pytest tests/test_e2e_fake_node.py
```

`tests/test_e2e_fake_node.py` runs the full flow: register, onboard, skill install, deploy with `MCP=true`, node MCP calls with Basic auth over a tunnel with a pinned CN=localhost certificate, and teardown. It runs against a fake node built from fake `ssh` and `docker` binaries, a stand-in Makefile, and an HTTPS FastMCP server.

## License

Apache-2.0