# Computer Use Adapter for DevSpace Ultra

A small stdio MCP adapter that makes desktop app discovery machine-readable while preserving native accessibility text and screenshot content.

It connects [DevSpace Ultra](https://github.com/enwong93-sketch/devspace-ultra) to [Open Computer Use](https://github.com/iFurySt/open-codex-computer-use). The desktop automation engine belongs to Open Computer Use; this repository contains the integration layer, parser, and tests. It is an independent community integration.

## What it does

```text
ChatGPT / MCP client
  → DevSpace capability_call
  → this adapter
  → Open Computer Use
  → accessibility tree + native PNG
```

- Adds `structuredContent.apps` to recognized `list_apps` results and advertises an output schema in the tool catalog.
- Preserves the original discovery text, upstream structured metadata, and error flags.
- Passes other tool results through, including native MCP image blocks.
- Does not guess app identifiers or special-case Calculator in the parser.
- Does not include a desktop automation engine or any production credentials/state.

The wrapper uses Node built-ins only. It launches the external command `npx -y open-computer-use mcp`; npm may download that upstream package on first use. The upstream version is not pinned by this adapter, so its output format and behavior can change. The SDK dependency in this repository is used only by the optional desktop smoke test.

## Requirements

- Node.js 22 or later and npm/npx on `PATH`. Desktop acceptance was performed on Windows with Node 24.19.0; other desktop platforms have not been accepted by this project.
- An interactive desktop session supported by Open Computer Use, with any permissions required by that engine.
- DevSpace Ultra with **native MCP content forwarding** for screenshots to reach the outer consumer. A text-only capability wrapper is insufficient.

Our accepted DevSpace build contained native forwarding and bounded MCP sessions at local commit `4d5951fa55e380d5b5d314e12aa792687d893f05`. Those core changes are separate from this adapter and are not included here; do not assume the upstream default branch or a package labeled `0.4.0` contains them. Text/structured discovery and native image support are distinct requirements.

## Install into DevSpace

Clone this repository on the machine running DevSpace:

```sh
git clone https://github.com/ftll574/devspace-computer-use-adapter.git
```

Call DevSpace's `capability_install` with the **absolute local path to the plugin subdirectory**, not the repository root:

```json
{
  "source": "<absolute-path>/devspace-computer-use-adapter/plugins/open-computer-use-devspace",
  "enable": true,
  "trust": true
}
```

Enabling and trusting a capability permits it to run local code and expose desktop controls. Review the adapter and upstream engine before enabling it. The adapter's instructions guide consumers; they are not a sandbox or enforcement boundary.

Then inspect the installed package:

```json
{"pluginId":"open-computer-use-devspace","probeMcp":true}
```

The tested upstream exposed nine tools: `list_apps`, `get_app_state`, `click`, `drag`, `perform_secondary_action`, `press_key`, `scroll`, `set_value`, and `type_text`. The live catalog remains authoritative.

## Discover before observing

Invoke `capability_call`:

```json
{
  "pluginId": "open-computer-use-devspace",
  "kind": "mcp",
  "serverId": "open-computer-use",
  "toolName": "list_apps",
  "arguments": {}
}
```

Example normalized entry (illustrative; never reuse this identifier without discovery):

```json
{
  "apps": [{
    "appId": "example-app",
    "name": "Example App",
    "running": true,
    "pid": 1234,
    "windowTitle": "Example Window"
  }]
}
```

Direct adapter clients receive `structuredContent.apps`. Through DevSpace's capability envelope this is `structuredContent.result.structuredContent.apps`.

Select an explicitly authorized app from that fresh result, then invoke `get_app_state` with its `appId` as `arguments.app`. Do not guess identifiers; stop if the target is absent or ambiguous. A working native-forwarding host returns text and image content blocks. There is no need to paste base64 into a prompt.

## Tests

Pure parser/response tests and syntax checks need no installed dependencies or desktop access:

```sh
npm test
npm run check
```

They cover Unicode, optional titles, malformed lines, invalid PIDs, metadata preservation, error passthrough, schema injection, and unchanged screenshot responses. GitHub Actions runs these checks on Linux and Windows with Node 22 and 24; CI does not automate a desktop.

Optional **read-only Windows Calculator** smoke test:

```sh
npm ci --ignore-scripts
npm run smoke:calculator
```

Open Calculator first and keep it fully visible. The smoke test lists apps, selects exactly one running Calculator, reads its state, and checks for text plus PNG. It sends no clicks or keystrokes and prints only summary counts. App discovery still exposes app names and window titles to the local client.

## Validation and limitations

The adapter plugin files originate from lab commit `1723cd25e69f10507ddb5a6010309e92e482d195`. They were accepted through both a direct MCP client and an existing ChatGPT → fixed endpoint → DevSpace connection: discover apps → identify Calculator → receive UI tree and native PNG. No app identifier was supplied to the ChatGPT consumer during that acceptance.

- Discovery parsing targets the observed `id -- name [running, pid=..., window=...]` format. Unrecognized lines remain in raw text; structured coverage may be partial or empty.
- If upstream already provides `structuredContent.apps`, it is preserved rather than rewritten.
- Screenshots may contain an occluding window. Keyboard reliability is an upstream limitation; this adapter does not fix either issue.
- General-purpose desktop actions, all Windows applications, and macOS/Linux desktop behavior have not been comprehensively validated here.
- DevSpace's session memory fix and production upgrade/rollback tooling are separate core/operations work, not features of this adapter.

## Contributing and license

Run `npm test` and `npm run check` before submitting a change. Use synthetic fixtures; do not attach tokens, desktop screenshots containing private information, SQLite state, or production logs to issues.

Adapter code is MIT licensed; see [LICENSE](LICENSE). Open Computer Use is a separate MIT-licensed upstream project, copyright its contributors. No upstream engine source is bundled or relicensed here. Dependencies retain their own licenses.