# windows2text

Current version: `0.5.0`.

`windows2text` directly reads the Windows Accessibility / UI Automation tree and emits structured JSON or compact model-friendly text. It does not use screenshots or OCR.

The native reader has no package dependencies and builds with the .NET Framework compiler included in Windows, so a separate .NET SDK is not required. The Node integration uses the official MCP TypeScript SDK and Zod.

## Build

```powershell
powershell -NoProfile -ExecutionPolicy Bypass -File .\build.ps1
```

The executable is written to `dist\windows2text.exe`. `windows2text.cmd` builds it automatically on first use.

Run the desktop smoke tests with:

```powershell
powershell -NoProfile -ExecutionPolicy Bypass -File .\test.ps1
```

## Usage

Read the foreground window as JSON:

```powershell
.\windows2text.cmd snapshot
```

Read it as compact text for an LLM or computer-use planner:

```powershell
.\windows2text.cmd text --max-depth 6 --max-nodes 300
```

List top-level windows, then capture one by handle or process ID:

```powershell
.\windows2text.cmd windows
.\windows2text.cmd snapshot --scope hwnd --hwnd 0x123456
.\windows2text.cmd snapshot --scope process --pid 1234
```

Inspect the element under the mouse pointer, including the raw accessibility tree:

```powershell
.\windows2text.cmd snapshot --scope cursor --view raw --max-depth 4
```

Supported scopes are `foreground`, `desktop`, `cursor`, `process`, and `hwnd`. Supported UIA views are `control`, `content`, and `raw`.

## Output contract

Each snapshot node can include:

- `ref`: snapshot-local ID such as `e12`
- `path`: tree path from the selected root
- `role`, `name`, `value`, and `text`
- `automationId`, `className`, and `framework`
- `processId`, `hwnd`, and UIA `runtimeId`
- `bounds`: physical screen rectangle for coordinate fallback
- `state`: enabled, focused, focusable, off-screen, selected, checked, etc.
- `patterns`: supported actions such as `Invoke`, `Value`, `Toggle`, and `Scroll`

`ref` and `runtimeId` are not durable across application restarts. A future action layer should resolve elements using a combination of process, window, AutomationId, role/name, tree path, and current bounds.

## Jev / computer-use integration

### MCP

The recommended agent integration is the stdio MCP server:

```powershell
npm install
npm run build
$env:TYPESAFE_API_KEY = "your-key"
npm run mcp
```

It exposes `windows_list`, `windows_observe`, `windows_snapshot`, `windows_decide`, `windows_act`, `windows_step`, and `windows_run`. Observation and explicit actions work without a Jev key; Jev-backed tools read `TYPESAFE_API_KEY` from the server process environment.

Use `windows_run` for multi-step tasks. The complete UIA/Jev/action loop stays inside the MCP server, so the host LLM makes one tool call and receives compact step summaries rather than a full tree after every action.

`windows_decide` and `windows_step` send the goal, application name, projected accessibility labels and values, notes, previous action, and proposed input text to the configured TypeSafe endpoint. Use observation-only tools when desktop content must remain local.

See [MCP.md](MCP.md) and the ready-to-edit configs under [`examples`](examples/).

### Modular Node API

The package exports three independent layers:

- `Windows2TextClient`: long-lived Windows UIA observation and action process
- `JevClient`: direct TypeSafe System One API client
- `WindowsComputerUse`: observation, Jev decision, confidence gate, and execution loop

Set the API key only in the process environment:

```powershell
$env:TYPESAFE_API_KEY = "your-key"
```

Do not put the key in source files or command arguments.

One gated decision without execution:

```js
import { WindowsComputerUse } from "./lib/index.mjs";

const computer = new WindowsComputerUse();
try {
  const result = await computer.step("Open the settings page");
  console.log(result.decision);
} finally {
  await computer.close();
}
```

Execute one step only when Jev's safety gate returns `execute`:

```js
const result = await computer.step("Open the settings page", { act: true });
```

Run a bounded loop:

```js
const result = await computer.run("Open the settings page", {
  act: true,
  maxSteps: 8,
  getText: async ({ decision }) => {
    // Jev selects the type action and target but does not generate text.
    return generateTextFor(decision);
  },
});
```

`WindowsComputerUse` accepts injected `windows` and `jev` clients, so applications can replace either side with a mock, another transport, or a custom policy.

### Combined CLI

Build first, then use the dependency-free Node CLI:

```powershell
npm run build
node .\bin\windows2text-jev.mjs observe
node .\bin\windows2text-jev.mjs decide --goal "Open Settings"
node .\bin\windows2text-jev.mjs step --goal "Open Settings" --act
```

`decide` is always dry-run. `step` and `run` are also dry-run unless `--act` is supplied. Even with `--act`, `confirm`, `escalate`, and `abort` gates are never executed automatically.

Run the synthetic end-to-end Jev test without touching user applications:

```powershell
powershell -NoProfile -ExecutionPolicy Bypass -File .\test-live.ps1
powershell -NoProfile -ExecutionPolicy Bypass -File .\test-live.ps1 -Act
```

The test compiles its own WinForms fixture, asks Jev to choose an action, and with `-Act` verifies the result by reading the fixture's UIA state again.

### Raw Protocol

For observations only, execute the CLI and parse stdout:

```text
windows2text.exe snapshot --scope foreground --max-depth 8 --max-nodes 500
```

For observation plus actions, keep one helper process running:

```text
windows2text.exe serve
```

Send one JSON request per stdin line and read one JSON response per stdout line:

```json
{"id":1,"method":"observe","params":{"scope":"foreground","maxDepth":8,"maxNodes":500}}
{"id":2,"method":"invoke","params":{"ref":"s1:e12"}}
{"id":3,"method":"type_text","params":{"ref":"s1:e18","text":"hello","clear":true}}
```

The server supports UIA actions (`invoke`, `set_value`, `set_range`, `toggle`, `select`, `expand`, `collapse`, `scroll`) and input fallbacks (`click`, `type_text`, `key`). See [PROTOCOL.md](PROTOCOL.md) for the complete transport and action contract.

Each server snapshot replaces the reference cache. References such as `s1:e12` therefore cannot accidentally resolve to a different element after a new observation.

The `observe` result contains `app`, an ordered Jev-ready `visible` string array, and an equally ordered `targets` array. This matches the public [`Jevbridge`](https://github.com/tacticocc/Jevbridge) computer-use shape: pass `goal`, `app`, and `visible` to `jev_computer_use`, obey its confidence gate, map the selected visible label back through `targets`, then execute the corresponding ref here.

[`examples/jev-adapter.mjs`](examples/jev-adapter.mjs) provides compatibility exports for the modular clients:

```js
import { WindowsComputerUse } from "./examples/jev-adapter.mjs";

const computer = new WindowsComputerUse();
const result = await computer.step("Open the settings page");
await computer.close();
```

Jevbridge actions map as follows: `click` uses `InvokePattern` when available and coordinate click otherwise; `type` uses `ValuePattern` or Unicode input; `scroll` uses UIA scrolling; `wait` stays local; `screenshot` requests a fresh accessibility observation; `done` and `abort` terminate the loop. A `confirm`, `escalate`, or `abort` gate is never executed automatically.

## Limitations

- UIA cannot read text painted on a canvas, games, video, or inaccessible custom controls. Those require OCR or vision fallback.
- A normal process cannot inspect elevated applications reliably. Run `windows2text` at the same integrity level as the target.
- The Windows secure desktop cannot be inspected.
- Browser and Electron accessibility trees depend on the application's accessibility support and settings.

## Related projects

- [FlaUI](https://github.com/FlaUI/FlaUI): a full-featured MIT-licensed .NET UIA2/UIA3 wrapper; recommended if the project later adopts a modern .NET SDK.
- [pywinauto](https://github.com/pywinauto/pywinauto): a mature BSD-licensed Python Win32/UIA automation library.
- [Windows-MCP](https://github.com/CursorTouch/Windows-MCP): useful reference for accessibility snapshots, action IDs, and MCP computer-use design.
- [Accessibility Insights for Windows](https://github.com/microsoft/accessibility-insights-windows): Microsoft's open-source accessibility inspection tool and a useful robustness reference.