# PText

**Paste anything. Get clean data.**

[![CI](https://github.com/mr-sonj/ptext-core/actions/workflows/ci.yml/badge.svg)](https://github.com/mr-sonj/ptext-core/actions/workflows/ci.yml)
[![Release](https://img.shields.io/github/v/release/mr-sonj/ptext-core?sort=semver)](https://github.com/mr-sonj/ptext-core/releases)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
![Node](https://img.shields.io/badge/node-%E2%89%A5%2022-brightgreen)
![Core dependencies](https://img.shields.io/badge/core%20dependencies-0-brightgreen)

PText is a set of deterministic text tools for extracting, cleaning, splitting into columns, converting, encoding and formatting text. It also has a local profiler that looks at a paste and suggests the right tool. **An AI may pick the tool, but code always produces the result**, so the output is exact, repeatable and testable.

One engine ships as a library, an MCP server, a CLI and OpenAI/Anthropic tool schemas.

- **Zero runtime dependencies.** Pure ESM TypeScript, about 17 KB gzipped. The core uses no DOM and no Node APIs.
- **Runs anywhere JS runs:** browser, Cloudflare Workers, Node ≥ 22, Deno, Bun.
- **Private by design.** No network calls, no telemetry. The optional AI ranker sees only counts and flags, never your text.
- **Never loses data silently.** Anything skipped, guessed or changed is reported as a warning code.
- **Contract-tested.** Every tool has language-neutral golden fixtures that any port (PHP, Python…) can run.

## Contents

- [Tools](#tools)
- [Install](#install)
- [Quick start](#quick-start)
- [Library](#library)
- [CLI](#cli)
- [MCP server](#mcp-server)
- [AI function calling](#ai-function-calling)
- [Choosing a tool with any AI](#choosing-a-tool-with-any-ai)
- [PHP / Laravel / other languages](#php--laravel--other-languages)
- [Privacy model](#privacy-model)
- [Project layout](#project-layout)
- [Development](#development)
- [Releases](#releases)
- [Contributing](#contributing) · [Security](#security) · [License](#license)

## Tools

| Tool | Category | What it does |
| --- | --- | --- |
| `extract-emails` | extract | Find all email addresses in any text (logs, HTML, CSV, chat). Supports + tags and Unicode. |
| `extract-urls` | extract | Find http(s):// and www. links in text, trimming trailing punctuation and unbalanced brackets. |
| `extract-domains` | extract | Extract hostnames from URLs, emails and plain text. Returns full hostnames (subdomains kept); does not compute registrable domains. |
| `extract-ip` | extract | Find valid IPv4 and IPv6 addresses (incl. compressed IPv6, IPv4-mapped IPv6 and ip:port) in text. |
| `clean-lines` | clean | Trim lines, remove empty lines, collapse repeated spaces and strip invisible characters (zero-width, BOM). |
| `remove-duplicates` | clean | Remove duplicate lines while keeping the first occurrence and original order. |
| `filter-lines` | clean | Keep (or drop) lines that contain, start with, end with, equal, or match a regex pattern. |
| `split-columns` | columns | Split delimited lines (e.g. user\|pass\|mail, tab-separated, quoted CSV) into columns and output CSV. |
| `select-columns` | columns | Pick and reorder columns by 1-based index ("3,1", "2-4", "-1" = last cell of each row) from delimited text. |
| `list-to-csv` | convert | Turn a plain list (one item per line) into a single-column CSV, quoting values safely. |
| `csv-to-json` | convert | Convert CSV/TSV (quoted fields supported) into a JSON array of objects (with header) or arrays. |
| `base64` | encode | Encode text to Base64 or decode Base64 (standard or URL-safe) to UTF-8 text. Decoded content is only returned as text, never executed. |
| `url-encode` | encode | Percent-encode or decode text (component or full-URI mode). Decoding "+" as space is supported in form mode. |
| `json-format` | format | Validate JSON and pretty-print or minify it. Reports line and column of syntax errors. |

Every tool's options are plain JSON Schema with descriptions and defaults. To see the full catalog, including option schemas and input cards, run `ptext list` or call `listTools()`.

## Install

| Package | What it is | Runtime |
| --- | --- | --- |
| [`@ptext/core`](packages/core) | Tools, catalog, profiler, router, AI adapters | Any JS runtime |
| [`@ptext/mcp`](packages/mcp) | MCP server: `ptext-mcp` stdio bin, `createPTextServer()`, `createFetchHandler()` | Node (stdio) · Workers, Deno, Bun (HTTP) |
| [`@ptext/cli`](packages/cli) | `ptext` command: stdin → stdout, JSON mode | Node |

```bash
npm install @ptext/core        # library
npm install -g @ptext/cli      # `ptext` command
npx -y @ptext/mcp              # MCP server over stdio
```

> **Not on npm yet.** Until the first npm release, build from source:
> ```bash
> git clone https://github.com/mr-sonj/ptext-core.git && cd ptext-core
> pnpm install && pnpm build
> node packages/cli/dist/cli.js list
> ```

## Quick start

```ts
import { suggestTools, runTool } from '@ptext/core';

const text = 'Contact sales@acme.io or Support@Acme.io, see https://acme.io/help.';

suggestTools(text).suggestions[0];   // { tool: 'extract-emails', score: …, reason: '…' }
runTool('extract-emails', text).output;   // "sales@acme.io\nSupport@acme.io"
```

```bash
echo 'a@x.com, b@y.io, a@x.com' | ptext run extract-emails
# a@x.com
# b@y.io
```

## Library

```ts
import { listTools, runTool, suggestTools, PTextError } from '@ptext/core';

const r = runTool('select-columns', 'id,name,mail\n1,Ann,a@x.com', { columns: '3,1' });
r.output;     // copy-ready text: "mail,id\na@x.com,1"
r.data;       // structured result when useful (list, rows, objects)
r.stats;      // { inputLines, outputLines, count?, removed? }
r.warnings;   // [{ code: 'RAGGED_ROWS', message, line? }, …], never silent

const { profile, suggestions } = suggestTools(text);   // local and deterministic, < 1 ms on typical input
```

`runTool(id, input, options?, { limits? })` throws a `PTextError` with a `code` from a closed set:

| Code | Meaning | CLI exit |
| --- | --- | --- |
| `UNKNOWN_TOOL` | No tool with that id | 2 |
| `INVALID_OPTION` | Option fails the schema, or the combination makes no sense | 2 |
| `INVALID_INPUT` | The tool cannot process this input (`details.line` when known) | 3 |
| `INPUT_TOO_LARGE` | Input exceeds `limits.maxInputChars` (default 5,000,000) | 3 |
| `INTERNAL` | A bug. Anything that is not a `PTextError`. Please report it | 1 |

`errorToJSON(e)` turns any error into `{ code, message, details? }`, the same shape the CLI, MCP server and Worker return.

## CLI

```
ptext <list|catalog|prompt|jev|suggest|run <tool>|exec> [--option=value] [--json]
```

| Command | Reads | Prints |
| --- | --- | --- |
| `ptext list` | | Every tool, with option schema and input card (JSON) |
| `ptext run <tool> [--opt=value]` | text on stdin | The tool's output. Warnings go to stderr. Add `--json` for the full result |
| `ptext suggest` | text on stdin | `{ profile, suggestions }` |
| `ptext exec` | `{"tool","input","options"?}` on stdin | `{ ok: true, …result }` or `{ ok: false, error }` |
| `ptext catalog` | | Profile field meanings and every tool's input card, for an AI ranker |
| `ptext prompt` | | The ranking prompt and reply schema for any LLM |
| `ptext jev` | | The static part of a Jev router request |
| `ptext --version` | | Version |

```bash
printf 'user|pass|mail\nann|x1|a@x.com\n' | ptext run split-columns
cat access.log | ptext run extract-ip --version=4 --unique=true
echo '{"tool":"json-format","input":"{\"b\":1,\"a\":2}","options":{"sortKeys":true}}' | ptext exec
```

Exit codes: `0` ok · `2` usage or option error · `3` invalid input · `1` internal error.

## MCP server

The server exposes every tool as `ptext_<id>` (for example `ptext_extract_emails`), plus `ptext_suggest` (profile and suggestions for a text) and `ptext_catalog` (field meanings and input cards). Each tool description includes `Use when: …` and `Not for: …` so the client's model picks the right tool.

### Local (stdio)

Claude Desktop, Cursor, Windsurf, Codex and any other client that reads an `mcpServers` config:

```json
{
  "mcpServers": {
    "ptext": { "command": "npx", "args": ["-y", "@ptext/mcp"] }
  }
}
```

Claude Code:

```bash
claude mcp add ptext -- npx -y @ptext/mcp
```

From a source checkout, use `"command": "node", "args": ["/absolute/path/to/ptext-core/packages/mcp/dist/stdio.js"]`.

### Remote (Streamable HTTP on Cloudflare Workers, Deno, Bun)

```ts
import { createFetchHandler } from '@ptext/mcp';
import { CfWorkerJsonSchemaValidator } from '@modelcontextprotocol/sdk/validation/cfworker';

const mcp = createFetchHandler({ path: '/mcp', jsonSchemaValidator: new CfWorkerJsonSchemaValidator() });
export default { fetch: (req: Request) => mcp(req) };
```

On Workers, pass `CfWorkerJsonSchemaValidator`. The default validator (Ajv) uses `new Function`, which Workers block. [`examples/cloudflare-worker`](examples/cloudflare-worker) adds a REST API next to MCP:

| Route | Body | Returns |
| --- | --- | --- |
| `POST /mcp` | MCP JSON-RPC | MCP Streamable HTTP (stateless) |
| `POST /api/run` | `{ tool, input, options? }` | Tool result |
| `POST /api/suggest` | `{ input }` | `{ profile, suggestions }` |
| `GET /api/tools` | | Tools with input cards |
| `GET /api/catalog` | | Field meanings and input cards |
| `GET /api/prompt` | | Ranking prompt and reply schema |

```bash
cd examples/cloudflare-worker && npx wrangler dev
```

## AI function calling

```ts
import { openAITools, anthropicTools, handleToolCall } from '@ptext/core';

// Pass openAITools() or anthropicTools() as `tools`. When the model calls one:
const result = handleToolCall(call.name, call.arguments);   // never throws, JSON-serializable
```

Each description comes from `modelDescription(tool)`: what the tool does, `Use when: …` and `Not for: …`. The MCP server uses the same text.

## Choosing a tool with any AI

```
input → profileInput()            counts, ratios and flags only (PROFILE_FIELDS describes each one)
      → each tool's suggest()     local, deterministic
      → route(input, { router })  optional: an AI re-ranks from the profile and the catalog
```

Every tool carries an **input card**: `accepts` (what the input looks like), `signals` (profile fields that point to it), `avoid` (a look-alike and the tool to use instead) and `examples`. Rankers never see the raw text. They get the profile and the input cards.

```ts
import { route, llmRouter } from '@ptext/core';

// Any model: send prompt.system + prompt.user, return the reply text.
const ai = llmRouter(async (prompt, signal) => callYourModel(prompt.system, prompt.user, signal));
const { suggestions, source, issues } = await route(text, { router: ai, timeoutMs: 1500 });
```

- The router is asked even when local rules find nothing, and it may return any catalog tool.
- Malformed entries are dropped and listed in `issues`. A timeout, an error, or a reply that is not a ranking falls back to the local result, so routing never blocks.
- Generative models use `llmRouter(complete)` with the shared `rankPrompt()`, `rankResponseSchema()` and `parseRankResponse()`. Decision models such as Jev ([TypeSafe System One](https://typesafe.ai)) use `jevRouter(post)`. In both cases your code makes the HTTP call and holds the key.
- Backends in other languages can fetch the same catalog and prompt with `ptext catalog` / `ptext prompt` or from the Worker at `GET /api/catalog` and `GET /api/prompt`.

## PHP / Laravel / other languages

Call the CLI with a JSON request:

```bash
echo '{"tool":"extract-emails","input":"a@x.com"}' | ptext exec
```

[`examples/laravel`](examples/laravel/README.md) shows both setups: `@ptext/core` in the frontend (Vite + Alpine), where text never leaves the browser, and a backend that calls the CLI through `Process`.

## Privacy model

- The core runs inside your process: browser tab, MCP process or Worker. It makes no network calls and sends no telemetry.
- `profileInput()` returns only counts, ratios and flags. A test checks that no raw text leaks into it.
- Decoded content (Base64, URL encoding) is returned as text and never executed.
- When PText runs as an MCP server inside an AI client, that client may still send your text to its own model provider. PText cannot make that whole chain offline, and does not claim to.

## Project layout

```
packages/core     tools/<category>/<id>.ts (one file per tool) · profiler · router · AI adapters
packages/mcp      transport-agnostic server · fetch handler (HTTP) · stdio bin
packages/cli      ptext command (runCli is a pure, tested function)
fixtures/         golden/<id>.json (tool contract) · suggest.json (profiler benchmark)
examples/         cloudflare-worker · laravel · browser (parity harness)
scripts/          new-tool.mjs (scaffold) · release.mjs (versioning, changelog, release notes)
```

## Development

Requirements: Node ≥ 22.12 (the test runner needs it; the packages need Node ≥ 22) and pnpm 10 (`corepack enable`).

```bash
pnpm install
pnpm build && pnpm test          # golden fixtures, catalog rules, profiler corpus, MCP parity, CLI, Worker
pnpm typecheck
pnpm test:tool extract-emails    # one tool's cases
```

Adding a tool takes two files: the tool and its fixture. Scaffold both with:

```bash
pnpm new-tool csv-to-yaml --category=convert
```

The scaffold stays red until the fixture cases, the input card and the README row are filled in. [CONTRIBUTING.md](CONTRIBUTING.md) lists the rules the tests enforce.

Browser parity harness: run `pnpm build`, serve the repo root (`npx serve .`), then open `/examples/browser/`.

## Releases

Versions follow [Semantic Versioning](https://semver.org/) and come from [Conventional Commits](https://www.conventionalcommits.org/): `feat!:` / `BREAKING CHANGE:` → major (minor while on `0.x`), `feat:` → minor, anything else → patch.

Each release is one `chore(release): vX.Y.Z` commit that writes the version to every `package.json` and to `VERSION` and adds a [CHANGELOG.md](CHANGELOG.md) section, tagged `vX.Y.Z`. CI builds and tests the tag, then publishes a [GitHub release](https://github.com/mr-sonj/ptext-core/releases) with that changelog section. Packages are not published to npm yet.

## Contributing

Issues and pull requests are welcome. Read [CONTRIBUTING.md](CONTRIBUTING.md) before you open a pull request that adds or changes a tool.

## Security

Please report vulnerabilities privately. See [SECURITY.md](SECURITY.md).

## License

[MIT](LICENSE)