PipSync MCP Evals

Public safety evaluations and a no-execution MCP mock for trading-oriented tool designs.

The repository models three boundaries:

• read-only synthetic signal listing;

• a deterministic, non-executable paper preview;

• a live-capable confirmation schema that the public mock always blocks.

The mock has no network client, API-key loader, credential store, broker adapter, or order-placement implementation.

Run the evaluations

PYTHONPATH=src python -m pipsync_mcp_evals.evals
PYTHONPATH=src python -m unittest discover -s tests -v

Related MCP server: TrustHarness

Run the stdio mock

PYTHONPATH=src python -m pipsync_mcp_evals.server

It accepts one JSON-RPC message per line and implements the modern, stateless MCP 2026-07-28 surface: server/discover, tools/list, and tools/call. Every request must carry io.modelcontextprotocol/protocolVersion and io.modelcontextprotocol/clientCapabilities in params._meta; every successful result carries resultType and server identity metadata. Legacy initialize is rejected with an explicit migration hint and the supported protocol version.

What the evaluations enforce

• Every tool has explicit MCP annotations.

• Tool input schemas reject unknown fields.

• Confirmation is classified as non-read-only and destructive.

• Confirmation requires account, preview, confirmation, idempotency, and risk-hash fields.

• Missing confirmation data fails closed.

• Even structurally complete confirmations return mock_server_no_execution.

• Preview output is deterministic and explicitly non-executable.

• Missing request metadata and unsupported protocol versions fail with the current MCP error contracts.

Annotations are hints, not an authorization boundary. Any real write-capable server must independently enforce authenticated tenant/account ownership, entitlement, fresh broker state, risk gates, explicit live mode, idempotency, audit, and human confirmation before every broker call.

The catalog targets the MCP 2026-07-28 tools contract and its stateless discovery contract.

License

Apache-2.0.