machinegrade validate

Validate AI-generated artifacts against a contract before you act on them:

• json_schema — validate artifact against a JSON Schema (ajv, all errors collected).

• openapi_response — validate a response body against the response schema for a given path + method + status in an OpenAPI spec.

• sql — check a SQL string for syntax errors in a given dialect.

Every check returns a verdict, not an error: {valid, errors, latency_ms}, HTTP 200 whether the artifact is valid or not. Only genuinely wrong requests (bad key, unsupported type, malformed body, over your limit) get typed HTTP errors.

Built on Hono — one codebase, runs locally on Node today and is written to be Cloudflare Workers-compatible for deploy later (see "Deploy" below).

Why

Agents that generate JSON, API responses, or SQL need a fast, cheap, machine-checkable pass/fail before they ship the result — cheaper than a full LLM-as-judge call, and deterministic.

Related MCP server: perf-mcp

Run it locally

npm install
npm run dev
# machinegrade validate listening on http://localhost:8787

3 runnable examples

1. curl

# Get an API key
curl -s -X POST http://localhost:8787/keys \
  -H 'content-type: application/json' \
  -d '{"email": "you@example.com"}'
# => {"key":"sk_..."}

# Validate a JSON artifact against a JSON Schema
curl -s -X POST http://localhost:8787/v1/validate \
  -H 'content-type: application/json' \
  -H 'X-Api-Key: sk_...' \
  -d '{
    "type": "json_schema",
    "artifact": {"name": "Ada", "age": 30},
    "contract": {
      "schema": {
        "type": "object",
        "required": ["name", "age"],
        "properties": {"name": {"type": "string"}, "age": {"type": "number"}}
      }
    }
  }'
# => {"valid":true,"errors":[],"latency_ms":1}

2. Python (requests)

import requests

base = "http://localhost:8787"

key = requests.post(f"{base}/keys", json={"email": "you@example.com"}).json()["key"]

resp = requests.post(
    f"{base}/v1/validate",
    headers={"X-Api-Key": key},
    json={
        "type": "sql",
        "artifact": "SELECT id, name FROM users WHERE id = 1",
        "contract": {"dialect": "mysql"},
    },
)
print(resp.status_code, resp.headers.get("X-Calls-Remaining"), resp.json())

3. MCP config snippet

mcp/server.ts exposes a single tool, validate, that forwards to POST /v1/validate. Point an MCP-compatible client at it:

{
  "mcpServers": {
    "machinegrade-validate": {
      "command": "npx",
      "args": ["tsx", "mcp/server.ts"],
      "cwd": "/path/to/validate",
      "env": {
        "SANDBOX_URL": "http://localhost:8787",
        "SANDBOX_API_KEY": "sk_..."
      }
    }
  }
}

API

See public/openapi.yaml for the full contract, or /v1/manifest for a machine-readable summary (types, limits, pricing, error codes) once the service is running. /llms.txt is a short pointer for LLM agents.

Pricing

• Free tier: 500 calls/month per key, 60 calls/minute rate limit.

• Paid tier: EUR 0.002/call beyond the free tier — opens soon. Request paid access via POST /v1/paid-request (requires X-Api-Key); you'll be notified when it's live.

Errors

Every error is typed JSON — {code, message, hint, docs_url} — never a free-form string:

A verdict ({valid, errors, latency_ms}) is never an error — an invalid artifact is a normal, expected outcome and returns HTTP 200.

Storage

src/storage.ts defines a Storage interface with two implementations:

• MemoryStorage — full in-memory implementation, used for npm run dev and the test suite.

• D1Storage — real Cloudflare D1 binding, backed by schema.sql (keys, events tables). Used in production; the Workers entry point in src/index.ts builds it from the DB binding on first request.

Apply schema.sql to a new D1 database with:

wrangler d1 execute machinegrade-validate-db --file=schema.sql          # local
wrangler d1 execute machinegrade-validate-db --file=schema.sql --remote # production

Testing

npm test          # vitest run, in-process via app.request(), MemoryStorage
npm run typecheck # tsc --noEmit

Tests cover: key issuance, happy + fail cases for each validator, typed 401/400/402/429 errors, the metering limits (both injectable in tests so they don't require looping hundreds of real requests), and /stats funnel counts.

Deploy

This template runs on Cloudflare Workers (Hono + D1 + Workers Static Assets). To deploy to a fresh Cloudflare account:

wrangler d1 create machinegrade-validate-db   # copy the returned database_id into wrangler.toml
wrangler d1 execute machinegrade-validate-db --file=schema.sql --remote
wrangler secret put ADMIN_TOKEN
wrangler deploy

Then bind a custom domain (e.g. api.machinegrade.dev) to the Worker via the Cloudflare dashboard or wrangler. CI can deploy on push to main once CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID repo secrets are set and the deploy job in .github/workflows/ci.yml is uncommented.

Two things worth knowing about the Workers port:

• GET /openapi.yaml and GET /llms.txt are served by the ASSETS binding ([assets] in wrangler.toml, pointing at public/) — Cloudflare serves them directly, without invoking the Worker. The routes in src/index.ts are a fallback for local Node dev/tests, where there's no ASSETS binding.

• The json_schema and openapi_response validators use @cfworker/json-schema, not ajv: ajv compiles schemas via new Function(...), which the Workers runtime disallows, and schemas here arrive dynamically per request (from the caller), so they can't be precompiled at build time either.

Status

Early stage, honestly so: this service is live and free-tier usage is real, and we're measuring whether it earns a paid tier. What you can rely on:

• The API contract (/v1/validate request/response shapes, typed error codes, verdict semantics) is stable — breaking changes only with a versioned path (/v2/...), never silently.

• The free tier (500 calls/month) stays.

• If we ever sunset the service, keys keep working for 90 days after the announcement, and the validators are open source in this repo — you can self-host the same behavior.

Feedback and integration stories are the most valuable thing you can give us right now: open an issue or use POST /v1/paid-request if you need more than the free tier.