Generates a tested connector that writes records into Airtable bases, mapping fields from an upstream event payload.
Generates a tested connector that reacts to Calendly scheduling events and forwards booking details to another service.
Generates a tested connector that adds or updates contacts in a Mailchimp audience, including applying tags derived from upstream event data.
Generates a tested connector that uses Shopify events (e.g. new orders) as a trigger and syncs the resulting records into another service.
Generates a tested connector that posts messages or notifications to Slack channels from an upstream event.
Generates a tested connector that reacts to Stripe events (e.g. a successful payment) and forwards the resulting data to another service.
New in 0.2.4:
patchbay audit. Free CLI to scan an MCP server for OAuth 2.1 gaps. Read the November 2026 sweep (25 public MCP servers scanned, 2 of 8 reachable-and-public ones expose mutating tools without auth) or try it:npx -y @patchbay-nebius/agent audit https://mcp.example.com/mcp21 live checks + 1 retired, against the MCP 2026-07-28 authorization spec and the OAuth 2.1 / RFC 8252 / 7591 / 8707 / 9068 / 9207 / 9728 stack it builds on. Context-sensitive severity,
--pr-draftfor CI,--authfor walled servers. Jump to the audit section for the full flag list.Hosted scheduled scans + alerts coming Q1 2027 for teams. CLI stays free and MIT forever. Interest check · email waitlist.
New in 0.4.0: shareable scan URLs + badge. Add
--publishto your audit and get back a public report URL plus an embeddable score badge:[](https://patchbay-mcp.dev/report/<id>)New in 0.5.0: AUTH-006 (RFC 8252 §7.3 loopback redirect check) + GitHub Action. Drop this in your CI to fail builds on MCP OAuth gaps:
- uses: GodSmasher/patchbay-audit@v1 with: url: https://your-mcp-server.example.com/mcp fail-on: highSeparate repo: GodSmasher/patchbay-audit · SARIF output into code scanning supported.
Describe an integration in one sentence. Get a tested connector (TypeScript or Python) that already passed its own tests.
# TypeScript (vitest + injected fetch)
patchbay build "When a Stripe payment succeeds, add the customer to a Mailchimp audience with the plan as a tag"
# Python 3.12 (pytest + injected async transport)
patchbay build --lang=python "When a Stripe payment succeeds, add the customer to a Mailchimp audience with the plan as a tag"90.2% green on TS Core 20 · 73.3% green on Python Core 20 · 70.0% on Python Brutal 20. Measured across 192 live Nebius Sandbox runs with per-run JSON committed to the repo. Fine-tune live on Nebius AI Cloud (Qwen3-30B-A3B on 173 of patchbay's own red→green repair pairs, served from an H100 NVLink GPU behind an OpenAI-compatible vLLM endpoint). Dual-language toggle on the live site at https://patchbay-mcp.dev, public uptime at https://patchbay.openstatus.dev, open-source training dataset on 🤗 patchbay-nebius/connector-repair-dataset.
Every row is a live Nebius Sandbox run with the per-attempt files, test report and sandbox operation id committed under bench/runs/. Not cherry-picked cases. Full tables and per-prompt breakdowns in bench/RESULTS-variance.md.
Set | Shape | Runs | Green | First-try | Cost / build |
TS Core 20 | common integrations (Stripe→Mailchimp, Shopify→Airtable, …) | 92 | 90.2% | 52.2% | $0.07 |
TS Brutal 10 | adversarial (OAuth refresh, WebSocket, HMAC, cursor paging) | 13 | 69.2% | 23.1% | $0.18 |
Python Core 20 | same 20 prompts as TS Core, Python+pytest sandbox | 60 | 73.3% | 31.7% | $0.07 / green build |
Python Brutal 20 | adversarial (SOAP, GraphQL federation, long-poll, dedup) | 40 | 70.0% | — | $0.16 |
TS × Nemotron Super ablation | strong-step downgraded 550B → 120B, same prompts | 60 | 46.7% | 8.3% | $0.06 |
Reading these: the common-case Python gap (17 points vs TS Core) closes on hard cases. Python Brutal matches TS Brutal within one point. The Super ablation shows Ultra isn't overkill: dropping to the mid-tier halves the green rate because the saved per-call cost gets eaten by extra repair attempts.
Related MCP server: Snyk API & Web MCP Server
patchbay runs five steps:
Plan. A fast model turns the sentence into source/target apps, trigger, action, docs queries, field mapping. Nemotron 3.5 Lightning.
Research. Tavily search + extract, biased toward each vendor's own domain, feeding a mid-tier model that produces a normalized API spec (endpoints, auth, example payloads). Nemotron 3 Super.
Generate. A strong model writes the connector, its tests and a README. TypeScript (src/connector.ts + vitest) by default; --lang=python writes connector.py + pytest. Nemotron 3 Ultra, 32k tokens, temperature 0.1.
Verify. A Nebius Token Factory sandbox pulls the right base image once and installs the test deps (vitest + typescript, or pytest + pytest-asyncio + pytest-json-report) into a checkpointed image. Every attempt forks that checkpoint with the generated files and networking disabled. For TS the gate is tsc --noEmit then vitest run; for Python it is python -m compileall then pytest -q --json-report.
Repair. When a run is red, the strong model gets the file blocks and the failed test output, then tries again. Up to PATCHBAY_MAX_ATTEMPTS times (default 3). Optional parallel mode generates N variants at once, verifies them in parallel sandbox forks, keeps the first green.
Every endpoint in the spec comes with an example payload taken from the documentation; the generated test harness runs the connector against that payload (the "contract" check), so a run only counts as green when the generated code actually works with the shape the real API returns.
TypeScript and Python share one plan + research step — only the generate prompt, the sandbox image and the test harness swap. The hosted demo at patchbay-mcp.dev has a TypeScript · Python · Both toggle; "Both" fires two live Nebius pipelines side-by-side so you watch the same prompt land in both languages at once.
The Python target ships fixtures.py with the same shape as src/fixtures.ts: SOURCE_EXAMPLE, TARGET_ENDPOINTS, contract_fetch(), fake_fetch(), json_response(). Connectors export async def handle(payload, config) with an injected async transport in config['fetch'], so tests stay hermetic (no network, no real httpx client). URL matcher handles placeholder segments ({id}, :id, <id>) and region subdomains. See bench/RESULTS-variance.md for the per-prompt breakdown.
Carve-outs: PATCHBAY_RUNNER=local only runs TS. The hosted site's recorded replays are TS-only recordings, so without live keys the Python toggle shows the TS replay with a note.
patchbay audit <url-or-directory> scans an MCP server or a connector repo for OAuth 2.1 and secret-hygiene gaps before you ship it. ~40% of open-source MCP servers have no auth at all, only ~8.5% use OAuth. patchbay audit is the drop-in check.
npx -y @patchbay-nebius/agent audit https://mcp.example.com/mcp
npx -y @patchbay-nebius/agent audit ./my-mcp-server --json
npx -y @patchbay-nebius/agent audit ./my-mcp-server --sarif > results.sarif21 live checks + 1 retired:
ID | Severity | What it catches |
| critical |
|
| critical | Static API key or bearer accepted, no |
| high | OAuth metadata without PKCE S256 in |
|
| Checked |
| high / info | High when neither CIMD nor DCR advertised. Info when only DCR is advertised (deprecated per MCP 2026-07-28). |
| high | Authorization server rejects loopback redirect URIs (RFC 8252 §7.3). Opt-in via |
| medium | Invalid bearer token returns a status other than HTTP 401 (MCP 2026-07-28 mandates 401). |
| critical | Server accepts bearer tokens minted for a sibling resource on the same AS (RFC 8707). Opt-in via |
| critical | Server accepts a bearer token whose JWT payload has no |
| high | Missing or malformed Protected Resource Metadata (RFC 9728). MCP MUST since 2025-06-18. |
| high | All tools share one scope or no scopes declared at all |
| medium | Write/destructive tools share scope with read tools |
| critical | Hardcoded |
| medium | Credential read from env but no refresh-token flow found |
| high | Remote server reachable over plain HTTP |
| medium | Server accepts a spoofed |
| medium | Token endpoint treats |
| medium | Weak transport-security response headers (wildcard CORS with credentials, missing HSTS on HTTPS, missing frame-options, missing nosniff). |
| high / medium | High for hidden Unicode in tool descriptions (zero-width, BOM, bidi, tag chars). Medium for imperative role-marker patterns ( |
| low | OAuth metadata does not advertise |
| varies | Tool manifest changed since last pinned scan ( |
| varies | Classifies the manifest delta into breaking (removed tool, removed param, type narrowed, enum shrunk, made-required), semantic-drift (description changed), and additive buckets. One finding per non-empty bucket. |
| info | Probe inconclusive, cannot judge auth posture |
See patchbay-phase2-plan.md for the roadmap (--with-auth connector templates, --pr-draft, GitHub Action).
Outputs: pretty terminal (default, with CTA), --json, --sarif for GitHub Actions. --ci exits non-zero on critical/high so the audit gates your pipeline. --no-cta keeps CI logs clean.
Pure-core design. packages/audit-core has zero I/O inside the check functions: probing collects a TargetSnapshot, checks are (snapshot) => Finding[]. Trivial to unit-test, trivial for the repair model to consume later.
Every patchbay benchmark run is a record of the agent failing and then fixing itself. 173 of those (red draft + failing test report) → green draft moments across the committed benchmark passes make a domain-matched training set — and Nebius Token Factory ships an OpenAI-compatible fine-tune endpoint that eats exactly this format. So we fine-tuned Qwen/Qwen3-30B-A3B-Instruct-2507 on patchbay's own repair history. The deployment path uses a second Nebius product: trained checkpoints come out via /fine_tuning/jobs/{id}/checkpoints, land on a Nebius AI Cloud GPU (H100 NVLink preemptible) behind a vLLM OpenAI-compatible endpoint, and PATCHBAY_REPAIR_LLM_BASE_URL points patchbay's repair step at the self-hosted model. Three Nebius products in one closed loop: Token Factory Inference (live agent) → Token Factory Post-training (fine-tune) → AI Cloud GPU (serve the fine-tune).
bench/training-{train,eval}.jsonl — 98 train + 11 eval real red→green repair pairs, extracted from committed benchmark runs. Also mirrored on HuggingFace as patchbay-nebius/connector-repair-dataset with a train/eval split.
scripts/finetune.ts — upload / start / watch Nebius fine-tune jobs (takes --context=32768 to fit our 20k-token repair pairs).
scripts/finetune-checkpoint.ts — poll a job to completion, fetch the checkpoint URL, print the exact AI Cloud deploy next steps.
scripts/vllm-deploy/ — Dockerfile + run.sh to serve the fine-tuned weights on an H100 / H200.
scripts/bench-finetuned-ablation.ts — side-by-side fine-tuned vs Ultra-baseline over the TS Core 20.
scripts/eval-repair.ts — format, file-coverage and line-overlap metrics against the held-out eval split.
Full plan + status (updated as the job progresses): docs/finetune.md.
Every live build offers a one-click Evidence .zip download on the site. Trust but verify — the whole run is auditable without re-executing anything:
connector/ final files (ready to drop into a repo)
trajectory/
attempt-1/
<files>
report.json the test report for that attempt
attempt-2/ … every repair attempt the agent made
prompt.txt the one-sentence prompt
plan.json source app, target app, field mapping
spec.json researched API spec (endpoints, auth, example payloads)
usage.json per-call model usage + cost + total
manifest.json issuedAt, finalResult, sandbox operation IDs, SHA-256 of every file, verify instructions
README.md what each file is + how to recompute hashesThe sandboxOperationIds in manifest.json are Nebius Token Factory operation IDs — your project's operation history retains the exact command patchbay ran, its exit code, and its stdout for each one. Recompute hashes locally with shasum -a 256; mismatch means somebody edited the artifact after export.
CLI – @patchbay-nebius/agent on npm
npx -y @patchbay-nebius/agent build "When a Stripe payment succeeds, add the customer to a Mailchimp audience"
# or
npm install -g @patchbay-nebius/agent
patchbay build [--record] [--lang=python] "<one-sentence integration>"
patchbay research [--lang=python] "<one-sentence integration>"
patchbay verify [--lang=python] <directory>
patchbay scan <directory-or-github-url> [--build-top=<n>] [--lang=python]
patchbay audit <url-or-directory> [--json] [--sarif] [--ci] [--no-cta]patchbay scan <dir-or-github-url> — autonomous repo scanner. Walks any repository (local directory OR a GitHub URL, which patchbay shallow-clones for you) and reports the integration opportunities patchbay could build: TypeScript Express/Next/Fastify webhook handlers that only contain TODO or throw not implemented, Python FastAPI/Flask routes with pass bodies or raise NotImplementedError, .env.example entries pointing at vendor APIs with no implementation, TODO/FIXME comments mentioning Stripe/Mailchimp/Shopify/… and vendor SDKs that are imported but barely used. Each opportunity ships with a ready-to-run prompt and a priority score; --build-top=N runs the full pipeline on the top N automatically.
npx -y @patchbay-nebius/agent scan https://github.com/<some>/<repo>MCP server – @patchbay-nebius/mcp on npm, stdio transport.
Four tools — scan_repo (any local path or GitHub URL, returns ranked opportunities with ready-to-run prompts; TS + Python aware), build_connector (full pipeline, lang: "ts" | "py"), research_api (plan + Tavily spec, no code), verify_code (sandbox verify + repair existing files, auto-detects language). Live progress + logging notifications.
claude mcp add patchbay -- npx -y @patchbay-nebius/mcpFull setup for Claude Code, Codex and Cursor in docs/mcp-setup.md.
Web app – apps/web, Next.js 14 on port 3400. Live demo at https://patchbay-mcp.dev. The same UI streams a live Nebius run or, past the 6/hour per-IP rate limit, the recorded replay. Shareable scan URLs (?scan=owner/name) and shareable run URLs. Public uptime at https://patchbay.openstatus.dev.
Slack – /patchbay <one-sentence integration> as a slash command. Setup in docs/slack-setup.md; the endpoint lives at apps/web/app/api/slack/command/route.ts on the same Vercel deploy.
GitHub PR – comment /patchbay build "<integration>" on any PR and the workflow in .github/workflows/patchbay-pr.yml runs the full pipeline, attaches the generated connector as an artifact and replies with the test score. Needs the three Nebius / Tavily secrets configured in the repo.
Same connector, same requirements. Rough per-connector cost:
Where | Cost | Factor |
patchbay (Nemotron on Nebius) | ~$0.07 | 1× |
Claude Code with Haiku 5.5 | ~$0.10 | 1.4× |
Claude Code with Sonnet 5.5 | ~$0.35 | 5× |
Claude Code with Opus 4.7 | ~$1.90 | 27× |
$0.07 vs ~$1.90 for the same connector. The tests run before you see the file.
The agent in your IDE is a generalist. For integration connectors — read the docs, map fields, write the thing, prove it works — that's work with hard structure, and a specialized pipeline on cheap models beats a general-purpose agent on expensive ones. Same reason you don't ask Opus to format JSON.
patchbay sits inside your editor as an MCP server (claude mcp add patchbay -- npx -y @patchbay-nebius/mcp), so you keep using Claude Code / Cursor / Codex for everything else — architecture, refactors, bug hunts — and offload connector work to patchbay. Your Claude usage bill goes down for the task Claude is slowest and most expensive at.
The generated connector is plain TypeScript — or Python 3.12 — in your repo, no runtime dependency on patchbay. Read it, change it, ship it.
git clone https://github.com/GodSmasher/patchbay
cd patchbay
cp .env.example .env.local # fill in NEBIUS + TAVILY keys
docker compose up --build
# open http://localhost:3400The Dockerfile is a three-stage build (deps → build → runtime) producing a ~90 MB image around Next.js's standalone server. docker-compose.yml wires the env vars through from .env.local; without live keys the web app streams the recorded replay instead.
What broke this week and how it got fixed is in docs/build-log.md. Short version: nested-contract regex (+2 benchmark points), parallel-repair fiasco (reverted), monotonic repair (never regress past the best attempt), timestamp prompt (don't reject the documented example), esbuild preflight (first-try rate 33% → 52%), Vercel monorepo pain (three fixes, one deploy), URL FIDELITY + idempotent-upsert worked examples for Python (closed 2 of the hardest TS→Python green-rate gaps).
npm install
cp .env.example .env.local # fill in NEBIUS_API_KEY, NEBIUS_AI_PROJECT, TAVILY_API_KEY
npm test # vitest, replay mode, no keys needed
npm run typecheck # tsc across workspaces
npm run check:nebius # auth probe for inference + sandboxes + Tavily
npm run patchbay -- build "..." # live run (needs keys)
npm run patchbay -- help # all commands and flags
npm run bench # the 20-prompt benchmark
npm run dev # Next.js app on :3400
npm run mcp # start the MCP server on stdioSee .env.example. Key knobs:
Variable | Default | What it does |
|
| Replay mode. Set to |
|
|
|
|
|
|
|
| Repair-loop cap. |
|
| N parallel repair branches per attempt (max 4). Best-of-N. |
| nvidia/Nemotron 3.5 Lightning / 3 Super / 3 Ultra | Model tiers. Any OpenAI-compatible base URL works. |
| (unset) / | Point patchbay at any OpenAI-compatible endpoint — including a self-hosted vLLM serving our fine-tuned Qwen3-30B. |
The generated connector has zero patchbay dependency – it's a plain TypeScript or Python file your project can drop in anywhere.
packages/
agent/ # the pipeline (plan, research, generate, verify, repair) + CLI + scanner
mcp/ # MCP server exposing scan_repo / build_connector / research_api / verify_code
audit-core/ # OAuth 2.1 / scope / secret checks for MCP servers (patchbay audit)
apps/
web/ # Next.js demo
bench/ # the 20-prompt benchmark + results + training data
docs/ # setup guides + build log + finetune plan
scripts/ # finetune launcher, checkpoint fetch, vllm-deploy, ablation bench
runs/ # recorded live runs (gitignored)The web demo is a Next.js app with one SSE route (/api/run). An in-memory IP rate limit caps live runs at PATCHBAY_MAX_RUNS_PER_HOUR (default 6) per hour and silently falls back to the recorded replay when exceeded, so a hug-of-death doesn't burn through credits. For Vercel: apps/web/vercel.json pins the route's function to a 300s timeout (needs a Pro plan; on Hobby, add ?replay=1 or set PATCHBAY_MOCK=true to keep every call under 10s).
Built for the Nebius × NVIDIA Global AI Hackathon. Track: Coding & Agentic Engineering.
Scan a remote MCP server URL (or a local MCP connector directory) for OAuth 2.1, RFC 8707 audience validation, RFC 8252 loopback redirects, OAuth metadata, scope posture, secret surface, tool manifest drift, and tool-description prompt injection patterns. Returns a letter grade, numeric score and the full findings list. Runs zero-auth probing by default (metadata-level checks only). Pass "authToken" to probe behind authentication. Pass "probeAudienceToken" to enable AUDIENCE-001 (cross-audience token check, RFC 8707). Pass "probeMissingAudToken" to enable AUDIENCE-002 (missing-aud-claim check, catches the if (aud && aud !== me) reject early-exit bug). Pass "probeDcrLoopback: true" to enable AUTH-006 (writes a throw-away client registration to the authorization server). The scanner never calls any of the server's tools.
Inputs: target (required), authToken, timeoutMs, probeDcrLoopback, +2 more.
Describe an integration in one sentence ("When X happens in A, do Y in B"). patchbay plans it, researches both APIs, generates a connector with tests and a README, then verifies it in a sandbox with up to N repair attempts. Default language is TypeScript (vitest); set lang: "py" for Python 3.12 (pytest). Falls back to the recorded TypeScript replay run when live API keys are missing.
Inputs: prompt (required), lang, maxAttempts.
Plan + Tavily-backed API research only. Returns the connector plan, the ranked documentation sources, and a normalized API spec (endpoints, auth, example payloads). No code is generated and no sandbox is used, so this is cheap and fast.
Inputs: prompt (required).
Walk a repository (local directory or shallow-cloned GitHub URL) and report the integration opportunities patchbay could build: webhook handlers with TODO/empty bodies, .env.example entries pointing at vendor APIs with no implementation, TODO/FIXME comments mentioning vendors, and vendor SDKs imported but barely used. Understands TypeScript/JavaScript (Express/Next/Fastify routes) AND Python (FastAPI/Flask routes, pass bodies, raise NotImplementedError). Returns a ranked list, each item with a ready-to-run prompt the agent can hand straight to build_connector. Instant, free (no LLM / sandbox), auditable regex-only output.
Inputs: path (required), topN.
Type-check and test an existing connector (files you provide as {path, content}) in an isolated Nebius sandbox, with up to N repair attempts from the strong model. For TypeScript (default) the gate is tsc + vitest; for Python (lang: "py") it is compileall + pytest. Requires live keys (NEBIUS_API_KEY, NEBIUS_AI_PROJECT); returns the final test report and the (possibly repaired) files.
Inputs: files (required), lang, maxAttempts.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
Scored across 5 tools
build_connector bundles research and verification that research_api and verify_code expose separately, so an agent may briefly consider which layer to use. The descriptions make the boundaries clear enough (cheap research-only vs. full build vs. existing-code verification), while scan_repo and audit_mcp_server are distinctly scoped.
All five tools follow a consistent snake_case verb_noun pattern: build_connector, research_api, scan_repo, verify_code, audit_mcp_server. The action and target are predictable across the set.
Five tools cleanly cover the core workflows: discover opportunities, research APIs, build connectors, verify code, and audit MCP security. There are no filler tools and the surface is not bloated.
The connector lifecycle is well covered from repository scanning through research, generation, and sandbox verification. Minor gaps exist around deployment/publishing or connector listing/management, and the MCP audit is somewhat orthogonal to the connector-building workflow.
Scan any MCP server for tool-poisoning, security, auth & license. Trust score before install.
Preflight MCP OAuth compatibility with structured evidence and remediation guidance.
Audits any remote agent or MCP URL for JSON-RPC compliance, latency SLA, OpenAPI schema, and HTTP.
Scan any website or MCP server for agent-trust-readiness; returns a signed, verifiable scorecard.
