🧪 LabBench

A personal platform for learning ML & systems by building — host entire courses (CS336-style problem sets), write multi-file solutions in the browser, run tests in sandboxes with arbitrary machine types, and get help from Claude both in-site and from Claude Code over MCP.

What it does

• Problem-set style problems — each problem has a multi-part markdown statement (KaTeX math supported) in a sidebar, closer to a pset than a LeetCode prompt. Courses are just directories of markdown + yaml in content/ (see content/README.md).

• Multi-file editor — Monaco-based workspace on the right: create/delete files, autosave, Cmd/Ctrl+S. Python + PyTorch out of the box; the runner itself is language-agnostic (anything the sandbox image can execute).

• Sandboxed test runs — hit Run tests and the platform executes the problem's run_tests.py against your files in a sandbox and renders per-case results + logs in the sidebar. Two backends:

  • local: subprocess on the server (dev, stdlib-only problems)

  • modal: Modal Sandboxes with per-problem machine specs — image, cpu, memory, GPUs (T4…H100), and pip packages like torch

• Google login, allowlist-only — only emails in ALLOWED_EMAILS can sign in.

• One shared MCP endpoint (/api/mcp) for Claude Code:

  tool	what it does
  list_problems / get_problem	browse all coursework
  get_workspace / get_run	read your current files, runs, logs
  run_tests	kick off a sandbox run
  post_hint	drop a hint into the problem's chat sidebar
  assign_task	assign you a brand-new project (statement + starter + tests), shows up under "Assigned by Claude"

• In-site tutor chat — per-problem Claude chat that sees the statement, your files, and your latest test run. MCP hints land in the same thread.

Related MCP server: Canvas LMS MCP Server

Setup

npm install
cp .env.example .env   # then fill it in
npx prisma db push     # create tables
npm run dev

You need:

1. Postgres — any DATABASE_URL (Neon / Supabase / Vercel Postgres / local).

2. Google OAuth — create an OAuth client at console.cloud.google.com/apis/credentials, authorized redirect URI https://<your-domain>/api/auth/callback/google (or http://localhost:3000/... for dev). Put the id/secret in AUTH_GOOGLE_ID / AUTH_GOOGLE_SECRET, and set AUTH_SECRET (openssl rand -base64 32).

3. ALLOWED_EMAILS — your email. Nobody else can log in.

Optional:

4. Anthropic — ANTHROPIC_API_KEY enables the in-site tutor chat.

5. Modal — MODAL_TOKEN_ID/MODAL_TOKEN_SECRET (from modal.com/settings/tokens) enables cloud sandboxes with GPUs and pip installs. With RUNNER_BACKEND=auto, problems run on Modal when it's configured and locally otherwise; per-problem machine.backend overrides.

6. MCP_TOKEN — openssl rand -hex 32; required for the MCP endpoint.

Deploying (Vercel)

Push this repo, import into Vercel, set the env vars above. npm run build runs prisma generate automatically; run npx prisma db push against your production DB once. The MCP endpoint and OAuth need the public URL, so set the Google redirect URI accordingly.

Connecting Claude Code (MCP)

claude mcp add --transport http labbench https://<your-domain>/api/mcp \
  --header "Authorization: Bearer <MCP_TOKEN>"

Then, in any Claude Code session:

• "Look at my workspace for cs336-mini/micrograd and give me a hint about the failing test" → Claude uses get_workspace/get_run, posts via post_hint.

• "Assign me a small project about building a compiler" → Claude uses assign_task; the task appears on your dashboard under Assigned by Claude, complete with starter files and tests.

The MCP server acts on behalf of the first email in ALLOWED_EMAILS (sign in once on the site before using MCP so the user row exists).

Authoring courses

See content/README.md. Short version:

content/courses/<course>/course.yaml
content/courses/<course>/problems/<slug>/{problem.yaml,statement.md,starter/,tests/}

tests/run_tests.py uses the injected harness:

from labbench import case, run

@case("part (a): thing works")
def test_thing():
    assert thing() == 42

run()

Validate with npm run content:check. The sample course CS336-mini ships three problems: a BPE tokenizer, a micrograd-style autograd engine (both run anywhere, stdlib-only), and attention-from-scratch in PyTorch (machine spec installs torch; give it a GPU by editing problem.yaml).

Architecture notes

• Next.js 15 App Router + Prisma/Postgres + NextAuth v5 (Google, JWT sessions).

• File-based content is read fresh from disk per request — edit markdown, refresh. DB-backed problems (from assign_task) merge into the same catalog under the virtual assigned course.

• Runs: POST /api/runs snapshots your files into a Run row, executes via next/server after() (works on Vercel), client polls GET /api/runs/:id. Test files are copied into the sandbox after user files, so tests can't be overwritten; they're never exposed in the UI or over MCP.

• The results protocol is one marker line of JSON printed by the harness — language-agnostic, so future non-Python runners only need to print the same line.