# 🛠️ ToolSmith Agent — a hand-built ReAct agent + MCP server, provable offline

[![CI](https://github.com/e-akgul/toolsmith-agent/actions/workflows/ci.yml/badge.svg)](https://github.com/e-akgul/toolsmith-agent/actions/workflows/ci.yml)
![python](https://img.shields.io/badge/python-3.12-blue)
![offline](https://img.shields.io/badge/runs-100%25%20offline%20(mock%20brain)-success)
![MCP](https://img.shields.io/badge/MCP-server%20included-8A2BE2)

A multi-tool task agent whose **one tool layer** (file search · read-only
SQLite/text-to-SQL · safe calculator · report writer) is driven **three ways from
a single source of truth**:

1. a **deterministic mock brain** → 100% offline, zero secrets, CI-gated;
2. an optional **Groq** free-tier model (one env var);
3. a real **MCP server** so **Claude Code / Claude Desktop / Cursor** can reason
   over the exact same tools — real NL→tool reasoning, for free.

> No paid API is needed to prove the engineering. The mock makes the whole agent
> reproducible and testable offline; the MCP path shows a *real* model driving
> the identical tools + guardrails at zero cost.

## Results (offline mock brain, `python -m eval.simple_eval`)

| Metric | Score |
|---|---|
| Task Success Rate | **6/6 = 1.00** |
| Tool-Trajectory accuracy | **6/6 = 1.00** |
| Self-correction / recovery (injected tool errors) | **2/2 = 1.00** |
| Unit tests (guardrails, loop, MCP parity, matcher) | **21 passing** |

Trajectory is asserted, not just the final answer — a right answer via the wrong
tool still fails. See the honesty notes below on what these numbers do and don't
mean.

## What one run looks like

```
▶ TASK (mock): List the top 3 products by revenue and save a report
  ├─ step 0 · db_schema()
  │    ↳ CREATE TABLE products ( id INTEGER PRIMARY KEY, name TEXT ... )
  ├─ step 1 · query_db(sql="SELECT p.name, SUM(s.amount) AS revenue ...")
  │    ↳ name | revenue  Gadget | 600.0  Widget | 375.0  Gizmo | 90.0
  ├─ step 2 · write_report(filename="top_products.md", ...)
  │    ↳ Wrote 79 chars to reports/top_products.md.
  └─ FINAL: Saved top_products.md. Gadget leads with 600.0 in revenue.
```

Self-correction (the `count_orders` task queries a table that doesn't exist):
`db_schema → query_db(orders) → ERROR → query_db(sales) → "There are 5 sales records."`

## Architecture — one tool layer, three brains, two surfaces

```
                         tools/  ← THE single source of truth (REGISTRY)
              search_files · db_schema · query_db · calculator · write_report
              (sandbox · read-only SQL · AST calc · write-gate guardrails)
                          │                │                    │
        ┌─────────────────┘                │                    └───────────────┐
        ▼                                   ▼                                     ▼
  agent/loop.py (ReAct)              Groq schema export                   mcp_server/server.py
  reason→act→observe                 (same schemas)                       (FastMCP, stdio)
        │                                                                        │
  LLMProvider seam  ── LLM_PROVIDER=mock (default) | groq ──┐          Claude Code / Desktop / Cursor
        │                                                    │          drive the SAME tools (real model)
  mock_llm (offline, CI) ─────────────────────────────────── groq_llm (free tier)
```

- **Hand-written ReAct loop** (no `create_react_agent`): reason → tool-select →
  validate args → execute → observe → repeat, under a max-steps cap with
  identical-action loop detection. Self-correction is emergent: a `ToolError`
  becomes an `ERROR:` observation the model re-plans from.
- **Two-tier memory** (scratchpad + persistent SQLite thread store) and a
  **JSONL trace** of every step.
- **Guardrails as tested code:** path-sandbox, read-only SQL (sqlglot + `mode=ro`),
  AST calculator (no `eval`), write-gate, and `<untrusted>` wrapping of tool
  output (prompt-injection defense). See [DECISIONS.md](DECISIONS.md).

## Quickstart (offline — no API key)

```bash
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt -r requirements-dev.txt

pytest -q                          # 21 tests, guardrail attacks included
python -m eval.simple_eval         # the agent eval gate (offline, deterministic)

python -m agent.run "What is the 8% sales commission on our total revenue?"
python scripts/render_trace.py     # pretty-print the latest ReAct trace
```

## Use it from Claude Code / Claude Desktop / Cursor (real model, free)

The MCP server exposes the same tools. Point a real client at it:

**Claude Code** (from the project dir):
```bash
claude mcp add toolsmith -- /absolute/path/to/toolsmith-agent/.venv/bin/python \
                             /absolute/path/to/toolsmith-agent/mcp_server/server.py
# then, inside Claude Code:  /mcp   (and ask a multi-tool question)
```
A committed [`.mcp.json`](.mcp.json) (uv-based) also works automatically if you
have [uv](https://docs.astral.sh/uv/) installed.

**Claude Desktop** — add to `~/Library/Application Support/Claude/claude_desktop_config.json`, then restart:
```json
{
  "mcpServers": {
    "toolsmith": {
      "command": "/absolute/path/to/toolsmith-agent/.venv/bin/python",
      "args": ["/absolute/path/to/toolsmith-agent/mcp_server/server.py"]
    }
  }
}
```

**Cursor** — same block in `.cursor/mcp.json`.

**Inspect the server** (Tools / Resources / Prompts UI):
```bash
npx @modelcontextprotocol/inspector .venv/bin/python mcp_server/server.py
```
Try calling `query_db` with `DROP TABLE sales` and watch it come back a clean,
guardrailed error.

## Optional: drive the standalone loop with a real model (Groq free tier)

```bash
pip install -r requirements-groq.txt
cp .env.example .env     # set GROQ_API_KEY, LLM_PROVIDER=groq
python -m agent.run "Which product earned the most, and what's 8% of it?"
```
The provider seam swaps with **zero changes to the loop**.

## Honest notes (because measuring is the point)

- **The mock proves** the loop's control flow, tool selection/dispatch, arg
  validation, termination + loop-detection, that every guardrail fires, that an
  ERROR observation triggers recovery, MCP tool parity, tracing, and full offline
  CI — **not** that a model reasons or generalizes.
- **Real reasoning** is covered, for free, by the **MCP-in-Claude-Code** path
  (identical tools + guardrails) and by the optional Groq provider.
- `pass^k` is trivially 1.0 under the deterministic mock; it's only meaningful
  re-run over a real model. This README does not headline it as a reliability
  number.

## Tech

Python 3.12 · MCP (official SDK / FastMCP) · Pydantic · sqlglot · SQLite ·
stdlib (ast, pathlib) · pytest · GitHub Actions · Docker · optional Groq.
Deliberately **torch-free**. Sibling project: **GroundedQA** (RAG) —
github.com/e-akgul/groundedqa.