# BAIC Data Layer MCP Server

An MCP server over the BAIC Data Layer: sources, knowledge bases, knowledge clusters, the three
data agents, and transformation pipelines.

API behaviour is documented in [DL-API-REFERENCE.md](DL-API-REFERENCE.md), verified live against
two environments. Read that first if anything here surprises you — several endpoints are named
misleadingly.

[SESSION-LOG.md](SESSION-LOG.md) records how this was built: the platform behaviours found, the
bugs fixed along the way, claims that were later corrected, and the open items.

## Setup

```bash
cp .env.example .env      # then fill in credentials + hosts
uv sync
```

`.env` is gitignored (as is any `.env*` except the example). Nothing environment-specific is
hardcoded: the tenant and user ids are derived from the access token at runtime, because the
platform uses different tenant ids on different endpoints and they vary per deployment.

### Pointing at a different environment

Edit `.env` — the hosts and credentials there are all that select a deployment:

```
BAIC_UI_URL  BAIC_BACKEND_URL  BAIC_USER_MGMT_URL  BAIC_USER_NAME  BAIC_USER_PASSWD
```

Keeping spare copies (`.env-udl-demo`, `.env-poc-farmers`) and copying one over `.env` works fine;
all `.env*` files are gitignored except `.env.example`.

`.env` is optional when the variables are supplied directly, which is how Docker (`--env-file`) and
fastmcp.cloud work — the image ships no dotenv file on purpose. Missing values fail at startup
naming the variable.

`whoami` reports the backend host, tenant and user, so you can always confirm where you are
pointed before running anything.

## Running

Built on [FastMCP](https://gofastmcp.com). Two transports, chosen by `BAIC_TRANSPORT`:

```bash
uv run baic-dl-mcp                          # stdio (default) - what Claude Desktop drives
BAIC_TRANSPORT=http uv run baic-dl-mcp      # HTTP on :8000
```

### Claude Desktop — local

```json
{
  "mcpServers": {
    "baic-dl": {
      "command": "uv",
      "args": ["--directory", "/absolute/path/to/BAIC-DL-MCP", "run", "baic-dl-mcp"]
    }
  }
}
```

### Claude Desktop — Docker

```bash
docker build -t baic-dl-mcp:latest .
```

```json
{
  "mcpServers": {
    "baic-dl": {
      "command": "docker",
      "args": ["run", "-i", "--rm",
               "--env-file", "/absolute/path/to/BAIC-DL-MCP/.env",
               "baic-dl-mcp:latest"]
    }
  }
}
```

`-i` is required — stdio needs stdin held open. Credentials are passed at run time via
`--env-file` and never baked into the image.

### fastmcp.cloud

`fastmcp.json` points at `server.py:mcp` and is ready to deploy. Connect the repo in
[fastmcp.cloud](https://fastmcp.cloud), then set these as environment variables **in the cloud UI**
— never commit them:

```
BAIC_USER_NAME  BAIC_USER_PASSWD  BAIC_UI_URL  BAIC_BACKEND_URL  BAIC_USER_MGMT_URL
```

Root `server.py` is a path shim, not a second copy of the server: hosted runners load that file
directly rather than importing the installed package, so the relative imports inside
`src/baic_dl_mcp/` need `src` on the path first.

### HTTP, for a shared deployment

```bash
docker compose up -d          # serves http://localhost:8000/mcp
```

Both paths are verified: 36 tools listed and tools invoked successfully over container stdio and
container HTTP.

Start with `whoami` to confirm which environment you're pointed at.

## Tools

36 tools.

### Connectivity

| Tool | Purpose |
| --- | --- |
| `whoami` | Confirm connectivity and show which environment, tenant, and user are in play |

### Sources

| Tool | Purpose |
| --- | --- |
| `list_sources` | List configured data sources (source connections), one page at a time |
| `get_source` | Full detail for one source, including its connection config (secrets masked) |
| `list_connector_types` | The connector engines this platform supports |
| `create_snowflake_source` | Create a new Snowflake source connection |
| `create_postgres_source` | Create a new Postgres source connection |

### Ingestion

| Tool | Purpose |
| --- | --- |
| `list_available_tables` | List the tables a source exposes, before importing any of them |
| `list_destinations` | List destinations - the vector stores or warehouses that ingestion writes into |
| `import_source_metadata` | Crawl a source's metadata into a new knowledge base |
| `import_and_wait` | Import a source's metadata into a knowledge base and wait for it to finish |
| `get_ingestion_task` | Check the status of a metadata import started by import_source_metadata |

### Knowledge bases

| Tool | Purpose |
| --- | --- |
| `list_knowledge_bases` | List knowledge bases (KBs) - the metadata imported via Data Ingestion, one page at a time |
| `list_kb_tables` | List the tables inside a knowledge base, one page at a time |
| `get_table_details` | Table descriptions and column-level detail for one or more tables |
| `get_table_relationships` | Discovered foreign-key style relationships between tables in a knowledge base |

### Knowledge resources

| Tool | Purpose |
| --- | --- |
| `list_golden_sql` | List the Golden SQL examples attached to a knowledge base |
| `add_golden_sql` | Save a SQL query as a Golden SQL example on a knowledge base |
| `delete_golden_sql` | Delete a Golden SQL example. Requires BAIC_ALLOW_DESTRUCTIVE=true |
| `get_kb_instructions` | Read the instructions attached to a knowledge base, with all versions |
| `update_kb_instructions` | Overwrite the active instructions on a knowledge base. Replaces, does not append |

### Knowledge clusters

| Tool | Purpose |
| --- | --- |
| `list_knowledge_clusters` | List knowledge clusters (KCs) - the groupings of knowledge bases used by the agents |
| `get_knowledge_cluster` | KC detail: member knowledge bases, their engines, and the KC instructions |
| `get_kc_instructions` | Read a knowledge cluster's instructions, with all versions |
| `update_kc_instructions` | Overwrite a knowledge cluster's active instructions. Replaces, does not append |
| `create_knowledge_cluster` | Create a knowledge cluster from knowledge bases, optionally with instructions |
| `check_kc_federation` | Check whether a set of knowledge bases could legally share one knowledge cluster |
| `delete_knowledge_cluster` | Delete a knowledge cluster. Requires BAIC_ALLOW_DESTRUCTIVE=true |

### Agents

| Tool | Purpose |
| --- | --- |
| `discover_tables` | Ask the Data Discovery agent which tables and columns are relevant to a question |
| `ask_data_analyst` | Ask the Data Analyst agent a question about a knowledge cluster |
| `generate_pipeline_sql` | Ask the Data Engineering agent to write pipeline SQL |
| `approve_table_selection` | Confirm the table selection and generate the SQL. Step 2 of 2 |

### Pipelines

| Tool | Purpose |
| --- | --- |
| `list_pipelines` | List data pipelines, optionally only those in one knowledge cluster |
| `create_pipeline` | Create a data pipeline |
| `get_pipeline_sql` | Read a pipeline's saved SQL and visual pipeline graph. Defaults to the active version |
| `save_pipeline_sql` | Save SQL and/or the visual pipeline graph onto a pipeline version. Write-only |
| `delete_pipeline` | Delete a pipeline. Requires BAIC_ALLOW_DESTRUCTIVE=true |

## Two things worth knowing before you use it

### Saving pipeline SQL is a write, not a review

Pass `save_to_version_id` to `generate_pipeline_sql` or `approve_table_selection` and the SQL is
generated **and saved in that one call**. You get back a `saved` receipt plus a short
`sql_preview`; the full SQL body and graph are deliberately not returned.

That shape is intentional. Returning 1-2 kB of SQL to the model invites it to re-read the SQL,
call `get_table_details` to check the column names, and then write several paragraphs explaining
the query and flagging joins for review. None of that is wanted — the SQL comes from the
platform's own engineering agent, generated from the knowledge cluster's metadata and
instructions. With nothing to summarise, there is nothing to summarise.

`save_pipeline_sql` behaves the same way if you call it directly: compact receipt, no SQL echo.
Its description tells the model not to validate, not to reformat, and not to restate. Use
`get_pipeline_sql` when you actually want the SQL back.

Two related things worth knowing:

- **Empty `columns` from `get_table_details` is normal**, not a verification failure. It means
  enrichment has not run on that table. The response now says so explicitly, because reading it
  as "I cannot verify this" was what triggered the hedging in the first place. The agents read the
  source schema directly and do not depend on that endpoint.
- **Branch on `outcome`, don't assume a review.** `generate_pipeline_sql` usually returns
  `review_required` with `numbered_tables` for a human to pick from, but it sometimes returns
  `sql_generated` straight away, and occasionally `no_sql_returned`.

### `ask_data_analyst` retries on empty results, on purpose

**Snowflake** execution is intermittent: the identical request returned data in roughly 3 of 14
measured attempts. The UI behaves the same way, so it is not a client problem.
**Postgres** is reliable — 5/5 on the same kind of query.

Worse, a failed execution is reported as `IWX-AI-SUCCESS-001` with an empty result and the answer
"The query returned no results", so on Snowflake an empty result is indistinguishable from a
failure. (Tell-tale: a genuinely empty table returns one row, `[{"count": 0}]`; a failed execution
returns zero rows, `[]`.)

So the tool retries up to `max_attempts` (default 3) and reports per-attempt outcomes. Retrying
often recovers but is not dependable. If every attempt is empty you get an `execution_warning` —
do not relay that to a user as "there is no data". Evidence in DL-API-REFERENCE.md §5.1.

## The BAIC DataLayer skill

`skills/baic-datalayer/` covers the whole Data Layer, organised the way the UI is — Manage
Connector, Data Ingestion, Data Visualization, and the three Data Agents — with a table mapping
each UI surface to its tools, plus the flows for building a pipeline, asking a question, and
onboarding a source.

It leads with three rules that override default assistant behaviour, because each one breaks a
scripted demo:

1. **Pass user wording through verbatim.** The knowledge cluster's instructions already supply
   grain, dedup keys, null handling and dialect, so an "improved" prompt generates different SQL
   than the one you rehearsed.
2. **Never read, validate or summarise generated SQL.** No checking column names via
   `get_table_details`, no explaining the design, no flagging joins for review. Saves are reported
   in one line.
3. **Chain to the next tool call instead of narrating.**

`reference/quirks.md` carries the platform behaviours worth not re-deriving — the intermittent
Snowflake execution, the review gate that does not always fire, empty `columns` being normal.

### Adding it to Claude Desktop

The skill links to the API reference, which Desktop cannot reach because uploads have no access to
repo files. Build the self-contained bundle first — it copies the referenced docs inside and
rewrites the links:

```bash
uv run python scripts/package.py
# -> dist/baic-datalayer-skill.zip
```

Then in Claude Desktop:

1. **Settings → Capabilities → Skills** (on some builds: Settings → Features → Skills)
2. **Upload skill** and choose `dist/baic-datalayer-skill.zip`
3. Confirm `baic-datalayer` is listed and enabled
4. Start a new conversation — skills are picked up per conversation, not retroactively
5. Check it loaded by asking something that should trigger it, e.g. *"list the knowledge clusters
   in baic"*

The zip has the skill folder as its root, which is the layout Desktop expects:

```
baic-datalayer/
  SKILL.md
  reference/quirks.md
  reference/api-reference.md    <- copied in, links rewritten
```

Rebuild and re-upload whenever the skill or the tool set changes.

**Claude Code** does not load this skill in *this* repo, deliberately: it lives in `skills/`, not
`.claude/skills/`, because it tells an assistant how to *drive* the Data Layer, which is not what
you are doing when you are editing the server. To use it from Claude Code in another project, copy
`skills/baic-datalayer/` into that project's `.claude/skills/`, or unzip the bundle there.

### It stays environment-neutral

The skill names no deployment: no hostnames, user emails, ids, or row counts. It tells the model to
call `whoami` first and to discover ids at run time, because all of those change when you point
`.env` somewhere else.

`DL-API-REFERENCE.md` is deliberately **not** bundled into the skill. It is a human record of
building this server — raw endpoints, one deployment's hosts, ids and counts — and shipping it made
the skill assert things that become false the moment you switch environment. The skill drives MCP
tools, not HTTP, so it does not need it. That dropped the bundle from 31 KB to 5 KB.

`scripts/package.py` fails the build if the bundle mentions a specific host, email, or UUID, so
this cannot creep back in.

### Keeping it honest

```bash
uv run python scripts/check_skill.py
```

Fails if the skill names a tool that does not exist, and warns if a tool is never mentioned. A
skill that points at a renamed tool sends the model down a dead end, so this is checked rather than
trusted.

### It is a nudge, not a guarantee

Skills and tool descriptions steer; they do not enforce. The enforcement is structural:

- Pass `save_to_version_id` so SQL is saved server-side and **never returned to the model**. SQL
  that is not in the response cannot be summarised.
- `save_pipeline_sql` returns a compact receipt, not the SQL body.
- `get_table_details` labels an empty `columns` list as expected, so it stops reading as a
  verification failure.

`instruction` and `question` are documented as verbatim in the tool descriptions and in the
server-level instructions too, so the rule still holds when the skill is not loaded.

## Guard rails

Two flags in `.env`, both default-off:

- `BAIC_ALLOW_DESTRUCTIVE` — gates `delete_pipeline`, `delete_golden_sql`,
  `delete_knowledge_cluster`
- `BAIC_ALLOW_AUTO_APPROVE` — reserved for chaining the pipeline review gate automatically

`create_knowledge_cluster` also refuses knowledge bases that would need federation. All structured
KBs in a cluster must share one engine and one account — the platform cannot query across them,
and the API will not stop you. Unstructured (document) KBs are exempt.

## Packaging and release

```bash
uv run python scripts/package.py        # build skill bundle + validate all targets
uv run python scripts/package.py --check # validate only
```

Three deployment targets are kept working at once, and the validator fails if any breaks:

| Target        | Entry                               | Transport     |
| ------------- | ----------------------------------- | ------------- |
| Local         | `baic-dl-mcp` console script        | stdio         |
| Docker        | `Dockerfile` / `docker-compose.yml` | stdio or http |
| fastmcp.cloud | `fastmcp.json` → `server.py:mcp`    | http          |

Before a demo or a release, run the sequence in the `package-baic-mcp` skill — it regenerates the
README tool table, checks the skill for stale tool names, builds the bundle, rebuilds the image, and
finishes with a live smoke test against the configured environment.

Credentials never enter an artefact: `.gitignore` covers `.env*` (except `.env.example`),
`.dockerignore` keeps them out of the image, and both are asserted by the validator. Docker and
fastmcp.cloud take them as run-time environment variables.

## Layout

```
src/baic_dl_mcp/
  config.py            env loading
  client.py            auth, token refresh, envelope normalisation, paging
  domain.py            the operations, and the reasons behind each workaround
  server.py            FastMCP tool surface
server.py              path shim for hosted runners (fastmcp.cloud) - no logic
fastmcp.json           fastmcp.cloud deployment config
Dockerfile             stdio + http, credentials injected at run time
docker-compose.yml     HTTP deployment
scripts/
  gen_readme_tools.py  regenerates the Tools table from the running server
  check_skill.py       fails if the skill names a tool that does not exist
  package.py           builds the Desktop skill bundle, validates all targets
skills/
  baic-datalayer/      how to drive the Data Layer - ships to Desktop, not loaded here
.claude/skills/
  package-baic-mcp/    the release workflow, used while working in this repo
```

The transport layer is deliberately thin: `config`/`client`/`domain` know nothing about MCP, which
is why swapping the server from the low-level SDK to FastMCP touched only `server.py`.

The Tools table above is generated from the running server, so it cannot drift out of date:

```bash
uv run python scripts/gen_readme_tools.py
```

It fails if a tool exists that the table does not list, or vice versa. Run it after adding a tool.

## A note on tool schemas

Tools are registered with FastMCP's decorator and no hand-rolled wrapper. An earlier version wrapped
each tool in `*args, **kwargs` and copied `__annotations__` but not `__signature__`; MCP introspects
the callable it is handed, so every tool advertised two bogus string parameters named `args` and
`kwargs`, and clients sent those instead of the real arguments. If you add a wrapper here, set
`__signature__` — or better, don't wrap.