Skip to main content
Glama
Deriva MCP Server

Deriva MCP Server

Official

Deriva MCP Server

Model Context Protocol (MCP) server that exposes Deriva catalog operations and DerivaML ML workflow tools for LLM applications.

Overview

This MCP server provides an interface to Deriva catalogs and DerivaML, enabling AI assistants like Claude to:

  • Connect to and manage Deriva catalogs

  • Create and manage datasets with versioning

  • Work with controlled vocabularies

  • Define and execute ML workflows

  • Create and manage features for ML experiments

For full ML workflow management, this server is designed to work alongside the GitHub MCP Server to enable:

  • Storing and versioning hydra-zen configurations in GitHub repositories

  • Managing workflow code and model implementations

  • Collaborative development of ML experiments

Related MCP server: mlops-mcp-server

Prerequisites

Deriva Authentication

DerivaML uses Globus for authentication. Before using the MCP server, you must authenticate with your Deriva server:

# Install deriva-ml if not already installed
pip install deriva-ml

# Authenticate with your Deriva server
python -c "from deriva_ml import DerivaML; DerivaML.globus_login('your-server.org')"

This opens a browser window for Globus authentication. Credentials are cached locally and persist across sessions.

Alternatively, use the Deriva Auth Agent for browser-based authentication:

  1. Install the Deriva Auth Agent from deriva-py

  2. Run deriva-globus-auth-utils login --host your-server.org

GitHub Authentication (for configuration management)

Create a GitHub Personal Access Token (PAT) for the GitHub MCP Server:

  1. Go to GitHub Settings > Personal Access Tokens

  2. Create a fine-grained token with these permissions:

    • Repository access: Select repositories containing your ML configurations

    • Permissions:

      • Contents: Read and write (for pushing configs)

      • Pull requests: Read and write (optional, for PR workflows)

      • Issues: Read (optional, for tracking)

  3. Copy the token securely - you'll need it for configuration

Installation

Docker provides the simplest setup with no Python environment management. The image is automatically built and published to GitHub Container Registry on every commit to main.

# Pull the latest image from GitHub Container Registry
docker pull ghcr.io/informatics-isi-edu/deriva-mcp:latest

# Run the server (for testing)
docker run --rm -it ghcr.io/informatics-isi-edu/deriva-mcp:latest --help

To build locally instead:

git clone https://github.com/informatics-isi-edu/deriva-mcp.git
cd deriva-mcp
./scripts/docker-build.sh

Using uv

uv pip install deriva-mcp

Using pip

pip install deriva-mcp

From source

git clone https://github.com/informatics-isi-edu/deriva-mcp.git
cd deriva-mcp
uv sync

Claude Code Plugin (Skills)

The Deriva skills plugin for Claude Code provides 30+ skills that guide Claude through common Deriva and DerivaML workflows. The skills are maintained in a separate repository: deriva-skills.

Installing the Plugin

# Add the marketplace (one-time)
/plugin marketplace add informatics-isi-edu/deriva-skills

# Install the plugin
/plugin install deriva

Updating the Plugin

To update to the latest version:

/plugin install deriva

Or check your entire DerivaML ecosystem:

/deriva:check-versions

This checks three components — the deriva-ml Python package, the deriva-skills plugin, and the deriva-mcp MCP server — against upstream releases and offers to update outdated ones.

See the deriva-skills README for the full list of available skills.

Development: Testing Skills Locally

During development, load the plugin from a local path without installing:

claude --plugin-dir /path/to/deriva-skills

Configuration

Claude Desktop - Full Setup with GitHub Integration

For the complete ML workflow experience, configure both DerivaML and GitHub MCP servers together.

Configuration file locations:

  • macOS: ~/Library/Application Support/Claude/claude_desktop_config.json

  • Windows: %APPDATA%\Claude\claude_desktop_config.json

  • Linux: ~/.config/Claude/claude_desktop_config.json

Uses Docker for both MCP servers - most consistent setup:

{
  "mcpServers": {
    "deriva": {
      "type": "stdio",
      "command": "/bin/sh",
      "args": [
        "-c",
        "docker run -i --rm --add-host localhost:host-gateway -e HOME=$HOME -v $HOME/.deriva:$HOME/.deriva:ro -v $HOME/.bdbag:$HOME/.bdbag -v $HOME/.deriva-ml:$HOME/.deriva-ml ghcr.io/informatics-isi-edu/deriva-mcp:latest"
      ],
      "env": {}
    },
    "github": {
      "type": "stdio",
      "command": "docker",
      "args": [
        "run", "-i", "--rm",
        "-e", "GITHUB_PERSONAL_ACCESS_TOKEN",
        "ghcr.io/github/github-mcp-server"
      ],
      "env": {
        "GITHUB_PERSONAL_ACCESS_TOKEN": "ghp_your_token_here"
      }
    }
  }
}

Docker arguments explained:

  • --add-host localhost:host-gateway - Allows connecting to a Deriva server running on localhost

  • -e HOME=$HOME - Passes your home directory path into the container so mounted paths are found correctly

For localhost with self-signed certificates, the image defaults to using ~/.deriva/allCAbundle-with-local.pem as the CA bundle. See Troubleshooting for how to create this file.

Volume mounts explained:

  • $HOME/.deriva:$HOME/.deriva:ro - Mounts your Deriva credentials (read-only)

  • $HOME/.bdbag:$HOME/.bdbag - Mounts bdbag keychain for dataset download authentication (writable)

  • $HOME/.deriva-ml:$HOME/.deriva-ml - Working directory for execution outputs (writable)

Note: Create the workspace directory before first use:

mkdir -p ~/.deriva-ml

If the directory doesn't exist, Docker creates it as root, causing permission issues.

Option 2: Direct Install with GitHub Remote

Uses pip-installed DerivaML MCP with GitHub's hosted server:

{
  "mcpServers": {
    "deriva": {
      "type": "stdio",
      "command": "deriva-mcp",
      "env": {}
    },
    "github": {
      "type": "stdio",
      "command": "npx",
      "args": ["-y", "@anthropic-ai/github-mcp-server"],
      "env": {
        "GITHUB_PERSONAL_ACCESS_TOKEN": "ghp_your_token_here"
      }
    }
  }
}

Option 3: From Source (Development)

For development or customization:

{
  "mcpServers": {
    "deriva": {
      "type": "stdio",
      "command": "uv",
      "args": [
        "--directory",
        "/path/to/deriva-mcp",
        "run",
        "deriva-mcp"
      ],
      "env": {}
    },
    "github": {
      "type": "stdio",
      "command": "docker",
      "args": [
        "run", "-i", "--rm",
        "-e", "GITHUB_PERSONAL_ACCESS_TOKEN",
        "ghcr.io/github/github-mcp-server"
      ],
      "env": {
        "GITHUB_PERSONAL_ACCESS_TOKEN": "ghp_your_token_here"
      }
    }
  }
}

Option 4: DerivaML Only (No GitHub)

If you don't need GitHub integration:

{
  "mcpServers": {
    "deriva": {
      "type": "stdio",
      "command": "/bin/sh",
      "args": [
        "-c",
        "docker run -i --rm --add-host localhost:host-gateway -e HOME=$HOME -v $HOME/.deriva:$HOME/.deriva:ro -v $HOME/.bdbag:$HOME/.bdbag -v $HOME/.deriva-ml:$HOME/.deriva-ml ghcr.io/informatics-isi-edu/deriva-mcp:latest"
      ],
      "env": {}
    }
  }
}

Or with direct install:

{
  "mcpServers": {
    "deriva": {
      "type": "stdio",
      "command": "deriva-mcp",
      "env": {}
    }
  }
}

Claude Code

Add to ~/.mcp.json (global) or your project's .mcp.json file:

With Docker:

{
  "mcpServers": {
    "deriva": {
      "type": "stdio",
      "command": "/bin/sh",
      "args": [
        "-c",
        "docker run -i --rm --add-host localhost:host-gateway -e HOME=$HOME -v $HOME/.deriva:$HOME/.deriva:ro -v $HOME/.bdbag:$HOME/.bdbag -v $HOME/.deriva-ml:$HOME/.deriva-ml ghcr.io/informatics-isi-edu/deriva-mcp:latest"
      ],
      "env": {}
    },
    "github": {
      "type": "stdio",
      "command": "docker",
      "args": [
        "run", "-i", "--rm",
        "-e", "GITHUB_PERSONAL_ACCESS_TOKEN",
        "ghcr.io/github/github-mcp-server"
      ],
      "env": {
        "GITHUB_PERSONAL_ACCESS_TOKEN": "${GITHUB_PERSONAL_ACCESS_TOKEN}"
      }
    }
  }
}

With direct install:

{
  "mcpServers": {
    "deriva": {
      "type": "stdio",
      "command": "deriva-mcp",
      "env": {}
    },
    "github": {
      "type": "stdio",
      "command": "npx",
      "args": ["-y", "@anthropic-ai/github-mcp-server"],
      "env": {
        "GITHUB_PERSONAL_ACCESS_TOKEN": "ghp_your_token_here"
      }
    }
  }
}

Then enable in .claude/settings.local.json:

{
  "enableAllProjectMcpServers": true,
  "enabledMcpjsonServers": ["deriva", "github"]
}

HTTP Transport Mode (for Long-Running Operations)

For operations that take more than a few minutes (like catalog cloning), HTTP transport provides a persistent connection that survives client disconnects. The server runs as a background service and maintains task state across reconnections.

Starting the HTTP Server

Using Docker Compose (Recommended):

# Requires the deriva-localhost Docker network for localhost catalogs
docker-compose -f docker-compose.mcp.yaml up -d

# View logs
docker-compose -f docker-compose.mcp.yaml logs -f

# Stop the server
docker-compose -f docker-compose.mcp.yaml down

Using Docker directly:

docker run -d --name deriva-mcp \
  -p 8000:8000 \
  --network deriva-localhost_internal_network \
  -e HOME=$HOME \
  -e DERIVA_MCP_LOCALHOST_ALIAS=deriva-webserver \
  -v $HOME/.deriva:$HOME/.deriva:ro \
  -v $HOME/.bdbag:$HOME/.bdbag \
  -v $HOME/.deriva-ml:$HOME/.deriva-ml \
  ghcr.io/informatics-isi-edu/deriva-mcp:latest \
  deriva-mcp --transport streamable-http --host 0.0.0.0 --port 8000

The DERIVA_MCP_LOCALHOST_ALIAS variable tells the entrypoint to resolve the Docker DNS name deriva-webserver to an IP and add it to /etc/hosts as localhost. This avoids hardcoding container IPs that change when the network is recreated.

Running locally (development):

# From source
uv run deriva-mcp --transport streamable-http --host 0.0.0.0 --port 8000

# If installed via pip
deriva-mcp --transport streamable-http --host 0.0.0.0 --port 8000

Client Configuration for HTTP

Claude Code (~/.mcp.json):

{
  "mcpServers": {
    "deriva": {
      "type": "http",
      "url": "http://localhost:8000/mcp"
    }
  }
}

Claude Desktop:

{
  "mcpServers": {
    "deriva": {
      "type": "http",
      "url": "http://localhost:8000/mcp"
    }
  }
}

Benefits of HTTP Transport

  • No idle timeouts: Long-running operations complete without connection drops

  • Persistent server: Server survives client disconnects and restarts

  • Task state preservation: Background tasks continue even if you close Claude

  • Multiple clients: Multiple Claude sessions can share the same server

  • Health checks: Docker can automatically restart unhealthy servers

Verifying the HTTP Server

# Check server health (uses /health endpoint, not /mcp which creates sessions)
curl -s http://localhost:8000/health

# View server logs
docker-compose -f docker-compose.mcp.yaml logs -f

VS Code with Continue or Cline

Add to your MCP configuration (typically .vscode/mcp.json):

{
  "mcp": {
    "servers": {
      "deriva": {
        "type": "stdio",
        "command": "/bin/sh",
        "args": [
          "-c",
          "docker run -i --rm --add-host localhost:host-gateway -e HOME=$HOME -v $HOME/.deriva:$HOME/.deriva:ro -v $HOME/.bdbag:$HOME/.bdbag -v $HOME/.deriva-ml:$HOME/.deriva-ml ghcr.io/informatics-isi-edu/deriva-mcp:latest"
        ],
        "env": {}
      },
      "github": {
        "type": "stdio",
        "command": "docker",
        "args": [
          "run", "-i", "--rm",
          "-e", "GITHUB_PERSONAL_ACCESS_TOKEN",
          "ghcr.io/github/github-mcp-server"
        ],
        "env": {
          "GITHUB_PERSONAL_ACCESS_TOKEN": "ghp_your_token_here"
        }
      }
    }
  }
}

Environment Variables

For security, store tokens in environment variables instead of config files:

# Add to ~/.bashrc, ~/.zshrc, or equivalent
export GITHUB_PERSONAL_ACCESS_TOKEN="ghp_your_token_here"

Then reference in config:

{
  "mcpServers": {
    "github": {
      "type": "stdio",
      "command": "docker",
      "args": ["run", "-i", "--rm", "-e", "GITHUB_PERSONAL_ACCESS_TOKEN", "ghcr.io/github/github-mcp-server"],
      "env": {
        "GITHUB_PERSONAL_ACCESS_TOKEN": "${GITHUB_PERSONAL_ACCESS_TOKEN}"
      }
    }
  }
}

Verifying Your Setup

After configuration, verify both servers are working:

User: What MCP servers are available?

Claude: I have access to two MCP servers:
1. deriva - For managing ML workflows in Deriva catalogs
2. github - For managing GitHub repositories and configurations

User: Connect to the deriva catalog at example.org with ID 42

Claude: [Uses connect_catalog tool]
Connected to example.org, catalog 42. The domain schema is 'my_project'.

User: List the hydra-zen configs in the my-ml-project repo

Claude: [Uses GitHub get_file_contents tool]
Found configuration files in configs/:
- deriva.py - DerivaML connection settings
- datasets.py - Dataset specifications
- model.py - Model hyperparameters

Available Tools

Catalog Management

Tool

Description

connect_catalog

Connect to a DerivaML catalog

disconnect_catalog

Disconnect from the active catalog

list_connections

List all active connections

set_active_catalog

Set which connection is active

get_catalog_info

Get information about the active catalog

list_users

List users with catalog access

get_chaise_url

Get web interface URL for a table

resolve_rid

Find which table a RID belongs to

list_catalog_registry

List all catalogs and aliases on a server

create_catalog

Create a new DerivaML catalog (with optional alias)

delete_catalog

Permanently delete a catalog

clone_catalog

Clone a catalog to create a copy

Catalog Alias Management

Tool

Description

create_catalog_alias

Create an alias for a catalog

get_catalog_alias

Get alias metadata (target, owner)

update_catalog_alias

Update alias target or owner

delete_catalog_alias

Delete an alias (catalog not affected)

Dataset Management

Tool

Description

find_datasets

Find all datasets in the catalog

lookup_dataset

Look up detailed information about a dataset

create_dataset

Create a new dataset

list_dataset_members

List members of a dataset

add_dataset_members

Add members to a dataset

get_dataset_version_history

Get version history

increment_dataset_version

Update dataset version

delete_dataset

Delete a dataset

list_dataset_element_types

List valid element types

add_dataset_element_type

Enable a table as element type

Vocabulary Management

Tool

Description

list_vocabularies

List all vocabulary tables

list_vocabulary_terms

List terms in a vocabulary

lookup_term

Find a term by name or synonym

add_term

Add a term to a vocabulary

create_vocabulary

Create a new vocabulary table

Workflow Management

Tool

Description

find_workflows

Find all workflows

lookup_workflow

Find a workflow by URL/checksum

create_workflow

Create and register a workflow

list_workflow_types

List available workflow types

add_workflow_type

Add a new workflow type

Feature Management

Tool

Description

find_features

Find features for a table

lookup_feature

Get feature details

list_feature_values

Get all values for a feature

create_feature

Create a feature definition

delete_feature

Delete a feature

list_feature_names

List all feature names

Schema Management

Tool

Description

create_table

Create a new table in the domain schema

create_asset_table

Create an asset table for file management

list_assets

List all assets in an asset table

list_tables

List all tables in the domain schema

get_table_schema

Get column and key definitions for a table

list_asset_types

List available asset type terms

add_asset_type

Add a new asset type to the vocabulary

Execution Management

Tool

Description

create_execution

Create a new execution for ML workflows

start_execution

Start the active execution

stop_execution

Stop and complete the active execution

update_execution_status

Update execution status and message

get_execution_info

Get details about the active execution

restore_execution

Restore a previous execution by RID

asset_file_path

Register a file for upload as an execution output

commit_output_assets

Commit all registered outputs to the catalog

list_executions

List recent executions

create_execution_dataset

Create a dataset within an execution

download_execution_dataset

Download a dataset for processing

get_execution_working_dir

Get the working directory path

Execution Workflow

The typical execution workflow using the context manager:

with execution.execute() as exe:
    # Do your work here
    exe.asset_file_path(asset_name="Image", file_name="output.png")
    # ... more processing ...

# After context exits, commit output assets
execution.commit_output_assets()

Using MCP tools, the equivalent workflow is:

  1. create_execution() - Create the execution record with workflow info

  2. start_execution() - Mark execution as running, begin timing

  3. asset_file_path() - Register output files (repeat as needed)

  4. stop_execution() - Mark execution as complete

  5. commit_output_assets() - Required: Commit all registered files to catalog

Important: You must call commit_output_assets() after completing your work to commit any registered assets to the catalog. This is not automatic.

Available Resources

MCP resources provide read-only access to catalog information and configuration templates.

Static Resources - Configuration Templates

These resources provide code templates for configuring DerivaML with hydra-zen:

Resource URI

Description

deriva://config/deriva-ml-template

Hydra-zen configuration template for DerivaML connection

deriva://config/dataset-spec-template

Configuration template for dataset specifications

deriva://config/execution-template

Configuration template for ML executions

deriva://config/model-template

Configuration template for ML models with zen_partial

Dynamic Resources - Catalog Information

These resources return current catalog state (requires active connection):

Resource URI

Description

deriva://catalog/schema

Current catalog schema structure in JSON

deriva://catalog/vocabularies

All vocabulary tables and their terms

deriva://catalog/datasets

All datasets in the current catalog

deriva://catalog/workflows

All registered workflows

deriva://catalog/features

All feature names defined in the catalog

Template Resources - Parameterized

These resources accept parameters to return specific information:

Resource URI

Description

deriva://dataset/{dataset_rid}

Detailed information about a specific dataset

deriva://table/{table_name}/features

Features defined for a specific table

deriva://vocabulary/{vocab_name}

Terms in a specific vocabulary table

Documentation Resources

Documentation is fetched dynamically from GitHub repositories with 1-hour caching:

Resource URI

Description

deriva://docs/overview

DerivaML overview and architecture

deriva://docs/datasets

Guide to creating and managing datasets

deriva://docs/features

Guide to defining and using features

deriva://docs/execution-configuration

Guide to configuring ML executions

deriva://docs/hydra-zen

Guide to hydra-zen configuration

deriva://docs/file-assets

Guide to managing file assets

deriva://docs/notebooks

Guide to Jupyter notebook integration

deriva://docs/identifiers

Guide to RIDs, MINIDs, and identifiers

deriva://docs/install

Installation instructions

deriva://docs/ermrest/*

ERMrest API documentation

deriva://docs/chaise/*

Chaise UI documentation

deriva://docs/deriva-py/*

Deriva Python SDK documentation

Using Resources

Resources are accessed differently than tools - they provide static or semi-static data that can be read without side effects:

User: Show me the DerivaML configuration template

Claude: [Reads deriva://config/deriva-ml-template resource]
Here's a hydra-zen configuration template for DerivaML...

User: What datasets are in the catalog?

Claude: [Reads deriva://catalog/datasets resource]
Found the following datasets in your catalog...

Usage Examples

Discovering and Connecting to Catalogs

User: What catalogs are available on example.org?

Claude: [Uses list_catalog_registry tool]
Found 3 catalogs on example.org:
- ID: 21, Name: "ML Project", Persistent: true
- ID: 45, Name: "Test Environment", Persistent: true
- ID: 50, Name: "Clone of ML Project", Persistent: true

And 2 aliases:
- "ml-project" -> catalog 21
- "test-env" -> catalog 45

User: Connect to ml-project

Claude: [Uses connect_catalog tool with catalog_id="ml-project"]
Connected to example.org using alias "ml-project" (catalog 21).
The domain schema is 'my_domain'.

Cloning and Managing Catalogs

User: Clone the production catalog to create a test environment

Claude: [Uses clone_catalog tool]
Cloned catalog 21 to new catalog 52 with all data and policies.

User: Create an alias "dev-test" for the new catalog

Claude: [Uses create_catalog_alias tool]
Created alias "dev-test" pointing to catalog 52.
You can now connect using: connect_catalog("example.org", "dev-test")

Basic Catalog Operations

User: Connect to the deriva catalog at example.org with ID 123

Claude: I'll connect to that catalog for you.
[Uses connect_catalog tool]
Connected to example.org, catalog 123. The domain schema is 'my_project'.

User: What datasets are available?

Claude: Let me check what datasets exist.
[Uses find_datasets tool]
Found 5 datasets:
1. Training Images (v1.2.0) - 1500 images for model training
2. Validation Set (v1.0.0) - 300 images for validation
...

Configuration Workflow with GitHub

User: Create a new hydra-zen config for training on the cell-images dataset

Claude: I'll create a configuration file for you.
[Reads deriva://config/model-template resource for template]
[Uses GitHub create_or_update_file to save configs/cell_training.py]

Created configs/cell_training.py with:
- DerivaML connection to example.org
- Dataset spec for cell-images v2.0.0
- Model config with learning_rate=1e-3, epochs=50

User: Commit and push that config

Claude: [Uses GitHub create_or_update_file with commit message]
Committed and pushed to main branch.

Hydra-zen Configuration

DerivaML integrates with hydra-zen for configuration management, enabling reproducible ML workflows with structured configuration.

Basic Configuration

from hydra_zen import builds, instantiate
from deriva_ml import DerivaML
from deriva_ml.core.config import DerivaMLConfig

# Create a structured config using hydra-zen
DerivaMLConf = builds(DerivaMLConfig, populate_full_signature=True)

# Configure for your environment
conf = DerivaMLConf(
    hostname='deriva.example.org',
    catalog_id='42',
    domain_schema='my_domain',
)

# Instantiate to get a DerivaMLConfig object, then create DerivaML
config = instantiate(conf)
ml = DerivaML.instantiate(config)

Working Directory Configuration

DerivaML automatically configures Hydra's output directory based on your working_dir setting:

conf = DerivaMLConf(
    hostname='deriva.example.org',
    working_dir='/shared/ml_workspace',  # Custom working directory
)

Hydra outputs will be organized under: {working_dir}/{username}/deriva-ml/hydra/{timestamp}/

Configuration Composition

Create environment-specific configurations using hydra-zen's store:

from hydra_zen import store

# Development configuration
store(DerivaMLConf(
    hostname='dev.example.org',
    catalog_id='1',
), name='dev')

# Production configuration
store(DerivaMLConf(
    hostname='prod.example.org',
    catalog_id='100',
), name='prod')

Dataset Specification Configuration

Use DatasetSpecConfig for cleaner dataset specifications:

from deriva_ml.dataset import DatasetSpecConfig

# Create dataset specs (hydra-zen compatible)
training_data = DatasetSpecConfig(
    rid="1ABC",
    version="1.0.0",
    materialize=True,       # Download asset files
    description="Training images"
)

metadata_only = DatasetSpecConfig(
    rid="2DEF",
    version="2.0.0",
    materialize=False,      # Only download table data
)

# Use in hydra-zen store
from hydra_zen import store
datasets_store = store(group="datasets")
datasets_store([training_data], name="training")
datasets_store([metadata_only], name="metadata_only")

Asset Configuration

Use AssetRIDConfig for input assets (model weights, config files):

from deriva_ml.execution import AssetRIDConfig

# Define input assets
model_weights = AssetRIDConfig(rid="WXYZ", description="Pretrained model")
config_file = AssetRIDConfig(rid="ABCD", description="Hyperparameters")

# Store asset collections
assets_store = store(group="assets")
assets_store([model_weights, config_file], name="default_assets")

Execution Configuration

Configure ML executions with ExecutionConfiguration:

from hydra_zen import builds, instantiate
from deriva_ml.execution import ExecutionConfiguration
from deriva_ml.dataset import DatasetSpecConfig

# Build execution config
ExecConf = builds(ExecutionConfiguration, populate_full_signature=True)

# Configure execution with datasets and assets
conf = ExecConf(
    description="Training run",
    datasets=[
        DatasetSpecConfig(rid="1ABC", version="1.0.0", materialize=True),
    ],
    assets=["WXYZ", "ABCD"],  # Asset RIDs
)

exec_config = instantiate(conf)

Configuration Summary

Class

Module

Purpose

DerivaMLConfig

deriva_ml.core.config

Main DerivaML connection config

DatasetSpecConfig

deriva_ml.dataset

Dataset specification for executions

AssetRIDConfig

deriva_ml.execution

Input asset specification

ExecutionConfiguration

deriva_ml.execution

Full execution configuration

Workflow

deriva_ml.execution

Workflow definition

See the DerivaML Hydra-zen Guide for complete documentation.

Troubleshooting

Docker with Localhost Deriva Server

When running the MCP server in Docker and connecting to a Deriva server on your local machine, you need additional configuration depending on how Deriva is running.

Option A: Deriva Running Directly on Host (not in Docker)

If your Deriva server is running directly on the host machine (not in Docker), use host-gateway:

{
  "mcpServers": {
    "deriva": {
      "type": "stdio",
      "command": "/bin/sh",
      "args": [
        "-c",
        "docker run -i --rm --add-host localhost:host-gateway -e HOME=$HOME -v $HOME/.deriva:$HOME/.deriva:ro -v $HOME/.bdbag:$HOME/.bdbag -v $HOME/.deriva-ml:$HOME/.deriva-ml ghcr.io/informatics-isi-edu/deriva-mcp:latest"
      ],
      "env": {}
    }
  }
}

Option B: Deriva Running in Docker (deriva-localhost)

If your Deriva server is running in Docker (e.g., using deriva-localhost), the MCP container must join the same Docker network and map localhost to the webserver container.

First, find the webserver IP:

docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' deriva-webserver

Then use it in the --add-host argument (replace <WEBSERVER_IP> with the actual IP):

{
  "mcpServers": {
    "deriva": {
      "type": "stdio",
      "command": "/bin/sh",
      "args": [
        "-c",
        "docker run -i --rm --network deriva-localhost_internal_network --add-host localhost:<WEBSERVER_IP> -e HOME=$HOME -v $HOME/.deriva:$HOME/.deriva:ro -v $HOME/.bdbag:$HOME/.bdbag -v $HOME/.deriva-ml:$HOME/.deriva-ml ghcr.io/informatics-isi-edu/deriva-mcp:latest"
      ],
      "env": {}
    }
  }
}

Why this is needed: The MCP container needs to download dataset assets from the Deriva server. When Deriva runs in Docker, URLs in the dataset bags reference localhost, which must resolve to the Deriva webserver container. The entrypoint script in the MCP image automatically adjusts /etc/hosts so that the --add-host mapping takes effect.

SSL Certificate Configuration

If your localhost Deriva server uses a self-signed certificate (common for development), the container won't trust it by default. The image automatically sets REQUESTS_CA_BUNDLE to $HOME/.deriva/allCAbundle-with-local.pem, so you just need to create this file:

Creating the CA bundle with your local certificate (macOS):

# Export the local CA certificate from System Keychain
security find-certificate -a -c "DERIVA Dev Local CA" -p /Library/Keychains/System.keychain > /tmp/deriva-local-ca.pem

# Combine with existing CA bundle (if you have one)
cat ~/.deriva/allCAbundle.pem /tmp/deriva-local-ca.pem > ~/.deriva/allCAbundle-with-local.pem

# Or just use the local CA alone
cp /tmp/deriva-local-ca.pem ~/.deriva/allCAbundle-with-local.pem

To use a different CA bundle path, override with -e REQUESTS_CA_BUNDLE=/path/to/bundle.pem.

Deriva Authentication Issues

Error: "No credentials found"

# Re-authenticate with Deriva
python -c "from deriva_ml import DerivaML; DerivaML.globus_login('your-server.org')"

Error: "Token expired"

# Force re-authentication
python -c "from deriva_ml import DerivaML; DerivaML.globus_login('your-server.org', force=True)"

GitHub MCP Issues

Error: "Bad credentials"

  • Verify your PAT hasn't expired

  • Check the token has required permissions (Contents: Read/Write)

  • Ensure the token is correctly set in your config

Docker not found

  • Install Docker Desktop or use the npx method instead

  • On Linux, ensure your user is in the docker group

MCP Server Connection Issues

Server not responding

  1. Check the server is installed: which deriva-mcp

  2. Test manually: deriva-mcp (should start without errors)

  3. Check Claude Desktop logs for errors

Multiple server conflicts

  • Ensure each server has a unique name in the config

  • Restart Claude Desktop after config changes

Long-Running Operations (Catalog Cloning)

Catalog cloning operations can take several minutes for large catalogs. The MCP server provides async tools (clone_catalog_async, get_task_status, list_tasks) to handle this.

Recommended: Use HTTP Transport for Long Operations

The HTTP transport mode solves connection timeout issues by running the server as a persistent service:

# Start the server with HTTP transport
docker-compose -f docker-compose.mcp.yaml up -d

# Or run directly
deriva-mcp --transport streamable-http --host 0.0.0.0 --port 8000

Then configure your MCP client to use HTTP:

{
  "mcpServers": {
    "deriva": {
      "type": "http",
      "url": "http://localhost:8000/mcp"
    }
  }
}

See HTTP Transport Mode for complete setup instructions.

Alternative: STDIO with Async Tools

If using STDIO transport, the async workflow helps manage timeouts:

  1. Use clone_catalog_async() to start the clone - returns a task_id immediately

  2. Periodically check get_task_status(task_id) for progress

  3. When status is "completed", the result contains the new catalog info

STDIO Connection timeout issues: If connections drop during clone operations with STDIO transport:

  1. Check task status after reconnection - If the MCP connection drops, simply reconnect and call list_tasks() to find your running tasks. Note: Task state is stored in memory, so if the server process restarts (not just reconnects), running tasks will be lost.

  2. Consider switching to HTTP transport - HTTP connections don't have the idle timeout issues that STDIO connections have with some MCP clients.

Development

Running Tests

uv run pytest

Code Quality

uv run ruff check src/
uv run ruff format src/

Requirements

  • Python 3.12+

  • MCP SDK 1.2.0+

  • DerivaML 0.1.0+

  • Docker (optional, for GitHub MCP local server)

License

Apache 2.0

Available Tools

101 tools
add_asset_typeA

Add a new asset type to the Asset_Type vocabulary.

Args: type_name: Name for the asset type. description: What this asset type represents.

Returns: JSON with status, name, description, rid.

Example: add_asset_type("Segmentation Mask", "Binary mask images for segmentation")

Inputs: type_name (required), description (required). Returns: result

add_asset_type_to_assetA

Add an asset type to a specific asset.

Associates an asset with a type from the Asset_Type vocabulary. An asset can have multiple types.

Args: asset_rid: RID of the asset to modify. type_name: Name of the asset type to add (must exist in Asset_Type vocab).

Returns: JSON with status, asset_rid, and updated types list.

Example: add_asset_type_to_asset("3JSE", "Training_Data")

Inputs: asset_rid (required), type_name (required). Returns: result

add_columnA

Add a new column to an existing table.

Args: table_name: Name of the table to modify. column_name: Name for the new column. column_type: Data type - one of "text", "int2", "int4", "int8", "float4", "float8", "boolean", "date", "timestamp", "timestamptz", "json", "jsonb", "markdown" (default: "text"). nullok: Whether NULL values are allowed (default: True). default: Default value for new rows (optional). comment: Description of the column (optional).

Returns: JSON with status, table_name, column_name, column_type.

Example: add_column("Subject", "Age", "int4", nullok=True, comment="Subject age in years")

Inputs: table_name (required), column_name (required), nullok, comment, +2 more. Returns: result

add_dataset_childA

Add a dataset as a nested child of another dataset.

Creates a parent-child relationship between datasets. Common pattern: a "Complete" parent dataset contains "Training" and "Testing" children that partition the same data.

Args: parent_rid: RID of the parent dataset. child_rid: RID of the child dataset to nest.

Returns: JSON with status, parent_rid, child_rid.

Example: add_dataset_child("1-ABC", "1-DEF") -> nests 1-DEF inside 1-ABC

Inputs: child_rid (required), parent_rid (required). Returns: result

add_dataset_element_typeA

Register a domain table as a dataset element type.

After registration, records from this table can be added to datasets using add_dataset_members(). Creates an association table to link records to datasets.

Args: table_name: Name of the domain table to register (e.g., "Subject", "Image").

Returns: JSON with status, table_name, association_table.

Example: add_dataset_element_type("Subject") -> enables Subject records as dataset elements

Inputs: table_name (required). Returns: result

add_dataset_membersA

Add records as dataset elements. Auto-increments minor version.

Records must be from tables registered as dataset element types. Use add_dataset_element_type() to register a table, or list_dataset_element_types() to see which tables are already registered.

Accepts members in two forms:

List of RIDs (member_rids): Each RID is auto-resolved to its table. Simpler but slower for large numbers.

Dict by table name (members_by_table): Maps table names to RID lists. Faster (skips RID resolution) and lets you add members of different types in one call. Recommended when you know the table names.

Exactly one of member_rids or members_by_table must be provided.

Args: dataset_rid: The RID of the dataset to add members to. member_rids: List of RIDs to add (e.g., ["2-ABC", "2-DEF"]). Auto-resolves each RID to its table. members_by_table: Dict mapping table names to RID lists (e.g., {"Subject": ["2-ABC"], "Observation": ["2-DEF", "2-GHI"]}). Faster than member_rids for large datasets. description: Optional description for the version increment that records why these members were added. Stored in the dataset history.

Returns: JSON with status, added_count, dataset_rid.

Example: add_dataset_members("1-ABC", member_rids=["2-DEF", "2-GHI"]) add_dataset_members("1-ABC", members_by_table={"Subject": ["2-DEF"], "Image": ["2-GHI"]})

Inputs: dataset_rid (required), description, member_rids, members_by_table. Returns: result

add_dataset_typeA

Add a type to a dataset.

Adds a Dataset_Type vocabulary term to this dataset. The type must exist in the Dataset_Type vocabulary.

Args: dataset_rid: RID of the dataset. dataset_type: Name of the type to add (must exist in Dataset_Type vocabulary).

Returns: JSON with status, dataset_rid, dataset_types list.

Example: add_dataset_type("1-ABC", "Training") -> adds "Training" type to dataset

Inputs: dataset_rid (required), dataset_type (required). Returns: result

add_feature_valueA

Add feature values to one or more domain objects.

Associates feature values (terms, assets, or other) with target records. Accepts a list of entries, each mapping a target RID to a value. All entries are inserted in a single batch for efficiency.

If an execution is active, it will be used for provenance. Otherwise, provide an execution_rid explicitly.

For simple features (single term or asset column): Use this tool — each entry needs only target_rid and value.

For complex features (multiple columns per record): Use add_feature_value_record instead, which accepts arbitrary field dicts.

Args: table_name: Table the target records belong to (e.g., "Image"). feature_name: Name of the feature (e.g., "Diagnosis"). entries: List of dicts, each with: - target_rid (str): RID of the target record to annotate. - value (str): The feature value — a term name or asset RID. execution_rid: Execution RID for provenance (uses active if not provided).

Returns: JSON with status, feature_name, count, execution_rid, rids.

Examples: # Single value add_feature_value("Image", "Diagnosis", [{"target_rid": "1-ABC", "value": "Normal"}])

# Batch values
add_feature_value("Image", "Diagnosis", [
    {"target_rid": "1-ABC", "value": "Normal"},
    {"target_rid": "1-DEF", "value": "Abnormal"},
    {"target_rid": "1-GHI", "value": "Normal"},
])

Inputs: entries (required), table_name (required), feature_name (required), execution_rid. Returns: result

add_feature_value_recordA

Add feature values with multiple fields to one or more domain objects.

For features with multiple columns (e.g., a diagnosis with confidence score), use this tool to provide values for each field. Accepts a list of entries for batch insertion. Use lookup_feature first to see available fields.

Feature columns are dynamically generated based on the feature definition:

  • term_columns: Accept vocabulary term names (strings)

  • asset_columns: Accept asset RIDs (strings)

  • value_columns: Accept direct values (strings, numbers)

Args: table_name: Table the target records belong to (e.g., "Image"). feature_name: Name of the feature (e.g., "Diagnosis"). entries: List of dicts, each with: - target_rid (str, required): RID of the target record. - Plus any feature column names mapped to their values. Use lookup_feature to see available columns and types. execution_rid: Execution RID for provenance (uses active if not provided).

Returns: JSON with status, feature_name, count, execution_rid, rids.

Example: # First check the feature structure: lookup_feature("Image", "Diagnosis") # -> {"term_columns": {"Diagnosis_Type": {...}}, "value_columns": {"confidence": {...}}}

# Then add values (single or batch):
add_feature_value_record("Image", "Diagnosis", [
    {"target_rid": "1-ABC", "Diagnosis_Type": "Normal", "confidence": 0.95},
    {"target_rid": "1-DEF", "Diagnosis_Type": "Abnormal", "confidence": 0.87},
])

Inputs: entries (required), table_name (required), feature_name (required), execution_rid. Returns: result

add_nested_executionA

Add a child execution to a parent execution.

Creates a parent-child relationship between executions. Use this to group related executions, such as:

  • Parameter sweeps (parent = sweep, children = individual runs)

  • Pipelines (parent = pipeline, children = stages)

  • Cross-validation (parent = CV experiment, children = folds)

Args: parent_execution_rid: RID of the parent execution. child_execution_rid: RID of the child execution to nest. sequence: Optional ordering index (0, 1, 2...). Use None for parallel executions.

Returns: JSON with parent_rid, child_rid, sequence.

Example: # Create a sweep parent, then add child executions add_nested_execution("1-PARENT", "1-CHILD1", sequence=0) add_nested_execution("1-PARENT", "1-CHILD2", sequence=1)

Inputs: child_execution_rid (required), parent_execution_rid (required), sequence. Returns: result

add_synonymA

Add a synonym to an existing vocabulary term.

Synonyms are alternative names that can be used to look up a term.

Args: vocabulary_name: Name of the vocabulary table (e.g., "Dataset_Type"). term_name: Primary name of the term to add synonym to. synonym: Alternative name to add.

Returns: JSON with status, name, synonyms list.

Example: add_synonym("Dataset_Type", "Training", "train") -> adds "train" as synonym

Inputs: synonym (required), term_name (required), vocabulary_name (required). Returns: result

add_termA

Add a new term to a vocabulary.

Args: vocabulary_name: Name of the vocabulary table (e.g., "Dataset_Type"). term_name: Primary name for the term (must be unique). description: What this term means. synonyms: Alternative names that can also match this term.

Returns: JSON with status, name, description, synonyms, rid.

Example: add_term("Dataset_Type", "Validation", "Held-out data for validation", ["val", "valid"])

Inputs: term_name (required), description (required), vocabulary_name (required), synonyms. Returns: result

add_visible_columnA

Add a column to the visible-columns list for a specific context.

This is a convenience tool for adding columns without replacing the entire visible-columns annotation. Changes are staged until apply_annotations() is called.

Args: table_name: Name of the table. context: The context to modify. See Contexts below. column: Column to add. Can be: - String: column name (e.g., "Filename") - List: foreign key reference (e.g., ["schema", "fkey_name"]) - Dict: pseudo-column definition (see set_visible_columns) position: Position to insert at (0-indexed). If None, appends to end.

Contexts for visible-columns:

Context

Description

When Used

*

Default for all contexts

Fallback when specific context not set

compact

List/table view

Main record list, search results

compact/brief

Abbreviated list

Inline previews, tooltips

compact/brief/inline

Minimal inline

Foreign key cell display

compact/select

Selection modal

Picker dialogs for foreign keys

detailed

Full record view

Single record page

entry

Data entry forms

Both create and edit forms

entry/create

Create form only

New record creation

entry/edit

Edit form only

Editing existing records

export

Data export

CSV/JSON export

filter

Faceted search

Search sidebar (uses different format)

Returns: JSON with the updated column list for the context.

Examples: # Add column to end of compact view add_visible_column("Image", "compact", "Description")

# Add foreign key reference at position 1
add_visible_column("Image", "detailed", ["domain", "Image_Subject_fkey"], 1)

# Add pseudo-column
add_visible_column("Image", "compact", {
    "source": [{"outbound": ["domain", "Image_Subject_fkey"]}, "Name"],
    "markdown_name": "Subject"
})

Inputs: column (required), context (required), table_name (required), position. Returns: result

add_visible_foreign_keyA

Add a foreign key to the visible-foreign-keys list for a specific context.

This is a convenience tool for adding related tables without replacing the entire visible-foreign-keys annotation. Changes are staged until apply_annotations() is called.

Args: table_name: Name of the table. context: The context to modify (typically "detailed" or "*"). foreign_key: Foreign key to add. Can be: - List: inbound foreign key reference (e.g., ["schema", "Other_Table_fkey"]) - Dict: pseudo-column definition for complex relationships position: Position to insert at (0-indexed). If None, appends to end.

Contexts for visible-foreign-keys:

Context

Description

When Used

*

Default for all contexts

Fallback when specific context not set

detailed

Full record view

Related tables shown on single record page

Important: Only INBOUND foreign keys are valid - these are foreign keys from OTHER tables that reference THIS table. Use list_foreign_keys() to see which inbound foreign keys are available.

Returns: JSON with the updated foreign key list for the context.

Examples: # Add inbound foreign key to detailed view add_visible_foreign_key("Subject", "detailed", ["domain", "Image_Subject_fkey"])

# Add at specific position
add_visible_foreign_key("Subject", "detailed", ["domain", "Diagnosis_Subject_fkey"], 0)

# Add pseudo-column for complex relationship
add_visible_foreign_key("Subject", "detailed", {
    "source": [{"inbound": ["domain", "Image_Subject_fkey"]}],
    "markdown_name": "Subject Images"
})

Inputs: context (required), table_name (required), foreign_key (required), position. Returns: result

add_workflow_typeA

Add a new workflow type to the Workflow_Type vocabulary.

Args: type_name: Name for the new workflow type. description: What this type of workflow does.

Returns: JSON with status, name, description, rid.

Example: add_workflow_type("Data Augmentation", "Workflows that augment training data")

Inputs: type_name (required), description (required). Returns: result

apply_annotationsA

Apply all staged annotation changes to the catalog.

This commits any annotation changes made via set_display_annotation, set_visible_columns, set_visible_foreign_keys, set_table_display, or set_column_display to the remote catalog.

Returns: JSON with status and details of the apply operation.

Example workflow: 1. get_table_annotations("Image") # Check current state 2. set_display_annotation("Image", annotation={"name": "Images"}) 3. set_visible_columns("Image", {"compact": ["RID", "Filename"]}) 4. apply_annotations() # Commit all changes

Inputs: no named parameters. Returns: result

apply_catalog_annotationsA

Apply catalog-level annotations to initialize the Chaise web interface.

Chaise is Deriva's web-based data browser. This method sets up annotations that control how Chaise displays and organizes the catalog, including the navigation bar and display settings.

Navigation Bar Structure: Creates a navigation bar with organized dropdown menus:

  • User Info: Users, Groups, and RID Lease tables

  • Deriva-ML: Core ML tables (Workflow, Execution, Dataset, Dataset_Version, etc.)

  • WWW: Web content tables (Page, File)

  • {Domain Schema}: All domain-specific tables (excludes vocabularies/associations)

  • Vocabulary: All controlled vocabulary tables from ML and domain schemas

  • Assets: All asset tables from ML and domain schemas

  • Catalog Registry: Link to ermrest registry

  • Documentation: Links to ML docs and instructions

Display Settings:

  • Underscores in names displayed as spaces

  • System columns (RID) shown in views

  • Default landing page set to Dataset table

  • Faceted search and record deletion enabled

Bulk Upload: Configures drag-and-drop file upload for asset tables.

When to call: After creating the domain schema and all tables. The menus are dynamically built from the current schema structure.

Args: navbar_brand_text: Text in the navigation bar brand area (default: "ML Data Browser"). head_title: Browser tab title (default: "Catalog ML").

Returns: JSON with status and applied settings.

Example workflow: 1. create_catalog("localhost", "my_project") 2. create_vocabulary("Species", "Types of species") 3. create_asset("Image", ...) 4. apply_catalog_annotations("My ML Project", "ML Catalog")

Inputs: head_title, navbar_brand_text. Returns: result

bag_infoA

Get comprehensive info about a dataset bag: size, contents, and cache status.

Combines the size estimate (row counts, asset sizes per table) with local cache status. Use this to decide whether to cache a bag before running an experiment.

Cache status values:

  • "not_cached": No local copy exists

  • "cached_metadata_only": Table data downloaded, assets not fetched

  • "cached_materialized": Fully downloaded and validated

  • "cached_incomplete": Was cached but some assets are missing

Args: dataset_rid: RID of the dataset to inspect. version: Semantic version to inspect (e.g., "1.0.0"). exclude_tables: Optional list of table names to exclude from FK path traversal.

Returns: JSON with size info (tables, total_rows, total_asset_bytes, total_asset_size) plus cache_status and cache_path.

Inputs: version (required), dataset_rid (required), exclude_tables. Returns: result

cache_datasetA

Download a dataset bag or asset into the local cache without creating an execution.

Use this to warm the cache before running experiments. No execution or provenance records are created — this is purely a local download operation. After caching, subsequent download_dataset or download_execution_dataset calls will use the cached copy.

Provide either dataset_rid (for bags) or asset_rid (for individual assets), not both.

Args: dataset_rid: RID of a dataset to cache (mutually exclusive with asset_rid). asset_rid: RID of an asset to cache (mutually exclusive with dataset_rid). version: Dataset version to cache (required when using dataset_rid). materialize: If True (default), download all asset files in the bag. If False, download only table metadata (faster, smaller). Ignored for asset cache. exclude_tables: Optional list of table names to exclude from FK path traversal during bag export. Only applies to dataset cache.

Returns: JSON with cache results. For datasets: bag_info including cache_status and size. For assets: file path and metadata.

Inputs: version, asset_rid, dataset_rid, materialize, +1 more. Returns: result

cancel_taskA

Cancel a pending or running background task.

Args: task_id: The task ID to cancel.

Returns: JSON with cancellation status.

Note: Cancellation is best-effort. Long-running operations may not stop immediately.

Inputs: task_id (required). Returns: result

citeA

Generate a citation URL for a catalog entity.

Creates a permanent, citable URL for any catalog entity (dataset, execution, asset, etc.). By default, includes a snapshot timestamp for reproducibility. Use current=True for a link to the live data.

Args: rid: RID of the entity to cite (e.g., "1-ABC"). current: If True, return URL to current state without snapshot. If False (default), return permanent URL with snapshot timestamp.

Returns: JSON with: - url: The citation URL - rid: The entity RID - is_snapshot: Whether URL includes snapshot timestamp

Examples: cite("1-ABC") -> {"url": "https://host/id/catalog/1-ABC@2024-01-15", "is_snapshot": true}

cite("1-ABC", current=True)
-> {"url": "https://host/id/catalog/1-ABC", "is_snapshot": false}

Inputs: rid (required), current. Returns: result

clone_catalogA

Create an ML workspace by cloning data reachable from a root RID.

Creates a partial catalog clone containing only data reachable from the root RID (e.g., a project, dataset, or experiment). Uses the root table's export annotation (if available) to determine which tables and paths to follow, then fills in any uncovered tables (vocabularies, associations).

Uses a three-stage approach:

  1. Create schema WITHOUT foreign keys (only for included tables)

  2. Copy data asynchronously (export paths + fill-in tables)

  3. Apply foreign keys, handling violations based on orphan_strategy

Asset handling modes:

  • "none": Don't copy assets (asset columns will be empty)

  • "refs": Copy asset URLs only, files stay on source server (default)

  • "full": Download and re-upload all assets (fully independent clone)

Orphan handling: When source catalog policies hide some data but not references to it, cloning can result in dangling foreign keys. The orphan_strategy controls how these are handled.

Args: source_hostname: Source server hostname (e.g., "www.facebase.org"). source_catalog_id: ID of the catalog to clone. root_rid: The starting RID from which to trace reachability (e.g., a project RID like "3-HXMC"). dest_hostname: Destination hostname. If None, uses source hostname. alias: Optional alias name for the new catalog. add_ml_schema: If True, add the DerivaML schema to the clone. asset_mode: How to handle assets: "none", "refs" (default), or "full". copy_annotations: If True (default), copy all annotations. copy_policy: If True (default), copy ACL policies. exclude_schemas: List of schema names to exclude from cloning. exclude_objects: List of tables ("schema:table" format) to exclude. reinitialize_dataset_versions: If True (default), reinitialize dataset versions. orphan_strategy: How to handle orphan rows: "fail", "delete", or "nullify". prune_hidden_fkeys: If True, skip FKs with hidden reference data. truncate_oversized: If True, truncate values exceeding index size limits. include_tables: Additional tables to include. include_associations: If True, auto-include association tables. include_vocabularies: If True, auto-include vocabulary tables. table_concurrency: Max concurrent table copies during fill phase. Lower values reduce server load. Default: 1.

Returns: JSON with status, source info, destination info, and operation details including tables restored and orphan handling stats.

Examples: clone_catalog("www.facebase.org", "1", root_rid="3-HXMC", dest_hostname="localhost", alias="facebase-musmorph", add_ml_schema=True, orphan_strategy="delete")

Inputs: root_rid (required), source_hostname (required), source_catalog_id (required), alias, +15 more. Returns: result

clone_catalog_asyncA

Create an ML workspace by cloning data reachable from a root RID.

Starts the workspace creation in the background and immediately returns a task_id that you can use to check progress.

The operation uses export annotations (if available) to determine which tables and paths to follow from the root RID, then fills in any uncovered tables (vocabularies, associations). Uses async data copying for performance.

Use this for large catalogs or cross-server clones that may take several minutes to complete. Check progress with get_task_status(task_id).

Args: source_hostname: Source server hostname (e.g., "www.facebase.org"). source_catalog_id: ID of the catalog to clone. root_rid: The starting RID from which to trace reachability (e.g., "3-HXMC"). dest_hostname: Destination hostname. If None, uses source hostname. alias: Optional alias name for the new catalog. add_ml_schema: If True, add the DerivaML schema to the clone. asset_mode: How to handle assets: "none", "refs" (default), or "full". copy_annotations: If True (default), copy all annotations. copy_policy: If True (default), copy ACL policies. exclude_schemas: Schemas to exclude from cloning. exclude_objects: Tables ("schema:table") to exclude. reinitialize_dataset_versions: If True, reinitialize dataset versions. orphan_strategy: How to handle orphans: "fail", "delete", "nullify". prune_hidden_fkeys: Skip FKs with hidden reference data. truncate_oversized: Truncate values exceeding index limits. include_tables: Additional tables to include. include_associations: Include association tables. include_vocabularies: Include vocabulary tables. table_concurrency: Max concurrent table copies during fill phase. Lower values reduce server load. Default: 2.

Returns: JSON with task_id and status. Use get_task_status(task_id) to check progress.

Example: clone_catalog_async("www.facebase.org", "1", root_rid="3-HXMC", dest_hostname="localhost", alias="facebase-clone", orphan_strategy="delete") -> {"task_id": "abc123", "status": "started", ...}

# Check progress
get_task_status("abc123")
-> {"status": "running", "progress": {"percent_complete": 45.0, ...}}

# When done
get_task_status("abc123")
-> {"status": "completed", "result": {...}}

Inputs: root_rid (required), source_hostname (required), source_catalog_id (required), alias, +15 more. Returns: result

connect_catalogA

Connect to an existing DerivaML catalog. Must be called before using other tools.

On connection, an MCP workflow and execution are automatically created to track all operations performed through the MCP server. The workflow type "DerivaML MCP" is created if it doesn't exist.

Args: hostname: Server hostname (e.g., "dev.eye-ai.org", "www.atlas-d2k.org"). catalog_id: Catalog ID number (e.g., "1", "52"). domain_schema: Schema name for domain tables. Auto-detected if omitted. default_schema: Default schema for table creation and lookups. If omitted and there is exactly one domain schema, that schema is used. Required when multiple domain schemas exist and you want to avoid specifying the schema on every operation.

Returns: JSON with status, hostname, catalog_id, domain_schemas, default_schema, project_name, workflow_rid, execution_rid.

Example: connect_catalog("dev.eye-ai.org", "52") -> connects to eye-ai catalog connect_catalog("localhost", "10", domain_schema="isa", default_schema="isa")

Inputs: hostname (required), catalog_id (required), domain_schema, default_schema. Returns: result

create_asset_tableA

Create a new asset table for file management with automatic URL/checksum tracking.

Asset tables automatically include: URL, Filename, Length, MD5, Description. They integrate with executions for provenance tracking.

Args: asset_name: Name for the asset table (e.g., "Image", "Model", "Checkpoint"). columns: Additional columns beyond standard asset columns. referenced_tables: Tables this asset should have foreign keys to. comment: Description of the asset table's purpose. schema: Schema to create the table in. If not provided, uses the default domain schema.

Returns: JSON with status, table_name, schema, columns.

Example: create_asset_table("Image", [{"name": "Width", "type": "int4"}], ["Subject"])

Inputs: asset_name (required), schema, columns, comment, +1 more. Returns: result

create_catalogA

Create a new DerivaML catalog with all ML schema tables.

Creates a fresh catalog with Dataset, Execution, Workflow, Feature, and vocabulary tables. Automatically connects to the new catalog.

Args: hostname: Server hostname (e.g., "localhost", "deriva.example.org"). project_name: Name for the project, becomes the domain schema name. catalog_alias: Optional alias for the catalog. If provided, creates an alias that allows accessing the catalog by name instead of numeric ID (e.g., /ermrest/catalog/my-project instead of /ermrest/catalog/45).

Returns: JSON with status, hostname, catalog_id, catalog_alias (if created), domain_schema, project_name.

Example: create_catalog("localhost", "my_ml_project", "my-project")

Inputs: hostname (required), project_name (required), catalog_alias. Returns: result

create_catalog_aliasA

Create an alias for an existing catalog.

Aliases allow accessing a catalog by a memorable name instead of its numeric ID. For example, instead of /ermrest/catalog/21, you can use /ermrest/catalog/eye-ai.

Args: hostname: Server hostname (e.g., "www.eye-ai.org"). alias_name: The alias identifier (e.g., "my-project", "eye-ai"). Must be unique on the server. catalog_id: The numeric ID of the catalog to alias. name: Optional display name for the alias. description: Optional description of the alias.

Returns: JSON with status and alias details.

Example: create_catalog_alias("localhost", "my-project", "45", "My ML Project") -> {"status": "created", "alias": "my-project", "target": "45"}

Inputs: hostname (required), alias_name (required), catalog_id (required), name, +1 more. Returns: result

create_datasetA

Create a new empty dataset within an execution context.

The dataset is created through an execution for proper provenance tracking. Use add_dataset_members() to populate it after creation.

Assign Dataset_Type labels to categorize the dataset's role (e.g., "Training", "Testing", "Validation").

Args: description: Human-readable description of the dataset's purpose. dataset_types: Type labels from Dataset_Type vocabulary (e.g., ["Training", "Image"]). version: Initial version string (default: "0.1.0").

Returns: JSON with status, rid, description, dataset_types, version, execution_rid.

Example: create_dataset("Training images for model v2", ["Training"])

Inputs: version, description, dataset_types. Returns: result

create_dataset_type_termA

Create a new dataset type term in the Dataset_Type vocabulary.

This creates a new vocabulary term that can then be assigned to datasets using add_dataset_type(). Dataset types help categorize datasets by their role in ML workflows.

Common types include "Training", "Testing", "Validation", "Complete".

Args: type_name: Name for the dataset type (must be unique). description: What this type of dataset is used for. synonyms: Alternative names that can match this type (e.g., ["train"] for "Training").

Returns: JSON with status, name, description, synonyms, rid.

Example: create_dataset_type_term("Validation", "Held-out data for hyperparameter tuning", ["val", "valid"])

Inputs: type_name (required), description (required), synonyms. Returns: result

create_executionA

Create a new execution to track an ML workflow run with provenance.

This is the first step in the execution lifecycle. Specify input datasets and assets to establish provenance - these will be recorded as inputs to this workflow run.

LIFECYCLE (follow in order):

  1. create_execution() - You are here

  2. start_execution() - Begin timing

  3. [Run your ML workflow]

  4. stop_execution() - End timing

Args: workflow_name: Descriptive name (e.g., "ResNet50 Training Run 3"). workflow_type: Type from Workflow_Type vocabulary (e.g., "Training", "Inference"). description: What this execution does and why. dataset_rids: Input dataset RIDs for provenance tracking. asset_rids: Input asset RIDs for provenance tracking. dry_run: If True, download input datasets/assets but skip creating execution records in the catalog and skip uploading results. Useful for testing data loading, configuration, and model initialization without writing to the catalog.

Returns: JSON with execution_rid, workflow_rid, dataset_count, asset_count, dry_run.

Example: create_execution("CIFAR Training", "Training", "Train ResNet on CIFAR-10", ["1-ABC"]) create_execution("Test Run", "Training", "Debug data loading", dry_run=True)

Inputs: workflow_name (required), workflow_type (required), dry_run, asset_rids, +2 more. Returns: result

create_execution_datasetA

Create a new dataset as output from this execution.

Creates a dataset that is linked to this execution for provenance. Use this when your workflow produces a new curated collection of data (e.g., augmented training data, filtered results).

Args: description: What this dataset contains. dataset_types: Type labels (e.g., ["Training", "Augmented"]).

Returns: JSON with dataset_rid, execution_rid.

Inputs: description, dataset_types. Returns: result

create_featureA

Create a new feature definition to associate metadata with domain objects.

Features enable ML data engineering by linking labels, scores, or derived assets to domain objects. The feature definition specifies what types of values are valid.

What this creates:

  1. A new association table in the domain schema to store feature values

  2. A dynamically generated Pydantic model class for creating validated feature instances

The Pydantic model class (accessible via feature_record_class() in Python) provides type-safe construction of feature records with automatic validation against the feature's definition.

Feature types:

  • Term-based: Values come from controlled vocabularies (e.g., diagnosis labels)

  • Asset-based: Values reference asset files (e.g., segmentation masks)

  • Mixed: Can reference both terms and assets

The feature automatically tracks which Execution produced each value for provenance.

Args: table_name: Table to attach the feature to (e.g., "Image", "Subject"). feature_name: Unique name for the feature (e.g., "Diagnosis", "Quality_Score"). comment: Description of what this feature represents. terms: Vocabulary table names whose terms can be values (e.g., ["Diagnosis_Type"]). assets: Asset table names that can be referenced (e.g., ["Segmentation_Mask"]). metadata: Additional columns or table references to include in the feature. Each item can be: - A string: Treated as a table name (adds a foreign key reference) - A dict: Column definition with at minimum "name" and "type" keys. The "type" value should be a dict like {"typename": "float4"}. Valid type names: text, int2, int4, int8, float4, float8, boolean, date, timestamp, timestamptz, json, jsonb. Optional keys: "nullok" (bool), "default", "comment".

Returns: JSON with status, feature_name, target_table.

Examples: # Simple term-based feature create_feature("Image", "Diagnosis", "Clinical diagnosis label", terms=["Diagnosis_Type"])

# Feature with a confidence score column
create_feature("Image", "Diagnosis", "Diagnosis with confidence",
    terms=["Diagnosis_Type"],
    metadata=[{"name": "confidence", "type": {"typename": "float4"}}])

# Feature referencing another table
create_feature("Image", "Review", "Review annotations",
    terms=["Review_Status"],
    metadata=["Reviewer"])

Inputs: table_name (required), feature_name (required), terms, assets, +2 more. Returns: result

create_tableA

Create a new table in the domain schema.

This tool creates a standard table (not an asset table). For tables that store files with automatic URL/checksum tracking, use create_asset_table instead.

Process Overview:

  1. Define columns with names, types, and constraints

  2. Optionally define foreign keys to reference other tables

  3. The table is created in the domain schema

  4. The navigation bar is automatically updated

Args: table_name: Name for the new table (e.g., "Subject", "Experiment", "Protocol"). columns: Column definitions, each dict with: - name (str, required): Column name - type (str): One of "text", "int2", "int4", "int8", "float4", "float8", "boolean", "date", "timestamp", "timestamptz", "json", "jsonb", "markdown" (default: "text") - nullok (bool): Allow null values (default: True) - comment (str): Column description foreign_keys: Foreign key definitions, each dict with: - column (str, required): Column name in this table (must also be in columns list) - referenced_table (str, required): Name of the table to reference - referenced_column (str): Column in referenced table (default: "RID") - on_delete (str): Action on delete - "NO ACTION", "CASCADE", "SET NULL" (default: "NO ACTION") comment: Description of the table's purpose. schema: Schema to create the table in. If not provided, uses the default domain schema. Useful when a catalog has multiple domain schemas.

Returns: JSON with status, table_name, schema, columns.

Examples: Simple table: create_table("Subject", [ {"name": "Name", "type": "text", "nullok": false}, {"name": "Age", "type": "int4"}, {"name": "Notes", "type": "markdown"} ])

Table with foreign key:
    create_table("Sample", [
        {"name": "Name", "type": "text", "nullok": false},
        {"name": "Subject", "type": "text", "nullok": false},
        {"name": "Collection_Date", "type": "date"}
    ], foreign_keys=[
        {"column": "Subject", "referenced_table": "Subject", "on_delete": "CASCADE"}
    ])

Inputs: table_name (required), schema, columns, comment, +1 more. Returns: result

create_vocabularyA

Create a new vocabulary table for storing controlled terms.

Args: vocabulary_name: Name for the new vocabulary table. comment: Description of the vocabulary's purpose. schema: Schema to create in (default: domain schema).

Returns: JSON with status, name, schema, comment.

Example: create_vocabulary("Quality_Level", "Image quality ratings")

Inputs: vocabulary_name (required), schema, comment. Returns: result

create_workflowA

Create and register a new workflow definition.

Args: name: Display name for the workflow. workflow_type: Type from Workflow_Type vocabulary (e.g., "Training", "Inference"). description: What this workflow does.

Returns: JSON with status, rid, name, workflow_type, description.

Example: create_workflow("ResNet Training", "Training", "Trains ResNet50 on image data")

Inputs: name (required), workflow_type (required), description. Returns: result

delete_catalogA

PERMANENTLY DELETE a catalog and all its data. Cannot be undone.

Args: hostname: Server hostname. catalog_id: ID of the catalog to delete.

Returns: JSON with deletion status.

Inputs: hostname (required), catalog_id (required). Returns: result

delete_catalog_aliasA

Delete a catalog alias. The target catalog is NOT deleted.

Args: hostname: Server hostname (e.g., "www.eye-ai.org"). alias_name: The alias identifier to delete.

Returns: JSON with deletion status.

Example: delete_catalog_alias("localhost", "my-project") -> {"status": "deleted", "alias": "my-project"}

Inputs: hostname (required), alias_name (required). Returns: result

delete_datasetA

Soft-delete a dataset (marks deleted but preserves data).

Soft deletion hides the dataset from normal queries but keeps all data intact. For nested datasets, use recurse=True to also delete child datasets.

Args: dataset_rid: The RID of the dataset to delete. recurse: If True, also delete all nested child datasets.

Returns: JSON with status, dataset_rid, recursive.

Inputs: dataset_rid (required), recurse. Returns: result

delete_dataset_membersA

Remove records from a dataset. Auto-increments minor version.

Removes the specified records from the dataset's membership. The records themselves are not deleted from the catalog, only their association with this dataset is removed.

Removing members automatically increments the dataset's minor version for change tracking.

Args: dataset_rid: The RID of the dataset to remove members from. member_rids: List of RIDs to remove (e.g., ["2-ABC", "2-DEF", "2-GHI"]).

Returns: JSON with status, removed_count, dataset_rid.

Example: delete_dataset_members("1-ABC", ["2-DEF", "2-GHI"]) -> removes 2 records from dataset

Inputs: dataset_rid (required), member_rids (required). Returns: result

delete_dataset_type_termA

Delete a dataset type term from the Dataset_Type vocabulary.

WARNING: Only delete types that are not referenced by any datasets. If datasets use this type, the delete will fail with a foreign key error. Use remove_dataset_type() first to remove the type from all datasets.

Args: type_name: Name of the dataset type to delete.

Returns: JSON with status and deleted type name.

Example: delete_dataset_type_term("Obsolete") -> {"status": "deleted", "name": "Obsolete"}

Inputs: type_name (required). Returns: result

delete_featureA

Delete a feature definition and all its values. Cannot be undone.

WARNING: This permanently removes the feature table and all associated values. All provenance information for this feature will be lost.

Args: table_name: Table the feature is attached to. feature_name: Name of the feature to delete.

Returns: JSON with status, feature_name, table_name.

Inputs: table_name (required), feature_name (required). Returns: result

delete_termA

Delete a term from a vocabulary.

The term must not be in use by any records in the catalog. If the term is referenced by other records (e.g., datasets using this type), the delete will fail with an error listing how many records reference it.

Args: vocabulary_name: Name of the vocabulary table (e.g., "Dataset_Type"). term_name: Name of the term to delete.

Returns: JSON with status, vocabulary, and deleted term name.

Example: delete_term("Dataset_Type", "Obsolete") -> {"status": "deleted", ...}

Inputs: term_name (required), vocabulary_name (required). Returns: result

disconnect_catalogB

Disconnect from the currently active catalog.

Inputs: no named parameters. Returns: result

estimate_bag_sizeA

Estimate the size of a dataset bag before downloading.

Runs the same FK path traversal as a dataset bag download, then queries the snapshot catalog for row counts and asset file sizes. Use this to preview what a download will contain and how large it will be before committing to the full download.

Args: dataset_rid: RID of the dataset to estimate. version: Semantic version to estimate (e.g., "1.0.0"). exclude_tables: Optional list of table names to exclude from FK path traversal during bag export.

Returns: JSON with: - tables: dict of table name -> {row_count, is_asset, asset_bytes} - total_rows: total row count across all tables - total_asset_bytes: total asset size in bytes - total_asset_size: human-readable size (e.g., "1.2 GB")

Inputs: version (required), dataset_rid (required), exclude_tables. Returns: result

get_dataset_specA

Generate a DatasetSpecConfig string for use in Python configuration files.

Returns the exact Python code to use in hydra-zen config files. This ensures the RID and version are correctly formatted and match what's in the catalog.

IMPORTANT: Always prefer specifying explicit versions in configurations. Using current_version as a default can lead to unexpected changes in results if the dataset is modified after the configuration is written. Pin to a specific version for reproducibility.

Args: dataset_rid: The RID of the dataset (e.g., "28CT"). version: Specific version to use. If not provided, uses the dataset's current version (with a warning about reproducibility).

Returns: JSON with the Python code string and metadata including: - spec: The DatasetSpecConfig(...) string ready to paste into code - rid: The dataset RID - version: The version used - description: Dataset description for reference - warning: Present if using current_version (recommends explicit version)

Example: get_dataset_spec("28CT") -> {"spec": "DatasetSpecConfig(rid="28CT", version="0.21.0")", ...}

get_dataset_spec("28CT", "0.20.0")
-> {"spec": "DatasetSpecConfig(rid=\"28CT\", version=\"0.20.0\")", ...}

Inputs: dataset_rid (required), version. Returns: result

get_handlebars_template_variablesA

Get all available template variables for a table.

Returns the columns, foreign keys, and special variables that can be used in Handlebars templates (row_markdown_pattern, markdown_pattern, etc.) for the specified table.

Args: table_name: Name of the table to get variables for.

Returns: JSON with columns, foreign_keys, and special variables available for use in templates.

Example: get_handlebars_template_variables("Image") -> { "table": "Image", "columns": [ {"name": "RID", "type": "ermrest_rid", "template": "{{{RID}}}"}, {"name": "Filename", "type": "text", "template": "{{{Filename}}}"}, ... ], "foreign_keys": [ { "constraint": ["domain", "Image_Subject_fkey"], "to_table": "Subject", "values_template": "{{{$fkeys.domain.Image_Subject_fkey.values.column}}}", "row_name_template": "{{{$fkeys.domain.Image_Subject_fkey.rowName}}}" } ], "special_variables": {...} }

Inputs: table_name (required). Returns: result

get_recordA

Get a single record by its RID.

Args: table_name: Name of the table containing the record. rid: The RID of the record to fetch.

Returns: JSON with the complete record or error if not found.

Example: get_record("Image", "1-ABC") -> full image record

Inputs: rid (required), table_name (required). Returns: result

get_table_sample_dataA

Get sample row data from a table for template testing.

Retrieves a few sample rows from the table that can be used to test Handlebars templates. Use this to see real values that would be available in templates.

Args: table_name: Name of the table. limit: Number of sample rows to return (default: 3, max: 10).

Returns: JSON with sample rows and their column values.

Example: get_table_sample_data("Image", 2) -> { "table": "Image", "sample_rows": [ {"RID": "1-ABC", "Filename": "scan001.jpg", "Subject": "2-DEF", ...}, {"RID": "1-XYZ", "Filename": "scan002.jpg", "Subject": "2-DEF", ...} ], "template_test_suggestion": "Try: {{{Filename}}} - Subject: {{{Subject}}}" }

Inputs: table_name (required), limit. Returns: result

get_task_statusA

Get the status and progress of a background task.

Args: task_id: The task ID returned by an async operation. include_result: If True, include the full result when completed.

Returns: JSON with task status, progress, and optionally the result.

Example: get_task_status("abc123") -> { "task_id": "abc123", "status": "running", "progress": { "current_step": "Copying data", "percent_complete": 45.0, "message": "Copying table Subject..." } }

Inputs: task_id (required), include_result. Returns: result

increment_dataset_versionA

Manually increment a dataset's semantic version (major.minor.patch).

Description Handling (follows generate-descriptions prompt guidelines):

  1. If user provides description: Use it, potentially improving for clarity

  2. If description is empty: Generate from conversation context:

    • What catalog operations were performed since last version?

    • What was the user's stated goal for these changes?

    • Summarize the changes made (e.g., "Added X, fixed Y, modified Z")

Description Generation Guidelines:

  • Include WHAT changed (added images, fixed labels, new features)

  • Include WHY if known (QA review, batch import, schema update)

  • Include IMPACT if relevant (affects N records, breaking change)

  • Use markdown for complex descriptions (lists, tables)

Use this tool when:

  • You've modified catalog data and want changes visible in a dataset

  • You need to capture a snapshot of the current catalog state

  • You want to create a reproducible checkpoint before making changes

Args: dataset_rid: The RID of the dataset. description: What changed in this version. If empty, LLM should generate from context. Good descriptions include: - What was added, modified, or fixed - Why the change was made (if known) - Impact on users of this dataset

    Examples of good descriptions:
    - "Added 500 new labeled training images from batch 3"
    - "Fixed incorrect labels on 23 images identified in QA review"
    - "Schema change: added 'quality_score' column to Image table"
    - "Captured snapshot before label correction workflow"

component: Which part to increment: "major", "minor", or "patch".
    - major: Breaking changes or schema modifications
    - minor: New data added or non-breaking changes (default)
    - patch: Bug fixes or label corrections

Returns: JSON with status, new_version, previous_version, dataset_rid, description.

Examples: increment_dataset_version("1-ABC", "Added quality labels to all images", "minor") increment_dataset_version("1-ABC", "Fixed mislabeled cat images", "patch") increment_dataset_version("1-ABC", "Schema change: new metadata columns", "major")

Inputs: dataset_rid (required), component, description. Returns: result

insert_recordsA

Insert new records into a domain table.

IMPORTANT: This tool is for domain-specific tables only (e.g., Subject, Image metadata). Do NOT use for:

  • Datasets → use create_dataset(), add_dataset_members()

  • Features → use add_feature_value()

  • Vocabularies → use add_term()

  • Executions → use create_execution()

  • Workflows → use create_workflow()

  • Assets with files → use the DerivaML Python API execution workflow

Args: table_name: Name of the domain table to insert into. records: List of dictionaries with column values.

Returns: JSON with inserted_count and record RIDs.

Example: insert_records("Subject", [{"Name": "Patient A", "Age": 45}])

Inputs: records (required), table_name (required). Returns: result

invalidate_cacheA

Invalidate cached tabular query results.

Args: cache_key: Invalidate a specific cached result. source: Invalidate all results from this source ("bag" or "catalog"). If neither is provided, invalidates all cached results.

Returns: JSON with the number of entries invalidated.

Inputs: source, cache_key. Returns: result

list_asset_executionsA

List all executions associated with an asset.

Given an asset RID, returns a list of executions that created or used the asset, along with the role (Input/Output) in each execution. This is useful for provenance tracking - finding which execution created an asset or which executions used it as input.

Args: asset_rid: RID of the asset to look up. asset_role: Optional filter: "Input" or "Output". If omitted, returns all.

Returns: JSON array of execution records showing which executions are associated with this asset. Each record includes execution_rid, workflow_rid, status, and description.

Example: list_asset_executions("3JSE") -> finds all executions that created/used this asset list_asset_executions("3JSE", "Output") -> finds only the execution that created it

Inputs: asset_rid (required), asset_role. Returns: result

list_cached_resultsA

List all cached tabular query results.

Returns metadata for each cached result including the tool that produced it, parameters, row count, age, and cache key. Use the cache_key with query_cached_result to re-query with different sort/filter/pagination.

Returns: JSON with list of cached result entries.

Inputs: no named parameters. Returns: result

list_catalog_registryA

List all catalogs and aliases available on a Deriva server.

Queries the server's ermrest registry to discover available catalogs and their aliases. Use this to find catalogs before connecting.

Note: Returns all non-deleted catalogs and aliases. Typically bounded (servers have 10-50 catalogs), but could be larger on shared servers.

Args: hostname: Server hostname (e.g., "www.eye-ai.org", "dev.facebase.org").

Returns: JSON with: - hostname: The server queried - catalogs: List of {id, name, description} for each catalog - aliases: List of {id, alias_target, name, description} for each alias

Example: list_catalog_registry("www.eye-ai.org") -> {"hostname": "www.eye-ai.org", "catalogs": [...], "aliases": [...]}

Inputs: hostname (required). Returns: result

list_dataset_parentsA

List all parent datasets that contain this dataset as a child.

Args: dataset_rid: RID of the child dataset. recurse: If True, recursively list all ancestors (parents of parents). version: Semantic version to query (e.g., "1.0.0"). If not specified, uses the current version.

Returns: JSON array of parent datasets with {rid, description, dataset_types, current_version}.

Example: list_dataset_parents("1-ABC") -> direct parents only list_dataset_parents("1-ABC", recurse=True) -> all ancestors

Inputs: dataset_rid (required), recurse, version. Returns: result

list_nested_executionsA

List all child (nested) executions of an execution.

Args: execution_rid: RID of the parent execution. recurse: If True, return all descendants (children, grandchildren, etc.).

Returns: JSON array of {execution_rid, workflow_rid, status, description} for each child.

Example: list_nested_executions("1-PARENT") # Direct children only list_nested_executions("1-PARENT", recurse=True) # All descendants

Inputs: execution_rid (required), recurse. Returns: result

list_tasksA

List all background tasks for the current user.

Args: status: Filter by status: "pending", "running", "completed", "failed", "cancelled". task_type: Filter by type: "clone_catalog".

Returns: JSON list of tasks with their status and basic info.

Example: list_tasks(status="running") -> [{"task_id": "abc123", "status": "running", ...}]

Inputs: status, task_type. Returns: result

lookup_workflow_by_urlA

Find a workflow by its source URL.

Search for a workflow that was registered with the given source URL. Use this to check if a workflow for a specific script or notebook already exists before creating a new one.

Args: url: The source URL to search for (e.g., GitHub URL to script).

Returns: JSON with: - found: True if workflow exists, False otherwise - workflow: Full workflow details if found

Example: lookup_workflow_by_url("https://github.com/org/repo/blob/main/train.py")

Inputs: url (required). Returns: result

preview_denormalized_datasetA

Preview a denormalized (wide table) view of dataset tables.

Joins related dataset tables into a single wide table. Returns schema shape (columns, join path) and size estimates. Optionally returns actual row data when a dataset and limit are provided.

Modes:

  • No dataset_rid: Returns schema shape + global size estimates. Use this to explore what a denormalized join would look like.

  • With dataset_rid, limit=0: Returns shape + dataset-scoped estimates.

  • With dataset_rid, limit>0: Returns shape + estimates + row preview.

Tables are joined based on their foreign key relationships. Column names are prefixed with the source table name using dots (e.g., "Image.Filename", "Subject.RID"). Intermediate tables needed for the join are auto-discovered.

Args: include_tables: List of table names to include in the join. Tables are joined based on their foreign key relationships. Order doesn't matter - the join order is determined automatically. Add more tables iteratively to expand the denormalized view. dataset_rid: RID of the dataset to preview. If omitted, returns schema shape with global (catalog-wide) row counts. version: Semantic version to query (e.g., "1.0.0"). If not specified, uses the current version. Only used with dataset_rid. limit: Maximum rows to return (default: 0, max: 100). Only used with dataset_rid. Set to 0 for shape and estimates only.

Returns: JSON with columns, join_path, tables (per-table size info), total_rows, total_asset_bytes, total_asset_size. When limit > 0 with a dataset_rid, also includes rows and count.

Example: # Explore schema shape (no dataset needed) preview_denormalized_dataset(["Subject", "Report_HVF"]) -> {"columns": [...], "join_path": ["Report_HVF", "Observation", "Subject"], ...}

# Get dataset-scoped estimates
preview_denormalized_dataset(["Image", "Subject"], dataset_rid="1-ABC")
-> {"columns": [...], "tables": {"Image": {"row_count": 50}}, ...}

# Preview actual rows
preview_denormalized_dataset(["Image", "Subject"], dataset_rid="1-ABC", limit=10)
-> {"columns": [...], "tables": {...}, "rows": [...], "count": 10}

Inputs: include_tables (required), limit, version, dataset_rid. Returns: result

preview_tableA

Preview records from a table with optional column selection and filtering.

Returns a sample of records for understanding data structure and content. For bulk data access, use the DerivaML Python API directly.

Args: table_name: Name of the table to preview (e.g., "Image", "Subject", "Dataset"). columns: List of column names to return. Default: all columns. filters: Dictionary of {column: value} equality filters. limit: Maximum records to return (default: 25, max: 100). offset: Number of records to skip.

Returns: JSON with records array, count, and table name.

Examples: preview_table("Image") -> first 25 images preview_table("Image", columns=["RID", "Filename"], limit=10) preview_table("Subject", filters={"Species": "Human"})

Inputs: table_name (required), limit, offset, columns, +1 more. Returns: result

query_cached_resultA

Re-query a cached tabular result with different sort/filter/pagination.

Use list_cached_results to find available cache keys. This tool lets you paginate, sort, and filter previously computed results without re-executing the original query.

Args: cache_key: The cache key from a previous query result. sort_by: Column name to sort by (e.g., "Image.CDR"). sort_desc: Sort descending if True. filter_col: Column name to filter on. filter_val: Value to filter for (substring match, case-insensitive). limit: Maximum rows to return (default: 100). offset: Number of rows to skip for pagination.

Returns: JSON with columns, rows, count, and total_count.

Inputs: cache_key (required), limit, offset, sort_by, +3 more. Returns: result

rag_add_sourceA

Register a new documentation source for RAG indexing.

After adding a source, run rag_ingest(source_name=name) to index it.

Args: name: Unique name for this source (e.g., "my-project-docs") repo_owner: GitHub repository owner (e.g., "informatics-isi-edu") repo_name: GitHub repository name (e.g., "deriva-ml") branch: Git branch to index (default "main") path_prefix: Only index files under this path (default "docs/") include_patterns: File patterns to include (default ["*.md"]) doc_type: Document type tag for filtering (default "user-guide")

Returns: Dict confirming the source was added.

Inputs: name (required), repo_name (required), repo_owner (required), branch, +3 more.

rag_index_schemaA

Re-index the connected catalog's schema for RAG search.

Fetches the current schema from the connected catalog and indexes it for semantic search. This happens automatically on connect_catalog, but can be called manually after schema changes (e.g., after creating tables, adding columns, or creating features).

Uses schema hashing — returns immediately if unchanged.

Returns: Dict with indexing statistics (status, chunks_created, schema_hash).

Inputs: no named parameters.

rag_ingestA

Full crawl and index of documentation sources.

Crawls GitHub repositories, fetches all documentation files, chunks them, and indexes them for semantic search. This is a long-running operation that runs in the background.

Args: source_name: Specific source to ingest (e.g., "deriva-ml-docs"). If None, ingests all configured sources.

Returns: Dict with task ID for tracking progress, or immediate results if the operation completes quickly.

Inputs: source_name.

rag_remove_sourceA

Remove a documentation source and its indexed chunks.

This deletes all indexed chunks for the source and removes it from the configuration.

Args: name: Name of the source to remove (e.g., "my-project-docs")

Returns: Dict with removal status and number of chunks deleted.

Inputs: name (required).

rag_statusA

Get the status of the RAG documentation index.

Returns information about the index including total chunks, configured sources, and last update times.

Returns: Dict with index status, source configurations, and statistics.

Inputs: no named parameters.

rag_updateA

Incremental update of documentation index.

Checks for changed files in source repositories and only re-indexes files that have been added, modified, or deleted. Much faster than full ingestion when few files have changed.

Args: source_name: Specific source to update (e.g., "deriva-ml-docs"). If None, updates all configured sources.

Returns: Dict with task ID for tracking progress.

Inputs: source_name.

remove_asset_type_from_assetA

Remove an asset type from a specific asset.

Removes the association between an asset and a type.

Args: asset_rid: RID of the asset to modify. type_name: Name of the asset type to remove.

Returns: JSON with status, asset_rid, and updated types list.

Example: remove_asset_type_from_asset("3JSE", "Training_Data")

Inputs: asset_rid (required), type_name (required). Returns: result

remove_dataset_typeA

Remove a type from a dataset.

Removes a Dataset_Type vocabulary term from this dataset. The type must exist in the Dataset_Type vocabulary.

Args: dataset_rid: RID of the dataset. dataset_type: Name of the type to remove.

Returns: JSON with status, dataset_rid, dataset_types list.

Example: remove_dataset_type("1-ABC", "Training") -> removes "Training" type from dataset

Inputs: dataset_rid (required), dataset_type (required). Returns: result

remove_synonymA

Remove a synonym from an existing vocabulary term.

Args: vocabulary_name: Name of the vocabulary table (e.g., "Dataset_Type"). term_name: Primary name of the term to remove synonym from. synonym: Alternative name to remove.

Returns: JSON with status, name, updated synonyms list.

Example: remove_synonym("Dataset_Type", "Training", "train") -> removes "train" as synonym

Inputs: synonym (required), term_name (required), vocabulary_name (required). Returns: result

remove_visible_columnA

Remove a column from the visible-columns list for a specific context.

This is a convenience tool for removing columns without replacing the entire visible-columns annotation. Changes are staged until apply_annotations() is called.

Args: table_name: Name of the table. context: The context to modify (e.g., "compact", "detailed"). column: Column to remove. Can be: - String: column name to find and remove - List: foreign key reference [schema, constraint] to find and remove - Integer: index position to remove (0-indexed)

Returns: JSON with the updated column list for the context.

Examples: # Remove by column name remove_visible_column("Image", "compact", "Description")

# Remove by foreign key reference
remove_visible_column("Image", "detailed", ["domain", "Image_Subject_fkey"])

# Remove by position (first column)
remove_visible_column("Image", "compact", 0)

Inputs: column (required), context (required), table_name (required). Returns: result

remove_visible_foreign_keyA

Remove a foreign key from the visible-foreign-keys list for a specific context.

This is a convenience tool for removing related tables without replacing the entire visible-foreign-keys annotation. Changes are staged until apply_annotations() is called.

Args: table_name: Name of the table. context: The context to modify (e.g., "detailed", "*"). foreign_key: Foreign key to remove. Can be: - List: foreign key reference [schema, constraint] to find and remove - Integer: index position to remove (0-indexed)

Returns: JSON with the updated foreign key list for the context.

Examples: # Remove by foreign key reference remove_visible_foreign_key("Subject", "detailed", ["domain", "Image_Subject_fkey"])

# Remove by position (first foreign key)
remove_visible_foreign_key("Subject", "detailed", 0)

Inputs: context (required), table_name (required), foreign_key (required). Returns: result

reorder_visible_columnsA

Reorder columns in the visible-columns list for a specific context.

This is a convenience tool for reordering columns without manually reconstructing the list. Changes are staged until apply_annotations() is called.

Args: table_name: Name of the table. context: The context to modify (e.g., "compact", "detailed"). new_order: The new order specification. Can be: - List of indices: [2, 0, 1, 3] reorders by current positions - List of column names/refs: ["Name", "RID", ...] specifies exact order

Returns: JSON with the reordered column list for the context.

Examples: # Reorder by indices (move item at index 2 to front) reorder_visible_columns("Image", "compact", [2, 0, 1, 3, 4])

# Reorder by specifying exact column order
reorder_visible_columns("Image", "compact", ["Filename", "Subject", "RID"])

# Note: When using column names, all columns must be included
# or unmentioned columns will be removed from the list

Inputs: context (required), new_order (required), table_name (required). Returns: result

reorder_visible_foreign_keysA

Reorder foreign keys in the visible-foreign-keys list for a specific context.

This is a convenience tool for reordering related tables without manually reconstructing the list. Changes are staged until apply_annotations() is called.

Args: table_name: Name of the table. context: The context to modify (e.g., "detailed", "*"). new_order: The new order specification. Can be: - List of indices: [2, 0, 1] reorders by current positions - List of foreign key refs: [["schema", "fkey1"], ...] specifies exact order

Returns: JSON with the reordered foreign key list for the context.

Examples: # Reorder by indices (move item at index 2 to front) reorder_visible_foreign_keys("Subject", "detailed", [2, 0, 1])

# Reorder by specifying exact foreign key order
reorder_visible_foreign_keys("Subject", "detailed", [
    ["domain", "Diagnosis_Subject_fkey"],
    ["domain", "Image_Subject_fkey"]
])

Inputs: context (required), new_order (required), table_name (required). Returns: result

restore_executionA

Restore a previous execution to continue working with it.

Args: execution_rid: RID of the execution to restore (e.g., "1-ABC").

Returns: JSON with execution_rid, workflow_rid, dataset_count.

Inputs: execution_rid (required). Returns: result

set_active_catalogA

Switch the active catalog when multiple catalogs are connected.

Args: hostname: Server hostname of the catalog to activate. catalog_id: Catalog ID to activate.

Inputs: hostname (required), catalog_id (required). Returns: result

set_column_descriptionA

Set or update the description (comment) for a column.

Args: table_name: Name of the table containing the column. column_name: Name of the column to update. description: New description for the column.

Returns: JSON with status, table_name, column_name, description.

Example: set_column_description("Subject", "Age", "Subject age in years at enrollment")

Inputs: table_name (required), column_name (required), description (required). Returns: result

set_column_displayA

Set the column-display annotation on a column.

Controls how a column's values are rendered, including custom formatting and markdown patterns.

Changes are staged locally until apply_annotations() is called.

Args: table_name: Name of the table containing the column. column_name: Name of the column. annotation: The column-display annotation value. Set to null/None to remove.

Column-Display Annotation Schema (tag:isrd.isi.edu,2016:column-display):

{
    "*": {                                    // Default for all contexts
        "pre_format": {
            "format": "%.2f",                 // printf-style format string
            "bool_true_value": "Yes",         // Display for true
            "bool_false_value": "No"          // Display for false
        },
        "markdown_pattern": "**{{{value}}}**",  // Markdown template
        "template_engine": "handlebars",        // or "mustache"
        "column_order": false                   // Disable sorting, or specify sort
    },
    "compact": {...},                         // Compact/list view options
    "detailed": {...},                        // Detailed/record view options
    "entry": {...},                           // Entry form options
    "entry/create": {...},                    // Create form options
    "entry/edit": {...}                       // Edit form options
}

Available options:

  • pre_format: Pre-processing before display

    • format: printf-style format (e.g., "%.2f" for 2 decimal places)

    • bool_true_value: Text to show for boolean true

    • bool_false_value: Text to show for boolean false

  • markdown_pattern: Template using {{{column_name}}} substitution

  • template_engine: "handlebars" (default) or "mustache"

  • column_order: Sort configuration or false to disable sorting

Template variables in markdown_pattern:

  • {{{_value}}} or {{{value}}}: The column's value

  • {{{_row.column_name}}}: Another column's value from same row

  • {{{$fkeys.schema.fkey.values.column}}}: Value from related table

Returns: JSON with status and the target column.

Examples: # Format numbers with 2 decimal places set_column_display("Measurement", "Value", { "*": {"pre_format": {"format": "%.2f"}} })

# Display boolean as Yes/No
set_column_display("Subject", "Active", {
    "*": {
        "pre_format": {
            "bool_true_value": "Active",
            "bool_false_value": "Inactive"
        }
    }
})

# Custom markdown pattern
set_column_display("Image", "URL", {
    "detailed": {
        "markdown_pattern": "[![Image]({{{_value}}})]({{{_value}}})"
    }
})

# Disable column sorting
set_column_display("Subject", "Notes", {
    "*": {"column_order": false}
})

Inputs: table_name (required), column_name (required), annotation. Returns: result

set_column_display_nameA

Set the display name shown in the UI for a column.

This is a convenience tool for setting just the display name. For setting multiple display properties at once (name, markdown_name, etc.), use set_display_annotation with a column_name parameter instead.

Args: table_name: Name of the table containing the column. column_name: Name of the column to update. display_name: Human-readable name to display in the UI.

Returns: JSON with status, table_name, column_name, display_name.

Example: set_column_display_name("Subject", "DOB", "Date of Birth")

Inputs: table_name (required), column_name (required), display_name (required). Returns: result

set_column_nullokA

Set whether a column allows NULL values.

Args: table_name: Name of the table containing the column. column_name: Name of the column to update. nullok: True to allow NULL values, False to require values.

Returns: JSON with status, table_name, column_name, nullok.

Note: Setting nullok=False will fail if the column contains NULL values.

Example: set_column_nullok("Subject", "Name", False) # Make Name required

Inputs: nullok (required), table_name (required), column_name (required). Returns: result

set_dataset_descriptionA

Set or update the description for a dataset.

Updates the dataset's description in the catalog. Good descriptions help users understand the dataset's purpose, contents, and intended use.

Args: dataset_rid: RID of the dataset to update. description: New description text.

Returns: JSON with status, dataset_rid, description.

Example: set_dataset_description("1-ABC", "Training images for CIFAR-10 classification")

Inputs: dataset_rid (required), description (required). Returns: result

set_default_schemaA

Set the default schema for the active catalog connection.

When a catalog has multiple domain schemas, many operations require knowing which schema to use. Setting a default schema avoids having to specify it on every call.

Args: schema_name: Name of the domain schema to set as default (e.g., "isa", "my_project"). Must be one of the catalog's domain schemas.

Returns: JSON with status, default_schema, and domain_schemas.

Example: set_default_schema("isa") -> sets "isa" as the default schema

Inputs: schema_name (required). Returns: result

set_display_annotationA

Set the display annotation on a table or column.

The display annotation controls basic naming and display options. Changes are staged locally until apply_annotations() is called.

Args: table_name: Name of the table. column_name: Name of the column (optional). If provided, sets the annotation on the column; otherwise sets it on the table. annotation: The display annotation value. Set to null/None to remove.

Display Annotation Schema (tag:isrd.isi.edu,2015:display):

{
    "name": "string",           // Display name (mutually exclusive with markdown_name)
    "markdown_name": "string",  // Markdown-formatted name (mutually exclusive with name)
    "name_style": {
        "underline_space": true,  // Replace underscores with spaces
        "title_case": true,       // Apply title case
        "markdown": true          // Interpret as markdown
    },
    "comment": "string",        // Tooltip/description text
    "show_null": {              // How to display null values per context
        "*": true,              // true = show "No value", false = hide, "string" = custom
        "compact": false,
        "detailed": ""N/A""
    },
    "show_foreign_key_link": {  // Show FK as link per context
        "*": true,
        "compact": false
    }
}

Valid contexts for show_null/show_foreign_key_link:

  • "*" (all contexts)

  • "compact", "compact/select", "compact/brief", "compact/brief/inline"

  • "detailed"

Returns: JSON with status and the target (table or column).

Examples: # Set table display name set_display_annotation("Image", annotation={"name": "Images"})

# Set column display name
set_display_annotation("Image", "Filename", {"name": "File Name"})

# Set name style to replace underscores with spaces
set_display_annotation("Image", annotation={"name_style": {"underline_space": true}})

# Remove display annotation
set_display_annotation("Image", annotation=null)

Inputs: table_name (required), annotation, column_name. Returns: result

set_execution_descriptionA

Set or update the description for an execution.

Updates the execution's description in the catalog. Good descriptions help users understand what the execution accomplished and any notable results.

Args: execution_rid: RID of the execution to update. description: New description text.

Returns: JSON with status, execution_rid, description.

Example: set_execution_description("2-XYZ", "Training run with lr=0.001, achieved 95% accuracy")

Inputs: description (required), execution_rid (required). Returns: result

set_row_name_patternA

Set the pattern used to display row names in the UI.

The pattern uses Handlebars syntax with triple braces for column values.

Args: table_name: Name of the table to update. pattern: Handlebars template (e.g., "{{{Name}}}" or "{{{FirstName}}} {{{LastName}}}").

Returns: JSON with status, table_name, pattern.

Examples: set_row_name_pattern("Subject", "{{{Name}}}") set_row_name_pattern("Image", "{{{Filename}}} ({{{RID}}})")

Inputs: pattern (required), table_name (required). Returns: result

set_table_descriptionA

Set or update the description (comment) for a table.

Args: table_name: Name of the table to update. description: New description for the table.

Returns: JSON with status, table_name, description.

Example: set_table_description("Image", "Medical images for analysis")

Inputs: table_name (required), description (required). Returns: result

set_table_displayA

Set the table-display annotation on a table.

Controls table-level display options like row naming patterns, page size, and row ordering.

Changes are staged locally until apply_annotations() is called.

Args: table_name: Name of the table. annotation: The table-display annotation value. Set to null/None to remove.

Table-Display Annotation Schema (tag:isrd.isi.edu,2016:table-display):

{
    "row_name": {                              // How to display row identifiers
        "row_markdown_pattern": "{{{Name}}}",  // Template for row display
        "template_engine": "handlebars"        // or "mustache"
    },
    "detailed": {                              // Options for detailed view
        "hide_column_headers": true,           // Hide column headers
        "collapse_toc_panel": true             // Collapse table of contents
    },
    "compact": {                               // Options for compact/list view
        "page_size": 25,                       // Rows per page
        "row_order": [                         // Default sort order
            {"column": "RCT", "descending": true},
            "Name"
        ]
    },
    "*": {                                     // Default for all contexts
        "page_size": 10,
        "row_order": ["Name"]
    }
}

Context-specific options:

  • row_name: Special context for row identifier display

  • detailed: Detailed/record view

  • compact, compact/select, compact/brief: List views

  • *: Default for unspecified contexts

Available options per context:

  • row_order: Array of sort keys (column name or {column, descending})

  • page_size: Number of rows per page

  • collapse_toc_panel: Boolean to collapse TOC (detailed only)

  • hide_column_headers: Boolean to hide headers (detailed only)

  • row_markdown_pattern: Template string using {{{column}}} syntax

  • page_markdown_pattern: Template for entire page

  • separator_markdown: Separator between rows

  • prefix_markdown: Content before rows

  • suffix_markdown: Content after rows

  • template_engine: "handlebars" or "mustache"

Returns: JSON with status and the table name.

Examples: # Set row name pattern set_table_display("Subject", { "row_name": { "row_markdown_pattern": "{{{Name}}} ({{{Species}}})" } })

# Set default sort order and page size
set_table_display("Image", {
    "compact": {
        "row_order": [{"column": "RCT", "descending": true}],
        "page_size": 50
    }
})

# Configure detailed view
set_table_display("Page", {
    "detailed": {
        "hide_column_headers": true,
        "collapse_toc_panel": true
    }
})

Inputs: table_name (required), annotation. Returns: result

set_table_display_nameA

Set the display name shown in the UI for a table.

This is a convenience tool for setting just the display name. For setting multiple display properties at once (name, markdown_name, etc.), use set_display_annotation instead.

Args: table_name: Name of the table to update. display_name: Human-readable name to display in the UI.

Returns: JSON with status, table_name, display_name.

Example: set_table_display_name("Image", "Medical Images")

Inputs: table_name (required), display_name (required). Returns: result

set_visible_columnsA

Set the visible-columns annotation on a table.

Controls which columns appear in different UI contexts and their order. Changes are staged locally until apply_annotations() is called.

Args: table_name: Name of the table. annotation: The visible-columns annotation value. Set to null/None to remove.

Visible-Columns Annotation Schema (tag:isrd.isi.edu,2016:visible-columns):

{
    "compact": [...],           // Columns for compact/list view
    "detailed": [...],          // Columns for detailed/record view
    "entry": [...],             // Columns for data entry (create/edit)
    "entry/create": [...],      // Columns for create only
    "entry/edit": [...],        // Columns for edit only
    "export": [...],            // Columns for export
    "filter": {                 // Faceted search configuration
        "and": [...]
    },
    "*": [...]                  // Default for all contexts
}

Column directive formats (items in column lists):

  1. Simple column name (string):

    "RID"
  2. Foreign key reference (array of [schema, constraint_name]):

    ["schema_name", "fkey_constraint_name"]
  3. Pseudo-column (object with source path):

    {
        "source": "column_name",           // Simple column
        "source": [                        // Or path through foreign keys
            {"outbound": ["schema", "fkey"]},
            "target_column"
        ],
        "sourcekey": "predefined_key",    // OR reference to source-definitions
        "entity": true,                   // Show as entity (row) vs scalar value
        "aggregate": "array",             // Aggregation: min, max, cnt, cnt_d, array, array_d
        "self_link": true,                // Link to current row
        "markdown_name": "Display Name",  // Custom column header
        "comment": "Tooltip text",        // Column tooltip
        "display": {                      // Display options
            "markdown_pattern": "{{{value}}}",
            "template_engine": "handlebars",  // or "mustache"
            "show_foreign_key_link": true,
            "array_ux_mode": "csv"           // raw, csv, olist, ulist
        }
    }

Filter context (for faceted search):

{
    "filter": {
        "and": [
            {
                "source": "column_name",
                "markdown_name": "Filter Label",
                "open": true,              // Expand by default
                "ux_mode": "choices",      // choices, ranges, check_presence
                "bar_plot": true,          // Show distribution chart
                "hide_null_choice": true,  // Hide "No value" option
                "choices": ["val1", "val2"], // Preset choices
                "ranges": [{"min": 0, "max": 100}]  // Preset ranges
            }
        ]
    }
}

Returns: JSON with status and the table name.

Examples: # Simple column list for compact view set_visible_columns("Image", { "compact": ["RID", "Filename", "Subject"], "detailed": ["RID", "Filename", "Subject", "Description", "URL"] })

# Include foreign key as a column
set_visible_columns("Image", {
    "compact": ["Filename", ["domain", "Image_Subject_fkey"]]
})

# Pseudo-column traversing foreign key
set_visible_columns("Image", {
    "detailed": [
        "Filename",
        {
            "source": [{"outbound": ["domain", "Image_Subject_fkey"]}, "Name"],
            "markdown_name": "Subject Name"
        }
    ]
})

# Configure faceted search
set_visible_columns("Image", {
    "filter": {
        "and": [
            {"source": "Species", "open": true},
            {"source": "Quality", "ux_mode": "choices"}
        ]
    }
})

Inputs: table_name (required), annotation. Returns: result

set_visible_foreign_keysA

Set the visible-foreign-keys annotation on a table.

Controls which related tables (via inbound foreign keys) appear in different UI contexts and their order. These show as "related tables" sections in the detailed view.

Changes are staged locally until apply_annotations() is called.

Args: table_name: Name of the table. annotation: The visible-foreign-keys annotation value. Set to null/None to remove.

Visible-Foreign-Keys Annotation Schema (tag:isrd.isi.edu,2016:visible-foreign-keys):

{
    "detailed": [...],  // Related tables in detailed view
    "*": [...]          // Default for all contexts
}

Foreign key directive formats (items in the lists):

  1. Inbound foreign key reference (array of [schema, constraint_name]):

    ["schema_name", "fkey_constraint_name"]

    The constraint must be an INBOUND foreign key (i.e., another table references this table).

  2. Pseudo-column for related entities (object):

    {
        "source": [
            {"inbound": ["schema", "fkey_to_this_table"]},
            {"outbound": ["schema", "fkey_to_related"]},
            "column_name"
        ],
        "sourcekey": "predefined_key",    // OR reference to source-definitions
        "markdown_name": "Related Items",
        "comment": "Tooltip text",
        "display": {
            "markdown_pattern": "...",
            "template_engine": "handlebars"
        }
    }

Returns: JSON with status and the table name.

Examples: # Show specific related tables in detailed view set_visible_foreign_keys("Subject", { "detailed": [ ["domain", "Image_Subject_fkey"], ["domain", "Diagnosis_Subject_fkey"] ] })

# Hide all related tables
set_visible_foreign_keys("Subject", {"detailed": []})

# Pseudo-column for complex relationship
set_visible_foreign_keys("Subject", {
    "detailed": [
        {
            "source": [{"inbound": ["domain", "Image_Subject_fkey"]}],
            "markdown_name": "Subject Images"
        }
    ]
})

Inputs: table_name (required), annotation. Returns: result

set_workflow_descriptionA

Set or update the description for a workflow.

Updates the workflow's description in the catalog. Good descriptions help users understand what the workflow does and how to use it.

Args: workflow_rid: RID of the workflow to update. description: New description text.

Returns: JSON with status, workflow_rid, description.

Example: set_workflow_description("3-WKF", "Trains CNN on image data with augmentation")

Inputs: description (required), workflow_rid (required). Returns: result

split_datasetA

Split a dataset into training, testing, and optionally validation subsets.

Creates a new dataset hierarchy with full provenance tracking:

  • Split (parent, type: "Split")

    • Training (child, type: "Training" + training_types)

    • Validation (child, type: "Validation" + validation_types) # if val_size

    • Testing (child, type: "Testing" + testing_types)

The API follows scikit-learn's train_test_split conventions for test_size, train_size, val_size, shuffle, and seed parameters.

Splitting strategies:

  • Random (default): Shuffles members and splits at the boundary. No denormalization needed. Fast for any dataset size.

  • Stratified: Maintains class distribution across splits. Requires stratify_by_column and include_tables. Uses scikit-learn internally.

Column naming for stratification:

When using stratify_by_column, the column name must match the denormalized DataFrame format: {TableName}_{ColumnName}. For example, to stratify by the Image_Class column from the Image_Classification feature table, use Image_Classification_Image_Class.

Derive the column name from the table schema (via the deriva://catalog/schema or deriva://catalog/features resource) rather than calling denormalize_dataset().

Args: source_dataset_rid: RID of the source dataset to split. test_size: Test set size as a fraction (0-1) or absolute count. Default: 0.2 (20% of data). train_size: Train set size as a fraction (0-1) or absolute count. Default: None (complement of test_size and val_size). val_size: Validation set size as a fraction (0-1) or absolute count. Default: None (no validation split, two-way only). When provided, creates a three-way train/val/test split. seed: Random seed for reproducibility. Default: 42. shuffle: Whether to shuffle before splitting. Default: True. stratify_by_column: Column name in the denormalized DataFrame for stratified splitting. Maintains class distribution across all partitions. Requires include_tables. Example: "Image_Classification_Image_Class". stratify_missing: Policy for null values in the stratify column. "error" (default): raise if any nulls exist, reporting count and percentage. "drop": exclude rows with null values from the split. "include": treat nulls as a separate class. Only used when stratify_by_column is set. element_table: Element table to split (e.g., "Image"). If not specified, auto-detected from the dataset's members. include_tables: Tables to include when denormalizing. Required when using stratify_by_column. Example: ["Image", "Image_Classification"]. training_types: Additional dataset types for the training set beyond "Training". Example: ["Labeled"]. testing_types: Additional dataset types for the testing set beyond "Testing". Example: ["Labeled"]. validation_types: Additional dataset types for the validation set beyond "Validation". Example: ["Labeled"]. Ignored when val_size is None. split_description: Description for the parent Split dataset. dry_run: If True, return what would happen without modifying the catalog. Useful for previewing split sizes.

Returns: JSON with split results including: - split: RID, version, and count of the parent Split dataset - training: RID, version, and count of the Training dataset - validation: RID, version, and count of the Validation dataset (if val_size) - testing: RID, version, and count of the Testing dataset - source: RID of the source dataset

Example: # Random 80/20 split split_dataset("28D0", test_size=0.2, seed=42)

# Three-way train/val/test split
split_dataset("28D0", test_size=0.2, val_size=0.1, seed=42)

# Stratified split maintaining class balance
split_dataset("28D0", test_size=0.2,
             stratify_by_column="Image_Classification_Image_Class",
             include_tables=["Image", "Image_Classification"])

# Fixed-count split with labeled types
split_dataset("28D0", train_size=400, test_size=100,
             training_types=["Labeled"], testing_types=["Labeled"])

# Dry run to preview
split_dataset("28D0", test_size=0.2, dry_run=True)

Inputs: source_dataset_rid (required), seed, dry_run, shuffle, +11 more. Returns: result

start_executionA

Start timing the active execution. Call after create_execution().

Records the start timestamp for duration tracking. The execution status changes to "running".

Inputs: no named parameters. Returns: result

stop_executionA

Stop timing and mark execution complete.

Records the stop timestamp and calculates duration. Call this after your ML workflow completes.

Inputs: no named parameters. Returns: result

update_catalog_aliasA

Update an existing catalog alias.

Can change the target catalog the alias points to, or update the owner ACL.

Args: hostname: Server hostname (e.g., "www.eye-ai.org"). alias_name: The alias identifier to update. alias_target: New target catalog ID. If None, target is unchanged. Pass empty string "" to unbind the alias from any catalog. owner: New owner ACL (list of user/group identifiers). If None, owner is unchanged.

Returns: JSON with status and updated alias details.

Example: update_catalog_alias("localhost", "my-project", alias_target="50") -> {"status": "updated", "alias": "my-project", "target": "50"}

Inputs: hostname (required), alias_name (required), owner, alias_target. Returns: result

update_execution_statusA

Update the execution status with a progress message.

Args: status: One of "pending", "running", "completed", "failed". message: Progress message or error description.

Returns: JSON with execution_rid, new_status, message.

Inputs: status (required), message (required). Returns: result

update_recordA

Update fields in an existing record.

Args: table_name: Name of the table containing the record. rid: The RID of the record to update. updates: Dictionary of {column: new_value} updates.

Returns: JSON with update status.

Example: update_record("Subject", "1-ABC", {"Age": 46, "Status": "Active"})

Inputs: rid (required), updates (required), table_name (required). Returns: result

update_term_descriptionA

Update the description of a vocabulary term.

Args: vocabulary_name: Name of the vocabulary table (e.g., "Dataset_Type"). term_name: Primary name of the term to update. description: New description for the term.

Returns: JSON with status, name, updated description.

Example: update_term_description("Dataset_Type", "Training", "Data used to train models")

Inputs: term_name (required), description (required), vocabulary_name (required). Returns: result

validate_ridsA

Validate that RIDs exist in the catalog before running experiments.

Performs batch validation of RIDs to catch configuration errors early with clear error messages. Use this before running experiments to ensure all referenced datasets, assets, and other entities actually exist.

Args: dataset_rids: List of dataset RIDs to validate. asset_rids: List of asset RIDs to validate (model weights, etc.). dataset_versions: Dictionary mapping dataset RID to required version string (e.g., {"1-ABC": "0.4.0"}). Validates version exists. workflow_rids: List of workflow RIDs to validate. execution_rids: List of execution RIDs to validate. warn_missing_descriptions: If True (default), include warnings for datasets missing descriptions.

Returns: JSON with: - is_valid: True if all validations passed - errors: List of error messages - warnings: List of warning messages - validated_rids: Dictionary of validated RID info

Example: validate_rids( dataset_rids=["1-ABC", "2-DEF"], dataset_versions={"1-ABC": "0.4.0"}, asset_rids=["3-GHI"] ) -> { "is_valid": true, "errors": [], "warnings": [], "validated_rids": {...} }

Inputs: asset_rids, dataset_rids, workflow_rids, execution_rids, +2 more. Returns: result

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    B3.4/5.0

    Scored across 101 tools

    Disambiguation2/5

    The annotation tools heavily overlap: set_visible_columns vs. add/remove/reorder_visible_column, set_display_annotation vs. set_table_display/set_column_display, and set_table_display vs. set_table_display_name are easy to confuse. There are also near-duplicate pairs like clone_catalog/clone_catalog_async and create_dataset_type_term/add_term, plus similar-sounding create_execution and create_execution_dataset. With 101 tools, an agent would frequently struggle to pick the right one.

    Naming Consistency3/5

    Most names use snake_case with a verb prefix, but the conventions are mixed: set_*, add_*, remove_*, create_*, delete_*, update_*, and apply_* are used inconsistently for parallel operations (e.g., set_visible_columns vs. add_visible_column). Some names break the pattern entirely (bag_info, rag_update, cite), and the RAG tools mix noun-first and verb-first naming. Overall it is readable but not predictable.

    Tool Count1/5

    101 tools is an extreme count for a single MCP server and creates a massive selection burden. Even accounting for the broad DerivaML domain, many tools are convenience variants or specialized duplicates that bloat the surface. This is well beyond the range where an agent can efficiently discover and choose the correct tool.

    Completeness2/5

    The server covers many workflows but has notable gaps: tools referenced in examples like get_table_annotations, list_foreign_keys, and lookup_feature are not actually provided, and there are no direct list/get tools for core entities such as datasets, workflows, or features. These missing read/discovery operations force agents to use workarounds like preview_table or rag_search, and can cause failures when following described workflows.

    Maintenance

    ActivityNo data
    ResponsivenessOutdated data

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      D
      maintenance
      Enables AI assistants to interact with Databricks workspaces programmatically, providing comprehensive tools for cluster management, notebook operations, job orchestration, Unity Catalog data governance, user management, permissions control, and FinOps cost analytics.
      MIT
    • A
      license
      D
      quality
      D
      maintenance
      Enables AI assistants to perform MLOps workflows such as experiment tracking, model registry, dataset management, pipeline orchestration, and data lineage by wrapping DVC, MLflow, and Git.
      100
      1
      MIT
    • A
      license
      Not graded
      quality
      D
      maintenance
      Enables AI systems to manage Verodat accounts, workspaces, and datasets, including creating datasets, querying data, and executing AI-powered queries through natural language.
      9 npm
      Apache 2.0