Deriva MCP Server
OfficialDesigned to work alongside the GitHub MCP Server for version control of ML configurations, managing workflow code and model implementations, and collaborative development of ML experiments.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@Deriva MCP Server create a new dataset version named 'experiment-v2'"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Deriva MCP Server
Model Context Protocol (MCP) server that exposes Deriva catalog operations and DerivaML ML workflow tools for LLM applications.
Overview
This MCP server provides an interface to Deriva catalogs and DerivaML, enabling AI assistants like Claude to:
Connect to and manage Deriva catalogs
Create and manage datasets with versioning
Work with controlled vocabularies
Define and execute ML workflows
Create and manage features for ML experiments
For full ML workflow management, this server is designed to work alongside the GitHub MCP Server to enable:
Storing and versioning hydra-zen configurations in GitHub repositories
Managing workflow code and model implementations
Collaborative development of ML experiments
Related MCP server: mlops-mcp-server
Prerequisites
Deriva Authentication
DerivaML uses Globus for authentication. Before using the MCP server, you must authenticate with your Deriva server:
# Install deriva-ml if not already installed
pip install deriva-ml
# Authenticate with your Deriva server
python -c "from deriva_ml import DerivaML; DerivaML.globus_login('your-server.org')"This opens a browser window for Globus authentication. Credentials are cached locally and persist across sessions.
Alternatively, use the Deriva Auth Agent for browser-based authentication:
Install the Deriva Auth Agent from deriva-py
Run
deriva-globus-auth-utils login --host your-server.org
GitHub Authentication (for configuration management)
Create a GitHub Personal Access Token (PAT) for the GitHub MCP Server:
Create a fine-grained token with these permissions:
Repository access: Select repositories containing your ML configurations
Permissions:
Contents: Read and write (for pushing configs)
Pull requests: Read and write (optional, for PR workflows)
Issues: Read (optional, for tracking)
Copy the token securely - you'll need it for configuration
Installation
Using Docker (Recommended)
Docker provides the simplest setup with no Python environment management. The image is automatically built and published to GitHub Container Registry on every commit to main.
# Pull the latest image from GitHub Container Registry
docker pull ghcr.io/informatics-isi-edu/deriva-mcp:latest
# Run the server (for testing)
docker run --rm -it ghcr.io/informatics-isi-edu/deriva-mcp:latest --helpTo build locally instead:
git clone https://github.com/informatics-isi-edu/deriva-mcp.git
cd deriva-mcp
./scripts/docker-build.shUsing uv
uv pip install deriva-mcpUsing pip
pip install deriva-mcpFrom source
git clone https://github.com/informatics-isi-edu/deriva-mcp.git
cd deriva-mcp
uv syncClaude Code Plugin (Skills)
The Deriva skills plugin for Claude Code provides 30+ skills that guide Claude through common Deriva and DerivaML workflows. The skills are maintained in a separate repository: deriva-skills.
Installing the Plugin
# Add the marketplace (one-time)
/plugin marketplace add informatics-isi-edu/deriva-skills
# Install the plugin
/plugin install derivaUpdating the Plugin
To update to the latest version:
/plugin install derivaOr check your entire DerivaML ecosystem:
/deriva:check-versionsThis checks three components — the deriva-ml Python package, the deriva-skills plugin, and the deriva-mcp MCP server — against upstream releases and offers to update outdated ones.
See the deriva-skills README for the full list of available skills.
Development: Testing Skills Locally
During development, load the plugin from a local path without installing:
claude --plugin-dir /path/to/deriva-skillsConfiguration
Claude Desktop - Full Setup with GitHub Integration
For the complete ML workflow experience, configure both DerivaML and GitHub MCP servers together.
Configuration file locations:
macOS:
~/Library/Application Support/Claude/claude_desktop_config.jsonWindows:
%APPDATA%\Claude\claude_desktop_config.jsonLinux:
~/.config/Claude/claude_desktop_config.json
Option 1: Both Servers with Docker (Recommended)
Uses Docker for both MCP servers - most consistent setup:
{
"mcpServers": {
"deriva": {
"type": "stdio",
"command": "/bin/sh",
"args": [
"-c",
"docker run -i --rm --add-host localhost:host-gateway -e HOME=$HOME -v $HOME/.deriva:$HOME/.deriva:ro -v $HOME/.bdbag:$HOME/.bdbag -v $HOME/.deriva-ml:$HOME/.deriva-ml ghcr.io/informatics-isi-edu/deriva-mcp:latest"
],
"env": {}
},
"github": {
"type": "stdio",
"command": "docker",
"args": [
"run", "-i", "--rm",
"-e", "GITHUB_PERSONAL_ACCESS_TOKEN",
"ghcr.io/github/github-mcp-server"
],
"env": {
"GITHUB_PERSONAL_ACCESS_TOKEN": "ghp_your_token_here"
}
}
}
}Docker arguments explained:
--add-host localhost:host-gateway- Allows connecting to a Deriva server running on localhost-e HOME=$HOME- Passes your home directory path into the container so mounted paths are found correctly
For localhost with self-signed certificates, the image defaults to using ~/.deriva/allCAbundle-with-local.pem as the CA bundle. See Troubleshooting for how to create this file.
Volume mounts explained:
$HOME/.deriva:$HOME/.deriva:ro- Mounts your Deriva credentials (read-only)$HOME/.bdbag:$HOME/.bdbag- Mounts bdbag keychain for dataset download authentication (writable)$HOME/.deriva-ml:$HOME/.deriva-ml- Working directory for execution outputs (writable)
Note: Create the workspace directory before first use:
mkdir -p ~/.deriva-mlIf the directory doesn't exist, Docker creates it as root, causing permission issues.
Option 2: Direct Install with GitHub Remote
Uses pip-installed DerivaML MCP with GitHub's hosted server:
{
"mcpServers": {
"deriva": {
"type": "stdio",
"command": "deriva-mcp",
"env": {}
},
"github": {
"type": "stdio",
"command": "npx",
"args": ["-y", "@anthropic-ai/github-mcp-server"],
"env": {
"GITHUB_PERSONAL_ACCESS_TOKEN": "ghp_your_token_here"
}
}
}
}Option 3: From Source (Development)
For development or customization:
{
"mcpServers": {
"deriva": {
"type": "stdio",
"command": "uv",
"args": [
"--directory",
"/path/to/deriva-mcp",
"run",
"deriva-mcp"
],
"env": {}
},
"github": {
"type": "stdio",
"command": "docker",
"args": [
"run", "-i", "--rm",
"-e", "GITHUB_PERSONAL_ACCESS_TOKEN",
"ghcr.io/github/github-mcp-server"
],
"env": {
"GITHUB_PERSONAL_ACCESS_TOKEN": "ghp_your_token_here"
}
}
}
}Option 4: DerivaML Only (No GitHub)
If you don't need GitHub integration:
{
"mcpServers": {
"deriva": {
"type": "stdio",
"command": "/bin/sh",
"args": [
"-c",
"docker run -i --rm --add-host localhost:host-gateway -e HOME=$HOME -v $HOME/.deriva:$HOME/.deriva:ro -v $HOME/.bdbag:$HOME/.bdbag -v $HOME/.deriva-ml:$HOME/.deriva-ml ghcr.io/informatics-isi-edu/deriva-mcp:latest"
],
"env": {}
}
}
}Or with direct install:
{
"mcpServers": {
"deriva": {
"type": "stdio",
"command": "deriva-mcp",
"env": {}
}
}
}Claude Code
Add to ~/.mcp.json (global) or your project's .mcp.json file:
With Docker:
{
"mcpServers": {
"deriva": {
"type": "stdio",
"command": "/bin/sh",
"args": [
"-c",
"docker run -i --rm --add-host localhost:host-gateway -e HOME=$HOME -v $HOME/.deriva:$HOME/.deriva:ro -v $HOME/.bdbag:$HOME/.bdbag -v $HOME/.deriva-ml:$HOME/.deriva-ml ghcr.io/informatics-isi-edu/deriva-mcp:latest"
],
"env": {}
},
"github": {
"type": "stdio",
"command": "docker",
"args": [
"run", "-i", "--rm",
"-e", "GITHUB_PERSONAL_ACCESS_TOKEN",
"ghcr.io/github/github-mcp-server"
],
"env": {
"GITHUB_PERSONAL_ACCESS_TOKEN": "${GITHUB_PERSONAL_ACCESS_TOKEN}"
}
}
}
}With direct install:
{
"mcpServers": {
"deriva": {
"type": "stdio",
"command": "deriva-mcp",
"env": {}
},
"github": {
"type": "stdio",
"command": "npx",
"args": ["-y", "@anthropic-ai/github-mcp-server"],
"env": {
"GITHUB_PERSONAL_ACCESS_TOKEN": "ghp_your_token_here"
}
}
}
}Then enable in .claude/settings.local.json:
{
"enableAllProjectMcpServers": true,
"enabledMcpjsonServers": ["deriva", "github"]
}HTTP Transport Mode (for Long-Running Operations)
For operations that take more than a few minutes (like catalog cloning), HTTP transport provides a persistent connection that survives client disconnects. The server runs as a background service and maintains task state across reconnections.
Starting the HTTP Server
Using Docker Compose (Recommended):
# Requires the deriva-localhost Docker network for localhost catalogs
docker-compose -f docker-compose.mcp.yaml up -d
# View logs
docker-compose -f docker-compose.mcp.yaml logs -f
# Stop the server
docker-compose -f docker-compose.mcp.yaml downUsing Docker directly:
docker run -d --name deriva-mcp \
-p 8000:8000 \
--network deriva-localhost_internal_network \
-e HOME=$HOME \
-e DERIVA_MCP_LOCALHOST_ALIAS=deriva-webserver \
-v $HOME/.deriva:$HOME/.deriva:ro \
-v $HOME/.bdbag:$HOME/.bdbag \
-v $HOME/.deriva-ml:$HOME/.deriva-ml \
ghcr.io/informatics-isi-edu/deriva-mcp:latest \
deriva-mcp --transport streamable-http --host 0.0.0.0 --port 8000The DERIVA_MCP_LOCALHOST_ALIAS variable tells the entrypoint to resolve the Docker DNS name deriva-webserver to an IP and add it to /etc/hosts as localhost. This avoids hardcoding container IPs that change when the network is recreated.
Running locally (development):
# From source
uv run deriva-mcp --transport streamable-http --host 0.0.0.0 --port 8000
# If installed via pip
deriva-mcp --transport streamable-http --host 0.0.0.0 --port 8000Client Configuration for HTTP
Claude Code (~/.mcp.json):
{
"mcpServers": {
"deriva": {
"type": "http",
"url": "http://localhost:8000/mcp"
}
}
}Claude Desktop:
{
"mcpServers": {
"deriva": {
"type": "http",
"url": "http://localhost:8000/mcp"
}
}
}Benefits of HTTP Transport
No idle timeouts: Long-running operations complete without connection drops
Persistent server: Server survives client disconnects and restarts
Task state preservation: Background tasks continue even if you close Claude
Multiple clients: Multiple Claude sessions can share the same server
Health checks: Docker can automatically restart unhealthy servers
Verifying the HTTP Server
# Check server health (uses /health endpoint, not /mcp which creates sessions)
curl -s http://localhost:8000/health
# View server logs
docker-compose -f docker-compose.mcp.yaml logs -fVS Code with Continue or Cline
Add to your MCP configuration (typically .vscode/mcp.json):
{
"mcp": {
"servers": {
"deriva": {
"type": "stdio",
"command": "/bin/sh",
"args": [
"-c",
"docker run -i --rm --add-host localhost:host-gateway -e HOME=$HOME -v $HOME/.deriva:$HOME/.deriva:ro -v $HOME/.bdbag:$HOME/.bdbag -v $HOME/.deriva-ml:$HOME/.deriva-ml ghcr.io/informatics-isi-edu/deriva-mcp:latest"
],
"env": {}
},
"github": {
"type": "stdio",
"command": "docker",
"args": [
"run", "-i", "--rm",
"-e", "GITHUB_PERSONAL_ACCESS_TOKEN",
"ghcr.io/github/github-mcp-server"
],
"env": {
"GITHUB_PERSONAL_ACCESS_TOKEN": "ghp_your_token_here"
}
}
}
}
}Environment Variables
For security, store tokens in environment variables instead of config files:
# Add to ~/.bashrc, ~/.zshrc, or equivalent
export GITHUB_PERSONAL_ACCESS_TOKEN="ghp_your_token_here"Then reference in config:
{
"mcpServers": {
"github": {
"type": "stdio",
"command": "docker",
"args": ["run", "-i", "--rm", "-e", "GITHUB_PERSONAL_ACCESS_TOKEN", "ghcr.io/github/github-mcp-server"],
"env": {
"GITHUB_PERSONAL_ACCESS_TOKEN": "${GITHUB_PERSONAL_ACCESS_TOKEN}"
}
}
}
}Verifying Your Setup
After configuration, verify both servers are working:
User: What MCP servers are available?
Claude: I have access to two MCP servers:
1. deriva - For managing ML workflows in Deriva catalogs
2. github - For managing GitHub repositories and configurations
User: Connect to the deriva catalog at example.org with ID 42
Claude: [Uses connect_catalog tool]
Connected to example.org, catalog 42. The domain schema is 'my_project'.
User: List the hydra-zen configs in the my-ml-project repo
Claude: [Uses GitHub get_file_contents tool]
Found configuration files in configs/:
- deriva.py - DerivaML connection settings
- datasets.py - Dataset specifications
- model.py - Model hyperparametersAvailable Tools
Catalog Management
Tool | Description |
| Connect to a DerivaML catalog |
| Disconnect from the active catalog |
| List all active connections |
| Set which connection is active |
| Get information about the active catalog |
| List users with catalog access |
| Get web interface URL for a table |
| Find which table a RID belongs to |
| List all catalogs and aliases on a server |
| Create a new DerivaML catalog (with optional alias) |
| Permanently delete a catalog |
| Clone a catalog to create a copy |
Catalog Alias Management
Tool | Description |
| Create an alias for a catalog |
| Get alias metadata (target, owner) |
| Update alias target or owner |
| Delete an alias (catalog not affected) |
Dataset Management
Tool | Description |
| Find all datasets in the catalog |
| Look up detailed information about a dataset |
| Create a new dataset |
| List members of a dataset |
| Add members to a dataset |
| Get version history |
| Update dataset version |
| Delete a dataset |
| List valid element types |
| Enable a table as element type |
Vocabulary Management
Tool | Description |
| List all vocabulary tables |
| List terms in a vocabulary |
| Find a term by name or synonym |
| Add a term to a vocabulary |
| Create a new vocabulary table |
Workflow Management
Tool | Description |
| Find all workflows |
| Find a workflow by URL/checksum |
| Create and register a workflow |
| List available workflow types |
| Add a new workflow type |
Feature Management
Tool | Description |
| Find features for a table |
| Get feature details |
| Get all values for a feature |
| Create a feature definition |
| Delete a feature |
| List all feature names |
Schema Management
Tool | Description |
| Create a new table in the domain schema |
| Create an asset table for file management |
| List all assets in an asset table |
| List all tables in the domain schema |
| Get column and key definitions for a table |
| List available asset type terms |
| Add a new asset type to the vocabulary |
Execution Management
Tool | Description |
| Create a new execution for ML workflows |
| Start the active execution |
| Stop and complete the active execution |
| Update execution status and message |
| Get details about the active execution |
| Restore a previous execution by RID |
| Register a file for upload as an execution output |
| Commit all registered outputs to the catalog |
| List recent executions |
| Create a dataset within an execution |
| Download a dataset for processing |
| Get the working directory path |
Execution Workflow
The typical execution workflow using the context manager:
with execution.execute() as exe:
# Do your work here
exe.asset_file_path(asset_name="Image", file_name="output.png")
# ... more processing ...
# After context exits, commit output assets
execution.commit_output_assets()Using MCP tools, the equivalent workflow is:
create_execution()- Create the execution record with workflow infostart_execution()- Mark execution as running, begin timingasset_file_path()- Register output files (repeat as needed)stop_execution()- Mark execution as completecommit_output_assets()- Required: Commit all registered files to catalog
Important: You must call commit_output_assets() after completing your work
to commit any registered assets to the catalog. This is not automatic.
Available Resources
MCP resources provide read-only access to catalog information and configuration templates.
Static Resources - Configuration Templates
These resources provide code templates for configuring DerivaML with hydra-zen:
Resource URI | Description |
| Hydra-zen configuration template for DerivaML connection |
| Configuration template for dataset specifications |
| Configuration template for ML executions |
| Configuration template for ML models with zen_partial |
Dynamic Resources - Catalog Information
These resources return current catalog state (requires active connection):
Resource URI | Description |
| Current catalog schema structure in JSON |
| All vocabulary tables and their terms |
| All datasets in the current catalog |
| All registered workflows |
| All feature names defined in the catalog |
Template Resources - Parameterized
These resources accept parameters to return specific information:
Resource URI | Description |
| Detailed information about a specific dataset |
| Features defined for a specific table |
| Terms in a specific vocabulary table |
Documentation Resources
Documentation is fetched dynamically from GitHub repositories with 1-hour caching:
Resource URI | Description |
| DerivaML overview and architecture |
| Guide to creating and managing datasets |
| Guide to defining and using features |
| Guide to configuring ML executions |
| Guide to hydra-zen configuration |
| Guide to managing file assets |
| Guide to Jupyter notebook integration |
| Guide to RIDs, MINIDs, and identifiers |
| Installation instructions |
| ERMrest API documentation |
| Chaise UI documentation |
| Deriva Python SDK documentation |
Using Resources
Resources are accessed differently than tools - they provide static or semi-static data that can be read without side effects:
User: Show me the DerivaML configuration template
Claude: [Reads deriva://config/deriva-ml-template resource]
Here's a hydra-zen configuration template for DerivaML...
User: What datasets are in the catalog?
Claude: [Reads deriva://catalog/datasets resource]
Found the following datasets in your catalog...Usage Examples
Discovering and Connecting to Catalogs
User: What catalogs are available on example.org?
Claude: [Uses list_catalog_registry tool]
Found 3 catalogs on example.org:
- ID: 21, Name: "ML Project", Persistent: true
- ID: 45, Name: "Test Environment", Persistent: true
- ID: 50, Name: "Clone of ML Project", Persistent: true
And 2 aliases:
- "ml-project" -> catalog 21
- "test-env" -> catalog 45
User: Connect to ml-project
Claude: [Uses connect_catalog tool with catalog_id="ml-project"]
Connected to example.org using alias "ml-project" (catalog 21).
The domain schema is 'my_domain'.Cloning and Managing Catalogs
User: Clone the production catalog to create a test environment
Claude: [Uses clone_catalog tool]
Cloned catalog 21 to new catalog 52 with all data and policies.
User: Create an alias "dev-test" for the new catalog
Claude: [Uses create_catalog_alias tool]
Created alias "dev-test" pointing to catalog 52.
You can now connect using: connect_catalog("example.org", "dev-test")Basic Catalog Operations
User: Connect to the deriva catalog at example.org with ID 123
Claude: I'll connect to that catalog for you.
[Uses connect_catalog tool]
Connected to example.org, catalog 123. The domain schema is 'my_project'.
User: What datasets are available?
Claude: Let me check what datasets exist.
[Uses find_datasets tool]
Found 5 datasets:
1. Training Images (v1.2.0) - 1500 images for model training
2. Validation Set (v1.0.0) - 300 images for validation
...Configuration Workflow with GitHub
User: Create a new hydra-zen config for training on the cell-images dataset
Claude: I'll create a configuration file for you.
[Reads deriva://config/model-template resource for template]
[Uses GitHub create_or_update_file to save configs/cell_training.py]
Created configs/cell_training.py with:
- DerivaML connection to example.org
- Dataset spec for cell-images v2.0.0
- Model config with learning_rate=1e-3, epochs=50
User: Commit and push that config
Claude: [Uses GitHub create_or_update_file with commit message]
Committed and pushed to main branch.Hydra-zen Configuration
DerivaML integrates with hydra-zen for configuration management, enabling reproducible ML workflows with structured configuration.
Basic Configuration
from hydra_zen import builds, instantiate
from deriva_ml import DerivaML
from deriva_ml.core.config import DerivaMLConfig
# Create a structured config using hydra-zen
DerivaMLConf = builds(DerivaMLConfig, populate_full_signature=True)
# Configure for your environment
conf = DerivaMLConf(
hostname='deriva.example.org',
catalog_id='42',
domain_schema='my_domain',
)
# Instantiate to get a DerivaMLConfig object, then create DerivaML
config = instantiate(conf)
ml = DerivaML.instantiate(config)Working Directory Configuration
DerivaML automatically configures Hydra's output directory based on your working_dir setting:
conf = DerivaMLConf(
hostname='deriva.example.org',
working_dir='/shared/ml_workspace', # Custom working directory
)Hydra outputs will be organized under: {working_dir}/{username}/deriva-ml/hydra/{timestamp}/
Configuration Composition
Create environment-specific configurations using hydra-zen's store:
from hydra_zen import store
# Development configuration
store(DerivaMLConf(
hostname='dev.example.org',
catalog_id='1',
), name='dev')
# Production configuration
store(DerivaMLConf(
hostname='prod.example.org',
catalog_id='100',
), name='prod')Dataset Specification Configuration
Use DatasetSpecConfig for cleaner dataset specifications:
from deriva_ml.dataset import DatasetSpecConfig
# Create dataset specs (hydra-zen compatible)
training_data = DatasetSpecConfig(
rid="1ABC",
version="1.0.0",
materialize=True, # Download asset files
description="Training images"
)
metadata_only = DatasetSpecConfig(
rid="2DEF",
version="2.0.0",
materialize=False, # Only download table data
)
# Use in hydra-zen store
from hydra_zen import store
datasets_store = store(group="datasets")
datasets_store([training_data], name="training")
datasets_store([metadata_only], name="metadata_only")Asset Configuration
Use AssetRIDConfig for input assets (model weights, config files):
from deriva_ml.execution import AssetRIDConfig
# Define input assets
model_weights = AssetRIDConfig(rid="WXYZ", description="Pretrained model")
config_file = AssetRIDConfig(rid="ABCD", description="Hyperparameters")
# Store asset collections
assets_store = store(group="assets")
assets_store([model_weights, config_file], name="default_assets")Execution Configuration
Configure ML executions with ExecutionConfiguration:
from hydra_zen import builds, instantiate
from deriva_ml.execution import ExecutionConfiguration
from deriva_ml.dataset import DatasetSpecConfig
# Build execution config
ExecConf = builds(ExecutionConfiguration, populate_full_signature=True)
# Configure execution with datasets and assets
conf = ExecConf(
description="Training run",
datasets=[
DatasetSpecConfig(rid="1ABC", version="1.0.0", materialize=True),
],
assets=["WXYZ", "ABCD"], # Asset RIDs
)
exec_config = instantiate(conf)Configuration Summary
Class | Module | Purpose |
|
| Main DerivaML connection config |
|
| Dataset specification for executions |
|
| Input asset specification |
|
| Full execution configuration |
|
| Workflow definition |
See the DerivaML Hydra-zen Guide for complete documentation.
Troubleshooting
Docker with Localhost Deriva Server
When running the MCP server in Docker and connecting to a Deriva server on your local machine, you need additional configuration depending on how Deriva is running.
Option A: Deriva Running Directly on Host (not in Docker)
If your Deriva server is running directly on the host machine (not in Docker), use host-gateway:
{
"mcpServers": {
"deriva": {
"type": "stdio",
"command": "/bin/sh",
"args": [
"-c",
"docker run -i --rm --add-host localhost:host-gateway -e HOME=$HOME -v $HOME/.deriva:$HOME/.deriva:ro -v $HOME/.bdbag:$HOME/.bdbag -v $HOME/.deriva-ml:$HOME/.deriva-ml ghcr.io/informatics-isi-edu/deriva-mcp:latest"
],
"env": {}
}
}
}Option B: Deriva Running in Docker (deriva-localhost)
If your Deriva server is running in Docker (e.g., using deriva-localhost), the MCP container must join the same Docker network and map localhost to the webserver container.
First, find the webserver IP:
docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' deriva-webserverThen use it in the --add-host argument (replace <WEBSERVER_IP> with the actual IP):
{
"mcpServers": {
"deriva": {
"type": "stdio",
"command": "/bin/sh",
"args": [
"-c",
"docker run -i --rm --network deriva-localhost_internal_network --add-host localhost:<WEBSERVER_IP> -e HOME=$HOME -v $HOME/.deriva:$HOME/.deriva:ro -v $HOME/.bdbag:$HOME/.bdbag -v $HOME/.deriva-ml:$HOME/.deriva-ml ghcr.io/informatics-isi-edu/deriva-mcp:latest"
],
"env": {}
}
}
}Why this is needed: The MCP container needs to download dataset assets from the Deriva server. When Deriva runs in Docker, URLs in the dataset bags reference localhost, which must resolve to the Deriva webserver container. The entrypoint script in the MCP image automatically adjusts /etc/hosts so that the --add-host mapping takes effect.
SSL Certificate Configuration
If your localhost Deriva server uses a self-signed certificate (common for development), the container won't trust it by default. The image automatically sets REQUESTS_CA_BUNDLE to $HOME/.deriva/allCAbundle-with-local.pem, so you just need to create this file:
Creating the CA bundle with your local certificate (macOS):
# Export the local CA certificate from System Keychain
security find-certificate -a -c "DERIVA Dev Local CA" -p /Library/Keychains/System.keychain > /tmp/deriva-local-ca.pem
# Combine with existing CA bundle (if you have one)
cat ~/.deriva/allCAbundle.pem /tmp/deriva-local-ca.pem > ~/.deriva/allCAbundle-with-local.pem
# Or just use the local CA alone
cp /tmp/deriva-local-ca.pem ~/.deriva/allCAbundle-with-local.pemTo use a different CA bundle path, override with -e REQUESTS_CA_BUNDLE=/path/to/bundle.pem.
Deriva Authentication Issues
Error: "No credentials found"
# Re-authenticate with Deriva
python -c "from deriva_ml import DerivaML; DerivaML.globus_login('your-server.org')"Error: "Token expired"
# Force re-authentication
python -c "from deriva_ml import DerivaML; DerivaML.globus_login('your-server.org', force=True)"GitHub MCP Issues
Error: "Bad credentials"
Verify your PAT hasn't expired
Check the token has required permissions (Contents: Read/Write)
Ensure the token is correctly set in your config
Docker not found
Install Docker Desktop or use the npx method instead
On Linux, ensure your user is in the docker group
MCP Server Connection Issues
Server not responding
Check the server is installed:
which deriva-mcpTest manually:
deriva-mcp(should start without errors)Check Claude Desktop logs for errors
Multiple server conflicts
Ensure each server has a unique name in the config
Restart Claude Desktop after config changes
Long-Running Operations (Catalog Cloning)
Catalog cloning operations can take several minutes for large catalogs. The MCP server provides async tools (clone_catalog_async, get_task_status, list_tasks) to handle this.
Recommended: Use HTTP Transport for Long Operations
The HTTP transport mode solves connection timeout issues by running the server as a persistent service:
# Start the server with HTTP transport
docker-compose -f docker-compose.mcp.yaml up -d
# Or run directly
deriva-mcp --transport streamable-http --host 0.0.0.0 --port 8000Then configure your MCP client to use HTTP:
{
"mcpServers": {
"deriva": {
"type": "http",
"url": "http://localhost:8000/mcp"
}
}
}See HTTP Transport Mode for complete setup instructions.
Alternative: STDIO with Async Tools
If using STDIO transport, the async workflow helps manage timeouts:
Use
clone_catalog_async()to start the clone - returns a task_id immediatelyPeriodically check
get_task_status(task_id)for progressWhen status is "completed", the result contains the new catalog info
STDIO Connection timeout issues: If connections drop during clone operations with STDIO transport:
Check task status after reconnection - If the MCP connection drops, simply reconnect and call
list_tasks()to find your running tasks. Note: Task state is stored in memory, so if the server process restarts (not just reconnects), running tasks will be lost.Consider switching to HTTP transport - HTTP connections don't have the idle timeout issues that STDIO connections have with some MCP clients.
Development
Running Tests
uv run pytestCode Quality
uv run ruff check src/
uv run ruff format src/Requirements
Python 3.12+
MCP SDK 1.2.0+
DerivaML 0.1.0+
Docker (optional, for GitHub MCP local server)
License
Apache 2.0
Related Projects
DerivaML - Core library for ML workflows on Deriva
Deriva - Python SDK for Deriva scientific data management
GitHub MCP Server - Official GitHub MCP server
MCP Python SDK - Official Python SDK for Model Context Protocol
Available Tools
101 toolsadd_asset_typeA
Add a new asset type to the Asset_Type vocabulary.
Args: type_name: Name for the asset type. description: What this asset type represents.
Returns: JSON with status, name, description, rid.
Example: add_asset_type("Segmentation Mask", "Binary mask images for segmentation")
Inputs: type_name (required), description (required). Returns: result
add_asset_type_to_assetA
Add an asset type to a specific asset.
Associates an asset with a type from the Asset_Type vocabulary. An asset can have multiple types.
Args: asset_rid: RID of the asset to modify. type_name: Name of the asset type to add (must exist in Asset_Type vocab).
Returns: JSON with status, asset_rid, and updated types list.
Example: add_asset_type_to_asset("3JSE", "Training_Data")
Inputs: asset_rid (required), type_name (required). Returns: result
add_columnA
Add a new column to an existing table.
Args: table_name: Name of the table to modify. column_name: Name for the new column. column_type: Data type - one of "text", "int2", "int4", "int8", "float4", "float8", "boolean", "date", "timestamp", "timestamptz", "json", "jsonb", "markdown" (default: "text"). nullok: Whether NULL values are allowed (default: True). default: Default value for new rows (optional). comment: Description of the column (optional).
Returns: JSON with status, table_name, column_name, column_type.
Example: add_column("Subject", "Age", "int4", nullok=True, comment="Subject age in years")
Inputs: table_name (required), column_name (required), nullok, comment, +2 more. Returns: result
add_dataset_childA
Add a dataset as a nested child of another dataset.
Creates a parent-child relationship between datasets. Common pattern: a "Complete" parent dataset contains "Training" and "Testing" children that partition the same data.
Args: parent_rid: RID of the parent dataset. child_rid: RID of the child dataset to nest.
Returns: JSON with status, parent_rid, child_rid.
Example: add_dataset_child("1-ABC", "1-DEF") -> nests 1-DEF inside 1-ABC
Inputs: child_rid (required), parent_rid (required). Returns: result
add_dataset_element_typeA
Register a domain table as a dataset element type.
After registration, records from this table can be added to datasets using add_dataset_members(). Creates an association table to link records to datasets.
Args: table_name: Name of the domain table to register (e.g., "Subject", "Image").
Returns: JSON with status, table_name, association_table.
Example: add_dataset_element_type("Subject") -> enables Subject records as dataset elements
Inputs: table_name (required). Returns: result
add_dataset_membersA
Add records as dataset elements. Auto-increments minor version.
Records must be from tables registered as dataset element types. Use add_dataset_element_type() to register a table, or list_dataset_element_types() to see which tables are already registered.
Accepts members in two forms:
List of RIDs (member_rids): Each RID is auto-resolved to its table. Simpler but slower for large numbers.
Dict by table name (members_by_table): Maps table names to RID lists. Faster (skips RID resolution) and lets you add members of different types in one call. Recommended when you know the table names.
Exactly one of member_rids or members_by_table must be provided.
Args: dataset_rid: The RID of the dataset to add members to. member_rids: List of RIDs to add (e.g., ["2-ABC", "2-DEF"]). Auto-resolves each RID to its table. members_by_table: Dict mapping table names to RID lists (e.g., {"Subject": ["2-ABC"], "Observation": ["2-DEF", "2-GHI"]}). Faster than member_rids for large datasets. description: Optional description for the version increment that records why these members were added. Stored in the dataset history.
Returns: JSON with status, added_count, dataset_rid.
Example: add_dataset_members("1-ABC", member_rids=["2-DEF", "2-GHI"]) add_dataset_members("1-ABC", members_by_table={"Subject": ["2-DEF"], "Image": ["2-GHI"]})
Inputs: dataset_rid (required), description, member_rids, members_by_table. Returns: result
add_dataset_typeA
Add a type to a dataset.
Adds a Dataset_Type vocabulary term to this dataset. The type must exist in the Dataset_Type vocabulary.
Args: dataset_rid: RID of the dataset. dataset_type: Name of the type to add (must exist in Dataset_Type vocabulary).
Returns: JSON with status, dataset_rid, dataset_types list.
Example: add_dataset_type("1-ABC", "Training") -> adds "Training" type to dataset
Inputs: dataset_rid (required), dataset_type (required). Returns: result
add_feature_valueA
Add feature values to one or more domain objects.
Associates feature values (terms, assets, or other) with target records. Accepts a list of entries, each mapping a target RID to a value. All entries are inserted in a single batch for efficiency.
If an execution is active, it will be used for provenance. Otherwise, provide an execution_rid explicitly.
For simple features (single term or asset column):
Use this tool — each entry needs only target_rid and value.
For complex features (multiple columns per record):
Use add_feature_value_record instead, which accepts arbitrary field dicts.
Args: table_name: Table the target records belong to (e.g., "Image"). feature_name: Name of the feature (e.g., "Diagnosis"). entries: List of dicts, each with: - target_rid (str): RID of the target record to annotate. - value (str): The feature value — a term name or asset RID. execution_rid: Execution RID for provenance (uses active if not provided).
Returns: JSON with status, feature_name, count, execution_rid, rids.
Examples: # Single value add_feature_value("Image", "Diagnosis", [{"target_rid": "1-ABC", "value": "Normal"}])
# Batch values
add_feature_value("Image", "Diagnosis", [
{"target_rid": "1-ABC", "value": "Normal"},
{"target_rid": "1-DEF", "value": "Abnormal"},
{"target_rid": "1-GHI", "value": "Normal"},
])
Inputs: entries (required), table_name (required), feature_name (required), execution_rid. Returns: result
add_feature_value_recordA
Add feature values with multiple fields to one or more domain objects.
For features with multiple columns (e.g., a diagnosis with confidence score),
use this tool to provide values for each field. Accepts a list of entries
for batch insertion. Use lookup_feature first to see available fields.
Feature columns are dynamically generated based on the feature definition:
term_columns: Accept vocabulary term names (strings)asset_columns: Accept asset RIDs (strings)value_columns: Accept direct values (strings, numbers)
Args:
table_name: Table the target records belong to (e.g., "Image").
feature_name: Name of the feature (e.g., "Diagnosis").
entries: List of dicts, each with:
- target_rid (str, required): RID of the target record.
- Plus any feature column names mapped to their values.
Use lookup_feature to see available columns and types.
execution_rid: Execution RID for provenance (uses active if not provided).
Returns: JSON with status, feature_name, count, execution_rid, rids.
Example: # First check the feature structure: lookup_feature("Image", "Diagnosis") # -> {"term_columns": {"Diagnosis_Type": {...}}, "value_columns": {"confidence": {...}}}
# Then add values (single or batch):
add_feature_value_record("Image", "Diagnosis", [
{"target_rid": "1-ABC", "Diagnosis_Type": "Normal", "confidence": 0.95},
{"target_rid": "1-DEF", "Diagnosis_Type": "Abnormal", "confidence": 0.87},
])
Inputs: entries (required), table_name (required), feature_name (required), execution_rid. Returns: result
add_nested_executionA
Add a child execution to a parent execution.
Creates a parent-child relationship between executions. Use this to group related executions, such as:
Parameter sweeps (parent = sweep, children = individual runs)
Pipelines (parent = pipeline, children = stages)
Cross-validation (parent = CV experiment, children = folds)
Args: parent_execution_rid: RID of the parent execution. child_execution_rid: RID of the child execution to nest. sequence: Optional ordering index (0, 1, 2...). Use None for parallel executions.
Returns: JSON with parent_rid, child_rid, sequence.
Example: # Create a sweep parent, then add child executions add_nested_execution("1-PARENT", "1-CHILD1", sequence=0) add_nested_execution("1-PARENT", "1-CHILD2", sequence=1)
Inputs: child_execution_rid (required), parent_execution_rid (required), sequence. Returns: result
add_synonymA
Add a synonym to an existing vocabulary term.
Synonyms are alternative names that can be used to look up a term.
Args: vocabulary_name: Name of the vocabulary table (e.g., "Dataset_Type"). term_name: Primary name of the term to add synonym to. synonym: Alternative name to add.
Returns: JSON with status, name, synonyms list.
Example: add_synonym("Dataset_Type", "Training", "train") -> adds "train" as synonym
Inputs: synonym (required), term_name (required), vocabulary_name (required). Returns: result
add_termA
Add a new term to a vocabulary.
Args: vocabulary_name: Name of the vocabulary table (e.g., "Dataset_Type"). term_name: Primary name for the term (must be unique). description: What this term means. synonyms: Alternative names that can also match this term.
Returns: JSON with status, name, description, synonyms, rid.
Example: add_term("Dataset_Type", "Validation", "Held-out data for validation", ["val", "valid"])
Inputs: term_name (required), description (required), vocabulary_name (required), synonyms. Returns: result
add_visible_columnA
Add a column to the visible-columns list for a specific context.
This is a convenience tool for adding columns without replacing the entire visible-columns annotation. Changes are staged until apply_annotations() is called.
Args: table_name: Name of the table. context: The context to modify. See Contexts below. column: Column to add. Can be: - String: column name (e.g., "Filename") - List: foreign key reference (e.g., ["schema", "fkey_name"]) - Dict: pseudo-column definition (see set_visible_columns) position: Position to insert at (0-indexed). If None, appends to end.
Contexts for visible-columns:
Context | Description | When Used |
| Default for all contexts | Fallback when specific context not set |
| List/table view | Main record list, search results |
| Abbreviated list | Inline previews, tooltips |
| Minimal inline | Foreign key cell display |
| Selection modal | Picker dialogs for foreign keys |
| Full record view | Single record page |
| Data entry forms | Both create and edit forms |
| Create form only | New record creation |
| Edit form only | Editing existing records |
| Data export | CSV/JSON export |
| Faceted search | Search sidebar (uses different format) |
Returns: JSON with the updated column list for the context.
Examples: # Add column to end of compact view add_visible_column("Image", "compact", "Description")
# Add foreign key reference at position 1
add_visible_column("Image", "detailed", ["domain", "Image_Subject_fkey"], 1)
# Add pseudo-column
add_visible_column("Image", "compact", {
"source": [{"outbound": ["domain", "Image_Subject_fkey"]}, "Name"],
"markdown_name": "Subject"
})
Inputs: column (required), context (required), table_name (required), position. Returns: result
add_visible_foreign_keyA
Add a foreign key to the visible-foreign-keys list for a specific context.
This is a convenience tool for adding related tables without replacing the entire visible-foreign-keys annotation. Changes are staged until apply_annotations() is called.
Args: table_name: Name of the table. context: The context to modify (typically "detailed" or "*"). foreign_key: Foreign key to add. Can be: - List: inbound foreign key reference (e.g., ["schema", "Other_Table_fkey"]) - Dict: pseudo-column definition for complex relationships position: Position to insert at (0-indexed). If None, appends to end.
Contexts for visible-foreign-keys:
Context | Description | When Used |
| Default for all contexts | Fallback when specific context not set |
| Full record view | Related tables shown on single record page |
Important: Only INBOUND foreign keys are valid - these are foreign keys from OTHER tables that reference THIS table. Use list_foreign_keys() to see which inbound foreign keys are available.
Returns: JSON with the updated foreign key list for the context.
Examples: # Add inbound foreign key to detailed view add_visible_foreign_key("Subject", "detailed", ["domain", "Image_Subject_fkey"])
# Add at specific position
add_visible_foreign_key("Subject", "detailed", ["domain", "Diagnosis_Subject_fkey"], 0)
# Add pseudo-column for complex relationship
add_visible_foreign_key("Subject", "detailed", {
"source": [{"inbound": ["domain", "Image_Subject_fkey"]}],
"markdown_name": "Subject Images"
})
Inputs: context (required), table_name (required), foreign_key (required), position. Returns: result
add_workflow_typeA
Add a new workflow type to the Workflow_Type vocabulary.
Args: type_name: Name for the new workflow type. description: What this type of workflow does.
Returns: JSON with status, name, description, rid.
Example: add_workflow_type("Data Augmentation", "Workflows that augment training data")
Inputs: type_name (required), description (required). Returns: result
apply_annotationsA
Apply all staged annotation changes to the catalog.
This commits any annotation changes made via set_display_annotation, set_visible_columns, set_visible_foreign_keys, set_table_display, or set_column_display to the remote catalog.
Returns: JSON with status and details of the apply operation.
Example workflow: 1. get_table_annotations("Image") # Check current state 2. set_display_annotation("Image", annotation={"name": "Images"}) 3. set_visible_columns("Image", {"compact": ["RID", "Filename"]}) 4. apply_annotations() # Commit all changes
Inputs: no named parameters. Returns: result
apply_catalog_annotationsA
Apply catalog-level annotations to initialize the Chaise web interface.
Chaise is Deriva's web-based data browser. This method sets up annotations that control how Chaise displays and organizes the catalog, including the navigation bar and display settings.
Navigation Bar Structure: Creates a navigation bar with organized dropdown menus:
User Info: Users, Groups, and RID Lease tables
Deriva-ML: Core ML tables (Workflow, Execution, Dataset, Dataset_Version, etc.)
WWW: Web content tables (Page, File)
{Domain Schema}: All domain-specific tables (excludes vocabularies/associations)
Vocabulary: All controlled vocabulary tables from ML and domain schemas
Assets: All asset tables from ML and domain schemas
Catalog Registry: Link to ermrest registry
Documentation: Links to ML docs and instructions
Display Settings:
Underscores in names displayed as spaces
System columns (RID) shown in views
Default landing page set to Dataset table
Faceted search and record deletion enabled
Bulk Upload: Configures drag-and-drop file upload for asset tables.
When to call: After creating the domain schema and all tables. The menus are dynamically built from the current schema structure.
Args: navbar_brand_text: Text in the navigation bar brand area (default: "ML Data Browser"). head_title: Browser tab title (default: "Catalog ML").
Returns: JSON with status and applied settings.
Example workflow: 1. create_catalog("localhost", "my_project") 2. create_vocabulary("Species", "Types of species") 3. create_asset("Image", ...) 4. apply_catalog_annotations("My ML Project", "ML Catalog")
Inputs: head_title, navbar_brand_text. Returns: result
bag_infoA
Get comprehensive info about a dataset bag: size, contents, and cache status.
Combines the size estimate (row counts, asset sizes per table) with local cache status. Use this to decide whether to cache a bag before running an experiment.
Cache status values:
"not_cached": No local copy exists
"cached_metadata_only": Table data downloaded, assets not fetched
"cached_materialized": Fully downloaded and validated
"cached_incomplete": Was cached but some assets are missing
Args: dataset_rid: RID of the dataset to inspect. version: Semantic version to inspect (e.g., "1.0.0"). exclude_tables: Optional list of table names to exclude from FK path traversal.
Returns: JSON with size info (tables, total_rows, total_asset_bytes, total_asset_size) plus cache_status and cache_path.
Inputs: version (required), dataset_rid (required), exclude_tables. Returns: result
cache_datasetA
Download a dataset bag or asset into the local cache without creating an execution.
Use this to warm the cache before running experiments. No execution or provenance records are created — this is purely a local download operation. After caching, subsequent download_dataset or download_execution_dataset calls will use the cached copy.
Provide either dataset_rid (for bags) or asset_rid (for individual assets), not both.
Args: dataset_rid: RID of a dataset to cache (mutually exclusive with asset_rid). asset_rid: RID of an asset to cache (mutually exclusive with dataset_rid). version: Dataset version to cache (required when using dataset_rid). materialize: If True (default), download all asset files in the bag. If False, download only table metadata (faster, smaller). Ignored for asset cache. exclude_tables: Optional list of table names to exclude from FK path traversal during bag export. Only applies to dataset cache.
Returns: JSON with cache results. For datasets: bag_info including cache_status and size. For assets: file path and metadata.
Inputs: version, asset_rid, dataset_rid, materialize, +1 more. Returns: result
cancel_taskA
Cancel a pending or running background task.
Args: task_id: The task ID to cancel.
Returns: JSON with cancellation status.
Note: Cancellation is best-effort. Long-running operations may not stop immediately.
Inputs: task_id (required). Returns: result
citeA
Generate a citation URL for a catalog entity.
Creates a permanent, citable URL for any catalog entity (dataset, execution, asset, etc.). By default, includes a snapshot timestamp for reproducibility. Use current=True for a link to the live data.
Args: rid: RID of the entity to cite (e.g., "1-ABC"). current: If True, return URL to current state without snapshot. If False (default), return permanent URL with snapshot timestamp.
Returns: JSON with: - url: The citation URL - rid: The entity RID - is_snapshot: Whether URL includes snapshot timestamp
Examples: cite("1-ABC") -> {"url": "https://host/id/catalog/1-ABC@2024-01-15", "is_snapshot": true}
cite("1-ABC", current=True)
-> {"url": "https://host/id/catalog/1-ABC", "is_snapshot": false}
Inputs: rid (required), current. Returns: result
clone_catalogA
Create an ML workspace by cloning data reachable from a root RID.
Creates a partial catalog clone containing only data reachable from the root RID (e.g., a project, dataset, or experiment). Uses the root table's export annotation (if available) to determine which tables and paths to follow, then fills in any uncovered tables (vocabularies, associations).
Uses a three-stage approach:
Create schema WITHOUT foreign keys (only for included tables)
Copy data asynchronously (export paths + fill-in tables)
Apply foreign keys, handling violations based on orphan_strategy
Asset handling modes:
"none": Don't copy assets (asset columns will be empty)
"refs": Copy asset URLs only, files stay on source server (default)
"full": Download and re-upload all assets (fully independent clone)
Orphan handling: When source catalog policies hide some data but not references to it, cloning can result in dangling foreign keys. The orphan_strategy controls how these are handled.
Args: source_hostname: Source server hostname (e.g., "www.facebase.org"). source_catalog_id: ID of the catalog to clone. root_rid: The starting RID from which to trace reachability (e.g., a project RID like "3-HXMC"). dest_hostname: Destination hostname. If None, uses source hostname. alias: Optional alias name for the new catalog. add_ml_schema: If True, add the DerivaML schema to the clone. asset_mode: How to handle assets: "none", "refs" (default), or "full". copy_annotations: If True (default), copy all annotations. copy_policy: If True (default), copy ACL policies. exclude_schemas: List of schema names to exclude from cloning. exclude_objects: List of tables ("schema:table" format) to exclude. reinitialize_dataset_versions: If True (default), reinitialize dataset versions. orphan_strategy: How to handle orphan rows: "fail", "delete", or "nullify". prune_hidden_fkeys: If True, skip FKs with hidden reference data. truncate_oversized: If True, truncate values exceeding index size limits. include_tables: Additional tables to include. include_associations: If True, auto-include association tables. include_vocabularies: If True, auto-include vocabulary tables. table_concurrency: Max concurrent table copies during fill phase. Lower values reduce server load. Default: 1.
Returns: JSON with status, source info, destination info, and operation details including tables restored and orphan handling stats.
Examples: clone_catalog("www.facebase.org", "1", root_rid="3-HXMC", dest_hostname="localhost", alias="facebase-musmorph", add_ml_schema=True, orphan_strategy="delete")
Inputs: root_rid (required), source_hostname (required), source_catalog_id (required), alias, +15 more. Returns: result
clone_catalog_asyncA
Create an ML workspace by cloning data reachable from a root RID.
Starts the workspace creation in the background and immediately returns a task_id that you can use to check progress.
The operation uses export annotations (if available) to determine which tables and paths to follow from the root RID, then fills in any uncovered tables (vocabularies, associations). Uses async data copying for performance.
Use this for large catalogs or cross-server clones that may take several
minutes to complete. Check progress with get_task_status(task_id).
Args: source_hostname: Source server hostname (e.g., "www.facebase.org"). source_catalog_id: ID of the catalog to clone. root_rid: The starting RID from which to trace reachability (e.g., "3-HXMC"). dest_hostname: Destination hostname. If None, uses source hostname. alias: Optional alias name for the new catalog. add_ml_schema: If True, add the DerivaML schema to the clone. asset_mode: How to handle assets: "none", "refs" (default), or "full". copy_annotations: If True (default), copy all annotations. copy_policy: If True (default), copy ACL policies. exclude_schemas: Schemas to exclude from cloning. exclude_objects: Tables ("schema:table") to exclude. reinitialize_dataset_versions: If True, reinitialize dataset versions. orphan_strategy: How to handle orphans: "fail", "delete", "nullify". prune_hidden_fkeys: Skip FKs with hidden reference data. truncate_oversized: Truncate values exceeding index limits. include_tables: Additional tables to include. include_associations: Include association tables. include_vocabularies: Include vocabulary tables. table_concurrency: Max concurrent table copies during fill phase. Lower values reduce server load. Default: 2.
Returns: JSON with task_id and status. Use get_task_status(task_id) to check progress.
Example: clone_catalog_async("www.facebase.org", "1", root_rid="3-HXMC", dest_hostname="localhost", alias="facebase-clone", orphan_strategy="delete") -> {"task_id": "abc123", "status": "started", ...}
# Check progress
get_task_status("abc123")
-> {"status": "running", "progress": {"percent_complete": 45.0, ...}}
# When done
get_task_status("abc123")
-> {"status": "completed", "result": {...}}
Inputs: root_rid (required), source_hostname (required), source_catalog_id (required), alias, +15 more. Returns: result
connect_catalogA
Connect to an existing DerivaML catalog. Must be called before using other tools.
On connection, an MCP workflow and execution are automatically created to track all operations performed through the MCP server. The workflow type "DerivaML MCP" is created if it doesn't exist.
Args: hostname: Server hostname (e.g., "dev.eye-ai.org", "www.atlas-d2k.org"). catalog_id: Catalog ID number (e.g., "1", "52"). domain_schema: Schema name for domain tables. Auto-detected if omitted. default_schema: Default schema for table creation and lookups. If omitted and there is exactly one domain schema, that schema is used. Required when multiple domain schemas exist and you want to avoid specifying the schema on every operation.
Returns: JSON with status, hostname, catalog_id, domain_schemas, default_schema, project_name, workflow_rid, execution_rid.
Example: connect_catalog("dev.eye-ai.org", "52") -> connects to eye-ai catalog connect_catalog("localhost", "10", domain_schema="isa", default_schema="isa")
Inputs: hostname (required), catalog_id (required), domain_schema, default_schema. Returns: result
create_asset_tableA
Create a new asset table for file management with automatic URL/checksum tracking.
Asset tables automatically include: URL, Filename, Length, MD5, Description. They integrate with executions for provenance tracking.
Args: asset_name: Name for the asset table (e.g., "Image", "Model", "Checkpoint"). columns: Additional columns beyond standard asset columns. referenced_tables: Tables this asset should have foreign keys to. comment: Description of the asset table's purpose. schema: Schema to create the table in. If not provided, uses the default domain schema.
Returns: JSON with status, table_name, schema, columns.
Example: create_asset_table("Image", [{"name": "Width", "type": "int4"}], ["Subject"])
Inputs: asset_name (required), schema, columns, comment, +1 more. Returns: result
create_catalogA
Create a new DerivaML catalog with all ML schema tables.
Creates a fresh catalog with Dataset, Execution, Workflow, Feature, and vocabulary tables. Automatically connects to the new catalog.
Args: hostname: Server hostname (e.g., "localhost", "deriva.example.org"). project_name: Name for the project, becomes the domain schema name. catalog_alias: Optional alias for the catalog. If provided, creates an alias that allows accessing the catalog by name instead of numeric ID (e.g., /ermrest/catalog/my-project instead of /ermrest/catalog/45).
Returns: JSON with status, hostname, catalog_id, catalog_alias (if created), domain_schema, project_name.
Example: create_catalog("localhost", "my_ml_project", "my-project")
Inputs: hostname (required), project_name (required), catalog_alias. Returns: result
create_catalog_aliasA
Create an alias for an existing catalog.
Aliases allow accessing a catalog by a memorable name instead of its numeric ID. For example, instead of /ermrest/catalog/21, you can use /ermrest/catalog/eye-ai.
Args: hostname: Server hostname (e.g., "www.eye-ai.org"). alias_name: The alias identifier (e.g., "my-project", "eye-ai"). Must be unique on the server. catalog_id: The numeric ID of the catalog to alias. name: Optional display name for the alias. description: Optional description of the alias.
Returns: JSON with status and alias details.
Example: create_catalog_alias("localhost", "my-project", "45", "My ML Project") -> {"status": "created", "alias": "my-project", "target": "45"}
Inputs: hostname (required), alias_name (required), catalog_id (required), name, +1 more. Returns: result
create_datasetA
Create a new empty dataset within an execution context.
The dataset is created through an execution for proper provenance tracking. Use add_dataset_members() to populate it after creation.
Assign Dataset_Type labels to categorize the dataset's role (e.g., "Training", "Testing", "Validation").
Args: description: Human-readable description of the dataset's purpose. dataset_types: Type labels from Dataset_Type vocabulary (e.g., ["Training", "Image"]). version: Initial version string (default: "0.1.0").
Returns: JSON with status, rid, description, dataset_types, version, execution_rid.
Example: create_dataset("Training images for model v2", ["Training"])
Inputs: version, description, dataset_types. Returns: result
create_dataset_type_termA
Create a new dataset type term in the Dataset_Type vocabulary.
This creates a new vocabulary term that can then be assigned to datasets using add_dataset_type(). Dataset types help categorize datasets by their role in ML workflows.
Common types include "Training", "Testing", "Validation", "Complete".
Args: type_name: Name for the dataset type (must be unique). description: What this type of dataset is used for. synonyms: Alternative names that can match this type (e.g., ["train"] for "Training").
Returns: JSON with status, name, description, synonyms, rid.
Example: create_dataset_type_term("Validation", "Held-out data for hyperparameter tuning", ["val", "valid"])
Inputs: type_name (required), description (required), synonyms. Returns: result
create_executionA
Create a new execution to track an ML workflow run with provenance.
This is the first step in the execution lifecycle. Specify input datasets and assets to establish provenance - these will be recorded as inputs to this workflow run.
LIFECYCLE (follow in order):
create_execution() - You are here
start_execution() - Begin timing
[Run your ML workflow]
stop_execution() - End timing
Args: workflow_name: Descriptive name (e.g., "ResNet50 Training Run 3"). workflow_type: Type from Workflow_Type vocabulary (e.g., "Training", "Inference"). description: What this execution does and why. dataset_rids: Input dataset RIDs for provenance tracking. asset_rids: Input asset RIDs for provenance tracking. dry_run: If True, download input datasets/assets but skip creating execution records in the catalog and skip uploading results. Useful for testing data loading, configuration, and model initialization without writing to the catalog.
Returns: JSON with execution_rid, workflow_rid, dataset_count, asset_count, dry_run.
Example: create_execution("CIFAR Training", "Training", "Train ResNet on CIFAR-10", ["1-ABC"]) create_execution("Test Run", "Training", "Debug data loading", dry_run=True)
Inputs: workflow_name (required), workflow_type (required), dry_run, asset_rids, +2 more. Returns: result
create_execution_datasetA
Create a new dataset as output from this execution.
Creates a dataset that is linked to this execution for provenance. Use this when your workflow produces a new curated collection of data (e.g., augmented training data, filtered results).
Args: description: What this dataset contains. dataset_types: Type labels (e.g., ["Training", "Augmented"]).
Returns: JSON with dataset_rid, execution_rid.
Inputs: description, dataset_types. Returns: result
create_featureA
Create a new feature definition to associate metadata with domain objects.
Features enable ML data engineering by linking labels, scores, or derived assets to domain objects. The feature definition specifies what types of values are valid.
What this creates:
A new association table in the domain schema to store feature values
A dynamically generated Pydantic model class for creating validated feature instances
The Pydantic model class (accessible via feature_record_class() in Python) provides
type-safe construction of feature records with automatic validation against the
feature's definition.
Feature types:
Term-based: Values come from controlled vocabularies (e.g., diagnosis labels)
Asset-based: Values reference asset files (e.g., segmentation masks)
Mixed: Can reference both terms and assets
The feature automatically tracks which Execution produced each value for provenance.
Args: table_name: Table to attach the feature to (e.g., "Image", "Subject"). feature_name: Unique name for the feature (e.g., "Diagnosis", "Quality_Score"). comment: Description of what this feature represents. terms: Vocabulary table names whose terms can be values (e.g., ["Diagnosis_Type"]). assets: Asset table names that can be referenced (e.g., ["Segmentation_Mask"]). metadata: Additional columns or table references to include in the feature. Each item can be: - A string: Treated as a table name (adds a foreign key reference) - A dict: Column definition with at minimum "name" and "type" keys. The "type" value should be a dict like {"typename": "float4"}. Valid type names: text, int2, int4, int8, float4, float8, boolean, date, timestamp, timestamptz, json, jsonb. Optional keys: "nullok" (bool), "default", "comment".
Returns: JSON with status, feature_name, target_table.
Examples: # Simple term-based feature create_feature("Image", "Diagnosis", "Clinical diagnosis label", terms=["Diagnosis_Type"])
# Feature with a confidence score column
create_feature("Image", "Diagnosis", "Diagnosis with confidence",
terms=["Diagnosis_Type"],
metadata=[{"name": "confidence", "type": {"typename": "float4"}}])
# Feature referencing another table
create_feature("Image", "Review", "Review annotations",
terms=["Review_Status"],
metadata=["Reviewer"])
Inputs: table_name (required), feature_name (required), terms, assets, +2 more. Returns: result
create_tableA
Create a new table in the domain schema.
This tool creates a standard table (not an asset table). For tables that store files with automatic URL/checksum tracking, use create_asset_table instead.
Process Overview:
Define columns with names, types, and constraints
Optionally define foreign keys to reference other tables
The table is created in the domain schema
The navigation bar is automatically updated
Args: table_name: Name for the new table (e.g., "Subject", "Experiment", "Protocol"). columns: Column definitions, each dict with: - name (str, required): Column name - type (str): One of "text", "int2", "int4", "int8", "float4", "float8", "boolean", "date", "timestamp", "timestamptz", "json", "jsonb", "markdown" (default: "text") - nullok (bool): Allow null values (default: True) - comment (str): Column description foreign_keys: Foreign key definitions, each dict with: - column (str, required): Column name in this table (must also be in columns list) - referenced_table (str, required): Name of the table to reference - referenced_column (str): Column in referenced table (default: "RID") - on_delete (str): Action on delete - "NO ACTION", "CASCADE", "SET NULL" (default: "NO ACTION") comment: Description of the table's purpose. schema: Schema to create the table in. If not provided, uses the default domain schema. Useful when a catalog has multiple domain schemas.
Returns: JSON with status, table_name, schema, columns.
Examples: Simple table: create_table("Subject", [ {"name": "Name", "type": "text", "nullok": false}, {"name": "Age", "type": "int4"}, {"name": "Notes", "type": "markdown"} ])
Table with foreign key:
create_table("Sample", [
{"name": "Name", "type": "text", "nullok": false},
{"name": "Subject", "type": "text", "nullok": false},
{"name": "Collection_Date", "type": "date"}
], foreign_keys=[
{"column": "Subject", "referenced_table": "Subject", "on_delete": "CASCADE"}
])
Inputs: table_name (required), schema, columns, comment, +1 more. Returns: result
create_vocabularyA
Create a new vocabulary table for storing controlled terms.
Args: vocabulary_name: Name for the new vocabulary table. comment: Description of the vocabulary's purpose. schema: Schema to create in (default: domain schema).
Returns: JSON with status, name, schema, comment.
Example: create_vocabulary("Quality_Level", "Image quality ratings")
Inputs: vocabulary_name (required), schema, comment. Returns: result
create_workflowA
Create and register a new workflow definition.
Args: name: Display name for the workflow. workflow_type: Type from Workflow_Type vocabulary (e.g., "Training", "Inference"). description: What this workflow does.
Returns: JSON with status, rid, name, workflow_type, description.
Example: create_workflow("ResNet Training", "Training", "Trains ResNet50 on image data")
Inputs: name (required), workflow_type (required), description. Returns: result
delete_catalogA
PERMANENTLY DELETE a catalog and all its data. Cannot be undone.
Args: hostname: Server hostname. catalog_id: ID of the catalog to delete.
Returns: JSON with deletion status.
Inputs: hostname (required), catalog_id (required). Returns: result
delete_catalog_aliasA
Delete a catalog alias. The target catalog is NOT deleted.
Args: hostname: Server hostname (e.g., "www.eye-ai.org"). alias_name: The alias identifier to delete.
Returns: JSON with deletion status.
Example: delete_catalog_alias("localhost", "my-project") -> {"status": "deleted", "alias": "my-project"}
Inputs: hostname (required), alias_name (required). Returns: result
delete_datasetA
Soft-delete a dataset (marks deleted but preserves data).
Soft deletion hides the dataset from normal queries but keeps all data intact. For nested datasets, use recurse=True to also delete child datasets.
Args: dataset_rid: The RID of the dataset to delete. recurse: If True, also delete all nested child datasets.
Returns: JSON with status, dataset_rid, recursive.
Inputs: dataset_rid (required), recurse. Returns: result
delete_dataset_membersA
Remove records from a dataset. Auto-increments minor version.
Removes the specified records from the dataset's membership. The records themselves are not deleted from the catalog, only their association with this dataset is removed.
Removing members automatically increments the dataset's minor version for change tracking.
Args: dataset_rid: The RID of the dataset to remove members from. member_rids: List of RIDs to remove (e.g., ["2-ABC", "2-DEF", "2-GHI"]).
Returns: JSON with status, removed_count, dataset_rid.
Example: delete_dataset_members("1-ABC", ["2-DEF", "2-GHI"]) -> removes 2 records from dataset
Inputs: dataset_rid (required), member_rids (required). Returns: result
delete_dataset_type_termA
Delete a dataset type term from the Dataset_Type vocabulary.
WARNING: Only delete types that are not referenced by any datasets. If datasets use this type, the delete will fail with a foreign key error. Use remove_dataset_type() first to remove the type from all datasets.
Args: type_name: Name of the dataset type to delete.
Returns: JSON with status and deleted type name.
Example: delete_dataset_type_term("Obsolete") -> {"status": "deleted", "name": "Obsolete"}
Inputs: type_name (required). Returns: result
delete_featureA
Delete a feature definition and all its values. Cannot be undone.
WARNING: This permanently removes the feature table and all associated values. All provenance information for this feature will be lost.
Args: table_name: Table the feature is attached to. feature_name: Name of the feature to delete.
Returns: JSON with status, feature_name, table_name.
Inputs: table_name (required), feature_name (required). Returns: result
delete_termA
Delete a term from a vocabulary.
The term must not be in use by any records in the catalog. If the term is referenced by other records (e.g., datasets using this type), the delete will fail with an error listing how many records reference it.
Args: vocabulary_name: Name of the vocabulary table (e.g., "Dataset_Type"). term_name: Name of the term to delete.
Returns: JSON with status, vocabulary, and deleted term name.
Example: delete_term("Dataset_Type", "Obsolete") -> {"status": "deleted", ...}
Inputs: term_name (required), vocabulary_name (required). Returns: result
disconnect_catalogB
Disconnect from the currently active catalog.
Inputs: no named parameters. Returns: result
estimate_bag_sizeA
Estimate the size of a dataset bag before downloading.
Runs the same FK path traversal as a dataset bag download, then queries the snapshot catalog for row counts and asset file sizes. Use this to preview what a download will contain and how large it will be before committing to the full download.
Args: dataset_rid: RID of the dataset to estimate. version: Semantic version to estimate (e.g., "1.0.0"). exclude_tables: Optional list of table names to exclude from FK path traversal during bag export.
Returns: JSON with: - tables: dict of table name -> {row_count, is_asset, asset_bytes} - total_rows: total row count across all tables - total_asset_bytes: total asset size in bytes - total_asset_size: human-readable size (e.g., "1.2 GB")
Inputs: version (required), dataset_rid (required), exclude_tables. Returns: result
get_dataset_specA
Generate a DatasetSpecConfig string for use in Python configuration files.
Returns the exact Python code to use in hydra-zen config files. This ensures the RID and version are correctly formatted and match what's in the catalog.
IMPORTANT: Always prefer specifying explicit versions in configurations. Using current_version as a default can lead to unexpected changes in results if the dataset is modified after the configuration is written. Pin to a specific version for reproducibility.
Args: dataset_rid: The RID of the dataset (e.g., "28CT"). version: Specific version to use. If not provided, uses the dataset's current version (with a warning about reproducibility).
Returns: JSON with the Python code string and metadata including: - spec: The DatasetSpecConfig(...) string ready to paste into code - rid: The dataset RID - version: The version used - description: Dataset description for reference - warning: Present if using current_version (recommends explicit version)
Example: get_dataset_spec("28CT") -> {"spec": "DatasetSpecConfig(rid="28CT", version="0.21.0")", ...}
get_dataset_spec("28CT", "0.20.0")
-> {"spec": "DatasetSpecConfig(rid=\"28CT\", version=\"0.20.0\")", ...}
Inputs: dataset_rid (required), version. Returns: result
get_handlebars_template_variablesA
Get all available template variables for a table.
Returns the columns, foreign keys, and special variables that can be used in Handlebars templates (row_markdown_pattern, markdown_pattern, etc.) for the specified table.
Args: table_name: Name of the table to get variables for.
Returns: JSON with columns, foreign_keys, and special variables available for use in templates.
Example: get_handlebars_template_variables("Image") -> { "table": "Image", "columns": [ {"name": "RID", "type": "ermrest_rid", "template": "{{{RID}}}"}, {"name": "Filename", "type": "text", "template": "{{{Filename}}}"}, ... ], "foreign_keys": [ { "constraint": ["domain", "Image_Subject_fkey"], "to_table": "Subject", "values_template": "{{{$fkeys.domain.Image_Subject_fkey.values.column}}}", "row_name_template": "{{{$fkeys.domain.Image_Subject_fkey.rowName}}}" } ], "special_variables": {...} }
Inputs: table_name (required). Returns: result
get_recordA
Get a single record by its RID.
Args: table_name: Name of the table containing the record. rid: The RID of the record to fetch.
Returns: JSON with the complete record or error if not found.
Example: get_record("Image", "1-ABC") -> full image record
Inputs: rid (required), table_name (required). Returns: result
get_table_sample_dataA
Get sample row data from a table for template testing.
Retrieves a few sample rows from the table that can be used to test Handlebars templates. Use this to see real values that would be available in templates.
Args: table_name: Name of the table. limit: Number of sample rows to return (default: 3, max: 10).
Returns: JSON with sample rows and their column values.
Example: get_table_sample_data("Image", 2) -> { "table": "Image", "sample_rows": [ {"RID": "1-ABC", "Filename": "scan001.jpg", "Subject": "2-DEF", ...}, {"RID": "1-XYZ", "Filename": "scan002.jpg", "Subject": "2-DEF", ...} ], "template_test_suggestion": "Try: {{{Filename}}} - Subject: {{{Subject}}}" }
Inputs: table_name (required), limit. Returns: result
get_task_statusA
Get the status and progress of a background task.
Args: task_id: The task ID returned by an async operation. include_result: If True, include the full result when completed.
Returns: JSON with task status, progress, and optionally the result.
Example: get_task_status("abc123") -> { "task_id": "abc123", "status": "running", "progress": { "current_step": "Copying data", "percent_complete": 45.0, "message": "Copying table Subject..." } }
Inputs: task_id (required), include_result. Returns: result
increment_dataset_versionA
Manually increment a dataset's semantic version (major.minor.patch).
Description Handling (follows generate-descriptions prompt guidelines):
If user provides description: Use it, potentially improving for clarity
If description is empty: Generate from conversation context:
What catalog operations were performed since last version?
What was the user's stated goal for these changes?
Summarize the changes made (e.g., "Added X, fixed Y, modified Z")
Description Generation Guidelines:
Include WHAT changed (added images, fixed labels, new features)
Include WHY if known (QA review, batch import, schema update)
Include IMPACT if relevant (affects N records, breaking change)
Use markdown for complex descriptions (lists, tables)
Use this tool when:
You've modified catalog data and want changes visible in a dataset
You need to capture a snapshot of the current catalog state
You want to create a reproducible checkpoint before making changes
Args: dataset_rid: The RID of the dataset. description: What changed in this version. If empty, LLM should generate from context. Good descriptions include: - What was added, modified, or fixed - Why the change was made (if known) - Impact on users of this dataset
Examples of good descriptions:
- "Added 500 new labeled training images from batch 3"
- "Fixed incorrect labels on 23 images identified in QA review"
- "Schema change: added 'quality_score' column to Image table"
- "Captured snapshot before label correction workflow"
component: Which part to increment: "major", "minor", or "patch".
- major: Breaking changes or schema modifications
- minor: New data added or non-breaking changes (default)
- patch: Bug fixes or label correctionsReturns: JSON with status, new_version, previous_version, dataset_rid, description.
Examples: increment_dataset_version("1-ABC", "Added quality labels to all images", "minor") increment_dataset_version("1-ABC", "Fixed mislabeled cat images", "patch") increment_dataset_version("1-ABC", "Schema change: new metadata columns", "major")
Inputs: dataset_rid (required), component, description. Returns: result
insert_recordsA
Insert new records into a domain table.
IMPORTANT: This tool is for domain-specific tables only (e.g., Subject, Image metadata). Do NOT use for:
Datasets → use create_dataset(), add_dataset_members()
Features → use add_feature_value()
Vocabularies → use add_term()
Executions → use create_execution()
Workflows → use create_workflow()
Assets with files → use the DerivaML Python API execution workflow
Args: table_name: Name of the domain table to insert into. records: List of dictionaries with column values.
Returns: JSON with inserted_count and record RIDs.
Example: insert_records("Subject", [{"Name": "Patient A", "Age": 45}])
Inputs: records (required), table_name (required). Returns: result
invalidate_cacheA
Invalidate cached tabular query results.
Args: cache_key: Invalidate a specific cached result. source: Invalidate all results from this source ("bag" or "catalog"). If neither is provided, invalidates all cached results.
Returns: JSON with the number of entries invalidated.
Inputs: source, cache_key. Returns: result
list_asset_executionsA
List all executions associated with an asset.
Given an asset RID, returns a list of executions that created or used the asset, along with the role (Input/Output) in each execution. This is useful for provenance tracking - finding which execution created an asset or which executions used it as input.
Args: asset_rid: RID of the asset to look up. asset_role: Optional filter: "Input" or "Output". If omitted, returns all.
Returns: JSON array of execution records showing which executions are associated with this asset. Each record includes execution_rid, workflow_rid, status, and description.
Example: list_asset_executions("3JSE") -> finds all executions that created/used this asset list_asset_executions("3JSE", "Output") -> finds only the execution that created it
Inputs: asset_rid (required), asset_role. Returns: result
list_cached_resultsA
List all cached tabular query results.
Returns metadata for each cached result including the tool that produced it, parameters, row count, age, and cache key. Use the cache_key with query_cached_result to re-query with different sort/filter/pagination.
Returns: JSON with list of cached result entries.
Inputs: no named parameters. Returns: result
list_catalog_registryA
List all catalogs and aliases available on a Deriva server.
Queries the server's ermrest registry to discover available catalogs and their aliases. Use this to find catalogs before connecting.
Note: Returns all non-deleted catalogs and aliases. Typically bounded (servers have 10-50 catalogs), but could be larger on shared servers.
Args: hostname: Server hostname (e.g., "www.eye-ai.org", "dev.facebase.org").
Returns: JSON with: - hostname: The server queried - catalogs: List of {id, name, description} for each catalog - aliases: List of {id, alias_target, name, description} for each alias
Example: list_catalog_registry("www.eye-ai.org") -> {"hostname": "www.eye-ai.org", "catalogs": [...], "aliases": [...]}
Inputs: hostname (required). Returns: result
list_dataset_parentsA
List all parent datasets that contain this dataset as a child.
Args: dataset_rid: RID of the child dataset. recurse: If True, recursively list all ancestors (parents of parents). version: Semantic version to query (e.g., "1.0.0"). If not specified, uses the current version.
Returns: JSON array of parent datasets with {rid, description, dataset_types, current_version}.
Example: list_dataset_parents("1-ABC") -> direct parents only list_dataset_parents("1-ABC", recurse=True) -> all ancestors
Inputs: dataset_rid (required), recurse, version. Returns: result
list_nested_executionsA
List all child (nested) executions of an execution.
Args: execution_rid: RID of the parent execution. recurse: If True, return all descendants (children, grandchildren, etc.).
Returns: JSON array of {execution_rid, workflow_rid, status, description} for each child.
Example: list_nested_executions("1-PARENT") # Direct children only list_nested_executions("1-PARENT", recurse=True) # All descendants
Inputs: execution_rid (required), recurse. Returns: result
list_tasksA
List all background tasks for the current user.
Args: status: Filter by status: "pending", "running", "completed", "failed", "cancelled". task_type: Filter by type: "clone_catalog".
Returns: JSON list of tasks with their status and basic info.
Example: list_tasks(status="running") -> [{"task_id": "abc123", "status": "running", ...}]
Inputs: status, task_type. Returns: result
lookup_workflow_by_urlA
Find a workflow by its source URL.
Search for a workflow that was registered with the given source URL. Use this to check if a workflow for a specific script or notebook already exists before creating a new one.
Args: url: The source URL to search for (e.g., GitHub URL to script).
Returns: JSON with: - found: True if workflow exists, False otherwise - workflow: Full workflow details if found
Example: lookup_workflow_by_url("https://github.com/org/repo/blob/main/train.py")
Inputs: url (required). Returns: result
preview_denormalized_datasetA
Preview a denormalized (wide table) view of dataset tables.
Joins related dataset tables into a single wide table. Returns schema shape (columns, join path) and size estimates. Optionally returns actual row data when a dataset and limit are provided.
Modes:
No dataset_rid: Returns schema shape + global size estimates. Use this to explore what a denormalized join would look like.
With dataset_rid, limit=0: Returns shape + dataset-scoped estimates.
With dataset_rid, limit>0: Returns shape + estimates + row preview.
Tables are joined based on their foreign key relationships. Column names are prefixed with the source table name using dots (e.g., "Image.Filename", "Subject.RID"). Intermediate tables needed for the join are auto-discovered.
Args: include_tables: List of table names to include in the join. Tables are joined based on their foreign key relationships. Order doesn't matter - the join order is determined automatically. Add more tables iteratively to expand the denormalized view. dataset_rid: RID of the dataset to preview. If omitted, returns schema shape with global (catalog-wide) row counts. version: Semantic version to query (e.g., "1.0.0"). If not specified, uses the current version. Only used with dataset_rid. limit: Maximum rows to return (default: 0, max: 100). Only used with dataset_rid. Set to 0 for shape and estimates only.
Returns: JSON with columns, join_path, tables (per-table size info), total_rows, total_asset_bytes, total_asset_size. When limit > 0 with a dataset_rid, also includes rows and count.
Example: # Explore schema shape (no dataset needed) preview_denormalized_dataset(["Subject", "Report_HVF"]) -> {"columns": [...], "join_path": ["Report_HVF", "Observation", "Subject"], ...}
# Get dataset-scoped estimates
preview_denormalized_dataset(["Image", "Subject"], dataset_rid="1-ABC")
-> {"columns": [...], "tables": {"Image": {"row_count": 50}}, ...}
# Preview actual rows
preview_denormalized_dataset(["Image", "Subject"], dataset_rid="1-ABC", limit=10)
-> {"columns": [...], "tables": {...}, "rows": [...], "count": 10}
Inputs: include_tables (required), limit, version, dataset_rid. Returns: result
preview_tableA
Preview records from a table with optional column selection and filtering.
Returns a sample of records for understanding data structure and content. For bulk data access, use the DerivaML Python API directly.
Args: table_name: Name of the table to preview (e.g., "Image", "Subject", "Dataset"). columns: List of column names to return. Default: all columns. filters: Dictionary of {column: value} equality filters. limit: Maximum records to return (default: 25, max: 100). offset: Number of records to skip.
Returns: JSON with records array, count, and table name.
Examples: preview_table("Image") -> first 25 images preview_table("Image", columns=["RID", "Filename"], limit=10) preview_table("Subject", filters={"Species": "Human"})
Inputs: table_name (required), limit, offset, columns, +1 more. Returns: result
query_cached_resultA
Re-query a cached tabular result with different sort/filter/pagination.
Use list_cached_results to find available cache keys. This tool lets you paginate, sort, and filter previously computed results without re-executing the original query.
Args: cache_key: The cache key from a previous query result. sort_by: Column name to sort by (e.g., "Image.CDR"). sort_desc: Sort descending if True. filter_col: Column name to filter on. filter_val: Value to filter for (substring match, case-insensitive). limit: Maximum rows to return (default: 100). offset: Number of rows to skip for pagination.
Returns: JSON with columns, rows, count, and total_count.
Inputs: cache_key (required), limit, offset, sort_by, +3 more. Returns: result
rag_add_sourceA
Register a new documentation source for RAG indexing.
After adding a source, run rag_ingest(source_name=name) to index it.
Args: name: Unique name for this source (e.g., "my-project-docs") repo_owner: GitHub repository owner (e.g., "informatics-isi-edu") repo_name: GitHub repository name (e.g., "deriva-ml") branch: Git branch to index (default "main") path_prefix: Only index files under this path (default "docs/") include_patterns: File patterns to include (default ["*.md"]) doc_type: Document type tag for filtering (default "user-guide")
Returns: Dict confirming the source was added.
Inputs: name (required), repo_name (required), repo_owner (required), branch, +3 more.
rag_index_schemaA
Re-index the connected catalog's schema for RAG search.
Fetches the current schema from the connected catalog and indexes it for semantic search. This happens automatically on connect_catalog, but can be called manually after schema changes (e.g., after creating tables, adding columns, or creating features).
Uses schema hashing — returns immediately if unchanged.
Returns: Dict with indexing statistics (status, chunks_created, schema_hash).
Inputs: no named parameters.
rag_ingestA
Full crawl and index of documentation sources.
Crawls GitHub repositories, fetches all documentation files, chunks them, and indexes them for semantic search. This is a long-running operation that runs in the background.
Args: source_name: Specific source to ingest (e.g., "deriva-ml-docs"). If None, ingests all configured sources.
Returns: Dict with task ID for tracking progress, or immediate results if the operation completes quickly.
Inputs: source_name.
rag_remove_sourceA
Remove a documentation source and its indexed chunks.
This deletes all indexed chunks for the source and removes it from the configuration.
Args: name: Name of the source to remove (e.g., "my-project-docs")
Returns: Dict with removal status and number of chunks deleted.
Inputs: name (required).
rag_searchA
Search Deriva documentation, catalog schema, and catalog data using semantic similarity.
Searches across three categories of indexed content:
Catalog schema (
doc_type="catalog-schema"): The connected catalog's structure, including:Tables with columns, types, nullability, and comments
Foreign key relationships between tables
Feature definitions with term columns, asset columns, and value columns
Vocabulary terms with names, descriptions, and synonyms Use this to understand what data exists, how tables relate, what features are defined, and what vocabulary values are available.
Catalog data (
doc_type="catalog-data"): User-specific records from the connected catalog, including:Datasets with types, versions, and descriptions
Executions with workflow names, status, and inputs Use this to find specific datasets by purpose or trace experiment provenance.
Documentation (
doc_type="user-guide","api-reference","sdk-reference"): Indexed docs from Deriva ecosystem repositories (deriva-ml, ermrest, chaise, deriva-py). Use this for API usage questions and how-to guidance.
Prefer this tool over reading raw resources for catalog exploration.
Use doc_type to focus results on the category you need.
Args:
query: Natural language search query (e.g., "how to create a dataset")
limit: Maximum number of results to return (default 10)
source: Filter by source name (e.g., "deriva-ml-docs", "ermrest-docs")
doc_type: Filter by document type. Key values:
- "catalog-schema": Tables, columns, FKs, features, vocab terms
- "catalog-data": Datasets and executions in the catalog
- "user-guide": DerivaML and Chaise documentation
- "api-reference": ERMrest API documentation
- "sdk-reference": deriva-py SDK documentation
include_schema: If True (default), include catalog schema results when
connected. Set to False to search only documentation.
include_data: If True (default), include per-user data index results
(datasets, executions) when connected. Set to False to
exclude user data from search results.
Returns: Dict with search results including text snippets, relevance scores, source metadata, and GitHub URLs.
Examples: # Explore catalog structure — tables, columns, relationships rag_search("Image tables and features", doc_type="catalog-schema")
# Find vocabulary terms by meaning
rag_search("classification categories", doc_type="catalog-schema")
# Find feature definitions and their columns
rag_search("diagnosis label confidence", doc_type="catalog-schema")
# Find datasets by description, type, or purpose
rag_search("training split labeled", doc_type="catalog-data")
# Find executions by workflow or status
rag_search("training experiment results", doc_type="catalog-data")
# Search Deriva API documentation
rag_search("how to create a dataset", include_schema=False, include_data=False)
# Search everything (docs + schema + data) — the default
rag_search("how are images classified")
Inputs: query (required), limit, source, doc_type, +2 more.
rag_statusA
Get the status of the RAG documentation index.
Returns information about the index including total chunks, configured sources, and last update times.
Returns: Dict with index status, source configurations, and statistics.
Inputs: no named parameters.
rag_updateA
Incremental update of documentation index.
Checks for changed files in source repositories and only re-indexes files that have been added, modified, or deleted. Much faster than full ingestion when few files have changed.
Args: source_name: Specific source to update (e.g., "deriva-ml-docs"). If None, updates all configured sources.
Returns: Dict with task ID for tracking progress.
Inputs: source_name.
remove_asset_type_from_assetA
Remove an asset type from a specific asset.
Removes the association between an asset and a type.
Args: asset_rid: RID of the asset to modify. type_name: Name of the asset type to remove.
Returns: JSON with status, asset_rid, and updated types list.
Example: remove_asset_type_from_asset("3JSE", "Training_Data")
Inputs: asset_rid (required), type_name (required). Returns: result
remove_dataset_typeA
Remove a type from a dataset.
Removes a Dataset_Type vocabulary term from this dataset. The type must exist in the Dataset_Type vocabulary.
Args: dataset_rid: RID of the dataset. dataset_type: Name of the type to remove.
Returns: JSON with status, dataset_rid, dataset_types list.
Example: remove_dataset_type("1-ABC", "Training") -> removes "Training" type from dataset
Inputs: dataset_rid (required), dataset_type (required). Returns: result
remove_synonymA
Remove a synonym from an existing vocabulary term.
Args: vocabulary_name: Name of the vocabulary table (e.g., "Dataset_Type"). term_name: Primary name of the term to remove synonym from. synonym: Alternative name to remove.
Returns: JSON with status, name, updated synonyms list.
Example: remove_synonym("Dataset_Type", "Training", "train") -> removes "train" as synonym
Inputs: synonym (required), term_name (required), vocabulary_name (required). Returns: result
remove_visible_columnA
Remove a column from the visible-columns list for a specific context.
This is a convenience tool for removing columns without replacing the entire visible-columns annotation. Changes are staged until apply_annotations() is called.
Args: table_name: Name of the table. context: The context to modify (e.g., "compact", "detailed"). column: Column to remove. Can be: - String: column name to find and remove - List: foreign key reference [schema, constraint] to find and remove - Integer: index position to remove (0-indexed)
Returns: JSON with the updated column list for the context.
Examples: # Remove by column name remove_visible_column("Image", "compact", "Description")
# Remove by foreign key reference
remove_visible_column("Image", "detailed", ["domain", "Image_Subject_fkey"])
# Remove by position (first column)
remove_visible_column("Image", "compact", 0)
Inputs: column (required), context (required), table_name (required). Returns: result
remove_visible_foreign_keyA
Remove a foreign key from the visible-foreign-keys list for a specific context.
This is a convenience tool for removing related tables without replacing the entire visible-foreign-keys annotation. Changes are staged until apply_annotations() is called.
Args: table_name: Name of the table. context: The context to modify (e.g., "detailed", "*"). foreign_key: Foreign key to remove. Can be: - List: foreign key reference [schema, constraint] to find and remove - Integer: index position to remove (0-indexed)
Returns: JSON with the updated foreign key list for the context.
Examples: # Remove by foreign key reference remove_visible_foreign_key("Subject", "detailed", ["domain", "Image_Subject_fkey"])
# Remove by position (first foreign key)
remove_visible_foreign_key("Subject", "detailed", 0)
Inputs: context (required), table_name (required), foreign_key (required). Returns: result
reorder_visible_columnsA
Reorder columns in the visible-columns list for a specific context.
This is a convenience tool for reordering columns without manually reconstructing the list. Changes are staged until apply_annotations() is called.
Args: table_name: Name of the table. context: The context to modify (e.g., "compact", "detailed"). new_order: The new order specification. Can be: - List of indices: [2, 0, 1, 3] reorders by current positions - List of column names/refs: ["Name", "RID", ...] specifies exact order
Returns: JSON with the reordered column list for the context.
Examples: # Reorder by indices (move item at index 2 to front) reorder_visible_columns("Image", "compact", [2, 0, 1, 3, 4])
# Reorder by specifying exact column order
reorder_visible_columns("Image", "compact", ["Filename", "Subject", "RID"])
# Note: When using column names, all columns must be included
# or unmentioned columns will be removed from the list
Inputs: context (required), new_order (required), table_name (required). Returns: result
reorder_visible_foreign_keysA
Reorder foreign keys in the visible-foreign-keys list for a specific context.
This is a convenience tool for reordering related tables without manually reconstructing the list. Changes are staged until apply_annotations() is called.
Args: table_name: Name of the table. context: The context to modify (e.g., "detailed", "*"). new_order: The new order specification. Can be: - List of indices: [2, 0, 1] reorders by current positions - List of foreign key refs: [["schema", "fkey1"], ...] specifies exact order
Returns: JSON with the reordered foreign key list for the context.
Examples: # Reorder by indices (move item at index 2 to front) reorder_visible_foreign_keys("Subject", "detailed", [2, 0, 1])
# Reorder by specifying exact foreign key order
reorder_visible_foreign_keys("Subject", "detailed", [
["domain", "Diagnosis_Subject_fkey"],
["domain", "Image_Subject_fkey"]
])
Inputs: context (required), new_order (required), table_name (required). Returns: result
restore_executionA
Restore a previous execution to continue working with it.
Args: execution_rid: RID of the execution to restore (e.g., "1-ABC").
Returns: JSON with execution_rid, workflow_rid, dataset_count.
Inputs: execution_rid (required). Returns: result
set_active_catalogA
Switch the active catalog when multiple catalogs are connected.
Args: hostname: Server hostname of the catalog to activate. catalog_id: Catalog ID to activate.
Inputs: hostname (required), catalog_id (required). Returns: result
set_column_descriptionA
Set or update the description (comment) for a column.
Args: table_name: Name of the table containing the column. column_name: Name of the column to update. description: New description for the column.
Returns: JSON with status, table_name, column_name, description.
Example: set_column_description("Subject", "Age", "Subject age in years at enrollment")
Inputs: table_name (required), column_name (required), description (required). Returns: result
set_column_displayA
Set the column-display annotation on a column.
Controls how a column's values are rendered, including custom formatting and markdown patterns.
Changes are staged locally until apply_annotations() is called.
Args: table_name: Name of the table containing the column. column_name: Name of the column. annotation: The column-display annotation value. Set to null/None to remove.
Column-Display Annotation Schema (tag:isrd.isi.edu,2016:column-display):
{
"*": { // Default for all contexts
"pre_format": {
"format": "%.2f", // printf-style format string
"bool_true_value": "Yes", // Display for true
"bool_false_value": "No" // Display for false
},
"markdown_pattern": "**{{{value}}}**", // Markdown template
"template_engine": "handlebars", // or "mustache"
"column_order": false // Disable sorting, or specify sort
},
"compact": {...}, // Compact/list view options
"detailed": {...}, // Detailed/record view options
"entry": {...}, // Entry form options
"entry/create": {...}, // Create form options
"entry/edit": {...} // Edit form options
}Available options:
pre_format: Pre-processing before displayformat: printf-style format (e.g., "%.2f" for 2 decimal places)bool_true_value: Text to show for boolean truebool_false_value: Text to show for boolean false
markdown_pattern: Template using {{{column_name}}} substitutiontemplate_engine: "handlebars" (default) or "mustache"column_order: Sort configuration or false to disable sorting
Template variables in markdown_pattern:
{{{_value}}}or{{{value}}}: The column's value{{{_row.column_name}}}: Another column's value from same row{{{$fkeys.schema.fkey.values.column}}}: Value from related table
Returns: JSON with status and the target column.
Examples: # Format numbers with 2 decimal places set_column_display("Measurement", "Value", { "*": {"pre_format": {"format": "%.2f"}} })
# Display boolean as Yes/No
set_column_display("Subject", "Active", {
"*": {
"pre_format": {
"bool_true_value": "Active",
"bool_false_value": "Inactive"
}
}
})
# Custom markdown pattern
set_column_display("Image", "URL", {
"detailed": {
"markdown_pattern": "[]({{{_value}}})"
}
})
# Disable column sorting
set_column_display("Subject", "Notes", {
"*": {"column_order": false}
})
Inputs: table_name (required), column_name (required), annotation. Returns: result
set_column_display_nameA
Set the display name shown in the UI for a column.
This is a convenience tool for setting just the display name. For setting multiple display properties at once (name, markdown_name, etc.), use set_display_annotation with a column_name parameter instead.
Args: table_name: Name of the table containing the column. column_name: Name of the column to update. display_name: Human-readable name to display in the UI.
Returns: JSON with status, table_name, column_name, display_name.
Example: set_column_display_name("Subject", "DOB", "Date of Birth")
Inputs: table_name (required), column_name (required), display_name (required). Returns: result
set_column_nullokA
Set whether a column allows NULL values.
Args: table_name: Name of the table containing the column. column_name: Name of the column to update. nullok: True to allow NULL values, False to require values.
Returns: JSON with status, table_name, column_name, nullok.
Note: Setting nullok=False will fail if the column contains NULL values.
Example: set_column_nullok("Subject", "Name", False) # Make Name required
Inputs: nullok (required), table_name (required), column_name (required). Returns: result
set_dataset_descriptionA
Set or update the description for a dataset.
Updates the dataset's description in the catalog. Good descriptions help users understand the dataset's purpose, contents, and intended use.
Args: dataset_rid: RID of the dataset to update. description: New description text.
Returns: JSON with status, dataset_rid, description.
Example: set_dataset_description("1-ABC", "Training images for CIFAR-10 classification")
Inputs: dataset_rid (required), description (required). Returns: result
set_default_schemaA
Set the default schema for the active catalog connection.
When a catalog has multiple domain schemas, many operations require knowing which schema to use. Setting a default schema avoids having to specify it on every call.
Args: schema_name: Name of the domain schema to set as default (e.g., "isa", "my_project"). Must be one of the catalog's domain schemas.
Returns: JSON with status, default_schema, and domain_schemas.
Example: set_default_schema("isa") -> sets "isa" as the default schema
Inputs: schema_name (required). Returns: result
set_display_annotationA
Set the display annotation on a table or column.
The display annotation controls basic naming and display options. Changes are staged locally until apply_annotations() is called.
Args: table_name: Name of the table. column_name: Name of the column (optional). If provided, sets the annotation on the column; otherwise sets it on the table. annotation: The display annotation value. Set to null/None to remove.
Display Annotation Schema (tag:isrd.isi.edu,2015:display):
{
"name": "string", // Display name (mutually exclusive with markdown_name)
"markdown_name": "string", // Markdown-formatted name (mutually exclusive with name)
"name_style": {
"underline_space": true, // Replace underscores with spaces
"title_case": true, // Apply title case
"markdown": true // Interpret as markdown
},
"comment": "string", // Tooltip/description text
"show_null": { // How to display null values per context
"*": true, // true = show "No value", false = hide, "string" = custom
"compact": false,
"detailed": ""N/A""
},
"show_foreign_key_link": { // Show FK as link per context
"*": true,
"compact": false
}
}Valid contexts for show_null/show_foreign_key_link:
"*" (all contexts)
"compact", "compact/select", "compact/brief", "compact/brief/inline"
"detailed"
Returns: JSON with status and the target (table or column).
Examples: # Set table display name set_display_annotation("Image", annotation={"name": "Images"})
# Set column display name
set_display_annotation("Image", "Filename", {"name": "File Name"})
# Set name style to replace underscores with spaces
set_display_annotation("Image", annotation={"name_style": {"underline_space": true}})
# Remove display annotation
set_display_annotation("Image", annotation=null)
Inputs: table_name (required), annotation, column_name. Returns: result
set_execution_descriptionA
Set or update the description for an execution.
Updates the execution's description in the catalog. Good descriptions help users understand what the execution accomplished and any notable results.
Args: execution_rid: RID of the execution to update. description: New description text.
Returns: JSON with status, execution_rid, description.
Example: set_execution_description("2-XYZ", "Training run with lr=0.001, achieved 95% accuracy")
Inputs: description (required), execution_rid (required). Returns: result
set_row_name_patternA
Set the pattern used to display row names in the UI.
The pattern uses Handlebars syntax with triple braces for column values.
Args: table_name: Name of the table to update. pattern: Handlebars template (e.g., "{{{Name}}}" or "{{{FirstName}}} {{{LastName}}}").
Returns: JSON with status, table_name, pattern.
Examples: set_row_name_pattern("Subject", "{{{Name}}}") set_row_name_pattern("Image", "{{{Filename}}} ({{{RID}}})")
Inputs: pattern (required), table_name (required). Returns: result
set_table_descriptionA
Set or update the description (comment) for a table.
Args: table_name: Name of the table to update. description: New description for the table.
Returns: JSON with status, table_name, description.
Example: set_table_description("Image", "Medical images for analysis")
Inputs: table_name (required), description (required). Returns: result
set_table_displayA
Set the table-display annotation on a table.
Controls table-level display options like row naming patterns, page size, and row ordering.
Changes are staged locally until apply_annotations() is called.
Args: table_name: Name of the table. annotation: The table-display annotation value. Set to null/None to remove.
Table-Display Annotation Schema (tag:isrd.isi.edu,2016:table-display):
{
"row_name": { // How to display row identifiers
"row_markdown_pattern": "{{{Name}}}", // Template for row display
"template_engine": "handlebars" // or "mustache"
},
"detailed": { // Options for detailed view
"hide_column_headers": true, // Hide column headers
"collapse_toc_panel": true // Collapse table of contents
},
"compact": { // Options for compact/list view
"page_size": 25, // Rows per page
"row_order": [ // Default sort order
{"column": "RCT", "descending": true},
"Name"
]
},
"*": { // Default for all contexts
"page_size": 10,
"row_order": ["Name"]
}
}Context-specific options:
row_name: Special context for row identifier displaydetailed: Detailed/record viewcompact,compact/select,compact/brief: List views*: Default for unspecified contexts
Available options per context:
row_order: Array of sort keys (column name or {column, descending})page_size: Number of rows per pagecollapse_toc_panel: Boolean to collapse TOC (detailed only)hide_column_headers: Boolean to hide headers (detailed only)row_markdown_pattern: Template string using {{{column}}} syntaxpage_markdown_pattern: Template for entire pageseparator_markdown: Separator between rowsprefix_markdown: Content before rowssuffix_markdown: Content after rowstemplate_engine: "handlebars" or "mustache"
Returns: JSON with status and the table name.
Examples: # Set row name pattern set_table_display("Subject", { "row_name": { "row_markdown_pattern": "{{{Name}}} ({{{Species}}})" } })
# Set default sort order and page size
set_table_display("Image", {
"compact": {
"row_order": [{"column": "RCT", "descending": true}],
"page_size": 50
}
})
# Configure detailed view
set_table_display("Page", {
"detailed": {
"hide_column_headers": true,
"collapse_toc_panel": true
}
})
Inputs: table_name (required), annotation. Returns: result
set_table_display_nameA
Set the display name shown in the UI for a table.
This is a convenience tool for setting just the display name. For setting multiple display properties at once (name, markdown_name, etc.), use set_display_annotation instead.
Args: table_name: Name of the table to update. display_name: Human-readable name to display in the UI.
Returns: JSON with status, table_name, display_name.
Example: set_table_display_name("Image", "Medical Images")
Inputs: table_name (required), display_name (required). Returns: result
set_visible_columnsA
Set the visible-columns annotation on a table.
Controls which columns appear in different UI contexts and their order. Changes are staged locally until apply_annotations() is called.
Args: table_name: Name of the table. annotation: The visible-columns annotation value. Set to null/None to remove.
Visible-Columns Annotation Schema (tag:isrd.isi.edu,2016:visible-columns):
{
"compact": [...], // Columns for compact/list view
"detailed": [...], // Columns for detailed/record view
"entry": [...], // Columns for data entry (create/edit)
"entry/create": [...], // Columns for create only
"entry/edit": [...], // Columns for edit only
"export": [...], // Columns for export
"filter": { // Faceted search configuration
"and": [...]
},
"*": [...] // Default for all contexts
}Column directive formats (items in column lists):
Simple column name (string):
"RID"Foreign key reference (array of [schema, constraint_name]):
["schema_name", "fkey_constraint_name"]Pseudo-column (object with source path):
{ "source": "column_name", // Simple column "source": [ // Or path through foreign keys {"outbound": ["schema", "fkey"]}, "target_column" ], "sourcekey": "predefined_key", // OR reference to source-definitions "entity": true, // Show as entity (row) vs scalar value "aggregate": "array", // Aggregation: min, max, cnt, cnt_d, array, array_d "self_link": true, // Link to current row "markdown_name": "Display Name", // Custom column header "comment": "Tooltip text", // Column tooltip "display": { // Display options "markdown_pattern": "{{{value}}}", "template_engine": "handlebars", // or "mustache" "show_foreign_key_link": true, "array_ux_mode": "csv" // raw, csv, olist, ulist } }
Filter context (for faceted search):
{
"filter": {
"and": [
{
"source": "column_name",
"markdown_name": "Filter Label",
"open": true, // Expand by default
"ux_mode": "choices", // choices, ranges, check_presence
"bar_plot": true, // Show distribution chart
"hide_null_choice": true, // Hide "No value" option
"choices": ["val1", "val2"], // Preset choices
"ranges": [{"min": 0, "max": 100}] // Preset ranges
}
]
}
}Returns: JSON with status and the table name.
Examples: # Simple column list for compact view set_visible_columns("Image", { "compact": ["RID", "Filename", "Subject"], "detailed": ["RID", "Filename", "Subject", "Description", "URL"] })
# Include foreign key as a column
set_visible_columns("Image", {
"compact": ["Filename", ["domain", "Image_Subject_fkey"]]
})
# Pseudo-column traversing foreign key
set_visible_columns("Image", {
"detailed": [
"Filename",
{
"source": [{"outbound": ["domain", "Image_Subject_fkey"]}, "Name"],
"markdown_name": "Subject Name"
}
]
})
# Configure faceted search
set_visible_columns("Image", {
"filter": {
"and": [
{"source": "Species", "open": true},
{"source": "Quality", "ux_mode": "choices"}
]
}
})
Inputs: table_name (required), annotation. Returns: result
set_visible_foreign_keysA
Set the visible-foreign-keys annotation on a table.
Controls which related tables (via inbound foreign keys) appear in different UI contexts and their order. These show as "related tables" sections in the detailed view.
Changes are staged locally until apply_annotations() is called.
Args: table_name: Name of the table. annotation: The visible-foreign-keys annotation value. Set to null/None to remove.
Visible-Foreign-Keys Annotation Schema (tag:isrd.isi.edu,2016:visible-foreign-keys):
{
"detailed": [...], // Related tables in detailed view
"*": [...] // Default for all contexts
}Foreign key directive formats (items in the lists):
Inbound foreign key reference (array of [schema, constraint_name]):
["schema_name", "fkey_constraint_name"]The constraint must be an INBOUND foreign key (i.e., another table references this table).
Pseudo-column for related entities (object):
{ "source": [ {"inbound": ["schema", "fkey_to_this_table"]}, {"outbound": ["schema", "fkey_to_related"]}, "column_name" ], "sourcekey": "predefined_key", // OR reference to source-definitions "markdown_name": "Related Items", "comment": "Tooltip text", "display": { "markdown_pattern": "...", "template_engine": "handlebars" } }
Returns: JSON with status and the table name.
Examples: # Show specific related tables in detailed view set_visible_foreign_keys("Subject", { "detailed": [ ["domain", "Image_Subject_fkey"], ["domain", "Diagnosis_Subject_fkey"] ] })
# Hide all related tables
set_visible_foreign_keys("Subject", {"detailed": []})
# Pseudo-column for complex relationship
set_visible_foreign_keys("Subject", {
"detailed": [
{
"source": [{"inbound": ["domain", "Image_Subject_fkey"]}],
"markdown_name": "Subject Images"
}
]
})
Inputs: table_name (required), annotation. Returns: result
set_workflow_descriptionA
Set or update the description for a workflow.
Updates the workflow's description in the catalog. Good descriptions help users understand what the workflow does and how to use it.
Args: workflow_rid: RID of the workflow to update. description: New description text.
Returns: JSON with status, workflow_rid, description.
Example: set_workflow_description("3-WKF", "Trains CNN on image data with augmentation")
Inputs: description (required), workflow_rid (required). Returns: result
split_datasetA
Split a dataset into training, testing, and optionally validation subsets.
Creates a new dataset hierarchy with full provenance tracking:
Split (parent, type: "Split")
Training (child, type: "Training" + training_types)
Validation (child, type: "Validation" + validation_types) # if val_size
Testing (child, type: "Testing" + testing_types)
The API follows scikit-learn's train_test_split conventions for test_size, train_size, val_size, shuffle, and seed parameters.
Splitting strategies:
Random (default): Shuffles members and splits at the boundary. No denormalization needed. Fast for any dataset size.
Stratified: Maintains class distribution across splits. Requires stratify_by_column and include_tables. Uses scikit-learn internally.
Column naming for stratification:
When using stratify_by_column, the column name must match the
denormalized DataFrame format: {TableName}_{ColumnName}.
For example, to stratify by the Image_Class column from the
Image_Classification feature table, use
Image_Classification_Image_Class.
Derive the column name from the table schema (via the
deriva://catalog/schema or deriva://catalog/features resource)
rather than calling denormalize_dataset().
Args: source_dataset_rid: RID of the source dataset to split. test_size: Test set size as a fraction (0-1) or absolute count. Default: 0.2 (20% of data). train_size: Train set size as a fraction (0-1) or absolute count. Default: None (complement of test_size and val_size). val_size: Validation set size as a fraction (0-1) or absolute count. Default: None (no validation split, two-way only). When provided, creates a three-way train/val/test split. seed: Random seed for reproducibility. Default: 42. shuffle: Whether to shuffle before splitting. Default: True. stratify_by_column: Column name in the denormalized DataFrame for stratified splitting. Maintains class distribution across all partitions. Requires include_tables. Example: "Image_Classification_Image_Class". stratify_missing: Policy for null values in the stratify column. "error" (default): raise if any nulls exist, reporting count and percentage. "drop": exclude rows with null values from the split. "include": treat nulls as a separate class. Only used when stratify_by_column is set. element_table: Element table to split (e.g., "Image"). If not specified, auto-detected from the dataset's members. include_tables: Tables to include when denormalizing. Required when using stratify_by_column. Example: ["Image", "Image_Classification"]. training_types: Additional dataset types for the training set beyond "Training". Example: ["Labeled"]. testing_types: Additional dataset types for the testing set beyond "Testing". Example: ["Labeled"]. validation_types: Additional dataset types for the validation set beyond "Validation". Example: ["Labeled"]. Ignored when val_size is None. split_description: Description for the parent Split dataset. dry_run: If True, return what would happen without modifying the catalog. Useful for previewing split sizes.
Returns: JSON with split results including: - split: RID, version, and count of the parent Split dataset - training: RID, version, and count of the Training dataset - validation: RID, version, and count of the Validation dataset (if val_size) - testing: RID, version, and count of the Testing dataset - source: RID of the source dataset
Example: # Random 80/20 split split_dataset("28D0", test_size=0.2, seed=42)
# Three-way train/val/test split
split_dataset("28D0", test_size=0.2, val_size=0.1, seed=42)
# Stratified split maintaining class balance
split_dataset("28D0", test_size=0.2,
stratify_by_column="Image_Classification_Image_Class",
include_tables=["Image", "Image_Classification"])
# Fixed-count split with labeled types
split_dataset("28D0", train_size=400, test_size=100,
training_types=["Labeled"], testing_types=["Labeled"])
# Dry run to preview
split_dataset("28D0", test_size=0.2, dry_run=True)
Inputs: source_dataset_rid (required), seed, dry_run, shuffle, +11 more. Returns: result
start_executionA
Start timing the active execution. Call after create_execution().
Records the start timestamp for duration tracking. The execution status changes to "running".
Inputs: no named parameters. Returns: result
stop_executionA
Stop timing and mark execution complete.
Records the stop timestamp and calculates duration. Call this after your ML workflow completes.
Inputs: no named parameters. Returns: result
update_catalog_aliasA
Update an existing catalog alias.
Can change the target catalog the alias points to, or update the owner ACL.
Args: hostname: Server hostname (e.g., "www.eye-ai.org"). alias_name: The alias identifier to update. alias_target: New target catalog ID. If None, target is unchanged. Pass empty string "" to unbind the alias from any catalog. owner: New owner ACL (list of user/group identifiers). If None, owner is unchanged.
Returns: JSON with status and updated alias details.
Example: update_catalog_alias("localhost", "my-project", alias_target="50") -> {"status": "updated", "alias": "my-project", "target": "50"}
Inputs: hostname (required), alias_name (required), owner, alias_target. Returns: result
update_execution_statusA
Update the execution status with a progress message.
Args: status: One of "pending", "running", "completed", "failed". message: Progress message or error description.
Returns: JSON with execution_rid, new_status, message.
Inputs: status (required), message (required). Returns: result
update_recordA
Update fields in an existing record.
Args: table_name: Name of the table containing the record. rid: The RID of the record to update. updates: Dictionary of {column: new_value} updates.
Returns: JSON with update status.
Example: update_record("Subject", "1-ABC", {"Age": 46, "Status": "Active"})
Inputs: rid (required), updates (required), table_name (required). Returns: result
update_term_descriptionA
Update the description of a vocabulary term.
Args: vocabulary_name: Name of the vocabulary table (e.g., "Dataset_Type"). term_name: Primary name of the term to update. description: New description for the term.
Returns: JSON with status, name, updated description.
Example: update_term_description("Dataset_Type", "Training", "Data used to train models")
Inputs: term_name (required), description (required), vocabulary_name (required). Returns: result
validate_ridsA
Validate that RIDs exist in the catalog before running experiments.
Performs batch validation of RIDs to catch configuration errors early with clear error messages. Use this before running experiments to ensure all referenced datasets, assets, and other entities actually exist.
Args: dataset_rids: List of dataset RIDs to validate. asset_rids: List of asset RIDs to validate (model weights, etc.). dataset_versions: Dictionary mapping dataset RID to required version string (e.g., {"1-ABC": "0.4.0"}). Validates version exists. workflow_rids: List of workflow RIDs to validate. execution_rids: List of execution RIDs to validate. warn_missing_descriptions: If True (default), include warnings for datasets missing descriptions.
Returns: JSON with: - is_valid: True if all validations passed - errors: List of error messages - warnings: List of warning messages - validated_rids: Dictionary of validated RID info
Example: validate_rids( dataset_rids=["1-ABC", "2-DEF"], dataset_versions={"1-ABC": "0.4.0"}, asset_rids=["3-GHI"] ) -> { "is_valid": true, "errors": [], "warnings": [], "validated_rids": {...} }
Inputs: asset_rids, dataset_rids, workflow_rids, execution_rids, +2 more. Returns: result
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 101 tools
The annotation tools heavily overlap: set_visible_columns vs. add/remove/reorder_visible_column, set_display_annotation vs. set_table_display/set_column_display, and set_table_display vs. set_table_display_name are easy to confuse. There are also near-duplicate pairs like clone_catalog/clone_catalog_async and create_dataset_type_term/add_term, plus similar-sounding create_execution and create_execution_dataset. With 101 tools, an agent would frequently struggle to pick the right one.
Most names use snake_case with a verb prefix, but the conventions are mixed: set_*, add_*, remove_*, create_*, delete_*, update_*, and apply_* are used inconsistently for parallel operations (e.g., set_visible_columns vs. add_visible_column). Some names break the pattern entirely (bag_info, rag_update, cite), and the RAG tools mix noun-first and verb-first naming. Overall it is readable but not predictable.
101 tools is an extreme count for a single MCP server and creates a massive selection burden. Even accounting for the broad DerivaML domain, many tools are convenience variants or specialized duplicates that bloat the surface. This is well beyond the range where an agent can efficiently discover and choose the correct tool.
The server covers many workflows but has notable gaps: tools referenced in examples like get_table_annotations, list_foreign_keys, and lookup_feature are not actually provided, and there are no direct list/get tools for core entities such as datasets, workflows, or features. These missing read/discovery operations force agents to use workarounds like preview_table or rag_search, and can cause failures when following described workflows.
Maintenance
Related MCP Connectors
Manage ConfigDirector feature flags, remote configs, and targeting rules from AI assistants.
Connect AI assistants to GitHub - manage repos, issues, PRs, and workflows through natural language.
MCP Hub: AI service discovery, per-user OAuth, and multi-service workflow orchestration
Connect AI clients to biomedical data and tools.
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceEnables AI assistants to interact with Databricks workspaces programmatically, providing comprehensive tools for cluster management, notebook operations, job orchestration, Unity Catalog data governance, user management, permissions control, and FinOps cost analytics.MIT
- AlicenseDqualityDmaintenanceEnables AI assistants to perform MLOps workflows such as experiment tracking, model registry, dataset management, pipeline orchestration, and data lineage by wrapping DVC, MLflow, and Git.1001MIT
- AlicenseNot gradedqualityCmaintenanceEnables AI assistants to interact with OSDU platform services including search, data management, and schema operations.9Apache 2.0
- AlicenseNot gradedqualityDmaintenanceEnables AI systems to manage Verodat accounts, workspaces, and datasets, including creating datasets, querying data, and executing AI-powered queries through natural language.9 npmApache 2.0