Set the display annotation on a table or column.

The display annotation controls basic naming and display options. Changes are staged locally until apply_annotations() is called.

Args: table_name: Name of the table. column_name: Name of the column (optional). If provided, sets the annotation on the column; otherwise sets it on the table. annotation: The display annotation value. Set to null/None to remove.

Display Annotation Schema (tag:isrd.isi.edu,2015:display):

{
    "name": "string",           // Display name (mutually exclusive with markdown_name)
    "markdown_name": "string",  // Markdown-formatted name (mutually exclusive with name)
    "name_style": {
        "underline_space": true,  // Replace underscores with spaces
        "title_case": true,       // Apply title case
        "markdown": true          // Interpret as markdown
    },
    "comment": "string",        // Tooltip/description text
    "show_null": {              // How to display null values per context
        "*": true,              // true = show "No value", false = hide, "string" = custom
        "compact": false,
        "detailed": ""N/A""
    },
    "show_foreign_key_link": {  // Show FK as link per context
        "*": true,
        "compact": false
    }
}

Valid contexts for show_null/show_foreign_key_link:

• "*" (all contexts)

• "compact", "compact/select", "compact/brief", "compact/brief/inline"

• "detailed"

Returns: JSON with status and the target (table or column).

Examples: # Set table display name set_display_annotation("Image", annotation={"name": "Images"})

# Set column display name
set_display_annotation("Image", "Filename", {"name": "File Name"})

# Set name style to replace underscores with spaces
set_display_annotation("Image", annotation={"name_style": {"underline_space": true}})

# Remove display annotation
set_display_annotation("Image", annotation=null)

Set the visible-columns annotation on a table.

Controls which columns appear in different UI contexts and their order. Changes are staged locally until apply_annotations() is called.

Args: table_name: Name of the table. annotation: The visible-columns annotation value. Set to null/None to remove.

Visible-Columns Annotation Schema (tag:isrd.isi.edu,2016:visible-columns):

{
    "compact": [...],           // Columns for compact/list view
    "detailed": [...],          // Columns for detailed/record view
    "entry": [...],             // Columns for data entry (create/edit)
    "entry/create": [...],      // Columns for create only
    "entry/edit": [...],        // Columns for edit only
    "export": [...],            // Columns for export
    "filter": {                 // Faceted search configuration
        "and": [...]
    },
    "*": [...]                  // Default for all contexts
}

Column directive formats (items in column lists):

1. Simple column name (string):

   "RID"

2. Foreign key reference (array of [schema, constraint_name]):

   ["schema_name", "fkey_constraint_name"]

3. Pseudo-column (object with source path):

   {
       "source": "column_name",           // Simple column
       "source": [                        // Or path through foreign keys
           {"outbound": ["schema", "fkey"]},
           "target_column"
       ],
       "sourcekey": "predefined_key",    // OR reference to source-definitions
       "entity": true,                   // Show as entity (row) vs scalar value
       "aggregate": "array",             // Aggregation: min, max, cnt, cnt_d, array, array_d
       "self_link": true,                // Link to current row
       "markdown_name": "Display Name",  // Custom column header
       "comment": "Tooltip text",        // Column tooltip
       "display": {                      // Display options
           "markdown_pattern": "{{{value}}}",
           "template_engine": "handlebars",  // or "mustache"
           "show_foreign_key_link": true,
           "array_ux_mode": "csv"           // raw, csv, olist, ulist
       }
   }

Filter context (for faceted search):

{
    "filter": {
        "and": [
            {
                "source": "column_name",
                "markdown_name": "Filter Label",
                "open": true,              // Expand by default
                "ux_mode": "choices",      // choices, ranges, check_presence
                "bar_plot": true,          // Show distribution chart
                "hide_null_choice": true,  // Hide "No value" option
                "choices": ["val1", "val2"], // Preset choices
                "ranges": [{"min": 0, "max": 100}]  // Preset ranges
            }
        ]
    }
}

Returns: JSON with status and the table name.

Examples: # Simple column list for compact view set_visible_columns("Image", { "compact": ["RID", "Filename", "Subject"], "detailed": ["RID", "Filename", "Subject", "Description", "URL"] })

# Include foreign key as a column
set_visible_columns("Image", {
    "compact": ["Filename", ["domain", "Image_Subject_fkey"]]
})

# Pseudo-column traversing foreign key
set_visible_columns("Image", {
    "detailed": [
        "Filename",
        {
            "source": [{"outbound": ["domain", "Image_Subject_fkey"]}, "Name"],
            "markdown_name": "Subject Name"
        }
    ]
})

# Configure faceted search
set_visible_columns("Image", {
    "filter": {
        "and": [
            {"source": "Species", "open": true},
            {"source": "Quality", "ux_mode": "choices"}
        ]
    }
})

Set the visible-foreign-keys annotation on a table.

Controls which related tables (via inbound foreign keys) appear in different UI contexts and their order. These show as "related tables" sections in the detailed view.

Changes are staged locally until apply_annotations() is called.

Args: table_name: Name of the table. annotation: The visible-foreign-keys annotation value. Set to null/None to remove.

Visible-Foreign-Keys Annotation Schema (tag:isrd.isi.edu,2016:visible-foreign-keys):

{
    "detailed": [...],  // Related tables in detailed view
    "*": [...]          // Default for all contexts
}

Foreign key directive formats (items in the lists):

1. Inbound foreign key reference (array of [schema, constraint_name]):

   ["schema_name", "fkey_constraint_name"]

   The constraint must be an INBOUND foreign key (i.e., another table references this table).

2. Pseudo-column for related entities (object):

   {
       "source": [
           {"inbound": ["schema", "fkey_to_this_table"]},
           {"outbound": ["schema", "fkey_to_related"]},
           "column_name"
       ],
       "sourcekey": "predefined_key",    // OR reference to source-definitions
       "markdown_name": "Related Items",
       "comment": "Tooltip text",
       "display": {
           "markdown_pattern": "...",
           "template_engine": "handlebars"
       }
   }

Returns: JSON with status and the table name.

Examples: # Show specific related tables in detailed view set_visible_foreign_keys("Subject", { "detailed": [ ["domain", "Image_Subject_fkey"], ["domain", "Diagnosis_Subject_fkey"] ] })

# Hide all related tables
set_visible_foreign_keys("Subject", {"detailed": []})

# Pseudo-column for complex relationship
set_visible_foreign_keys("Subject", {
    "detailed": [
        {
            "source": [{"inbound": ["domain", "Image_Subject_fkey"]}],
            "markdown_name": "Subject Images"
        }
    ]
})

Set the table-display annotation on a table.

Controls table-level display options like row naming patterns, page size, and row ordering.

Changes are staged locally until apply_annotations() is called.

Args: table_name: Name of the table. annotation: The table-display annotation value. Set to null/None to remove.

Table-Display Annotation Schema (tag:isrd.isi.edu,2016:table-display):

{
    "row_name": {                              // How to display row identifiers
        "row_markdown_pattern": "{{{Name}}}",  // Template for row display
        "template_engine": "handlebars"        // or "mustache"
    },
    "detailed": {                              // Options for detailed view
        "hide_column_headers": true,           // Hide column headers
        "collapse_toc_panel": true             // Collapse table of contents
    },
    "compact": {                               // Options for compact/list view
        "page_size": 25,                       // Rows per page
        "row_order": [                         // Default sort order
            {"column": "RCT", "descending": true},
            "Name"
        ]
    },
    "*": {                                     // Default for all contexts
        "page_size": 10,
        "row_order": ["Name"]
    }
}

Context-specific options:

• row_name: Special context for row identifier display

• detailed: Detailed/record view

• compact, compact/select, compact/brief: List views

• *: Default for unspecified contexts

Available options per context:

• row_order: Array of sort keys (column name or {column, descending})

• page_size: Number of rows per page

• collapse_toc_panel: Boolean to collapse TOC (detailed only)

• hide_column_headers: Boolean to hide headers (detailed only)

• row_markdown_pattern: Template string using {{{column}}} syntax

• page_markdown_pattern: Template for entire page

• separator_markdown: Separator between rows

• prefix_markdown: Content before rows

• suffix_markdown: Content after rows

• template_engine: "handlebars" or "mustache"

Returns: JSON with status and the table name.

Examples: # Set row name pattern set_table_display("Subject", { "row_name": { "row_markdown_pattern": "{{{Name}}} ({{{Species}}})" } })

# Set default sort order and page size
set_table_display("Image", {
    "compact": {
        "row_order": [{"column": "RCT", "descending": true}],
        "page_size": 50
    }
})

# Configure detailed view
set_table_display("Page", {
    "detailed": {
        "hide_column_headers": true,
        "collapse_toc_panel": true
    }
})

Set the column-display annotation on a column.

Controls how a column's values are rendered, including custom formatting and markdown patterns.

Changes are staged locally until apply_annotations() is called.

Args: table_name: Name of the table containing the column. column_name: Name of the column. annotation: The column-display annotation value. Set to null/None to remove.

Column-Display Annotation Schema (tag:isrd.isi.edu,2016:column-display):

{
    "*": {                                    // Default for all contexts
        "pre_format": {
            "format": "%.2f",                 // printf-style format string
            "bool_true_value": "Yes",         // Display for true
            "bool_false_value": "No"          // Display for false
        },
        "markdown_pattern": "**{{{value}}}**",  // Markdown template
        "template_engine": "handlebars",        // or "mustache"
        "column_order": false                   // Disable sorting, or specify sort
    },
    "compact": {...},                         // Compact/list view options
    "detailed": {...},                        // Detailed/record view options
    "entry": {...},                           // Entry form options
    "entry/create": {...},                    // Create form options
    "entry/edit": {...}                       // Edit form options
}

Available options:

• pre_format: Pre-processing before display

  • format: printf-style format (e.g., "%.2f" for 2 decimal places)

  • bool_true_value: Text to show for boolean true

  • bool_false_value: Text to show for boolean false

• markdown_pattern: Template using {{{column_name}}} substitution

• template_engine: "handlebars" (default) or "mustache"

• column_order: Sort configuration or false to disable sorting

Template variables in markdown_pattern:

• {{{_value}}} or {{{value}}}: The column's value

• {{{_row.column_name}}}: Another column's value from same row

• {{{$fkeys.schema.fkey.values.column}}}: Value from related table

Returns: JSON with status and the target column.

Examples: # Format numbers with 2 decimal places set_column_display("Measurement", "Value", { "*": {"pre_format": {"format": "%.2f"}} })

# Display boolean as Yes/No
set_column_display("Subject", "Active", {
    "*": {
        "pre_format": {
            "bool_true_value": "Active",
            "bool_false_value": "Inactive"
        }
    }
})

# Custom markdown pattern
set_column_display("Image", "URL", {
    "detailed": {
        "markdown_pattern": "[![Image]({{{_value}}})]({{{_value}}})"
    }
})

# Disable column sorting
set_column_display("Subject", "Notes", {
    "*": {"column_order": false}
})

Apply all staged annotation changes to the catalog.

This commits any annotation changes made via set_display_annotation, set_visible_columns, set_visible_foreign_keys, set_table_display, or set_column_display to the remote catalog.

Returns: JSON with status and details of the apply operation.

Example workflow: 1. get_table_annotations("Image") # Check current state 2. set_display_annotation("Image", annotation={"name": "Images"}) 3. set_visible_columns("Image", {"compact": ["RID", "Filename"]}) 4. apply_annotations() # Commit all changes

Add a column to the visible-columns list for a specific context.

This is a convenience tool for adding columns without replacing the entire visible-columns annotation. Changes are staged until apply_annotations() is called.

Args: table_name: Name of the table. context: The context to modify. See Contexts below. column: Column to add. Can be: - String: column name (e.g., "Filename") - List: foreign key reference (e.g., ["schema", "fkey_name"]) - Dict: pseudo-column definition (see set_visible_columns) position: Position to insert at (0-indexed). If None, appends to end.

Contexts for visible-columns:

Returns: JSON with the updated column list for the context.

Examples: # Add column to end of compact view add_visible_column("Image", "compact", "Description")

# Add foreign key reference at position 1
add_visible_column("Image", "detailed", ["domain", "Image_Subject_fkey"], 1)

# Add pseudo-column
add_visible_column("Image", "compact", {
    "source": [{"outbound": ["domain", "Image_Subject_fkey"]}, "Name"],
    "markdown_name": "Subject"
})

Remove a column from the visible-columns list for a specific context.

This is a convenience tool for removing columns without replacing the entire visible-columns annotation. Changes are staged until apply_annotations() is called.

Args: table_name: Name of the table. context: The context to modify (e.g., "compact", "detailed"). column: Column to remove. Can be: - String: column name to find and remove - List: foreign key reference [schema, constraint] to find and remove - Integer: index position to remove (0-indexed)

Returns: JSON with the updated column list for the context.

Examples: # Remove by column name remove_visible_column("Image", "compact", "Description")

# Remove by foreign key reference
remove_visible_column("Image", "detailed", ["domain", "Image_Subject_fkey"])

# Remove by position (first column)
remove_visible_column("Image", "compact", 0)

Reorder columns in the visible-columns list for a specific context.

This is a convenience tool for reordering columns without manually reconstructing the list. Changes are staged until apply_annotations() is called.

Args: table_name: Name of the table. context: The context to modify (e.g., "compact", "detailed"). new_order: The new order specification. Can be: - List of indices: [2, 0, 1, 3] reorders by current positions - List of column names/refs: ["Name", "RID", ...] specifies exact order

Returns: JSON with the reordered column list for the context.

Examples: # Reorder by indices (move item at index 2 to front) reorder_visible_columns("Image", "compact", [2, 0, 1, 3, 4])

# Reorder by specifying exact column order
reorder_visible_columns("Image", "compact", ["Filename", "Subject", "RID"])

# Note: When using column names, all columns must be included
# or unmentioned columns will be removed from the list

Add a foreign key to the visible-foreign-keys list for a specific context.

This is a convenience tool for adding related tables without replacing the entire visible-foreign-keys annotation. Changes are staged until apply_annotations() is called.

Args: table_name: Name of the table. context: The context to modify (typically "detailed" or "*"). foreign_key: Foreign key to add. Can be: - List: inbound foreign key reference (e.g., ["schema", "Other_Table_fkey"]) - Dict: pseudo-column definition for complex relationships position: Position to insert at (0-indexed). If None, appends to end.

Contexts for visible-foreign-keys:

Important: Only INBOUND foreign keys are valid - these are foreign keys from OTHER tables that reference THIS table. Use list_foreign_keys() to see which inbound foreign keys are available.

Returns: JSON with the updated foreign key list for the context.

Examples: # Add inbound foreign key to detailed view add_visible_foreign_key("Subject", "detailed", ["domain", "Image_Subject_fkey"])

# Add at specific position
add_visible_foreign_key("Subject", "detailed", ["domain", "Diagnosis_Subject_fkey"], 0)

# Add pseudo-column for complex relationship
add_visible_foreign_key("Subject", "detailed", {
    "source": [{"inbound": ["domain", "Image_Subject_fkey"]}],
    "markdown_name": "Subject Images"
})

Remove a foreign key from the visible-foreign-keys list for a specific context.

This is a convenience tool for removing related tables without replacing the entire visible-foreign-keys annotation. Changes are staged until apply_annotations() is called.

Args: table_name: Name of the table. context: The context to modify (e.g., "detailed", "*"). foreign_key: Foreign key to remove. Can be: - List: foreign key reference [schema, constraint] to find and remove - Integer: index position to remove (0-indexed)

Returns: JSON with the updated foreign key list for the context.

Examples: # Remove by foreign key reference remove_visible_foreign_key("Subject", "detailed", ["domain", "Image_Subject_fkey"])

# Remove by position (first foreign key)
remove_visible_foreign_key("Subject", "detailed", 0)

Reorder foreign keys in the visible-foreign-keys list for a specific context.

This is a convenience tool for reordering related tables without manually reconstructing the list. Changes are staged until apply_annotations() is called.

Args: table_name: Name of the table. context: The context to modify (e.g., "detailed", "*"). new_order: The new order specification. Can be: - List of indices: [2, 0, 1] reorders by current positions - List of foreign key refs: [["schema", "fkey1"], ...] specifies exact order

Returns: JSON with the reordered foreign key list for the context.

Examples: # Reorder by indices (move item at index 2 to front) reorder_visible_foreign_keys("Subject", "detailed", [2, 0, 1])

# Reorder by specifying exact foreign key order
reorder_visible_foreign_keys("Subject", "detailed", [
    ["domain", "Diagnosis_Subject_fkey"],
    ["domain", "Image_Subject_fkey"]
])

Get all available template variables for a table.

Returns the columns, foreign keys, and special variables that can be used in Handlebars templates (row_markdown_pattern, markdown_pattern, etc.) for the specified table.

Args: table_name: Name of the table to get variables for.

Returns: JSON with columns, foreign_keys, and special variables available for use in templates.

Example: get_handlebars_template_variables("Image") -> { "table": "Image", "columns": [ {"name": "RID", "type": "ermrest_rid", "template": "{{{RID}}}"}, {"name": "Filename", "type": "text", "template": "{{{Filename}}}"}, ... ], "foreign_keys": [ { "constraint": ["domain", "Image_Subject_fkey"], "to_table": "Subject", "values_template": "{{{$fkeys.domain.Image_Subject_fkey.values.column}}}", "row_name_template": "{{{$fkeys.domain.Image_Subject_fkey.rowName}}}" } ], "special_variables": {...} }

Get sample row data from a table for template testing.

Retrieves a few sample rows from the table that can be used to test Handlebars templates. Use this to see real values that would be available in templates.

Args: table_name: Name of the table. limit: Number of sample rows to return (default: 3, max: 10).

Returns: JSON with sample rows and their column values.

Example: get_table_sample_data("Image", 2) -> { "table": "Image", "sample_rows": [ {"RID": "1-ABC", "Filename": "scan001.jpg", "Subject": "2-DEF", ...}, {"RID": "1-XYZ", "Filename": "scan002.jpg", "Subject": "2-DEF", ...} ], "template_test_suggestion": "Try: {{{Filename}}} - Subject: {{{Subject}}}" }

Create an ML workspace by cloning data reachable from a root RID.

Starts the workspace creation in the background and immediately returns a task_id that you can use to check progress.

The operation uses export annotations (if available) to determine which tables and paths to follow from the root RID, then fills in any uncovered tables (vocabularies, associations). Uses async data copying for performance.

Use this for large catalogs or cross-server clones that may take several minutes to complete. Check progress with get_task_status(task_id).

Args: source_hostname: Source server hostname (e.g., "www.facebase.org"). source_catalog_id: ID of the catalog to clone. root_rid: The starting RID from which to trace reachability (e.g., "3-HXMC"). dest_hostname: Destination hostname. If None, uses source hostname. alias: Optional alias name for the new catalog. add_ml_schema: If True, add the DerivaML schema to the clone. asset_mode: How to handle assets: "none", "refs" (default), or "full". copy_annotations: If True (default), copy all annotations. copy_policy: If True (default), copy ACL policies. exclude_schemas: Schemas to exclude from cloning. exclude_objects: Tables ("schema:table") to exclude. reinitialize_dataset_versions: If True, reinitialize dataset versions. orphan_strategy: How to handle orphans: "fail", "delete", "nullify". prune_hidden_fkeys: Skip FKs with hidden reference data. truncate_oversized: Truncate values exceeding index limits. include_tables: Additional tables to include. include_associations: Include association tables. include_vocabularies: Include vocabulary tables. table_concurrency: Max concurrent table copies during fill phase. Lower values reduce server load. Default: 2.

Returns: JSON with task_id and status. Use get_task_status(task_id) to check progress.

Example: clone_catalog_async("www.facebase.org", "1", root_rid="3-HXMC", dest_hostname="localhost", alias="facebase-clone", orphan_strategy="delete") -> {"task_id": "abc123", "status": "started", ...}

# Check progress
get_task_status("abc123")
-> {"status": "running", "progress": {"percent_complete": 45.0, ...}}

# When done
get_task_status("abc123")
-> {"status": "completed", "result": {...}}

Get the status and progress of a background task.

Args: task_id: The task ID returned by an async operation. include_result: If True, include the full result when completed.

Returns: JSON with task status, progress, and optionally the result.

Example: get_task_status("abc123") -> { "task_id": "abc123", "status": "running", "progress": { "current_step": "Copying data", "percent_complete": 45.0, "message": "Copying table Subject..." } }

List all background tasks for the current user.

Args: status: Filter by status: "pending", "running", "completed", "failed", "cancelled". task_type: Filter by type: "clone_catalog".

Returns: JSON list of tasks with their status and basic info.

Example: list_tasks(status="running") -> [{"task_id": "abc123", "status": "running", ...}]

Cancel a pending or running background task.

Args: task_id: The task ID to cancel.

Returns: JSON with cancellation status.

Note: Cancellation is best-effort. Long-running operations may not stop immediately.

Connect to an existing DerivaML catalog. Must be called before using other tools.

On connection, an MCP workflow and execution are automatically created to track all operations performed through the MCP server. The workflow type "DerivaML MCP" is created if it doesn't exist.

Args: hostname: Server hostname (e.g., "dev.eye-ai.org", "www.atlas-d2k.org"). catalog_id: Catalog ID number (e.g., "1", "52"). domain_schema: Schema name for domain tables. Auto-detected if omitted. default_schema: Default schema for table creation and lookups. If omitted and there is exactly one domain schema, that schema is used. Required when multiple domain schemas exist and you want to avoid specifying the schema on every operation.

Returns: JSON with status, hostname, catalog_id, domain_schemas, default_schema, project_name, workflow_rid, execution_rid.

Example: connect_catalog("dev.eye-ai.org", "52") -> connects to eye-ai catalog connect_catalog("localhost", "10", domain_schema="isa", default_schema="isa")

Disconnect from the currently active catalog.

Switch the active catalog when multiple catalogs are connected.

Args: hostname: Server hostname of the catalog to activate. catalog_id: Catalog ID to activate.

Set the default schema for the active catalog connection.

When a catalog has multiple domain schemas, many operations require knowing which schema to use. Setting a default schema avoids having to specify it on every call.

Args: schema_name: Name of the domain schema to set as default (e.g., "isa", "my_project"). Must be one of the catalog's domain schemas.

Returns: JSON with status, default_schema, and domain_schemas.

Example: set_default_schema("isa") -> sets "isa" as the default schema

Create a new DerivaML catalog with all ML schema tables.

Creates a fresh catalog with Dataset, Execution, Workflow, Feature, and vocabulary tables. Automatically connects to the new catalog.

Args: hostname: Server hostname (e.g., "localhost", "deriva.example.org"). project_name: Name for the project, becomes the domain schema name. catalog_alias: Optional alias for the catalog. If provided, creates an alias that allows accessing the catalog by name instead of numeric ID (e.g., /ermrest/catalog/my-project instead of /ermrest/catalog/45).

Returns: JSON with status, hostname, catalog_id, catalog_alias (if created), domain_schema, project_name.

Example: create_catalog("localhost", "my_ml_project", "my-project")

PERMANENTLY DELETE a catalog and all its data. Cannot be undone.

Args: hostname: Server hostname. catalog_id: ID of the catalog to delete.

Returns: JSON with deletion status.

Apply catalog-level annotations to initialize the Chaise web interface.

Chaise is Deriva's web-based data browser. This method sets up annotations that control how Chaise displays and organizes the catalog, including the navigation bar and display settings.

Navigation Bar Structure: Creates a navigation bar with organized dropdown menus:

• User Info: Users, Groups, and RID Lease tables

• Deriva-ML: Core ML tables (Workflow, Execution, Dataset, Dataset_Version, etc.)

• WWW: Web content tables (Page, File)

• {Domain Schema}: All domain-specific tables (excludes vocabularies/associations)

• Vocabulary: All controlled vocabulary tables from ML and domain schemas

• Assets: All asset tables from ML and domain schemas

• Catalog Registry: Link to ermrest registry

• Documentation: Links to ML docs and instructions

Display Settings:

• Underscores in names displayed as spaces

• System columns (RID) shown in views

• Default landing page set to Dataset table

• Faceted search and record deletion enabled

Bulk Upload: Configures drag-and-drop file upload for asset tables.

When to call: After creating the domain schema and all tables. The menus are dynamically built from the current schema structure.

Args: navbar_brand_text: Text in the navigation bar brand area (default: "ML Data Browser"). head_title: Browser tab title (default: "Catalog ML").

Returns: JSON with status and applied settings.

Example workflow: 1. create_catalog("localhost", "my_project") 2. create_vocabulary("Species", "Types of species") 3. create_asset("Image", ...) 4. apply_catalog_annotations("My ML Project", "ML Catalog")

Create an alias for an existing catalog.

Aliases allow accessing a catalog by a memorable name instead of its numeric ID. For example, instead of /ermrest/catalog/21, you can use /ermrest/catalog/eye-ai.

Args: hostname: Server hostname (e.g., "www.eye-ai.org"). alias_name: The alias identifier (e.g., "my-project", "eye-ai"). Must be unique on the server. catalog_id: The numeric ID of the catalog to alias. name: Optional display name for the alias. description: Optional description of the alias.

Returns: JSON with status and alias details.

Example: create_catalog_alias("localhost", "my-project", "45", "My ML Project") -> {"status": "created", "alias": "my-project", "target": "45"}

Update an existing catalog alias.

Can change the target catalog the alias points to, or update the owner ACL.

Args: hostname: Server hostname (e.g., "www.eye-ai.org"). alias_name: The alias identifier to update. alias_target: New target catalog ID. If None, target is unchanged. Pass empty string "" to unbind the alias from any catalog. owner: New owner ACL (list of user/group identifiers). If None, owner is unchanged.

Returns: JSON with status and updated alias details.

Example: update_catalog_alias("localhost", "my-project", alias_target="50") -> {"status": "updated", "alias": "my-project", "target": "50"}

Delete a catalog alias. The target catalog is NOT deleted.

Args: hostname: Server hostname (e.g., "www.eye-ai.org"). alias_name: The alias identifier to delete.

Returns: JSON with deletion status.

Example: delete_catalog_alias("localhost", "my-project") -> {"status": "deleted", "alias": "my-project"}

Create an ML workspace by cloning data reachable from a root RID.

Creates a partial catalog clone containing only data reachable from the root RID (e.g., a project, dataset, or experiment). Uses the root table's export annotation (if available) to determine which tables and paths to follow, then fills in any uncovered tables (vocabularies, associations).

Uses a three-stage approach:

1. Create schema WITHOUT foreign keys (only for included tables)

2. Copy data asynchronously (export paths + fill-in tables)

3. Apply foreign keys, handling violations based on orphan_strategy

Asset handling modes:

• "none": Don't copy assets (asset columns will be empty)

• "refs": Copy asset URLs only, files stay on source server (default)

• "full": Download and re-upload all assets (fully independent clone)

Orphan handling: When source catalog policies hide some data but not references to it, cloning can result in dangling foreign keys. The orphan_strategy controls how these are handled.

Args: source_hostname: Source server hostname (e.g., "www.facebase.org"). source_catalog_id: ID of the catalog to clone. root_rid: The starting RID from which to trace reachability (e.g., a project RID like "3-HXMC"). dest_hostname: Destination hostname. If None, uses source hostname. alias: Optional alias name for the new catalog. add_ml_schema: If True, add the DerivaML schema to the clone. asset_mode: How to handle assets: "none", "refs" (default), or "full". copy_annotations: If True (default), copy all annotations. copy_policy: If True (default), copy ACL policies. exclude_schemas: List of schema names to exclude from cloning. exclude_objects: List of tables ("schema:table" format) to exclude. reinitialize_dataset_versions: If True (default), reinitialize dataset versions. orphan_strategy: How to handle orphan rows: "fail", "delete", or "nullify". prune_hidden_fkeys: If True, skip FKs with hidden reference data. truncate_oversized: If True, truncate values exceeding index size limits. include_tables: Additional tables to include. include_associations: If True, auto-include association tables. include_vocabularies: If True, auto-include vocabulary tables. table_concurrency: Max concurrent table copies during fill phase. Lower values reduce server load. Default: 1.

Returns: JSON with status, source info, destination info, and operation details including tables restored and orphan handling stats.

Examples: clone_catalog("www.facebase.org", "1", root_rid="3-HXMC", dest_hostname="localhost", alias="facebase-musmorph", add_ml_schema=True, orphan_strategy="delete")

Validate that RIDs exist in the catalog before running experiments.

Performs batch validation of RIDs to catch configuration errors early with clear error messages. Use this before running experiments to ensure all referenced datasets, assets, and other entities actually exist.

Args: dataset_rids: List of dataset RIDs to validate. asset_rids: List of asset RIDs to validate (model weights, etc.). dataset_versions: Dictionary mapping dataset RID to required version string (e.g., {"1-ABC": "0.4.0"}). Validates version exists. workflow_rids: List of workflow RIDs to validate. execution_rids: List of execution RIDs to validate. warn_missing_descriptions: If True (default), include warnings for datasets missing descriptions.

Returns: JSON with: - is_valid: True if all validations passed - errors: List of error messages - warnings: List of warning messages - validated_rids: Dictionary of validated RID info

Example: validate_rids( dataset_rids=["1-ABC", "2-DEF"], dataset_versions={"1-ABC": "0.4.0"}, asset_rids=["3-GHI"] ) -> { "is_valid": true, "errors": [], "warnings": [], "validated_rids": {...} }

Generate a citation URL for a catalog entity.

Creates a permanent, citable URL for any catalog entity (dataset, execution, asset, etc.). By default, includes a snapshot timestamp for reproducibility. Use current=True for a link to the live data.

Args: rid: RID of the entity to cite (e.g., "1-ABC"). current: If True, return URL to current state without snapshot. If False (default), return permanent URL with snapshot timestamp.

Returns: JSON with: - url: The citation URL - rid: The entity RID - is_snapshot: Whether URL includes snapshot timestamp

Examples: cite("1-ABC") -> {"url": "https://host/id/catalog/1-ABC@2024-01-15", "is_snapshot": true}

cite("1-ABC", current=True)
-> {"url": "https://host/id/catalog/1-ABC", "is_snapshot": false}

List all catalogs and aliases available on a Deriva server.

Queries the server's ermrest registry to discover available catalogs and their aliases. Use this to find catalogs before connecting.

Note: Returns all non-deleted catalogs and aliases. Typically bounded (servers have 10-50 catalogs), but could be larger on shared servers.

Args: hostname: Server hostname (e.g., "www.eye-ai.org", "dev.facebase.org").

Returns: JSON with: - hostname: The server queried - catalogs: List of {id, name, description} for each catalog - aliases: List of {id, alias_target, name, description} for each alias

Example: list_catalog_registry("www.eye-ai.org") -> {"hostname": "www.eye-ai.org", "catalogs": [...], "aliases": [...]}

Create a new empty dataset within an execution context.

The dataset is created through an execution for proper provenance tracking. Use add_dataset_members() to populate it after creation.

Assign Dataset_Type labels to categorize the dataset's role (e.g., "Training", "Testing", "Validation").

Args: description: Human-readable description of the dataset's purpose. dataset_types: Type labels from Dataset_Type vocabulary (e.g., ["Training", "Image"]). version: Initial version string (default: "0.1.0").

Returns: JSON with status, rid, description, dataset_types, version, execution_rid.

Example: create_dataset("Training images for model v2", ["Training"])

Generate a DatasetSpecConfig string for use in Python configuration files.

Returns the exact Python code to use in hydra-zen config files. This ensures the RID and version are correctly formatted and match what's in the catalog.

IMPORTANT: Always prefer specifying explicit versions in configurations. Using current_version as a default can lead to unexpected changes in results if the dataset is modified after the configuration is written. Pin to a specific version for reproducibility.

Args: dataset_rid: The RID of the dataset (e.g., "28CT"). version: Specific version to use. If not provided, uses the dataset's current version (with a warning about reproducibility).

Returns: JSON with the Python code string and metadata including: - spec: The DatasetSpecConfig(...) string ready to paste into code - rid: The dataset RID - version: The version used - description: Dataset description for reference - warning: Present if using current_version (recommends explicit version)

Example: get_dataset_spec("28CT") -> {"spec": "DatasetSpecConfig(rid="28CT", version="0.21.0")", ...}

get_dataset_spec("28CT", "0.20.0")
-> {"spec": "DatasetSpecConfig(rid=\"28CT\", version=\"0.20.0\")", ...}

Add records as dataset elements. Auto-increments minor version.

Records must be from tables registered as dataset element types. Use add_dataset_element_type() to register a table, or list_dataset_element_types() to see which tables are already registered.

Accepts members in two forms:

List of RIDs (member_rids): Each RID is auto-resolved to its table. Simpler but slower for large numbers.

Dict by table name (members_by_table): Maps table names to RID lists. Faster (skips RID resolution) and lets you add members of different types in one call. Recommended when you know the table names.

Exactly one of member_rids or members_by_table must be provided.

Args: dataset_rid: The RID of the dataset to add members to. member_rids: List of RIDs to add (e.g., ["2-ABC", "2-DEF"]). Auto-resolves each RID to its table. members_by_table: Dict mapping table names to RID lists (e.g., {"Subject": ["2-ABC"], "Observation": ["2-DEF", "2-GHI"]}). Faster than member_rids for large datasets. description: Optional description for the version increment that records why these members were added. Stored in the dataset history.

Returns: JSON with status, added_count, dataset_rid.

Example: add_dataset_members("1-ABC", member_rids=["2-DEF", "2-GHI"]) add_dataset_members("1-ABC", members_by_table={"Subject": ["2-DEF"], "Image": ["2-GHI"]})

Remove records from a dataset. Auto-increments minor version.

Removes the specified records from the dataset's membership. The records themselves are not deleted from the catalog, only their association with this dataset is removed.

Removing members automatically increments the dataset's minor version for change tracking.

Args: dataset_rid: The RID of the dataset to remove members from. member_rids: List of RIDs to remove (e.g., ["2-ABC", "2-DEF", "2-GHI"]).

Returns: JSON with status, removed_count, dataset_rid.

Example: delete_dataset_members("1-ABC", ["2-DEF", "2-GHI"]) -> removes 2 records from dataset

Manually increment a dataset's semantic version (major.minor.patch).

Description Handling (follows generate-descriptions prompt guidelines):

1. If user provides description: Use it, potentially improving for clarity

2. If description is empty: Generate from conversation context:

   • What catalog operations were performed since last version?

   • What was the user's stated goal for these changes?

   • Summarize the changes made (e.g., "Added X, fixed Y, modified Z")

Description Generation Guidelines:

• Include WHAT changed (added images, fixed labels, new features)

• Include WHY if known (QA review, batch import, schema update)

• Include IMPACT if relevant (affects N records, breaking change)

• Use markdown for complex descriptions (lists, tables)

Use this tool when:

• You've modified catalog data and want changes visible in a dataset

• You need to capture a snapshot of the current catalog state

• You want to create a reproducible checkpoint before making changes

Args: dataset_rid: The RID of the dataset. description: What changed in this version. If empty, LLM should generate from context. Good descriptions include: - What was added, modified, or fixed - Why the change was made (if known) - Impact on users of this dataset

    Examples of good descriptions:
    - "Added 500 new labeled training images from batch 3"
    - "Fixed incorrect labels on 23 images identified in QA review"
    - "Schema change: added 'quality_score' column to Image table"
    - "Captured snapshot before label correction workflow"

component: Which part to increment: "major", "minor", or "patch".
    - major: Breaking changes or schema modifications
    - minor: New data added or non-breaking changes (default)
    - patch: Bug fixes or label corrections

Returns: JSON with status, new_version, previous_version, dataset_rid, description.

Examples: increment_dataset_version("1-ABC", "Added quality labels to all images", "minor") increment_dataset_version("1-ABC", "Fixed mislabeled cat images", "patch") increment_dataset_version("1-ABC", "Schema change: new metadata columns", "major")

Soft-delete a dataset (marks deleted but preserves data).

Soft deletion hides the dataset from normal queries but keeps all data intact. For nested datasets, use recurse=True to also delete child datasets.

Args: dataset_rid: The RID of the dataset to delete. recurse: If True, also delete all nested child datasets.

Returns: JSON with status, dataset_rid, recursive.

Set or update the description for a dataset.

Updates the dataset's description in the catalog. Good descriptions help users understand the dataset's purpose, contents, and intended use.

Args: dataset_rid: RID of the dataset to update. description: New description text.

Returns: JSON with status, dataset_rid, description.

Example: set_dataset_description("1-ABC", "Training images for CIFAR-10 classification")

Add a type to a dataset.

Adds a Dataset_Type vocabulary term to this dataset. The type must exist in the Dataset_Type vocabulary.

Args: dataset_rid: RID of the dataset. dataset_type: Name of the type to add (must exist in Dataset_Type vocabulary).

Returns: JSON with status, dataset_rid, dataset_types list.

Example: add_dataset_type("1-ABC", "Training") -> adds "Training" type to dataset

Remove a type from a dataset.

Removes a Dataset_Type vocabulary term from this dataset. The type must exist in the Dataset_Type vocabulary.

Args: dataset_rid: RID of the dataset. dataset_type: Name of the type to remove.

Returns: JSON with status, dataset_rid, dataset_types list.

Example: remove_dataset_type("1-ABC", "Training") -> removes "Training" type from dataset

Register a domain table as a dataset element type.

After registration, records from this table can be added to datasets using add_dataset_members(). Creates an association table to link records to datasets.

Args: table_name: Name of the domain table to register (e.g., "Subject", "Image").

Returns: JSON with status, table_name, association_table.

Example: add_dataset_element_type("Subject") -> enables Subject records as dataset elements

Add a dataset as a nested child of another dataset.

Creates a parent-child relationship between datasets. Common pattern: a "Complete" parent dataset contains "Training" and "Testing" children that partition the same data.

Args: parent_rid: RID of the parent dataset. child_rid: RID of the child dataset to nest.

Returns: JSON with status, parent_rid, child_rid.

Example: add_dataset_child("1-ABC", "1-DEF") -> nests 1-DEF inside 1-ABC

List all parent datasets that contain this dataset as a child.

Args: dataset_rid: RID of the child dataset. recurse: If True, recursively list all ancestors (parents of parents). version: Semantic version to query (e.g., "1.0.0"). If not specified, uses the current version.

Returns: JSON array of parent datasets with {rid, description, dataset_types, current_version}.

Example: list_dataset_parents("1-ABC") -> direct parents only list_dataset_parents("1-ABC", recurse=True) -> all ancestors

Estimate the size of a dataset bag before downloading.

Runs the same FK path traversal as a dataset bag download, then queries the snapshot catalog for row counts and asset file sizes. Use this to preview what a download will contain and how large it will be before committing to the full download.

Args: dataset_rid: RID of the dataset to estimate. version: Semantic version to estimate (e.g., "1.0.0"). exclude_tables: Optional list of table names to exclude from FK path traversal during bag export.

Returns: JSON with: - tables: dict of table name -> {row_count, is_asset, asset_bytes} - total_rows: total row count across all tables - total_asset_bytes: total asset size in bytes - total_asset_size: human-readable size (e.g., "1.2 GB")

Get comprehensive info about a dataset bag: size, contents, and cache status.

Combines the size estimate (row counts, asset sizes per table) with local cache status. Use this to decide whether to cache a bag before running an experiment.

Cache status values:

• "not_cached": No local copy exists

• "cached_metadata_only": Table data downloaded, assets not fetched

• "cached_materialized": Fully downloaded and validated

• "cached_incomplete": Was cached but some assets are missing

Args: dataset_rid: RID of the dataset to inspect. version: Semantic version to inspect (e.g., "1.0.0"). exclude_tables: Optional list of table names to exclude from FK path traversal.

Returns: JSON with size info (tables, total_rows, total_asset_bytes, total_asset_size) plus cache_status and cache_path.

Download a dataset bag or asset into the local cache without creating an execution.

Use this to warm the cache before running experiments. No execution or provenance records are created — this is purely a local download operation. After caching, subsequent download_dataset or download_execution_dataset calls will use the cached copy.

Provide either dataset_rid (for bags) or asset_rid (for individual assets), not both.

Args: dataset_rid: RID of a dataset to cache (mutually exclusive with asset_rid). asset_rid: RID of an asset to cache (mutually exclusive with dataset_rid). version: Dataset version to cache (required when using dataset_rid). materialize: If True (default), download all asset files in the bag. If False, download only table metadata (faster, smaller). Ignored for asset cache. exclude_tables: Optional list of table names to exclude from FK path traversal during bag export. Only applies to dataset cache.

Returns: JSON with cache results. For datasets: bag_info including cache_status and size. For assets: file path and metadata.

Preview a denormalized (wide table) view of dataset tables.

Joins related dataset tables into a single wide table. Returns schema shape (columns, join path) and size estimates. Optionally returns actual row data when a dataset and limit are provided.

Modes:

• No dataset_rid: Returns schema shape + global size estimates. Use this to explore what a denormalized join would look like.

• With dataset_rid, limit=0: Returns shape + dataset-scoped estimates.

• With dataset_rid, limit>0: Returns shape + estimates + row preview.

Tables are joined based on their foreign key relationships. Column names are prefixed with the source table name using dots (e.g., "Image.Filename", "Subject.RID"). Intermediate tables needed for the join are auto-discovered.

Args: include_tables: List of table names to include in the join. Tables are joined based on their foreign key relationships. Order doesn't matter - the join order is determined automatically. Add more tables iteratively to expand the denormalized view. dataset_rid: RID of the dataset to preview. If omitted, returns schema shape with global (catalog-wide) row counts. version: Semantic version to query (e.g., "1.0.0"). If not specified, uses the current version. Only used with dataset_rid. limit: Maximum rows to return (default: 0, max: 100). Only used with dataset_rid. Set to 0 for shape and estimates only.

Returns: JSON with columns, join_path, tables (per-table size info), total_rows, total_asset_bytes, total_asset_size. When limit > 0 with a dataset_rid, also includes rows and count.

Example: # Explore schema shape (no dataset needed) preview_denormalized_dataset(["Subject", "Report_HVF"]) -> {"columns": [...], "join_path": ["Report_HVF", "Observation", "Subject"], ...}

# Get dataset-scoped estimates
preview_denormalized_dataset(["Image", "Subject"], dataset_rid="1-ABC")
-> {"columns": [...], "tables": {"Image": {"row_count": 50}}, ...}

# Preview actual rows
preview_denormalized_dataset(["Image", "Subject"], dataset_rid="1-ABC", limit=10)
-> {"columns": [...], "tables": {...}, "rows": [...], "count": 10}

Create a new dataset type term in the Dataset_Type vocabulary.

This creates a new vocabulary term that can then be assigned to datasets using add_dataset_type(). Dataset types help categorize datasets by their role in ML workflows.

Common types include "Training", "Testing", "Validation", "Complete".

Args: type_name: Name for the dataset type (must be unique). description: What this type of dataset is used for. synonyms: Alternative names that can match this type (e.g., ["train"] for "Training").

Returns: JSON with status, name, description, synonyms, rid.

Example: create_dataset_type_term("Validation", "Held-out data for hyperparameter tuning", ["val", "valid"])

Delete a dataset type term from the Dataset_Type vocabulary.

WARNING: Only delete types that are not referenced by any datasets. If datasets use this type, the delete will fail with a foreign key error. Use remove_dataset_type() first to remove the type from all datasets.

Args: type_name: Name of the dataset type to delete.

Returns: JSON with status and deleted type name.

Example: delete_dataset_type_term("Obsolete") -> {"status": "deleted", "name": "Obsolete"}

Split a dataset into training, testing, and optionally validation subsets.

Creates a new dataset hierarchy with full provenance tracking:

• Split (parent, type: "Split")

  • Training (child, type: "Training" + training_types)

  • Validation (child, type: "Validation" + validation_types) # if val_size

  • Testing (child, type: "Testing" + testing_types)

The API follows scikit-learn's train_test_split conventions for test_size, train_size, val_size, shuffle, and seed parameters.

Splitting strategies:

• Random (default): Shuffles members and splits at the boundary. No denormalization needed. Fast for any dataset size.

• Stratified: Maintains class distribution across splits. Requires stratify_by_column and include_tables. Uses scikit-learn internally.

Column naming for stratification:

When using stratify_by_column, the column name must match the denormalized DataFrame format: {TableName}_{ColumnName}. For example, to stratify by the Image_Class column from the Image_Classification feature table, use Image_Classification_Image_Class.

Derive the column name from the table schema (via the deriva://catalog/schema or deriva://catalog/features resource) rather than calling denormalize_dataset().

Args: source_dataset_rid: RID of the source dataset to split. test_size: Test set size as a fraction (0-1) or absolute count. Default: 0.2 (20% of data). train_size: Train set size as a fraction (0-1) or absolute count. Default: None (complement of test_size and val_size). val_size: Validation set size as a fraction (0-1) or absolute count. Default: None (no validation split, two-way only). When provided, creates a three-way train/val/test split. seed: Random seed for reproducibility. Default: 42. shuffle: Whether to shuffle before splitting. Default: True. stratify_by_column: Column name in the denormalized DataFrame for stratified splitting. Maintains class distribution across all partitions. Requires include_tables. Example: "Image_Classification_Image_Class". stratify_missing: Policy for null values in the stratify column. "error" (default): raise if any nulls exist, reporting count and percentage. "drop": exclude rows with null values from the split. "include": treat nulls as a separate class. Only used when stratify_by_column is set. element_table: Element table to split (e.g., "Image"). If not specified, auto-detected from the dataset's members. include_tables: Tables to include when denormalizing. Required when using stratify_by_column. Example: ["Image", "Image_Classification"]. training_types: Additional dataset types for the training set beyond "Training". Example: ["Labeled"]. testing_types: Additional dataset types for the testing set beyond "Testing". Example: ["Labeled"]. validation_types: Additional dataset types for the validation set beyond "Validation". Example: ["Labeled"]. Ignored when val_size is None. split_description: Description for the parent Split dataset. dry_run: If True, return what would happen without modifying the catalog. Useful for previewing split sizes.

Returns: JSON with split results including: - split: RID, version, and count of the parent Split dataset - training: RID, version, and count of the Training dataset - validation: RID, version, and count of the Validation dataset (if val_size) - testing: RID, version, and count of the Testing dataset - source: RID of the source dataset

Example: # Random 80/20 split split_dataset("28D0", test_size=0.2, seed=42)

# Three-way train/val/test split
split_dataset("28D0", test_size=0.2, val_size=0.1, seed=42)

# Stratified split maintaining class balance
split_dataset("28D0", test_size=0.2,
             stratify_by_column="Image_Classification_Image_Class",
             include_tables=["Image", "Image_Classification"])

# Fixed-count split with labeled types
split_dataset("28D0", train_size=400, test_size=100,
             training_types=["Labeled"], testing_types=["Labeled"])

# Dry run to preview
split_dataset("28D0", test_size=0.2, dry_run=True)

Add a new term to a vocabulary.

Args: vocabulary_name: Name of the vocabulary table (e.g., "Dataset_Type"). term_name: Primary name for the term (must be unique). description: What this term means. synonyms: Alternative names that can also match this term.

Returns: JSON with status, name, description, synonyms, rid.

Example: add_term("Dataset_Type", "Validation", "Held-out data for validation", ["val", "valid"])

Create a new vocabulary table for storing controlled terms.

Args: vocabulary_name: Name for the new vocabulary table. comment: Description of the vocabulary's purpose. schema: Schema to create in (default: domain schema).

Returns: JSON with status, name, schema, comment.

Example: create_vocabulary("Quality_Level", "Image quality ratings")

Add a synonym to an existing vocabulary term.

Synonyms are alternative names that can be used to look up a term.

Args: vocabulary_name: Name of the vocabulary table (e.g., "Dataset_Type"). term_name: Primary name of the term to add synonym to. synonym: Alternative name to add.

Returns: JSON with status, name, synonyms list.

Example: add_synonym("Dataset_Type", "Training", "train") -> adds "train" as synonym

Remove a synonym from an existing vocabulary term.

Args: vocabulary_name: Name of the vocabulary table (e.g., "Dataset_Type"). term_name: Primary name of the term to remove synonym from. synonym: Alternative name to remove.

Returns: JSON with status, name, updated synonyms list.

Example: remove_synonym("Dataset_Type", "Training", "train") -> removes "train" as synonym

Update the description of a vocabulary term.

Args: vocabulary_name: Name of the vocabulary table (e.g., "Dataset_Type"). term_name: Primary name of the term to update. description: New description for the term.

Returns: JSON with status, name, updated description.

Example: update_term_description("Dataset_Type", "Training", "Data used to train models")

Delete a term from a vocabulary.

The term must not be in use by any records in the catalog. If the term is referenced by other records (e.g., datasets using this type), the delete will fail with an error listing how many records reference it.

Args: vocabulary_name: Name of the vocabulary table (e.g., "Dataset_Type"). term_name: Name of the term to delete.

Returns: JSON with status, vocabulary, and deleted term name.

Example: delete_term("Dataset_Type", "Obsolete") -> {"status": "deleted", ...}

Find a workflow by its source URL.

Search for a workflow that was registered with the given source URL. Use this to check if a workflow for a specific script or notebook already exists before creating a new one.

Args: url: The source URL to search for (e.g., GitHub URL to script).

Returns: JSON with: - found: True if workflow exists, False otherwise - workflow: Full workflow details if found

Example: lookup_workflow_by_url("https://github.com/org/repo/blob/main/train.py")

Create and register a new workflow definition.

Args: name: Display name for the workflow. workflow_type: Type from Workflow_Type vocabulary (e.g., "Training", "Inference"). description: What this workflow does.

Returns: JSON with status, rid, name, workflow_type, description.

Example: create_workflow("ResNet Training", "Training", "Trains ResNet50 on image data")

Set or update the description for a workflow.

Updates the workflow's description in the catalog. Good descriptions help users understand what the workflow does and how to use it.

Args: workflow_rid: RID of the workflow to update. description: New description text.

Returns: JSON with status, workflow_rid, description.

Example: set_workflow_description("3-WKF", "Trains CNN on image data with augmentation")

Add a new workflow type to the Workflow_Type vocabulary.

Args: type_name: Name for the new workflow type. description: What this type of workflow does.

Returns: JSON with status, name, description, rid.

Example: add_workflow_type("Data Augmentation", "Workflows that augment training data")

Create a new feature definition to associate metadata with domain objects.

Features enable ML data engineering by linking labels, scores, or derived assets to domain objects. The feature definition specifies what types of values are valid.

What this creates:

1. A new association table in the domain schema to store feature values

2. A dynamically generated Pydantic model class for creating validated feature instances

The Pydantic model class (accessible via feature_record_class() in Python) provides type-safe construction of feature records with automatic validation against the feature's definition.

Feature types:

• Term-based: Values come from controlled vocabularies (e.g., diagnosis labels)

• Asset-based: Values reference asset files (e.g., segmentation masks)

• Mixed: Can reference both terms and assets

The feature automatically tracks which Execution produced each value for provenance.

Args: table_name: Table to attach the feature to (e.g., "Image", "Subject"). feature_name: Unique name for the feature (e.g., "Diagnosis", "Quality_Score"). comment: Description of what this feature represents. terms: Vocabulary table names whose terms can be values (e.g., ["Diagnosis_Type"]). assets: Asset table names that can be referenced (e.g., ["Segmentation_Mask"]). metadata: Additional columns or table references to include in the feature. Each item can be: - A string: Treated as a table name (adds a foreign key reference) - A dict: Column definition with at minimum "name" and "type" keys. The "type" value should be a dict like {"typename": "float4"}. Valid type names: text, int2, int4, int8, float4, float8, boolean, date, timestamp, timestamptz, json, jsonb. Optional keys: "nullok" (bool), "default", "comment".

Returns: JSON with status, feature_name, target_table.

Examples: # Simple term-based feature create_feature("Image", "Diagnosis", "Clinical diagnosis label", terms=["Diagnosis_Type"])

# Feature with a confidence score column
create_feature("Image", "Diagnosis", "Diagnosis with confidence",
    terms=["Diagnosis_Type"],
    metadata=[{"name": "confidence", "type": {"typename": "float4"}}])

# Feature referencing another table
create_feature("Image", "Review", "Review annotations",
    terms=["Review_Status"],
    metadata=["Reviewer"])

Delete a feature definition and all its values. Cannot be undone.

WARNING: This permanently removes the feature table and all associated values. All provenance information for this feature will be lost.

Args: table_name: Table the feature is attached to. feature_name: Name of the feature to delete.

Returns: JSON with status, feature_name, table_name.

Add feature values to one or more domain objects.

Associates feature values (terms, assets, or other) with target records. Accepts a list of entries, each mapping a target RID to a value. All entries are inserted in a single batch for efficiency.

If an execution is active, it will be used for provenance. Otherwise, provide an execution_rid explicitly.

For simple features (single term or asset column): Use this tool — each entry needs only target_rid and value.

For complex features (multiple columns per record): Use add_feature_value_record instead, which accepts arbitrary field dicts.

Args: table_name: Table the target records belong to (e.g., "Image"). feature_name: Name of the feature (e.g., "Diagnosis"). entries: List of dicts, each with: - target_rid (str): RID of the target record to annotate. - value (str): The feature value — a term name or asset RID. execution_rid: Execution RID for provenance (uses active if not provided).

Returns: JSON with status, feature_name, count, execution_rid, rids.

Examples: # Single value add_feature_value("Image", "Diagnosis", [{"target_rid": "1-ABC", "value": "Normal"}])

# Batch values
add_feature_value("Image", "Diagnosis", [
    {"target_rid": "1-ABC", "value": "Normal"},
    {"target_rid": "1-DEF", "value": "Abnormal"},
    {"target_rid": "1-GHI", "value": "Normal"},
])

Add feature values with multiple fields to one or more domain objects.

For features with multiple columns (e.g., a diagnosis with confidence score), use this tool to provide values for each field. Accepts a list of entries for batch insertion. Use lookup_feature first to see available fields.

Feature columns are dynamically generated based on the feature definition:

• term_columns: Accept vocabulary term names (strings)

• asset_columns: Accept asset RIDs (strings)

• value_columns: Accept direct values (strings, numbers)

Args: table_name: Table the target records belong to (e.g., "Image"). feature_name: Name of the feature (e.g., "Diagnosis"). entries: List of dicts, each with: - target_rid (str, required): RID of the target record. - Plus any feature column names mapped to their values. Use lookup_feature to see available columns and types. execution_rid: Execution RID for provenance (uses active if not provided).

Returns: JSON with status, feature_name, count, execution_rid, rids.

Example: # First check the feature structure: lookup_feature("Image", "Diagnosis") # -> {"term_columns": {"Diagnosis_Type": {...}}, "value_columns": {"confidence": {...}}}

# Then add values (single or batch):
add_feature_value_record("Image", "Diagnosis", [
    {"target_rid": "1-ABC", "Diagnosis_Type": "Normal", "confidence": 0.95},
    {"target_rid": "1-DEF", "Diagnosis_Type": "Abnormal", "confidence": 0.87},
])

Create a new table in the domain schema.

This tool creates a standard table (not an asset table). For tables that store files with automatic URL/checksum tracking, use create_asset_table instead.

Process Overview:

1. Define columns with names, types, and constraints

2. Optionally define foreign keys to reference other tables

3. The table is created in the domain schema

4. The navigation bar is automatically updated

Args: table_name: Name for the new table (e.g., "Subject", "Experiment", "Protocol"). columns: Column definitions, each dict with: - name (str, required): Column name - type (str): One of "text", "int2", "int4", "int8", "float4", "float8", "boolean", "date", "timestamp", "timestamptz", "json", "jsonb", "markdown" (default: "text") - nullok (bool): Allow null values (default: True) - comment (str): Column description foreign_keys: Foreign key definitions, each dict with: - column (str, required): Column name in this table (must also be in columns list) - referenced_table (str, required): Name of the table to reference - referenced_column (str): Column in referenced table (default: "RID") - on_delete (str): Action on delete - "NO ACTION", "CASCADE", "SET NULL" (default: "NO ACTION") comment: Description of the table's purpose. schema: Schema to create the table in. If not provided, uses the default domain schema. Useful when a catalog has multiple domain schemas.

Returns: JSON with status, table_name, schema, columns.

Examples: Simple table: create_table("Subject", [ {"name": "Name", "type": "text", "nullok": false}, {"name": "Age", "type": "int4"}, {"name": "Notes", "type": "markdown"} ])

Table with foreign key:
    create_table("Sample", [
        {"name": "Name", "type": "text", "nullok": false},
        {"name": "Subject", "type": "text", "nullok": false},
        {"name": "Collection_Date", "type": "date"}
    ], foreign_keys=[
        {"column": "Subject", "referenced_table": "Subject", "on_delete": "CASCADE"}
    ])

Create a new asset table for file management with automatic URL/checksum tracking.

Asset tables automatically include: URL, Filename, Length, MD5, Description. They integrate with executions for provenance tracking.

Args: asset_name: Name for the asset table (e.g., "Image", "Model", "Checkpoint"). columns: Additional columns beyond standard asset columns. referenced_tables: Tables this asset should have foreign keys to. comment: Description of the asset table's purpose. schema: Schema to create the table in. If not provided, uses the default domain schema.

Returns: JSON with status, table_name, schema, columns.

Example: create_asset_table("Image", [{"name": "Width", "type": "int4"}], ["Subject"])

List all executions associated with an asset.

Given an asset RID, returns a list of executions that created or used the asset, along with the role (Input/Output) in each execution. This is useful for provenance tracking - finding which execution created an asset or which executions used it as input.

Args: asset_rid: RID of the asset to look up. asset_role: Optional filter: "Input" or "Output". If omitted, returns all.

Returns: JSON array of execution records showing which executions are associated with this asset. Each record includes execution_rid, workflow_rid, status, and description.

Example: list_asset_executions("3JSE") -> finds all executions that created/used this asset list_asset_executions("3JSE", "Output") -> finds only the execution that created it

Add a new asset type to the Asset_Type vocabulary.

Args: type_name: Name for the asset type. description: What this asset type represents.

Returns: JSON with status, name, description, rid.

Example: add_asset_type("Segmentation Mask", "Binary mask images for segmentation")

Add an asset type to a specific asset.

Associates an asset with a type from the Asset_Type vocabulary. An asset can have multiple types.

Args: asset_rid: RID of the asset to modify. type_name: Name of the asset type to add (must exist in Asset_Type vocab).

Returns: JSON with status, asset_rid, and updated types list.

Example: add_asset_type_to_asset("3JSE", "Training_Data")

Remove an asset type from a specific asset.

Removes the association between an asset and a type.

Args: asset_rid: RID of the asset to modify. type_name: Name of the asset type to remove.

Returns: JSON with status, asset_rid, and updated types list.

Example: remove_asset_type_from_asset("3JSE", "Training_Data")

Set or update the description (comment) for a table.

Args: table_name: Name of the table to update. description: New description for the table.

Returns: JSON with status, table_name, description.

Example: set_table_description("Image", "Medical images for analysis")

Set the display name shown in the UI for a table.

This is a convenience tool for setting just the display name. For setting multiple display properties at once (name, markdown_name, etc.), use set_display_annotation instead.

Args: table_name: Name of the table to update. display_name: Human-readable name to display in the UI.

Returns: JSON with status, table_name, display_name.

Example: set_table_display_name("Image", "Medical Images")

Set the pattern used to display row names in the UI.

The pattern uses Handlebars syntax with triple braces for column values.

Args: table_name: Name of the table to update. pattern: Handlebars template (e.g., "{{{Name}}}" or "{{{FirstName}}} {{{LastName}}}").

Returns: JSON with status, table_name, pattern.

Examples: set_row_name_pattern("Subject", "{{{Name}}}") set_row_name_pattern("Image", "{{{Filename}}} ({{{RID}}})")

Add a new column to an existing table.

Args: table_name: Name of the table to modify. column_name: Name for the new column. column_type: Data type - one of "text", "int2", "int4", "int8", "float4", "float8", "boolean", "date", "timestamp", "timestamptz", "json", "jsonb", "markdown" (default: "text"). nullok: Whether NULL values are allowed (default: True). default: Default value for new rows (optional). comment: Description of the column (optional).

Returns: JSON with status, table_name, column_name, column_type.

Example: add_column("Subject", "Age", "int4", nullok=True, comment="Subject age in years")

Set or update the description (comment) for a column.

Args: table_name: Name of the table containing the column. column_name: Name of the column to update. description: New description for the column.

Returns: JSON with status, table_name, column_name, description.

Example: set_column_description("Subject", "Age", "Subject age in years at enrollment")

Set the display name shown in the UI for a column.

This is a convenience tool for setting just the display name. For setting multiple display properties at once (name, markdown_name, etc.), use set_display_annotation with a column_name parameter instead.

Args: table_name: Name of the table containing the column. column_name: Name of the column to update. display_name: Human-readable name to display in the UI.

Returns: JSON with status, table_name, column_name, display_name.

Example: set_column_display_name("Subject", "DOB", "Date of Birth")

Set whether a column allows NULL values.

Args: table_name: Name of the table containing the column. column_name: Name of the column to update. nullok: True to allow NULL values, False to require values.

Returns: JSON with status, table_name, column_name, nullok.

Note: Setting nullok=False will fail if the column contains NULL values.

Example: set_column_nullok("Subject", "Name", False) # Make Name required

Create a new execution to track an ML workflow run with provenance.

This is the first step in the execution lifecycle. Specify input datasets and assets to establish provenance - these will be recorded as inputs to this workflow run.

LIFECYCLE (follow in order):

1. create_execution() - You are here

2. start_execution() - Begin timing

3. [Run your ML workflow]

4. stop_execution() - End timing

Args: workflow_name: Descriptive name (e.g., "ResNet50 Training Run 3"). workflow_type: Type from Workflow_Type vocabulary (e.g., "Training", "Inference"). description: What this execution does and why. dataset_rids: Input dataset RIDs for provenance tracking. asset_rids: Input asset RIDs for provenance tracking. dry_run: If True, download input datasets/assets but skip creating execution records in the catalog and skip uploading results. Useful for testing data loading, configuration, and model initialization without writing to the catalog.

Returns: JSON with execution_rid, workflow_rid, dataset_count, asset_count, dry_run.

Example: create_execution("CIFAR Training", "Training", "Train ResNet on CIFAR-10", ["1-ABC"]) create_execution("Test Run", "Training", "Debug data loading", dry_run=True)

Start timing the active execution. Call after create_execution().

Records the start timestamp for duration tracking. The execution status changes to "running".

Stop timing and mark execution complete.

Records the stop timestamp and calculates duration. Call this after your ML workflow completes.

Update the execution status with a progress message.

Args: status: One of "pending", "running", "completed", "failed". message: Progress message or error description.

Returns: JSON with execution_rid, new_status, message.

Set or update the description for an execution.

Updates the execution's description in the catalog. Good descriptions help users understand what the execution accomplished and any notable results.

Args: execution_rid: RID of the execution to update. description: New description text.

Returns: JSON with status, execution_rid, description.

Example: set_execution_description("2-XYZ", "Training run with lr=0.001, achieved 95% accuracy")

Restore a previous execution to continue working with it.

Args: execution_rid: RID of the execution to restore (e.g., "1-ABC").

Returns: JSON with execution_rid, workflow_rid, dataset_count.

Create a new dataset as output from this execution.

Creates a dataset that is linked to this execution for provenance. Use this when your workflow produces a new curated collection of data (e.g., augmented training data, filtered results).

Args: description: What this dataset contains. dataset_types: Type labels (e.g., ["Training", "Augmented"]).

Returns: JSON with dataset_rid, execution_rid.

Add a child execution to a parent execution.

Creates a parent-child relationship between executions. Use this to group related executions, such as:

• Parameter sweeps (parent = sweep, children = individual runs)

• Pipelines (parent = pipeline, children = stages)

• Cross-validation (parent = CV experiment, children = folds)

Args: parent_execution_rid: RID of the parent execution. child_execution_rid: RID of the child execution to nest. sequence: Optional ordering index (0, 1, 2...). Use None for parallel executions.

Returns: JSON with parent_rid, child_rid, sequence.

Example: # Create a sweep parent, then add child executions add_nested_execution("1-PARENT", "1-CHILD1", sequence=0) add_nested_execution("1-PARENT", "1-CHILD2", sequence=1)

List all child (nested) executions of an execution.

Args: execution_rid: RID of the parent execution. recurse: If True, return all descendants (children, grandchildren, etc.).

Returns: JSON array of {execution_rid, workflow_rid, status, description} for each child.

Example: list_nested_executions("1-PARENT") # Direct children only list_nested_executions("1-PARENT", recurse=True) # All descendants

Preview records from a table with optional column selection and filtering.

Returns a sample of records for understanding data structure and content. For bulk data access, use the DerivaML Python API directly.

Args: table_name: Name of the table to preview (e.g., "Image", "Subject", "Dataset"). columns: List of column names to return. Default: all columns. filters: Dictionary of {column: value} equality filters. limit: Maximum records to return (default: 25, max: 100). offset: Number of records to skip.

Returns: JSON with records array, count, and table name.

Examples: preview_table("Image") -> first 25 images preview_table("Image", columns=["RID", "Filename"], limit=10) preview_table("Subject", filters={"Species": "Human"})

Insert new records into a domain table.

IMPORTANT: This tool is for domain-specific tables only (e.g., Subject, Image metadata). Do NOT use for:

• Datasets → use create_dataset(), add_dataset_members()

• Features → use add_feature_value()

• Vocabularies → use add_term()

• Executions → use create_execution()

• Workflows → use create_workflow()

• Assets with files → use the DerivaML Python API execution workflow

Args: table_name: Name of the domain table to insert into. records: List of dictionaries with column values.

Returns: JSON with inserted_count and record RIDs.

Example: insert_records("Subject", [{"Name": "Patient A", "Age": 45}])

Get a single record by its RID.

Args: table_name: Name of the table containing the record. rid: The RID of the record to fetch.

Returns: JSON with the complete record or error if not found.

Example: get_record("Image", "1-ABC") -> full image record

Update fields in an existing record.

Args: table_name: Name of the table containing the record. rid: The RID of the record to update. updates: Dictionary of {column: new_value} updates.

Returns: JSON with update status.

Example: update_record("Subject", "1-ABC", {"Age": 46, "Status": "Active"})

List all cached tabular query results.

Returns metadata for each cached result including the tool that produced it, parameters, row count, age, and cache key. Use the cache_key with query_cached_result to re-query with different sort/filter/pagination.

Returns: JSON with list of cached result entries.

Re-query a cached tabular result with different sort/filter/pagination.

Use list_cached_results to find available cache keys. This tool lets you paginate, sort, and filter previously computed results without re-executing the original query.

Args: cache_key: The cache key from a previous query result. sort_by: Column name to sort by (e.g., "Image.CDR"). sort_desc: Sort descending if True. filter_col: Column name to filter on. filter_val: Value to filter for (substring match, case-insensitive). limit: Maximum rows to return (default: 100). offset: Number of rows to skip for pagination.

Returns: JSON with columns, rows, count, and total_count.

Invalidate cached tabular query results.

Args: cache_key: Invalidate a specific cached result. source: Invalidate all results from this source ("bag" or "catalog"). If neither is provided, invalidates all cached results.

Returns: JSON with the number of entries invalidated.

Search Deriva documentation, catalog schema, and catalog data using semantic similarity.

Searches across three categories of indexed content:

1. Catalog schema (doc_type="catalog-schema"): The connected catalog's structure, including:

   • Tables with columns, types, nullability, and comments

   • Foreign key relationships between tables

   • Feature definitions with term columns, asset columns, and value columns

   • Vocabulary terms with names, descriptions, and synonyms Use this to understand what data exists, how tables relate, what features are defined, and what vocabulary values are available.

2. Catalog data (doc_type="catalog-data"): User-specific records from the connected catalog, including:

   • Datasets with types, versions, and descriptions

   • Executions with workflow names, status, and inputs Use this to find specific datasets by purpose or trace experiment provenance.

3. Documentation (doc_type="user-guide", "api-reference", "sdk-reference"): Indexed docs from Deriva ecosystem repositories (deriva-ml, ermrest, chaise, deriva-py). Use this for API usage questions and how-to guidance.

Prefer this tool over reading raw resources for catalog exploration. Use doc_type to focus results on the category you need.

Args: query: Natural language search query (e.g., "how to create a dataset") limit: Maximum number of results to return (default 10) source: Filter by source name (e.g., "deriva-ml-docs", "ermrest-docs") doc_type: Filter by document type. Key values: - "catalog-schema": Tables, columns, FKs, features, vocab terms - "catalog-data": Datasets and executions in the catalog - "user-guide": DerivaML and Chaise documentation - "api-reference": ERMrest API documentation - "sdk-reference": deriva-py SDK documentation include_schema: If True (default), include catalog schema results when connected. Set to False to search only documentation. include_data: If True (default), include per-user data index results (datasets, executions) when connected. Set to False to exclude user data from search results.

Returns: Dict with search results including text snippets, relevance scores, source metadata, and GitHub URLs.

Examples: # Explore catalog structure — tables, columns, relationships rag_search("Image tables and features", doc_type="catalog-schema")

# Find vocabulary terms by meaning
rag_search("classification categories", doc_type="catalog-schema")

# Find feature definitions and their columns
rag_search("diagnosis label confidence", doc_type="catalog-schema")

# Find datasets by description, type, or purpose
rag_search("training split labeled", doc_type="catalog-data")

# Find executions by workflow or status
rag_search("training experiment results", doc_type="catalog-data")

# Search Deriva API documentation
rag_search("how to create a dataset", include_schema=False, include_data=False)

# Search everything (docs + schema + data) — the default
rag_search("how are images classified")

Full crawl and index of documentation sources.

Crawls GitHub repositories, fetches all documentation files, chunks them, and indexes them for semantic search. This is a long-running operation that runs in the background.

Args: source_name: Specific source to ingest (e.g., "deriva-ml-docs"). If None, ingests all configured sources.

Returns: Dict with task ID for tracking progress, or immediate results if the operation completes quickly.

Incremental update of documentation index.

Checks for changed files in source repositories and only re-indexes files that have been added, modified, or deleted. Much faster than full ingestion when few files have changed.

Args: source_name: Specific source to update (e.g., "deriva-ml-docs"). If None, updates all configured sources.

Returns: Dict with task ID for tracking progress.

Get the status of the RAG documentation index.

Returns information about the index including total chunks, configured sources, and last update times.

Returns: Dict with index status, source configurations, and statistics.

Register a new documentation source for RAG indexing.

After adding a source, run rag_ingest(source_name=name) to index it.

Args: name: Unique name for this source (e.g., "my-project-docs") repo_owner: GitHub repository owner (e.g., "informatics-isi-edu") repo_name: GitHub repository name (e.g., "deriva-ml") branch: Git branch to index (default "main") path_prefix: Only index files under this path (default "docs/") include_patterns: File patterns to include (default ["*.md"]) doc_type: Document type tag for filtering (default "user-guide")

Returns: Dict confirming the source was added.

Remove a documentation source and its indexed chunks.

This deletes all indexed chunks for the source and removes it from the configuration.

Args: name: Name of the source to remove (e.g., "my-project-docs")

Returns: Dict with removal status and number of chunks deleted.