Runtime Copilot MCP Server
Provides tools for interacting with a Databricks-like toy platform, including versioning, partitions, DAGs, and event-driven workflows, for learning warehouse internals.
Provides tools for interacting with a PostgreSQL-like toy engine, including heap tables, B-tree indexing, query plan selection, and WAL/checkpoint operations, for learning storage and planner internals.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@Runtime Copilot MCP Server explain the latest regression failure"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Interactive Data Systems Lab
Runnable Python and Rust data-system internals with an MCP-native runtime copilot layer.
This repository combines two things:
a local-first engineering lab for PostgreSQL-like and Databricks-like internals,
a Runtime Copilot MCP surface for diagnostics, explainability, regression checks, and operational memory.
Core capabilities
Storage and planner internals: heap tables, B-tree indexing, selectivity, and plan choice.
Persistence and replay: WAL/checkpoint style flows and deterministic state transitions.
Workflow and write-path modeling: event-first architecture, idempotency, retry semantics.
Explainable runtime operations: traced runs, failure summaries, regression verdicts, baseline compare.
MCP access: machine-usable operational interface instead of ad hoc shell scripts.
Related MCP server: Alma Atlas
Who this is for
Data engineers learning warehouse and query-engine internals.
Platform and infrastructure engineers teaching storage and execution fundamentals.
Teams building onboarding labs, workshops, and demo environments.
Quick start
Requirements:
Python 3.10+ (tested with Python 3.14)
Setup:
python3 -m venv .venv
source .venv/bin/activate
python -m pip install "psycopg[binary]"Run end-to-end flow:
cargo run --bin e2e_flowRun core demos:
.venv/bin/python mini_pg_like.py
.venv/bin/python mini_databricks_clone.py
cargo run --bin mini_pg_like
cargo run --bin mini_databricks_cloneWhat this repository contains
mini_pg_like.py: PostgreSQL-like toy engine with heap table, B-tree index, and planner output.mini_databricks_clone.py: Databricks-like toy platform with versioning, partitions, DAGs, and events.src/bin/mini_pg_like.rs: Rust PostgreSQL-like demo.src/bin/mini_databricks_clone.rs: Rust Databricks-like demo.src/lib.rs,src/common.rs,src/pg.rs: shared Rust core modules.mcp_engine_server.py: MCP runtime adapter for diagnostics and regression workflows.
Why this exists
Most internals content stops at diagrams. This project stays runnable and inspectable:
compare Python and Rust implementations of the same system ideas,
trace write-path behavior with concrete events and state transitions,
run explainable regression checks through MCP,
turn runtime operations into a discoverable control surface for AI clients.
MCP Adapter Layer
This repo also includes a minimal MCP server that wraps the lab operations:
mcp_engine_server.pyCursor config:
.cursor/mcp.json
Current MCP tool list for this release (47 tools total):
Engine state and runtime:
init_engineinsert_rowupsert_rowcreate_indexexplain_customerreindex_projectrun_e2e_flow
Explainability and demos:
explain_rundemo_explain_rundemo_explain_run_failuredemo_explain_semantic_failuredemo_explain_idempotency_conflictdemo_explain_concurrency_failure_stormexplain_regression_suite
Trace and retrieval:
record_tool_tracesimilar_incidentsrefresh_trace_pathrefresh_docs_pathmemory_upsertmemory_search
SLO and ROI:
health_checkbenchmark_callsscenario_load_testcapture_roi_baselinereport_drift_bugdecision_gate
Schema evaluation (verdict + report via MCP, no external Postgres):
schema_load_toolschema_explain_toolschema_evaluate_toolschema_evaluate_full_tool
Project contract and regression:
project_manifestproject_capabilitiesproject_tool_catalogproject_get_defaultsproject_run_regressionproject_capture_baselineproject_compare_baseline
Generic project state:
project_list_entitiesproject_get_entityproject_upsert_entityproject_delete_entityproject_append_eventproject_ingest_traceproject_explain_runproject_export_state
Generic heuristics:
project_list_heuristicsproject_run_heuristic
For machine-readable discovery, prefer:
project_tool_catalogproject_get_defaults
Current heuristic profiles available through project_run_heuristic:
pain_structurenaive_biasprice_distributionliquidity_signalsprice_liquidity_matrixcross_categorysale_formatspeed_signalstrust_signals
If Cursor MCP auto-discovery is enabled, restart Cursor and connect mini-data-engine.
Default MCP runtime data paths are under tests/artifacts/mcp/*.
Cursor approval setup (reduce repeated prompts)
If Cursor keeps asking for MCP or command approval on every call, apply this once:
Enable workspace trust in Cursor user settings:
"security.workspace.trust.enabled": trueIn Cursor, open
Settings -> Agents -> Auto-Runand set:Auto-run mode:Run in SandboxMCP Allowlist: addmini-data-enginetools you use oftenCommand Allowlist: add frequently used safe commands
Keep this repo opened as the same trusted workspace and reload the window once.
Notes:
MCP server approval and per-tool allowlist behavior are enforced by Cursor security settings.
In some Cursor versions, allowlist behavior can be best-effort and still prompt in edge cases.
Fastest way to see the new explainability use case in action through MCP:
demo_explain_runThat single tool call creates a traced run, records step-level events under one run_id,
and returns an explanation with:
ordered timeline
tool path
total elapsed time
failure summary if anything breaks
You can then replay the same explanation directly with:
explain_run(run_id="...")Explain Regression Suite
Use explain_regression_suite when you want regression checks to run through MCP and come back as explainable run summaries instead of isolated test output.
The suite drives the current validation surface through the MCP layer, attaches run_id traces, and returns explain output for each check so regressions can be inspected with the same mechanism used for runtime incidents.
It currently runs:
Python unit tests via
python -m unittest discoverRust tests via
cargo test, including the currentengine_cliintegration testshealth_checkbenchmark_callsscenario_load_testexplainability control demos
The explainability demos intentionally include both positive and negative controls:
demo_explain_runasexpected_successdemo_explain_run_failureasexpected_failuredemo_explain_semantic_failureasexpected_failuredemo_explain_idempotency_conflictasexpected_failuredemo_explain_concurrency_failure_stormasexpected_failure
That means the suite is not only checking that the happy path stays green. It also checks that the explain layer still classifies and summarizes known failure classes correctly.
The current regression surface covers:
happy-path traced execution
runtime/path failures
semantic data validation failures
idempotency conflict failures
concurrency and failure-storm control scenarios
sampled benchmark and scenario SLO regressions
Fastest MCP call for the full regression bundle:
explain_regression_suiteUse it as the top-level MCP regression entrypoint when you want one answer that includes:
which checks passed
which failures were expected controls
explain summaries for each traced run
early signals that a latency or behavior regression appeared
The MCP layer is an access interface, not the core product idea. The core of the repository is the runnable lab itself.
Product note:
PRODUCT_NOTE_RUNTIME_EXPLAINABILITY.mdShort note describing the runtime explainability use case, the required signals, and theexplain_runMVP.PRODUCT_NOTE_RUNTIME_COPILOT.mdProduct framing forRuntime Copilotas an MCP-native operational brain.EXPLAIN_REGRESSION_SUITE_FEASIBILITY.mdShort article describing what this repository validated about explain-first regression suites and where the current denominator still stays narrow.
Use in Codex:
skill package:
codex/skills/runtime-copilot/SKILL.mdautomation examples:
codex/automationsguide:
docs/use-in-codex.md
Run persistent engine CLI (productization path):
# Initialize storage
cargo run --bin engine_cli -- init ./tests/artifacts/engine/data orders
# Insert and upsert (WAL append)
cargo run --bin engine_cli -- insert ./tests/artifacts/engine/data orders 1 4242 50
cargo run --bin engine_cli -- upsert ./tests/artifacts/engine/data orders 1 4242 55
# Build index and explain
cargo run --bin engine_cli -- index ./tests/artifacts/engine/data orders
cargo run --bin engine_cli -- explain ./tests/artifacts/engine/data orders 4242
# Write snapshot and truncate WAL
cargo run --bin engine_cli -- checkpoint ./tests/artifacts/engine/data orders
# Transaction simulation: begin/commit/rollback semantics,
# per-table write lock, snapshot read, and conflict detection
cargo run --bin engine_cli -- tx-demo ./tests/artifacts/engine/data orders
# Crash/restart recovery for transaction journals
cargo run --bin engine_cli -- tx-recovery-list ./tests/artifacts/engine/data orders
cargo run --bin engine_cli -- tx-recovery-commit ./tests/artifacts/engine/data orders <tx_id>
cargo run --bin engine_cli -- tx-recovery-rollback ./tests/artifacts/engine/data orders <tx_id>What You Should See
In
mini_pg_like.py: selective predicate switches toIndex Scan; non-selective staysSeq Scan.In
mini_databricks_clone.py: layer-by-layer demo output, workflow DAG order/metrics, and canonical events count from the single write path.In
mini_pg_like(Rust): same planner behavior with shared core modules.In
mini_databricks_clone(Rust): same layered demo using shared Rust library code.In
engine_cli(Rust): persistent snapshot + WAL replay flow with simple operational commands.In
engine_cli tx-demo: explicit transaction scopes, snapshot reads, per-table write lock, and concurrent upsert conflict detection.In
engine_cli tx-recovery-*: staged transaction operations survive process restarts via per-transaction journal files and can be committed or rolled back explicitly.In
e2e_flow: one command runs write path, checkpoint, bronze->silver transform, planner explain, and DuckDB SQL validation on persisted data.
Technical Design Backbone
TECHNICAL_DESIGN_GENERIC.md captures the architectural discipline behind the code:
cross-layer reasoning (
Idea -> API -> Runtime -> Storage -> Perf),deterministic state transitions,
event-first design,
adapter contracts,
DAG-driven orchestration,
measurable go/no-go criteria.
It is not a separate product claim. It is the review and implementation spine used across the lab.
Docker package
Use the local build (recommended for development)
Build the image from this repo so MCP uses your local code (including schema tools) instead of the GitHub image:
./scripts/docker-build-local.shThis builds mini-data-engine:local. To drive another project (e.g. threads) with this MCP, set Cursor MCP to use the local image and mount that project as workspace:
Copy .cursor/mcp.docker.local.json into your project’s
.cursor/mcp.json(or merge themcpServersentry into your Cursor user config).Open the project you want to drive (e.g. threads).
${workspaceFolder}will be that project; the container getsWORKSPACE_ROOT=/workspaceand your project mounted at/workspace, so e.g.schema_path="schema.sql"resolves to that project’s file.
Example local config (uses mini-data-engine:local and mounts current workspace as /workspace):
{
"mcpServers": {
"mini-data-engine": {
"command": "docker",
"args": [
"run", "--rm", "-i",
"-e", "WORKSPACE_ROOT=/workspace",
"-v", "${workspaceFolder}:/workspace",
"-v", "${workspaceFolder}/tests/artifacts:/app/tests/artifacts",
"mini-data-engine:local"
]
}
}
}Use the published image (GHCR)
Image is published to GHCR:
ghcr.io/kroq86/data-engineering-runtime-lab:latest
Pull:
docker pull ghcr.io/kroq86/data-engineering-runtime-lab:latestUse in Cursor MCP config (example):
{
"mcpServers": {
"mini-data-engine": {
"command": "docker",
"args": [
"run",
"--rm",
"-i",
"-v",
"${workspaceFolder}/tests/artifacts:/app/tests/artifacts",
"ghcr.io/kroq86/data-engineering-runtime-lab:latest"
]
}
}
}Loom stack
MCP surface for loom-ops and ops runbooks. Ecosystem: ECOSYSTEM.md
pip install ops-runtime-mcp
ops-runtime-mcp # stdio MCP; see docs/use-in-codex.mdAvailable Tools
47 toolsbenchmark_callsC
Benchmark MCP operations with SLO-style summary metrics.
Inputs: iterations, root_dir, table, min_success_rate, +1 more. Returns: object
capture_roi_baselineC
Capture baseline KPI snapshot for ROI Phase 0.
Inputs: root_dir, table, benchmark_iterations, scenario_iterations, +1 more. Returns: object
create_indexC
Create customer index for engine table.
Inputs: root_dir, table. Returns: object
decision_gateC
Evaluate migration triggers and return pass/fail gate.
Inputs: trace_db_path, baseline_path, drift_counter_path, need_rust_portfolio, +3 more. Returns: object
demo_explain_concurrency_failure_stormC
Run a traced concurrency conflict plus failure-storm scenario.
Inputs: root_dir, table, trace_db_path, workers. Returns: object
demo_explain_idempotency_conflictB
Run a traced idempotency-conflict flow through WriteCore.
Inputs: trace_db_path. Returns: object
demo_explain_runC
Run a traced demo flow, then explain the run immediately.
Inputs: root_dir, table, customer_id, trace_db_path. Returns: object
demo_explain_run_failureC
Run a traced failing flow, then explain the failed run immediately.
Inputs: root_dir, table, trace_db_path. Returns: object
demo_explain_semantic_failureC
Run a traced semantic-corruption flow and explain the failed validation.
Inputs: root_dir, table, customer_id, trace_db_path. Returns: object
explain_customerC
Run EXPLAIN ANALYZE style output by customer filter.
Inputs: root_dir, table, customer_id. Returns: object
explain_regression_suiteC
Run regression checks and return explain output for each traced run.
Inputs: root_prefix, trace_db_path, benchmark_iterations, scenario_iterations. Returns: object
explain_runC
Explain one recorded run_id from the local trace store.
Inputs: run_id (required), trace_db_path, max_timeline_events. Returns: object
health_checkC
Run a quick MCP smoke flow and summarize status.
Inputs: root_dir, table. Returns: object
init_engineD
Initialize persistent engine storage.
Inputs: root_dir, table. Returns: object
insert_rowC
Insert one row into the persistent engine.
Inputs: root_dir, table, order_id, customer_id, +1 more. Returns: object
memory_searchC
Search memory entries by semantic similarity and metadata filters.
Inputs: query_text (required), top_k, min_score, status, +6 more. Returns: object
memory_upsertC
Upsert one operational memory entry for semantic recall.
Inputs: run_id (required), summary (required), status, tool_name, +9 more. Returns: object
project_append_eventC
Append a generic project event to the local event log.
Inputs: event_type, entity_type, entity_id, payload_json, +3 more. Returns: object
project_capabilitiesB
Return declared runtime capabilities and contract coverage.
Inputs: no named parameters. Returns: object
project_capture_baselineC
Capture a baseline snapshot and return a unified verdict envelope.
Inputs: root_dir, table, benchmark_iterations, scenario_iterations, +1 more. Returns: object
project_compare_baselineC
Compare current benchmark/scenario results against a stored baseline.
Inputs: baseline_path, root_dir, table, benchmark_iterations, +4 more. Returns: object
project_delete_entityC
Delete a declared entity from the generic state store.
Inputs: entity_type, entity_id, root_dir, dry_run. Returns: object
project_explain_runC
Read one run explanation through the generic project explain entrypoint.
Inputs: run_id (required), trace_db_path, max_timeline_events. Returns: object
project_export_stateC
Export generic project state as a JSON snapshot.
Inputs: root_dir, entity_type, output_path, include_events. Returns: object
project_get_defaultsA
Return default workspace, paths, runtime mode, and project metadata.
Inputs: no named parameters. Returns: object
project_get_entityC
Load one declared project entity by identity key.
Inputs: entity_type, entity_id, root_dir. Returns: object
project_ingest_traceC
Append one normalized trace record through the generic project ingest path.
Inputs: run_id (required), tool_name (required), status (required), summary (required), +10 more. Returns: object
project_list_entitiesC
List declared project entities from the generic state store.
Inputs: entity_type, root_dir, limit. Returns: object
project_list_heuristicsA
List declared heuristic profiles available for generic project analysis.
Inputs: no named parameters. Returns: object
project_manifestB
Describe project state roots, schemas, and supported regression primitives.
Inputs: no named parameters. Returns: object
project_run_heuristicC
Run one declared heuristic profile over a source and persist the analysis through project state.
Inputs: heuristic_name (required), source_path (required), source_kind, root_dir, +3 more. Returns: object
project_run_regressionD
Run the explain-first regression bundle and return a unified verdict.
Inputs: root_prefix, trace_db_path, benchmark_iterations, scenario_iterations. Returns: object
project_tool_catalogB
Return the full MCP tool catalog with groups, entrypoints, and summaries.
Inputs: group. Returns: object
project_upsert_entityC
Create or update a declared entity in the generic state store.
Inputs: entity_type, entity_id, payload_json, root_dir, +2 more. Returns: object
record_tool_traceC
Append one MCP tool trace record to local trace store.
Inputs: run_id (required), tool_name (required), status (required), summary (required), +9 more. Returns: object
refresh_docs_pathC
Incrementally ingest project docs, code, and config files.
Inputs: source_dir, trace_db_path, refresh_state_path, scenario_id, +4 more. Returns: object
refresh_trace_pathC
Incrementally ingest new lines from source path into trace store.
Inputs: source_path (required), trace_db_path, refresh_state_path, scenario_id. Returns: object
reindex_projectC
Re-index this project dataset (engine_cli index).
Inputs: root_dir, table. Returns: object
report_drift_bugC
Increment and persist split-logic drift bug counter.
Inputs: note, counter_path. Returns: object
run_e2e_flowC
Execute full MiniPG + MiniDatabricks + DuckDB end-to-end flow.
Inputs: root_dir. Returns: object
scenario_load_testD
Mixed workload load test: insert/upsert/explain/reindex/e2e and summary metrics.
Inputs: iterations, root_dir, table, min_success_rate, +2 more. Returns: object
schema_evaluate_full_toolC
One-shot: load schema → run EXPLAIN (from query_profiles_json/seed_sql_json) → evaluate. Generic: you supply queries and optional seed SQL.
Inputs: schema_path, ddl_text, schema_entity_id, query_profiles_json, +4 more. Returns: object
schema_evaluate_toolB
Build verdict from schema metadata and EXPLAIN outputs; write evaluation_report.json and verdict.md. If query_profile_names empty, discovers explain_*.txt in artifacts.
Inputs: schema_entity_id, query_profile_names, root_dir, artifacts_dir. Returns: object
schema_explain_toolB
Run EXPLAIN for each profile in query_profiles_json (JSON: name -> SQL). Optional seed_sql_json (JSON array of SQL) runs before EXPLAIN. Writes explain_.txt.
Inputs: schema_entity_id, query_profiles_json, seed_sql_json, root_dir, +1 more. Returns: object
schema_load_toolA
Ingest DDL from file or raw text, validate with DuckDB, store metadata in project state (schemas/).
Inputs: schema_path, ddl_text, schema_entity_id, root_dir. Returns: object
similar_incidentsC
Find semantically similar historical incidents.
Inputs: query_text (required), top_k, min_score, status, +5 more. Returns: object
upsert_rowC
Upsert one row by order_id.
Inputs: root_dir, table, order_id, customer_id, +1 more. Returns: object
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 47 tools
Many tools have overlapping purposes, particularly the multiple 'explain' and 'project_*' tools are hard to distinguish. The demo, benchmark, and schema tools also have unclear boundaries, making selection confusing for an agent.
Naming conventions are inconsistent: some use underscore (benchmark_calls), some use project_ prefix, others schema_ prefix, and demo_explain_*. While groups have internal consistency, the overall pattern is mixed and unpredictable.
47 tools is excessive for a single server. Many tools could be consolidated or split into separate servers. The count feels bloated and hard to navigate.
The tool set appears to cover many areas (benchmarking, demo, schema management, project state) but lacks clear CRUD coverage for key entities and has dead ends (e.g., no delete for most entities). Gaps in lifecycle operations are notable.
Maintenance
Related MCP Connectors
Governed data discovery, exact queries, decisions, simulations, and runtime utilities over MCP.
Hosted MCP memory and agent control plane for durable conversations, jobs, and operations.
Analytical memory for AI agents: a real Postgres queried in plain English over MCP. One command.
MCP observability. Query live traffic, errors, duration, and alerts from your AI agent.
Related MCP Servers
- AlicenseBqualityCmaintenanceA read-only MCP server that exposes dbt project artifacts and data quality result tables (BigQuery/Postgres) to LLM clients, enabling deep introspection, run-history analysis, source freshness, test coverage, and lineage walks.2742 npmMIT
- AlicenseNot gradedqualityDmaintenanceEnables AI agents to query live schema, lineage, and query-context across data warehouses, dbt projects, orchestration systems, and BI tools via MCP tools.Apache 2.0
- AlicenseBqualityBmaintenanceEnables MCP hosts like ChatGPT and Claude to query data platform metadata, explore schemas and tables, inspect SQL explain plans, and monitor DAG and ETL status through standardized tools.21MIT
- AlicenseNot gradedqualityCmaintenanceEnables MCP clients to ask bilingual data questions, inspect database tables and schemas, and execute validated read-only SQL through exposed tools.MIT