Provides natural language querying capabilities for Karenina verification results stored in SQLite databases, with hierarchical schema discovery and SQL query generation for analyzing benchmarking data.
Click on "Install Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@Karenina MCPshow me the top performing models on biology questions"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
karenina-mcp
Experimental - This is an experimental MCP server for inspecting Karenina verification results through natural language queries.
Overview
karenina-mcp provides an MCP (Model Context Protocol) interface that allows AI assistants like Claude to explore and analyze verification results stored in a Karenina SQLite database. Instead of writing SQL queries manually, you can ask questions in natural language and the assistant will translate them into appropriate queries.
How It Works
The server uses a hierarchical context exposition approach to help the assistant understand your database efficiently:
Step 1: Configure the Database
First, call configure_database with the path to your SQLite results database. This connects the server and returns a list of available tables and views.
Step 2: Query with Natural Language
Once configured, the agent uses hierarchical schema discovery to answer your questions:
Schema Awareness - View summaries are embedded in the
get_schematool description, so the agent sees all available views without any tool callSelective Deep-Dive - The agent calls
get_schema([view_names])only for views relevant to your questionQuery Generation - With precise schema knowledge, it generates accurate SQL queries
Results Interpretation - Results are returned as formatted markdown tables
This approach minimizes context usage while ensuring the assistant has the precise information needed to answer your questions accurately.
Installation
Usage
Run the server (STDIO mode)
Run as HTTP server
Start the MCP server as an HTTP server for remote or web-based access:
The server will be available at http://localhost:8000. You can also specify a custom host:
Configure in Claude Code
Add to your Claude Code settings (.claude/settings.local.json or global settings):
Replace /path/to/karenina-mcp with the absolute path to the karenina-mcp directory.
Configure in Claude Desktop
Add to your Claude Desktop config (~/Library/Application Support/Claude/claude_desktop_config.json):
Tools
configure_database
Initialize the server with your results database.
Returns confirmation with list of available tables and views.
get_schema
Get detailed schema documentation for specific views. The tool description itself contains one-line summaries of all available views, so the agent can identify relevant views without calling the tool.
Returns full column documentation, types, primary/foreign keys, join information, and example queries for the requested views.
Example Questions
Once the database is configured, you can ask questions like:
"What's the overall pass rate across all models?"
"Show me the questions where "mcp-local" was correct but "mcp-remote" failed;
"Compute pass rates by question keywords and sort them in increasing performance"
Show me results to question from the last run where more than one but not all of the replicates failed;
Related Projects
Karenina - Core benchmarking framework