🔭 EOSC Data Commons Search server

A server for the EOSC Data Commons project MatchMaker service, providing natural language search over open-access datasets. It exposes an HTTP POST endpoint and supports the Model Context Protocol (MCP) to help users discover datasets and tools via a Large Language Model–assisted search.

🧩 Endpoints

The HTTP API comprises 2 main endpoints:

• /mcp: MCP server that searches for relevant data to answer a user question using the EOSC Data Commons OpenSearch service

  • Uses Streamable HTTP transport

  • Available tools:

    • Search datasets

    • Get metadata for the files in a dataset (name, description, type of files)

    • Search tools

    • Search citations related to datasets or tools

• /chat: HTTP POST endpoint (JSON) for chatting with the MCP server tools via an LLM provider (API key provided through env variable at deployment)

  • Streams Server-Sent Events (SSE) response complying with the AG-UI protocol.

Related MCP server: Datos.gob.es-MCP

🔌 Connect to the MCP server

The system can be used directly as a MCP server using either STDIO, or Streamable HTTP transport.

Follow the instructions of your client, and use the /mcp URL of the public server: https://matchmaker.eosc-data-commons.eu/api/search/mcp

To add a new MCP server to VSCode GitHub Copilot:

• Open the Command Palette (ctrl+shift+p or cmd+shift+p)

• Search for MCP: Add Server...

• Choose HTTP, and provide the MCP server URL: https://matchmaker.eosc-data-commons.eu/api/search/mcp

Your VSCode mcp.json should look like:

{
    "servers": {
        "data-commons-search-http": {
            "url": "https://matchmaker.eosc-data-commons.eu/api/search/mcp",
            "type": "http"
        }
    },
    "inputs": []
}

🛠️ Development

📥 Install dev dependencies

uv sync --all-extras

Install pre-commit hooks:

uv run --all-extras pre-commit install

Create a keys.env file with your LLM provider API key(s), and optionally other configurations:

CESNET_API_KEY=YOUR_API_KEY
MISTRAL_API_KEY=YOUR_API_KEY

OIDC_CLIENT_ID=
OIDC_CLIENT_SECRET=
LANGFUSE_PUBLIC_KEY=
LANGFUSE_SECRET_KEY=
POSTGRES_HOST=localhost
POSTGRES_USER=app
POSTGRES_PASSWORD=app_password

RATE_LIMITING_ENABLED=False
LOG_LEVEL=DEBUG
LOG_JSON=false

OPENSEARCH_URL=http://localhost:9200

💾 Database

The search system needs to connect to a PostgreSQL database to store authenticated users conversations.

Deploy and initialize the metadata-warehouse, in these instructions we expect the metadata-warehouse folder to be alongside the data-commons-search,in the same folder.

cd ../metadata-warehouse
docker compose up postgres

To initialize db, run from the metadata-warehouse repo:

uv run --directory scripts/postgres_data create_db.py --db appdb --reset

ALTER USER app WITH PASSWORD 'newpassword';

Reset db:

docker compose down --volumes --remove-orphans

Export the schema from db.py to the metadata-warehouse (command to run at the root of the data-commons-search repo):

uv run scripts/export_db_schema.py ../metadata-warehouse/scripts/postgres_data/create_sql/appdb/tables.sql

⚡️ Start dev server

Start the server in dev at http://localhost:8000, with MCP endpoint at http://localhost:8000/mcp pointing to a running OpenSearch instance:

uv run --all-extras uvicorn src.data_commons_search.main:app --reload

Default OPENSEARCH_URL=http://localhost:9200

Customize server port through environment variable:

OPENSEARCH_URL=http://localhost:9200 SERVER_PORT=8001 uv run --all-extras uvicorn src.data_commons_search.main:app --host 0.0.0.0 --port 8001 --reload

cd ../matchmaker
npm run dev

curl -X POST http://localhost:8000/chat -H "Content-Type: application/json" \
	-d '{"items": [{"type": "message", "role": "user", "content": [{"text": "Educational datasets from Switzerland covering student assessments, language competencies, and learning outcomes, including experimental or longitudinal studies on pupils or students."}]}], "model": "cesnet/agentic"}'

curl -X POST http://localhost:8000/chat -H "Content-Type: application/json" \
-H "Cookie: access_token=$ACCESS_TOKEN" \
-d '{"items": [{"type": "message", "role": "user", "content": [{"text": "Educational datasets from Switzerland covering student assessments, language competencies, and learning outcomes, including experimental or longitudinal studies on pupils or students."}]}], "model": "cesnet/agentic"}'

curl -X GET "http://localhost:8000/conversation/$(curl -s http://localhost:8000/conversations -H "Content-Type: application/json" -H "Cookie: access_token=$ACCESS_TOKEN" | jq -r '.[-1].thread_id')" -H "Content-Type: application/json" -H "Cookie: access_token=$ACCESS_TOKEN"

curl -H "Authorization: Bearer $CESNET_API_KEY" https://llm.ai.e-infra.cz/v1/models | jq ".data[].id"

🔐 Secrets Store

EGI Secret Store, get the token from aai.egi.eu/token (decode the JWT to get the actual access token)

export BASE="https://matchmaker.eosc-data-commons.eu"
curl -s "$BASE/auth/user" --cookie "access_token=$TOKEN"

curl -s -X PUT "$BASE/auth/keys/vip" --cookie "access_token=$TOKEN" \
  -H "Content-Type: application/json" -d '{"key_value":"sk-123"}'

curl -s "$BASE/auth/keys" --cookie "access_token=$TOKEN"
curl -s "$BASE/auth/keys/all" --cookie "access_token=$TOKEN"
curl -s "$BASE/auth/keys/vip" --cookie "access_token=$TOKEN"
curl -s -X DELETE "$BASE/auth/keys/vip" --cookie "access_token=$TOKEN"

🐳 Deploy with Docker

Create a keys.env file with the API keys (see above for complete example):

CESNET_API_KEY=YOUR_API_KEY
MISTRAL_API_KEY=YOUR_API_KEY
SEARCH_API_KEY=SECRET_KEY_YOU_CAN_USE_IN_FRONTEND_TO_AVOID_SPAM

You can use the prebuilt docker image ghcr.io/eosc-data-commons/data-commons-search:main

Example compose.yml:

services:
  mcp:
    image: ghcr.io/eosc-data-commons/data-commons-search:main
    ports:
      - "127.0.0.1:8000:8000"
    environment:
      OPENSEARCH_URL: "http://opensearch:9200"
      CESNET_API_KEY: "${CESNET_API_KEY}"

Build and deploy the service:

docker compose up

📦 Build for production

Build package in dist/:

uv build

✅ Run tests

uv run pytest

Run benchmark (check success of a set of search queries):

uv run tests/benchmark.py

Run LLM jailbreak tests with garak:

PYTHONPATH=tests/security uv run garak --config tests/security/garak.yaml

Run stress tests (20 concurrent uses) of the API:

uv run tests/stress_api.py -c 20

🧹 Format code and type check

uvx ruff format && uvx ruff check --fix && uvx ty check

♻️ Reset the environment

Upgrade uv:

uv self update

Clean uv cache:

uv cache clean

🔧 Maintenance

Pre-compute stats for the datasets in the db to src/data_commons_search/stats.json:

POSTGRES_DB=datasetdb uv run scripts/compute_stats.py

Update dependencies in pyproject.toml:

uvx uv-bump

🏷️ Release process

Run the release script providing the version bump: fix, minor, or major

.github/release.sh fix

This will create a git tag, github release, and publish a docker image

🤝 Acknowledments

The LLM provider cesnet is a service provided by e-INFRA CZ and operated by CERIT-SC Masaryk University

Computational resources were provided by the e-INFRA CZ project (ID:90254), supported by the Ministry of Education, Youth and Sports of the Czech Republic.

The authentication provider is EGI Check-in.