IBM i MCP Server

# MCP Server Evals This directory contains evaluation configurations and test cases for the MCP Server. These evaluations are designed to test various functionalities and performance aspects of the server. ## MCPJam Evals CLI ### Install the CLI ```bash npm install -g @mcpjam/cli ``` ### Set up tests To set up, create a new folder directory for your test. In that directory, create three files: 1. `environment.json`: This file contains the environment configuration for the tests. 2. `llms.json`: This file specifies the language models to be used in the tests 3. `tests.json`: This file contains the actual test cases to be executed. ```bash cp templates/enronment.template.json environment.json cp templates/llms.template.json llms.json cp templates/tests.template.json tests.json ``` ## Server connection file (environment.json) This file is configured very similar to a mcp.json file. For servers with OAuth, you must provide your own Bearer API tokens. MCPJam CLI does not handle OAuth flows / DCR. For bearer tokens, make sure to wrap your header with requestInit. ### Server connection file (environment.json) This file is configured very similar to a `mcp.json` file. For servers with OAuth, you must provide your own `Bearer` API tokens. MCPJam CLI does not handle OAuth flows / DCR. For bearer tokens, make sure to wrap your header with `requestInit`. ```json { "servers": { "ibmi-mcp-server": { "url": "http://localhost:3010/mcp" }, "ibmi-mcp-server-stdio": { "command": "npx", "args": ["/path/to/your/ibmi-mcp-server/dist/index.js"], "env": { "TOOLS_YAML_PATH": "tools", "NODE_OPTIONS": "--no-deprecation", "DB2i_HOST": "your-host.example.com", "DB2i_USER": "your-username", "DB2i_PASS": "your-password", "DB2i_PORT": "your-port", "MCP_TRANSPORT_TYPE": "stdio" } } } } ``` ### Tests file (tests.json) The test file is an array of tests. ```json [ { "title": "Workspace test", "query": "What is my asana workspace?", "runs": 1, "model": "anthropic/claude-3.7-sonnet", "provider": "openrouter", "expectedToolCalls": ["asana_list_workspaces"] }, { "title": "Workspace users test", "query": "Can you figure out who is in the workspace?", "runs": 1, "model": "anthropic/claude-3.7-sonnet", "provider": "openrouter", "expectedToolCalls": ["asana_list_workspaces", "asana_get_workspace_users"] } ] ``` ### LLM API key file (llms.json) ```json { "anthropic": "<ANTHROPIC_API_KEY>", "openai": "<OPENAI_API_KEY>", "openrouter": "<OPENROUTER_API_KEY>" } ``` ### Run MCP Eval ```bash mcpjam evals run --tests tests.json --environment environment.json --llms llms.json ``` #### Short flags ```bash mcpjam evals run -t tests.json -e environment.json -l llms.json ``` #### CLI Options - `--tests, -t <file>`: Path to the tests configuration file (required) - `--environment, -e <file>`: Path to the environment configuration file (required) - `--llms, -l <file>`: Path to the LLM API key configuration file - `--help, -h`: Show help information - `--version, -V`: Display version number