OpenXAI MCP Server

Instructions

Implementation Reference

• index.js:670-766 (handler)

  The primary handler function `listMetrics(metricType)` that executes the tool logic. It defines metrics grouped by categories (faithfulness, stability, ground_truth) and returns filtered list based on input metric_type or 'all', formatted as MCP content response.

  async listMetrics(metricType) {
    const metrics = {
      faithfulness: {
        PGI: {
          name: 'Prediction Gap on Important feature perturbation',
          description: 'Measures the difference in prediction probability when perturbing important features',
          higher_is_better: true
        },
        PGU: {
          name: 'Prediction Gap on Unimportant feature perturbation',
          description: 'Measures the difference in prediction probability when perturbing unimportant features',
          higher_is_better: false
        }
      },
      stability: {
        RIS: {
          name: 'Relative Input Stability',
          description: 'Measures maximum change in explanation relative to changes in inputs',
          higher_is_better: false
        },
        RRS: {
          name: 'Relative Representation Stability',
          description: 'Measures maximum change in explanation relative to changes in internal representation',
          higher_is_better: false
        },
        ROS: {
          name: 'Relative Output Stability',
          description: 'Measures maximum change in explanation relative to changes in output predictions',
          higher_is_better: false
        }
      },
      ground_truth: {
        FA: {
          name: 'Feature Agreement',
          description: 'Fraction of top-K features common between explanation and ground truth',
          higher_is_better: true
        },
        RA: {
          name: 'Rank Agreement',
          description: 'Fraction of top-K features with same rank in explanation and ground truth',
          higher_is_better: true
        },
        SA: {
          name: 'Sign Agreement',
          description: 'Fraction of top-K features with same sign in explanation and ground truth',
          higher_is_better: true
        },
        SRA: {
          name: 'Signed Rank Agreement',
          description: 'Fraction of top-K features with same sign and rank in explanation and ground truth',
          higher_is_better: true
        },
        RC: {
          name: 'Rank Correlation',
          description: 'Spearman rank correlation between explanation and ground truth rankings',
          higher_is_better: true
        },
        PRA: {
          name: 'Pairwise Rank Agreement',
          description: 'Fraction of feature pairs with same relative ordering in explanation and ground truth',
          higher_is_better: true
        }
      }
    };
  
    let result = [];
    if (metricType === 'all') {
      result = Object.entries(metrics).map(([category, categoryMetrics]) => ({
        category,
        metrics: Object.entries(categoryMetrics).map(([key, value]) => ({
          metric: key,
          ...value
        }))
      }));
    } else {
      const categoryMetrics = metrics[metricType];
      if (categoryMetrics) {
        result = [{
          category: metricType,
          metrics: Object.entries(categoryMetrics).map(([key, value]) => ({
            metric: key,
            ...value
          }))
        }];
      }
    }
  
    return {
      content: [
        {
          type: 'text',
          text: `Available OpenXAI evaluation metrics:\n\n` +
                JSON.stringify(result, null, 2)
        }
      ]
    };
  }

• index.js:157-171 (registration)

  Registration of the 'list_metrics' tool in the ListToolsRequest handler, including name, description, and inputSchema.

  {
    name: 'list_metrics',
    description: 'List available evaluation metrics in OpenXAI',
    inputSchema: {
      type: 'object',
      properties: {
        metric_type: {
          type: 'string',
          description: 'Filter by metric type (faithfulness, stability, fairness)',
          enum: ['faithfulness', 'stability', 'fairness', 'all']
        }
      },
      required: []
    }
  },

• index.js:160-169 (schema)

  The inputSchema for the 'list_metrics' tool, defining optional metric_type parameter with enum ['faithfulness', 'stability', 'fairness', 'all']. Note: enum has 'fairness' but code uses 'ground_truth'; possible discrepancy.

  inputSchema: {
    type: 'object',
    properties: {
      metric_type: {
        type: 'string',
        description: 'Filter by metric type (faithfulness, stability, fairness)',
        enum: ['faithfulness', 'stability', 'fairness', 'all']
      }
    },
    required: []

• index.js:273-274 (handler)

  The switch case dispatcher in CallToolRequestHandler that routes 'list_metrics' calls to the listMetrics method.

  case 'list_metrics':
    return await this.listMetrics(args.metric_type || 'all');