OpenXAI MCP Server

Instructions

Implementation Reference

• index.js:847-1001 (handler)

  The handler function that executes the get_framework_info tool. It takes an infoType parameter and returns the corresponding detailed text information about the OpenXAI framework (overview, features, paper, installation, or quickstart) wrapped in MCP content format.
  async getFrameworkInfo(infoType) { const info = { overview: `OpenXAI Framework Overview OpenXAI is a comprehensive and extensible open-source framework for evaluating and benchmarking post hoc explanation methods. It provides: 🔍 **Evaluation Framework**: Systematic evaluation of explanation methods with 22+ quantitative metrics 📊 **Datasets**: Collection of synthetic and real-world datasets with ground truth explanations 🤖 **Models**: Pre-trained models for various machine learning tasks 🔬 **Explainers**: Implementations of state-of-the-art explanation methods (LIME, SHAP, etc.) 📈 **Leaderboards**: Public XAI leaderboards for transparent benchmarking 🛠️ **Extensibility**: Easy integration of custom datasets, models, and explanation methods Key Features: - Model-agnostic explanation methods - Ground truth faithfulness metrics - Predicted faithfulness metrics - Stability and robustness evaluation - Fairness assessment across subgroups - Comprehensive benchmarking pipeline`, features: `OpenXAI Key Features 🎯 **Explanation Methods**: - LIME (Local Interpretable Model-agnostic Explanations) - SHAP (SHapley Additive exPlanations) - Integrated Gradients - Grad-CAM - Guided Backpropagation - And more... 📊 **Evaluation Metrics**: - Faithfulness: PGI, PGU - Stability: RIS, RRS, ROS - Ground Truth: FA, RA, SA, SRA, RC, PRA - Fairness: Subgroup analysis 🗂️ **Datasets**: - Synthetic datasets with ground truth - Real-world datasets (German Credit, COMPAS, Adult Income) - Tabular, image, and text data support 🤖 **Models**: - Neural Networks (ANN) - Logistic Regression - Random Forest - Support Vector Machine - XGBoost 🏆 **Leaderboards**: - Public benchmarking platform - Transparent evaluation results - Community-driven improvements`, paper: `OpenXAI Research Paper Title: "OpenXAI: Towards a Transparent Evaluation of Model Explanations" Authors: Chirag Agarwal, Satyapriya Krishna, Eshika Saxena, Martin Pawelczyk, Nari Johnson, Isha Puri, Marinka Zitnik, Himabindu Lakkaraju Abstract: While several types of post hoc explanation methods have been proposed in recent literature, there is little to no work on systematically benchmarking these methods in an efficient and transparent manner. OpenXAI introduces a comprehensive framework for evaluating and benchmarking post hoc explanation methods with synthetic data generators, real-world datasets, pre-trained models, and quantitative metrics. 📄 Paper: https://arxiv.org/abs/2206.11104 🌐 Website: https://open-xai.github.io/ 📚 GitHub: https://github.com/AI4LIFE-GROUP/OpenXAI Citation: @inproceedings{agarwal2022openxai, title={OpenXAI: Towards a Transparent Evaluation of Model Explanations}, author={Agarwal, Chirag and Krishna, Satyapriya and Saxena, Eshika and others}, booktitle={NeurIPS 2022 Datasets and Benchmarks Track}, year={2022} }`, installation: `OpenXAI Installation Guide 📦 **Installation**: \`\`\`bash # Install from PyPI pip install openxai # Or install from source git clone https://github.com/AI4LIFE-GROUP/OpenXAI.git cd OpenXAI pip install -e . \`\`\` 📋 **Requirements**: - Python 3.7+ - PyTorch or TensorFlow (for neural network models) - scikit-learn - pandas - numpy - matplotlib 🔧 **Optional Dependencies**: - For image explanations: Pillow, opencv-python - For text explanations: transformers, torch-text - For advanced visualizations: plotly, seaborn ✅ **Verification**: \`\`\`python import openxai print(openxai.__version__) \`\`\``, quickstart: `OpenXAI Quickstart Guide 🚀 **Quick Start Example**: \`\`\`python from openxai.dataloader import ReturnLoaders from openxai import LoadModel, Explainer, Evaluator # 1. Load dataset trainloader, testloader = ReturnLoaders(data_name='german', download=True) # 2. Load pre-trained model model = LoadModel(data_name='german', ml_model='ann', pretrained=True) # 3. Generate explanations explainer = Explainer(method='lime', model=model) inputs, labels = next(iter(testloader)) explanations = explainer.get_explanations(inputs) # 4. Evaluate explanations evaluator = Evaluator(model, metric='PGI') score = evaluator.evaluate(inputs=inputs, labels=labels, explanations=explanations) print(f"PGI Score: {score}") \`\`\` 🎯 **Common Workflows**: 1. **Benchmarking**: Compare multiple explanation methods 2. **Evaluation**: Assess explanation quality with metrics 3. **Leaderboards**: Submit results to public benchmarks 4. **Research**: Develop new explanation methods 📚 **Next Steps**: - Explore different datasets and models - Try various explanation methods - Evaluate with different metrics - Contribute to leaderboards` }; return { content: [ { type: 'text', text: info[infoType] || info.overview } ] }; }
• index.js:220-229 (schema)

  The input schema for the get_framework_info tool, defining an optional 'info_type' parameter with allowed enum values.
  inputSchema: { type: 'object', properties: { info_type: { type: 'string', description: 'Type of information to retrieve', enum: ['overview', 'features', 'paper', 'installation', 'quickstart'] } }, required: []
• index.js:217-231 (registration)

  Registration of the get_framework_info tool in the ListTools handler response, including name, description, and input schema.
  { name: 'get_framework_info', description: 'Get information about OpenXAI framework', inputSchema: { type: 'object', properties: { info_type: { type: 'string', description: 'Type of information to retrieve', enum: ['overview', 'features', 'paper', 'installation', 'quickstart'] } }, required: [] } },