🏦 Agent Risk AI — ML + MCP Server

Agent Risk AI: Your autonomous credit intelligence and risk analyst via MCP. A credit card default prediction model, trained with methodological rigor (stratified CV, Bayesian tuning with Optuna, optimized threshold, explainability via SHAP) and exposed as an MCP server — queryable directly by Claude Desktop/Code and AI agents in natural language.

📌 Why this project is different from "just training a model"

Most portfolio projects stop at training the model and showing a .ipynb with metrics. This one goes a step further: the model is encapsulated in an MCP server (Model Context Protocol) with 6 business tools, which means any compatible LLM host (Claude Desktop, Claude Code) can query the model in natural language, without writing code:

🗣️ "What is the default risk for this customer: age 46, income R$107,934, credit score 544, 2 previous defaults?" 🤖 → calls predict_default → responds with probability, class, and SHAP explanation.

This is exactly the pattern emerging in risk/data teams that want to put production models "into the conversation," not behind a static dashboard.

Related MCP server: CreddyMCP

🗂️ The business problem

Dataset of 45,528 credit card customers with demographic, income, and credit behavior variables. Target: credit_card_default (binary), with a real imbalance of 8.1% default rate — a typical credit risk scenario, where naive accuracy is a misleading metric.

🏗️ How the System Works (Simple Architecture)

The project turns raw credit data into actionable, auditable decisions consumed by AI agents through 4 integrated stages:

flowchart LR
    A["📁 1. Dados Brutos<br/><b>train.csv / test.csv</b>"] --> B["🧹 2. Limpeza & Features<br/><b>DTI, Limite, Flags</b>"]
    B --> C["🤖 3. Cérebro Preditivo<br/><b>XGBoost + Optuna + SHAP</b>"]
    C --> D["🔌 4. Servidor MCP<br/><b>6 Ferramentas de Negócio</b>"]
    D --> E["💬 5. Agente de IA<br/><b>Claude / Cursor / LLMs</b>"]

The 4-Step Flow:

1. 📁 1. Processing & Financial Intelligence (data_processing.py / feature_engineering.py)

   • Removes sensitive data (PII) and handles dataset anomalies (such as the retiree sentinel).

   • Creates real financial indicators: Debt-to-Income (DTI), credit utilization, and per capita income.

2. 🤖 2. Machine Learning Pipeline (pipeline.py / train.py)

   • Runs transformations (imputation, one-hot encoding, and scaling) in a leak-proof manner (no data leakage).

   • Trains and tunes XGBoost via Optuna (25 trials) in 5-fold cross-validation, calibrating the optimal decision threshold ($F_1 = 0.875$).

3. 🧠 3. Explainability & Auditing (inference.py / evaluate.py)

   • Persists the winning model and the SHAP TreeExplainer to decompose exactly which variables increase or reduce each customer's risk in real time.

4. 🔌 4. MCP Agentic Layer (mcp_server/server.py)

   • Exposes 6 ready-made tools so any AI assistant or agent (Claude Desktop, Claude Code, etc.) can query the model, simulate scenarios, and evaluate entire portfolios in natural language.

🔬 Domain-driven feature engineering

Instead of "throwing everything into XGBoost," each derived feature has an explicit credit risk rationale:

🧪 Methodology and statistical rigor

• Winsorization learned only on training (99.5th percentile) and reapplied on test/holdout — no data leakage.

• Single sklearn pipeline (ColumnTransformer + model) — imputation and encoding are recalculated at each cross-validation fold, not just once on the entire dataset (a common mistake that artificially inflates metrics).

• Selection metric: PR-AUC (Average Precision), not ROC-AUC or accuracy — the right choice for 8% prevalence of the positive class.

• 15% holdout never seen during Optuna tuning — the final metrics below reflect real generalization, not overfitting to the search process.

• Recalibrated decision threshold maximizing F1 on the holdout precision-recall curve (0.875), instead of blindly using 0.5 — essential when the positive class is rare.

• Explainability via SHAP TreeExplainer — every MCP server prediction can be audited factor by factor (relevant for credit regulatory compliance).

📊 Results and Performance Metrics

All metrics below were calculated on the holdout set (6,830 customers), completely isolated during Optuna hyperparameter search:

1. Model Comparison (Stratified 5-Fold Cross-Validation)

2. Holdout Performance Metrics (Champion Model)

3. Detailed Confusion Matrix on Holdout

4. Winning Hyperparameters (Optuna — 25 Trials)

{
  "n_estimators": 500,
  "max_depth": 4,
  "learning_rate": 0.0121,
  "subsample": 0.7244,
  "colsample_bytree": 0.7301,
  "min_child_weight": 8,
  "gamma": 3.1878,
  "reg_lambda": 3.5388,
  "reg_alpha": 0.0774,
  "scale_pos_weight": 11.3164
}

5. Top 10 Auditable Risk Factors (Mean $|\text{SHAP}|$)

📈 Visual Artifacts in reports/figures/:

• roc_curve.png — ROC curve with random baseline.

• precision_recall_curve.png — Precision-Recall curve compared to base prevalence.

• confusion_matrix.png — Confusion matrix at the optimal threshold.

• shap_summary.png — Beeswarm summary plot of global explainability.

🔒 All metrics above are reproducible and are saved in the audit metadata in models/model_metadata.json.

💡 Guide to Interpreting the Results (For Laypeople and Business)

To facilitate communication between data scientists, credit analysts, and non-technical directors, each system output has a direct business meaning:

1. 📈 Probability of Default (PD) & Action Bands

• What it is: The estimated probability (from 0% to 100%) that the customer will be more than 90 days late on their bill payment in the following months.

• How to act based on the band:

  • 🟢 MUITO_BAIXO (< 5%) and BAIXO (5% to 15%): Credit granting and limit increases recommended automatically with competitive rates.

  • 🟡 MODERADO (15% to 35%): Borderline customer. Conservative initial limit or income proof request recommended.

  • 🔴 ALTO (35% to 60%) and MUITO_ALTO (≥ 60%): High default risk. Proposal rejection or requirement of guarantors/real collateral recommended.

2. 📊 How to Read the SHAP Explainability Chart

• 🔴 Bars to the RIGHT (Positive Contribution): Registration or behavioral factors that push risk UP (e.g., low score, excessive revolving limit usage, prior default).

• 🟢 Bars to the LEFT (Negative Contribution): Healthy factors that protect the customer and push risk DOWN (e.g., years of job stability, high income, high score).

• 📏 Bar Length: The longer the bar, the more decisive that variable was for the AI's final verdict.

3. 📉 What is the What-If Simulation?

• It allows simulating the impact of changes in rules or guiding denied customers. For example: "If you reduce your limit utilization from 73% to 30%, your risk will drop from 68% to 22%, allowing your card to be approved."

4. 💰 Total Exposure and Expected Loss of the Portfolio

• Total Exposure: The total financial volume the institution put at stake (sum of granted credit limits).

• Expected Loss ($PD \times \text{Exposure}$): The amount in Reais the institution projects to statistically lose from default if no action is taken.

• Loss Rate (%): Direct basis for the Allowance for Doubtful Accounts (PDD / IFRS 9).

🔌 The MCP server — 6 business tools

Risk bands used by the server: MUITO_BAIXO (<5%) · BAIXO (5–15%) · MODERADO (15–35%) · ALTO (35–60%) · MUITO_ALTO (≥60%).

🌐 Web Chat Interface in the Browser (Streamlit)

The project includes a complete conversational web interface built in Streamlit for demonstrations, quick tests, and operational use by credit and underwriting teams:

make web
# ou: streamlit run app.py

Access it in your browser: http://localhost:8501

✨ Main Features of the Web Interface:

• 💬 Natural Language Chat: Ask free-form questions about customers, simulations, or portfolios in Portuguese.

• ⚡ Quick Actions (All 5 Risk Bands): Instantly load representative profiles of each band with 1 click:

  • 🟢 1. Very Low (<5%): Prime Customer (high income, score 910, limit usage 10%).

  • 🟢 2. Low (5–15%): Healthy Customer (score 810, limit usage 25%, 0 defaults).

  • 🟡 3. Moderate (15–35%): Borderline Customer (score 580, limit usage 50%, no late payments).

  • 🔴 4. High (35–60%): Alert Customer (score 580, limit usage 50%, 1 recent default).

  • ⛔ 5. Very High (≥60%): Critical Customer (score 544, limit usage 73%, 2 defaults).

• 🛠️ Suggested Query Grid:

  • 📊 Technical Sheet: Displays validation metrics, ROC-AUC, PR-AUC, and accuracy.

  • 📁 CSV Portfolio: Evaluates entire portfolios with vectorized scoring of 11,000 customers in 0.7s, calculating the Expected Loss (R$) and total exposure.

  • 📉 What-If Simulation: Simulate limit reductions (30%), debt settlement, or score increases (+150 points).

  • 🔬 SHAP Audit: Ranking and bar charts with the biggest credit risk drivers.

• 💡 Expandable Guides for Laypeople: Each response contains a didactic caption explaining the meaning of SHAP charts, probability deltas, and loss provisioning.

🔌 Option 2: MCP Server (Claude Desktop / Claude Code)

# 1. Instalar dependências
pip install -r requirements.txt --break-system-packages   # ou use um venv

# 2. Treinar o modelo (gera models/*.joblib e model_metadata.json)
python -m src.train

# 3. (Opcional) Gerar os gráficos de avaliação em reports/figures/
python -m src.evaluate

# 4. Rodar os testes
pytest -v

# 5. Subir o servidor MCP (stdio)
python -m mcp_server.server

Connect to Claude Desktop / Claude Code

Copy mcp_server/claude_desktop_config.example.json to the MCP configuration file of your client, adjusting the absolute paths:

{
  "mcpServers": {
    "agent-risk-ai": {
      "command": "python",
      "args": ["-m", "mcp_server.server"],
      "cwd": "/caminho/absoluto/para/agent-risk-ai",
      "env": { "PYTHONPATH": "/caminho/absoluto/para/agent-risk-ai" }
    }
  }
}

Restart the client and ask, for example: "Using the agent-risk-ai server, what is the risk of this customer: ..."

📁 Project structure

agent-risk-ai/
├── app.py                       # Interface Web Chat conversacional no navegador (Streamlit)
├── data/raw/                    # train.csv, test.csv, sample_submission.csv
├── src/
│   ├── config.py                 # caminhos, sementes, regras de negócio centralizadas
│   ├── data_processing.py        # limpeza (sentinelas, winsorização, PII)
│   ├── feature_engineering.py    # features de domínio (DTI, utilização, tenure...)
│   ├── pipeline.py                # ColumnTransformer sklearn (sem vazamento)
│   ├── train.py                   # baselines + Optuna + XGBoost + SHAP + persistência
│   ├── evaluate.py                # gera gráficos (ROC, PR, confusão, SHAP)
│   └── inference.py                # camada de predição reutilizada pelo MCP e Web Chat
├── mcp_server/
│   ├── server.py                   # servidor MCP com as 6 ferramentas
│   └── claude_desktop_config.example.json
├── models/                         # modelo treinado + metadados (gerado por train.py)
├── reports/figures/                 # gráficos de avaliação (gerado por evaluate.py)
├── tests/test_pipeline.py            # 7 testes unitários (pytest)
├── requirements.txt
├── Makefile
└── README.md

⚠️ Known limitations and next steps

Transparency about limitations is part of doing serious data science:

• LGD assumed at 100% in the expected loss calculation (portfolio_risk_summary) for simplicity — in production, this would come from historical recovery data.

• No drift monitoring — the natural next step would be to instrument predict_default with feature distribution logging over time.

• Probability calibration was not validated with CalibratedClassifierCV — the probabilities are discriminative (good for ranking risk), but may not be perfectly calibrated on an absolute scale.

• occupation_type = "Unknown" is the most frequent category (~31% of the base) and coincides with the retired/unemployed flag — a future refinement would be to break down this category.

🧠 Technical stack

Python 3.12 · pandas · scikit-learn · XGBoost · Optuna (Bayesian tuning via TPE) · SHAP (explainability) · matplotlib · pytest · MCP Python SDK