🔍 Kibana MCP (Model Context Protocol) Server

A powerful, high-performance server that provides seamless access to Kibana and Periscope logs through a unified API. Built with modular architecture, in-memory caching, HTTP/2 support, and OpenTelemetry tracing.

📋 Table of Contents

• Overview

• Features

• What's New in v2.0.0

• Setup

  • Prerequisites

  • Installation

  • Configuration

• Authentication

• Running the Server

• API Reference

• Available Indexes

• Example Usage

• Troubleshooting

• Performance Features

• Architecture

• AI Integration

• License

🌟 Overview

This project bridges the gap between your applications and Kibana/Periscope logs by providing:

1. Modular Architecture: Clean separation of concerns with dedicated modules for clients, services, and API layers

2. Dual Interface Support: Both Kibana (KQL) and Periscope (SQL) querying

3. Multi-Index Access: Query across 9 different log indexes (1.3+ billion logs)

4. Performance Optimized: In-memory caching, HTTP/2, and connection pooling

5. Timezone-Aware: Full support for international timezones (IST, UTC, PST, etc.)

6. Production-Ready: Comprehensive error handling, retry logic, and observability

✨ Features

Core Features

• Simple API: Easy-to-use RESTful endpoints for log searching and analysis

• Dual Log System Support:

  • Kibana: KQL-based querying for application logs

  • Periscope: SQL-based querying for HTTP access logs

• Multi-Index Support: Access to 9 indexes with 1.3+ billion logs

• Flexible Authentication: API-based token management for both Kibana and Periscope

• Time-Based Searching: Absolute and relative time ranges with full timezone support

• Real-Time Streaming: Monitor logs as they arrive

Performance Features (New in v2.0.0)

• ⚡ In-Memory Caching:

  • Schema cache: 1 hour TTL

  • Search cache: 5 minutes TTL

• 🚀 HTTP/2 Support: Multiplexed connections for faster requests

• 🔄 Connection Pooling: 200 max connections, 50 keepalive

• 📊 OpenTelemetry Tracing: Distributed tracing for monitoring and debugging

• 🌍 Timezone-Aware: Support for any IANA timezone without manual UTC conversion

AI & Analysis Features

• 🧠 AI-Powered Analysis: Intelligent log summarization using Neurolink

• Smart Chunking: Automatic handling of large log sets

• Pattern Analysis: Tools to identify log patterns and extract errors

• Cross-Index Correlation: Track requests across multiple log sources

🆕 What's New in v2.0.0

Modular Architecture

• ✅ Clean separation: clients/, services/, api/, models/, utils/

• ✅ Improved testability and maintainability

• ✅ Better error handling and logging

• ✅ Type-safe with Pydantic models

Performance Enhancements

• ✅ In-memory caching reduces API calls

• ✅ HTTP/2 support for better throughput

• ✅ Connection pooling for efficiency

• ✅ OpenTelemetry tracing for observability

Multi-Index Support

• ✅ 9 indexes accessible (7 with active data)

• ✅ 1.3+ billion logs available

• ✅ Index discovery and selection API

• ✅ Universal timestamp field compatibility

Enhanced Timezone Support

• ✅ Periscope queries with timezone parameter

• ✅ No manual UTC conversion needed

• ✅ Support for IST, UTC, PST, and all IANA timezones

Configuration Improvements

• ✅ Optimized config.yaml (36% smaller)

• ✅ Dynamic configuration via API

• ✅ Only essential parameters included

🚀 Setup

Prerequisites

• Python 3.8+

• Access to Kibana instance (for Kibana features)

• Access to Periscope instance (optional, for Periscope features)

• Authentication tokens for the services you want to use

Installation

1. Clone this repository:

   git clone https://github.com/gaharivatsa/KIBANA_SERVER.git cd KIBANA_SERVER

2. Create a virtual environment:

   python -m venv KIBANA_E # On macOS/Linux source KIBANA_E/bin/activate # On Windows KIBANA_E\Scripts\activate

3. Install dependencies:

   pip install -r requirements.txt

4. Make the start script executable:

   chmod +x ./run_kibana_mcp.sh

5. Optional: Set up AI-powered log analysis:

   # Install Node.js if not already installed (required for Neurolink) # Visit https://nodejs.org/ or use your package manager # Set your AI provider API key export GOOGLE_AI_API_KEY="your-google-ai-api-key" # Recommended (free tier) # OR export OPENAI_API_KEY="your-openai-key" # Neurolink will be automatically set up when you start the server

Configuration

The server comes with an optimized config.yaml that works out of the box. Key settings:

Dynamic Configuration (optional):

🔐 Authentication

Kibana Authentication

Set via API (Recommended):

How to Get Your Token:

1. Log in to Kibana in your browser

2. Open developer tools (F12)

3. Go to Application → Cookies

4. Find the authentication cookie (e.g., JWT token)

5. Copy the complete value

Periscope Authentication

How to Get Periscope Token:

1. Log in to Periscope in your browser

2. Open developer tools (F12)

3. Go to Application → Cookies

4. Find the auth_tokens cookie

5. Copy its value (base64 encoded)

🖥️ Running the Server

Start the server:

The server will be available at http://localhost:8000

Health Check:

Response:

📡 API Reference

Kibana Endpoints

Periscope Endpoints

Utility Endpoints

🗂️ Available Indexes

The server provides access to 9 log indexes (7 with active data):

Active Indexes

Empty Indexes

• wallet-product-v2* - No data

• core-ledger-v2* - No data

Total: ~1.3 Billion logs across all indexes

📝 Example Usage

1. Discover and Set Index

2. Search Logs (Kibana)

Basic Search:

Search with Time Range (Timezone-Aware):

Session-Based Search:

3. Search Periscope Logs (SQL)

Find 5XX Errors:

Search with Timezone (NEW!):

Quick Error Search:

4. AI-Powered Analysis

Response (example):

5. Cross-Index Correlation

Track a request across multiple indexes:

🔧 Troubleshooting

Common Issues

1. Timestamp Field Errors

Problem: "No mapping found for [timestamp] in order to sort on"

Solution: The server uses timestamp field which works for all indexes. This error should not occur in v2.0.0.

If you see it:

2. Authentication Errors (401)

Problem: "Unauthorized" or "Invalid token"

Solution:

• Token expired - get a fresh token from browser

• Re-authenticate using /api/set_auth_token

3. No Results Returned

Checklist:

1. ✅ Is the correct index set?

2. ✅ Is the time range correct?

3. ✅ Try a broader query ("*")

4. ✅ Check timezone offset

4. Slow Queries

Solutions:

• Reduce max_results

• Narrow time range

• Add specific query terms

• Check if caching is working (should be faster on repeated queries)

Testing

⚡ Performance Features

In-Memory Caching

Automatic caching reduces load on backend systems:

• Schema Cache: 1 hour TTL (Periscope stream schemas)

• Search Cache: 5 minutes TTL (recent queries)

Benefits:

• Faster repeated queries

• Reduced API calls

• Lower backend load

HTTP/2 Support

• Multiplexed connections

• Faster concurrent requests

• Better throughput for parallel queries

Connection Pooling

• Max connections: 200

• Keepalive connections: 50

• Efficient connection reuse

• Reduced latency

OpenTelemetry Tracing

• Distributed request tracing

• Performance monitoring

• Debug distributed issues

• Track request flow across components

🏗️ Architecture

Modular Structure

Legacy vs Modular

🤖 AI Integration

For AI Assistants

Use the provided AI_rules.txt for generic product documentation or AI_rules_file.txt for company-specific usage.

Key Requirements:

• ✅ Always authenticate first

• ✅ Discover and set index before searching

• ✅ Use timestamp field for sorting

• ✅ Include session_id in queries when tracking sessions

• ✅ Use ISO timestamps with timezone

Example AI Workflow

1. Authenticate:

   POST /api/set_auth_token

2. Discover Indexes:

   GET /api/discover_indexes

3. Set Index:

   POST /api/set_current_index

4. Search Logs:

   POST /api/search_logs

5. Analyze (Optional):

   POST /api/summarize_logs

For complete AI integration instructions, refer to AI_rules.txt (generic) or AI_rules_file.txt (company-specific).

📚 Documentation

• AI_rules.txt - Generic product usage guide

• AI_rules_file.txt - Company-specific usage (internal)

• CONFIG_USAGE_ANALYSIS.md - Configuration reference (deleted, info in this README)

• KIBANA_INDEXES_COMPLETE_ANALYSIS.md - Index details (deleted, info in this README)

🔄 Migration from v1.x

If upgrading from v1.x:

1. Update imports: Change from kibana_mcp_server.py to main.py

2. Update config: Remove unused parameters (see config.yaml)

3. Update queries: Use timestamp field instead of @timestamp or start_time

4. Test endpoints: All endpoints remain compatible

5. Enjoy performance: Automatic caching and HTTP/2 benefits

📊 Performance Benchmarks

• Cache Hit Rate: ~80% for repeated queries

• Response Time: 30-50% faster with HTTP/2

• Connection Reuse: 90%+ with pooling

• Memory Usage: <200MB with full cache

🤝 Contributing

This is a proprietary project. For issues or feature requests, contact the maintainers.

📜 License

This project is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License (CC BY-NC-ND 4.0).

This license requires that reusers:

• ✅ Give appropriate credit (Attribution)

• ❌ Do not use for commercial purposes (NonCommercial)

• ❌ Do not distribute modified versions (NoDerivatives)

For more information, see the LICENSE file.

Version: 2.0.0 (Modular)
Last Updated: October 2025
Total Logs: 1.3+ Billion
Indexes: 9 (7 active)
Status: Production Ready ✅