Kibana MCP Server

# 📊 KIBANA LOGS TOOL - AI USAGE SPECIFICATION ## 🚨 MANDATORY REQUIREMENTS ### 📋 STEP 0: INITIAL USER INTERACTION (FIRST STEP - ALWAYS REQUIRED) 1. **ASK USER: "Do you want to debug using Periscope or Kibana?"** - This determines which workflow to follow - User response will branch to either Periscope or Kibana workflow - **CRITICAL:** Wait for user response before proceeding 2. **CREATE MEMORY BOARD (MANDATORY SECOND STEP)** - **Before any analysis**, create an investigation board: `POST /api/memory/create` - Name it descriptively: `{"name": "RCA: [Issue Description]"}` - Store the returned `board_id` for the entire session - **Use this board throughout** to track findings, critical points, patterns - Check board when needed: `GET /api/memory/{board_id}` for context ### 🔧 CONFIGURATION SETUP (IF DEFAULTS NOT SUITABLE) 3. **SETUP DYNAMIC CONFIGURATION (OPTIONAL)** - Before proceeding to any other steps, if the default server configurations are not suitable for the user's environment or task, you MUST use the `/api/set_config` endpoint to adjust them. - SAY TO USER WHAT ALL CONFIGS TO SET - This allows tailoring the server's behavior (e.g., target Kibana instance, log processing parameters) without manual `config.yaml` edits or server restarts. - **Endpoint:** `POST /api/set_config` - **Request Body Example:** ```json { "key_path": "elasticsearch.host", "value": "your-company-kibana.example.com" } ``` - **Commonly Used Configuration Keys (key_path: value_type - example_value):** - `elasticsearch.host`: string - e.g., "kibana.example.com" (Target Kibana host) - `elasticsearch.port`: integer - e.g., 443 (Target Kibana port) - `elasticsearch.timestamp_field`: string - e.g., "@timestamp", "timestamp", "start_time" (Primary timestamp field in logs) - `elasticsearch.verify_ssl`: boolean - e.g., true, false (Whether to verify SSL for Kibana connection) - `mcp_server.log_level`: string - e.g., "info", "debug", "error" (Server's own logging verbosity) - `processing.max_logs`: integer - e.g., 500 (Default max logs for `search_logs` if not specified in request) - `processing.default_time_range`: string - e.g., "1h", "24h", "7d" (Default time range for queries if not specified) - `processing.cache_ttl_seconds`: integer - e.g., 300 (Cache duration for repeated queries) - `processing.timeout_seconds`: integer - e.g., 60 (Timeout for Kibana queries) - `ai_response_settings.temperature`: float - e.g., 0.3 (Controls AI creativity, lower is more factual) - `ai_response_settings.max_tokens`: integer - e.g., 2000000 (Max tokens for AI generated responses) - `ai_response_settings.detailed_mode`: boolean - e.g., true (Enable more detailed AI responses) - `ai_providers.google_ai_api_key`: string - e.g., "AIza..." (Set Google AI API key; **Caution: handle API keys securely**) - `ai_providers.openai_api_key`: string - e.g., "sk-..." (Set OpenAI API key; **Caution: handle API keys securely**) - **Note:** Only set these if you understand their impact or if instructed by the user. Incorrect values might lead to errors. The server will use values from `config.yaml` if not dynamically overridden. ### 📌 GENERAL REQUIREMENTS 4. **ALWAYS request `session_id` from the user, if not given ask whether can i use order id to get id and proceed. if yes then do so** - this is required for all queries 5. **ALWAYS include `session_id` in queries** using appropriate format (KQL for Kibana, SQL for Periscope) 6. **ALWAYS use correct query language:** - **Kibana:** KQL (Kibana Query Language) - **Periscope:** SQL with special functions like `match_all()` 7. **ALWAYS use ISO format timestamps** (e.g., "2023-06-15T00:00:00Z") or relative time strings ("24h", "7d") 8. **Parse and present results clearly** in a structured format 9. **dont use summarize_logs by yourself, use only when user asks for log summary** 10. **Add findings to memory board** as you discover them throughout the investigation 11. **Check memory board** when you need context or want to track progress How to Perform Root Cause Analysis (RCA) Start with the Session ID: Begin the RCA process immediately after obtaining the session_id. Initial Log Extraction: Fetch the first 200 logs in ascending order using only the session_id as the query. This sets the baseline timeline. Understand the Anomaly: Analyze the user-reported issue and begin identifying patterns or anomalies in the initial log batch. Craft Intelligent Queries: Use insights from the codebase to construct advanced, context-aware queries that help narrow down potential causes. Correlate Logs with Code: As logs are fetched, map them to related functions, services, and error flows in the codebase to deepen understanding. Iterate & Converge: Refine your queries recursively, incorporating knowledge from previous logs and the codebase. Keep digging until you clearly identify the root cause. Optimize Query Strategy: ⚠️ Always choose the most effective query for faster convergence. Maintain clarity and precision at every step—accuracy is non-negotiable. ## 🔑 AUTHENTICATION SETUP (REQUIRED FIRST STEP) ```bash curl -X POST http://localhost:8000/api/set_auth_token \ -H "Content-Type: application/json" \ -d '{"auth_token":"YOUR_AUTH_TOKEN_HERE"}' ``` Example: ```bash curl -X POST http://localhost:8000/api/set_auth_token -H "Content-Type: application/json" -d '{"auth_token":"asdahduwdnsaiuefehasadflhy"}' ``` ## 🗂️ INDEX SELECTION (SECOND STEP FOR NEW USERS) 1. **Discover available indexes:** ```bash curl -X GET http://localhost:8000/api/discover_indexes ``` 2. **Set the index to use:** ```bash curl -X POST http://localhost:8000/api/set_current_index \ -H "Content-Type: application/json" \ -d '{"index_pattern": "YOUR_INDEX_PATTERN"}' ``` Example: ```bash curl -X POST http://localhost:8000/api/set_current_index \ -H "Content-Type: application/json" \ -d '{"index_pattern": "estio-logs-*"}' ``` 3. **When a user says they're from a specific company:** - First discover available indexes - Look for index patterns that might match their company name - Suggest setting the index that seems most appropriate - If unsure, ask the user to select from the available indexes 4. **Index preference persistence:** - Once set, the index preference will persist until changed - If a user switches companies or needs a different index, use set_current_index again To extract the **Session ID** using the **Order ID** from Kibana logs during the `callStartPayment` phase. ### 🔍 Step-by-Step Rule: 1. **Use the extract_session_id Endpoint (Preferred Method):** ```bash curl -X POST http://localhost:8000/api/extract_session_id \ -H "Content-Type: application/json" \ -d '{ "order_id": "YOUR_ORDER_ID_HERE" }' ``` * This endpoint uses Neurolink AI to automatically extract session IDs * It follows all the rules below and returns structured results 2. **Fallback Method (If extract_session_id Fails):** Use the search_logs endpoint with this query: ```kibana {Order ID} AND "callStartPayment" ``` * Replace `{Order ID}` with the actual order ID. * Use Kibana Query Language (KQL) in the query box. * max_results must always be 1-3 3. **Analyze Response Logs (For Fallback Method):** * Look for log lines that start with this pattern: ``` message:{num} | {some_id} | {session_id} | ``` * The line always starts with a prefix like `message:`num followed by two pipe-separated values. * The **third segment** (after the second `|`) is the **Session ID**. * max_results must always be 1-3 4. **IMPORTANT: If the extract_session_id endpoint fails, you MUST use the fallback method with search_logs** * Parse the logs manually using the pattern above * Extract the session ID from the third segment after the second pipe character * Proceed with the extracted session ID for subsequent queries --- ** Function based mode Begin ** this will be triggered by user saying search based on functions - if this mode is used : all your query text must have {session_id} AND ("FunctionCallResult" OR "FunctionCalled") + query terms - always use sort by timestamp and sort order asc in this mode ** Function based mode End. ** ## 📋 API ENDPOINTS REFERENCE | Endpoint | Primary Use Case | Key Parameters | |----------|------------------|----------------| | `/api/search_logs` | **MAIN ENDPOINT** - Find specific logs | `query_text`, `max_results`, time filters | | `/api/extract_session_id` | **EXTRACT SESSION ID** - Get session ID from order ID | `order_id` | | `/api/get_recent_logs` | View latest logs | `count`, `level` | | `/api/analyze_logs` | Identify patterns | `time_range`, `group_by` | | `/api/extract_errors` | Find errors | `hours`, `include_stack_traces`, `limit` | | `/api/summarize_logs` | 🧠 **AI-POWERED** - Generate intelligent log analysis | Same as `search_logs` + AI analysis | | `/api/discover_indexes` | List available indexes | None | | `/api/set_current_index` | Select index for searches | `index_pattern` | ## 🧠 AI-POWERED MEMORY BOARD The Memory Board is a stateful, short-term memory system that allows an AI agent to build context during a Root Cause Analysis (RCA) session. It helps the AI to track findings, avoid redundant work, and form more intelligent hypotheses. ### Endpoints | Endpoint | Method | Purpose | Key Parameters | |---|---|---|---| | `/api/memory/create` | POST | Creates a new, named investigation board. | `name` | | `/api/memory/all` | GET | Lists all active investigation boards (ID and name). | None | | `/api/memory/{board_id}` | GET | Retrieves all findings for a specific board. | `board_id` | | `/api/memory/{board_id}/add_finding`| POST | Adds a structured finding to a board. | `board_id`, finding data | | `/api/memory/{board_id}/clear` | POST | Clears and deletes an investigation board. | `board_id` | ### AI Workflow for Using the Memory Board 1. **Start Task & Check for Existing Work:** When a new RCA task begins, first call `GET /api/memory/all` to get a list of all active investigations. * Check if an investigation for the same issue already exists to avoid redundant work. * Check for related investigations that might provide context (e.g., a systemic error board). 2. **Create an Investigation Board:** If no relevant board exists, create a new one. * **Action:** Call `POST /api/memory/create`. * **Payload:** `{"name": "RCA for OrderID 12345 payment failure"}` * **Store the returned `board_id`** for all subsequent operations in this investigation. 3. **Add Findings as You Discover Them:** As you query logs and find important information (errors, anomalies, key events), add them to the board. * **Action:** Call `POST /api/memory/{board_id}/add_finding`. * **Payload:** A structured JSON object for the finding. ```json { "timestamp": "2023-10-27T10:05:15Z", "finding": "Payment failed due to 'Insufficient Funds' from Stripe.", "source_log": { "...raw log object..." }, "attention_weight": 8, "implication": "The issue is external, not a system bug. Verify the request sent to Stripe was correct." } ``` 4. **Consult the Board to Inform Next Steps:** Before each new query, get the current state of the investigation to make a more intelligent decision. * **Action:** Call `GET /api/memory/{board_id}`. * **Logic:** Review the existing findings and their implications to decide what to query for next. Avoid re-discovering facts. 5. **End the Investigation:** Once the RCA is complete and you have reported the final summary to the user: * **Action:** Call `POST /api/memory/{board_id}/clear` to clean up the in-memory board. ## 🔎 SEARCH_LOGS ENDPOINT - DETAILED SPECIFICATION ### Required Parameters: - `query_text`: String using KQL format - MUST include `{session_id}` + query terms - `max_results`: Integer (default: 100) - Number of results to return (adjust to your needs) ### Optional Parameters: - `start_time`/`end_time`: ISO timestamps (e.g., "2023-06-15T00:00:00Z") - `levels`: Array of log levels to include (e.g., ["ERROR", "WARN"]) - `include_fields`/`exclude_fields`: Arrays of field names - `sort_by`: Field to sort by - MUST use a valid timestamp field name (`timestamp`, `@timestamp`, or `start_time`) - `sort_order`: "asc" or "desc" (default: "desc") ### KQL Query Examples: ``` "{session_id} AND error" "{session_id} AND payment" "{session_id} AND verifyPaymentAttempt" "a71e84b1ee00f6247347 AND callStartPayment" ``` ### Complete Examples: #### 1. Basic Search with Session ID ```bash curl -X POST http://localhost:8000/api/search_logs \ -H "Content-Type: application/json" \ -d '{ "query_text": "{session_id} AND payment error", "max_results": 50 }' ``` #### 2. Search with Absolute Time Range ```bash curl -X POST http://localhost:8000/api/search_logs \ -H "Content-Type: application/json" \ -d '{ "query_text": "{session_id} AND payment", "start_time": "2023-06-15T00:00:00Z", "end_time": "2023-06-16T00:00:00Z" }' ``` #### 3. Search with Specific Transaction ```bash curl -X POST http://localhost:8000/api/search_logs \ -H "Content-Type: application/json" \ -d '{ "query_text": "{session_id} AND verifyPaymentAttempt", "max_results": 1 }' ``` #### 4. Search with Correct Sorting ```bash curl -X POST http://localhost:8000/api/search_logs \ -H "Content-Type: application/json" \ -d '{ "query_text": "{session_id} AND payment", "sort_by": "timestamp", "sort_order": "desc" }' ``` #### 5. Extract Session ID from Order ID ```bash curl -X POST http://localhost:8000/api/extract_session_id \ -H "Content-Type: application/json" \ -d '{ "order_id": "ORDER_12345" }' ``` ## 📊 OTHER ENDPOINTS - REFERENCE EXAMPLES ### Analyze Logs for Patterns ```bash curl -X POST http://localhost:8000/api/analyze_logs \ -H "Content-Type: application/json" \ -d '{ "time_range": "24h", "group_by": "level" }' ``` ### Fetch Recent Logs ```bash curl -X POST http://localhost:8000/api/get_recent_logs \ -H "Content-Type: application/json" \ -d '{ "count": 10, "level": "ERROR" }' ``` !CAUTION DONT USE THIS ENDPOINT WITHOUT USER EXPLICITLY ASKING FOR IT (summarize_logs) ### 🧠 AI-Powered Log Analysis (SUMMARIZE_LOGS) ```bash curl -X POST http://localhost:8000/api/summarize_logs \ -H "Content-Type: application/json" \ -d '{ "query_text": "{session_id} AND {other query}", "max_results": 50, "start_time": "2023-06-15T00:00:00Z", "end_time": "2023-06-16T00:00:00Z" }' ``` **What it does:** - Searches logs using the same parameters as `search_logs` - Uses AI (Neurolink) to generate intelligent analysis - Provides structured insights including: - Summary of activities and systems involved - Key insights and patterns - Errors and exceptions with details - Function calls and critical methods - Chronological flow of events - Anomalies and suspicious behavior - Focus areas needing attention - Recommendations for fixes **When to use:** - When you need to understand complex log patterns - For root cause analysis of issues - When dealing with large volumes of logs - For generating reports or summaries - When you need actionable insights from logs **Requirements:** - Requires Node.js and Neurolink (automatically set up when starting server) - Requires AI provider API key for full AI features. Set via environment variable (e.g., `export GOOGLE_AI_API_KEY=...`) OR in `config.yaml` under `ai_providers` (config values take precedence). Google AI Studio key is recommended (has a free tier). - May take longer than regular search due to AI processing ## 🔄 AI DECISION FLOW ### STEP 1: INITIAL SETUP 1. **Ask user: "Do you want to debug using Periscope or Kibana?"** - Wait for user's choice before proceeding 2. **Create Memory Board (MANDATORY)** ```bash curl -X POST http://localhost:8000/api/memory/create \ -H "Content-Type: application/json" \ -d '{"name": "RCA: [Issue Description]"}' ``` - Store the returned `board_id` for the entire session ### STEP 2: BRANCH BASED ON USER CHOICE #### 🔵 IF USER CHOOSES KIBANA: 3. **Set Kibana Auth Token** ```bash curl -X POST http://localhost:8000/api/set_auth_token \ -H "Content-Type: application/json" \ -d '{"auth_token":"USER_TOKEN"}' ``` 4. **Discover Available Indexes** ```bash curl -X GET http://localhost:8000/api/discover_indexes ``` 5. **Ask user which index to use** (based on discovered indexes) 6. **Set the selected index** ```bash curl -X POST http://localhost:8000/api/set_current_index \ -H "Content-Type: application/json" \ -d '{"index_pattern": "USER_SELECTED_INDEX"}' ``` 7. **Request `session_id` from user** (if not provided, offer to extract from order_id) 8. **Search logs using KQL** ```bash curl -X POST http://localhost:8000/api/search_logs \ -H "Content-Type: application/json" \ -d '{ "query_text": "{session_id} AND <query terms>", "max_results": 200, "sort_order": "asc" }' ``` 9. **Add findings to memory board as you discover them** ```bash curl -X POST http://localhost:8000/api/memory/{board_id}/add_finding \ -H "Content-Type: application/json" \ -d '{ "finding": "Description of finding", "attention_weight": 8, "implication": "What this means" }' ``` #### 🟢 IF USER CHOOSES PERISCOPE: 3. **Set Periscope Auth Token** ```bash curl -X POST http://localhost:8000/api/set_periscope_auth_token \ -H "Content-Type: application/json" \ -d '{"auth_token":"USER_PERISCOPE_TOKEN"}' ``` 4. **Discover Available Streams** ```bash curl -X GET http://localhost:8000/api/get_periscope_streams ``` 5. **Ask user which stream to use** (e.g., vayu_logs, envoy_logs, etc.) 6. **Get schema of selected stream (CRITICAL - USE THIS KNOWLEDGE)** ```bash curl -X POST http://localhost:8000/api/get_periscope_stream_schema \ -H "Content-Type: application/json" \ -d '{"stream_name": "USER_SELECTED_STREAM"}' ``` - **Study the schema:** field names, types, available columns - **Use this knowledge** when constructing SQL queries 7. **Request `session_id` from user** (if not provided, offer to extract from order_id) 8. **Search logs using SQL with `match_all()` for robustness** ```bash curl -X POST http://localhost:8000/api/search_periscope_logs \ -H "Content-Type: application/json" \ -d '{ "sql_query": "SELECT * FROM \"stream_name\" WHERE match_all('\''session_id'\'') ORDER BY _timestamp ASC", "start_time": "7d", "max_results": 200 }' ``` - **IMPORTANT:** Use `match_all()` function for session_id searches (handles leading spaces) - Alternative: Use `WHERE session_id = ' SESSION_ID'` (note the leading space) - See "PERISCOPE QUERY METHODS" section below for details 9. **Add findings to memory board as you discover them** ```bash curl -X POST http://localhost:8000/api/memory/{board_id}/add_finding \ -H "Content-Type: application/json" \ -d '{ "finding": "Description of finding", "attention_weight": 8, "implication": "What this means" }' ``` ### STEP 3: THROUGHOUT INVESTIGATION 10. **Check memory board when needed** for context and progress tracking ```bash curl -X GET http://localhost:8000/api/memory/{board_id} ``` 11. **Continue adding findings** as you discover: - Errors and exceptions - Critical log entries - Patterns and anomalies - Key business events - Performance issues - Any insights that help RCA 12. **Present results** in a clear, structured format to the user ## 🔭 PERISCOPE INTEGRATION - SQL-BASED LOG ANALYSIS ### 📊 Overview Periscope is a separate log analysis system that uses SQL queries instead of KQL. It provides: - SQL-based querying for logs - Stream-based organization (envoy_logs, vayu_logs, etc.) - Schema discovery with field types and statistics - Time-based filtering in microseconds ### 🔑 Authentication Setup (Required First Step) ```bash curl -X POST http://localhost:8000/api/set_periscope_auth_token \ -H "Content-Type: application/json" \ -d '{"auth_token":"YOUR_PERISCOPE_AUTH_TOKEN_HERE"}' ``` **Note:** The Periscope auth token is the base64-encoded `auth_tokens` cookie value from your Periscope session. ### 📋 Available Periscope Endpoints | Endpoint | Purpose | Key Parameters | |----------|---------|----------------| | `/api/set_periscope_auth_token` | Set Periscope authentication | `auth_token` | | `/api/get_periscope_streams` | List available log streams | `org_identifier` (optional) | | `/api/get_periscope_stream_schema` | Get schema for a specific stream | `stream_name`, `org_identifier` | | `/api/get_all_periscope_schemas` | Get schemas for all streams | `org_identifier` (optional) | | `/api/search_periscope_logs` | **MAIN ENDPOINT** - Search using SQL | `sql_query`, `start_time`, `end_time`, `max_results` | | `/api/search_periscope_errors` | Find error logs (4xx/5xx) | `hours`, `stream`, `error_codes` | ### 🔍 Step-by-Step Workflow #### 1. Discover Available Streams ```bash curl -X GET http://localhost:8000/api/get_periscope_streams ``` **Common Streams:** - `envoy_logs` - Envoy proxy access logs (HTTP requests/responses) - `vayu_logs` - Vayu service logs - `breeze_logs` - Breeze application logs - `lighthouse_logs` - Lighthouse service logs - `nimble_logs` - Nimble service logs - `istio_logs` - Istio service mesh logs - `squid_logs` - Squid proxy logs #### 2. Get Schema for a Stream ```bash curl -X POST http://localhost:8000/api/get_periscope_stream_schema \ -H "Content-Type: application/json" \ -d '{ "stream_name": "envoy_logs" }' ``` **Returns:** - Field names and types (e.g., `status_code: Utf8`, `timestamp: Int64`) - Statistics (doc count, time range, storage size) - Settings (retention, partitioning) #### 3. Search Logs with SQL ```bash curl -X POST http://localhost:8000/api/search_periscope_logs \ -H "Content-Type: application/json" \ -d '{ "sql_query": "SELECT * FROM \"envoy_logs\" WHERE status_code LIKE '\''5%'\''", "start_time": "24h", "max_results": 50 }' ``` ### 📝 SQL Query Syntax **Basic Query Structure:** ```sql SELECT * FROM "stream_name" WHERE condition ``` **Important:** - Stream names must be in double quotes: `"envoy_logs"` - String literals use single quotes: `'value'` - In curl commands, escape quotes: `'\''value'\''` **Common Query Patterns:** 1. **Search by Status Code:** ```sql SELECT * FROM "envoy_logs" WHERE status_code LIKE '5%' ``` 2. **Search by Method and Path:** ```sql SELECT * FROM "envoy_logs" WHERE method = 'POST' AND path LIKE '/api%' ``` 3. **Search by Session ID:** ```sql SELECT * FROM "vayu_logs" WHERE session_id = 'your-session-id' ``` 4. **Search by Time Range and Condition:** ```sql SELECT * FROM "envoy_logs" WHERE status_code >= 400 AND method = 'GET' ``` 5. **Limit Results:** ```sql SELECT * FROM "envoy_logs" WHERE status_code = '500' LIMIT 10 ``` ### ⏰ Time Format Options & Timezone Support Periscope uses **microseconds** since epoch internally, but the API accepts flexible formats with **full timezone support**: 1. **Relative Time:** `"24h"`, `"1d"`, `"7d"`, `"1w"`, `"1m"` 2. **ISO Timestamps with Timezone:** - UTC: `"2025-10-04T04:50:00Z"` or `"2025-10-04T04:50:00+00:00"` - IST: `"2025-10-04T10:20:00+05:30"` - US Eastern: `"2025-10-04T00:50:00-05:00"` 3. **Naive Datetime with Timezone Parameter:** - `"2025-10-04 10:20:00"` with `"timezone": "Asia/Kolkata"` - `"2025-10-04T10:20:00"` with `"timezone": "Asia/Kolkata"` 4. **Microseconds:** `1759527600000000` **🌍 NEW TIMEZONE FEATURE:** You can now specify times in any timezone and the system will automatically convert to UTC microseconds: ```bash # Using IST (India Standard Time) directly - NO manual UTC conversion needed! curl -X POST http://localhost:8000/api/search_periscope_logs \ -H "Content-Type: application/json" \ -d '{ "sql_query": "SELECT COUNT(*) FROM \"envoy_logs\"", "start_time": "2025-10-04 10:20:00", "end_time": "2025-10-04 11:00:00", "timezone": "Asia/Kolkata" }' # Alternative: ISO format with timezone offset (no timezone parameter needed) curl -X POST http://localhost:8000/api/search_periscope_logs \ -H "Content-Type: application/json" \ -d '{ "sql_query": "SELECT COUNT(*) FROM \"envoy_logs\"", "start_time": "2025-10-04T10:20:00+05:30", "end_time": "2025-10-04T11:00:00+05:30" }' ``` **Common Timezone Names:** - India: `"Asia/Kolkata"` or `"Asia/Calcutta"` - UTC: `"UTC"` - US Eastern: `"US/Eastern"` or `"America/New_York"` - US Pacific: `"US/Pacific"` or `"America/Los_Angeles"` - Singapore: `"Asia/Singapore"` - UK: `"Europe/London"` **Examples:** ```bash # Last 24 hours "start_time": "24h" # Last 7 days "start_time": "7d" # Specific date range in UTC "start_time": "2025-10-01T00:00:00Z", "end_time": "2025-10-03T00:00:00Z" # Specific date range in IST using timezone parameter "start_time": "2025-10-04 10:20:00", "end_time": "2025-10-04 11:00:00", "timezone": "Asia/Kolkata" # Microseconds (always UTC) "start_time": 1759440000000000, "end_time": 1759527600000000 ``` **IMPORTANT NOTES:** - If `timezone` parameter is provided with naive datetime, it will be applied - If datetime has explicit timezone (like `+05:30`), timezone parameter is ignored - Naive datetimes without timezone parameter default to UTC - All conversions result in UTC microseconds internally - The `_timestamp` field in logs may be in local server time (check your specific logs) ### 🔍 PERISCOPE QUERY METHODS (CRITICAL) **⚠️ IMPORTANT: `match_all()` is PERISCOPE-ONLY, NOT for Kibana!** #### Problem: Leading Spaces in Field Values Many Periscope streams (especially `vayu_logs`) have **leading spaces** in field values: ```json { "session_id": " McErP7Cg2jG8X_Slg2foh", // Note the space before 'M'! "device_id": " h2-BsSfwasVTRHUnAOdwo", // Space before 'h' "shop_url": " https://artociti.myshopify.com" // Space before 'h' } ``` This causes **direct equality queries to fail**: ```sql -- ❌ FAILS: Returns 0 results WHERE session_id = 'McErP7Cg2jG8X_Slg2foh' ``` #### 3 Solutions (Tested & Verified) All three methods below successfully return **2,016 logs** for test session `McErP7Cg2jG8X_Slg2foh`: **Method 1: Include Leading Space (FASTEST)** ```sql SELECT * FROM "vayu_logs" WHERE session_id = ' McErP7Cg2jG8X_Slg2foh' ↑ Note the space after quote! ``` - ✅ **Pros:** Fastest performance, exact match - ❌ **Cons:** Must remember to add space, not obvious from error messages - **Use when:** You know the field has leading spaces **Method 2: Use LIKE Wildcard** ```sql SELECT * FROM "vayu_logs" WHERE session_id LIKE '%McErP7Cg2jG8X_Slg2foh%' ``` - ✅ **Pros:** Handles any whitespace, works without knowing exact format - ❌ **Cons:** Slower on large datasets (full scan), may match partial IDs - **Use when:** Uncertain about whitespace, small datasets, debugging **Method 3: Use match_all() Full-Text Search (RECOMMENDED)** ```sql SELECT * FROM "vayu_logs" WHERE match_all('McErP7Cg2jG8X_Slg2foh') ``` - ✅ **Pros:** Robust, searches across ALL fields, handles whitespace - ✅ **Pros:** More forgiving, good for exploratory searches - ⚠️ **Cons:** Searches all fields (may match unintended data), slightly slower than Method 1 - **Use when:** Not sure which field contains the value, want maximum coverage #### Performance Comparison | Method | Query Time | Results | Best For | |--------|-----------|---------|----------| | **1. With space** | ~26s | 2,016 | Known field, exact match | | **2. LIKE** | ~30s | 2,016 | Uncertain whitespace | | **3. match_all()** | ~29s | 2,016 | **Exploratory search, maximum robustness** | #### Recommendation **For session_id queries:** 1. **First try:** `match_all('SESSION_ID')` (most robust, works across fields) 2. **If you need speed:** `session_id = ' SESSION_ID'` (with leading space) 3. **For debugging:** `session_id LIKE '%SESSION_ID%'` (handles any format) #### Complete Examples ```bash # Method 1: With leading space curl -X POST http://localhost:8000/api/search_periscope_logs \ -H "Content-Type: application/json" \ -d '{ "sql_query": "SELECT * FROM \"vayu_logs\" WHERE session_id = '\'' McErP7Cg2jG8X_Slg2foh'\'' ORDER BY _timestamp ASC", "start_time": "7d", "max_results": 200 }' # Method 2: Using LIKE curl -X POST http://localhost:8000/api/search_periscope_logs \ -H "Content-Type: application/json" \ -d '{ "sql_query": "SELECT * FROM \"vayu_logs\" WHERE session_id LIKE '\''%McErP7Cg2jG8X_Slg2foh%'\''", "start_time": "7d", "max_results": 200 }' # Method 3: Using match_all() (RECOMMENDED) curl -X POST http://localhost:8000/api/search_periscope_logs \ -H "Content-Type: application/json" \ -d '{ "sql_query": "SELECT * FROM \"vayu_logs\" WHERE match_all('\''McErP7Cg2jG8X_Slg2foh'\'') ORDER BY _timestamp ASC", "start_time": "7d", "max_results": 200 }' ``` **Remember:** `match_all()` is a **Periscope-specific** SQL function. Do NOT use it with Kibana/Elasticsearch (use KQL instead). ### 🔴 Finding Errors - Convenience Function For quickly finding 4xx/5xx errors: ```bash curl -X POST http://localhost:8000/api/search_periscope_errors \ -H "Content-Type: application/json" \ -d '{ "hours": 24, "stream": "envoy_logs", "error_codes": "5%" }' ``` **Parameters:** - `hours`: Number of hours to look back (default: 24) - `stream`: Stream name (default: "envoy_logs") - `error_codes`: Pattern like `"4%"` (4xx), `"5%"` (5xx), or `null` for all >=400 - `org_identifier`: Organization (default: "default") ### 📊 Complete Examples #### Example 1: Find All 5xx Errors in Last 24 Hours ```bash curl -X POST http://localhost:8000/api/search_periscope_errors \ -H "Content-Type: application/json" \ -d '{ "hours": 24, "stream": "envoy_logs", "error_codes": "5%" }' ``` #### Example 2: Search for Specific Endpoint Errors ```bash curl -X POST http://localhost:8000/api/search_periscope_logs \ -H "Content-Type: application/json" \ -d '{ "sql_query": "SELECT * FROM \"envoy_logs\" WHERE path LIKE '\''/api/payment%'\'' AND status_code >= 400", "start_time": "24h", "max_results": 100 }' ``` #### Example 3: Analyze Slow Requests ```bash curl -X POST http://localhost:8000/api/search_periscope_logs \ -H "Content-Type: application/json" \ -d '{ "sql_query": "SELECT * FROM \"envoy_logs\" WHERE duration_ms > 1000 ORDER BY duration_ms DESC LIMIT 50", "start_time": "24h" }' ``` #### Example 4: Find Requests by Session ID ```bash curl -X POST http://localhost:8000/api/search_periscope_logs \ -H "Content-Type: application/json" \ -d '{ "sql_query": "SELECT * FROM \"vayu_logs\" WHERE session_id = '\''abc-123-def'\''", "start_time": "7d", "max_results": 200 }' ``` #### Example 5: Get All Schemas for Reference ```bash curl -X GET http://localhost:8000/api/get_all_periscope_schemas ``` ### 🎯 Common Use Cases **1. Error Analysis:** - Find all 5xx errors: `WHERE status_code LIKE '5%'` - Find specific error: `WHERE status_code = '500'` - Find client errors: `WHERE status_code LIKE '4%'` **2. Performance Analysis:** - Slow requests: `WHERE duration_ms > 1000` - Fast requests: `WHERE duration_ms < 100` - Average duration: Use aggregation functions **3. Traffic Analysis:** - By endpoint: `WHERE path LIKE '/api/specific%'` - By method: `WHERE method = 'POST'` - By user agent: `WHERE user_agent LIKE '%Chrome%'` **4. Session Tracking:** - All session logs: `WHERE session_id = 'session-id'` - Session with errors: `WHERE session_id = 'session-id' AND status_code >= 400` ### 🔄 Periscope vs Kibana Decision Flow **Use Periscope when:** - ✅ You need SQL-based querying - ✅ Analyzing HTTP access logs (envoy_logs) - ✅ Need schema information and field types - ✅ Working with structured log data - ✅ Need precise time-based filtering **Use Kibana when:** - ✅ You need KQL-based querying - ✅ Working with application logs - ✅ Need session-based tracking - ✅ Following existing Kibana workflows - ✅ Need AI-powered summarization ### ⚠️ Important Notes 1. **Stream Names:** Always use double quotes around stream names in SQL 2. **String Values:** Use single quotes for string literals in WHERE clauses 3. **Time Range:** Periscope uses microseconds internally but accepts flexible formats 4. **Default Org:** Most queries use `org_identifier: "default"` unless specified 5. **Field Names:** Check schema first to know available fields and types 6. **Escaping:** In curl commands with bash, escape single quotes as `'\''` 7. **🚨 CRITICAL: Leading Spaces in `vayu_logs` Fields** - All field values in `vayu_logs` stream have leading spaces: ```json { "session_id": " _58ZElUxpcb86PNSbUjnZ", // Note the space before underscore! "device_id": " hX5GdI0KqkTS5FDonNZPB", // Space before 'h' "shop_url": " https://5d818d.myshopify.com" // Space before 'h' } ``` **Impact:** Exact match queries will fail if you don't include the leading space: - ❌ Wrong: `WHERE session_id = 'jxz6F5eKlFjs2yrxD18h2'` (returns 0 results) - ✅ Correct Option 1: `WHERE session_id = ' jxz6F5eKlFjs2yrxD18h2'` (with leading space) - ✅ Correct Option 2: `WHERE session_id LIKE '%jxz6F5eKlFjs2yrxD18h2%'` (wildcard) - ✅ Correct Option 3: `WHERE match_all('jxz6F5eKlFjs2yrxD18h2')` (RECOMMENDED - most robust) **📖 See "PERISCOPE QUERY METHODS" section above for detailed comparison, performance metrics, and complete examples of all 3 methods.** ### 🧪 Testing Your Setup 1. **Test Authentication:** ```bash curl -X POST http://localhost:8000/api/set_periscope_auth_token \ -H "Content-Type: application/json" \ -d '{"auth_token":"your-token"}' ``` 2. **List Streams:** ```bash curl -X GET http://localhost:8000/api/get_periscope_streams ``` 3. **Get Schema:** ```bash curl -X POST http://localhost:8000/api/get_periscope_stream_schema \ -H "Content-Type: application/json" \ -d '{"stream_name": "envoy_logs"}' ``` 4. **Search for Errors:** ```bash curl -X POST http://localhost:8000/api/search_periscope_errors \ -H "Content-Type: application/json" \ -d '{"hours": 1, "stream": "envoy_logs", "error_codes": "5%"}' ``` ### 🔍 Complete RCA Example Using Timezone Feature **Scenario:** CPU spike alert at 10:45 AM IST on Oct 4, 2025. Investigate 10:20-11:00 AM IST window. **Step 1: Set Authentication** ```bash curl -X POST http://localhost:8000/api/set_periscope_auth_token \ -H "Content-Type: application/json" \ -d '{"auth_token":"your-periscope-token"}' ``` **Step 2: Get Total Traffic (Using NEW Timezone Feature)** ```bash curl -X POST http://localhost:8000/api/search_periscope_logs \ -H "Content-Type: application/json" \ -d '{ "sql_query": "SELECT COUNT(*) as total_logs FROM \"envoy_logs\"", "start_time": "2025-10-04 10:20:00", "end_time": "2025-10-04 11:00:00", "timezone": "Asia/Kolkata", "max_results": 1 }' ``` **Step 3: Find Peak Traffic Minutes** ```bash curl -X POST http://localhost:8000/api/search_periscope_logs \ -H "Content-Type: application/json" \ -d '{ "sql_query": "SELECT COUNT(*) as req_per_minute, CAST(_timestamp / 60000000 AS BIGINT) as minute_bucket FROM \"envoy_logs\" GROUP BY minute_bucket ORDER BY req_per_minute DESC LIMIT 5", "start_time": "2025-10-04 10:20:00", "end_time": "2025-10-04 11:00:00", "timezone": "Asia/Kolkata", "max_results": 5 }' ``` **Step 4: Analyze Failures** ```bash curl -X POST http://localhost:8000/api/search_periscope_logs \ -H "Content-Type: application/json" \ -d '{ "sql_query": "SELECT COUNT(*) as failure_count, upstream_host FROM \"envoy_logs\" WHERE response_flags = '\''DC'\'' GROUP BY upstream_host ORDER BY failure_count DESC", "start_time": "2025-10-04 10:20:00", "end_time": "2025-10-04 11:00:00", "timezone": "Asia/Kolkata", "max_results": 10 }' ``` **Step 5: Error Breakdown** ```bash curl -X POST http://localhost:8000/api/search_periscope_logs \ -H "Content-Type: application/json" \ -d '{ "sql_query": "SELECT COUNT(*) as error_count, response_flags, status_code FROM \"envoy_logs\" WHERE status_code >= '\''400'\'' OR status_code = '\''0'\'' GROUP BY response_flags, status_code ORDER BY error_count DESC", "start_time": "2025-10-04 10:20:00", "end_time": "2025-10-04 11:00:00", "timezone": "Asia/Kolkata", "max_results": 10 }' ``` **Benefits of Using Timezone Parameter:** - ✅ No manual UTC conversion required - ✅ Search using local business hours (IST) - ✅ Easier to correlate with alerts and monitoring dashboards - ✅ Reduces human error in timezone calculations - ✅ Works with any IANA timezone name ## 📌 IMPORTANT NOTES - Always replace `{session_id}` with the actual session ID in queries - Always use ISO format timestamps (e.g., "2023-06-15T00:00:00Z") instead of relative time strings - Include relevant context in queries to narrow results - Use field filters (`include_fields` and `exclude_fields`) for large result sets - Set appropriate `max_results` to balance detail vs. performance - **For Periscope:** Use SQL syntax and check stream schemas first - **For Periscope:** Time values can be relative ("24h"), ISO, or microseconds ## ⚠️ TROUBLESHOOTING COMMON ERRORS ### Timestamp Field Issues - When using `sort_by`, you MUST use a valid timestamp field that exists in the indices - Different indices use different field names for timestamps: - `timestamp` - Used in breeze-logs source - `@timestamp` - Standard Elasticsearch default - `start_time` - Used in main elasticsearch config - If you see "No mapping found for [field] in order to sort on" error: 1. Try using a different timestamp field name (`timestamp`, `@timestamp`, or `start_time`) 2. The server will automatically retry without sorting if it encounters this error 3. You can omit the `sort_by` parameter to let the server use its default sorting logic ### Example with Correct Timestamp Field ```bash # Correct - Using a valid timestamp field: curl -X POST http://localhost:8000/api/search_logs \ -H "Content-Type: application/json" \ -d '{ "query_text": "{session_id} AND error", "sort_by": "timestamp", "sort_order": "desc" }' ```