AI_rules_file.txt•30.4 kB
# 📊 KIBANA LOGS TOOL - AI USAGE SPECIFICATION
## 🚨 MANDATORY REQUIREMENTS
1. **SETUP DYNAMIC CONFIGURATION (FIRST STEP - MANDATORY IF DEFAULTS ARE NOT SUITABLE)**
- Before proceeding to any other steps, if the default server configurations are not suitable for the user's environment or task, you MUST use the `/api/set_config` endpoint to adjust them.
- SAY TO USER WHAT ALL CONFIGS TO SET
- This allows tailoring the server's behavior (e.g., target Kibana instance, log processing parameters) without manual `config.yaml` edits or server restarts.
- **Endpoint:** `POST /api/set_config`
- **Request Body Example:**
```json
{
"key_path": "elasticsearch.host",
"value": "your-company-kibana.example.com"
}
```
- **Commonly Used Configuration Keys (key_path: value_type - example_value):**
- `elasticsearch.host`: string - e.g., "kibana.example.com" (Target Kibana host)
- `elasticsearch.port`: integer - e.g., 443 (Target Kibana port)
- `elasticsearch.timestamp_field`: string - e.g., "@timestamp", "timestamp", "start_time" (Primary timestamp field in logs)
- `elasticsearch.verify_ssl`: boolean - e.g., true, false (Whether to verify SSL for Kibana connection)
- `mcp_server.log_level`: string - e.g., "info", "debug", "error" (Server's own logging verbosity)
- `processing.max_logs`: integer - e.g., 500 (Default max logs for `search_logs` if not specified in request)
- `processing.default_time_range`: string - e.g., "1h", "24h", "7d" (Default time range for queries if not specified)
- `processing.cache_ttl_seconds`: integer - e.g., 300 (Cache duration for repeated queries)
- `processing.timeout_seconds`: integer - e.g., 60 (Timeout for Kibana queries)
- `ai_response_settings.temperature`: float - e.g., 0.3 (Controls AI creativity, lower is more factual)
- `ai_response_settings.max_tokens`: integer - e.g., 2000000 (Max tokens for AI generated responses)
- `ai_response_settings.detailed_mode`: boolean - e.g., true (Enable more detailed AI responses)
- `ai_providers.google_ai_api_key`: string - e.g., "AIza..." (Set Google AI API key; **Caution: handle API keys securely**)
- `ai_providers.openai_api_key`: string - e.g., "sk-..." (Set OpenAI API key; **Caution: handle API keys securely**)
- **Note:** Only set these if you understand their impact or if instructed by the user. Incorrect values might lead to errors. The server will use values from `config.yaml` if not dynamically overridden.
2. **Set auth token first** before using any other endpoint
3. **NEXT ask user which index to use** - after setting auth token but before asking for session ID
- If user asks to "get index" or "show indexes", call discover_indexes
- Have user select an index using set_current_index before proceeding
4. **ALWAYS request `session_id` from the user , if not given ask whether can i use order id to get id and proceed . if yes then do so ** - this is required for all queries
5. **ALWAYS include `session_id` in KQL queries** using format: `"{session_id} AND additional_query"`
6. **ALWAYS use Kibana Query Language (KQL)** for all query formatting
7. **ALWAYS use ISO format timestamps** (e.g., "2023-06-15T00:00:00Z") instead of relative time strings
8. **Parse and present results clearly** in a structured format
9. **dont use summarize_logs by yourself , use only when user asks for log summary .
10. **Detect and use the right index** - allow users to discover and select the appropriate index for their company
How to Perform Root Cause Analysis (RCA)
Start with the Session ID:
Begin the RCA process immediately after obtaining the session_id.
Initial Log Extraction:
Fetch the first 200 logs in ascending order using only the session_id as the query. This sets the baseline timeline.
Understand the Anomaly:
Analyze the user-reported issue and begin identifying patterns or anomalies in the initial log batch.
Craft Intelligent Queries:
Use insights from the codebase to construct advanced, context-aware queries that help narrow down potential causes.
Correlate Logs with Code:
As logs are fetched, map them to related functions, services, and error flows in the codebase to deepen understanding.
Iterate & Converge:
Refine your queries recursively, incorporating knowledge from previous logs and the codebase. Keep digging until you clearly identify the root cause.
Optimize Query Strategy:
⚠️ Always choose the most effective query for faster convergence. Maintain clarity and precision at every step—accuracy is non-negotiable.
## 🔑 AUTHENTICATION SETUP (REQUIRED FIRST STEP)
```bash
curl -X POST http://localhost:8000/api/set_auth_token \
-H "Content-Type: application/json" \
-d '{"auth_token":"YOUR_AUTH_TOKEN_HERE"}'
```
Example:
```bash
curl -X POST http://localhost:8000/api/set_auth_token -H "Content-Type: application/json" -d '{"auth_token":"asdahduwdnsaiuefehasadflhy"}'
```
## 🗂️ INDEX SELECTION (SECOND STEP FOR NEW USERS)
1. **Discover available indexes:**
```bash
curl -X GET http://localhost:8000/api/discover_indexes
```
2. **Set the index to use:**
```bash
curl -X POST http://localhost:8000/api/set_current_index \
-H "Content-Type: application/json" \
-d '{"index_pattern": "YOUR_INDEX_PATTERN"}'
```
Example:
```bash
curl -X POST http://localhost:8000/api/set_current_index \
-H "Content-Type: application/json" \
-d '{"index_pattern": "estio-logs-*"}'
```
3. **When a user says they're from a specific company:**
- First discover available indexes
- Look for index patterns that might match their company name
- Suggest setting the index that seems most appropriate
- If unsure, ask the user to select from the available indexes
4. **Index preference persistence:**
- Once set, the index preference will persist until changed
- If a user switches companies or needs a different index, use set_current_index again
To extract the **Session ID** using the **Order ID** from Kibana logs during the `callStartPayment` phase.
### 🔍 Step-by-Step Rule:
1. **Use the extract_session_id Endpoint (Preferred Method):**
```bash
curl -X POST http://localhost:8000/api/extract_session_id \
-H "Content-Type: application/json" \
-d '{
"order_id": "YOUR_ORDER_ID_HERE"
}'
```
* This endpoint uses Neurolink AI to automatically extract session IDs
* It follows all the rules below and returns structured results
2. **Fallback Method (If extract_session_id Fails):**
Use the search_logs endpoint with this query:
```kibana
{Order ID} AND "callStartPayment"
```
* Replace `{Order ID}` with the actual order ID.
* Use Kibana Query Language (KQL) in the query box.
* max_results must always be 1-3
3. **Analyze Response Logs (For Fallback Method):**
* Look for log lines that start with this pattern:
```
message:{num} | {some_id} | {session_id} |
```
* The line always starts with a prefix like `message:`num followed by two pipe-separated values.
* The **third segment** (after the second `|`) is the **Session ID**.
* max_results must always be 1-3
4. **IMPORTANT: If the extract_session_id endpoint fails, you MUST use the fallback method with search_logs**
* Parse the logs manually using the pattern above
* Extract the session ID from the third segment after the second pipe character
* Proceed with the extracted session ID for subsequent queries
---
** Function based mode Begin **
this will be triggered by user saying search based on functions
- if this mode is used : all your query text must have {session_id} AND ("FunctionCallResult" OR "FunctionCalled") + query terms
- always use sort by timestamp and sort order asc in this mode
** Function based mode End. **
## 📋 API ENDPOINTS REFERENCE
| Endpoint | Primary Use Case | Key Parameters |
|----------|------------------|----------------|
| `/api/search_logs` | **MAIN ENDPOINT** - Find specific logs | `query_text`, `max_results`, time filters |
| `/api/extract_session_id` | **EXTRACT SESSION ID** - Get session ID from order ID | `order_id` |
| `/api/get_recent_logs` | View latest logs | `count`, `level` |
| `/api/analyze_logs` | Identify patterns | `time_range`, `group_by` |
| `/api/extract_errors` | Find errors | `hours`, `include_stack_traces`, `limit` |
| `/api/summarize_logs` | 🧠 **AI-POWERED** - Generate intelligent log analysis | Same as `search_logs` + AI analysis |
| `/api/discover_indexes` | List available indexes | None |
| `/api/set_current_index` | Select index for searches | `index_pattern` |
## 🧠 AI-POWERED MEMORY BOARD
The Memory Board is a stateful, short-term memory system that allows an AI agent to build context during a Root Cause Analysis (RCA) session. It helps the AI to track findings, avoid redundant work, and form more intelligent hypotheses.
### Endpoints
| Endpoint | Method | Purpose | Key Parameters |
|---|---|---|---|
| `/api/memory/create` | POST | Creates a new, named investigation board. | `name` |
| `/api/memory/all` | GET | Lists all active investigation boards (ID and name). | None |
| `/api/memory/{board_id}` | GET | Retrieves all findings for a specific board. | `board_id` |
| `/api/memory/{board_id}/add_finding`| POST | Adds a structured finding to a board. | `board_id`, finding data |
| `/api/memory/{board_id}/clear` | POST | Clears and deletes an investigation board. | `board_id` |
### AI Workflow for Using the Memory Board
1. **Start Task & Check for Existing Work:** When a new RCA task begins, first call `GET /api/memory/all` to get a list of all active investigations.
* Check if an investigation for the same issue already exists to avoid redundant work.
* Check for related investigations that might provide context (e.g., a systemic error board).
2. **Create an Investigation Board:** If no relevant board exists, create a new one.
* **Action:** Call `POST /api/memory/create`.
* **Payload:** `{"name": "RCA for OrderID 12345 payment failure"}`
* **Store the returned `board_id`** for all subsequent operations in this investigation.
3. **Add Findings as You Discover Them:** As you query logs and find important information (errors, anomalies, key events), add them to the board.
* **Action:** Call `POST /api/memory/{board_id}/add_finding`.
* **Payload:** A structured JSON object for the finding.
```json
{
"timestamp": "2023-10-27T10:05:15Z",
"finding": "Payment failed due to 'Insufficient Funds' from Stripe.",
"source_log": { "...raw log object..." },
"attention_weight": 8,
"implication": "The issue is external, not a system bug. Verify the request sent to Stripe was correct."
}
```
4. **Consult the Board to Inform Next Steps:** Before each new query, get the current state of the investigation to make a more intelligent decision.
* **Action:** Call `GET /api/memory/{board_id}`.
* **Logic:** Review the existing findings and their implications to decide what to query for next. Avoid re-discovering facts.
5. **End the Investigation:** Once the RCA is complete and you have reported the final summary to the user:
* **Action:** Call `POST /api/memory/{board_id}/clear` to clean up the in-memory board.
## 🔎 SEARCH_LOGS ENDPOINT - DETAILED SPECIFICATION
### Required Parameters:
- `query_text`: String using KQL format - MUST include `{session_id}` + query terms
- `max_results`: Integer (default: 100) - Number of results to return (adjust to your needs)
### Optional Parameters:
- `start_time`/`end_time`: ISO timestamps (e.g., "2023-06-15T00:00:00Z")
- `levels`: Array of log levels to include (e.g., ["ERROR", "WARN"])
- `include_fields`/`exclude_fields`: Arrays of field names
- `sort_by`: Field to sort by - MUST use a valid timestamp field name (`timestamp`, `@timestamp`, or `start_time`)
- `sort_order`: "asc" or "desc" (default: "desc")
### KQL Query Examples:
```
"{session_id} AND error"
"{session_id} AND payment"
"{session_id} AND verifyPaymentAttempt"
"a71e84b1ee00f6247347 AND callStartPayment"
```
### Complete Examples:
#### 1. Basic Search with Session ID
```bash
curl -X POST http://localhost:8000/api/search_logs \
-H "Content-Type: application/json" \
-d '{
"query_text": "{session_id} AND payment error",
"max_results": 50
}'
```
#### 2. Search with Absolute Time Range
```bash
curl -X POST http://localhost:8000/api/search_logs \
-H "Content-Type: application/json" \
-d '{
"query_text": "{session_id} AND payment",
"start_time": "2023-06-15T00:00:00Z",
"end_time": "2023-06-16T00:00:00Z"
}'
```
#### 3. Search with Specific Transaction
```bash
curl -X POST http://localhost:8000/api/search_logs \
-H "Content-Type: application/json" \
-d '{
"query_text": "{session_id} AND verifyPaymentAttempt",
"max_results": 1
}'
```
#### 4. Search with Correct Sorting
```bash
curl -X POST http://localhost:8000/api/search_logs \
-H "Content-Type: application/json" \
-d '{
"query_text": "{session_id} AND payment",
"sort_by": "timestamp",
"sort_order": "desc"
}'
```
#### 5. Extract Session ID from Order ID
```bash
curl -X POST http://localhost:8000/api/extract_session_id \
-H "Content-Type: application/json" \
-d '{
"order_id": "ORDER_12345"
}'
```
## 📊 OTHER ENDPOINTS - REFERENCE EXAMPLES
### Analyze Logs for Patterns
```bash
curl -X POST http://localhost:8000/api/analyze_logs \
-H "Content-Type: application/json" \
-d '{
"time_range": "24h",
"group_by": "level"
}'
```
### Fetch Recent Logs
```bash
curl -X POST http://localhost:8000/api/get_recent_logs \
-H "Content-Type: application/json" \
-d '{
"count": 10,
"level": "ERROR"
}'
```
!CAUTION DONT USE THIS ENDPOINT WITHOUT USER EXPLICITLY ASKING FOR IT (summarize_logs)
### 🧠 AI-Powered Log Analysis (SUMMARIZE_LOGS)
```bash
curl -X POST http://localhost:8000/api/summarize_logs \
-H "Content-Type: application/json" \
-d '{
"query_text": "{session_id} AND {other query}",
"max_results": 50,
"start_time": "2023-06-15T00:00:00Z",
"end_time": "2023-06-16T00:00:00Z"
}'
```
**What it does:**
- Searches logs using the same parameters as `search_logs`
- Uses AI (Neurolink) to generate intelligent analysis
- Provides structured insights including:
- Summary of activities and systems involved
- Key insights and patterns
- Errors and exceptions with details
- Function calls and critical methods
- Chronological flow of events
- Anomalies and suspicious behavior
- Focus areas needing attention
- Recommendations for fixes
**When to use:**
- When you need to understand complex log patterns
- For root cause analysis of issues
- When dealing with large volumes of logs
- For generating reports or summaries
- When you need actionable insights from logs
**Requirements:**
- Requires Node.js and Neurolink (automatically set up when starting server)
- Requires AI provider API key for full AI features.
Set via environment variable (e.g., `export GOOGLE_AI_API_KEY=...`)
OR in `config.yaml` under `ai_providers` (config values take precedence).
Google AI Studio key is recommended (has a free tier).
- May take longer than regular search due to AI processing
## 🔄 AI DECISION FLOW
1. **Set auth token** (if not already set)
2. **Ask which index to use** and call discover_indexes if needed
3. **Request `session_id` from user** (if not provided)
4. **Determine appropriate endpoint** based on user intent:
- For specific searches → `search_logs`
- For recent errors → `get_recent_logs` or `extract_errors`
- For pattern analysis → `analyze_logs`
- **For intelligent analysis/insights → `summarize_logs` (AI-powered)**
4. **Construct query** using KQL format: `"{session_id} AND <query terms>"`
5. **Select appropriate parameters** (time range, result limits, etc.)
6. **Execute request** using the curl command
7. **Parse and present results** in a clear, structured format
## 🔭 PERISCOPE INTEGRATION - SQL-BASED LOG ANALYSIS
### 📊 Overview
Periscope is a separate log analysis system that uses SQL queries instead of KQL. It provides:
- SQL-based querying for logs
- Stream-based organization (envoy_logs, vayu_logs, etc.)
- Schema discovery with field types and statistics
- Time-based filtering in microseconds
### 🔑 Authentication Setup (Required First Step)
```bash
curl -X POST http://localhost:8000/api/set_periscope_auth_token \
-H "Content-Type: application/json" \
-d '{"auth_token":"YOUR_PERISCOPE_AUTH_TOKEN_HERE"}'
```
**Note:** The Periscope auth token is the base64-encoded `auth_tokens` cookie value from your Periscope session.
### 📋 Available Periscope Endpoints
| Endpoint | Purpose | Key Parameters |
|----------|---------|----------------|
| `/api/set_periscope_auth_token` | Set Periscope authentication | `auth_token` |
| `/api/get_periscope_streams` | List available log streams | `org_identifier` (optional) |
| `/api/get_periscope_stream_schema` | Get schema for a specific stream | `stream_name`, `org_identifier` |
| `/api/get_all_periscope_schemas` | Get schemas for all streams | `org_identifier` (optional) |
| `/api/search_periscope_logs` | **MAIN ENDPOINT** - Search using SQL | `sql_query`, `start_time`, `end_time`, `max_results` |
| `/api/search_periscope_errors` | Find error logs (4xx/5xx) | `hours`, `stream`, `error_codes` |
### 🔍 Step-by-Step Workflow
#### 1. Discover Available Streams
```bash
curl -X GET http://localhost:8000/api/get_periscope_streams
```
**Common Streams:**
- `envoy_logs` - Envoy proxy access logs (HTTP requests/responses)
- `vayu_logs` - Vayu service logs
- `breeze_logs` - Breeze application logs
- `lighthouse_logs` - Lighthouse service logs
- `nimble_logs` - Nimble service logs
- `istio_logs` - Istio service mesh logs
- `squid_logs` - Squid proxy logs
#### 2. Get Schema for a Stream
```bash
curl -X POST http://localhost:8000/api/get_periscope_stream_schema \
-H "Content-Type: application/json" \
-d '{
"stream_name": "envoy_logs"
}'
```
**Returns:**
- Field names and types (e.g., `status_code: Utf8`, `timestamp: Int64`)
- Statistics (doc count, time range, storage size)
- Settings (retention, partitioning)
#### 3. Search Logs with SQL
```bash
curl -X POST http://localhost:8000/api/search_periscope_logs \
-H "Content-Type: application/json" \
-d '{
"sql_query": "SELECT * FROM \"envoy_logs\" WHERE status_code LIKE '\''5%'\''",
"start_time": "24h",
"max_results": 50
}'
```
### 📝 SQL Query Syntax
**Basic Query Structure:**
```sql
SELECT * FROM "stream_name" WHERE condition
```
**Important:**
- Stream names must be in double quotes: `"envoy_logs"`
- String literals use single quotes: `'value'`
- In curl commands, escape quotes: `'\''value'\''`
**Common Query Patterns:**
1. **Search by Status Code:**
```sql
SELECT * FROM "envoy_logs" WHERE status_code LIKE '5%'
```
2. **Search by Method and Path:**
```sql
SELECT * FROM "envoy_logs" WHERE method = 'POST' AND path LIKE '/api%'
```
3. **Search by Session ID:**
```sql
SELECT * FROM "vayu_logs" WHERE session_id = 'your-session-id'
```
4. **Search by Time Range and Condition:**
```sql
SELECT * FROM "envoy_logs" WHERE status_code >= 400 AND method = 'GET'
```
5. **Limit Results:**
```sql
SELECT * FROM "envoy_logs" WHERE status_code = '500' LIMIT 10
```
### ⏰ Time Format Options & Timezone Support
Periscope uses **microseconds** since epoch internally, but the API accepts flexible formats with **full timezone support**:
1. **Relative Time:** `"24h"`, `"1d"`, `"7d"`, `"1w"`, `"1m"`
2. **ISO Timestamps with Timezone:**
- UTC: `"2025-10-04T04:50:00Z"` or `"2025-10-04T04:50:00+00:00"`
- IST: `"2025-10-04T10:20:00+05:30"`
- US Eastern: `"2025-10-04T00:50:00-05:00"`
3. **Naive Datetime with Timezone Parameter:**
- `"2025-10-04 10:20:00"` with `"timezone": "Asia/Kolkata"`
- `"2025-10-04T10:20:00"` with `"timezone": "Asia/Kolkata"`
4. **Microseconds:** `1759527600000000`
**🌍 NEW TIMEZONE FEATURE:**
You can now specify times in any timezone and the system will automatically convert to UTC microseconds:
```bash
# Using IST (India Standard Time) directly - NO manual UTC conversion needed!
curl -X POST http://localhost:8000/api/search_periscope_logs \
-H "Content-Type: application/json" \
-d '{
"sql_query": "SELECT COUNT(*) FROM \"envoy_logs\"",
"start_time": "2025-10-04 10:20:00",
"end_time": "2025-10-04 11:00:00",
"timezone": "Asia/Kolkata"
}'
# Alternative: ISO format with timezone offset (no timezone parameter needed)
curl -X POST http://localhost:8000/api/search_periscope_logs \
-H "Content-Type: application/json" \
-d '{
"sql_query": "SELECT COUNT(*) FROM \"envoy_logs\"",
"start_time": "2025-10-04T10:20:00+05:30",
"end_time": "2025-10-04T11:00:00+05:30"
}'
```
**Common Timezone Names:**
- India: `"Asia/Kolkata"` or `"Asia/Calcutta"`
- UTC: `"UTC"`
- US Eastern: `"US/Eastern"` or `"America/New_York"`
- US Pacific: `"US/Pacific"` or `"America/Los_Angeles"`
- Singapore: `"Asia/Singapore"`
- UK: `"Europe/London"`
**Examples:**
```bash
# Last 24 hours
"start_time": "24h"
# Last 7 days
"start_time": "7d"
# Specific date range in UTC
"start_time": "2025-10-01T00:00:00Z",
"end_time": "2025-10-03T00:00:00Z"
# Specific date range in IST using timezone parameter
"start_time": "2025-10-04 10:20:00",
"end_time": "2025-10-04 11:00:00",
"timezone": "Asia/Kolkata"
# Microseconds (always UTC)
"start_time": 1759440000000000,
"end_time": 1759527600000000
```
**IMPORTANT NOTES:**
- If `timezone` parameter is provided with naive datetime, it will be applied
- If datetime has explicit timezone (like `+05:30`), timezone parameter is ignored
- Naive datetimes without timezone parameter default to UTC
- All conversions result in UTC microseconds internally
- The `_timestamp` field in logs may be in local server time (check your specific logs)
### 🔴 Finding Errors - Convenience Function
For quickly finding 4xx/5xx errors:
```bash
curl -X POST http://localhost:8000/api/search_periscope_errors \
-H "Content-Type: application/json" \
-d '{
"hours": 24,
"stream": "envoy_logs",
"error_codes": "5%"
}'
```
**Parameters:**
- `hours`: Number of hours to look back (default: 24)
- `stream`: Stream name (default: "envoy_logs")
- `error_codes`: Pattern like `"4%"` (4xx), `"5%"` (5xx), or `null` for all >=400
- `org_identifier`: Organization (default: "default")
### 📊 Complete Examples
#### Example 1: Find All 5xx Errors in Last 24 Hours
```bash
curl -X POST http://localhost:8000/api/search_periscope_errors \
-H "Content-Type: application/json" \
-d '{
"hours": 24,
"stream": "envoy_logs",
"error_codes": "5%"
}'
```
#### Example 2: Search for Specific Endpoint Errors
```bash
curl -X POST http://localhost:8000/api/search_periscope_logs \
-H "Content-Type: application/json" \
-d '{
"sql_query": "SELECT * FROM \"envoy_logs\" WHERE path LIKE '\''/api/payment%'\'' AND status_code >= 400",
"start_time": "24h",
"max_results": 100
}'
```
#### Example 3: Analyze Slow Requests
```bash
curl -X POST http://localhost:8000/api/search_periscope_logs \
-H "Content-Type: application/json" \
-d '{
"sql_query": "SELECT * FROM \"envoy_logs\" WHERE duration_ms > 1000 ORDER BY duration_ms DESC LIMIT 50",
"start_time": "24h"
}'
```
#### Example 4: Find Requests by Session ID
```bash
curl -X POST http://localhost:8000/api/search_periscope_logs \
-H "Content-Type: application/json" \
-d '{
"sql_query": "SELECT * FROM \"vayu_logs\" WHERE session_id = '\''abc-123-def'\''",
"start_time": "7d",
"max_results": 200
}'
```
#### Example 5: Get All Schemas for Reference
```bash
curl -X GET http://localhost:8000/api/get_all_periscope_schemas
```
### 🎯 Common Use Cases
**1. Error Analysis:**
- Find all 5xx errors: `WHERE status_code LIKE '5%'`
- Find specific error: `WHERE status_code = '500'`
- Find client errors: `WHERE status_code LIKE '4%'`
**2. Performance Analysis:**
- Slow requests: `WHERE duration_ms > 1000`
- Fast requests: `WHERE duration_ms < 100`
- Average duration: Use aggregation functions
**3. Traffic Analysis:**
- By endpoint: `WHERE path LIKE '/api/specific%'`
- By method: `WHERE method = 'POST'`
- By user agent: `WHERE user_agent LIKE '%Chrome%'`
**4. Session Tracking:**
- All session logs: `WHERE session_id = 'session-id'`
- Session with errors: `WHERE session_id = 'session-id' AND status_code >= 400`
### 🔄 Periscope vs Kibana Decision Flow
**Use Periscope when:**
- ✅ You need SQL-based querying
- ✅ Analyzing HTTP access logs (envoy_logs)
- ✅ Need schema information and field types
- ✅ Working with structured log data
- ✅ Need precise time-based filtering
**Use Kibana when:**
- ✅ You need KQL-based querying
- ✅ Working with application logs
- ✅ Need session-based tracking
- ✅ Following existing Kibana workflows
- ✅ Need AI-powered summarization
### ⚠️ Important Notes
1. **Stream Names:** Always use double quotes around stream names in SQL
2. **String Values:** Use single quotes for string literals in WHERE clauses
3. **Time Range:** Periscope uses microseconds internally but accepts flexible formats
4. **Default Org:** Most queries use `org_identifier: "default"` unless specified
5. **Field Names:** Check schema first to know available fields and types
6. **Escaping:** In curl commands with bash, escape single quotes as `'\''`
### 🧪 Testing Your Setup
1. **Test Authentication:**
```bash
curl -X POST http://localhost:8000/api/set_periscope_auth_token \
-H "Content-Type: application/json" \
-d '{"auth_token":"your-token"}'
```
2. **List Streams:**
```bash
curl -X GET http://localhost:8000/api/get_periscope_streams
```
3. **Get Schema:**
```bash
curl -X POST http://localhost:8000/api/get_periscope_stream_schema \
-H "Content-Type: application/json" \
-d '{"stream_name": "envoy_logs"}'
```
4. **Search for Errors:**
```bash
curl -X POST http://localhost:8000/api/search_periscope_errors \
-H "Content-Type: application/json" \
-d '{"hours": 1, "stream": "envoy_logs", "error_codes": "5%"}'
```
### 🔍 Complete RCA Example Using Timezone Feature
**Scenario:** CPU spike alert at 10:45 AM IST on Oct 4, 2025. Investigate 10:20-11:00 AM IST window.
**Step 1: Set Authentication**
```bash
curl -X POST http://localhost:8000/api/set_periscope_auth_token \
-H "Content-Type: application/json" \
-d '{"auth_token":"your-periscope-token"}'
```
**Step 2: Get Total Traffic (Using NEW Timezone Feature)**
```bash
curl -X POST http://localhost:8000/api/search_periscope_logs \
-H "Content-Type: application/json" \
-d '{
"sql_query": "SELECT COUNT(*) as total_logs FROM \"envoy_logs\"",
"start_time": "2025-10-04 10:20:00",
"end_time": "2025-10-04 11:00:00",
"timezone": "Asia/Kolkata",
"max_results": 1
}'
```
**Step 3: Find Peak Traffic Minutes**
```bash
curl -X POST http://localhost:8000/api/search_periscope_logs \
-H "Content-Type: application/json" \
-d '{
"sql_query": "SELECT COUNT(*) as req_per_minute, CAST(_timestamp / 60000000 AS BIGINT) as minute_bucket FROM \"envoy_logs\" GROUP BY minute_bucket ORDER BY req_per_minute DESC LIMIT 5",
"start_time": "2025-10-04 10:20:00",
"end_time": "2025-10-04 11:00:00",
"timezone": "Asia/Kolkata",
"max_results": 5
}'
```
**Step 4: Analyze Failures**
```bash
curl -X POST http://localhost:8000/api/search_periscope_logs \
-H "Content-Type: application/json" \
-d '{
"sql_query": "SELECT COUNT(*) as failure_count, upstream_host FROM \"envoy_logs\" WHERE response_flags = '\''DC'\'' GROUP BY upstream_host ORDER BY failure_count DESC",
"start_time": "2025-10-04 10:20:00",
"end_time": "2025-10-04 11:00:00",
"timezone": "Asia/Kolkata",
"max_results": 10
}'
```
**Step 5: Error Breakdown**
```bash
curl -X POST http://localhost:8000/api/search_periscope_logs \
-H "Content-Type: application/json" \
-d '{
"sql_query": "SELECT COUNT(*) as error_count, response_flags, status_code FROM \"envoy_logs\" WHERE status_code >= '\''400'\'' OR status_code = '\''0'\'' GROUP BY response_flags, status_code ORDER BY error_count DESC",
"start_time": "2025-10-04 10:20:00",
"end_time": "2025-10-04 11:00:00",
"timezone": "Asia/Kolkata",
"max_results": 10
}'
```
**Benefits of Using Timezone Parameter:**
- ✅ No manual UTC conversion required
- ✅ Search using local business hours (IST)
- ✅ Easier to correlate with alerts and monitoring dashboards
- ✅ Reduces human error in timezone calculations
- ✅ Works with any IANA timezone name
## 📌 IMPORTANT NOTES
- Always replace `{session_id}` with the actual session ID in queries
- Always use ISO format timestamps (e.g., "2023-06-15T00:00:00Z") instead of relative time strings
- Include relevant context in queries to narrow results
- Use field filters (`include_fields` and `exclude_fields`) for large result sets
- Set appropriate `max_results` to balance detail vs. performance
- **For Periscope:** Use SQL syntax and check stream schemas first
- **For Periscope:** Time values can be relative ("24h"), ISO, or microseconds
## ⚠️ TROUBLESHOOTING COMMON ERRORS
### Timestamp Field Issues
- When using `sort_by`, you MUST use a valid timestamp field that exists in the indices
- Different indices use different field names for timestamps:
- `timestamp` - Used in breeze-logs source
- `@timestamp` - Standard Elasticsearch default
- `start_time` - Used in main elasticsearch config
- If you see "No mapping found for [field] in order to sort on" error:
1. Try using a different timestamp field name (`timestamp`, `@timestamp`, or `start_time`)
2. The server will automatically retry without sorting if it encounters this error
3. You can omit the `sort_by` parameter to let the server use its default sorting logic
### Example with Correct Timestamp Field
```bash
# Correct - Using a valid timestamp field:
curl -X POST http://localhost:8000/api/search_logs \
-H "Content-Type: application/json" \
-d '{
"query_text": "{session_id} AND error",
"sort_by": "timestamp",
"sort_order": "desc"
}'
```