Skip to main content
Glama
Log Intelligence MCP

Log Intelligence MCP

Log Intelligence MCP

Semantic log ingestion, hybrid retrieval, and cleanup for the Rapid7 SI Triage Automation POC. This is the "Application Logs MCP" in the architecture diagram: it turns raw log files (downloaded from Jira tickets by the companion Jira/Confluence MCP) into a queryable vector index, and serves the most relevant log chunks back to the triage agent during defect analysis.


What it does

  1. Ingest — reads raw log files for a ticket, parses them into logical entries (a header line plus its stack-trace/continuation lines), groups them into semantic, token-budgeted chunks, embeds each chunk, and stores the vectors in a per-ticket collection.

  2. Query — given a natural-language question, runs hybrid retrieval (dense vector similarity + BM25 keyword matching, fused with Reciprocal Rank Fusion) and returns the top-k chunks with full provenance (source file, line range, time span, log levels, trace ids).

  3. Stats — cheap aggregate view of a ticket's logs (level histogram, error count, time span, distinct trace ids).

  4. Delete — after the defect pipeline finishes, removes the ticket's vectors and the raw local log files, freeing disk and clearing stale data.

Related MCP server: log-mcp

Tools

Tool

Purpose

ingest_ticket_logs(ticket_id, paths?)

Parse → chunk → embed → store all logs for a ticket. Reads from the shared logs/<ticket_id>/ dir by default, or an explicit paths list.

query_logs(ticket_id, query, top_k=5)

Hybrid semantic + keyword retrieval of the most relevant chunks.

get_log_stats(ticket_id)

Ingestion manifest + stored-chunk count + entry summary.

delete_ticket_logs(ticket_id, delete_raw=true)

Remove vectors and (optionally) raw files. Cleanup step.


The chunking strategy (why it's built this way)

Chunk quality decides retrieval quality, so the chunker is the heart of this MCP.

  • Entry-aware. Logs are first assembled into entries: a timestamped header plus every continuation line (\tat …, Caused by:, … N more, wrapped messages). An entry is atomic — it is never split across chunks, which is what guarantees a stack trace always travels with the ERROR line that produced it.

  • Token-budgeted for Claude. Chunks target ~1000 tokens and are capped at 1600 (CHUNK_*_TOKENS). Large enough to hold a full error + stack trace + surrounding context; small enough that top-k results stay focused and the agent's Phase-1 prompt stays bounded.

  • Semantically grouped. Packing prefers to break at natural boundaries — a new trace/correlation id, or a fresh ERROR — so related lines for one request land in the same chunk.

  • Overlap without cutting. Each chunk is seeded with the trailing whole entries of the previous chunk (~150 tokens) so context isn't lost at boundaries, but entries are never sliced mid-way.

  • Oversized entries. A single entry larger than the hard max (e.g. a giant stack trace) is emitted whole and flagged oversized rather than truncated.

Token counting uses a fast, conservative character-based estimate (logs are punctuation-heavy, so this slightly over-estimates and keeps chunks safely under budget). Set USE_ANTHROPIC_TOKENIZER=1 to use exact Claude token counts when network is available.

Hybrid retrieval

Dense and sparse retrieval catch different things: embeddings capture semantic similarity ("payment failed" ≈ "authorization error"), while BM25 nails exact identifiers (TokenVaultException, a trace id, a filename). We run both and fuse their rankings with Reciprocal Rank Fusion:

rrf_score(d) = Σ_retriever  weight / (RRF_K + rank_retriever(d))

RRF fuses ranks rather than raw scores, so the two different score scales don't need fragile normalisation. Tunables: RRF_K (default 60), DENSE_WEIGHT, SPARSE_WEIGHT, CANDIDATE_POOL, DEFAULT_TOP_K.


Backends (production vs. offline)

Every heavy dependency sits behind an adapter with a real pure-Python fallback, so the whole pipeline runs and is testable with no network, and flips to the production backend by changing one env var.

Concern

Production (default when installed)

Offline fallback (real, not mock)

Embeddings

sentence-transformers all-mpnet-base-v2 (768-dim)

Deterministic hashed n-gram TF-IDF on numpy

Vector store

Chroma (persistent)

Per-ticket numpy .npz + JSON, real cosine search

Sparse

Pure-Python BM25 (always)

same

Token count

Anthropic exact counter (optional)

character estimate

EMBED_BACKEND=auto uses sentence-transformers if importable, else the hashing embedder. VECTOR_BACKEND=auto uses Chroma if importable, else the numpy store. Force a backend with EMBED_BACKEND=sentence-transformers|bedrock|hashing and VECTOR_BACKEND=chroma|numpy.

The offline fallbacks are genuine implementations (real vectors, real persistence, real similarity search) — they exist so the POC runs anywhere, not to fake results.


Install & run

cd log-intelligence-mcp
python -m venv .venv && source .venv/bin/activate     # Windows: .venv\Scripts\activate
pip install -e .            # installs mcp, chromadb, sentence-transformers, numpy, uvicorn
cp .env.example .env        # adjust if needed

# stdio (for a local MCP client / Claude Desktop):
python -m log_intelligence_mcp --transport stdio

# HTTP (streamable-http, served at http://127.0.0.1:8081/mcp):
python -m log_intelligence_mcp --transport http

This server is one of four processes in the SI Triage POC (this + the Jira/Confluence and Historical KB MCPs + the orchestrator). For the full multi-service manual startup sequence, .env layout across all four repos, and end-to-end test steps, see orchestrator-agent/si-triage-automation/README.md → "Running the full system manually".

The first sentence-transformers run downloads the model (needs network once). With no network / no heavy deps installed, it automatically uses the offline fallbacks — the server still starts and every tool works.

Register with an MCP client (stdio example)

{
  "mcpServers": {
    "log-intelligence": {
      "command": "python",
      "args": ["-m", "log_intelligence_mcp", "--transport", "stdio"],
      "env": { "SI_DATA_DIR": "/absolute/path/to/si_data" }
    }
  }
}

How it coordinates with the Jira/Confluence MCP

Both servers share one directory tree, SI_DATA_DIR (default ./si_data) — set it to the same absolute path for both.

si_data/
  logs/<ticket_id>/…      # written by the Jira MCP, read by this MCP
  vector_store/           # owned by this MCP
  meta/<ticket_id>.json   # ingestion manifest written by this MCP

Typical flow: Jira MCP get_ticket downloads log attachments into logs/<ticket_id>/ → this MCP ingest_ticket_logs(ticket_id) indexes them → agent calls query_logs(...) during analysis → delete_ticket_logs(ticket_id) cleans up at the end.


Tests

pytest                      # in the POC environment (needs `pip install pytest`)
python tests/_runner.py     # offline harness used when pytest isn't installed

The suite covers entry assembly, the chunker invariants (no split entry, token budget respected, stack trace kept whole, overlap present, every line covered, oversized handling), BM25, hashed embeddings, the numpy store round-trip, hybrid fusion, and a full ingest → query → stats → delete end-to-end. 20 tests, all offline.

Configuration reference

See .env.example for every variable. Key ones: SI_DATA_DIR, CHUNK_TARGET_TOKENS/CHUNK_MAX_TOKENS/CHUNK_MIN_TOKENS/CHUNK_OVERLAP_TOKENS, EMBED_BACKEND/EMBED_MODEL, VECTOR_BACKEND, RRF_K/DENSE_WEIGHT/SPARSE_WEIGHT, DEFAULT_TOP_K, MCP_HTTP_HOST/MCP_HTTP_PORT (default 8081), LOG_LEVEL/LOG_JSON.

Available Tools

4 tools
delete_ticket_logsA

Clean up a ticket after the pipeline completes: remove embeddings and raw files.

Deletes the ticket's vectors from the store and (by default) the raw log files the Jira MCP downloaded locally, freeing disk and clearing stale data.

Inputs: ticket_id (required), delete_raw.

get_log_statsA

Cheap aggregate stats for a ticket's ingested logs (levels, errors, time span).

Inputs: ticket_id (required).

ingest_ticket_logsA

Parse, semantically chunk, embed, and store all log files for a ticket.

Reads raw files the Jira MCP downloaded to <data_dir>/logs/<ticket_id>/ (or an explicit paths list). Chunks are entry-aware and token-budgeted for Claude Sonnet/Opus. Returns ingestion stats including token distribution.

Inputs: ticket_id (required), paths.

query_logsA

Hybrid semantic + keyword (BM25) retrieval of the most relevant log chunks.

Combines dense vector similarity with BM25 lexical matching, fused via Reciprocal Rank Fusion. Returns ranked chunks with provenance metadata (source file, line range, time span, levels, trace ids).

Inputs: query (required), ticket_id (required), top_k.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A4.3/5.0

    Scored across 4 tools

    Disambiguation5/5

    Each tool targets a distinct operation: ingest loads and embeds logs, query retrieves relevant chunks, get_log_stats returns aggregates, and delete removes data. There is no meaningful overlap between the two read tools because one returns ranked search results and the other returns summary statistics.

    Naming Consistency5/5

    All tool names follow a clean verb_noun snake_case pattern (ingest_ticket_logs, query_logs, get_log_stats, delete_ticket_logs). Object naming varies slightly between ticket_logs and logs, but this does not break the overall consistency.

    Tool Count5/5

    Four tools cover the full log-intelligence pipeline for a ticket: ingestion, retrieval, stats, and cleanup. The count is appropriately scoped; every tool earns its place.

    Completeness5/5

    The set provides the necessary lifecycle: create/ingest, read via query and stats, and delete/cleanup. Logs are effectively immutable, so an update operation is not a meaningful gap.

    Maintenance

    ActivityInactive
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers