# Log Intelligence MCP

Semantic log ingestion, **hybrid retrieval**, and cleanup for the Rapid7 SI
Triage Automation POC. This is the "Application Logs MCP" in the architecture
diagram: it turns raw log files (downloaded from Jira tickets by the companion
Jira/Confluence MCP) into a queryable vector index, and serves the most relevant
log chunks back to the triage agent during defect analysis.

---

## What it does

1. **Ingest** — reads raw log files for a ticket, parses them into logical
   *entries* (a header line plus its stack-trace/continuation lines), groups them
   into **semantic, token-budgeted chunks**, embeds each chunk, and stores the
   vectors in a per-ticket collection.
2. **Query** — given a natural-language question, runs **hybrid retrieval**
   (dense vector similarity **+** BM25 keyword matching, fused with Reciprocal
   Rank Fusion) and returns the top-k chunks with full provenance (source file,
   line range, time span, log levels, trace ids).
3. **Stats** — cheap aggregate view of a ticket's logs (level histogram, error
   count, time span, distinct trace ids).
4. **Delete** — after the defect pipeline finishes, removes the ticket's vectors
   **and** the raw local log files, freeing disk and clearing stale data.

## Tools

| Tool | Purpose |
|------|---------|
| `ingest_ticket_logs(ticket_id, paths?)` | Parse → chunk → embed → store all logs for a ticket. Reads from the shared `logs/<ticket_id>/` dir by default, or an explicit `paths` list. |
| `query_logs(ticket_id, query, top_k=5)` | Hybrid semantic + keyword retrieval of the most relevant chunks. |
| `get_log_stats(ticket_id)` | Ingestion manifest + stored-chunk count + entry summary. |
| `delete_ticket_logs(ticket_id, delete_raw=true)` | Remove vectors and (optionally) raw files. Cleanup step. |

---

## The chunking strategy (why it's built this way)

Chunk quality decides retrieval quality, so the chunker is the heart of this MCP.

* **Entry-aware.** Logs are first assembled into entries: a timestamped header
  plus every continuation line (`\tat …`, `Caused by:`, `… N more`, wrapped
  messages). An entry is atomic — **it is never split across chunks**, which is
  what guarantees a stack trace always travels with the ERROR line that produced
  it.
* **Token-budgeted for Claude.** Chunks target ~**1000 tokens** and are capped at
  **1600** (`CHUNK_*_TOKENS`). Large enough to hold a full error + stack trace +
  surrounding context; small enough that top-k results stay focused and the
  agent's Phase-1 prompt stays bounded.
* **Semantically grouped.** Packing prefers to break at natural boundaries — a
  new trace/correlation id, or a fresh ERROR — so related lines for one request
  land in the same chunk.
* **Overlap without cutting.** Each chunk is seeded with the trailing *whole
  entries* of the previous chunk (~150 tokens) so context isn't lost at
  boundaries, but entries are never sliced mid-way.
* **Oversized entries.** A single entry larger than the hard max (e.g. a giant
  stack trace) is emitted whole and flagged `oversized` rather than truncated.

Token counting uses a fast, conservative character-based estimate (logs are
punctuation-heavy, so this slightly over-estimates and keeps chunks safely under
budget). Set `USE_ANTHROPIC_TOKENIZER=1` to use exact Claude token counts when
network is available.

## Hybrid retrieval

Dense and sparse retrieval catch different things: embeddings capture semantic
similarity ("payment failed" ≈ "authorization error"), while BM25 nails exact
identifiers (`TokenVaultException`, a trace id, a filename). We run both and fuse
their **rankings** with Reciprocal Rank Fusion:

```
rrf_score(d) = Σ_retriever  weight / (RRF_K + rank_retriever(d))
```

RRF fuses ranks rather than raw scores, so the two different score scales don't
need fragile normalisation. Tunables: `RRF_K` (default 60), `DENSE_WEIGHT`,
`SPARSE_WEIGHT`, `CANDIDATE_POOL`, `DEFAULT_TOP_K`.

---

## Backends (production vs. offline)

Every heavy dependency sits behind an adapter with a **real** pure-Python
fallback, so the whole pipeline runs and is testable with no network, and flips
to the production backend by changing one env var.

| Concern | Production (default when installed) | Offline fallback (real, not mock) |
|---------|-------------------------------------|-----------------------------------|
| Embeddings | `sentence-transformers` `all-mpnet-base-v2` (768-dim) | Deterministic hashed n-gram TF-IDF on numpy |
| Vector store | Chroma (persistent) | Per-ticket numpy `.npz` + JSON, real cosine search |
| Sparse | Pure-Python BM25 (always) | same |
| Token count | Anthropic exact counter (optional) | character estimate |

`EMBED_BACKEND=auto` uses sentence-transformers if importable, else the hashing
embedder. `VECTOR_BACKEND=auto` uses Chroma if importable, else the numpy store.
Force a backend with `EMBED_BACKEND=sentence-transformers|bedrock|hashing` and
`VECTOR_BACKEND=chroma|numpy`.

> The offline fallbacks are genuine implementations (real vectors, real
> persistence, real similarity search) — they exist so the POC runs anywhere,
> not to fake results.

---

## Install & run

```bash
cd log-intelligence-mcp
python -m venv .venv && source .venv/bin/activate     # Windows: .venv\Scripts\activate
pip install -e .            # installs mcp, chromadb, sentence-transformers, numpy, uvicorn
cp .env.example .env        # adjust if needed

# stdio (for a local MCP client / Claude Desktop):
python -m log_intelligence_mcp --transport stdio

# HTTP (streamable-http, served at http://127.0.0.1:8081/mcp):
python -m log_intelligence_mcp --transport http
```

> This server is one of four processes in the SI Triage POC (this + the
> Jira/Confluence and Historical KB MCPs + the orchestrator). For the full
> multi-service manual startup sequence, `.env` layout across all four repos,
> and end-to-end test steps, see **`orchestrator-agent/si-triage-automation/README.md`
> → "Running the full system manually"**.

The first sentence-transformers run downloads the model (needs network once).
With no network / no heavy deps installed, it automatically uses the offline
fallbacks — the server still starts and every tool works.

### Register with an MCP client (stdio example)

```json
{
  "mcpServers": {
    "log-intelligence": {
      "command": "python",
      "args": ["-m", "log_intelligence_mcp", "--transport", "stdio"],
      "env": { "SI_DATA_DIR": "/absolute/path/to/si_data" }
    }
  }
}
```

---

## How it coordinates with the Jira/Confluence MCP

Both servers share one directory tree, `SI_DATA_DIR` (default `./si_data`) — set
it to the **same absolute path** for both.

```
si_data/
  logs/<ticket_id>/…      # written by the Jira MCP, read by this MCP
  vector_store/           # owned by this MCP
  meta/<ticket_id>.json   # ingestion manifest written by this MCP
```

Typical flow: Jira MCP `get_ticket` downloads log attachments into
`logs/<ticket_id>/` → this MCP `ingest_ticket_logs(ticket_id)` indexes them →
agent calls `query_logs(...)` during analysis → `delete_ticket_logs(ticket_id)`
cleans up at the end.

---

## Tests

```bash
pytest                      # in the POC environment (needs `pip install pytest`)
python tests/_runner.py     # offline harness used when pytest isn't installed
```

The suite covers entry assembly, the chunker invariants (no split entry, token
budget respected, stack trace kept whole, overlap present, every line covered,
oversized handling), BM25, hashed embeddings, the numpy store round-trip, hybrid
fusion, and a full ingest → query → stats → delete end-to-end. 20 tests,
all offline.

## Configuration reference

See `.env.example` for every variable. Key ones: `SI_DATA_DIR`,
`CHUNK_TARGET_TOKENS`/`CHUNK_MAX_TOKENS`/`CHUNK_MIN_TOKENS`/`CHUNK_OVERLAP_TOKENS`,
`EMBED_BACKEND`/`EMBED_MODEL`, `VECTOR_BACKEND`, `RRF_K`/`DENSE_WEIGHT`/`SPARSE_WEIGHT`,
`DEFAULT_TOP_K`, `MCP_HTTP_HOST`/`MCP_HTTP_PORT` (default 8081), `LOG_LEVEL`/`LOG_JSON`.