# lockwood-group-observability-mcp

Semantic log clustering for incident triage, exposed as an MCP server.

## What it does and why

During an incident, the log stream is mostly repetition: the same failure
printed thousands of times with different ids, plus a few distinct signals
buried in the noise. This server takes a window of log lines and groups them
by semantic similarity, then ranks the groups so the ones that matter for
triage surface first. Instead of scrolling raw logs, an operator (or an agent
driving the MCP tool) sees "3 clusters: a CRITICAL checkout 500 seen 240x, an
ERROR payment timeout seen 1900x, ...".

Triage runs on the incident path, so latency is a hard requirement, not a
nice-to-have: a `triage_logs` call has a **p99 budget under 150ms**. That
budget shapes the whole design (see Architecture and `docs/adr/`).

## Architecture

```
inkwellobservability/
  types.py                 domain types: LogRecord, Cluster, TriageResult, Severity
  clustering.py            TriageEngine + normalize_message (the core algorithm)
  config.py                env-driven Config with startup validation
  errors.py                ValidationError / ResourceLimitError / ConfigError
  logging_setup.py         JSON-lines logging to stderr
  service.py               argument validation + engine orchestration
  server.py                MCP JSON-RPC 2.0 server over stdio
  __main__.py              entry point (python -m inkwellobservability)
  bench.py                 latency benchmark (make bench)
  providers/
    base.py                EmbeddingProvider interface
    stub.py                deterministic offline embeddings (hot path)
    real.py                HTTP embedding client (optional, never used in tests)
tests/
docs/adr/                  architecture decision records
```

The request path: `server` parses a JSON-RPC line and dispatches `tools/call`
to `service`, which validates arguments, enforces resource guards, and hands
`LogRecord`s to the `TriageEngine`. The engine normalizes each message to a
template, embeds it via the configured `EmbeddingProvider`, and assigns it to
the nearest cluster in a single greedy pass.

Two mechanisms make the 150ms budget a property of the design rather than a
hope:

- **Bounded cluster count `K`.** Assignment scans existing centroids, so
  per-record cost is `O(K * dim)`. Once `K` clusters exist, further distinct
  log shapes fold into the nearest cluster instead of spawning new scans.
  Total cost is `O(N * K * dim)` with a known constant.
- **A wall-clock deadline.** The clustering loop checks elapsed time on a
  coarse stride and, once the budget is spent, folds the remaining records
  into existing clusters and flags the result `degraded` — a partial but
  usable ranking, returned inside the budget, rather than running long.

Embeddings on the hot path come from a deterministic feature-hashing provider
(`StubEmbeddingProvider`) with no network hop. `RealEmbeddingProvider` is an
optional quality upgrade behind the same interface; it is never required for
the tests, which run fully offline. The reasoning behind these calls is
recorded in `docs/adr/`.

## Install

```
make venv       # python3 -m venv .venv
make install    # .venv/bin/python -m pip install -e .[dev]
```

All Makefile targets use `$(PY)` (default `.venv/bin/python`); override it to
use a different interpreter, e.g. `make test PY=python3`.

## Quickstart

Run the test suite and the latency benchmark:

```
make test PY=python3
make bench PY=python3
```

`make bench` prints, for a 2000-record window:

```
records/window : 2000
budget         : 150.0 ms
p50            : ...
p99            : ...
within budget  : True
```

Use the engine directly as a library:

```python
from inkwellobservability import TriageEngine, LogRecord

engine = TriageEngine()  # defaults: threshold 0.55, K=64, budget 150ms
records = [
    LogRecord.of("payment gateway timeout for order 4821", level="ERROR"),
    LogRecord.of("payment gateway timeout for order 90", level="ERROR"),
    LogRecord.of("user 12 login failed", level="WARNING"),
]
result = engine.triage(records)
for cluster in result.clusters:            # ranked, worst first
    print(cluster.size, cluster.max_level.name, cluster.template)
print("degraded:", result.degraded, "elapsed_ms:", result.elapsed_ms)
```

Ranking weights frequency by worst severity seen (`size * 4**level`): a small
burst of CRITICAL out-ranks a flood of INFO, while a large ERROR cluster still
out-ranks a single CRITICAL. See `docs/adr/0004-severity-weighted-ranking.md`.

### As an MCP server

The entry point speaks MCP (JSON-RPC 2.0) over stdio, one JSON object per line:

```
python -m inkwellobservability          # serve on stdin/stdout
python -m inkwellobservability --version
python -m inkwellobservability --help
```

After `make install` the same server is on `PATH` as `inkwell-observability`.

It exposes one tool, `triage_logs`, taking either inline `records` or a `path`
to a JSON-lines file, plus an optional `budget_ms`. Piping two requests in:

```
printf '%s\n' \
  '{"jsonrpc":"2.0","id":1,"method":"initialize"}' \
  '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"triage_logs","arguments":{"records":[{"message":"payment timeout order 1","level":"ERROR"},{"message":"payment timeout order 2","level":"ERROR"}]}}}' \
  | python -m inkwellobservability
```

The tool result carries both a text block (JSON) and `structuredContent` with
`clusters` (ranked worst-first), `total_records`, `cluster_count`,
`elapsed_ms`, `budget_ms`, and `degraded`.

Logs are JSON lines on **stderr** (stdout is reserved for protocol messages).
Every triage call logs its timing; a call that spends its budget logs at
WARNING with `degraded: true`.

## Configuration reference

All configuration is environment-driven and validated at startup; an invalid
value exits non-zero before the server starts serving.

| Variable | Default | Meaning |
| --- | --- | --- |
| `INKWELL_PROVIDER` | `stub` | `stub` (offline) or `real` (HTTP embeddings) |
| `INKWELL_EMBEDDING_DIM` | `128` | embedding dimension |
| `INKWELL_SIMILARITY_THRESHOLD` | `0.55` | cosine threshold for joining a cluster |
| `INKWELL_MAX_CLUSTERS` | `64` | cluster cap `K` (bounds per-record cost) |
| `INKWELL_BUDGET_MS` | `150.0` | per-call latency budget |
| `INKWELL_MAX_RECORDS` | `20000` | resource guard: max records per call |
| `INKWELL_MAX_MESSAGE_BYTES` | `16384` | resource guard: max bytes per message |
| `INKWELL_LOG_LEVEL` | `INFO` | `DEBUG`/`INFO`/`WARNING`/`ERROR` |

When `INKWELL_PROVIDER=real`, the real provider also reads
`INKWELL_EMBED_ENDPOINT` and `INKWELL_EMBED_API_KEY`. If either is missing (or
the backend is unreachable at construction), the service logs a warning and
degrades to the stub provider rather than failing to start.

## Failure handling

- Malformed JSON-RPC line, bad envelope, or unknown method -> JSON-RPC error.
- Bad tool arguments, missing/unreadable file, malformed JSONL line, or a
  resource guard trip -> a `tools/call` result with `isError: true` and a
  message; the server loop stays up.
- Oversized batch or message is rejected up front so one request cannot blow
  the latency budget for others.

## Known limitations

- **Per-call windows, no streaming state.** Each `triage_logs` call clusters
  the window it is given; clusters are not carried across calls. A rolling
  window with centroid eviction is deferred until the ingestion shape is fixed
  (TODO in `clustering.py`). See `docs/adr/0001`.
- **Feature-hashing embeddings are shallow.** The default provider matches on
  shared tokens, not learned meaning, so paraphrases with no common words
  ("disk full" vs "no space left on device") will not cluster together. Point
  `INKWELL_PROVIDER=real` at an embedding backend for semantic recall, at the
  cost of the offline guarantee. See `docs/adr/0002`.
- **Greedy clustering is order-sensitive.** Assignment depends on arrival
  order; a different permutation can yield slightly different centroids. This
  is an accepted tradeoff for the single-pass cost bound (`docs/adr/0001`).
- **Single tool, no auth.** The MCP surface is `triage_logs` only, and the
  stdio transport assumes a trusted local client. Network transport and
  authentication are out of scope for this deliverable.

---

*Lockwood Group is an illustrative client; this repository is a self-directed reference implementation built to work end to end.*