Skip to main content
Glama
Local Knowledge Desk

Local Knowledge Desk

by MatiasLaukka

Knotty

Evidence-grounded support-ticket triage, diagnostics, and human-governed AI workflows.

Imagine a customer reporting, “The campaign editor does not work.” Knotty brings the conversation and available evidence together, identifies missing information, and can propose a diagnosis for human review. A later reply or failed fix can change that conclusion. The local workflow preserves decisions and their supporting history so they can be inspected, replayed and considered for reviewed reuse.

This is an experimental project using controlled demonstration data, not a production customer-support platform.

View the Engineering Case Study (PDF) for an illustrated overview of the architecture, engineering decisions, evaluation evidence and limitations.

Knotty began as a personal experiment with AI agent Skills and Model Context Protocol (MCP) servers. Connecting an assistant to a few local tools raised harder questions: How can an agent reason about incomplete evidence? What happens when an investigation changes? Who can authorize a fix? Can earlier decisions be replayed after a restart?

What began as a tool-integration exercise evolved into a local diagnostic and workflow engine—and, more recently, an evaluation platform for hybrid retrieval, AI reasoning, and reliability engineering.

Knotty's central question is:

How do we build AI systems that do more than generate plausible answers—systems whose decisions are evidence-grounded, traceable, testable, and safe to act on?

Knotty is an experimental engineering project, not a deployed customer-support platform. The local application operates on synthetic tickets and clean-room knowledge articles from Northstar Marketing Cloud, a fictional ecommerce marketing platform. There is no live help-desk connection or outgoing customer email. Separately, the research harness evaluates historical public Home Assistant issues and documentation—not live Home Assistant tickets.

Try the demo · Understand the architecture · Explore Hybrid research · Read the evaluation evidence · Detailed engineering reference

Quick start

Run the local Approval Desk to see a ticket move through classification, evidence gathering, human approval, diagnosis, verification, and closure.

Requirements: Node.js ^20.19.0, ^22.12.0 or >=24.0.0; npm; and a local clone. The commands below use PowerShell, but the npm commands work from other shells.

npm ci
npm run build

# Stop other Knotty runtime/MCP processes before resetting.
npm run reset:demo
npm run approval-desk

Open the URL printed by the server (normally http://127.0.0.1:5177). Use TKT-1010 first: a vague report becomes an evidence-based product-performance investigation after a customer reply. The local demonstration can add a controlled synthetic reply after you approve an outbound update. For an incident-routing example, try TKT-1001.

npm run reset:demo is destructive to local demo runtime state. Stop all MCP and Approval Desk processes first. The reset restores both the authoritative operational state and advisory learning state.

For narrower resets, use npm run reset:operational-demo (tickets and workflow history) or npm run reset:learning-demo (advisory learning data). To retain changes across restarts, stop the Approval Desk with Ctrl+C and rerun npm run approval-desk without a reset. The reset controller uses leases, validation, rollback, and backup checks; its complete behavior is documented in the engineering reference.

Knotty Approval Desk overview showing local synthetic tickets and workflow controls

The browser interface provides the conversation timeline, current recommendation, evidence checklist, human-review controls, and audit context. Other views: conversation workspace and recommendation panel.

Related MCP server: Beeper MCP Note Server

What Knotty can do today

Capability

Status and boundary

Local MCP server, Codex Skill, browser Approval Desk

Implemented for synthetic support tickets

Conversation-aware classification and evidence checks

Implemented; deterministic safety policy has final authority

Diagnostic lifecycle, diagnosis review, fixes, verification and closure

Implemented with explicit human gates

Optional GPT classification, diagnosis advice and response drafting

Advisory, contract-checked, with local fallback

Knowledge discovery, operator promotion and controlled reuse

Implemented with a separate advisory learning ledger

B3 semantic/lexical retrieval, B4 ranking, B5 applicability

Research or shadow paths; no automatic operational cutover

H1–H5b Hybrid reasoning

Latest shadow architecture, not fully integrated into the operational diagnosis path

SIMPLE vs. HYBRID / Qwen vs. Luna vs. Sol

Historical Home Assistant experiment ongoing; no diagnostic-accuracy winner

Live Zendesk/Jira/email integration, production identity boundary

Not included

The application supports a ticket lifecycle rather than a one-shot classifier. A customer can provide new evidence, changing the classification, missing-evidence requirements, plausible diagnosis, or next permitted action. A failed fix or later reply can invalidate an earlier review. Neither a good-looking model explanation nor a retrieved article automatically proves a root cause.

Architecture in one picture

Knotty architecture overview with human/agent interfaces, shared deterministic service, transactional operational state, governed knowledge and isolated Hybrid research

Open the system architecture at full size

One shared TriageService is authoritative. The browser UI and MCP tools are adapters to the same application rules—not separate sources of truth. Four boundaries organize the system:

  1. Operational: deterministic policy, approval checks, ticket lifecycle, causal events, and transactional state.

  2. Advisory AI: optional model classification signals and drafts; outputs must pass validators and cannot authorize changes.

  3. Governed learning: reviewed knowledge candidates become usable only through explicit promotion and approved evidence policies.

  4. Research/shadow: retrieval, applicability and Hybrid reasoning experiments record observations without changing authoritative tickets.

How a recommendation is produced

Knotty triage and recommendation flow, from untrusted ticket text and evidence to validated draft, human decision and audited local action

Open the recommendation diagram at full size

  • Input remains untrusted. Ticket text, customer replies, and embedded instructions are evidence—not policy or approval.

  • Deterministic policy wins. Security, outage, SLA, evidence-readiness, revision, and escalation rules cannot be overridden by an LLM.

  • Drafting is bounded. Optional model-generated customer text must satisfy local response contracts; unavailable or invalid provider output falls back to deterministic drafting.

  • Nothing is automatically sent. Approving a customer-response field records a local workflow event; the project has no external messaging integration.

The full classifier, drafting and safety details are in the technical reference and threat model.

Human approval and the ticket lifecycle

Human-governed ticket triage sequence: investigate, save a pending proposal, require explicit approval, validate and commit, then read back

Open the approval sequence at full size

A proposal is not a ticket change. The operator first reviews evidence, risks, citations, proposed fields, and the customer-facing response. The service checks explicit actor/field authorization, source revision, routing, and lifecycle state before committing approved mutations. Readback confirms what changed and what stayed unchanged.

The lifecycle extends past this single approval:

evaluate → collect evidence → record diagnosis → human diagnosis review → customer update → fix or escalation → verification → ready-to-close → explicit closure

Exact actions vary by evidence and state. The system can escalate unresolved ambiguity instead of manufacturing a definitive answer. Fix approval is distinct from closure; new evidence can require review again.

The Approval Desk's Workflow Bar manages the current action, and the separate Pattern Bar appears for governed knowledge discovery. Their blocking reviews are enforced through the same backend service.

For a chronological MCP replay without network calls:

npm run evaluate:lifecycle-replay

Detailed diagnosis-and-fix walkthrough · Lifecycle replay guide · Full MCP action contracts

Hybrid diagnostic reasoning

Hybrid is the latest architectural expansion, but it must not be mistaken for an operationally deployed replacement for the existing SIMPLE workflow.

Knotty Hybrid diagnostic research architecture showing time-eligible historical evidence, B3/B4/B5 retrieval research, SIMPLE and HYBRID model lanes, blinded human review and no operational cutover

Open the Hybrid architecture at full size

The research architecture keeps the following pieces separate:

Research component

What is implemented or being investigated

B3 discovery

Local-first lexical retrieval, optional semantic embeddings, and provenance-preserving candidate sets across articles, known causes, playbooks and eligible resolved cases

B4 ranking

Offline lexical/semantic/reciprocal-rank-fusion comparisons; development evidence was inconclusive, so no ranking policy was promoted to authority

B5 applicability

Research on whether evidence actually applies to a case, including abstention and negative controls

H1–H5b Hybrid reasoning

Frozen, causally eligible inputs; provider-neutral contracts; immutable shadow runs; replay, idempotency and safe failure handling

Operational integration

Not completed; shadow output cannot approve, diagnose authoritatively, fix or close tickets

The purpose is not merely to supply more context to an LLM. It's to ask whether combining retrieval, structured evidence, diagnostic taxonomy, and model reasoning produces better-supported hypotheses without encouraging unsupported certainty or leaking knowledge from the future.

The experiments include an existing SIMPLE baseline and a HYBRID shadow lane. Provider/model identity is kept separate from the case opportunity and the human review packet. A claim that Hybrid works better requires scored, comparable results—not just successful execution.

Semantic retrieval is an optional research channel, with working components and bounded development evaluations, rather than a proven production capability. RAG (retrieval-augmented generation) supplies retrieved material to a model; B5 applicability assessment investigates whether that material actually supports the current case. Neither retrieval nor a valid model response establishes a correct diagnosis.

Bounded Luna comparisons in the case study (29 October 2026)

The presentation's fresh SIMPLE/Luna and B5/H5b HYBRID/Luna comparisons used lexical retrieval, with semantic vectors inactive. They ran against an isolated local evaluation revision, not an operational cutover on this repository's main branch.

The separate ranking follow-up compared the full candidate pool with existing B4 lexical selection plus preserved references on four paired synthetic cases (eight outputs). API-reported input tokens fell from 85,300 to 43,559, a 48.9% reduction, while 8/8 labeled relevant resource/case pairs were retained. This is a bounded context-selection result. It does not establish improved diagnostic quality, semantic-retrieval benefit, or billing savings. Cache conditions differed between the two inputs, so fewer input tokens cannot be read as a measured cost saving.

Human quality review remained pending at the time of the presentation. The PDF preserves that dated result and its limitations; it does not report a diagnostic-accuracy winner. These synthetic Luna experiments are separate from the historical Home Assistant E1a experiment below and the earlier Qwen retrieval study.

Hybrid shadow runner design · B3 retrieval plan · B4 ranking · B5 applicability · Hybrid implementation PR #30

Home Assistant historical-case experiment (E1a)

Knotty's earlier demonstrations use fictional support tickets. The E1a research goes further by studying historical public Home Assistant GitHub issues and time-eligible documentation. This is a controlled, local historical-case experiment, not an integration with Home Assistant support or an endorsed benchmark.

Every stage is limited by an asOf cutoff. A model should not see later GitHub comments, future solutions, or current documentation substituted for a missing historical version. Eligible documentation must have exact historical provenance and content hashes; otherwise it stays unavailable.

Preliminary technical results — October 2026

Measure

Status

Public historical cases

24

Time-scoped evidence checkpoints

64

SIMPLE/HYBRID × Qwen/Luna/Sol attempts

384

Structurally valid provider outputs

341

Invalid provider outputs

43

Valid-output packets prepared for blinded review

341

Completed diagnostic-accuracy comparison

Not yet available

341/384 is an output-validity result, not diagnosis accuracy. Malformed outputs remain reliability failures but do not receive invented human semantic ratings. Model and architecture identities remain concealed during blinded review; no provider ranking, Hybrid superiority claim, or winner is justified while adjudication and analysis are unfinished.

The E1a harness and raw evaluation artifacts currently reside in a separate local experimental worktree rather than in a reproducible benchmark command on this repository's main branch. Earlier local Qwen semantic-retrieval metrics and other controlled GPT comparisons are different experiments.

Governed knowledge evolution

Knotty governed knowledge evolution: completed diagnoses, candidate discovery, human promotion, immutable versions, advisory learning ledger and future evidence-gated reuse

Open the knowledge evolution diagram at full size

Knotty can suggest reusable knowledge after supported diagnoses, but this is not autonomous model retraining: no model can promote its own candidates or convert a discovered pattern into policy.

The sequence is: observed evidence → completed diagnosis → deterministic candidate discovery → optional advisory drafting → human policy review → explicit promotion → versioned future reuse. Candidates with incomplete or undecided evidence policies stay blocked. An approved known cause can still require new evidence in a later ticket; rejected, stale and contradicted objects do not silently become operational truth.

The SQLite learning ledger tracks candidate maturity and health, explicit promotions, immutable versions, reuse events and contradictions. Previously evaluated tickets retain their original source versions, so historical recommendations remain unchanged even as the approved knowledge catalog evolves.

npm run demo:knowledge-evolution
npm run demo:learning-ledger
npm run evaluate:knowledge-holdout

Detailed evidence-policy rules · Learning and operational failure boundaries

Persistence and reliability

Knotty reliability architecture with command IDs, deterministic policy validation, atomic operational SQLite, isolated learning outbox and restart-safe readback

Open the persistence and reliability diagram at full size

The authoritative operational plane uses transactional SQLite, not a set of mutable JSON/JSONL ticket records. Governed commands commit the projection/revision, ticket-local causal event, safe trace, replay result and any learning-outbox intent together.

Problem

Design choice

Stale human decisions

Optimistic source-revision and approval-time checks

Duplicate clicks or lost responses

Stable command IDs and persistent idempotent results

Ordering and auditability

Causal ticket-local event sequences and safe read models

Crash/restart uncertainty

Replayed command receipts; no reapplication of successful mutation

Learning repository unavailable

Operational success is isolated; advisory delivery retries or reports unavailability

Demo reset during a running process

Usage leases, guarded reset, verification and rollback

These are local reliability guarantees. SQLite files can still be modified by someone with filesystem access; this is neither a distributed multi-tenant help desk nor a cryptographically tamper-evident audit ledger.

npm run demo:operational-persistence
npm run evaluate:lifecycle-replay

For actual migration states, import contracts, reset safeguards, and crash/recovery restrictions, use the engineering reference.

Using the MCP server and Codex Skill

The repository-local Skill teaches Codex to inspect the current workflow, gather evidence, propose the next action and pause at the approval boundary.

  1. Run npm ci and npm run build in the repository root.

  2. Open the root as a project in Codex desktop. Review and trust .codex/config.toml; it starts node dist/src/index.js using the project working directory.

  3. Start a new thread and invoke the triaging-support-tickets Skill.

Use $triaging-support-tickets to review TKT-1005.
Show evidence, citations, escalation risks and proposed fields.
Stop and wait for my explicit approval before changing anything.

The standalone npm start script runs the stdio MCP server, not an HTTP page. An idle terminal is expected while it waits for MCP messages.

The server exposes 27 typed tools, 4 resources and 3 prompts (at the documented implementation baseline). Useful starting points include list_tickets, get_ticket_workflow, search_knowledge, evaluate_ticket, record_diagnosis, review_diagnosis, mark_response_done, and get_audit_events. Every mutation has specific schema and policy constraints; use the complete read, mutation, resource and prompt reference.

Optional model assistance

By default, the Approval Desk uses deterministic response drafting and needs no API key. Explicit opt-in example:

$env:OPENAI_API_KEY = 'set-locally-do-not-commit'
$env:APPROVAL_DRAFT_PROVIDER = 'openai'
npm run approval-desk

Provider availability is not required for deterministic routing, lifecycle, audit, or approval checks. Don't put credentials in tickets, screenshots or committed configs.

Useful configuration defaults

Environment setting

Default

TRIAGE_DATA_ROOT

data/runtime

TRIAGE_SEED_FILE

data/seed/tickets.json

TRIAGE_KNOWLEDGE_ROOT

data/knowledge

OPERATIONAL_DB_PATH

data/runtime/operational.sqlite

TRIAGE_LEARNING_LEDGER_PATH

data/runtime/knowledge-evolution/learning.sqlite

TRIAGE_KNOWLEDGE_APPROVERS

support-lead,reviewer,approval-desk

TRIAGE_MINUTES_SAVED

8 (an assumption, not measured labor time)

Advanced environment/provider options · Safe reset guidance · Retrieval index controls

Evaluation and current results

Different harnesses answer different questions. Fixture agreement, local controlled simulation, provider-output validity, and live-model diagnostic quality are not interchangeable metrics.

Command

What it proves (and what it does not)

npm test

Build, typecheck and local Vitest regression tests

npm run evaluate

Agreement between 30 preconstructed recommendations and expected outcomes; not unseen-ticket accuracy

npm run evaluate:diagnostics

11 controlled scenarios across 8 diagnostic families; nonmutating checks

npm run evaluate:lifecycle

Deterministic audit across all 30 seeded tickets; not 30 full chronological replays

npm run evaluate:lifecycle-replay

Real MCP lifecycle and read-before-action gates using temporary state

npm run evaluate:knowledge-holdout

Controlled future-turn evidence and approved-knowledge reuse

npm run evaluate:ai-comparison

Controlled local deterministic/advisory classifier and draft lanes; --live is explicitly opt-in

npm run evaluate:ai-diagnosis

Article-backed diagnoses and controlled optional advisory providers

npm run evaluate:retrieval

Local retrieval metrics; optional separately configured semantic experiments

npm run evaluate:ranking / npm run evaluate:applicability

B4/B5 development research, not an authorized operational ranking policy

To run the broad local verification (no live-model calls by default):

npm ci
npm test
npm run verify:portfolio

The 30-ticket seeded evaluate sample is intentionally constructed to match its reference outcomes; a printed perfect accuracy from that sample is not evidence of real-world diagnostic performance. Likewise, older GPT response-quality figures represent specific dated controlled or authenticated runs with separately documented limitations. E1a human evaluation is still in progress.

Full metrics, historical scorecards and commands · Diagnostic harness methods · AI comparison example · Retrieval reports

Trust boundaries and limitations

  • Synthetic local application: No live customer data, messaging, Zendesk/Jira updates, paging, or identity-provider integration. Home Assistant issues are historical external research inputs only.

  • Prompt injection: Ticket text and retrieved content cannot authorize actions, rewrite policy, or approve fields.

  • Explicit human intent: Pending proposals need review; an MCP payload alone cannot establish that a human personally intended an approval.

  • Causal time: Historical research must exclude future comments and documentation versions unavailable by the case cutoff.

  • Provider uncertainty: Malformed, contradictory, or unsupported model output can fail, abstain or fall back. Research gains are not assumed.

  • Audit and persistence: Local SQLite transactions are durable within documented conditions but aren't distributed, authenticated, or tamper-proof.

  • Reported savings: Queue estimatedMinutesSaved uses a configured assumption per approved recommendation, not stopwatch measurements.

  • Fixed fixtures: Synthetic SLA deadlines use June 10, 2026; use historical asOf where time dependence matters.

See SECURITY.md for the full threat model and limitations in the engineering reference.

Repository guide

Start with the current module map for source responsibilities, entry points, state ownership, execution paths, and test navigation. The documentation index distinguishes current guides from historical audits and proposed work.

Documentation: Architecture audit · Engineering reference · Case study · Demo walkthrough · Knowledge evolution · Lifecycle viewer · Evaluation guide · Roadmap · Capture guide.


Knotty is an ongoing personal engineering project. The implemented system is built around human oversight, deterministic policy and reproducible evidence; the Hybrid research asks how far that architecture can be improved without losing those boundaries.

Available Tools

6 tools
create_noteCreate noteA

Create a new local Markdown note without overwriting an existing note.

Inputs: id (required), title (required), body (required).

delete_noteDelete noteA
Destructive

Delete a local note after explicit confirmation.

Inputs: id (required), confirm (required).

list_notesList notesB
Read-onlyIdempotent

List local note IDs and titles.

Inputs: no named parameters.

read_noteRead noteB
Read-onlyIdempotent

Read one local Markdown note by ID.

Inputs: id (required).

search_notesSearch notesB
Read-onlyIdempotent

Search local note titles and bodies.

Inputs: query (required).

workspace_summaryWorkspace summaryA
Read-onlyIdempotent

Summarize the number and titles of local notes.

Inputs: no named parameters.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.7/5.0

    Scored across 6 tools

    Disambiguation5/5

    Each tool has a distinct purpose: create, delete, list, read, search, and summarize. No overlaps or ambiguities exist.

    Naming Consistency4/5

    All tool names use imperative verbs and snake_case, with 'workspace_summary' being a noun-based outlier but still following the same formatting.

    Tool Count5/5

    Six tools cover the essential CRUD operations plus search and summary without being excessive.

    Completeness3/5

    The set lacks an 'update_note' tool, which is a notable gap for full lifecycle management. Create and delete are present, but editing requires deletion and recreation.

    Maintenance

    ActivityActive
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    Appeared in Searches