Agentoom Knowledge

Agentoom Knowledge is a self-hosted Knowledge Server that exposes trusted enterprise context through the Model Context Protocol (MCP). It provides a unified retrieval layer over heterogeneous knowledge sources, ensuring that AI agents receive the most relevant context without needing to understand the underlying infrastructure.

Knowledge is heterogeneous. Retrieval should be too.

📖 Read the full article →  |  🚀 Installation Guide →  |  📚 How to Use →  |  🔌 Extending →

The Problem

Enterprise knowledge is inherently heterogeneous. It lives in fragmented systems, each optimized for a specific purpose:

• Documentation and Policies reside in Markdown or PDF files.

• Customer and Order Data live in relational SQL databases.

• Invoices and Receipts are stored as structured documents.

• System Configurations are managed in YAML or JSON.

• External APIs provide real-time state from third-party services.

Current AI retrieval systems often attempt to force these diverse sources into a single strategy. Some systems convert everything into vectors for semantic search, losing the precision of structured data. Others attempt to model everything as a graph, introducing unnecessary complexity for simple document retrieval.

While each strategy—semantic search, SQL queries, or API calls—is excellent within its own domain, no single strategy is optimal for every type of knowledge.

Related MCP server: MCP Local Context

Philosophy

Agentoom Knowledge is built on the principle that the retrieval strategy should match the nature of the knowledge.

Our goal is not to replace RAG, SQL, knowledge graphs, or APIs. Instead, the goal is to orchestrate them. Agentoom Knowledge acts as a deterministic abstraction layer that selects the most appropriate retrieval strategy for each query.

By separating the request for information from the execution of retrieval, we ensure that AI models remain focused on reasoning while the infrastructure handles the complexity of context gathering.

The Solution

Agentoom Knowledge is a self-hosted Knowledge Server designed for the enterprise. It exposes a trusted context window to AI agents via the Model Context Protocol (MCP).

Unlike orchestration frameworks that mix logic with retrieval, Agentoom Knowledge is focused purely on trusted context. The AI should never care whether a piece of information came from a vector database, a legacy SQL table, or a cloud API. It simply receives the unified context it needs to perform its task.

The Query Planner

At the heart of Agentoom Knowledge is the Query Planner. Inspired by database query optimizers, the planner is responsible for decomposing a context request into a set of executable tasks.

The Query Planner is deterministic and does not rely on an LLM for its core logic. This design choice provides several critical advantages:

• Predictability: The same request will always result in the same retrieval plan.

• Explainability: Every step of the retrieval process can be audited and understood.

• Performance: Deterministic planning has significantly lower latency than LLM-based reasoning.

• Cost Efficiency: No token costs are incurred during the planning phase.

The planner identifies which knowledge providers are relevant to a query, executes them in parallel or sequence, and merges the results into a coherent, ranked context window.

How It Works

The planning process follows a deterministic, multi-step pipeline:

1. Query Analysis: The planner receives a request containing the search query, optional namespaces, and requested search types (e.g., semantic or structured).

2. Provider Discovery: It consults the Metadata Registry to identify all registered knowledge providers that match the query's scope.

3. Capability-aware Routing: For each relevant provider, the planner determines the optimal operation based on the provider's capabilities. If a specific search type is requested and supported, it is prioritized; otherwise, it defaults to a general search.

4. Execution Planning: A structured Execution Plan is generated, consisting of discrete steps for each provider. Each step includes the specific parameters and operations needed.

5. Parallel Execution: The Retrieval Engine takes the plan and executes all steps in parallel using Laravel's concurrency features, ensuring that slow providers don't block the entire request.

6. Result Fusion: Finally, results from all providers are merged using Reciprocal Rank Fusion (RRF). This ensures that the most relevant information from diverse sources (e.g., a SQL table and a vector index) is ranked appropriately in the final context window.

Handling Conflicts, Priority, and Freshness

To ensure the context window is both accurate and authoritative, the Query Planner employs several advanced strategies:

Conflicting Information

The planner does not attempt to "resolve" factual conflicts at the retrieval layer. Instead, it uses Reciprocal Rank Fusion (RRF) to score snippets based on their consistency and relevance across multiple sources.

• Deduplication: Results are keyed by content hash or unique ID. Identical information from different sources is merged, increasing its overall rank.

• Contextual Diversity: When sources disagree, the planner preserves the conflicting perspectives in the final context. This allows the AI agent to see the "full picture" and apply its own reasoning to the evidence provided.

Source Priority & Authority

Not all knowledge is equal. The Query Planner respects the Source Priority defined in the Metadata Registry:

• Weighted Planning: The planner sorts execution steps based on the priority of the underlying Knowledge Source. High-priority sources (e.g., Official Policies) are prioritized in the execution plan over lower-priority sources (e.g., Community Wikis).

• Namespace Isolation: Authority is further enforced through Namespaces. By scoping queries to specific namespaces, users can ensure that only vetted, authoritative providers are consulted for sensitive requests.

Real-time vs. Cached Data

Agentoom Knowledge balances the speed of indexed search with the accuracy of live data:

• Hybrid Execution: The system simultaneously queries real-time providers (SQL, Filesystems) and indexed providers (Typesense).

• Discovery Caching: While retrieval is often real-time, the Metadata Registry is cached. This ensures the planner can identify the best sources in milliseconds without hitting the database on every request.

• Parallel Resilience: By using Laravel's concurrency layer, the planner ensures that slow real-time lookups (like a complex SQL join) do not delay the delivery of faster cached results.

Retrieval Philosophy

Agentoom Knowledge combines retrieval strategies rather than forcing a "one size fits all" approach.

Architecture

Agentoom Knowledge follows a deterministic flow from client request to context delivery.

graph TD
    Client[AI Client]
    MCP[MCP Interface]
    QP[Query Planner]
    Providers[Knowledge Providers]
    
    subgraph Retrieval [Retrieval Strategies]
        Semantic[Semantic Search]
        SQL[SQL Queries]
        YAML[YAML / JSON]
        Filesystem[Filesystem]
        Markdown[Markdown]
        Web[Web Crawler]
        Federation[Federated Servers]
    end
    
    Context[Unified Context]
    
    Client --> MCP
    MCP --> QP
    QP --> Providers
    Providers --> Semantic
    Providers --> SQL
    Providers --> YAML
    Providers --> Filesystem
    Providers --> Markdown
    Providers --> Web
    Providers --> Federation
    Semantic --> Context
    SQL --> Context
    YAML --> Context
    Filesystem --> Context
    Markdown --> Context
    Web --> Context
    Federation --> Context
    Context --> Client

Implementation Details

Embedding Generation

Agentoom Knowledge uses Managed Embeddings handled internally by the vector store (Typesense).

• Internal Processing: When a document is processed by the DocumentPipeline, the IndexChunk job sends raw text content to Typesense.

• No External Latency: By offloading vectorization to Typesense's built-in machine learning capabilities, the system avoids the latency, cost, and privacy concerns associated with calling external services like OpenAI or Cohere during the indexing loop.

• Consistency: This ensures that the same model is used for both indexing and query vectorization, maintained entirely within your self-hosted infrastructure.

Typesense Schema Management

The system avoids the complexity of mapping diverse enterprise schemas into a vector store by using a Hybrid Storage Strategy:

• Unified Document Index: Unstructured data (Markdown, PDFs, etc.) is parsed and decomposed into a unified knowledge_chunks collection. This collection uses a fixed, flattened schema that includes content, sequence, and source metadata.

• Native Structured Retrieval: For SQL databases and YAML structures, the system does not attempt to force them into Typesense collections. Instead, the SqlProvider and YamlProvider query the source data natively and in real-time.

• Precision over Flattening: This approach preserves the relational integrity and precision of structured data while allowing semantic search to operate on the types of knowledge where it excels (documentation and prose).

Authentication & Authorization

Security is a first-class citizen in Agentoom Knowledge, especially given the sensitivity of enterprise data:

• MCP API Guard: All requests to the MCP server are protected by a custom mcp_api authentication guard.

• API Keys: Access is managed through API Keys with granular scopes (e.g., mcp:use, admin:*). These keys must be provided as Bearer tokens in the MCP connection.

• Multi-tenancy Ready: While currently focused on single-instance enterprise deployment, the core data models (KnowledgeSource, Provider, Document, ApiKey) include tenant_id columns with foreign key constraints, ensuring that future multi-tenant deployments have logical isolation at the database and retrieval layers.

• Rate Limiting: The MCP API endpoint is protected by configurable per-API-key rate limiting (default: 60 requests/minute). Limits are managed from the Admin UI under Settings → Rate Limiting and can be disabled per deployment.

Lifecycle Automation & Syncing

The system handles the complexity of keeping knowledge sources in sync through an automated lifecycle:

• Source Observers: When a KnowledgeSource is created or updated in the Admin panel, Eloquent Observers automatically manage the underlying Provider models and their technical configurations.

• Pipeline Orchestration: New documents are automatically routed through a multi-stage pipeline (Discover → Parse → Normalize → Chunk → Enrich → Index) via batched queue jobs.

• Artisan Commands: knowledge:pipeline:run triggers document discovery and processing for active sources.

• Scheduled Maintenance: The Laravel scheduler runs periodic tasks — Horizon metric snapshots every 5 minutes, federation capability sync every 15 minutes, and daily retrieval log pruning (configurable from Settings → Maintenance).

Observability & The Search Playground

Agentoom Knowledge provides deep visibility into its "black box" retrieval logic:

• Retrieval Logging: Every request processed by the engine is logged with its full query, deterministic execution plan, fused results, and precise latency metrics.

• Search Playground: An interactive internal tool allows administrators to simulate agent requests. It visualizes the Reasoning (the step-by-step execution plan) alongside the Evidence (the final ranked results), making it easy to debug retrieval quality.

• Performance Metrics: The dashboard tracks real-time health, including Horizon queue status and vector store (Typesense) metrics, ensuring the system remains responsive under load.

• Health Endpoint: A GET /health JSON endpoint reports the status of database, Redis, Typesense, and storage. Designed for Docker healthchecks, Kubernetes probes, and load-balancer monitoring — returns 200 when all services are healthy, 503 if any critical service is down.

• Notification Pipeline: Configurable email and webhook alerts for operational events — high search latency, consecutive sync failures, and federation errors. Thresholds, alert types, and cooldown windows are managed from the Admin UI under Settings → Notifications.

Enterprise Administration

Phase 9 adds the operational controls enterprises need to audit and maintain a knowledge deployment:

• Activity / Audit Trail: Every knowledge source, API key, and settings mutation is recorded in an append-only activity_log with the actor, action, subject, IP address, and a property diff. Sensitive values — passwords, tokens, API keys, and hashes — are redacted before storage, so encrypted SQL credentials and key hashes never reach the audit trail. Browse and filter it under Admin → Activity Log (admin/operator only); settings deletion is audited too, and the Danger Zone reset truncates the same table.

• Document Reprocessing: ParseDocument retries transient Tika/OCR failures (3 attempts with 30s/120s backoff) before leaving a document in error. PipelineOrchestrator::reprocess() resets an errored, non-web document to discovered, de-indexes and clears its stale chunks, and re-queues parsing — reachable from the document detail page, the file manager, or the knowledge:documents:reprocess artisan command (--source= and --limit=). Web documents are re-fetched through their source pipeline, never reparsed from URL paths.

• Knowledge Source Templates: The create wizard ships versioned presets (markdown_docs, filesystem_documents, web_docs, sql_table) that prefill name, namespace, and provider config. Templates never contain credentials, and a slug-collision check surfaces a field error instead of a database unique-key exception.

Federation

Agentoom Knowledge servers can be federated so that a single instance queries multiple servers transparently:

• FederatedServer Model: Each remote server is registered with an endpoint URL, encrypted API token, and priority.

• FederationPlanner: Extends the query planner to include federation steps alongside local providers. Local results take priority; remote results augment with lower rank weight.

• FederationProvider: Acts as an MCP client — translates local SearchQuery objects into JSON-RPC tools/call requests to the remote server's search_knowledge endpoint. Results are tagged with _federation_source for traceability.

• Result Fusion: Remote results are fused with local results via RRF, so the AI receives a single unified context window regardless of how many servers contributed.

• Admin UI: Full CRUD for federation servers with connection testing and remote capability syncing.

Chunking Strategies

The document pipeline uses content-type-aware chunking to preserve semantic meaning:

• MarkdownChunking: Heading-aware splitting — splits on # headers, falls back to paragraph boundaries.

• SemanticChunking: Paragraph and sentence boundary detection — produces coherent chunks that never break mid-thought.

• SlidingWindowChunking: Overlapping windows for code and structured data — ensures no context is lost at chunk boundaries.

• FixedSizeChunking: Character-based with word-boundary respect — the safe fallback.

The ChunkingStrategyRegistry automatically selects the best strategy based on MIME type and file extension.

Token-aware enforcement: Every chunk produced by any strategy passes through the TokenAwareChunker, which caps chunks at the configured knowledge.chunk_max_tokens ceiling (default 384, safely inside the installed managed embedding model's 512-token window) and splits oversized chunks at token boundaries with knowledge.chunk_overlap_tokens overlap (default 64). The TokenCounter uses a deterministic UTF-8 tokenizer so the persisted token_count, the indexing metadata, and the document detail view all report the same value. Both settings are managed under Settings → Search Config; on re-chunking, prior chunk vectors are de-indexed first so retries or config changes never leave orphaned vectors behind.

Search Quality

Phase 7 introduced four orthogonal search quality improvements that operate across the retrieval pipeline:

Hybrid Keyword+Vector Search

When search_type=hybrid, the SemanticProvider sends both a keyword query and a vector query to Typesense in a single request. Typesense fuses the results using its built-in hybrid ranking, balancing keyword precision with semantic recall.

• Configurable Alpha: The keyword vs. vector weight is controlled by the knowledge.hybrid_alpha setting (0.0–1.0, default 0.5). Admin UI provides a slider under Settings → Search Config.

• A/B Testing in Playground: The Search Playground includes a side-by-side comparison mode — toggle "A/B Compare" to run the same query with two different search types simultaneously and see which results each strategy surfaces (or misses), with rank-change indicators and a unique-chunk diff summary. This makes tuning the alpha slider actionable rather than guesswork.

• Managed Embeddings: Uses the existing ts/all-MiniLM-L12-v2 model configured during indexing — no external embedding calls needed.

• Fully Optional: Default search (search_type null/absent) remains keyword-only for backward compatibility.

Content Deduplication via SHA-256

Duplicate content is detected and blocked at multiple stages of the document pipeline to prevent redundant indexing:

• Upload-time dedup: The FileManager computes hash_file('sha256', …) on every uploaded file and filters out records whose hash already exists in the database (excluding stale/error documents). Within-batch duplicates (same file dragged twice) are also caught via a local dedup map.

• Parse-stage dedup: ParseDocument computes hash('sha256', $content) after Tika extraction. If the content hash matches an existing non-stale document, the document is marked status = 'duplicate' (orange badge in the admin UI) and its chunks are de-indexed.

• Sync-time dedup: SyncKnowledgeSource filters filesystem scans against known content hashes before inserting new records.

• Result filtering: The SemanticProvider cross-references document IDs against the database, excluding any document whose status is not indexed from search results.

Configurable Synonym Expansion

Query-time synonym expansion rewrites search terms using administrator-defined synonym groups before sending the query to Typesense:

• Synonym Groups: Defined via the Admin → Synonyms page. Each group is a set of equivalent terms (e.g., ["car", "automobile", "vehicle"]).

• Query Rewriting: The QueryRewriter expands matching tokens by appending synonym terms to the query (Typesense's q parameter works with term-appended expansion, not boolean OR syntax). For example, "deployment pipeline" becomes "deployment pipeline release automation \"continuous delivery\"".

• Toggleable: Controlled by the knowledge.synonym_expansion_enabled setting in Settings → Search Config. Disabling stops all expansion — no reindexing needed.

• Expansion Cap: A configurable knowledge.synonym_expansion_max_terms setting (default 10, range 2–100) caps how many synonym variants are appended per token. Prevents over-expansion from large synonym groups causing query bloat and precision loss.

• Ranking behavior: Typesense's TF-IDF scoring treats synonym terms with equal weight to original query terms — a document matching 4 synonym terms may outrank one matching 2 original terms. The expansion cap is the primary defense: keep it low (default 10) for precision-biased deployments, raise it for recall-heavy ones.

• Multi-word phrases: Multi-word synonyms are wrapped in double quotes to preserve phrase matching (e.g., "continuous delivery").

Synonym Weighting

When synonym groups grow large, expanded terms can drown out original-query matches in ranking. Phase 9 adds an optional two-pass weighting pass in the SemanticProvider:

• Original-query first: When enabled, the provider runs a second search over the expanded query. Items matching the original terms keep their position, while synonym-only items are appended with their score multiplied by knowledge.synonym_penalty_factor (default 0.5).

• Bounded recall: Both passes pull from a recall pool capped at 250 (min(250, max(maxResults × 2, 50))), then truncate to maxResults before the indexed-document filter.

• Toggleable: Controlled by knowledge.synonym_weighting_enabled in Settings → Search Config. Disabled (default) preserves the legacy single-pass ranking exactly.

• Local-provider only: Weighting applies to the local SemanticProvider; federated results are unchanged since their protocol carries no original-term provenance.

Recency-Aware Reciprocal Rank Fusion

The RRF fusion strategy was extended with an optional recency boost that gives fresher content a scoring advantage:

• Exponential Decay Formula: final_score = rrf_score × (1 + boostFactor × e^(-λ × days_since_indexed)) where λ = ln(2) / halfLifeDays.

• Configurable parameters: knowledge.recency_boost_enabled (toggle), knowledge.recency_boost_factor (0.0–1.0, default 0.3), and knowledge.recency_boost_half_life_days (1–365, default 30). Managed from Settings → Search Config.

• Neutral for old content: Items without timestamps or very old content get a multiplier of ~1.0 — they are never penalized, just not boosted.

• Timestamp source: The SemanticProvider maps Typesense's auto-added created_at field into the result item's indexed_at for recency scoring. Items from federation providers without timestamps receive neutral treatment.

• Backward compatible: The RecencyBoostConfig parameter on ResultFusionStrategy::fuse() is nullable — passing null preserves the original RRF behavior unchanged.

Retrieval Quality Metrics

Recency boost follows exponential decay — the boost halves every half-life period. A concrete example with defaults (boostFactor = 0.3, halfLifeDays = 30):

Items without timestamps (e.g., from federation providers that don't return indexed_at) receive a multiplier of exactly 1.0 — they are never disadvantaged, just not boosted.

Concrete Quality Evidence

Two A/B comparisons demonstrating Phase 7 improvements on real documents:

Hybrid vs. Keyword-only — query: "quarterly revenue"

Synonyms vs. No Synonyms — query: "deployment pipeline"
Synonym groups: ["deployment", "release", "ship"], ["pipeline", "automation", "continuous delivery"]

Without synonyms, infra.txt — which describes the same concepts using different vocabulary — would never appear. Synonym expansion bridges that gap.

These comparisons were produced using the A/B Compare toggle in the Search Playground, which shows results side-by-side with rank-change indicators and a diff summary of chunks unique to each strategy.

Web Provider & Crawling

The WebProvider fetches and converts content from configured URLs into searchable Markdown via league/html-to-markdown. For larger documentation sites, it supports recursive crawling:

• Crawl Configuration: Set max_depth, max_pages, allowed_domains, and politeness_delay_ms per source.

• Robots.txt Compliance: The RobotsTxt utility fetches and caches robots.txt rules, respecting Disallow directives per user agent.

• Content Extraction: Navigation, footers, headers, scripts, and styles are stripped before Markdown conversion, leaving clean, structured text.

• Recursive Discovery: The CrawlWebSource job discovers <a href> links from each page and dispatches child jobs for the next depth level, respecting domain and pattern exclusions.

• Batched Processing: Each crawled page is stored as a Document and flows through the standard parsing/chunking/indexing pipeline.

Features

• Self-hosted: Total control over your data and infrastructure — runs on Docker.

• MCP Server: Native Model Context Protocol endpoint with search_knowledge, list_sources, and get_source_schema tools.

• Deterministic Query Planner: Reliable, auditable retrieval — no LLM in the retrieval path. Federation-aware: queries local and remote servers transparently.

• Hybrid Providers: Filesystem (multi-format), Markdown, SQL, YAML, JSON, Web, Vector (Typesense), and Federation providers built in. Each provider searches in its native format — YAML returns key-value hits, SQL returns row results, Markdown uses heading-aware chunking. All filesystem-backed providers support UI uploads via a built-in file manager.

• Recursive Web Crawling: Domain-scoped crawling with configurable depth, politeness, robots.txt compliance, and link exclusion patterns — ingested straight into the document pipeline.

• HTML-to-Markdown: Web content and crawled pages are converted to clean Markdown via league/html-to-markdown, preserving headings, lists, code blocks, and links.

• Advanced Chunking: Four chunking strategies (Semantic, Sliding Window, Markdown, Fixed Size) with content-type-aware auto-selection.

• Reciprocal Rank Fusion: Results from multiple providers merged and deduplicated by rank.

• Metadata Registry: Centralized, cached registry of all knowledge source capabilities and schemas.

• Document Pipeline: Automated multi-stage pipeline (Discover → Parse → Chunk → Enrich → Index).

• MCP Federation: Connect multiple Agentoom Knowledge servers together — queries execute across all federated peers with unified result fusion.

• Provider SDK: Extensible architecture with 10 contracts, make:knowledge-provider generator, auto-discovery via config, and full extension guide.

• Admin UI: Livewire + Flux admin panel for managing sources, providers, federation servers, users, and settings.

• Retrieval Logging: Full audit trail with query text, execution plans, fused results, and latency.

• Search Playground: Interactive sandbox to test queries, visualize retrieval plans in real time, and A/B compare search types side-by-side with rank-change indicators and unique-result diff summaries.

• Danger Zone Reset: One-click app reset from Settings — clears all knowledge data, search indexes, and logs while preserving users and configuration.

• API Key Auth: Scoped API key authentication for MCP access with separate user/service-account keys and prefix-optimized lookups.

• Rate Limiting: Configurable per-API-key rate limiting on the MCP endpoint to prevent abuse.

• Health Endpoint: GET /health JSON endpoint for Docker healthchecks and load-balancer probes — checks database, Redis, Typesense, and storage.

• Notification Pipeline: Email and webhook alerts for high search latency, sync failures, and federation errors — configurable thresholds, alert types, and cooldown windows.

• Scheduler Maintenance: Automated Horizon metric snapshots, federation capability sync, and retrieval log pruning via Laravel's scheduler.

• Horizon Queues: Background indexing and document processing via Redis queues.

• Hybrid Search: Combined keyword+vector search in Typesense with configurable alpha weighting — balances precision and semantic recall in a single query.

• Content Deduplication: SHA-256 hashing at upload, parse, and sync stages prevents duplicate content from entering the index — duplicates are flagged and their chunks de-indexed.

• Synonym Expansion: Configurable synonym groups append equivalent terms to queries at search time (e.g., "car" → "car automobile vehicle") with a per-token expansion cap to prevent query bloat — toggleable per deployment with no reindexing.

• Recency-aware Ranking: Exponential-decay recency boost in RRF fusion — fresh content surfaces higher (configurable boost factor and half-life).

• Activity / Audit Trail: Append-only audit log of knowledge source, API key, and setting changes with actor, action, subject, IP, and redacted properties — browsable under Admin → Activity Log.

• Token-Aware Chunking: A deterministic UTF-8 tokenizer caps every persisted chunk within the configured LLM context window (default 384 tokens), regardless of the active chunking strategy.

• Document Reprocessing: ParseDocument retries transient Tika/OCR failures automatically (3 attempts), and errored documents can be re-queued from the UI or the knowledge:documents:reprocess command.

• Knowledge Source Templates: One-click presets (Markdown docs, filesystem, web, SQL table) prefill the create wizard without shipping credentials.

• Synonym Weighting: Two-pass search scores original-query matches above synonym-only matches (configurable penalty) to protect precision as synonym groups grow.

• Apache Tika Integration: Robust parsing for hundreds of document formats (PDF, DOCX, etc.).

• External Embedding Providers: OpenAI, Cohere, and local HuggingFace implementations through the EmbeddingProvider contract.

• MCP Resources: Browse documents and knowledge sources through MCP resources.

• OCR Fallback: Local PaddleOCR processing for image documents when Apache Tika returns empty or near-empty content.

• Passkeys + 2FA: Fortify-powered authentication with passkeys and TOTP two-factor auth.

• Role-based Access: Admin, Operator, and Viewer roles for UI authorization.

Why Laravel?

Agentoom Knowledge is implemented using the Laravel framework. We chose Laravel because it is the premier ecosystem for building robust, maintainable infrastructure.

While Python is the standard for training AI models, this project is about orchestration and infrastructure. Laravel excels in the areas that matter most for a Knowledge Server:

• Orchestration: Powerful dependency injection and service container.

• Reliability: Mature queue systems (Horizon) and event broadcasting.

• Extensibility: A world-class package ecosystem and clean architectural patterns.

• Administration: Elegant tools for building secure, intuitive management interfaces.

• Integration: Seamless handling of databases, filesystems, and external APIs.

By leveraging Laravel, we provide a stable, scalable foundation that enterprise engineers can trust and contribute to.

What This Project Is NOT

To maintain a focused vision, it is important to define what Agentoom Knowledge is not:

• ❌ AI Agent Framework: We provide context; we don't build the agents.

• ❌ Workflow Engine: We don't manage complex multi-step AI business logic.

• ❌ Prompt Library: We focus on data retrieval, not prompt engineering.

• ❌ LLM Wrapper: We are a standalone infrastructure component.

• ❌ Vector Database: We integrate with vector stores but provide much more.

• ❌ Graph Database: We are a retrieval layer, not a primary storage engine.

• ❌ AI Model: we do not train or host LLMs.

• ❌ RAG Framework: We are a complete Knowledge Server, not just a library.

Installation

See the Installation Guide for step-by-step setup instructions covering both development and production environments.

For day-to-day usage after installation, see the How to Use guide. To build custom providers, see the Extending guide.

Prerequisites

• Docker and Docker Compose

• Laravel Sail (included)

Quick Start (Development)

git clone https://github.com/agentoom/knowledge.git
cd knowledge

# Copy and configure environment
cp .env.example .env

# Start Docker containers (PostgreSQL, Redis, Typesense)
vendor/bin/sail up -d

# Install dependencies
vendor/bin/sail composer install
vendor/bin/sail npm install

# Generate app key and run migrations
vendor/bin/sail artisan key:generate
vendor/bin/sail artisan migrate --seed

# Index demo data and build the metadata registry
vendor/bin/sail artisan knowledge:chunks:index
vendor/bin/sail artisan knowledge:providers:sync
vendor/bin/sail artisan knowledge:registry:refresh

# Build frontend assets
vendor/bin/sail npm run build

# Start the dev server
vendor/bin/sail artisan serve

The application will be available at http://localhost:8000.

Testing

A .env.testing file is provided with CI-friendly defaults (SQLite in-memory database, array cache and session drivers, sync queue). To run tests without Docker services:

cp .env.testing .env
vendor/bin/sail artisan test --compact

Note: some tests that exercise the vector store (Typesense) or Horizon will need Docker services running.

Default Admin User

The seeder creates an admin user. By default the email is admin@agentoom.com and a random password is generated and printed to the console during seeding (also written to storage/app/initial-admin-password.txt).

To use a fixed password instead (e.g., for automated evaluation environments), set both ADMIN_EMAIL and ADMIN_PASSWORD in your .env file before seeding.

# Example: override with known credentials
ADMIN_EMAIL=ci@example.com
ADMIN_PASSWORD=my-secure-password

The seeder is idempotent — you can safely re-run migrate --seed without duplicate key errors.

Queue Worker

For background document processing, start a queue worker:

vendor/bin/sail artisan horizon

Screenshots

Screenshots coming soon.

Roadmap

• Phase 1: Core MCP implementation, basic SQL/Semantic providers. ✅

• Phase 2: Query Planner strategies, Reciprocal Rank Fusion, observability. ✅

• Phase 3: Web Provider, Document Pipeline automation, Admin UI. ✅

• Phase 4: Advanced chunking strategies, HTML-to-Markdown conversion, provider SDK formalization. ✅

• Phase 5: True web crawling (domain recursive), MCP federation. ✅

• Phase 6: Production hardening — Laravel scheduler for periodic maintenance (Horizon snapshots, federation sync, log pruning), rate limiting on the MCP API endpoint, health-check endpoint for Docker and load balancers, notification pipeline for sync failures and high-latency alerts. ✅

• Phase 7: Search quality — hybrid keyword+vector search in Typesense, content deduplication via SHA-256 hashing in the parse stage, configurable synonym expansion for query rewriting, recency-aware scoring in Reciprocal Rank Fusion so fresher content surfaces higher. ✅

• Phase 8: Provider completeness — external embedding provider implementation (OpenAI, Cohere, local HuggingFace) through the existing EmbeddingProvider contract, MCP resources for document and source browsing, OCR fallback for images: a local OCR engine (e.g., PaddleOCR) running in the same Docker stack that the pipeline calls only when Tika returns empty or near-empty content from image files (jpg, png, tiff, etc.), so image-based documents become searchable without external API dependencies. Non-image parsing stays on Tika; OCR is a targeted gap-filler, not a replacement. ✅

• Phase 9: Enterprise features — activity/audit trail tracking who changed what (sources, API keys, settings), token-aware chunking that respects LLM context windows, retry/reprocess mechanism for documents stuck in error status after transient Tika failures, knowledge source templates for one-click setup of common configurations, synonym weighting — score documents matching original query terms higher than those matching only expanded synonyms (two-pass search or post-retrieval score penalty) to prevent recall from drowning precision when synonym groups grow large. ✅

• Phase 10: Test coverage — dedicated tests for each provider (Yaml, Json, Markdown, Web, Sql), chunking strategy tests for all four strategies, Livewire component tests for Playground, ApiKeys, DangerZone, and Dashboard.

• Phase 11: Future horizons — knowledge graph traversal, semantic caching of retrieval results, multi-tenancy with row-level data isolation (schema already has tenant_id columns), fine-tuned embedding models for domain-specific knowledge, and additional OCR/parser improvements.

Plugin Ecosystem

Agentoom Knowledge is designed to be extended. Its modular architecture allows developers to contribute to a growing ecosystem:

• Providers: Add support for new data sources (e.g., Jira, Salesforce, Slack).

• Vector Stores: Integrate with Pinecone, Milvus, or Weaviate.

• Embedding Providers: Use OpenAI, Cohere, or local HuggingFace models.

• Planner Strategies: Implement custom logic for specialized domain retrieval.

• Parsers: Extend Apache Tika with specialized document extractors.

Design Principles

1. Knowledge is heterogeneous. Retrieval should be too. Strategy must match the data.

2. Deterministic over magical. Prefer explainable logic over black-box LLM reasoning.

3. Infrastructure over framework. Be a reliable component, not a restrictive cage.

4. Self-hosted first. Privacy and data sovereignty are non-negotiable.

5. Open standards. Build on MCP and other interoperable protocols.

6. Replaceable components. Use Laravel Contracts to allow swapping any major driver.

7. Database-driven configuration. Manage the system through UI and API, not just files.

Future

The architecture of Agentoom Knowledge is built to support the next generation of AI infrastructure. See Phase 11 for forward-looking items including knowledge graphs, semantic caching, multi-tenancy, and fine-tuned embeddings.

Relationship with Agentoom

Agentoom Knowledge is extracted from the architecture behind Agentoom and released as a standalone open-source project because we believe the community benefits from a high-quality, self-hosted knowledge infrastructure that anyone can use.

License

The Agentoom Knowledge server is open-source software licensed under the MIT license.