Dense Mem

Dense-Mem is a standalone HTTP MCP memory server using Streamable HTTP. It stages exact evidence, derives semantic state through validated server policy, and returns active evidence contexts with graph-shaped Relationship handles. PostgreSQL is the durable authority for knowledge, lifecycle, provenance, search, authorization, and audit; Redis is coordination only. A single-node deployment may use process-local coordination; a multi-instance deployment requires Redis or an equivalent distributed coordination implementation.

The host LLM owns conversation and judgment. Dense-Mem owns durable evidence, owner authorization, lifecycle events, support eligibility, and bounded recall. The external memory automation contract is MCP at /mcp; browser routes are first-party interfaces, not an alternative public automation API.

Dense-Mem is part of the research preprint Governed Enterprise AI Memory Beyond RAG: From Vector Retrieval to Permissioned Knowledge Graphs.

Try the Hosted Demo

Create a temporary isolated team at https://demo-dense-mem.markhuang.ai to test disposable data before self-hosting.

Related MCP server: Strata

Why Dense-Mem

• Evidence is exact, durable, and append-only. A lifecycle action changes its effective state without deleting provenance or trace lineage.

• Entity and typed Value are semantic nodes. Owner-alias-owned Relationships become active graph edges only when their evidence support is eligible.

• Provider output is a proposal. Closed-schema validation and deterministic server policy decide durable state.

• Default recall excludes candidates and Hypotheses and returns evidence only when its active Relationship support path is eligible for the requested time.

• Team visibility and owner mutation authority are distinct. An author can change only their own evidence or owned semantic records.

Authentication resolves one immutable actor as team + identity + membership + permanent owner alias + optional credential. An SSO browser session uses the selected membership's permanent owner alias and has no direct credential. An API-key request carries a credential whose stable ID is also its permanent owner alias. Team, identity, membership, and credential fields never let a client choose or replace the semantic owner.

60-Second Quickstart

Download the local compose example and environment template, configure the required secrets, and start Dense-Mem:

mkdir dense-mem-local
cd dense-mem-local

curl -fsSLo docker-compose.yml \
  https://raw.githubusercontent.com/markhuangai/dense-mem/main/examples/docker-compose.base.yml
curl -fsSLo .env.example \
  https://raw.githubusercontent.com/markhuangai/dense-mem/main/examples/.env.example

cp .env.example .env
# Fill in POSTGRES_PASSWORD, CONTROL_PORTAL_TOKEN, and AI_API_KEY.
${EDITOR:-vi} .env

docker compose up -d

The base stack uses PostgreSQL with pgvector as the only durable authority. The v2.6.2 release requires the compatible cutover marker created by the stopped-service migration; it has no legacy database runtime or fallback. The local ports are:

MCP:            http://127.0.0.1:8080/mcp
User portal:    http://127.0.0.1:8080/ui
Control portal: http://127.0.0.1:8090/

Open the control portal with CONTROL_PORTAL_TOKEN, then create a team and its first credential/API key. For control-plane automation, use the same private API:

control_token="<CONTROL_PORTAL_TOKEN from .env>"

curl -fsS -X POST http://127.0.0.1:8090/control/api/teams \
  -H "Authorization: Bearer ${control_token}" \
  -H "Content-Type: application/json" \
  -d '{"name":"primary-memory"}'

curl -fsS -X POST http://127.0.0.1:8090/control/api/teams/<team-id>/credentials \
  -H "Authorization: Bearer ${control_token}" \
  -H "Content-Type: application/json" \
  -d '{"name":"default credential"}'

Operational diagnostics use one process logger for console and the private control portal. LOG_LEVEL accepts trace, debug, info, warn, error, or fatal and defaults to trace; fatal records severity and does not stop the process. PostgreSQL slow-query logging uses POSTGRES_SLOW_QUERY_THRESHOLD_MS, which defaults to 200 and must be a positive value. The operation-log sink is required for readiness and reports gaps and recovery when it is unavailable. Startup and the OAuth compatibility harness use console-only logging until a PostgreSQL operation-log sink is available; they do not claim that pre-attachment events were persisted.

The control portal's Logs view filters by severity, correlation ID, request hash, Remember attempt, execution/replay/conflict classification, and retry eligibility. A team's Remember diagnostics open on admitted calls and link to canonical attempts. Capture bodies are loaded only for an authorized detail view and are labelled as observed preparation, write, disconnect, or unknown receipt; a captured response does not prove delivery to the caller. Existing seven-day capture expiry, legal holds, erasure, and bounded unavailable or truncated states remain in force.

The release image contains one project executable, /app/server. It applies pending PostgreSQL migrations under a database session lock before serving, so the Compose stack does not need a separate migration container. Multiple server replicas that share one writable primary serialize this startup step. Keep ordinary rolling-deployment migrations backward compatible with the previous app version. The v2.6.1 synchronous Remember migration remains the stopped-service boundary; v2.6.2 evidence-first activation runs after it. For the v2.6.1 migration, stop every server instance, take the required PostgreSQL snapshot, and apply the stopped-service boundary before starting the new binary. Independent databases must each be migrated by a server connected to that database. Administration stays on the private control portal/API, while dreaming and automatic conflict review run as server background workers.

The image healthcheck allows the default 30-minute migration window and becomes active after its first success. If POSTGRES_MIGRATION_TIMEOUT_SECONDS is set above 1800, override the deployment healthcheck start period to at least the same duration.

Release candidates use vX.Y.Z-rc.N and demo-vX.Y.Z-rc.N. Stable releases use vX.Y.Z, latest, and demo-vX.Y.Z; there is no rolling demo tag.

The server requires complete embedding and verifier configuration at startup: AI_API_URL, AI_API_KEY, AI_API_EMBEDDING_MODEL, AI_API_EMBEDDING_DIMENSIONS, and AI_VERIFIER_MODEL. The compose examples provide OpenAI defaults for embeddings; choose the chat models explicitly in .env. AI_API_EMBEDDING_MAX_BATCH_ITEMS limits texts per provider HTTP request. It defaults to 256, matching the application batch limit; set it to 100 for Cloudflare Workers AI BGE-M3. One application batch still returns one ordered result, and a failed request fails the whole batch. Successful chunks stay private within the call; transient failures retry only the failed chunk, sharing three retries across the complete batch. Search reconciliation selects at most min(256, AI_API_EMBEDDING_MAX_BATCH_ITEMS) documents per hourly pass and retains its 10-second provider deadline.

For bounded live BGE-M3 measurements, set CLOUDFLARE_ACCOUNT_ID and CLOUDFLARE_API_TOKEN in the environment and run:

mkdir -p tmp/embedding-measurements
DENSE_MEM_CLOUDFLARE_BATCH_MEASUREMENT=1 \
  DENSE_MEM_CLOUDFLARE_MEASUREMENT_OUTPUT=tmp/embedding-measurements/batches.jsonl \
  go test ./internal/embedding -run '^TestCloudflareBatchMeasurement$' -count=1 -v

Use a new output filename for each run. The harness measures five repetitions each of 100, 200, and 256 documents, then compares failed-chunk and prior whole-batch retries with injected later-chunk 429/503 responses. Each batch has a 60-second measurement bound; the output records whether it exceeds the reconciliation deadline, HTTP attempts, resent documents, and reported input tokens. Neurons are calculated from reported tokens using the published BGE-M3 rate of 1075 per million input tokens; they are not measured billing usage. Missing provider usage remains unavailable.

AI_REMEMBER_MODEL, AI_CONFLICT_REVIEW_MODEL, AI_DREAM_GRAPH_MODEL, AI_DREAM_EVIDENCE_MODEL, and AI_COMMUNITY_SUMMARY_MODEL are optional overrides for their existing AI sessions. An unset or whitespace-only override uses AI_VERIFIER_MODEL; a configured model failure is returned without trying the fallback model.

Verifier and assessor calls send temperature: 0 by default. Set AI_VERIFIER_DISABLE_TEMPERATURE=true to omit the field for providers or models that reject temperature.

Fully Local Setup (Ollama)

Any OpenAI-compatible endpoint can provide embeddings and verification. With Ollama running on the Docker host:

ollama pull nomic-embed-text
ollama pull llama3.1:8b

AI_API_URL=http://host.docker.internal:11434/v1
AI_API_KEY=ollama
AI_API_EMBEDDING_MODEL=nomic-embed-text
AI_API_EMBEDDING_DIMENSIONS=768
AI_VERIFIER_MODEL=llama3.1:8b
AI_VERIFIER_TIMEOUT_SECONDS=300

Use host.docker.internal, not 127.0.0.1, because the server calls the provider from the compose network. AI_API_KEY must remain non-empty because startup validation requires a complete provider configuration.

• Set AI_VERIFIER_MODEL to a model that exists on the selected chat endpoint. Startup validates the model configuration before the service accepts memory writes. A 7B-8B class model works for local smoke tests; larger models can exceed the default 60-second timeout while they load. Remember runs synchronously within the request deadline; use the idempotency key to retry after a client timeout or transient provider failure.

Evidence Lifecycle

remember durably stages exact evidence, completes provider validation and the semantic commit, then returns a terminal result. The response includes the owner-scoped processing and search state; it does not expose provider output or internal run IDs.

Callers submit logical Entity, predicate, and Value proposals rather than text offsets. remember does not accept span, surface, or relationship supports fields. Each optional Relationship lists the zero-based evidence_indices that support it; proposals may be omitted, and their indices do not need to cover every evidence item. The single assessor session reviews every evidence item for security and grounds or normalizes only submitted Relationship proposals against their cited evidence. It never searches memory or discovers Relationships. Closed-schema validation and deterministic server policy decide what is safe to commit.

Each Relationship may also list up to 20 explicit known_evidence_ids. These UUIDs are resolved only from evidence visible to the caller and remain read-only assessor context; they do not receive security results. A stored Relationship must still cite at least one submitted evidence_indices span. Entity grounding accepts current canonical or alias names, and a pronoun only when the assessor is given a server-issued anchor for an earlier exact name span. Inaccessible or stale known evidence leaves the Relationship unsupported without revealing whether an ID exists. The aggregate known evidence content in one request is bounded to 20,000 Unicode code points before assessor boundary expansion; larger requests return input_budget_exceeded. The current public contract is dense-mem.v2.6.3; dense-mem.v2.6.2 remains accepted for compatible replays.

To replace a specific current evidence item you own, put its UUID in the new item's supersedes_evidence_ids. Direct targeting is separate from advancing a source revision with previous_source_revision; do not combine them.

{
  "evidence": [
    {
      "content": "Dense-Mem now uses PostgreSQL as its only deployment target.",
      "source_type": "manual",
      "supersedes_evidence_ids": ["<owned-current-evidence-uuid>"]
    }
  ],
  "relationships": [
    {
      "ref": "deployment-target",
      "subject": {
        "name": "Dense-Mem",
        "entity_kind": "project"
      },
      "predicate": {
        "proposed_key": "uses_as_only_deployment_target"
      },
      "object": {
        "entity": {
          "name": "PostgreSQL",
          "entity_kind": "product"
        }
      },
      "polarity": "+",
      "evidence_indices": [0]
    }
  ],
  "idempotency_key": "deployment-target-correction-batch-20260729"
}

Direct supersession is staged with the complete batch. The target is retired only inside the accepted semantic transaction; a failed submission leaves it active. This prevents a replacement that never becomes supported memory from invalidating current evidence.

Remember uses one assessor conversation for the complete batch. Every evidence item receives a security result, including evidence-only submissions. Unsafe evidence fails the complete batch with submission_policy_rejected and no semantic, search, or embedding writes. Safe evidence is stored and indexed even when no Relationship is proposed or accepted. The assessor may ground and normalize only submitted Relationship proposals against their cited evidence; it does not search memory, find support for evidence, or discover Relationships. Every submitted Relationship ref gets a stored or not_stored disposition; unsupported proposals are completed-result warnings. Exact client-owned changes after staging are reported as stale_input. Provider, configuration, database, and internal faults are typed operational failures. All accepted semantic effects commit atomically, with no partial replacement or interactive placement review.

Remember requires one top-level idempotency_key; evidence-level and derived keys are not accepted. If a complete batch needs correction, submit the entire batch again with a new key.

To retract evidence without a replacement, call retract_evidence with owned current IDs, a bounded reason, and an idempotency key:

{
  "evidence_ids": ["<owned-current-evidence-uuid>"],
  "reason": "The source was withdrawn.",
  "idempotency_key": "withdrawn-source-20260729"
}

Both operations append lifecycle events. They never physically delete evidence or trace lineage. Current recall excludes retired evidence, while a historical known_at view before the event can still show what the system knew then.

correct_relationship replaces a specific active Relationship owned by the caller's permanent owner alias. It does not rewrite or delete the original record. The caller supplies the current Relationship version, its exact effective evidence spans, a bounded reason, and only the endpoints or predicate that need correction:

{
  "action": "submit",
  "relationship_id": "<owned-active-relationship-uuid>",
  "expected_version": 1,
  "patch": {
    "object_entity": {
      "entity_id": "<correct-same-team-entity-uuid>"
    }
  },
  "supports": [
    { "evidence_id": "<supporting-evidence-uuid>", "start": 0, "end": 38 }
  ],
  "reason": "The object was resolved to the wrong Entity.",
  "idempotency_key": "relationship-correction-20260808"
}

On acceptance, Dense-Mem atomically supersedes the original Relationship, creates or reuses the active successor, copies the effective support lineage, and appends the correction event and corrects cross-reference. A different owner in the same team may read team-visible memory but cannot correct the author's Relationship. Ambiguous Entity names require one owner confirmation; the original remains active until that confirmation succeeds.

Recall and Graph State

recall_memory is evidence-first but support-path gated. Its results[] contain evidence contexts only after final hydration proves an active, query-relevant Relationship support path remains eligible for the requested valid_at and known_at view. Related Relationships, communities, and Hypotheses are separate bounded fields; candidates and Hypotheses are not default memory results.

remember evidence (+ optional Entity/Relationship proposals)
        |
        v
durable staging -> validated terminal commit -> active eligible Relationships
        |                                      |
        +-- lifecycle event -------------------+
                                               |
                                               v
                         support-gated evidence recall and trace lineage

MCP Tool Catalog

Discover the current closed-schema catalog with MCP tools/list; callers do not select a contract version. The server applies the same scope, feature, and visibility checks to tools/call.

The private control portal exposes bounded Dream investigation reads at /control/api/teams/:teamId/dreaming/runs/:runId/diagnostics and /control/api/teams/:teamId/dreams/:dreamId/diagnostics. They explain run, proposal, feedback, and confirmation outcomes without entering the memory graph. Retained provider projections expire after seven days and report unavailable or expired captures explicitly.

The production release binary is compiled without the evaluation build tag, so no environment variable or control-panel setting can register evaluation tools in a live release. The evaluation target adds only the three harness tools above. eval_get_manifest, eval_get_knowledge_item, eval_list_recall_feedback_events, eval_get_recall_feedback_event, and eval_score_retrieval_case are removed because the current harness does not use them.

When recall feedback is enabled and the feedback snapshot is stored, recall_memory.suggested_actions points to submit_recall_session_feedback with the matching recall ID. When effective team Dreaming is enabled and recall returns Hypotheses, it also points to resolve_dream_feedback: confirm true or false only with independent evidence, and leave uncertain Hypotheses unresolved.

For local evaluation, the committed compose example builds the evaluation target and loads the ignored repository-root .env by default:

docker compose -p densemem_eval \
  -f examples/docker-compose.evaluation.yml up -d --build

go run ./cmd/eval-seedgen \
  --preset local_eval_100 \
  --out tests/eval/seeds/local_eval_100 \
  --suite tests/eval/suites/local_eval_100.jsonl

The local_eval_100 CLI preset emits the versioned local_eval_100_v2 seed identity with 100 corpus rows and 25 scored cases. It is a smoke check for the evaluation image and harness plumbing, not a replacement for the approved deterministic 1k release gate. Use IMPORT_CONCURRENCY=5 for this smoke; the full evaluation remains configurable up to the harness limit of 10.

Memory-pack export emits the current dense-mem.memory-pack.v2.4 artifact. Import and candidate-discovery workflows are not part of the public contract.

Supported HTTP Surfaces

There is no supported public REST memory API. Do not automate browser routes or depend on retired /api/v1 paths.

Telemetry Overlay

Prometheus telemetry is optional and off by default. To collect HTTP, embedding, verifier, assessor, recall feedback, Remember, conflict-review, cost, Dream, and Relationship lifecycle telemetry, start the base stack with the telemetry overlay:

export TELEMETRY_SCRAPE_TOKEN="$(openssl rand -hex 32)"
export GRAFANA_ADMIN_PASSWORD="$(openssl rand -hex 32)"
docker compose \
  -f examples/docker-compose.base.yml \
  -f examples/docker-compose.telemetry.yml \
  -f examples/docker-compose.grafana.yml \
  up -d

The telemetry overlay starts Prometheus at 127.0.0.1:9090, and the optional Grafana overlay starts Grafana at 127.0.0.1:3000. It provisions the Prometheus datasource and dashboards from examples/grafana/; use the Grafana time picker for chart ranges and the Rolling totals selector for counters and canonical ledger windows. Select the Prometheus datasource and bounded scrape job in Grafana; dashboard JSON contains no credentials or installation-specific IDs.

Existing scoped telemetry metrics retain team and profile labels for the first-party API. Grafana aggregates them into system-wide values without selecting or grouping by those labels. New operational metrics add no identity labels, and metric labels contain no evidence or request text. Missing provider usage or pricing stays no-data and is called out separately from a real zero. Ledger collector failure suppresses its gauges and exposes collector health. Counter measures aggregate across instances and preserve the telemetry service's sparse first-sample behavior; range rates use Grafana's query step. Durable ledger gauges use max across replicas and never use rate or increase. The operator dashboards are system-wide because canonical lifecycle measures carry no team or credential dimensions.

The control Metrics and user Usage dashboards were removed after real Grafana and Prometheus parity validation. Team-overview request summaries, private diagnostics, settings, and the control telemetry reader used by conflict-queue health remain. See examples/grafana/README.md for provisioning and the series map.

Grafana series map

internal/operations/telemetry_catalog.go defines the retained control telemetry series and their availability rules. Grafana panel descriptions map to these identifiers. The metric names and labels remain available to operators.

Additional operational families are densemem_mcp_transport_requests_total, densemem_mcp_transport_duration_seconds, densemem_mcp_tool_results_total, densemem_logical_operation_attempts_total, densemem_logical_operation_duration_seconds, densemem_logical_operation_recoveries_total, densemem_remember_phase_duration_seconds, densemem_dream_cycle_attempts_total, densemem_dream_cycle_duration_seconds, densemem_dream_provider_attempts_total, densemem_dream_provider_duration_seconds, densemem_dream_feedback_actions_total, densemem_recall_hypothesis_expansions_total, densemem_recall_hypotheses_returned_total, densemem_operation_provider_tokens_total, and densemem_operation_provider_usage_unpriced_total. They distinguish MCP HTTP status from logical tool outcome, Remember execution/replay/conflict/recovery, bounded phase and provider usage, Dream run/provider outcomes, Recall hypothesis expansion, and canonical ledger state. New families use closed labels without team, profile, request, model, or content values. Their histogram buckets include the 180-second Remember budget and larger overruns. The canonical ledger collector reports densemem_operational_ledger_collection_success; a zero collection status means its other gauge families are omitted for that scrape. It uses a two-second read-only collection deadline.

To compare baseline and candidate overhead, run compare_telemetry_load.py against isolated deployments with matching data and configuration. It warms both servers, sends paired authenticated tools/list requests, and scrapes both metrics endpoints during the measured run. It fails when candidate p95 latency or throughput regresses by more than 10 percent. Set a shared DENSE_MEM_LOAD_TOKEN or the per-deployment DENSE_MEM_BASELINE_LOAD_TOKEN and DENSE_MEM_CANDIDATE_LOAD_TOKEN; set DENSE_MEM_LOAD_SCRAPE_TOKEN for both metrics endpoints. Use the optional DENSE_MEM_BASELINE_SCRAPE_TOKEN and DENSE_MEM_CANDIDATE_SCRAPE_TOKEN when the two scrape endpoints use different credentials. The JSON receipt is written under the ignored evaluation runtime directory.

Responsibility Boundary

Data Egress and Consistency

Dense-Mem can send evidence text, proposal context, and recall queries to the configured embedding and verifier providers. Self-hosted providers keep that traffic within your boundary; hosted providers do not. Embeddings are derived, versioned state and cannot overwrite newer sources. Startup checks prevent mixing incompatible embedding models or dimensions.

Remote development tests

Commit and push a worktree branch, then open Actions → Development E2E → Run workflow. Select main for the workflow and enter the branch or full commit SHA in source_ref. This route requires repository-admin permission and works before a pull request exists.

Enter registered scenario names in scenarios, separated by commas, or all for the entire scenario registry. Leave it blank for PostgreSQL-only testing. For postgres_shards, enter 0, 1, 2, a comma-separated subset, or all; leave it blank for E2E-only testing. Select at least one group. The scenario named full runs its existing dreaming/telemetry/portal coverage; all runs every registered scenario. Available names live in scripts/e2e-scenarios.json.

gh workflow run development-e2e.yml --ref main \
  -f source_ref=issue/my-change \
  -f scenarios=conflict,mcp_oauth \
  -f postgres_shards=all

The workflow resolves one commit, builds one native production image in the dedicated dense-mem-e2e GHCR package, and gives every selected job the same image digest and source SHA. E2E and PostgreSQL groups run independently, with four concurrent scenarios and three PostgreSQL shards per run. The Actions summary records the exact SHA, digest, selections, and results. Inspect the matching Clean development E2E images run as well: it removes the owned image after completion, including failures and cancellations. An hourly sweep recovers missed cleanup and preserves active runs or unexpected image ownership.

Start a fresh dispatch for retries. Each privileged job rechecks the acting accounts and rejects job reruns, which can reuse cached authorization or an image that has already been deleted. Successful remote results can satisfy the corresponding focused E2E and real PostgreSQL checks for their exact commit. Keep lightweight local checks and attach the run URL and summary to the PR. Every PR still requires its current-head full production-image E2E gate through the ordinary validation route.

New manual workflows become dispatchable after their workflow files reach the default branch. Live rollout must verify E2E-only, PostgreSQL-only, combined, failed, and cancelled runs together with GHCR cleanup.

Fork pull-request validation

Public fork PRs use contributor-owned Cloudflare credentials for production E2E. The upstream repository never forwards its Cloudflare token to a fork.

1. In your public fork, enable Actions and configure the repository variable CLOUDFLARE_ACCOUNT_ID and secret CLOUDFLARE_API_TOKEN. The token needs Workers AI access. Keep the token out of workflow inputs and PR comments.

2. Ask a repository administrator to apply deploy-test-image to the upstream PR. After the preview finishes, its comment contains the approved source SHA, image digest, preview run ID/attempt, and trusted upstream revision.

3. In your fork's Actions page, manually run Contributor fork E2E request on the branch at that exact PR head. Enter the upstream PR number, source_sha, preview_run_id, preview_run_attempt, and trusted_revision from the current preview receipt. The workflow must already be present on your fork's default branch for GitHub to offer manual dispatch.

4. Keep the PR head unchanged while all 24 scenarios, three PostgreSQL prechecks, rootless-controller checks, and cleanup finish. The upstream verifier waits up to 120 minutes and automatically verifies the signed receipt before reporting Production image E2E success.

The trusted upstream workflow signs in a separate job that runs no candidate code and receives no Cloudflare secret. Verification binds the signer revision, PR head, image digest, both run attempts, and complete results. Missing credentials, failed/skipped jobs, forged signatures, stale approvals, and timeouts fail validation. If the head or approved upstream workflow changes, request a fresh administrator-approved preview and start its matching run. Same-repository PRs retain the ordinary production E2E route.

Issue #508's live public-fork acceptance run is deferred by maintainer direction. Signed-fixture, receipt-policy, and workflow tests cover the bridge locally and in CI; they do not establish that live fork acceptance has completed.

Documentation

License

Apache-2.0