Dense Mem

Dense-Mem is a standalone HTTP MCP memory server using Streamable HTTP. It stages exact evidence, derives semantic state through validated server policy, and returns eligible Evidence with provenance and direct topic follow-up. PostgreSQL is the durable authority for knowledge, lifecycle, provenance, search, authorization, and audit; Redis is coordination only. A single-node deployment may use process-local coordination; a multi-instance deployment requires Redis or an equivalent distributed coordination implementation.

The host LLM owns conversation and judgment. Dense-Mem owns durable evidence, owner authorization, lifecycle events, support eligibility, and bounded recall. The external memory automation contract is MCP at /mcp; browser routes are first-party interfaces, not an alternative public automation API.

Dense-Mem is part of the research preprint Governed Enterprise AI Memory Beyond RAG: From Vector Retrieval to Permissioned Knowledge Graphs.

Try the Hosted Demo

Create a temporary isolated team at https://demo-dense-mem.markhuang.ai to test disposable data before self-hosting.

Related MCP server: Strata

Why Dense-Mem

• Evidence is exact, durable, and append-only. A lifecycle action changes its effective state without deleting provenance or trace lineage.

• Exact Evidence, occurrences, and source revisions retain their provenance. Owner-authorized supersession and retraction append lifecycle history.

• Recall selects eligible Evidence directly. Topics organize shared Evidence without creating factual claims or expanding descendant memberships.

Authentication resolves one immutable actor as team + identity + membership + permanent owner alias + optional credential. An SSO browser session uses the selected membership's permanent owner alias and has no direct credential. An API-key request carries a credential whose stable ID is also its permanent owner alias. Team, identity, membership, and credential fields never let a client choose or replace the semantic owner.

60-Second Quickstart

Download the local compose example and environment template, configure the required secrets, and start Dense-Mem:

mkdir dense-mem-local
cd dense-mem-local

curl -fsSLo docker-compose.yml \
  https://raw.githubusercontent.com/markhuangai/dense-mem/main/examples/docker-compose.base.yml
curl -fsSLo .env.example \
  https://raw.githubusercontent.com/markhuangai/dense-mem/main/examples/.env.example

cp .env.example .env
# Fill in POSTGRES_PASSWORD, CONTROL_PORTAL_TOKEN, and AI_API_KEY.
${EDITOR:-vi} .env

docker compose up -d

The base stack uses PostgreSQL with pgvector as the only durable authority. The v2.6.2 release requires the compatible cutover marker created by the stopped-service migration; it has no legacy database runtime or fallback. The local ports are:

MCP:            http://127.0.0.1:8080/mcp
User portal:    http://127.0.0.1:8080/ui
Control portal: http://127.0.0.1:8090/

Open the control portal with CONTROL_PORTAL_TOKEN, create a team from All teams, then open Credentials for its first API key. The sidebar groups team work, operations, and administration; its team picker changes the active workspace. The user portal offers Search memory, Ontology, and the management pages allowed by the session. Connect shows the team ID, MCP endpoint, and memory access. Both portals support System, Light, and Dark appearance, collapsible navigation, and section links that survive refresh and browser history. For control-plane automation, use the same private API:

control_token="<CONTROL_PORTAL_TOKEN from .env>"

curl -fsS -X POST http://127.0.0.1:8090/control/api/teams \
  -H "Authorization: Bearer ${control_token}" \
  -H "Content-Type: application/json" \
  -d '{"name":"primary-memory"}'

curl -fsS -X POST http://127.0.0.1:8090/control/api/teams/<team-id>/credentials \
  -H "Authorization: Bearer ${control_token}" \
  -H "Content-Type: application/json" \
  -d '{"name":"default credential"}'

Operational diagnostics use one process logger for console and the private control portal. LOG_LEVEL accepts trace, debug, info, warn, error, or fatal and defaults to trace; fatal records severity and does not stop the process. PostgreSQL slow-query logging uses POSTGRES_SLOW_QUERY_THRESHOLD_MS, which defaults to 200 and must be a positive value. The operation-log sink is required for readiness and reports gaps and recovery when it is unavailable. Startup and the OAuth compatibility harness use console-only logging until a PostgreSQL operation-log sink is available; they do not claim that pre-attachment events were persisted.

The control portal's Logs view filters by severity, correlation ID, request hash, Remember attempt, execution/replay/conflict classification, and retry eligibility. A team's Remember diagnostics open on admitted calls and link to canonical attempts. Capture bodies are loaded only for an authorized detail view and are labelled as observed preparation, write, disconnect, or unknown receipt; a captured response does not prove delivery to the caller. Existing seven-day capture expiry, legal holds, erasure, and bounded unavailable or truncated states remain in force.

The release image contains one project executable, /app/server. It applies pending PostgreSQL migrations under a database session lock before serving, so the Compose stack does not need a separate migration container. Multiple server replicas that share one writable primary serialize this startup step. Keep ordinary rolling-deployment migrations backward compatible with the previous app version. The v2.6.1 synchronous Remember migration remains the stopped-service boundary; v2.6.2 evidence-first activation runs after it. For the v2.6.1 migration, stop every server instance, take the required PostgreSQL snapshot, and apply the stopped-service boundary before starting the new binary. Independent databases must each be migrated by a server connected to that database. Administration stays on the private control portal/API, while ontology maintenance and automatic conflict review run as server background workers.

Migrations have no overall deadline by default (POSTGRES_MIGRATION_TIMEOUT_SECONDS=0). Set positive seconds, up to 86400, to choose a limit. Process shutdown still cancels migration, and progress is logged every 10 seconds. The server remains unready until migration completes. The image healthcheck has a 30-minute start period and becomes active after its first success; for larger migrations, extend the deployment's startup/healthcheck grace and rollout deadlines so the orchestrator does not restart or abandon the server while it is still migrating.

Release candidates use vX.Y.Z-rc.N and demo-vX.Y.Z-rc.N. Stable releases use vX.Y.Z, latest, and demo-vX.Y.Z; there is no rolling demo tag.

The server requires complete embedding and verifier configuration at startup: AI_API_URL, AI_API_KEY, AI_API_EMBEDDING_MODEL, AI_API_EMBEDDING_DIMENSIONS, and AI_VERIFIER_MODEL. The compose examples provide OpenAI defaults for embeddings; choose the chat models explicitly in .env. AI_API_EMBEDDING_MAX_BATCH_ITEMS limits texts per provider HTTP request. It defaults to 256, matching the application batch limit; set it to 100 for Cloudflare Workers AI BGE-M3. One application batch still returns one ordered result, and a failed request fails the whole batch. Successful chunks stay private within the call; transient failures retry only the failed chunk, sharing three retries across the complete batch. Search reconciliation selects at most min(256, AI_API_EMBEDDING_MAX_BATCH_ITEMS) documents per hourly pass and retains its 10-second provider deadline.

For bounded live BGE-M3 measurements, set CLOUDFLARE_ACCOUNT_ID and CLOUDFLARE_API_TOKEN in the environment and run:

mkdir -p tmp/embedding-measurements
DENSE_MEM_CLOUDFLARE_BATCH_MEASUREMENT=1 \
  DENSE_MEM_CLOUDFLARE_MEASUREMENT_OUTPUT=tmp/embedding-measurements/batches.jsonl \
  go test ./internal/embedding -run '^TestCloudflareBatchMeasurement$' -count=1 -v

Use a new output filename for each run. The harness measures five repetitions each of 100, 200, and 256 documents, then compares failed-chunk and prior whole-batch retries with injected later-chunk 429/503 responses. Each batch has a 60-second measurement bound; the output records whether it exceeds the reconciliation deadline, HTTP attempts, resent documents, and reported input tokens. Neurons are calculated from reported tokens using the published BGE-M3 rate of 1075 per million input tokens; they are not measured billing usage. Missing provider usage remains unavailable.

AI_REMEMBER_MODEL and AI_CONFLICT_REVIEW_MODEL are optional overrides for their existing AI sessions. An unset or whitespace-only override uses AI_VERIFIER_MODEL; a configured model failure is returned without trying the fallback model.

Verifier and assessor calls send temperature: 0 by default. Set AI_VERIFIER_DISABLE_TEMPERATURE=true to omit the field for providers or models that reject temperature.

Optional typed decisions

Configure an independent decision provider to enable evidence deduplication, security review, conflict review, and recall reranking automatically:

AI_DECISION_PROVIDER=openai
AI_DECISION_API_URL=https://api.openai.com/v1
AI_DECISION_API_KEY=<private-credential>
AI_DECISION_MODEL=gpt-6-luna
AI_DECISION_MAX_INPUT_TOKENS=32000
AI_DECISION_TIMEOUT_SECONDS=10
AI_DECISION_MAX_CONCURRENCY=4

The 32k input setting is a conservative deployment limit. OpenAI's Decisions endpoint supports up to 400 questions per request; its endpoint-specific token ceiling is unconfirmed. The general model window is not an endpoint guarantee. For Jev, use AI_DECISION_PROVIDER=typesafe, AI_DECISION_API_URL=https://api.typesafe.ai/v1, and pinned AI_DECISION_MODEL=jev-1.14.0. Set request and question limits separately with AI_DECISION_MAX_INPUT_TOKENS=64000 and AI_DECISION_MAX_QUESTION_INPUT_TOKENS=32000. Jev's explicit AI_DECISION_LONG_CONTEXT_ENABLED=true opt-in raises both allowed ceilings to 262000, with documented latency and accuracy trade-offs. Input counts use the configured verifier tokenizer as estimates; reported usage is recorded separately.

An unset provider or AI_DECISION_PROVIDER=disabled retains existing behavior. Enabled providers require complete independent credentials and limits; decision credentials never inherit from generation. Under Settings → Decisions, save individual task switches in PostgreSQL. Missing preferences default to on after a successful settings read; explicit off survives restarts and provider changes. There is no workload environment variable. Changes reach new operations through the five-second settings refresh; running operations retain one revision. Refresh failures retain the last valid snapshot marked stale, and required work fails if no valid snapshot exists. The private control listener exposes strict GET/PATCH /control/api/config/decisions using the existing configuration item format and { "data": ... } envelope. It never returns provider credentials.

Decisions replace only confident bounded tasks. Deduplication requires exactly one equivalence probability ≥0.99 with every alternative ≤0.01; all ≤0.01 means new evidence. Security passes only when every hazard probability is ≤0.01. Conflict selection requires both position and sufficiently-supported probabilities ≥0.99. Uncertainty retains complete generative review and exact security spans. Generative extraction and other semantic fields remain required. Required decision failures are visible and never silently invoke generation. Recall scores hydrated authorized shortlists of at most 50 Evidence items per space, outside transactions; optional failures retain the original ranking with an explicit degradation. Existing temporal, grouping, and cross-space fusion rules still apply. Reranking adds provider work; no live speed or quality improvement is claimed.

Set LOG_LEVEL=debug for bounded, credential-protected payloads, question IDs, criteria, authorized references, distributions, thresholds, and ranking changes. Info records task actions, uncertainty routing, settings revisions, usage, and durations; warnings identify stale settings, rate limiting, overload, refusals, context rejection, missing usage, and optional degradation. Errors identify required provider, configuration, validation, and persistence failures. Correlation and operation IDs connect task, exchange, routing, and existing commit logs. Logging works with telemetry disabled. Diagnose recorded operations and exchanges; this feature adds no probe endpoints or diagnostic model calls.

See ADR 0015 and the validation record. Live OpenAI comparison is deferred by maintainer direction because credentials are unavailable; Jev live quality and latency remain pending access. HTTP fixtures do not establish model accuracy or performance.

Fully Local Setup (Ollama)

Any OpenAI-compatible endpoint can provide embeddings and verification. With Ollama running on the Docker host:

ollama pull nomic-embed-text
ollama pull llama3.1:8b

AI_API_URL=http://host.docker.internal:11434/v1
AI_API_KEY=ollama
AI_API_EMBEDDING_MODEL=nomic-embed-text
AI_API_EMBEDDING_DIMENSIONS=768
AI_VERIFIER_MODEL=llama3.1:8b
AI_VERIFIER_TIMEOUT_SECONDS=300

Use host.docker.internal, not 127.0.0.1, because the server calls the provider from the compose network. AI_API_KEY must remain non-empty because startup validation requires a complete provider configuration.

• Set AI_VERIFIER_MODEL to a model that exists on the selected chat endpoint. Startup validates the model configuration before the service accepts memory writes. A 7B-8B class model works for local smoke tests; larger models can exceed the default 60-second timeout while they load. Remember runs synchronously within the request deadline; use the idempotency key to retry after a client timeout or transient provider failure.

Private and Team Memory

Every API key has its own private space, including keys issued by an SSO user. An SSO user's browser sessions and OAuth tokens share that user's private space within the selected team. Keys and SSO sessions have separate private memory.

The calling LLM must supply target_space: "private" or target_space: "team_shared" on every remember request. The target applies to the entire submission and participates in its idempotency hash. There is no default or redirection after a denied shared write. Authentication determines the actual space, team, and owner; clients cannot supply a destination ID.

Team-read-only keys can remember, correct, retract, and erase their own private memory and use feature-gated private ingest_session. Shared mutations require current write permission and ownership of the affected records. Both portals show effective memory_access.private.{read,write} and memory_access.team_shared.{read,write}. Supplied memory_binding fields are rejected, including null and empty values.

The public contract is dense-mem.v2.8.0. Refresh tools/list after upgrading. Retired fields fail validation before provider work. Legacy Remember operation keys retain their original identity and outcome and return non-retryable retired_contract guidance.

Upgrade Existing Private History

Server startup automatically upgrades existing private history before starting HTTP listeners or workers. No separate migration command, preflight approval, or maintenance-confirmation flag is required. Use the matching binary and existing environment configuration; keep a verified restorable backup before upgrading. Do not run an older binary across this incompatible schema cutover.

Startup installs the migration boundary, validates immutable writer attribution, captures the source fingerprint, moves private rows in bounded resumable batches, and finishes the schema migration. Concurrent starters serialize the data move; ordinary writes stay gated until completion. An interrupted upgrade resumes on the next startup using its retained fingerprint and progress.

Ambiguous ownership or invalid cross-space references still block startup with an error; automation never guesses an owner. Inspect those blockers with the optional read-only preflight, which starts no server and applies no migrations:

go run ./cmd/server private-memory-preflight

The content-free report includes source_fingerprint, ownership mappings, and blockers. Resolve attribution problems without guessing ownership, then restart the matching server. An optional private-memory-migrate command uses the same automatic migration path without starting the server; --source-fingerprint can constrain it to an inspected snapshot. Historical --maintenance-confirmed arguments remain accepted for compatibility but are not required.

Catalog and constraint changes use maintenance locks; allow disk and WAL capacity for necessary supporting copies and search rebuilding. Verify readiness, isolation, and effective capabilities before restoring traffic.

Migration preserves shared history and splits private records by immutable writer. Necessary support copies retain provenance without adding independent corroboration; old handles resolve only within the authorized destination and generation. Mixed context is sealed, projections are rebuilt, and unfinished session checkpoints are recomputed from the original intake.

Erasing a writer's private memory also purges its migration copies in other private spaces and invalidates dependent memories. Applicable legal holds block physical purge. Unaffected destination records survive with a new generation, which fences stale jobs; erased operation keys cannot restore content.

Before accepting new writes, rollback requires restoring the verified backup and matching application/schema together with all instances stopped. After new writes, use forward recovery. Down does not discard history or restore legacy bindings. Deploying the new server also runs its required data migrations.

Evidence Lifecycle

remember accepts exact Evidence and provenance under dense-mem.v2.8.0. Choose an explicit authenticated destination for the whole batch. Each item is bounded to 999 Unicode code points. Retired graph proposal fields fail validation before provider work.

{
  "target_space": "team_shared",
  "idempotency_key": "deployment-source-20261110",
  "evidence": [{
    "content": "Dense-Mem stores durable memory in PostgreSQL.",
    "source_type": "manual",
    "source": "Deployment decision"
  }]
}

Remember durably stages exact Evidence and processing intent before assessment. Complete security, equivalence, and Evidence-conflict decisions are validated before required embeddings. Evidence, occurrences, source revisions, lifecycle, search documents, and the terminal receipt commit atomically. A failed required provider call or cancellation cannot partially accept the batch. Initial security rejection remains content-free in diagnostics.

Retain the complete request and operation key. Replay the unchanged request to reconcile a timeout; a changed request with the same key conflicts. Operational failures retain typed retry guidance. Stored context never becomes new submitted Evidence merely because assessment or Recall used it.

For replacement, supply supersedes_evidence_ids on the new Evidence item. Alternatively advance a source revision with previous_source_revision; do not combine these mechanisms. Replacement retires its owned current targets only inside successful atomic acceptance. retract_evidence accepts owned IDs, a bounded reason, and an idempotency key. Both append history; they preserve exact content and provenance. Shared reads do not grant mutation authority.

Feature-gated ingest_session accepts exact user-authored events into private memory. session_evidence_excerpts_v2 selects exact current-event excerpts with original Unicode offsets. Prior context is read-only. Empty extraction returns an explicit memory_outcome: "no_memory" terminal result. Event identity, request hashing, replay, cancellation, and generation fences remain enforced.

Recall, topics, Trace, and export

recall_memory retrieves eligible Evidence directly with bounded exact context, equivalence IDs, explicit search states, conflicts, and degradation reporting. known_evidence_ids suppress already known eligible results.

related_topics supplies topic identity, name, description, eligible unseen Evidence count, and rank. topics_truncated reports omitted hints. topic_limit defaults to three and accepts 0–10. Follow up with topic_ids: 1–10 unique UUIDs select the union of current direct memberships without descendant expansion. Evidence lifecycle, time, freshness, ownership, and known exclusions apply before selection and counting. Missing, foreign, and stale topics produce the same bounded topic_unavailable error for explicit filters. Ordinary Recall reports optional topic degradation. Topic metadata remains current during temporal reads.

Ontology organizes only team-shared Evidence through topics, equivalence, and manager overrides. Private memory is excluded. Existing enable, pause, disable, concurrency, and generation fences remain authoritative. Pausing preserves eligible publications; disabling removes organization from Recall. Topic parent links are organizational metadata and do not establish factual support.

trace_memory takes an evidence_id. It returns bounded Evidence decisions, occurrences, source revisions, and lifecycle. include_evidence_content and include_transitions control detail. Defaults are 100 events and 2,000 content runes; caps are 500 and 8,000.

export_memory_pack accepts a name and 1–500 Evidence IDs. Its dense-mem.memory-pack.v3.0 artifact preserves provenance, canonical hashes, content bounds, and explicit omissions. Authorization applies to every target.

Python Core Client

The Python package uses the official MCP SDK 1.x and supports Python 3.11–3.14. It provides async/sync clients, Google ADK ingestion, and a LangChain retriever using the same Evidence-only schemas. See its README for complete requests, terminal handling, and unchanged-request replay.

MCP Tool Catalog

Discover closed schemas with tools/list. The registry applies identical scope, feature, version, and visibility checks to tools/call.

The evaluation build adds eval_list_knowledge_refs and eval_run_recall_case. Production builds cannot register them through configuration. Factual graph, Dream, and community contracts are retired from both profiles.

Before upgrading, stop incompatible instances and verify a matching restorable backup. Normal startup runs the ordered Evidence-only migration before workers or listeners. It preserves Evidence fingerprints and scoped operation identity, retires graph-specific data and mixed payloads, and blocks on legal holds or unexpected dependencies. No provider calls occur during migration. Before commit, errors roll back. After destructive commit, restore the matching backup; after new writes, recover forward unless separately approved. Never restart an old binary over the migrated database.

The production log default remains TRACE. DEBUG includes protected assessment exchanges, exact admitted fields, topic selection reasons, and authorized Trace/export limits and omissions. WARN reports optional topic degradation and retired contracts; INFO records session resume and migration milestones; ERROR records blocked or failed migrations. Capture complete stdout operation windows before teardown and verify 2,048-byte chunks, hashes, and counts. Existing 16 MiB provider-body, 64 MiB session-envelope, and seven-day diagnostic retention bounds remain. Missing, truncated, or interrupted capture is explicit.

Supported HTTP Surfaces

There is no supported public REST memory API. Do not automate browser routes or depend on retired /api/v1 paths.

Telemetry Overlay

Prometheus telemetry is optional and off by default. To collect HTTP, embedding, verifier, assessor, recall feedback, Remember, cost, and Evidence lifecycle telemetry, start the base stack with the telemetry overlay:

export TELEMETRY_SCRAPE_TOKEN="$(openssl rand -hex 32)"
export GRAFANA_ADMIN_PASSWORD="$(openssl rand -hex 32)"
docker compose \
  -f examples/docker-compose.base.yml \
  -f examples/docker-compose.telemetry.yml \
  -f examples/docker-compose.grafana.yml \
  up -d

The telemetry overlay starts Prometheus at 127.0.0.1:9090, and the optional Grafana overlay starts Grafana at 127.0.0.1:3000. It provisions the Prometheus datasource and dashboards from examples/grafana/; use the Grafana time picker for chart ranges and the Rolling totals selector for counters and canonical ledger windows. Select the Prometheus datasource and bounded scrape job in Grafana; dashboard JSON contains no credentials or installation-specific IDs.

Existing scoped telemetry metrics retain team and profile labels for the first-party API. Grafana aggregates them into system-wide values without selecting or grouping by those labels. New operational metrics add no identity labels, and metric labels contain no evidence or request text. Missing provider usage or pricing stays no-data and is called out separately from a real zero. Ledger collector failure suppresses its gauges and exposes collector health. Counter measures aggregate across instances and preserve the telemetry service's sparse first-sample behavior; range rates use Grafana's query step. Durable ledger gauges use max across replicas and never use rate or increase. The operator dashboards are system-wide because canonical lifecycle measures carry no team or credential dimensions.

The control Metrics and user Usage dashboards were removed after real Grafana and Prometheus parity validation. Team-overview request summaries, private diagnostics, settings, and control telemetry remain. See examples/grafana/README.md for provisioning and the series map.

Grafana series map

internal/operations/telemetry_catalog.go defines the retained control telemetry series and their availability rules. Grafana panel descriptions map to these identifiers. The metric names and labels remain available to operators.

Additional operational families are densemem_mcp_transport_requests_total, densemem_mcp_transport_duration_seconds, densemem_mcp_tool_results_total, densemem_logical_operation_attempts_total, densemem_logical_operation_duration_seconds, densemem_logical_operation_recoveries_total, densemem_remember_phase_duration_seconds, densemem_operation_provider_tokens_total, and densemem_operation_provider_usage_unpriced_total. They distinguish MCP HTTP status from logical tool outcome, Remember execution/replay/conflict/recovery, bounded phase and provider usage, and canonical ledger state. New families use closed labels without team, profile, request, model, or content values. Their histogram buckets include the 180-second Remember budget and larger overruns. The canonical ledger collector reports densemem_operational_ledger_collection_success; a zero collection status means its other gauge families are omitted for that scrape. It uses a two-second read-only collection deadline.

To compare baseline and candidate overhead, run compare_telemetry_load.py against isolated deployments with matching data and configuration. It warms both servers, sends paired authenticated tools/list requests, and scrapes both metrics endpoints during the measured run. It fails when candidate p95 latency or throughput regresses by more than 10 percent. Set a shared DENSE_MEM_LOAD_TOKEN or the per-deployment DENSE_MEM_BASELINE_LOAD_TOKEN and DENSE_MEM_CANDIDATE_LOAD_TOKEN; set DENSE_MEM_LOAD_SCRAPE_TOKEN for both metrics endpoints. Use the optional DENSE_MEM_BASELINE_SCRAPE_TOKEN and DENSE_MEM_CANDIDATE_SCRAPE_TOKEN when the two scrape endpoints use different credentials. The JSON receipt is written under the ignored evaluation runtime directory.

Responsibility Boundary

Data Egress and Consistency

Dense-Mem can send evidence text, proposal context, and recall queries to the configured embedding and verifier providers. Self-hosted providers keep that traffic within your boundary; hosted providers do not. Embeddings are derived, versioned state and cannot overwrite newer sources. Startup checks prevent mixing incompatible embedding models or dimensions.

Remote development tests

Commit and push a worktree branch, then open Actions → Development E2E → Run workflow. Select main for the workflow and enter the branch or full commit SHA in source_ref. This route requires repository-admin permission and works before a pull request exists.

Enter registered scenario names in scenarios, separated by commas, or all for the entire scenario registry. Leave it blank for PostgreSQL-only testing. For postgres_shards, enter 0 through 5, a comma-separated subset, or all; leave it blank for E2E-only testing. Select at least one group. The scenario named full runs Evidence lifecycle, telemetry, and portal coverage; all runs every registered scenario. Available names live in scripts/e2e-scenarios.json.

gh workflow run development-e2e.yml --ref main \
  -f source_ref=issue/my-change \
  -f scenarios=conflict,mcp_oauth \
  -f postgres_shards=all

The workflow resolves one commit, builds one native production image in the dedicated dense-mem-e2e GHCR package, and gives every selected job the same image digest and source SHA. E2E and PostgreSQL groups run independently, with eight concurrent scenarios and six PostgreSQL shards per run. The backlog regression has a dedicated shard; shard selections cover every registered case exactly once. Execution jobs are capped at 30 minutes, with bounded test time before cleanup. Production validation starts scenarios after the rootless controller check while PostgreSQL shards continue independently, and its final gate requires every group to pass. The Actions summary records the exact SHA, digest, selections, and results. Inspect the matching Clean development E2E images run as well: it removes eligible owned images after completion, including failures and cancellations. GitHub prevents deleting a package's final version, so cleanup retains that verified completed image and reports its version, digest, and run identity. An hourly sweep recovers missed cleanup and preserves active runs or unexpected image ownership.

Start a fresh dispatch for retries. Each privileged job rechecks the acting accounts and rejects job reruns, which can reuse cached authorization or an image that has already been deleted. Successful remote results can satisfy the corresponding focused E2E and real PostgreSQL checks for their exact commit. Keep lightweight local checks and attach the run URL and summary to the PR. Every PR still requires its current-head full production-image E2E gate through the ordinary validation route.

New manual workflows become dispatchable after their workflow files reach the default branch. Live rollout must verify E2E-only, PostgreSQL-only, combined, failed, and cancelled runs together with GHCR cleanup.

Shared CI runs repository/browser checks, Go tests and analysis, and core and PostgreSQL coverage producers independently. Its required merge gate verifies that all seven raw Go profiles belong to the same source commit and workflow attempt, then applies the unchanged deduplicated coverage threshold above 90%. The local scripts/coverage-report.sh --complete command still runs both parts and the merge. PR validation and post-merge CI use the same shared workflow.

Repository admins can benchmark changes to CI/E2E workflows before merge using production-image-e2e.yml on their own current PR branch with pr_number. The manual trial requires the trusted preview receipt and runs candidate shared CI alongside every database shard and registered scenario. It does not replace the normal required PR checks. Record the exact source SHA, image digest, workflow attempt, preview-build duration, queue and execution times, and cleanup results; accept the latency change only when all execution jobs finish below 30 minutes and the trial plus preview-build allowance finishes within 60 minutes.

Fork pull-request validation

Public fork PRs use contributor-owned Cloudflare credentials for production E2E. The upstream repository never forwards its Cloudflare token to a fork.

1. In your public fork, enable Actions and configure the repository variable CLOUDFLARE_ACCOUNT_ID and secret CLOUDFLARE_API_TOKEN. The token needs Workers AI access. Keep the token out of workflow inputs and PR comments.

2. Ask a repository administrator to apply deploy-test-image to the upstream PR. After the preview finishes, its comment contains the approved source SHA, image digest, preview run ID/attempt, and trusted upstream revision.

3. In your fork's Actions page, manually run Contributor fork E2E request on the branch at that exact PR head. Enter the upstream PR number, source_sha, preview_run_id, preview_run_attempt, and trusted_revision from the current preview receipt. The workflow must already be present on your fork's default branch for GitHub to offer manual dispatch.

4. Keep the PR head unchanged while all 24 scenarios, three PostgreSQL prechecks, rootless-controller checks, and cleanup finish. The upstream verifier waits up to 120 minutes and automatically verifies the signed receipt before reporting Production image E2E success.

The trusted upstream workflow signs in a separate job that runs no candidate code and receives no Cloudflare secret. Verification binds the signer revision, PR head, image digest, both run attempts, and complete results. Missing credentials, failed/skipped jobs, forged signatures, stale approvals, and timeouts fail validation. If the head or approved upstream workflow changes, request a fresh administrator-approved preview and start its matching run. Same-repository PRs retain the ordinary production E2E route.

Issue #508's live public-fork acceptance run is deferred by maintainer direction. Signed-fixture, receipt-policy, and workflow tests cover the bridge locally and in CI; they do not establish that live fork acceptance has completed.

Documentation

License

Apache-2.0

Evidence-only migration progress

The Evidence-only upgrade emits structured evidence_only_migration_progress events before listeners start. At LOG_LEVEL=info or more verbose (the default is trace), collect one execution from the server console:

docker compose -f examples/docker-compose.base.yml logs --no-color --no-log-prefix server |
  jq -Rc --arg run "$MIGRATION_EXECUTION_ID" \
    'fromjson? | select(.migration_execution_id == $run)'

Each observation identifies the phase, table or work unit, phase start and elapsed time, metric and unit, completed/total/remaining counts, and the last meaningful progress time. The ten-second heartbeat is liveness; it never advances that progress time. A long SQL statement can have a known table and an unknown row-progress count until it finishes. counted is an exact count, affected is PostgreSQL's command row count (not a measurement of scanned rows or changed values), matched counts hold matches, and verified counts retained fingerprints only after the equality check succeeds. Fingerprint-copy command counts use affected rows; they do not claim canonical Evidence was rewritten. Table completion counts table work units, including checked absent tables. No catalog estimate is represented as an exact count. Dependency closure reports discovered records and iterations; its remaining work is unknown until convergence. ETA is unavailable because the remaining phases have no stable cost model.

All phase counts describe attempted transaction-local work. Only evidence_only_migration_completed with transaction_outcome=committed and its verified receipt confirms durable results. A validation or hold failure identifies the bounded reason and failing phase. rolled_back requires PostgreSQL to report the exact migration transaction aborted. In-progress completion is observed for at most five seconds; another starter's commit cannot replace this attempt's outcome. unconfirmed means outcome verification failed or the state could not be reconciled. Do not infer success from the SQL stream finishing. Already-applied or concurrently completed upgrades emit an explicit skipped event with no newly attempted or committed migration work; previous_commit_verified describes their existing receipt.

evidence_only_migration_wait records the current phase, wait type/event and blocker count without query text. The migration session lock can be waiting between PostgreSQL lock probes. A wait event explains a stall; it does not authorize terminating its blockers. Repetitive progress is coalesced to at most once per second; phase/table boundaries and final counts are retained. evidence_only_migration_capture_incomplete and capture_reason identify missing, invalid, oversized or interrupted notices and unavailable wait observations. wait_observation_unavailable=true separately retains wait-probe degradation without replacing an existing notice-capture reason. The first failed wait sample emits a warning with a bounded wait_observation_failure_reason; repeated failed samples do not repeat that warning. The flag remains on subsequent progress and final records, including when wait sampling recovers. Investigate both notice decoding and the classified observation failure when both causes are present. A committed migration can have incomplete diagnostic capture. Preserve the log and investigate the stated capture reason rather than treating missing counts as zero. collection_queries counts additional client round trips; notice-helper SQL executes inside the original batches and is represented by sql_notices and measured elapsed overhead. WARN-level filtering excludes INFO progress; keep INFO or more verbose when collecting migration evidence. These events contain no memory text or row identifiers and use the existing credential-protected root logger.