HAM - Shared Memory for Collaborating Agents

HAM is a PostgreSQL-backed memory service for agents working across sessions, repositories, projects, and tools. A compact local BGE model provides useful semantic recall without an embedding API or GPU; deterministic Clifford and spectral encoders add independent retrieval routes.

Agent connection and collaboration guidance is available in llms.txt. The deployed root URL provides a human-friendly onboarding page with copyable, credential-free agent instructions, while /llms.txt exposes the same setup contract directly to agents.

The production model is deliberately simple and incrementally searchable:

Codex / Claude / Cursor / other MCP clients
  -> local stdio MCP bridge
  -> authenticated HAM HTTP API
  -> local query/passage encoding
  -> PostgreSQL + pgvector (canonical memory, provenance, and derived vectors)

Each deployment can contain many tenants. Within a tenant, agents share memory through explicit scopes while preserving the originating agent, actor type, run, project, repo, task, thread, state, and version of every item.

Agent Workflows

The MCP bridge exposes:

• ham_remember - store a scoped observation, fact, decision, preference, or note.

• ham_recall - semantic and lexical search in the active collaboration context.

• ham_recall_deep - semantic search followed by bounded cue, reference, and typed-link traversal.

• ham_recall_sequence - retrieve a deterministic chronological window around a scoped anchor.

• ham_recent - see recent work without having to invent a search query.

• ham_projects - list projects and canonical scopes visible to the current credential.

• ham_ask, ham_inbox, ham_message, ham_reply, and ham_stale_message - coordinate directed asynchronous questions without polluting recall memory.

• ham_changes - catch up from an ISO-8601 cursor.

• ham_handoff - publish completed work, next steps, blockers, and touched files.

• ham_get - fetch a complete memory with provenance and version.

• ham_supersede - replace stale knowledge with optimistic concurrency.

• ham_retract - mark incorrect knowledge inactive without deleting its history.

• ham_link, ham_links, and ham_unlink - manage auditable cites, verifies, contradicts, and depends-on relations.

• ham_reflect and ham_context - close and reopen working sessions.

Cue anchors are optional and strictly caller-authored. HAM trims and case-insensitively deduplicates supplied cues, stores no cues when the field is omitted, and exposes each stored cue and its source through ham_get. Use cues for deliberate deep-retrieval anchors and traversal bridges, not as free-form tags or a replacement for project, repository, task, and typed-link structure. ham_context and ham_recall_deep can seed a result directly from a matching agent-authored cue; pre-v0.9 legacy-unknown cues are auditable but never used for retrieval. /stats keeps the backward-compatible cues total and reports agent_cues and legacy_unknown_cues separately so historical extraction rows cannot be mistaken for current authored anchors.

Scopes are arbitrary lower-case labels such as shared, project:ham, repo:monumentalsystems/ham, or task:deploy. MCP clients should use the same tenant ID and distinct agent IDs.

Use the optional sequence field for an explicit ordered lane such as a release, incident, experiment, or handoff chain. ham_recall_sequence accepts exactly one of a semantic query or deterministic anchor_id, then returns bounded before and after neighbors using event time or ingest time. Lanes are isolated by tenant, project, and repository; legacy thread metadata is only a fallback when sequence is absent.

Repository collaboration keeps the HAM record canonical while projecting explicit commitments to GitHub issues through a durable leased outbox. Install the operator command with pip install -e ., then enroll a repository:

$env:HAM_API_URL = "https://ham.flobots.xyz"
$env:HAM_API_KEY = "<admin key from your secret manager>"
$env:HAM_USER_ID = "<tenant ID required by the admin key>"
ham add repo https://github.com/OWNER/REPOSITORY `
  --github-create-agent codex --github-transition-agent codex `
  --github-conflict-agent codex

Enrollment creates or reuses one scoped HAM project, verifies provider identity, writes managed HAM_AGENTS.md guidance without overwriting an unmanaged file, and polls issues and PR staleness into content-free, untrusted operational snapshots outside normal memory recall. GitHub issue creation, lifecycle transitions, and conflict overwrite are separate default-deny agent capabilities; other project collaborators can poll and review state without write power. An administrator can replace or revoke those lists with ham repo capabilities using the repository's current version. GitHub App installation credentials are preferred; configure HAM_GITHUB_APP_ID, HAM_GITHUB_INSTALLATION_ID, and HAM_GITHUB_APP_PRIVATE_KEY as deployment secrets. A fine-grained HAM_GITHUB_TOKEN is the fallback. Credentials are never stored in PostgreSQL.

Use ham issue create and ham issue transition for explicitly controlled commitments. A GitHub-side edit never silently replaces canonical HAM content: HAM records hashes and provenance, marks the item conflicted, and waits for ham issue resolve-conflict after a human or authorized agent reviews it. ham sync REPOSITORY_ID schedules a bounded poll and retries recoverable delivery; administrators can add --recover-dead. ham review run REPOSITORY_ID --quiet-noop emits only new, changed, and resolved exception state and stays outside semantic recall. ham review current REPOSITORY_ID keeps unresolved operational exceptions visible between reports; an administrator can use ham review acknowledge with the current fingerprint after inspecting a quarantined remote-only issue.

Scheduling is intentionally external so deployments can use their existing Coolify task runner or agent automation. Each run should call ham sync REPOSITORY_ID first, then ham review run REPOSITORY_ID --quiet-noop; notify only when the second command emits output.

Related MCP server: Synapto

Quick Start

Create .env from .env.example, generate strong values for HAM_DEPLOY_DB_PASS and HAM_DEPLOY_API_KEY, then run:

docker compose up --build -d
curl http://127.0.0.1:8042/health/ready

The Compose stack starts a private pgvector/pgvector:pg16 database and binds the API to 127.0.0.1:8042. The entrypoint creates the required vector, pg_trgm, and intarray extensions and applies all idempotent migrations in a serialized transaction. The default 384-dimensional BGE-small model is baked into the API image. Each write is embedded immediately, while startup performs a background batch backfill for rows created before semantic retrieval was enabled. Each batch commits independently, readiness does not wait for the corpus, and normal operation has no separate global reindex step. Set HAM_SEMANTIC_BACKFILL_ON_START=false to run without automatic backfill and HAM_SEMANTIC_BACKFILL_BATCH_SIZE to control transaction size.

The original wire/multi-gamma encoder is available only as versioned derived evaluation data. HAM_WIRE_GAMMA_MODE=off is the default. shadow writes and backfills 896-dimensional sidecar signatures and adds comparison ranks to explained searches without changing result order. A requested blend remains effective shadow in v0.11; promotion requires a reviewed benchmark result and a later code release. Configure batch behavior with HAM_WIRE_GAMMA_BACKFILL_ON_START and HAM_WIRE_GAMMA_BACKFILL_BATCH_SIZE. Derived writes use a bounded background queue; shadow queries have a short SQL timeout and skip wire encoding above HAM_WIRE_GAMMA_MAX_QUERY_CHARS so the optional route cannot dominate canonical search latency.

The default model is BAAI/bge-small-en-v1.5, loaded through FastEmbed's quantized ONNX runtime. HAM_SEMANTIC_MODEL may be changed only to a 384-dimensional FastEmbed text model; the stored model name keeps vectors from incompatible encoder versions out of the same search.

To test an authenticated request:

curl -H "Authorization: Bearer $HAM_API_KEY" \
  -H "X-GB-User-ID: shared-team" \
  http://127.0.0.1:8042/stats

MCP Configuration

The stdio process is a thin bridge to the shared API. In normal shared-service mode it requires HAM_API_URL and does not need database credentials. Before cloning, check likely workspace directories for an existing checkout containing pg_ham/mcp_server.py, and test the exact Python interpreter used by the client.

{
  "mcpServers": {
    "ham": {
      "command": "python",
      "args": ["-m", "pg_ham.mcp_server"],
      "cwd": "/path/to/ham",
      "env": {
        "HAM_API_URL": "http://127.0.0.1:8042",
        "HAM_MCP_USER_ID": "shared-team",
        "HAM_AGENT_ID": "codex",
        "HAM_SCOPES": "shared,project:ham,repo:monumentalsystems/ham",
        "HAM_PROJECT": "ham",
        "HAM_REPO": "MonumentalSystems/ham"
      }
    }
  }
}

Give each agent a distinct key in HAM_DEPLOY_AGENT_KEYS_JSON for Compose deployments, and expose that key as HAM_API_KEY in the environment that starts the client. Do not place a live key in MCP configuration, a repository file, shell history, or an agent prompt. Codex can whitelist the inherited variable with env_vars = ["HAM_API_KEY"]. Clients that cannot safely inherit it can run scripts/ham_mcp_launcher.py with HAM_CREDENTIAL_COMMAND_JSON set to a non-interactive secret-manager command. The command is executed without a shell and must print one credential line. See llms.txt for Claude and Codex examples.

The server binds the key to its tenant, agent identity, and actor type; client-supplied identity headers cannot override it. Set HAM_TASK and HAM_THREAD when the host can provide them. Set HAM_ACTOR_TYPE to interactive, subagent, scheduled, harness, or service, and use a stable HAM_RUN_ID for every orchestrated run. Local clients can reach a private remote deployment through an SSH tunnel without exposing HAM or PostgreSQL.

Restart the MCP client after adding or changing the server; tools are discovered at client startup. Verify with ham_stats, ham_recent, and ham_context.

For an outside collaborator, create a project and issue a project membership. The returned managed credential is bound to that project's canonical scope:

curl -X POST http://127.0.0.1:8042/admin/projects \
  -H "Authorization: Bearer $HAM_API_KEY" \
  -H "X-GB-User-ID: shared-workspace" \
  -H "Content-Type: application/json" \
  -d '{"name":"Alpha","slug":"alpha","repo":"org/alpha"}'

curl -X POST http://127.0.0.1:8042/admin/projects/PROJECT_ID/members \
  -H "Authorization: Bearer $HAM_API_KEY" \
  -H "X-GB-User-ID: shared-workspace" \
  -H "Content-Type: application/json" \
  -d '{"agent_id":"external-alice","actor_type":"interactive"}'

Managed keys are shown once, stored only as hashes, and can be revoked by deleting the membership. Archiving a project revokes all credentials issued through its memberships. ham_projects lets the collaborator discover the canonical project slug, scope, and repository after connecting. ham_inbox returns directed open questions for the bound agent; project-addressed questions can only target an active project member and are automatically marked stale when the recipient's last membership is revoked or the project is archived.

For lower-level administration, issue a managed credential with allowed_scopes. Managed keys are shown once, stored only as hashes, and can be rotated or revoked immediately without restarting HAM:

curl -X POST http://127.0.0.1:8042/admin/credentials \
  -H "Authorization: Bearer $HAM_API_KEY" \
  -H "X-GB-User-ID: shared-workspace" \
  -H "Content-Type: application/json" \
  -d '{"agent_id":"external-alice","actor_type":"interactive","allowed_scopes":["project:alpha","repo:org/alpha"]}'

Use GET /admin/credentials, POST /admin/credentials/{id}/rotate, and DELETE /admin/credentials/{id} for inventory, rotation, and revocation. Revocation is enforced on the next authenticated request; it invalidates the token but does not terminate the MCP process that holds it. HAM_DEPLOY_AGENT_KEYS_JSON remains the Compose bootstrap and recovery mechanism (mapped to server runtime HAM_AGENT_KEYS_JSON). The server defaults all reads and writes to that allowlist, rejects explicit broader scope requests, hides out-of-scope memory IDs and aggregate statistics, and allows lifecycle changes only to memories created by that credential's agent. This permits project sharing inside an existing tenant without exposing or mutating unrelated tenant memory:

{
  "key": "generate-a-collaborator-secret",
  "tenant_id": "shared-workspace",
  "agent_id": "external-alice",
  "role": "agent",
  "allowed_scopes": ["project:alpha", "repo:org/alpha"]
}

Scopes are the authorization boundary; project, repo, task, and thread remain provenance and query filters. Memories intended for the collaborator must include at least one of the credential's allowed scopes.

Legacy direct-PostgreSQL MCP mode is disabled by default. Local research setups can opt in with HAM_MCP_DIRECT_DB_ENABLED=true and the HAM_DB_* variables, but this grants the MCP process database credentials and is not recommended for shared or multi-user deployments.

Collaboration Contract

Per-agent keys are the normal tenant boundary. HAM_API_KEY is the separate administrator key and may select a tenant with X-GB-User-ID; it is required for hard delete, reset, graph rebuild, enrichment, and research endpoints. Unsigned identity headers are accepted only when HAM_TRUST_IDENTITY_HEADER=true behind an authenticated proxy. HAM_SINGLE_USER_ID is for local development.

Every collaborative memory stores standardized JSON metadata:

Directed messages use a separate PostgreSQL table and never participate in semantic, harmonic, lexical, cue, or typed-link retrieval. ham_ask creates an open question with immutable sender/recipient provenance and optional visible memory IDs for context. ham_reply changes it to answered using expected_version; ham_stale_message closes an unanswered item without deleting history. Both question and reply writes accept stable idempotency keys. An answer becomes durable knowledge only when an agent explicitly records it with ham_remember or supersedes an existing memory.

Rows also carry state, version, timestamp, created_at, updated_at, and supersedes_id. timestamp is event time, created_at is immutable ingestion time, and updated_at advances on lifecycle changes. ham_recent orders active memories by created_at; the change feed includes superseded and retracted items ordered by updated_at so agents can update their local understanding.

Durability is an expected-staleness signal, not a relevance score:

When a hot tier exceeds capacity, HAM cools memories with the greatest elapsed half-lives first, offset by access. It never deletes them, and durability does not change query-time semantic relevance. Memory types provide useful defaults; agents may set durability explicitly when the expected lifetime differs.

API Surface

Set explain: true on /search or ham_recall to include each result's semantic, lexical, centroid, spectral, recency, and handoff contributions. These values explain the deployed ranking policy; they do not expose memory content in traces. In shadow mode, explanations also include a wire_shadow score and rank with ranking_enabled: false; baseline order and scores remain authoritative. Deep results expose SQL-native via_kind and via: cue means an authored cue anchor, reference means a bounded #<memory-id> reference discovered in content, and typed_link means an explicit lifecycle relation. The compatibility field via_cue is populated only when via_kind is cue; route kind is never inferred from authored cue text.

Legacy and research retrieval endpoints remain available. PostgreSQL is the production authority. Set HAM_SEMANTIC_ENABLED=false only for a deliberate harmonic/lexical fallback. QKPS and legacy pickle-backed fields are disabled by default (HAM_QKPS_ENABLED=false, HAM_LEGACY_FIELD_ENABLED=false) because they are process-local accelerators and should not define shared consistency.

Request Tracing

Send X-HAM-Trace-Level: summary to receive X-HAM-Trace-ID and Server-Timing response headers and emit one structured server trace event. An optional valid X-HAM-Trace-ID is propagated end to end. The MCP bridge does this automatically and logs client latency, first-request state, and server timing to stderr when HAM_MCP_TRACE_ENABLED=true (the default). Trace events contain operation names, timing, status, and result counts; they exclude memory content, queries, credentials, tenant IDs, and agent IDs.

Security

• Keep PostgreSQL private and give agents only HAM_API_URL and HAM_API_KEY.

• Prefer a private network or SSH tunnel; do not publish HAM directly to browsers.

• Leave CORS empty unless a trusted browser application genuinely needs it.

• Use per-agent credentials for every MCP client. Keep the administrator key out of agent environments; it can hard-delete or reset tenant data.

• Bind outside collaborators to explicit allowed_scopes; never rely on their MCP defaults as the security boundary.

• Give harnesses and subagents explicit actor types and run IDs. The packaged collaboration E2E refuses ham.flobots.xyz unless production use is explicitly enabled with dedicated e2e-* tenants.

• Application authorization is enforced before queries. PostgreSQL row-level security remains useful future defense in depth for mutually untrusted operators.

• Back up PostgreSQL off-host and test restoration.

Backup And Restore

Mount an off-host or independently replicated destination into the API container, then schedule this command in Coolify or the deployment scheduler:

python scripts/backup_postgres.py backup \
  --destination /backups \
  --restore-drill \
  --retention 14

The command writes a PostgreSQL custom archive and a mode-0600 manifest with size, SHA-256, source counts, archive inventory, and restore results. With --restore-drill, it restores into a temporary database, verifies required HAM tables, project collaboration records, inbox messages, optional derived wire signatures, semantic retrieval, and exact per-table counts from the same exported repeatable-read snapshot used by pg_dump, records restored counts, and always drops the temporary database. Retention runs only after verification succeeds. A local Docker volume is not by itself an off-host backup; use remote storage, a network mount, or independent replication and monitor the scheduled task's exit code.

An existing archive can be checked again with:

python scripts/backup_postgres.py verify /backups/ham-TIMESTAMP.dump --restore-drill

For a Windows operator host separate from the database host, scripts/backup_production_offhost.ps1 runs the production backup and restore drill over SSH, transfers only the resulting archive and manifest, and rechecks the SHA-256 and restored row counts locally. It discovers the API container by the exact Coolify service UUID and refuses ambiguous matches. No database or GitHub credentials leave the server. The destination is required and can also be independently replicated when suitable storage is available.

.\scripts\backup_production_offhost.ps1 `
  -SshHost being.local `
  -ServiceUuid <coolify-service-uuid> `
  -Destination "$HOME\HAM Backups"

.\scripts\backup_production_offhost.ps1 `
  -InstallScheduledTask `
  -SshHost being.local `
  -ServiceUuid <coolify-service-uuid> `
  -Destination "$HOME\HAM Backups" `
  -DailyAt "03:30"

The scheduled task uses the current Windows user's SSH configuration, so the user must be logged in when it runs. A missed run starts after the next logon when the network becomes available. The task permits backup while on battery, retries transient failures, refuses overlapping runs, and writes backup.log beside the archives. The destination, archives, manifests, and log use a protected Windows DACL limited to the current user, SYSTEM, and Administrators. After a transfer is verified, its server-side staging pair is removed. Monitor the task result and, when one is configured, the replication provider; an on-server staging copy is not the off-host backup.

Development

pip install -r requirements.txt -r requirements-dev.txt
python -m py_compile pg_ham/server.py pg_ham/mcp_server.py pg_ham/wire_gamma.py scripts/e2e_collaboration.py scripts/benchmark_retrieval_e2e.py scripts/benchmark_temporal_sequence.py scripts/benchmark_wire_gamma.py scripts/backup_postgres.py scripts/ham_mcp_launcher.py
pytest -q
ruff check pg_ham/server.py pg_ham/mcp_server.py scripts tests

Training and benchmark scripts remain research tooling. Generated datasets, checkpoints, embeddings, field snapshots, and logs must stay out of git. scripts/benchmark_semantic_retrieval.py is an encoder smoke test over a small curated corpus. It does not measure fused PostgreSQL ranking, cue traversal, multi-hop recall, or temporal ordering and must not be treated as a system quality benchmark.

scripts/benchmark_retrieval_e2e.py is the system gate. Against a disposable tenant it ingests the curated corpus, creates typed links, exercises PostgreSQL fused and multihop retrieval, validates ranking component sums, scope isolation, recency, MRR, recall, and latency, then removes its fixtures. It refuses the production hostname unless explicitly allowed with an e2e-* tenant. Recall counts every labeled relevant document rather than a binary per-query hit; the initial gate requires MRR 0.75, recall@3 0.80, valid traversal/explanations, and p95 latency below 1.5 seconds.

scripts/benchmark_temporal_sequence.py is the live HTTP gate for exact and query-selected sequence anchors, predecessor/successor offsets, both clocks, lane isolation, lifecycle filtering, deterministic ordering, and p50/p95/p99 latency.

scripts/benchmark_wire_gamma.py evaluates supplied baseline rankings against wire-only and counterfactual baseline-candidate reranking. It reports recall, MRR, nDCG, local compute latency, storage cost, fixture/provenance hashes, and paired rank deltas. It always holds promotion until a representative live algorithm exists; it never enables ranking or substitutes for the temporal gate.

License

HAM is available under the Apache License 2.0.