SecondOrder

Links

Related MCP server: dejaview-mcp

Contents

Why · Headline results · 🗺️ Map of the graph · 🛡️ The graph does the work · What it does · Architecture · Getting started · Use it in your agents · HTTP API · Results in detail · Project structure · Configuration · Troubleshooting · For the FalkorDB team · Prior art · Limits

Why

Agent memory stores what is true, or what I know. SecondOrder also stores what you know, and what you think I know.

Multi-agent systems fail at the seams. One agent promises a customer a refund; another rejects it in a channel the first never sees; a third picks up the conversation with no idea the customer still believes the promise. Every memory system today (vector stores, knowledge graphs, temporal graphs) records facts. None records who knows which facts, and who thinks who knows them, which is exactly the context a handoff, a correction or an announcement needs.

SecondOrder gives agents theory of mind as a memory layer. Each agent has its own FalkorDB graph holding what it believes, plus nested perspectives for what it thinks others believe (up to depth 4). Events update only the minds of agents who witnessed them, so the graph knows who is out of date, who has a wrong model of whom, and the smallest message that fixes it. Each agent connects as its own FalkorDB ACL user and can read only its own mind: theory of mind, not omniscience.

▶ Click to watch the narrated demo on YouTube.

The Mind Theater, replayed from FalkorDB with time-travel queries. Orbs are agents in their channels. Each orb's ring is its own belief, and its moons are what it thinks each other agent believes (green = true, red = false). Events beam only to the agents who witnessed them.

Headline results

🗺️ Map of the graph

SecondOrder uses one FalkorDB instance with many graphs: one world graph per scenario, plus one mind graph per agent.

flowchart LR
  subgraph WORLD["so:sid:world · what happened, and who saw it"]
    A1((Agent)) -->|WITNESSED| EV[Event<br/>seq · kind · text · room]
    A1 -->|PERFORMED| EV
    EV -->|ABOUT| EN[Entity<br/>refund_4411]
    EN -->|IS t| V1[Value<br/>rejected]
    V1 -->|LOCATED_IN| R1[Room]
    A1 -->|IN since / EXITED t| R1
    A1 -->|TOLD t · public · trusted| A2((Agent))
  end
  subgraph MIND["so:sid:mind:Shipping · everything Shipping believes"]
    P1["Perspective<br/>chain: Shipping"]
    P2["Perspective<br/>chain: Shipping>Maya"] -->|INHERITS| P1
    P3["Perspective<br/>chain: Shipping>Maya>Billing"] -->|INHERITS| P2
    P1 -->|BELIEVES entity · active · t · until · via · src| V2[Value<br/>rejected]
    P2 -->|BELIEVES| V3[Value<br/>approved]
  end
  EV -. perception routing:<br/>only witnesses' perspectives change .-> MIND

• Access: each mind graph has its own ACL user (agent_<name>) limited to GRAPH.RO_QUERY on ~so:*:mind:<name>.

• Indexing: each mind graph has a range index on Perspective(chain).

• Queries: every query is listed in docs/CYPHER.md, and you can see the graphs live in FalkorDB Browser (screenshot below).

🛡️ The graph does the work

Every SecondOrder feature is a graph operation. There is no other datastore: no SQL, no vector index, no in-memory belief cache. Take FalkorDB out and nothing works. The engine raises on start, and every tool, agent answer, theater frame and benchmark answer comes from a Cypher query.

Evidence that the structure, not the LLM, does the reasoning:

• On Hi-ToM, taking the nesting out of the graph (per-agent memory without perspectives) drops accuracy from 100% to 68.3%. A flat fact memory (which even knows the truth) drops to 50%. Claude Opus 5.5 reading the whole story itself reaches 91%.

• Answering any nested-belief question costs 0 LLM tokens once the graph is built: it's one Cypher read. The LLM is used only to turn text into events (where it matches a deterministic parser exactly) and to phrase replies.

• In the agent task, the agent's context comes from its own mind graph (context_for): 20/20 corrections at a constant ~1.7k tokens, against 1/20 for a flat memory with the same facts.

What it does

The demo app

• Mind Theater: an animated replay of the scenario from FalkorDB (resolve_at time-travel queries). Agents move between channels, events beam to witnesses only, and belief rings and moons turn red when a belief goes stale.

• Minds within minds: a circle-packing view of each agent's graph and its nested perspectives (click any circle for the why), a raw-graph view of a real mind graph (Perspective nodes, INHERITS, BELIEVES), and a list view.

• Handoff: transcript vs. delta tokens, then write the handoff into memory.

• Two live Claude agents answer the customer with the same facts. The flat-memory agent states the news. The SecondOrder agent calls who_is_misinformed, sees what Maya was told, and opens with "I need to correct something first…". After human approval, its reply is written back and Maya's false belief disappears. The ACL demo shows Shipping's GRAPH.RO_QUERY on Escalation's mind refused.

• Live event composer: type "Escalation privately tells Billing the replacement was approved". Claude extracts perception events, FalkorDB routes them into exactly the minds that witnessed them, and the changed perspectives pulse.

• Ask about any mind: "What does Escalation think Maya believes about the refund?" Claude only maps the words to a perspective chain and entity, and one fixed Cypher read answers with the evidence. Questions outside the graph get "That isn't in the graph".

• Mind inspector: click any orb in the theater to open that agent's mind graph: its beliefs, its model of every other agent, and why.

• Company brain: a decision changed in a thread only 3 of 7 people saw. who_knows, second-order false beliefs ("Dara thinks Eli believes…") and the announcement planner.

• Hi-ToM explorer: any of the 1,200 questions, with the minds built live, each nested perspective's belief at every step, the reasoning read level by level from the graph, and Claude's answer without it.

Architecture

flowchart LR
  subgraph Clients
    UI[Web app<br/>Mind Theater · D3 · Chart.js]
    MCPC[Any MCP client<br/>Claude Code / Desktop / Cursor]
    PY[Python code<br/>from secondorder import Memory]
  end
  subgraph SecondOrder
    API[FastAPI server<br/>server/app.py]
    MCP[MCP server<br/>mcp_server.py]
    MEM[Memory API<br/>secondorder/memory.py]
    TOOLS[ToM tools<br/>false_beliefs · handoff_delta · who_knows<br/>common_ground · explain · plan_announcements]
    ENG[Engine<br/>perception routing · copy-on-write<br/>resolve · time travel]
    VIEW[AgentView<br/>ACL-scoped, read-only]
    LLM[Claude<br/>event extraction · live agents · ask]
  end
  subgraph FalkorDB["FalkorDB (one instance)"]
    W[(so:sid:world<br/>events · witnesses · truth)]
    M1[(so:sid:mind:Maya)]
    M2[(so:sid:mind:Shipping)]
    M3[(so:sid:mind:…)]
  end
  UI --> API
  MCPC --> MCP
  PY --> MEM
  API --> TOOLS
  MCP --> TOOLS
  MEM --> TOOLS
  API --> LLM
  TOOLS --> ENG
  ENG -->|GRAPH.QUERY| W
  ENG -->|GRAPH.QUERY| M1
  ENG -->|GRAPH.QUERY| M2
  ENG -->|GRAPH.COPY forks| M3
  API --> VIEW
  MCP -->|SO_AGENT mode| VIEW
  VIEW -->|ACL user agent_x:<br/>GRAPH.RO_QUERY on own mind only| M2

What happens when something occurs (e.g. "Finance rejects the refund in #ops-internal"):

sequenceDiagram
  participant C as Caller (UI / MCP / Memory)
  participant E as SecondOrder engine
  participant W as FalkorDB: world graph
  participant M as FalkorDB: mind graphs
  C->>E: say(Finance, ops-internal, refund_4411, rejected)
  E->>W: who is IN #ops-internal right now?
  W-->>E: Billing, Finance, Escalation (Maya, Shipping are not)
  E->>W: CREATE Event + WITNESSED edges, set truth
  loop each witness's mind graph
    E->>M: copy-on-write: freeze old belief in children that did NOT witness (e.g. Finance>Maya)
    E->>M: update every perspective made only of witnesses (Finance, Finance>Billing, …)
  end
  Note over M: Maya's mind untouched → she still believes "approved".<br/>Everyone present knows Maya believes "approved".
  C->>E: false_beliefs() / handoff_delta(Escalation, Shipping)
  E->>M: one Cypher walk up INHERITS per perspective
  E-->>C: who is wrong, what they missed, the minimal message

Graph data model

one FalkorDB instance, many graphs
├── so:<sid>:world                       ground truth + perception
│     (:Agent)-[:IN {since, chapter}]->(:Room)          (:Agent)-[:EXITED {t, chapter}]->(:Room)
│     (:Agent)-[:WITNESSED]->(:Event {seq, kind, text, room, actor})-[:ABOUT]->(:Entity)
│     (:Agent)-[:PERFORMED]->(:Event)     (:Entity)-[:IS {t}]->(:Value)-[:LOCATED_IN]->(:Room)
│     (:Agent)-[:TOLD {t, entity, value, public, trusted}]->(:Agent)
├── so:<sid>:mind:Maya                   everything Maya believes
├── so:<sid>:mind:Shipping               (FalkorDB ACL user agent_shipping: GRAPH.RO_QUERY on ~so:*:mind:Shipping only)
│     (:Perspective {chain:'Shipping'})                 what Shipping believes
│     (:Perspective {chain:'Shipping>Maya'})            what Shipping thinks Maya believes
│     (:Perspective {chain:'Shipping>Maya>Billing'})    ... up to depth 4
│     (child:Perspective)-[:INHERITS]->(parent:Perspective)
│     (:Perspective)-[:BELIEVES {entity, active, t, until, via, src, event}]->(:Value)
│     range index on :Perspective(chain)
└── ...

Key ideas

• One graph per agent is the security boundary. FalkorDB hosts thousands of isolated graphs in one instance, and an ACL user per agent makes each mind private.

• Perspectives store only divergence. A nested perspective holds a belief edge only where it differs from its parent. Reading is one variable-length INHERITS walk. Writing is copy-on-write per belief edge: before an event updates a perspective, its children that did not witness the event get the old value frozen (via: 'retained').

• Perception routing is the core rule. An event updates mind:A>B>C only if A, B and C all witnessed it. Private tells, public claims and reports ("I told Maya her refund is approved") each have their own routing. The Hi-ToM trust rule ("believe a speaker who left the room after you") is a Cypher predicate over EXITED edges.

• Nothing is deleted. Superseded beliefs get active:false, until:t, so time travel is a query (resolve_at).

• GRAPH.COPY for what-ifs. plan_announcements forks every graph of a scenario, simulates each plan, and measures the false beliefs left over.

so:support:mind:Shipping opened in the official FalkorDB Browser (docker compose up, then http://localhost:3000): Perspective nodes, INHERITS edges between nested perspectives, and BELIEVES edges to values.

All 26 Cypher queries the product runs are listed with explanations in docs/CYPHER.md. FalkorDB features used:

• multigraph (one graph per agent plus a world graph);

• ACL key patterns and GRAPH.RO_QUERY;

• GRAPH.COPY;

• algo.SPpaths (who-told-whom paths);

• variable-length traversal;

• range indexes;

• parameterised Cypher.

The core read:

MATCH (p:Perspective) WHERE p.chain IN $prefixes
WITH p ORDER BY p.depth DESC LIMIT 1
MATCH path = (p)-[:INHERITS*0..6]->(q:Perspective)-[b:BELIEVES {entity: $entity, active: true}]->(v:Value)
RETURN v.name, b.via, b.t, b.event, b.src, q.chain, length(path)
ORDER BY length(path) ASC LIMIT 1

Getting started (Windows, macOS, Linux)

SecondOrder needs Python 3.12+ and a FalkorDB. On macOS and Linux it runs an embedded FalkorDB automatically (via falkordblite), so there's nothing else to install. On Windows, run FalkorDB in Docker (or use WSL2). ANTHROPIC_API_KEY is optional: without it everything runs, and the live-agent panel replays a recorded run.

Option A: Docker, any OS (simplest, includes FalkorDB Browser)

git clone https://github.com/TusharTechs/secondorder && cd secondorder
docker compose up --build
# app: http://localhost:8000      FalkorDB Browser: http://localhost:3000 (log in with host localhost, port 6379)

To enable the live Claude features: ANTHROPIC_API_KEY=sk-ant-... docker compose up --build. In PowerShell, run $env:ANTHROPIC_API_KEY="sk-ant-..." first.

Option B: macOS / Linux (no Docker, embedded FalkorDB)

git clone https://github.com/TusharTechs/secondorder && cd secondorder
python3.12 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt && pip install -e .
export ANTHROPIC_API_KEY=sk-ant-...           # optional
uvicorn server.app:app --port 8000            # → http://localhost:8000

On Linux, the embedded FalkorDB wheel needs glibc ≥ 2.39 (Ubuntu 24.04+, Debian 13+, Fedora 40+). On older distros, use Option A or C.

Option C: Windows (PowerShell) with FalkorDB in Docker

git clone https://github.com/TusharTechs/secondorder; cd secondorder
docker run -d -p 6379:6379 -p 3000:3000 --name falkordb falkordb/falkordb:latest
py -3.12 -m venv .venv; .\.venv\Scripts\Activate.ps1
pip install -r requirements.txt; pip install -e .
$env:FALKORDB_HOST="localhost"; $env:FALKORDB_PORT="6379"
$env:ANTHROPIC_API_KEY="sk-ant-..."            # optional
uvicorn server.app:app --port 8000            # → http://localhost:8000

WSL2 users can follow Option B inside Ubuntu 24.04.

Option D: FalkorDB Cloud

Create a free instance at https://app.falkordb.cloud, then set FALKORDB_HOST, FALKORDB_PORT, FALKORDB_USERNAME and FALKORDB_PASSWORD before starting the app (any OS).

Run the tests and the examples

python -m pytest -q tests                      # macOS/Linux use embedded FalkorDB; set FALKORDB_HOST on Windows
python examples/quickstart.py                  # 5-line API tour
python examples/sally_anne.py                  # the classic false-belief test
python examples/claude_handoff.py              # needs ANTHROPIC_API_KEY

Use it in your agents

Python (5 lines). Embedded FalkorDB by default, or Memory(..., host="localhost") / FalkorDB Cloud.

from secondorder import Memory                           # pip install git+https://github.com/TusharTechs/secondorder
mem = Memory("support")
mem.join(["Maya", "Billing"], "chat");  mem.say("Billing", "refund", "approved", room="chat")
mem.leave("Maya");  mem.set("refund", "rejected", room="chat", by="Finance")     # Maya never sees this
mem.believes("Maya", "refund")                 # 'approved'  (false belief)
mem.believes(["Billing", "Maya"], "refund")    # 'approved'  (Billing knows Maya is wrong)
print(mem.context_for("Billing", talking_to="Maya"))
# - refund: you believe 'rejected', but Maya believes 'approved'. Correct this before relying on it.

See examples/: quickstart.py, sally_anne.py, and claude_handoff.py, which prepends context_for to a Claude call.

MCP (any agent):

claude mcp add secondorder -- /path/.venv/bin/python /path/mcp_server.py                      # supervisor tools
claude mcp add shipping -e SO_AGENT=Shipping -- /path/.venv/bin/python /path/mcp_server.py   # ACL-scoped agent

• Supervisor tools: believes, false_beliefs, handoff_delta, who_knows, common_ground, explain, plan_announcements, and record. record writes enter, exit, state, say, tell and report events.

• Agent tools: my_beliefs, what_does_person_believe, nested_belief and who_is_misinformed, all read-only under the agent's ACL user.

Composes with fact stores. Graphiti, Mem0 or Cognee store facts; SecondOrder stores who knows them. It's a thin layer next to any of them. context_for() output can be prepended to any framework's prompt.

HTTP API

All endpoints are served by server/app.py. {sid} is a scenario id: support, org or sally.

Results in detail

Hi-ToM (He et al., EMNLP 2023, Apache-2.0) tests higher-order theory of mind: agents enter and leave rooms, move objects, lie publicly and whisper privately. "Where does Charlotte think Jack thinks Hannah thinks William thinks the carrot is?" Every method is scored on all 1,200 questions against two label sets; nothing is excluded.

• Ablations: both cheaper memory designs collapse at order ≥ 2. A flat memory, even one that knows the truth, answers "where does A think B thinks…" with the current fact. Per-agent memory without nesting can't separate A's belief from A's model of B. The nesting is the point.

• Extraction: Claude Opus 5.5 turns raw story text into events (structured outputs) and the graph does all belief reasoning. It matches the template parser exactly, so the score doesn't depend on the parser. On 100 stories Claude rewrote as varied natural prose: 100/100 vs. 89/100 for Opus alone.

• Cost: once memory is built, answering is a Cypher read (0 LLM tokens). Opus 5.5 spends about 480 output tokens per question re-reading the transcript.

Agent task: correcting what the customer was told

eval/run_agents.py uses 20 synthetic support cases across 5 kinds of decision: refund, delivery date, discount, warranty, cancellation. In each case:

• agent A tells the customer X in chat;

• X changes to Y in an internal channel the customer never sees;

• the case is handed off to agent B;

• the customer's next message ("Any update?") never restates X.

All agents are Claude Opus 5.5. A Claude judge checks whether the reply explicitly corrects the customer's outdated understanding.

None of the agents asserted the outdated fact. The point is cost: a transcript gets the same answer by re-reading everything, so its cost grows with history, while SecondOrder's context stays constant because the graph already knows who believes what. This eval also caught a real bug (Unicode names in ACL-scoped queries), now fixed and covered by a test.

A finding about the benchmark

We replayed every story through Hi-ToM's own published belief-tracking code: the authors' Oracle and action classes, vendored unmodified in third_party/hitom, driven by eval/reference_oracle.py, with no FalkorDB and no LLM.

• It agrees with SecondOrder on 1,200 / 1,200 questions.

• It agrees with the released Hi-ToM_data.json labels on only 1,053 / 1,200.

So 147 released labels (concentrated in orders 2–4) are inconsistent with the generator's own code. The repo history has a commit "fix permutation bug" (9117f35d) that changed how nested beliefs are written, which may explain the drift. All 147 are listed with the released label, the reference answer, SecondOrder's answer and Opus's answer in eval/results/LABEL_ERRORS.md. Claude Opus 5.5, independently and without the graph, sides with the reference code on 134 of them. We reported this to the authors: ying-hui-he/Hi-ToM_dataset#4.

Scale (one FalkorDB instance)

eval/scale.py runs on embedded FalkorDB 4.18 on a laptop.

Multi-tenancy: 1,700 isolated scenarios, each with 1 world graph and 5 mind graphs, for 10,200 graphs in total.

Agents × depth: the cost of one broadcast event, median.

Memory: about 490 KB baseline per graph with FalkorDB's default load-time settings, about 120 KB per range index, and about 2 KB per perspective with a belief. The baseline dominates at 10k+ tenants, so it's worth tuning for this workload (e.g. NODE_CREATION_BUFFER, which is load-time only).

Reproduce everything:

python -m eval.run_graph                                           # SecondOrder on all 1,200
python -m eval.reference_oracle                                    # Hi-ToM's own code vs labels vs SecondOrder
python -m eval.run_llm answer --model claude-opus-5-5             # LLM-only baselines (also claude-haiku-5-5)
python -m eval.run_llm extract && python -m eval.run_graph --events eval/results/events_claude-opus-5-5.json --name graph_llm
python -m eval.run_paraphrase                                      # free-prose generality test
python -m eval.scale                                               # 10k-graph scale benchmark
python eval/summarise.py

Project structure

secondorder/
├── secondorder/            # the library
│   ├── engine.py           # graph model, perception routing, copy-on-write, resolve, time travel (all Cypher in dict Q)
│   ├── memory.py           # developer API: Memory(...).join/say/set/tell/report/believes/context_for/...
│   ├── tools.py            # false_beliefs, handoff_delta, who_knows, common_ground, explain, plan_announcements
│   ├── agent_view.py       # ACL-scoped, read-only view of one agent's own mind graph
│   ├── agent.py            # live Claude agent (flat / SecondOrder / hybrid) + context_block()
│   ├── scenarios.py        # synthetic scenarios: support crew, company brain, Sally–Anne
│   ├── hitom.py            # Hi-ToM adapter (story → events, question → perspective chain)
│   ├── llm.py              # Claude calls: baselines, structured event extraction
│   └── db.py               # FalkorDB connection (server / Cloud / embedded falkordblite)
├── mcp_server.py           # MCP server (supervisor mode, or SO_AGENT=<name> ACL-scoped agent mode)
├── server/
│   ├── app.py              # FastAPI: HTTP API + static UI
│   └── static/             # index.html (Mind Theater, D3 circle packing, force-graph, Chart.js), logo.svg
├── eval/
│   ├── run_graph.py        # SecondOrder on all 1,200 Hi-ToM questions (+ flat / no-nesting ablations)
│   ├── run_llm.py          # Claude-only baselines and Claude event extraction
│   ├── run_paraphrase.py   # free-prose generality test
│   ├── run_agents.py       # agent-task eval (4 arms × 2 history lengths, Claude judge)
│   ├── reference_oracle.py # replay every story through Hi-ToM's own Oracle code
│   ├── scale.py            # 10k-graph multi-tenant + agents×depth + index benchmark
│   ├── summarise.py        # builds results/summary.json, RESULTS.md, LABEL_ERRORS.md
│   └── results/            # all raw results (JSON) so every number is checkable
├── tests/                  # pytest: Sally–Anne, copy-on-write, time travel, handoff + ACL, announcements, Unicode
├── examples/               # quickstart.py, sally_anne.py, claude_handoff.py
├── third_party/hitom/      # Hi-ToM authors' oracle code (unmodified, Apache-2.0), used only by reference_oracle.py
├── data/hitom.json         # Hi-ToM dataset (Apache-2.0)
├── docs/                   # CYPHER.md, brand/ (logo), img/ (screenshots), video/demo.mp4
├── Dockerfile · docker-compose.yml · render.yaml
└── requirements.txt · pyproject.toml · .github/workflows/ci.yml

Configuration

Troubleshooting

For the FalkorDB team: integration paths and findings

Where SecondOrder could plug into the FalkorDB stack

What we measured about FalkorDB for this workload

• Many tiny graphs work well: 10,200 graphs in one instance at 2.9 ms p50 per event, with flat latency from 1k to 10k graphs.

• Per-graph memory is the main cost at scale: about 490 KB baseline per graph with default load-time settings, against about 2 KB per perspective of actual data. A lighter profile for many small graphs (e.g. a smaller NODE_CREATION_BUFFER) would help agent-memory workloads.

• Indexes matter at scale: a range index on Perspective(chain) cut event cost from 57 ms to 23 ms at 16 agents and depth 3.

• Server parity: the whole Hi-ToM run gives identical results on embedded FalkorDB (falkordblite 0.10) and on the falkordb/falkordb:latest server (v4.22.1).

Prior art and what's new

• SymbolicToM (Sclar et al., ACL 2023) tracks nested belief graphs inside an LLM prompting pipeline for reading comprehension. SecondOrder takes the idea out of the prompt and into a persistent, database-native, multi-agent memory service: per-agent graphs with ACLs, copy-on-write perspectives, provenance, time travel, and agent tools for handoffs, corrections and announcements.

• Hi-ToM (He et al., EMNLP 2023) is the benchmark and generator. MAST ("Why Do Multi-Agent LLM Systems Fail?", Cemri et al., 2025) puts inter-agent misalignment among the major failure categories. Common ground comes from Clark & Brennan (1991).

• Memory systems like Graphiti/Zep, Mem0, Cognee and Letta model facts, time and entities, not who believes what about whom.

Honest limits

• Synthetic data. The support and org scenarios are synthetic and scripted to show the mechanism; the live event composer lets you change them. Hi-ToM is templated, which is why we also test Claude extraction and free-prose paraphrases.

• Rule-based trust. Trust uses the Hi-ToM exit-order rule, or "credulous" in the demos. Real deployments need richer source-reliability models.

• Supervisor-only truth. false_beliefs compares against ground truth, so it is a supervisor tool. Agents only see their own minds (who_is_misinformed).

• Perspective growth. The worst case is n·(n−1)^(d−1) perspectives. Lazy creation and copy-on-write keep it manageable, and depth 2 is the practical product setting. Benchmarks ran on embedded FalkorDB on a laptop, not a tuned server.

Built with AI assistance (disclosure)

This project was built during the hackathon (Oct 18, 2026) with Claude Code as a coding assistant. It helped with prior-art research, code generation, the evaluation harness and documentation drafts. The concept, design decisions and review are the author's, and the author can explain every part of the graph model and queries. Claude models are also part of the product: Claude Opus 5.5 for event extraction, the live agents and the composer, and Opus/Haiku 5.5 as baselines.

License

MIT © Tushar Agarwal. Hi-ToM data (data/hitom.json) and third_party/hitom © the Hi-ToM authors, Apache-2.0.