The math behind a deterministic agent memory

I built N-Dimensional Memory (NDM) as an external memory for LLM agents. This post explains the math inside it. The math is counting and nothing fancier: it counts which events mention which things, then uses those counts to find connections, order history, spot unclear references and choose what to show the model. There is no neural network and no learned weights, so the same input always produces the same output.

System view

The language model reads each turn on its own and lists the entities it mentions. From there on the model is not needed until the final answer. Everything in between is arithmetic on those lists.

One tiny conversation

Every step below uses the same four turns between Alice and Bob:

Step 1: one table

Write each entity as a row and each turn (event) as a column, with a 1 wherever the entity appears. This is the incidence matrix B, and it is the only thing the engine computes on.

Read along a row to see where an entity shows up; read down a column to see what one event involves. Two rules keep the table trustworthy. Each event gets its own column, so facts from different turns cannot blur together. Names are normalised before they become rows, so "the vase" and "vase" share a row, while "my vase" said by Alice and "my vase" said by Bob do not.

Step 2: which events connect

Two events collide when they mention the same entity. Counting shared entities for every pair of events is one matrix product:

Speakers are excluded from this count. In a two-person chat they appear in almost every turn, so counting them would connect everything to everything. They are hubs, and hubs carry no signal.

The chain t0 → t1 → t2 is one connected story: a pottery class, the teacher, the teacher praising a vase. Nobody said so in one sentence; the link comes only from shared entities. The bike (t3) connects to nothing. The math says that events connect, never what the connection means. Whether Carla also teaches Alice is not stated, so the engine does not claim it.

Step 3: how strongly two entities are tied

Flip the question: count the events two entities share. It is the same product the other way round:

Not every shared event counts equally. An event that mentions two things says a lot about how they relate; an event that mentions twenty says little about any one pair. So each shared event is weighted down by how crowded it is (Adamic–Adar weighting):

It is a similarity score, not a distance: a higher number means more closely tied in this conversation, nothing more.

Step 4: history in order, with two clocks

An entity's trajectory is its row of B read in time order. Carla's is t1 (1 May, introduced as Bob's teacher), then t2 (8 May, praised Alice's vase). Every event carries two clocks:

The clocks can disagree. If Bob says on 9 May, "Last month I sold my bike", the event gets the newest tick (t3) but its date anchor is 9 May, so "last month" means April. This mattered in practice: an early version stored ticks but not dates and answered 1 of 7 time questions on a real benchmark conversation. With dates added it answered 5 of 5 on a new conversation. Those are small samples, so read them as a design signal, not a measurement.

Step 5: unclear words become a count

Suppose a fifth turn arrives on 10 May: Alice says, "She said it was my best work." The model reading that sentence alone cannot know who "she" is or what "it" is, so it marks both unresolved. The engine counts the candidates in the last three events:

The whole rule is a count: one candidate fills it in and records the evidence; two or more keep the question open until someone says which; zero records it as unknown. A guess never becomes a fact unless the count is unambiguous or a person states the answer.

Step 6: what to show the model

When a question arrives, the engine scores every event and shows the model only the best ones, in date order, within a word budget:

For “When was Alice's vase praised?”, the vase is a non-speaker entity and Alice is a speaker:

Then four fixed steps: keep the top events (t2, then t0); add the turn just before each kept event, because in dialogue an answer often depends on the line before it; sort by date, label who said each line, and cut the lowest scores first if over budget; hand the result to the model.

The model answers “8 May” from three lines instead of the whole conversation.

Step 7: measuring whether it works

Stability. The model may word things differently each time it reads the same conversation, so two runs may build different connections. Compare the sets of colliding event pairs with the Jaccard index:

If run A finds {t0–t1, t1–t2} and run B writes “pottery class” in t0 but “pottery” in t1, the t0–t1 link disappears and run B finds only {t1–t2}. J = 1 ÷ 2 = 0.5. On a real conversation the measured value was 0.22, which is how name drift was discovered and then fixed by normalising names (Step 1).

Accuracy. Benchmark questions (LoCoMo, with answers written by other researchers) are answered three ways, and every answer is read and scored by hand: correct = 1, partly correct = 0.5, wrong = 0.

The two headline ratios are plain division: 24 ÷ 26 ≈ 92% of the accuracy, using 571 ÷ 1,734 ≈ 33% of the words.

What the math does not do

• It does not understand language. The model reads each sentence; the math only counts what the model listed.
• It does not guess. Connections come from shared entities, and unclear words are filled in only when exactly one candidate exists.
• It does not learn or drift. There are no trained weights or embeddings in the engine, so the same conversation always gives the same table, the same connections and the same depiction.
• It does not prove meaning. A connection says two events share something; what it means is left to the model reading the depiction, with every line attributed to whoever said it.

The method on one page