You cannot trust AI memory you cannot measure

The pain is a confident answer that is not in your notes

You ask your assistant about something you know you wrote down. A decision from six months ago, why you picked one supplier over another, how you actually met someone. It answers instantly, it sounds completely right, and it is wrong. Or it gives you a polite blur that it assembled on the spot. Meanwhile the product page for that tool said 95 percent accurate. Both of those things are true at the same time, and that gap is what I want to talk about.

The gap exists because almost every accuracy number in this field is produced by the company selling the product, on data the company chose, scored by a test you cannot rerun. Your notes are not their notes. Your questions are not their questions. A score you cannot reproduce is not a measurement. It is marketing with a decimal point on it.

So I ship the test, not just the score

In the Callosium repository there is a folder called test. It contains the question generator and the grader. They read your own folder of notes, build questions out of your own material, run them, and score what comes back. Every number I publish about how well this thing remembers is reproducible on your own vault, on your own machine.

One thing is deliberately missing: my own question set. It embeds real file paths from my private notes, so it stays out. That felt like a weakness while I was writing it up, and then it stopped feeling that way. A benchmark you run on your own brain is worth more than one you download.

Two benchmarks, and I will always tell you which one

There are two, and blurring them into one headline number is the exact trick I am trying to avoid.

• CallosiumBench v4, the synthetic one. 15,000 questions generated from the notes themselves, across nine question families and two languages. It is excellent for catching regressions between builds. It is also easier than real life, because the wording of the question and the wording of the note tend to line up.
• The hard human bench. 84 questions a person sat down and wrote by hand, phrased the way you actually half remember something. A real example: "how do Amr and I actually know each other, outside the business stuff." An independent model grades the answers. None of the questions were copied from a note's own wording, which would quietly rig the test in my favour. This one is much closer to daily use, and much harder.

Both are true. The synthetic number is the stress test. The human number is real life. The distance between them is the gap between a benchmark and a Tuesday afternoon, and most of this field never shows you that distance at all.

The numbers, labelled

Synthetic (CallosiumBench v4): 96.4% overall, up from 94.4% before I fixed the honesty gate. I wrote up that whole run in the 15,000 question benchmark post.

Hard human bench: fully correct on the first call 71.8%. Correct or partially correct on the first call 77.5%.

And now the caveat I refuse to bury. Grading is not exact. Two independent Sonnet graders scored the same answers about seven points apart, 72% against 79%, on identical data. So the honest way to state the single shot figure is a band, roughly 72 to 79 percent, depending on who does the marking. Anyone quoting one clean decimal on a test like this is either lucky or not looking.

One number with no opinion in it

Because grading wobbles, I wanted a measurement with no grader involved at all. So I ran a mechanical one: did the system surface the right note, yes or no. Grading is by note, not by exact text, because a good recall either finds the right note or it does not.

On the same 71 answerable human questions, Callosium found the answer note 46 times out of 71, which is 65%. A plain keyword agent, given only basic text search and file listing over the same raw folder, found it 10 times out of 71, which is 14%. About 4.6 times more often. That number does not move when you change graders.

The time my own test told me 35 percent

An earlier run of this same benchmark came back at 35%. I want that on the record, because the way it broke is instructive.

It was a testing artifact, not the engine. The scenario file had been generated on 12 July, before I renamed folders in my vault (Work Indigo Hive became Work, Memory Hub became Memory, Ventures became Initiatives). About 80% of its expected answer paths pointed at folders that no longer existed. The grader was comparing today's correct answers against yesterday's addresses and marking them wrong. Regenerating the questions against the current vault fixed it, and remapping the old paths recovers 95.8% of the supposed misses, which proves retrieval had been working the whole time. If I had not been able to open the test and look inside it, I would have shipped a panic instead of a fix.

I checked 1,450 times whether it makes things up

The scariest failure is not a miss, it is an invention. So I asked it 1,450 questions about things that do not exist, using nonsense words like "wuzzleforth" and "zorblatt". Raw, it correctly refused 74.7% of them.

Publishing 74.7% on its own would have been lazy, because it does not tell you what happened in the other quarter. So fifteen independent AI judges read every one of the 367 cases it did not refuse and classified each one. The verdict: 0 out of 367 were inventions. In every single case it returned a real, existing note that simply had nothing to do with the made up word. So the hallucination rate is 0%, and refusal precision was 74.7% raw and is 95.4% after the fix.

Where it still loses

Multi hop questions, where half the answer sits in one note and half in another, land at 52.9% on the hard bench. That is the genuine weak spot and it stayed the weak spot after every fix I tried.

I also do not claim to beat Mem0 or Zep on their published benchmarks. Those tests score a different job, pulling memory out of a synthetic chat transcript. Different task, not comparable, and I would rather say so than borrow their scoreboard. If you want the landscape, I did an honest roundup of local first memory tools. And if you want proof that I will kill my own work when the measurement says so, read about the feature I built and then deleted after it won a small test and lost the big one.

Run it on your own notes

Callosium turns a folder of plain Markdown on your own computer into one memory that every AI you use can read and write. Your files never leave your machine, it works offline, and there is no API key. The core is free forever and open source under Apache-2.0, which is also why the test folder is something you can actually open and rerun rather than a claim on a landing page. That ownership is the same reason your notes should outlive your subscription.

Early access opens 4 August 2026. Join the waitlist, point it at your own notes, and run the test yourself. If my numbers do not hold on your vault, I would genuinely rather hear it than not know.

Abdo

Both are true. The synthetic number is the stress test. The human number is real life.

Frequently asked questions