We Tested Our AI Memory on 15,000 Questions. Here Is What Broke.

Why we tested it 15,000 times before you saw it

A demo is designed to succeed. Someone picks the question, knows the answer is sitting right there, and shows you the one path that works. Real life is messier. You ask at 2am, you misspell a name, you switch to Arabic mid-sentence, or you ask about something you never actually saved.

So before we let anyone near our AI memory, we built a benchmark: 15,000 real questions in English and Arabic, plus more than 10,000 trick questions with no answer at all. An AI memory benchmark like this is the only honest way to measure RAG accuracy, the rate at which an assistant actually retrieves the right thing from your notes. A tested AI memory is not a marketing line, it is a number you can check. A demo hides the failures. A benchmark counts them. We wanted the count.

The invisible character that switched off typo correction

Early in testing, misspelled questions scored around 48 percent. That was strange, because we had built typo correction on purpose. It was supposed to shrug off a wrong letter and find the note anyway.

The culprit was a single invisible character. Someone had left a stray backspace inside one line of code (a template literal, if you write code), and it silently switched off the typo-correction step. Nothing looked wrong on screen. Nothing threw an error. The feature was just quietly dead. The benchmark caught it, we removed one invisible character, and typo accuracy jumped to about 92 percent. Without the test, we would have shipped a memory that quietly failed every time you fat-fingered a word.

The one Arabic question mark that refused every Arabic question

Callosium treats English and Arabic as equals, tested to the same bar, using an on-device multilingual model. So one result stopped us cold: for a while, the engine refused every Arabic question.

The cause was one Arabic question mark. A single punctuation character was tripping the parser and making it reject the whole query before it ever searched. One character, and an entire language went dark. We found it, fixed it, and Arabic went straight back to parity with English. If you want the longer story on why most tools quietly fail non-English users, we wrote about an AI that works in Arabic.

Why it says "that's not in your brain" instead of guessing

The scariest failure in any AI memory is not a wrong answer. It is a confident wrong answer. This is AI hallucination: the model invents something plausible because guessing feels more helpful than admitting it does not know. For a second brain, that is poison. You cannot trust a memory that makes things up.

So we threw more than 10,000 trick questions at it, questions whose answers were simply not in the files. It made up zero answers. One hundred percent of the time it said, in effect, "that's not in your brain," instead of guessing. That honesty is the whole point. An assistant that keeps forgetting you is frustrating. An assistant that confidently misremembers you is dangerous.

Up to 20 AI reviewers hunting for bugs before launch

Fifteen thousand questions test what the memory does. They do not test the code running underneath it. So we ran adversarial code reviews with up to 20 AI reviewer agents, each one told to hunt for bugs, edge cases, and quiet failures like the backspace.

Twenty reviewers disagree with each other, and that is the point. One catches a broken Arabic character. Another questions how the AI retrieval ranks results. The ranking itself is deterministic (the same query always returns the same order, with no cloud calls in the loop), which makes every bug reproducible instead of random. Boring and reproducible is exactly what you want from the thing that remembers your life.

What "19 out of 20" actually means for you

After the fixes, the finished engine answers about 19 out of 20 questions (95.6 percent), and a later run reached 98.8 percent on 10,000 questions, at a median of about 28 milliseconds each. Fast, accurate, and honest when it does not know.

For contrast, we ran the most-starred open-source competitor (Gbrain, around 25,000 GitHub stars) the way a normal person installs it, with no paid API keys. It answered roughly 1 to 6 questions out of 100. Callosium answered more than 90 of 100, offline, about 45 times faster. To be fair, with paid cloud AI keys plugged in, competitors score far better. Our claim is specifically about the free, offline, day-one experience most people actually get.

This is why we test instead of trusting a demo, and it is why your memory should be something you actually own. Early access opens 4 August 2026 at callosium.com. The core engine is open source and free, forever.

Frequently asked questions