I built a feature that tested great. Then I deleted it.

The week I almost shipped a number I liked

I build Callosium. It turns a folder of plain text notes on your own computer into one memory that every AI you use can read and write from, so your assistant stops forgetting you between sessions. The files never leave your machine. The entire point of the thing is recall you can trust, which means the numbers behind that recall have to be honest, especially the unflattering ones. This is the story of a feature I built, switched on by default, measured properly, and then deleted from the product. The measuring is the part that matters.

What the feature was, in plain words

When you ask your notes a question, the normal search looks at your question and at each note separately, turns both into a kind of numeric fingerprint, and pulls back the notes whose fingerprints sit closest to yours. It works well and it is fast. But it never actually reads the question and the note side by side.

A reranker does. It is a second-pass scorer: take the handful of notes the first pass found, then read the question and each note together and score how well that note truly answers that question. The theory is sound, and it is why plenty of search systems use one. It should be strongest exactly where plain matching is weakest, on the questions you phrased nothing like the note you actually wrote.

The small test said yes

I keep two benchmarks and I try never to blur them. The first is CallosiumBench v4: 15,000 questions generated automatically from the notes themselves, nine families of question, two languages. It is synthetic, which makes it easier than real life, and it is excellent at catching regressions. The second is the hard human bench: 84 questions written by hand, the way you actually half-remember something, graded by an independent model. It is much closer to daily use and much harder.

I built the reranker for the hard human bench. On a small hand-built set of exactly the awkwardly worded questions it was meant to rescue, it looked like a clear win: single-shot accuracy went from 71.8% to 76.1%, about 4.3 points. So I switched it on by default and felt rather pleased with myself. (That 71.8% carries a caveat I am keeping in: it sits in a band of roughly 72 to 79 percent depending on who grades the answers. Two independent Sonnet graders scored the same answers about seven points apart.)

Then the big test said no

Then I ran the full 15,000-question synthetic benchmark with it and without it. Same engine, same questions, the reranker the only difference.

• Reranker off: 96.4% overall. A typical answer in 49 milliseconds, 77 at the ninetieth percentile, 116 for the slowest one in a hundred.
• Reranker on: 94.0% overall. Typical 51 milliseconds, but 1,732 at the ninetieth percentile and 2,439 for the slowest one in a hundred.

Worse and much slower. Not slower everywhere, which is the sneaky part. Its confidence gate fired across the whole broad band of low-coverage and close-race questions, so roughly one query in eight paid one to two seconds instead of ninety milliseconds. That is the difference between a memory that feels instant and one you sit and wait on.

Where the damage actually landed

The overall average hides the interesting bit.

• Deep recall of a rare phrase fell about 8 points. The second-pass scorer second-guesses a correct literal match, which is the one place you least want cleverness.
• Arabic ambiguity handling fell 31 points. Where the engine should say "there are two notes that fit, which one do you mean", the reranker collapsed that into a single confident, wrong pick.
• Multi-target comparisons fell 2 points.
• Its only wins were a fraction of a point on typos and on Arabic refusals.

Confidently wrong is the failure mode I care about most. This feature bought me a fraction of a point by manufacturing more of it.

The obvious fix that did not exist

My first instinct was not to delete it. It was to narrow the trigger so it only fires where it helps. That turned out to be impossible, and the reason is neat enough that I want it written down.

A "close race" (the second-best note scoring within 90% of the best) sits entirely inside the "ambiguous" band (within 88%), which is precisely where the encoder does its worst damage. The two conditions overlap by construction. I searched 23,100 candidate trigger settings, and not one survived the basic honesty check of being fitted on half the questions and then scored on the other half. Nothing generalized. There was no safe window to aim at.

So I removed it, rather than shipping it turned off

A feature sitting in the product with a default of "off" is still a promise. It is a switch somebody will find, flip, and quietly get worse answers from, and it is code I have to keep alive forever. So it is gone, not hidden. Deleting it also removed a 300MB model download that every single user was carrying around for a feature that made their search worse.

What ships is the reranker-off configuration: 96.4% on the synthetic benchmark, 116 milliseconds for the slowest query in a hundred.

The rule I hold myself to now

You do not ship a feature off a favourable micro-benchmark. If you build a test set for a feature, the feature will pass it. That is not evidence, that is a mirror. The number that counts is the one from the full benchmark you were already running before the feature existed, plus the harder human test you did not design around it.

I am publishing the whole result, wins and losses, so the next person building AI memory does not spend a month re-deriving it. If you use any tool that claims to remember you, ask it what it measured, on what, and what got worse. Better still, run the test yourself. If you are shopping around, there is also a roundup of local-first tools worth reading.

Callosium opens early access on 4 August 2026. The core is free forever and open source. If you want the honest version of AI memory, including the parts that did not work, join the waitlist.

Abdo

If you build a test set for your own feature, the feature will pass it. That is not evidence, that is a mirror.

Frequently asked questions