We Killed Our Own Feature With a Preregistration

If you lock the pass/fail rule before you see the results, you have to live with it, including when it kills something that looked like it worked. That is the reason for locking it. In November 2026 we ran two preregistered experiments on FRIDAY's memory search. The first failed outright. The second cleared its effect bar and then failed a hard identity gate, and afterward we found evidence that failure came from noise in our own measurement. We kept the kill. A badly designed gate gets fixed with a better gate next time. You never fix it by reinterpreting the verdict you already have. This is the record of that preregistration feature kill, what we learned from it, and why we would make the same call again.

What we were trying to fix

FRIDAY is the shared operational memory that our AI agents and engineering sessions read before they act. When an agent is about to touch a system, it searches FRIDAY first for the rules, incidents and decisions that apply. If that search misses, the agent acts without context it should have had.

The search is hybrid. A full-text lexical arm matches words, and a semantic arm matches embeddings. We mined about 3,800 logged searches from 2026-10-15 to 2026-11-08 and looked for misses, meaning searches that should have surfaced a known record and didn't. We found roughly 75 of them. About half were vocabulary misses: the searcher used a different name for the same thing the record described.

That gave us two candidate fixes and a firm decision to test both under preregistration. We would write the gate, hash it, lock it, and only then build.

Experiment 1: synonym expansion

The change was simple. We built a reviewed alias file of 30 groups. When a query word matched a group, the lexical arm searched for every alias in that group, joined with OR.

We preregistered and hash-locked the gate before writing a single alias. It would ship only if held-out vocabulary misses improved, with a paired-bootstrap 95% confidence interval above zero, and with zero regressions on every set.

The result was unambiguous. Zero of 25 vocabulary misses moved into the top five: 0 of 12 in the held-out set and 0 of 13 in the development set. On 97 control queries that already worked, 4 lost their correct record from the top five, and the hit rate dropped from 0.949 to 0.907.

Why widening the search made it worse

The lexical arm requires every query term to match. Expanding one name into all of its aliases didn't help the target record satisfy the other required terms, so the target still missed. The expansion did pull in records that mention many aliases at once. Those records matched the widened term several times over, and they outranked the precise record.

The queries that hurt most were the precise ones, where someone named a system exactly and used to get exactly that system back. Widening the key space diffused the search. Precision comes from constraining the search, and synonym expansion moved it the opposite way.

Why we stopped early

We had planned a second variant in case the first failed. When we looked at it again, it kept the same OR-expansion, so it would have carried the same mechanism into the same failure. We stopped.

Stopping early in the no-ship direction makes no claim. It ships nothing, asserts nothing works, and can't game the preregistration. Stopping early in the ship direction is the dangerous move, because that is where you catch a lucky streak and call it a result. We allow the first kind of early stop and never the second.

Experiment 2: filter relaxation

The second change targeted a different miss. Some searches are filtered by domain or record type, and the record the searcher needs sits just outside the filter. The fix was to run the filtered search as usual, also run it unfiltered, and return up to three clearly labelled records from outside the filter. The served results would stay exactly as they were. The extra records would sit beside them, marked as outside-filter.

Because the served results were not supposed to change, we made that a hard gate. The locked rule had three parts:

• Mean improvement in the share of filter-miss cases found of at least 0.15, across 31 cases.
• A bootstrap 95% confidence interval with its lower bound above zero.
• The served top five identical for every query in every set.

The effect cleared its bar. The change found 5 of the 31 misses, with a mean improvement of 0.161 and an interval of 0.032 to 0.290. Control queries showed no changes.

On one case, though, the fifth served result changed. The rule said identical for every query in every set, and this one wasn't identical. The locked verdict was FAIL, and we killed the change.

The diagnostic that pointed to noise

After the verdict, we ran a diagnostic. It was not a re-grade, and its result could not move the verdict. We ran the unchanged baseline twice more, against itself, with no feature switched on.

The same case flipped in the same fifth slot between two identical baseline runs.

That told us what the gate had actually measured. Query embeddings fetched fresh on each run are not bit-identical. Two records near-tied at the tail of a similarity-ordered list can trade places when the query vector moves by a rounding error. Our identity gate couldn't tell a change caused by the feature from a change caused by the embedding call, so it tripped on the embedding call.

So the change that failed probably did nothing wrong to the served results. We still killed it.

Why the kill stood

The whole value of a locked gate comes from the fact that the people who wrote it can't argue with it later. Suppose we had re-graded after the A/A run. Every future gate would then carry an unwritten clause saying it holds unless we find a good reason afterward. We can always find a good reason afterward. That clause would turn the preregistration back into the judgment call it existed to replace.

There's a second, narrower reason. The A/A diagnostic tells us the flip in the baseline was noise. It doesn't prove the flip in the treatment arm was the same noise. A real effect could produce an identical-looking swap. We designed a gate that couldn't separate the two, and the honest consequence of a gate that can't separate signal from noise is that it can't pass you.

We could be wrong about how strict to be here. If the cost of delaying this change were high, say if agents were acting on stale context in ways that hurt a production system, we would weigh a fast, fresh, properly controlled rerun more heavily. That situation would argue for running the next experiment sooner. It would not argue for overturning this one.

This is the same discipline we apply to the checkers that gate our code. A gate you're willing to overrule after the fact has no measured catch rate, and an unmeasured gate is a guess. We made that argument at length in Who Checks the Checker? Managing Agents Is a Measurement Problem.

Five lessons we wrote down

1. Pin the query embeddings

Compute each case's query embedding once, store it, and feed the same vector to both arms. Then any difference between baseline and treatment comes from the change being tested, because the input is byte-for-byte the same.

2. Run an A/A control before trusting any identity gate

Run the baseline against itself before you run the experiment. If the A/A fails, the run is invalid, not failed. That distinction belongs in the preregistration. Written in advance, it lets a noisy instrument void a run without anyone reinterpreting a result. We should have had this rule in place for experiment 2, and next time we will.

3. Stop early only in the no-ship direction

Stopping toward no-ship costs you a possible improvement and nothing else. Stopping toward ship lets the data choose when the test ends, and that is how a preregistration gets gamed without anyone intending to.

4. Never reuse exposed cases

The 31 filter-miss cases are spent. We've seen how both arms behave on them, so any new gate tuned on them would be tuned to them. The clean rerun uses fresh cases collected after the cutoff, with pinned embeddings and an A/A control, and it's planned for about late December 2026. Nothing gets built until then.

5. A recorded failure beats a shipped feature you can't defend

We now know three things we didn't know in October. Alias expansion against an all-terms lexical arm hurts precise queries. Filter relaxation shows a real-sized effect on found share. And our identity gate was measuring embedding jitter. All of that is written down, with the locked gates and the raw results beside it. If we had shipped experiment 2 on a re-grade, we would own a feature we couldn't defend to anyone who read the preregistration. The same principle sits behind our rule to replay the pipeline, not the function: measure the thing that actually runs, under conditions you fixed in advance.

What this means for your operation

Every operation that adds automation will face this moment. A change will look good, the test you agreed on will say no, and someone will have a reasonable-sounding argument for why the test was wrong. A process that bends at that point can't be trusted at any other point, including the times its tests say yes.

When we build operational software, we deploy it in modules that run in parallel with what you already have. We wrote more about why that matters for systems that learn in Self-Improving Systems: Proof, Not Promises. If you want automation whose ship decisions you can audit line by line, book a discovery call.