Replay the Pipeline, Not the Function

Here's the question every data-transforming change eventually forces: how do you know it does what you think it does, before it touches real orders? Our answer is replay testing production data: take the change, run it against a real, stored slice of history, and look at what comes out the other end. Not a mock. Not a synthetic fixture built to flatter the change. The actual data your system has already seen.

We take a side on this early, because the alternative gets sold as equivalent and it isn't. Unit tests confirm a function does what you told it to do. Replay confirms the pipeline does what you need it to do, on data you didn't write to be convenient. For any change that transforms data on its way through a system, live order data, extracted fields, customer records, that difference is the whole game.

But replay done halfway is worse than no replay at all, because it reports back with total confidence and no dissenting opinion. We've now watched two separate ways a replay can lie to you with a straight face. One comes from replaying the wrong scope: the function, not the pipeline it lives in. The other comes from replaying the wrong sample: only the cases the change should touch, never the ones it shouldn't. Both mistakes produced a clean, convincing, wrong answer. This piece is about both, plus the mechanism that turned one of them from a slow bleed into a same-day fix.

Why Replay Beats Unit Tests for Data-Transforming Changes

A unit test asks: given this input, does the function return the expected output? That's a fine question for a function that lives alone. It's an incomplete question for a function that lives inside a pipeline with other stages before and after it, stages that read state, apply rules, and sometimes reverse what the function just decided.

Data-transforming changes rarely live alone. Extraction logic feeds a rules engine. A rules engine feeds an approval gate. An approval gate feeds a release step. Change any one link, and a unit test on that link will pass every time, truthfully, while telling you nothing about what happens three links downstream when a later gate quietly overrides your change on some fraction of cases.

Replay closes that gap because it runs the whole chain, on data drawn from what the operation actually produced, and reports what actually came out. It's slower to build than a unit test suite. It's also the only method that gives you an honest answer to "what does this change do to real orders," instead of "what does this function do to inputs I chose."

We've written elsewhere about why the deployment posture that surrounds a change matters as much as the change itself, see Parallel Operation Deployment: Zero-Disruption Automation and Risk Mitigation Through Modular Software Implementation. Replay is the testing half of that same instinct: don't cut over on faith, verify on real data first, in increments you can inspect.

Lesson One: Pipeline, Not Function

We built an impact estimate for a decision function we'd changed. The method was direct and looked defensible: call the changed function on a batch of real historical cases, compare its new decisions to its old ones, and count how many outcomes flipped. The number came back big. Bigger than we expected, and bigger, it turned out, than reality.

The function did not operate alone. Downstream of it sat a gate, later in the pipeline, that reversed some of those outcomes under its own logic regardless of what the earlier function had decided. Calling the function directly and measuring its output as if that output were final skipped the gate entirely. The estimate was measuring the function's opinion, not the pipeline's behavior. Those are different things whenever something downstream gets a vote, and here something downstream had a vote on a real share of the cases.

The fix wasn't a smarter estimate. It was a different harness: instead of calling the changed function directly, replay the full pipeline, old code path first, and confirm it reproduces the number we'd already observed in production. That reproduction step matters more than it sounds like it should. If your corrected harness can't get back to the number you know already happened, under the unchanged code, you have no basis for trusting what it tells you about the changed code. We ran the old pipeline through the new harness, watched it land on the known figure, and only then trusted the harness enough to run the new function through it and read the real delta. The gap between the naive estimate and the honest one was entirely explained by the gate reversing outcomes the direct-call method never saw.

The lesson generalizes past this one incident. Any time you estimate the impact of a change by calling the changed unit directly, you're implicitly asserting nothing downstream matters. That assertion is sometimes true. It should never be assumed. Reproduce the old, known number first, with the old code, in the same harness you'll use for the new code. If your harness can't do that, it can't be trusted to tell you about the new code either, and you should stop before you ship on the strength of a number the harness invented for you.

Lesson Two: Negative Controls, or the Rule That Ate Approved Orders

The second failure came from a rule built to clean up a known mess, and it passed every test we wrote for it. Every single test case in that suite was a case the rule should catch and correct. None was a case the rule should recognize and leave alone. The suite was internally consistent and completely blind to the failure mode that actually happened: the rule went live, did its job on the cases it was meant for, and then reached into orders customers had just approved and archived them.

Nothing in the test suite would have caught this, because nothing in the test suite represented what a correct, hands-off case looked like. A test suite built entirely from positive cases, cases the logic is supposed to act on, can prove the logic acts. It cannot prove the logic knows when to stop acting. That second proof needs its own category of test: negative controls, cases specifically constructed to represent what the change must never touch, run through the same pipeline, checked for the absence of an effect rather than the presence of one.

We didn't have negative controls in that suite, and the gap cost us: freshly approved orders got archived by a rule that had no way of knowing it wasn't supposed to touch them, because nothing had ever told it what "supposed to touch" excluded. Once we saw it, the response had two parts. First, we hit the kill switch, which we'll come back to. Second, we rebuilt the test suite to include cases the fix is required to leave alone, not just cases it's required to fix, and we wrote the rule into how future replays get judged: a replay must go red if the fix that stops this is ever removed. Not a nice-to-have regression test sitting somewhere in a folder. A required, blocking check.

That last part is the real fix, more than the specific rule change. A replay suite with only positive cases will always look green and always be one refactor away from repeating this, because nothing in it represents the boundary the change isn't supposed to cross. Include the cases that must not change. Watch for the absence of an effect as carefully as you watch for its presence. A replay that only checks what should happen will confirm your fix and hide your regression in the same breath.

The Kill Switch That Made Recovery Same-Day

None of the above matters much if the response to a bad deploy takes days. It didn't, here, because the rule that archived approved orders had a kill switch built in before it needed one. We didn't build the ability to turn it off after we found the problem. It already existed, as a condition of shipping the rule at all.

When the archiving started, the sequence was: switch it off, fix the underlying logic, restore the affected orders, same day. Not a multi-day incident review followed by a patch the following sprint. A same-day close, because turning the thing off didn't require a deploy, a rollback, or a conversation about blast radius. It required flipping a switch that was already there.

That's the actual argument for building kill switches into anything that transforms live data automatically: not because you expect to need one, but because the day you need one, you need it to already exist. A kill switch you build after the incident is a post-mortem action item. A kill switch you build before is the reason the incident stayed small. We've made this same case about human approval points elsewhere, see Human-in-the-Loop Is Not a Safeguard: the gate only protects you if it's positioned to catch the failure and cheap enough to use the moment you see it. A kill switch is the same idea at the infrastructure layer. It's not a safeguard if reaching it costs you the rest of the day.

Restoring the affected orders the same day also depended on something upstream of the incident: knowing exactly which orders the rule had touched. That's a logging question as much as a testing one, and it's worth stating plainly, because it's easy to build a kill switch and forget the second half of the requirement. You need to be able to answer, precisely, "which records did this touch," or the kill switch stops the bleeding without telling you what to stitch back together.

What Should Have Been True From the Start

We'll say the honest part plainly, because the piece is weaker if we don't: both of these should have been caught before either shipped. A pipeline-scope replay would have caught the first gap immediately, because the true number, gate included, was always available to compare against, we just weren't looking at it that way. Negative controls are not an exotic testing technique, they're a basic complement to positive cases, and their absence from the second suite was a real gap in how the suite was built, not a subtle miss. Neither failure required bad luck. Both required an incomplete definition of what "tested" meant.

What would change our answer here: if a downstream gate never has authority to reverse an earlier decision, function-level replay is sufficient, because there's nothing downstream to miss. If a rule is scoped so narrowly it structurally cannot reach anything outside its target set, negative controls add less. Neither condition held in what we described above, and in our experience they rarely hold in a pipeline that's grown past its first version. Assume they don't hold until you've checked, not the other way around.

Checklist: Replay Before You Ship a Data-Transforming Change

• Replay the pipeline, not the function. If anything downstream can override, reverse, or filter the output of what you changed, testing the function alone measures the wrong thing.
• Reproduce the old number first. Before trusting a replay harness on the new code, run the old code through it and confirm it lands on a number you already know is true. If it can't reproduce the known past, it can't be trusted on the unknown future.
• Build negative controls, not just positive cases. For every case your change is supposed to affect, add a case it's supposed to leave alone, and check for absence of effect as rigorously as presence.
• Make the negative control a blocking check. A replay should go red if the protection against a known bad behavior is ever removed, not just at launch, on every future change to that code path.
• Build the kill switch before you need it. Retrofit the ability to turn a rule off after an incident and you've already lost the day. Build it in at ship time.
• Log what a rule touches, not just what it decided. A kill switch tells you when to stop. A touch log tells you what to restore.
• Treat a clean replay as a claim about scope, not a guarantee. A replay that only covers the function, or only covers cases the change should act on, will report success regardless of what's actually happening downstream or at the edges. Ask what it left out before you trust what it left in.

If you're deciding how much of this to build before your next data-transforming change ships, or you want a second set of eyes on a pipeline that's grown past the point where anyone fully holds it in their head, book a discovery call. We'd rather walk through your actual pipeline with you than have you find the gap the way we found ours.