Customer Change Requests Hide in Approvals & Emails

The change request that costs you money is never in the email that says "I'd like to make a change." It's in the approval click with a comment nobody read carefully. It's in the "where's my order" email that's actually a cancellation wearing a status question's clothes. If you're catching customer change requests by scanning for the word "change," you're catching the easy minority and missing the rest.

We built a keyword filter for exactly this problem, on our own order-to-release automation work for an operations-heavy client. It's the first thing anyone builds. It's also the thing you should plan to replace before you trust the pipeline downstream of it. Here's what we found and why we ended up detecting intent instead of words.

The approve button that carries a change

Most approvals carry no comment. A customer looks at a proof, clicks approve, done. That's the shape you design for, and it's correct most of the time.

But a small, steady fraction of commented approvals aren't approvals. They're something else wearing an approval's clothes. "Approved, but can you make the logo 10% bigger." "Approved, just change the ship-to to the warehouse this time." "Looks good, use the blue instead of navy." The customer clicked the button that means yes, and then wrote a sentence that means not quite.

If your system reads the button and ignores the comment, you ship the wrong logo size, to the wrong address, in the wrong color. The click said approved. The intent said conditional. A system that only listens to clicks will process these as clean approvals every time, and it will do it confidently, because nothing about the transaction looked wrong from the outside.

This is the trap with any interface that forces a binary choice on a decision that isn't binary. Approve or reject doesn't have a slot for "approve, if." Customers improvise one anyway, in the comment field, because that's the only place left to put it. The comment field becomes the actual decision, and the button becomes noise.

We ended up adding a third decision to the pipeline: approved with changes. Not a variant of approval. Not a subtype. A distinct outcome, routed like a rejection, because that's what it functionally is. A rejection with the courtesy of not making the customer start over. More on that below, but the point to sit with first is: if your system only has two outcomes and the input has three, the third one doesn't disappear. It gets misfiled into one of the two you built.

The status email that carries a cancellation

The other place change requests hide is inside the email that looks like a question. "Just checking on the status of my order" is the most common shape of inbound email in any order-driven business, and it is also, sometimes, not what it says it is.

"Just checking on the status of my order, actually go ahead and cancel it, we found a better price elsewhere" is a cancellation. It opens like a status question. If your triage step reads the first clause and routes on it, the customer gets an automated reply telling them their order is on track for Thursday, and the cancellation sits in an inbox unread until someone stumbles on it, usually after the order has shipped.

We watched this happen in a week of inbound customer email during our own build. Real changes, including cancellations, got classified as status questions and routed toward an automated answer. The automated answer isn't wrong on its own terms. It's answering the question that was asked in the first sentence. It's just answering the wrong question, because the email had two intents stacked on top of each other, and the system only read the first one before deciding it knew what it was looking at.

This is worse than the approval problem, not better, because a status question with an automated reply doesn't just fail to catch the change. It actively closes the loop in the wrong direction. The customer gets a confirmation that everything is fine. From their side, the system just told them their cancellation didn't happen, or worse, that it doesn't understand it was even sent. Now you've spent their patience and your credibility on a wrong answer to a question they didn't ask.

Why keyword rules fail

Our first attempt at solving this, like most first attempts, was a keyword filter. Scan inbound email for words like "change," "cancel," "different," "instead," "actually," and route anything that matches to a human. It's cheap, it's explainable, and it's the natural place to start when you don't yet trust anything more expensive.

In a week of real inbound email, the keyword filter caught only a minority of the real change requests in that inbox. Not most. Not even close to most. A minority.

The reason isn't that the filter was built badly. It's that customers don't write in keywords, they write in intent, and intent has more ways to express itself than any word list can enumerate in advance. "Use the other file I sent last week" is a change request with zero trigger words in it. "Can we push this a few days" is a change request about the ship date with no word that any reasonable filter would flag as urgent. "Actually my colleague will handle billing going forward" is a change to the account, phrased as an FYI.

Keyword rules are a special case of the deterministic-code preference we hold everywhere else in this work: write the rule as an if-statement when the input space is small and enumerable. The problem here is that natural-language customer intent isn't small and enumerable. It's the same reason plain RPA button-clicking breaks the moment a screen changes shape; we've written before about why that class of automation runs out of road fast (see Business Logic Chains Explained: Why RPA Is Just Button-Clicking). Keyword matching against free-text customer email is the email equivalent of RPA: it works exactly as long as everyone phrases things the way you guessed they would, and customers don't consult your guesses before they type.

The fix isn't a longer keyword list. We tried that too, and it buys you a slightly larger minority, not a majority. The fix is detecting intent rather than words: understanding what the email or comment is actually asking for, not whether it contains a string you pre-registered as suspicious. That's a different kind of system, and it needs a different kind of validation, which is the part most teams skip.

The third decision

Once you accept that "approved" and "approved with changes" are different outcomes, the architecture question is where the second one goes.

The tempting answer is to bolt it onto the approval path with a flag. Resist that. An approval with a change attached needs the same review, correction and reconfirmation that a rejection needs, because in operational terms it is a rejection: production cannot proceed on the original spec. If you route "approved with changes" down the fast lane reserved for clean approvals, you've built a system that looks like it caught the problem and then ships the original file anyway, because nothing downstream knew to stop and wait for the revised spec.

We routed approved-with-changes like a rejection: back to the person who owns the file, revised proof required, customer reconfirmation before it moves. Slower than a clean approval. Correct. The entire value of separating this into its own decision is that it gets its own path instead of inheriting one built for a different situation.

This matters more than it sounds like it should, because the cost of misrouting isn't symmetric. Routing a real change down the approval path means a proof goes to print with the wrong spec on it. Routing a clean approval down the rejection path costs you a delay and an apology. One of those is expensive in materials and customer trust. The other is expensive in a few minutes of someone's afternoon. When you're not sure which side an ambiguous case belongs on, that asymmetry is the tiebreaker, and it should push you toward routing conservatively rather than routing fast.

Validating a detector against what happened next

Here's the part that actually separates a detector you can trust from one you're hoping works: you don't validate it against whether it agrees with a human's label. You validate it against what happened next.

When our detector flagged an art change in an inbound email or comment, a proof revision usually followed, far more often than it did for similar emails the detector didn't flag. That's the test that matters. Not "did the model say this looked like a change request," but "did the operational record show a change actually happen afterward." The flag either predicted a real downstream event at a meaningfully higher rate than the baseline, or it didn't. Ours did, and that's the only kind of evidence that should move a detector from experimental to trusted.

This is a smaller version of an argument we've made about supervision generally: a checker that scores well on a hand-picked golden set and a checker that scores well against what actually happened next are not the same claim, and conflating them is how systems end up shipping change requests as clean approvals with high confidence attached to the wrong answer. We wrote about the gap between confidence and correctness at length in No learner without a decorrelated checker, and the same logic applies here: don't trust the detector's self-report, trust the trace of what happened downstream of its call.

It's also worth being honest about what "usually followed" doesn't mean. It doesn't mean every flagged email led to a revision, and it doesn't mean every unflagged email was actually clean. Some flags will be false positives that cost a few minutes of unnecessary review. Some misses will slip through, because no detector built on real customer language catches everything, and anyone who tells you theirs does hasn't looked hard enough at the misses. What the comparison tells you is direction and magnitude: is the flag doing meaningfully better than noise, by a wide enough margin to justify routing decisions on it. That's the bar. It's lower than perfect and higher than "seems reasonable," and it's the only bar that actually protects you operationally.

If you want the harder version of that argument, on why a human sitting in the approval loop doesn't automatically fix any of this, we made the case in Human-in-the-Loop Is Not a Safeguard. A reviewer who's staring at a "status question" that's actually a cancellation, inside a queue triaged by a keyword filter that already decided it wasn't urgent, isn't a safeguard. They're a second failure point stacked on the first one, reviewing the wrong bucket instead of the right one. The fix has to happen at triage, before the human ever sees the case, or the human inherits the triage system's blind spots along with its outputs.

Where this leaves the build decision

None of this is exotic. A keyword filter is the obvious first build, and it's also the wrong place to stop. The gap between "catches some change requests" and "catches the change requests that actually matter" is the gap between a system you demo and a system you run production on. Closing it takes intent detection instead of string matching, a third decision path with its own routing instead of a flag bolted onto approval, and a validation method that checks against what happened next instead of against a hand-labeled test set.

This is also, not coincidentally, the kind of workflow that off-the-shelf tools handle badly and custom software handles well, because the logic here is specific to how your customers actually write to you, not to a generic template of what a change request looks like. It's a business logic chain: approval click, comment field, downstream production step, reconfirmation, proof revision. Modeling that chain as it actually runs, rather than forcing your operation into a vendor's three pre-built statuses, is the difference between a detector that improves your throughput and one that quietly ships the wrong logo size at scale.

If change requests are getting lost in your approval queue or your support inbox right now, that's worth a conversation before it's worth a rebuild.

Related: Human-in-the-Loop Is Not a Safeguard · Who Checks the Checker? Managing Agents Is a Measurement Problem

Book a discovery call and we'll look at where your changes are actually hiding.