We Ran the Cascade on a Month of Real Purchase-Order Mail. Here Is What 300 Blind Labels Said.

Last week we argued that most operational AI decisions belong in a cascade: code first, a small decision model second, a frontier model only when the first two can't agree. (AI assistance vs automation: the decision tier that matters.) That was the theory. This is the test, on one ugly, real decision, with the numbers we got. It is also our clearest answer to the question behind every AI project we take on: what does it take to make AI reliable and dependable enough to run in production? Our answer is an order of operations, measured before anything ships.

The decision: which files in an inbound purchase-order email are usable artwork?

It sounds trivial. It is not. A promotional-products supplier receives thousands of purchase orders a month from distributors. Every email carries a pile of files: the PO itself, sometimes a quote or an invoice, sometimes the logo the end customer wants printed, and almost always junk. Email-signature logos. Social icons. Tracking pixels. Screenshots. The supplier's own logo quoted back in the reply chain. Get it wrong one way and an artist hunts through the email by hand. Get it wrong the other way and a distributor's signature logo lands on a proof sheet.

The incumbent was a hand-written rule set: filename keywords, a 100 KB size floor, a larger floor for images. Reasonable rules, written by reasonable people. We measured them.

The setup

We took 30 days of inbound mail, about 9,000 PDFs and images, and pulled every file plus the raw email it arrived in. Nothing ran in production. Everything below is an offline replay against the real corpus.

The architecture in one line: rules via code determinism, then our own System 1 model, then a cascade where disagreement means "unsure", then an LLM, then a human, with every human verdict flowing back as a label. Tier by tier:

1. Rules, as code. Deterministic disqualifiers. An inline image under 200 pixels wide is junk. The supplier's own logo is junk. Social-media icons are junk. An image that appears in ten or more different emails in a month, or arrives from five or more unrelated companies, is a signature, not a customer's art. A .ai or .eps file is art.
2. Our own System 1 model. Fast, single-pass judgment, no reasoning chain. Two small classifiers we train ourselves on frozen open encoders: one reads the pixels (SigLIP 2), one reads the text (the PDF's own text layer plus the filename). The text model answers in well under a millisecond. The pixel model takes about 150 milliseconds an image on an ordinary CPU and about 14 on a consumer GPU. We train the models ourselves, and every weight can run on our own hardware.
3. The cascade gate. When the two System 1 models agree, the file is decided. When they disagree, the answer is "unsure", and unsure escalates. We tried using a single model's confidence score as the gate and it failed: its confidence tracked how well it copied the old rules, not whether it was right. Disagreement between two models that look at the problem differently is the signal that held up.
4. The LLM, reading only. Escalated files, plus any large image pasted inline in the email body, go to an open-weights vision model (Gemma 4) with one narrow question: what kind of image is this, and whose brand is on it? Then code decides. Not a logo: junk. The supplier's or the sender's own logo: junk. Another company's logo: art.
5. A human. Anything still unsure goes to the art team. Their answers become labels, the labels retrain System 1 and re-audit every rule, and the loop tightens every day.

The result

We froze every prediction, hashed the file, and only then had a human label 300 files the system had never been tuned on, blind to what the system said.

On the real 30-day mix:

• Incumbent rules: 93.0% correct.
• Cascade: 97.3% correct, with 0.8% of files sent to a person as "unsure."

On roughly 9,000 files a month, that is about 630 wrong decisions under the old rules and about 240 under the cascade. Roughly 60% fewer mistakes, and the remaining uncertainty is routed to a human instead of guessed.

Where the volume went:

• About 13% decided by code alone. In the holdout, these rules were right on 48 of 48 files and dropped zero real artwork. (The social-icon and repeated-image rules were added after the holdout was frozen. They were checked against all 442 human labels and removed zero real art, but they were not part of the 97.3%.)
• About 71% decided by the two small models agreeing. 146 of 148 correct in the holdout.
• About 15% escalated to the frontier model. About 1,400 calls a month. At current list prices that is well under a dollar a month.

Five things we did not expect

1. Pixels alone lost. Our first pure vision classifier looked excellent against the old rules and then lost to them on the hard cases once a human judged the disagreements. The winning signal for PDFs was the text layer, which almost every digital PDF already carries for free. The vision model earned its keep on images, not documents.

2. A bigger model changed nothing. We tested the frontier tier with Gemma 4, Gemini 3.8 Flash and GPT-6 Luna. All three landed within noise of each other. Gemma was the fastest, about one second a call, and its weights are open, so the same model can move from a hosted API to our own hardware without re-testing anything. Once the cascade handles the easy 85%, the reader stops being the bottleneck.

3. "The model decides" was worse than "the model reads, code decides." Asked directly whether an inline logo was artwork, the model said yes to distributors' own logos. Asked only to read the brand, then handed to one line of business logic ("a distributor's own logo is not the order's art, unless it's a self-promo order"), the same model beat every alternative. Let models do what they are good at. Keep the business judgment in code, where it can be read, tested and audited.

4. The email itself was a better signal than the image. A file attached to an email is usually intentional. An image pasted inline is usually a signature. That one bit of MIME metadata, sitting in every email since the 1990s, separated art from junk better than image size or position ever did. It's not a perfect rule, because customers do paste their logo into the body, so it became a strong prior and an escalation trigger rather than a verdict.

5. OCR was slow and irrelevant. We ran a state-of-the-art local OCR model over several hundred scanned PDFs, including every scan in our labeled set. It averaged about ten seconds a page and changed zero labeled outcomes. The hard cases were hard because of intent, not because the text was unreadable.

Rules need a ratifier, not just an author

Deterministic rules have a quiet failure mode: they rot. The old 100 KB size floor was sensible the day it was written, and it silently dropped a customer's 15 KB vector logo. Nobody noticed, because nothing checks a rule after it ships.

So every rule in the cascade now lives in a registry with its source ("who said this, when, in their words"), the evidence behind it, and a status. Only ratified rules run. Every new human label re-audits every rule. If a rule ever removes a piece of real artwork, it is automatically demoted and waits for a person to re-ratify it.

It fired on its first run. One label said a .ai file wasn't artwork, which contradicted the ".ai is art" rule. The rule pulled itself. A person looked, confirmed the label was a slip, corrected it and re-ratified the rule. The whole loop took a few minutes, and it now works that way permanently.

The same loop on a second decision

The artwork cascade was not a one-off. The same day, we ran the pattern on a different decision in the same pipeline: is the customer email address an extraction model pulled from a purchase order the right one? The review step compared that field against nothing.

A fast model only works once it has learned the job. We tested a small contrastive model that scores a document against a fixed set of answers in tens of milliseconds. Out of the box it was unusable: on customers it had never seen, it ranked wrong emails as more likely correct than right ones. After training only its small decision head on about 3,500 past human verdicts, it reached 0.945 ROC-AUC on a fresh set of customers, caught 62% of the wrong emails at a 5% false-alarm rate, and its confidence scores were calibrated. The mistakes it had to catch were rarely typos. The wrong address was usually a real one in the email, belonging to the wrong person, which is exactly what string matching misses.

The labels were already being produced. Customer-service reps review the extracted fields as part of their normal work. Each edit is a rejection and each untouched approval is a ratification. The edits were stored. The approvals were not, so half of every verdict was lost. We are adding a record of both, at no extra cost to the reps.

Check the downstream system before you count a loss. Sorting a quarter of failed intake emails and reading the unexplained ones one by one, we estimated 45 to 47 orders that never made it in. Checking a sample against the ERP changed the answer: 16 of 20 had been entered there by hand, most within a few days. The real losses were closer to 8 a quarter. The larger cost was manual re-entry, and that is the number worth engineering down.

Where we could be wrong

One person labeled the 300-file holdout, and the same person set some of the business rules it was scored against. That's the right business truth for this supplier, but it's a single rater. The per-tier samples are small, so treat the tier-level numbers as direction and the overall 97.3% as the firm figure. The next step removes this caveat: the art team, who are better judges than any of us, will label a small daily batch, and their labels become the ground truth everything retrains and re-audits against.

What we'd do first

If you run any intake process where a person triages files (claims photos, freight paperwork, invoices versus quotes, compliance scans), start here:

1. Measure the rules you already have against a few hundred human labels. Ours were 93% right, which sounds fine until you count the 630 files a month.
2. Write the obvious rules as code, and give every rule a ratifier and an audit.
3. Add two cheap models that look at the problem differently, and treat their disagreement, not their confidence, as the signal to escalate.
4. Let the frontier model read, and let code decide.
5. Freeze your predictions before anyone labels the test set. Otherwise you're grading your own homework.

The cascade isn't a clever architecture. It's the boring order of operations: cheap and certain first, expensive and careful last, a person at the end, and the humans' answers flowing back in. On one month of real mail, that order of operations cut the mistakes by more than half.