A Dry Run Against Shared Production Isn't Safe

Read-only is not a safety property. That sentence should be obvious, and it wasn't, to us, in September 2026, until we watched it fail.

Here's the position, stated before the story: a dry run against a shared production connection carries the same blast radius as a write, because the setting that makes it "safe" doesn't stay scoped to the script that set it. If the connection is pooled, the setting rides along with the connection to whoever gets it next. That's not a rare edge case in how pooling works. That's the default behavior of every pool we know of. So the fix isn't "be careful with dry runs." The fix is: every dry run, replay, and backfill goes through the same pre-flight, no exceptions, and the pre-flight is a shared piece of tooling, not a line in someone's memory.

We're going to walk through what happened, because the shape of it is more useful than the summary.

The script that looked harmless

The job was a snapshot script. It connects to the production database, reads a slice of state, writes it out somewhere else for a replay we run downstream. It doesn't write to the database it's reading from. It never has. The person who wrote it did the responsible thing and set the session to read-only before running any queries, on the theory that if something in the query logic went sideways, the database itself would refuse the write and nothing bad could happen.

That theory is correct for a script running on its own dedicated connection. It is not correct for a script running on a connection borrowed from a shared pool.

The distinction matters because of what a connection pool actually is. A pool doesn't hand out a database connection and let it die when your script finishes. It hands out a connection, waits for you to release it, and then gives that exact same connection, with whatever session state you left on it, to the next thing that asks. If your script sets SET SESSION READONLY and then returns the connection to the pool without resetting it, the pool doesn't know or care that the setting was yours. It just sees an available connection. The live application asks for a connection next, gets the one your dry run just used, and inherits the read-only session your script set thirty seconds earlier.

That's what happened. The dry-run script shared a pooled connection with a live application. It set read-only. It returned the connection. The application picked it up and, for about fifteen minutes, a fraction of its writes started failing, rejected by the database as read-only violations. Some of those failed writes were records that never got a retry before whatever queued them moved on. Some were webhook deliveries, fire-and-forget by design, that just didn't fire. We lost some of both before anyone understood what was happening.

Why the error table under-counted it

The first place we looked, once things looked wrong, was the error table. We expected a spike. We got a shrug.

The problem is that "write failed: read-only transaction" doesn't look like an incident from inside a single service. It looks like a database error, one of dozens of database error strings that show up in logs on any given day for reasons that have nothing to do with each other: a lock timeout here, a constraint violation there, a connection reset somewhere else. Nothing about the error string itself says "your connection pool just leaked a session setting from an unrelated script." It just says the write failed and gives you the reason the database gave.

So the count of failures was real, but the count of incidents looked like zero, because nobody had connected fifteen minutes of scattered read-only errors across different code paths to one dry-run script that had finished running and exited cleanly minutes before. The script itself reported success. It did exactly what it was supposed to do: read some rows, write a snapshot file, exit zero. From the script's point of view, nothing went wrong at all. The damage was entirely downstream, in a system the script's author wasn't even looking at when they ran it.

This is the part we want to sit on for a second, because it's the part that generalizes past this one incident. A tool that succeeds on its own terms while breaking something adjacent is a worse failure mode than a tool that fails loudly, because nothing routes you to look. You don't get an alert from the dry-run script. You get an alert, if you get one at all, from a completely different service complaining about writes it can't explain, twenty minutes later, after the connection has probably already been recycled again and stopped mattering.

The second occurrence: a lesson that lived in a note

Here's the detail that actually changed how we operate, and it's not the outage itself. It's that this had happened before.

An earlier version of roughly the same mistake had occurred previously, been diagnosed, and been written down. Somewhere in our internal documentation there was a paragraph that described almost exactly this hazard: shared pooled connections, session-level settings, the risk of a script's read-only flag leaking to the app. The hazard had a name. It had an explanation. Nobody consulted it before writing the second script, because nobody thought to. The knowledge existed. It just didn't exist anywhere that stood between a person and the moment they were about to make the mistake again.

That's the actual lesson, and it's an uncomfortable one, because "write it down" is the default response to almost every incident retro, and it is close to worthless if the write-up lives in a wiki page that nobody opens at the moment they're writing a new script. A note is a record. It is not a control. The difference between the two is whether the system enforces the lesson or merely remembers it. Ours only remembered it, and memory is exactly the kind of safety property that fails silently, because nothing tells you it's failed until the incident repeats.

We've made this argument before in a different context, about checkers and catch rates: a safeguard you can't measure isn't a safeguard you can rely on, it's a story you tell yourself about your process. Human-in-the-Loop Is Not a Safeguard makes that case for approval gates staffed by people. The pooled-connection hazard is the same failure at a different layer. A written incident report is a human-in-the-loop control with a sample size of "whoever happens to read it before they need it." That's not a control. That's luck with a paper trail.

What we changed

We didn't respond to this by asking people to be more careful. We've stopped trusting "be more careful" as a fix for anything, for the same reason we don't trust unmeasured human review to catch what a system misses on its own: carefulness doesn't compound, and it doesn't transfer from the person who lived through the incident to the person writing the next script weeks later who never heard about it.

Three changes, in order of how much we lean on them:

A hard rule, not a guideline. No script that touches a shared connection pool is allowed to set a session-level setting on a borrowed connection. Full stop. If a script needs read-only behavior, it gets its own connection, opened directly, not pulled from the pool the application uses. This is enforced as a rule, meaning it's the kind of thing that gets checked, not the kind of thing that gets remembered.

A shared pre-flight every script imports. Instead of asking each script's author to reason correctly about pooling every time, we built one piece of shared tooling that every replay, backfill, and dry-run script now imports before it does anything else. The pre-flight does two things: it detects whether the connection it's about to use is a pooled one, and if it is, it refuses to proceed on that connection and opens a direct one instead. Then it scopes any read-only behavior to a single transaction, not the session, so even in the worst case the setting can't outlive the block of work it was meant to protect. Read-only inside one transaction dies when the transaction ends, whoever holds the connection next. Read-only at the session level lives until something explicitly resets it, and "something explicitly resets it" is exactly the step that gets skipped under a pool's normal churn.

Review checks that look for session-level settings specifically. Not a general "review this code" pass. A specific check, the kind you can write down as a single question: does this diff set anything at the session level on a connection that might be shared? If yes, it doesn't ship until it's rewritten to scope to a transaction or open its own connection. This is a narrow, cheap, high-signal check, and it's the kind of thing we think belongs in code review generally: not "is this good code" as a vibe, but a specific, answerable, checkable question that a reviewer can get right or wrong. We've written about this distinction before, in the context of what a checker actually catches versus what it's assumed to catch: Who Checks the Checker? Managing Agents Is a Measurement Problem. A review pass that's supposed to catch "anything bad" catches almost nothing reliably. A review pass built around one sharp, specific question catches that one thing, close to every time.

Why this isn't really a database story

We started this piece with a claim about read-only settings and connection pools, and that's a real, specific, technical thing worth getting right. But the pattern under it is broader than databases, and it's the same pattern we keep running into across the operational software we build: a safety property that holds under one set of assumptions (a dedicated connection, a script running alone) silently stops holding once the environment changes (a shared pool, concurrent traffic) and nothing in the system tells you the assumption broke.

This is the same shape of problem we've written about with automated decision systems generally. In No learner without a decorrelated checker, the argument is that a confidence score without a measured, independent check on it isn't confidence, it's a guess wearing a number. The read-only flag in this incident is the same thing wearing different clothes: a setting that looked like a guarantee and was actually a guess about how the connection would be used, unchecked against the reality of how the pool actually behaved.

The fix, in both cases, is the same shape too. Not "trust the setting more" or "be more careful with the setting." Build the actual check, make it structural, and stop relying on someone remembering the hazard at the right moment. A note in a doc is memory. A pre-flight every script imports is a guardrail. Only one of those survives the person who wrote it leaving the team, or the twenty other things they're thinking about the day they write the next script.

The plain admission

We caused this. It was our snapshot script, our pooled connection, our fifteen minutes of failed writes and lost webhook deliveries, on our own operational tooling. Nobody outside forced this mistake on us and nobody else's system absorbed the damage. We'd also seen a version of this exact hazard before and hadn't turned it into anything that would have stopped the second occurrence. That's on us too, and it's the more important admission of the two, because the first mistake is understandable and the second one is a process failure.

If you're running dry runs, replays, or backfills against any database your live application also touches through a pool, and your safety plan for those scripts is "we set it to read-only," stop and check one thing before anything else: is that setting scoped to a transaction, or to the session? If it's session-level and the connection is shared, you don't have a safety property. You have a setting waiting for the wrong process to pick it up next.

If you're building or re-architecting the operational software around your production systems, this is exactly the kind of failure mode that gets missed when software gets bolted on around an existing database rather than modeled against how your operation actually shares state. We build that kind of software deployed in increments, running parallel to what you already have, so problems like this one surface in a staging pass instead of in your production logs at 2pm on a Friday. If you want to talk through where your own pooled connections and shared state might be hiding a version of this, book a discovery call.