'It Passed Every Test' Doesn't Mean Your Software Is Safe
Green tests. Passing CI. A clean build that shipped on schedule.
None of that proves the software is safe.
Two real bugs made it through every test written for them — both shipped to production, both sat there until a security pass caught them by hand, both fixed the same day they were found.
We've written before about what actually makes software hold up under real use. The uncomfortable truth sits right next to that post: a green pipeline and a resilient system aren't the same claim. One says the code does what the tests told it to do. The other asks whether the code does the right thing when something unexpected happens. Tests only ever check the first one.
Why "All Tests Passed" Isn't The Same Claim As "This Is Safe"
A test is a question someone remembered to ask. It can only fail if the writer imagined the exact way things could go wrong. Both bugs below are ordinary — nobody wrote an exploit, nobody needed one. The code did something nobody had thought to check for, and every existing test kept passing right through it.
Bug One: A Signature Check That Verified The Wrong Bytes
A webhook handler is supposed to prove a request really came from the payment processor before trusting anything inside it. This one checked a signature correctly, by the book — except against the JSON after the framework had already parsed and re-serialized it, not against the raw bytes that arrived on the wire.
That sounds academic until you know JSON re-serialization doesn't reproduce the original byte order or whitespace. Svix's documentation on webhook signatures is explicit about why: the signature must be computed over the exact bytes received, since a re-serialized body can differ from the original even when the data is logically identical. The effect here: certain legitimate, correctly-signed upgrade events quietly failed verification and were dropped — meaning the paid upgrades they represented may never have applied. No test caught it, because the existing unit tests fed the handler a payload and its signature as a matched pair, generated the same way end to end — none fed it a request that had already passed through a parser.
Bug Two: A Filter That Forgot To Filter
The second bug lived in an API endpoint meant to return one tenant's records, scoped by a query parameter. Under one specific combination of inputs, that scoping parameter was silently stripped before the query ran — and the endpoint didn't error. It returned a 200, with the entire dataset, to whoever sent that combination.
This is the shape OWASP's API Security Top 10 calls out: Broken Object Level Authorization, the single most common API risk on their list — an endpoint leaning on a filter to do authorization work the application layer never enforces itself. It rhymes with the webhook idempotency bug we wrote up last year: both are places where "the request succeeded" and "the request did the right thing" quietly came apart.
What Both Bugs Have In Common: Neither Ever Turned Red
Same pattern both times — no crash, no exception, no error logged. Every existing test kept passing after the bug shipped, because the tests described the happy path a developer had in mind, not the input that actually arrived. Green checkmarks correctly told everyone the code did what it was written to do; nobody had asked whether that was still right once real input showed up — the same blind spot, a different shape, we found when we ran OWASP's agentic AI risk list against a live system.
Why Your Test Suite Is Structurally Blind To This
Unit tests encode assumptions the author already holds. If nobody imagined "what if the signature check runs against the parsed body instead of the raw one," there's no test for it — not missing, unaskable from inside that mental model. It's the same reason fractional CTOs exist as a category: a senior second set of eyes asks different questions than the team that built the thing, because they didn't write the assumptions into the code themselves.
A test proves the code does what you told it to do. An audit asks whether what you told it to do was ever the right thing.
The Fix Isn't "Write More Tests" — It's A Pass That Isn't Looking For What You Already Checked
More tests from the same team tend to encode the same assumptions, just more of them. What catches this class of bug is someone deliberately hunting the gap between "passes" and "safe": re-reading the authorization path line by line, re-deriving what should be impossible, and trying it anyway.
A Four-Week Plan To Run This Before Your Next Launch
- Week 1: List every place your system checks a signature, permission, or filter — not whether it exists, but what it's checked against.
- Week 2: For each one, write down the input that should be impossible, then try to produce it.
- Week 3: Fix what breaks — both bugs above were same-day fixes once found.
- Week 4: Re-run your suite to confirm it still passes, then add the cases you just found.
What To Measure
- Checks verified against their actual source, not just confirmed present.
- Time between a bug shipping and being found — same-day detection is a different risk profile than months-later.
- Findings from re-reading code versus an automated test — if that first number is always zero, nobody's looking.
A clean test run tells you what you already decided to check. Being honest about why a project slips comes from the same habit.
If You Want ZPro To Build This With You
We run the pass your own tests already confirmed can't catch this:
- Reading authorization and signature-verification paths line by line, not just confirming they exist
- Trying the specific input your team assumed couldn't happen
- Fixing what we find the same day, the way both bugs above got fixed
- Handing you a written record of what was checked and what changed, not a vague "looks good"
- Helping you decide when a second set of eyes earns its cost