The Bug That Taught Us to Change Our QA Process

Not every process improvement starts with a planning session. Some start with a bug that gets past every check meant to catch it - and forces an honest look at why the process let it through. This is one of those.

What Happened

A change to a client's order-processing logic passed code review, passed our existing test suite, and shipped. It worked correctly for the vast majority of orders. It failed silently for a specific edge case - orders that included a discount code applied to only part of a cart - producing an incorrect total that neither the system nor the customer flagged as wrong at the time. It wasn't caught by us. It was caught days later, when the client noticed a discrepancy while reconciling revenue.

Nothing about this bug was exotic. It was a straightforward logic gap. What made it worth changing our process over wasn't the bug itself - it was understanding exactly why three separate checkpoints all missed it.

Where the Process Actually Failed

Code review missed it because the change looked correct in isolation - the reviewer verified the new logic did what it claimed to do, without independently reconstructing the edge case that broke it. Our test suite missed it because our existing tests covered the common paths and a handful of edge cases we'd hit before, but partial-cart discounts weren't among them - nobody had written that test because nobody had thought to. And staging didn't catch it because our staging data didn't include a partial-discount order in its sample set, so the exact condition that triggered the bug simply never ran before launch.

Three checkpoints, three different reasons, one bug that slipped through all of them. That's the pattern that told us this wasn't a one-off mistake to fix and move past - it was a gap in how we design checkpoints in the first place.

What We Actually Changed

We didn't respond by adding more tests in general, which is the reflexive fix and not usually the right one. We changed how test cases get generated in the first place: for any logic involving combinable conditions - discounts, permissions, pricing tiers, anything where two features can interact - we now require test cases that explicitly cover the interaction, not just each condition on its own. This came directly from the root cause: the bug wasn't in the discount logic or the cart logic individually, it was in their interaction, and our old test-writing habit tested components rather than combinations.

We also changed what staging data looks like. Instead of a hand-picked sample meant to look "representative," staging now pulls a broader, semi-randomized slice of realistic order patterns, specifically to surface combinations nobody thought to construct by hand.

And we changed one thing about code review itself: for logic changes touching pricing, permissions, or anything with financial consequences, review now explicitly includes tracing through at least one combined-condition scenario, not just verifying the stated change works as described.

Why We're Being Specific About This

We could describe this generically - "we improved our QA process" - but that tells you nothing about whether it would prevent a bug like this happening to you. The specific failure mode here - checkpoints that each individually made sense but collectively had a shared blind spot - is common enough that we think it's worth naming precisely, rather than filing it under a vague commitment to "quality."

What Didn't Change

We didn't respond by slowing down releases across the board, and we didn't add review overhead to changes that don't touch pricing or permissions logic. The fix was targeted at the actual failure - combinable conditions in consequential logic - not a blanket reaction that would have added friction everywhere without addressing what actually went wrong.

The Broader Lesson

The most useful process changes we've made haven't come from anticipating every possible failure in advance. They've come from taking a real failure seriously enough to trace it back to its actual root cause, rather than patching the specific instance and calling it resolved. This bug cost us a client's trust for a few uncomfortable days. The version of it we didn't catch next time is the one this process change is meant to prevent.