Testing in Production vs Testing Before Production: Our Approach

"We test in production" is usually said as a joke, an admission that something shipped without enough validation. Said seriously, it means something different - and more useful. There are things you genuinely cannot verify before real traffic hits a system, and pretending otherwise just means finding out the hard way, unplanned, instead of on purpose. Here's how we actually think about the split.

What Has to Be Validated Before Production

Anything with a clear correct answer gets tested before launch, without exception. Core business logic, calculation accuracy, security boundaries, data integrity constraints - these have a right answer independent of real traffic, and there's no reason to wait for production to check them. Unit tests, integration tests, and staging environments with realistic data handle this category, and we don't treat "we'll catch it in prod" as an acceptable substitute for it.

What Only Production Can Actually Tell You

Some things don't have a knowable right answer until real usage exists. How a system behaves under genuinely unpredictable concurrent load, how caching performs against real user access patterns instead of synthetic ones, how a recommendation or ranking algorithm performs against actual user behavior rather than modeled behavior -staging can approximate these, but only production traffic tells you the real answer. Pretending staging fully substitutes for this is how systems pass every test and then behave unexpectedly on day one anyway.

How We Actually Test in Production, Safely

The phrase implies more recklessness than the practice actually involves. When we do validate something against real production conditions, it's through controlled mechanisms specifically built for it: feature flags that let us enable new logic for a small percentage of real traffic before a full rollout, canary deployments that route a fraction of requests to a new version while monitoring closely, and shadow traffic - where a new system processes a copy of real requests without its output affecting real users, purely to compare behavior. None of this is "ship it and see what breaks." It's structured exposure to real conditions with a fast, deliberate rollback path if something looks wrong.

The Line We Don't Cross

We don't use production testing for anything where a failure has an unrecoverable or high-cost consequence for a real user - financial transactions, data-altering operations, anything security-related. That category gets exhaustively tested before production, full stop, because the cost of being wrong in front of a real user isn't acceptable even at small scale. Production testing is reserved for things where a wrong result is observable, contained, and reversible.

Why We Don't Treat This as an Either/Or

The framing of "testing in production versus testing before production" implies you pick one philosophy and commit. We don't. Both exist in the same pipeline, applied to different categories of risk - pre-production testing for anything with a checkable right answer, structured production testing for anything that genuinely can't be known until real conditions exist. Treating them as competing philosophies usually means either over-relying on staging environments that never quite match reality, or under-testing before launch and calling it agile.

What This Looks Like on a Real Project

On a recent client project involving a personalized content feed, we validated data accuracy, security, and core logic entirely pre-production - that had a right answer and staging could confirm it. Ranking quality, however, only became measurable once real user interaction data existed, so we rolled the ranking logic out to five percent of traffic, watched engagement metrics against the existing system for two weeks, and expanded gradually as the numbers held up. Neither phase substituted for the other - each answered a question the other couldn't.