Are you measuring CRO wrong? More tests, less ego

I interviewed at a health-focused eComm startup recently, and they said something that completely flipped my mental model for CRO.

“We don’t optimise for conversion rate, we optimise for number of tests run.”

At first, I mentally raised an eyebrow. Surely the goal is winning tests, right? Lifts in revenue? Solid case studies? What’s the point of running more tests if they’re not moving the needle?

But the more I thought about it, the more it started to make sense…

Most companies want winners, not experiments

In a lot of businesses, testing isn’t really about experimentation – it’s about justifying decisions.

Stakeholders want high win rates. Nobody wants to run a test that drops revenue. And too often, the CRO process becomes this overly cautious, political minefield where you only get to test if the outcome is basically guaranteed.

The result? You get three tests a quarter, all based on safe ideas, and everyone’s shocked when nothing improves.

I’ve seen this play out first-hand. You fight tooth and nail just to get a variant live. It finally runs. It loses, or worse, it’s flat.

The whole thing gets quietly buried. Stakeholders retreat back to "let's just redesign the homepage" and testing momentum dies.

So when I heard a team say they don’t care about win rate, my first reaction was: that sounds chaotic. But also… freeing?

What if test volume was the better KPI?

Here’s the logic I’m starting to appreciate:

• More tests = more learning.

• You stop obsessing over perfection and start building a culture of curiosity.

• You uncover unexpected wins – and unexpected losses – faster.

• You move from ego-driven design (“my idea better work”) to systems thinking.

And honestly? It’s just more fun. Test ideas don’t have to be precious. Failures aren’t personal. You get to move fast, stay honest, and build momentum.

Kind of like product-led growth, but for ideas.

Maybe CRO isn’t about having the perfect batting average. Maybe it’s just about swinging more often.

How I’d run multiple tests (properly)

So what would I do if I had enough traffic to play with?

In VWO, you can run several campaigns in parallel as long as they target different pages (e.g. homepage, PDP, checkout). Since the changes load on different pages, there’s no conflict.

You can also run multiple campaigns on the same page, but only if they don’t touch the same elements. Overlapping changes (e.g. two tests editing the CTA) can break functionality or skew results. To avoid this, I’d plan tests around distinct elements or user segments.

For conflicting ideas (like two different PDP redesigns), I’d group them in a mutually exclusive campaign, so users only see one experience. Tools like VWO don’t enforce this automatically – you have to set it up intentionally using campaign settings or targeting logic.

I’d also build a repeatable system: test templates for common changes, a prioritised backlog, and light QA to keep velocity up. Most programs slow down not because of traffic limits – but because launching each test feels like a project.

So yes, multiple tests are doable. You just need structure, not chaos. With good scoping and a high-traffic site, it’s a powerful way to learn faster – without wrecking your data.

The reality: test velocity is hard

Now, to be clear – I’ve never been in an environment where high test volume was actually possible. Not yet, anyway.

Usually, I’m trying to convince a business why we should even run a test in the first place. Or why the “failures” are still valuable.

In most companies, testing at scale feels like a luxury:

• You need clean tracking.

• You need fast dev/design support.

• You need stakeholder buy-in.

• And most of all, you need a culture that doesn’t panic when a test loses.

But just because it’s hard doesn’t mean it’s wrong. It might actually be the thing that separates truly innovative teams from the rest.

Where I’d love to take things

I’d love to work somewhere that treats tests like ideas – not bets.

Where failure isn’t just tolerated, but expected. Where we run lots of small, fast, rough tests to chip away at real user behaviour. Not to “get lifts,” but to understand customers better.

That doesn’t mean we stop caring about results. But it does mean we care more about the system than the scoreboard.