Testing a video placement without fooling yourself

The usual version: change the placement, watch the numbers for a week, decide it worked. Nearly always that is a seasonal fluctuation being read as a result.

Why before-and-after usually lies

Ecommerce traffic is noisy. It moves with the weather, the day of the week, a competitor’s sale, an email you sent, and an algorithm update you never heard about.

Against that background, a five per cent change in conversion over a week is indistinguishable from nothing. Most placement decisions are made on differences smaller than the noise.

What a real test needs

• Two groups seeing different things at the same time. Same week, same weather, same email. This is the part that makes it a test rather than a story.
• Enough traffic. A few hundred sessions per arm will tell you nothing. If your store does not have the volume, be honest about that rather than running a test that cannot conclude.
• One change. Moving the placement and swapping the videos at the same time means you learn nothing about either.
• A decided-in-advance metric. Write down what you are measuring before you start, or you will find something that moved and call it the result.

The flicker problem

A technical detail that ruins many split tests. If the page renders one version and then swaps to the other after JavaScript runs, some visitors see the change happen. That flicker is itself an experience, it affects behaviour, and it contaminates the result.

Whatever splits the traffic has to do it before the page paints, so a visitor only ever sees one version. A test that flickers is measuring the flicker.

When not to test

Testing has a cost: time, complexity, and a period during which half your traffic is getting the worse version.

Do not test things that are obviously right. A video placement that pushes the buy button below the fold on mobile does not need a test; it needs moving. Save testing for genuine uncertainty between two reasonable options.

The pragmatic alternative

If you cannot run a proper test, run a long one-way change instead. Make the change, leave it for a full month, and compare against the same month last year as well as against last month. It is weaker evidence, and it is much better than a week.

And write down what you did and when. Six months later, the question “when did we move that” comes up constantly and nobody ever remembers.