An experiment is a way of being wrong cheaply, on purpose.
Every release is already a test. Usually you run it with no control group, no stated expectation, and no way to tell whether the number moved because of your change or because it was the first week of January. Doing it deliberately means writing the prediction down first — which metric, which direction, roughly how much — and then showing the change to some users and not others.
Experiments are not free. They spend traffic, and they spend weeks — enough that you cannot run one on every question. Keep them for decisions where you genuinely do not know the answer and where the answer would change what you build next. A test whose result you would ignore is theatre.
A/B Testing
Split users at random into two groups, show each a different version, and compare them on one metric named in advance.
Randomisation does the work. Assigning by signup date, country, or whoever opted in compares two populations, not two designs.
Change one thing. If your variant has a new headline, a new colour and a new price, a win tells you nothing about which of the three earned it.
Fix the sample size and the run length before you start, then leave them alone. Three numbers set the size: where the metric sits today, the smallest change that would actually alter your decision, and how sure you want to be. A sample size calculator does the arithmetic. Stopping the moment a live dashboard turns green is how teams ship noise and call it a finding. Run a full week at minimum — weekday and weekend behaviour are not the same.
Plenty of tests come back flat. That is still an answer.
- A/B Testing — What it is, examples, and best practices. — article
- The complete guide to A/B testing in 2026 — article
- What is A/B Testing in Design & User Experience Research? — video
- How To A/B Test a Product — video
Feature Flags
A flag is a switch in the code that decides, per user, whether a feature is on. It separates shipping the code from releasing the feature — two events that used to be one.
That buys you three things. Gradual rollout: one per cent of users, then ten, watching error rates and support volume between steps. Rollback that takes a toggle rather than a deploy, a build and a panicking engineer at eleven at night. And the delivery mechanism for A/B tests, since a flag is already a way of assigning users to variants.
The cost is accumulation. Every flag is a branch in the code and another state somebody has to test. Give each one an owner and a removal date the day you create it, or your codebase turns into a cupboard of dead switches nobody dares flip.
- What Are Feature Flags? — article
- What are Feature Flags? — video
- How To Build Feature Flags Like A Senior Dev In 20 Minutes — video