A/B Testing Chatbot Flows Without Fooling Yourself

Pick one metric before you start, run both variants concurrently on split traffic, set a minimum sample per variant, and do not look at the result until the test reaches it. The last part is where most chatbot tests fail.

Conversation flows are unusually tempting to call early because results appear within hours. A variant that is 40% ahead after 30 sessions feels decisive. It is noise - at that volume a single unusual afternoon moves the number more than the change you made.

Test things that change behaviour, not things that change appearance. In rough order of observed impact:

1. The opening message. The single highest-leverage element. It decides whether the conversation happens at all.
2. Question order. Asking for an email before delivering value drops completion; asking after it usually does not.
3. Number of questions. Every additional required field costs completions. Testing five fields against three answers a real business question.
4. Choice buttons vs free text. Buttons raise completion; free text collects richer data. The trade-off is worth measuring rather than assuming.
5. Where the human handoff is offered. Too early wastes agents, too late loses people.
6. Proactive trigger timing. When the widget invites rather than waits.

Not worth testing in isolation: button colours, avatar images, minor word changes. The effect sizes are smaller than the traffic most chatbots get, so you will spend weeks measuring nothing.

Declare the metric before you look at any data. Testing five metrics and reporting whichever moved is how teams talk themselves into changes that do nothing.

Pair the winner metric with one guardrail metric you refuse to damage. Optimising completion while satisfaction quietly falls is a bad trade you will not notice unless you named it up front.

The uncomfortable arithmetic: the smaller the effect you want to detect, the more traffic you need - and the relationship is steep, not linear. Halving the effect size roughly quadruples the sample required.

Practical consequences for chatbot testing:

• Large changes need modest traffic. Restructuring a flow that moves completion from 40% to 55% can resolve in a few hundred sessions per variant.
• Small changes need traffic most bots never get. Detecting a 1-2 point shift reliably takes thousands of sessions per variant. If your bot sees 500 sessions a month, that test will never conclude - do not start it.
• More variants split traffic further. A four-way test needs roughly twice the total traffic of a two-way test to give each arm the same certainty.

Set a minimum sample per variant before starting - 50 is a floor for coarse signals, several hundred is more honest for anything you plan to act on - and a maximum duration so an inconclusive test ends rather than running forever.

A 95% confidence level means that if the variants were genuinely identical, you would see a difference this large by chance about 5% of the time. It is a guard against fooling yourself, not a promise the winner is better.

Two implications teams routinely miss:

• Peeking inflates false positives. Checking daily and stopping the moment significance appears means you will hit that 5% threshold far more often than 5% of the time. Decide the sample size, then wait.
• The threshold is a business choice. 95% is conventional. For a low-risk copy change 90% may be fine; for a change to a checkout flow you may want 99%. Lower confidence means faster decisions and more wrong ones.

"Inconclusive" is a legitimate and common outcome. A test that ends without a winner has told you the change does not matter much - which is useful, and cheaper than shipping it and wondering.

1. Calling it early. The dominant failure. Early leads reverse constantly.
2. Sequential testing. Running variant A this week and B next week measures the difference between weeks, not variants. Always run concurrently.
3. Uneven traffic sources. If one variant catches a campaign spike, you have measured the campaign.
4. Changing several things at once. A wins, but you cannot tell which of the four edits caused it, so you learn nothing transferable.
5. Ignoring the guardrail. Completion up, satisfaction down, nobody noticed.
6. Not re-testing after the context changes. A flow tuned for one traffic mix may not hold when your audience shifts.

A repeatable sequence:

1. Write the hypothesis. "Moving the email request after the pricing answer will raise completion, because people resist giving contact details before receiving value." If you cannot write the "because", the test is a guess.
2. Pick the winner metric and one guardrail.
3. Set minimum sessions per variant and a maximum run time.
4. Split traffic evenly unless you have a reason to protect the control, in which case weight it deliberately.
5. Leave it alone until it reaches the sample floor.
6. Apply, document, and move on. Record what you tested and what happened, including the inconclusive ones - that log is what stops the team re-testing the same idea next quarter.

Conferbot supports two to four flow variants bound to saved flow versions, with even or custom traffic weights, a configurable minimum sessions per variant, a winner metric chosen from completion rate, conversion rate, average duration, average CSAT or leads collected, and a confidence level you set between 80% and 99%. Experiments can end automatically at a maximum wait time and optionally apply the winner, and results explicitly carry an inconclusive state rather than forcing a winner.