Choosing an LLM for a Chatbot: The Four Questions That Decide It

Most model comparisons open with a leaderboard, and almost nobody makes a real decision from one. Three reasons, and understanding them saves a great deal of reading.

They measure a different job

Graduate-level reasoning and competitive coding tell you very little about whether a model will answer a delivery question from your help centre in two sentences without inventing a refund policy. The benchmark tests the ceiling; a chatbot lives near the floor, doing an easy task reliably thousands of times.

The spread is narrow where it matters

For retrieval-grounded question answering - what most chatbots actually do - the mainstream models are close enough that the difference is swamped by your prompt and your content. The variance between a good and a bad knowledge base is larger than the variance between frontier models.

They go stale in weeks

Any page quoting a leaderboard is describing a moment, not a property. By the time it is indexed, two of the rows have changed.

What decides it instead

Four constraints, none of which appear on a leaderboard. Work through them in this order, because each one narrows the field before the next.

What a leaderboard cannot tell you

There is a further problem, subtler than staleness. Benchmarks are scored on whether the final answer is correct. A chatbot is judged on a dozen things that are not correctness: whether it stopped at two sentences, whether it refused gracefully, whether it kept your terminology, whether it escalated at the right moment. A model can win every benchmark row and still be the wrong choice because it writes four paragraphs when you needed one line.

There is also a selection effect worth knowing about. The models that top public leaderboards are the ones whose labs optimise for them, and the gap between benchmark performance and behaviour on messy real input is not constant across vendors. A model tuned hard for evaluation sets can be noticeably more brittle on the kind of half-formed question a real customer types at eleven at night.

Where comparison articles go wrong

Most published comparisons test models on questions the author invented. Invented questions are clean - correct spelling, one intent, enough context. Real questions are none of those. They contain typos, two questions at once, an order number with a missing digit, and an assumption about your business that is wrong. The gap between how models handle clean and messy input is larger than the gap between models on clean input, which is why your own twenty questions beat any article.

Start here, because it is binary and it eliminates options before capability is even considered. If a data protection assessment says customer text cannot leave a jurisdiction, no amount of capability makes a model in the wrong place acceptable.

The distinction teams miss

Storage location and inference location are different questions with different answers. A platform can store transcripts in Frankfurt and send every message to a model API in the United States, and both statements on its website are true. Ask which model provider by name, and in which country inference happens. The full data trail covers what else to ask.

The options by jurisdiction

The self-hosted row is the one people forget exists. Several strong models publish open weights, so "nothing may leave our network" is a constraint you can satisfy rather than a reason to abandon the project.

What "EU hosted" usually means

Three different things, and vendors are rarely precise about which. It can mean the application servers are in the EU, that transcripts are stored in the EU, or that model inference happens in the EU. The first two are common and the third is rarer, and only the third addresses the question a data protection officer is actually asking about AI processing.

Ask for it in writing, naming the sub-processor. A verbal assurance in a sales call is not something you can put in front of a reviewer eighteen months later when a customer questionnaire arrives.

The self-hosting option, honestly

Running an open-weight model on your own hardware genuinely answers "nothing may leave our network", and it is a real commitment rather than a checkbox. You own the GPUs, the uptime, the model updates and the capacity planning. Latency is whatever your hardware delivers, and scaling for a traffic spike is your problem rather than a provider's.

It suits organisations with existing GPU capacity and a hard constraint, and suits almost nobody else. The useful thing is that the option exists, so the constraint is satisfiable. The comparison that matters is not self-hosting against a frontier API, but self-hosting against not being allowed to deploy at all.

Sub-processors compound

Each provider you add is another entry on your sub-processor list, another location, another set of terms, and another party to notify customers about. Configuring a fallback provider in a different jurisdiction from your primary is operationally sensible and quietly doubles your disclosure obligations. Decide that deliberately rather than discovering it during a review.

Model price only matters above a threshold, and most teams either obsess over it below that threshold or ignore it above.

Do the arithmetic before the research

Take your monthly conversation count and your average messages per conversation. A lead bot handling two hundred conversations will not notice the difference between the cheapest and most expensive option - the cost is swamped by everything else you spend. A support bot handling ten thousand will notice nothing else.

Whose bill is it?

This decides whether a cheaper model helps you at all. Where you bring your own API key, token usage bills at the provider's published rates and switching to a cheaper model reduces your cost. Where AI is bundled into per-message pricing, switching saves the vendor money and you see nothing. Ask which model you are on before you buy, and it is worth raising during vendor evaluation rather than after.

The cheaper lever

Before changing model, check how much context you send on every turn. Pasting an entire document into the system prompt on each message is the commonest cause of a surprising bill, and a well-chunked knowledge base is both cheaper and more accurate than a large prompt.

The costs nobody models

Token price is the number on the pricing page and rarely the number that surprises people. Three others matter more in practice.

• Retries. A poorly grounded bot that gets the answer wrong first time costs two calls plus a frustrated customer. Grounding is a cost control as much as a quality control.
• Context length. A system prompt carrying your entire policy document is billed on every single turn of every single conversation. This is the commonest cause of a bill that is four times the estimate.
• Conversation length. Most APIs are stateless, so the whole history is resent each turn. A ten-turn conversation costs far more than ten times a one-turn conversation, because turn ten carries turns one through nine with it.

Reasoning models cost differently

Models that reason before answering bill for the reasoning as well as the answer, and that hidden portion can exceed the visible response. A flow that routes every question to a reasoning model when only one branch needs it can cost several times what it should. Route deliberately: chat model by default, reasoning model on the branch that genuinely reasons.

What to measure once live

Cost per resolved conversation, not cost per token. A cheaper model that resolves fewer conversations and escalates more is not cheaper once you count the human minutes on the other end of the handover. That number is the only one that connects model choice to the thing the business cares about.

Latency is felt, not measured, by the person waiting. The number that matters is time to first token, not total generation time, because a reply that starts streaming immediately feels fast even if it takes four seconds to finish.

Reasoning models pause

Models that reason before answering produce better results on multi-step problems and cost seconds of visible silence. On a high-traffic website widget that pause is expensive. On an eligibility check or a troubleshooting tree it is worth it.

Where speed is the whole product

Providers serving open models on custom inference hardware - Groq and Cerebras among them - exist because first-token latency is a real differentiator. If your widget sits on a high-traffic page, they are worth testing. The typing indicator and wait behaviour matter here too; a bot that says nothing for four seconds feels broken regardless of which model is behind it.

Perceived speed is not measured speed

A response that begins streaming in 400ms and completes in four seconds feels faster than one that arrives complete after two. People read as text appears, so streaming converts waiting into reading. If your platform supports streaming, enable it before optimising the model - it is usually a larger perceived improvement than any model change.

What breaks the illusion is a long silence before the first token. That is the moment a visitor decides the thing is broken, and it is why time to first token is the metric worth watching rather than total duration.

Where the latency actually comes from

• Retrieval - searching your knowledge base before the model is called.
• Any API call in the flow - an order lookup can easily exceed the model's own latency.
• Model inference - the part everyone optimises.
• Network distance to the provider region.

Teams change models to fix latency caused by a slow order lookup, which is a common and expensive mistake. Measure each hop before attributing the delay.

Cover the wait honestly

Where a wait is unavoidable, say so. "Checking that with the warehouse, about ten seconds" converts an unexplained pause into an expected one. Silence for the same duration reads as failure, and the difference in abandonment between the two is larger than the difference between a fast and a slow model.

The constraint that decides most real deployments, and the one no table can answer.

What actually differs

• Length. Some models write three sentences where you wanted one. On a phone, a long message pushes the input field off screen.
• Confidence. Some hedge, some assert. Hedging reads as unhelpful; over-confidence invents.
• Literalness. Some follow a system prompt word for word, others treat it as a suggestion. Literal models are easier to control and need more explicit instruction.
• Refusal behaviour. The most important one. What happens when the knowledge base does not cover the question - does it say so, or produce something plausible?

Refusal is the behaviour to test hardest

A bot that invents a refund window is worse than one that says it does not know, because the customer acts on it. This is a property of the model and the prompt, and it varies more between models than accuracy does. It also determines how often your fallback fires correctly rather than being bypassed by a confident guess.

Prompting narrows the gap, but not to zero

Much apparent model personality is promptable. A verbose model told to answer in under sixty words will mostly comply, and a hedging model told to state uncertainty once and move on will mostly comply. The residual differences are in how reliably they comply under pressure - on an ambiguous question, on turn fifteen, when the retrieved context partly contradicts itself.

That is why the test set should include the awkward cases. Any model handles the easy ones; you are buying behaviour at the edges.

Consistency matters more than any single reply

A model that produces a brilliant answer four times out of five and an eccentric one the fifth time is worse for support than one that is consistently good-not-brilliant. Customers experience variance as unreliability, and support teams cannot write process around a system that behaves differently each time. When comparing, run the same question several times and look at the spread, not just the best output.

Test the refusal explicitly

Write three questions your content genuinely cannot answer and watch what each model does. You want a clean admission and a handover. What you often get is a confident, plausible, invented answer - and you will only see it if you deliberately ask something unanswerable. This single test tells you more about production behaviour than every benchmark combined.

This takes an afternoon and beats every comparison article, including this one.

1. Take twenty real questions from your inbox, chat logs or search console. Real ones, with the typos and odd phrasing. Invented examples are always cleaner than reality and will mislead you.
2. Include five you cannot answer. Out of scope, ambiguous, or about something you do not sell. Behaviour with no grounded answer matters more than behaviour with one.
3. Hold everything else constant. Same flow, same prompt, same knowledge base. Change only the model.
4. Read the outputs side by side. Not scores - the text. Length, tone, hedging, invention, whether it handed over when it should have.
5. Time the first token on a few, because that is what a visitor feels.
6. Have someone who was not involved read them blind and pick which sounds like your company.

What you will notice

Accuracy differences are usually smaller than expected. Differences in length, confidence and willingness to say "I do not know" are usually larger. The last is the single most important behaviour in a support bot and appears on no benchmark.

How to score it without a spreadsheet

Resist scoring each answer out of ten - it produces a number that feels objective and hides what you actually noticed. Instead, for each question, mark one of three things: shipped (I would send this to a customer), fixable (right substance, wrong length or tone), or wrong (inaccurate, invented, or failed to hand over when it should).

Count only the third category. Everything fixable is a prompt problem you will solve regardless of model, so it tells you nothing about which model to buy. The wrong column is the one that differs meaningfully between candidates.

Who should read the outputs

Someone who answers these questions for real, not the person who built the flow. Builders unconsciously read past errors because they know what the bot meant. A support agent reading blind will spot the invented policy detail in three seconds, because they know the real one.

Run it again in six months

Models change under you. The provider ships a new version, the default endpoint moves, behaviour shifts. Keep the twenty questions in a file and re-run them after any provider update or when quality complaints appear. It takes twenty minutes the second time and it is the cheapest regression test available for the least deterministic part of your stack.

Before agonising over which model, check the thing with a larger effect than all of them: whether answers are grounded in your own content.

A mid-tier model answering from a well-organised knowledge base will outperform a frontier model guessing, on every metric you care about. Retrieval reduces invention far more reliably than any instruction telling a model not to invent, and it is the difference between a bot that quotes your actual refund window and one that produces a plausible number.

The order of work

Teams routinely work bottom to top, spend a week on model selection, and ship a bot that invents answers because nothing was grounded. Training a chatbot on a knowledge base covers steps one and two; the prompt engineering guide covers step three.

Why grounding beats capability

A model has two sources for any answer: what it absorbed in training, and what you put in front of it. Training data is stale, generic and knows nothing about your refund window. Retrieved content is current, specific and yours. When the two conflict, a well-grounded setup uses yours; an ungrounded one uses whatever sounds right.

That is the entire difference between a bot that says "returns within 30 days" because it is your policy and one that says it because 30 days is the most common policy on the internet.

What good grounding looks like

• Chunked sensibly - passages that answer one question, not whole pages.
• Current - with an owner who updates it when the product changes.
• Non-contradictory - two pages disagreeing about the refund window will produce a bot that disagrees with itself.
• Scoped - the model should be told to answer only from retrieved content and to say so when it cannot.

The audit worth doing first

Before any model comparison, take your top twenty questions and check whether an accurate, current answer exists in your content at all. Teams are routinely surprised: a third of their most common questions have no written answer anywhere, which no model can fix. That audit usually reorders the whole project plan, and it is an afternoon's work.

The most useful property of a model decision is being able to unmake it. If switching means rebuilding the flow, the choice carries weight it does not deserve. If it is a setting, twenty questions is enough to decide and you can revisit next quarter.

Three things to check before committing

• Is the provider a setting or an architecture? Where the flow, knowledge base, transcripts and widget are provider-agnostic, switching is a configuration change.
• Whose key is it? Bring-your-own-key means a cheaper model reduces your bill rather than the vendor's.
• Can you run two? A configured fallback means a provider outage degrades the bot instead of ending conversations, and it costs nothing until used.

Why a fallback matters more than the primary choice

Every provider has outages. A bot whose AI nodes error during one is a bot that failed in front of customers on the day you had least control. Configuring a second provider is the cheapest insurance available and the one teams add after their first incident rather than before it.

What actually locks you in

Rarely the model. It is the flow, the integrations and the transcripts. A platform where the conversation design is portable and the provider is a dropdown leaves you free; one where AI behaviour is woven through proprietary nodes does not, whatever its model list looks like.

The question to ask a vendor is not "which models do you support" but "if I switch model, what else changes?". If the answer involves rebuilding flows, the model list is marketing rather than flexibility.

Running two providers in practice

A fallback is not only for outages. It is also how you A/B a new model on live traffic without committing, and how you handle a provider rate-limiting you during a campaign. The operational cost is one more key to rotate and one more entry on the sub-processor list.

Configure it before you need it. The instinct after an incident is to add redundancy; the cheaper time is while nothing is wrong and nobody is watching.

Review it on a schedule

Put a calendar reminder for six months out to re-run the twenty questions against your current provider and one alternative. Prices fall, new models ship, and the correct answer changes more often in this category than in almost any other part of a stack. A decision you revisit deliberately is worth far more than one you agonised over once.

If you want a starting point rather than a framework:

Then run the twenty-question test against that default and one alternative. If they are indistinguishable on your questions, keep the cheaper one - which is a more common outcome than model marketing suggests.

Deviate when

• Your questions are in a language where you have tested and found a clear difference - see multilingual chatbots.
• Your flow genuinely reasons rather than looks up.
• A compliance constraint eliminates the default outright.

Conferbot supports twenty-one providers for exactly this reason, and the choice is per chatbot rather than per account. The full list, grouped by jurisdiction and open-weight status, is on the AI model providers page, and you can run the test on the free plan before committing. If you have not decided whether to build at all, when not to use a chatbot is worth reading first, and who should own it covers what happens after launch.