Evaluating a Chatbot Vendor: 24 Questions That Separate Them

Chatbot demos converge because they are all built to show the same thing: a clean flow, an AI answer, a handover. Thirty minutes in, three vendors look interchangeable, and the decision drifts toward price or whoever replied fastest.

The differences are real but they sit outside the demo - in what happens when the flow gets complicated, when the model is wrong, when you want to leave, and when someone using a screen reader opens the widget. These questions are grouped by what each one protects you from.

A practical note: ask for answers in writing. Vendors are more precise in a document than on a call, and a written answer is something you can hold them to later.

1. Where is conversation data stored, and in which country? Not "in the cloud". A named region.
2. Where does model inference happen, and by whom? Often a different answer from storage, and often a different company.
3. Is our conversation data used to train models? Get this in writing. "We do not train on customer data" should be a contractual term, not a marketing line.
4. What is the retention period, and can we set it? Some industries need short retention; some need long.
5. Can we export everything and delete everything? Ask what format. A CSV of messages without metadata is not a real export.
6. Who are the sub-processors? A model provider, a hosting provider, possibly an analytics vendor. You inherit all of them.

Answers that should worry you

Vagueness about inference location. Training rights buried in general terms. No documented deletion process. Any of these turn into a problem during a security review, which is usually after you have signed.

7. Can the bot call our systems mid-conversation? Not "we integrate with Zapier" - can a step in the flow call an endpoint and use the response.
8. What happens to that response? If it cannot be stored and referenced later, the integration is one-way and much less useful.
9. Can we choose the AI model? And can we bring our own key, or is AI bundled into per-message pricing?
10. What happens when the model provider is down? Is there a fallback, or does the bot stop?
11. How does handover work when nobody is available? Out of hours is most hours.
12. Can one flow deploy to several channels? Or is each channel a separate build to maintain?
13. What are the actual limits? Conversations, bots, seats, API calls. Ask for numbers, and check them against the plan you intend to buy rather than the enterprise tier.

Question seven separates platforms more than any other. A vendor whose only integration story is a third-party automation tool is adding a hop, a cost and another party to your data path.

Rarely asked, and the answers are unusually revealing because the work is either done or obviously not.

14. Can we see your VPAT or accessibility conformance report? A vendor who has tested has a document. "We're fully compliant" with nothing attached means nobody tested.
15. Does the message list use an ARIA live region? You can verify this in thirty seconds in dev tools on their own demo. It is the single best proxy for whether anyone considered it.
16. Can the widget be operated entirely by keyboard, including closing it? Ask for a demonstration rather than a yes.
17. Can we change the colours? A fixed launcher colour that fails contrast is a defect you cannot fix after buying.

Accessibility obligations attach to the business deploying the widget, not the vendor supplying it, so an inaccessible widget becomes your liability. The accessibility guide covers what to test.

Ask these before signing, when you have leverage, rather than in eighteen months when you do not.

18. What exactly can we export? Flows, transcripts, contacts, analytics - and in what format.
19. Is there a notice period or minimum term? Monthly plans that quietly annualise are common.
20. What happens to the widget on the day we cancel? A dead script on production pages is a real incident.
21. Do you charge for support, migration or onboarding?

The flow portability question

Conversation flows almost never export usefully between platforms - the node types differ. Assume a rebuild, and weight ease of building accordingly. A platform that takes three weeks to build in is a platform you are stuck with. Our migration guide covers what actually transfers.

22. What counts as a billable unit? A conversation, a message, a resolution and a session are four different things, and vendors are not consistent. A platform billing per message is far more expensive for a chatty bot than the headline suggests.
23. What happens at the limit? Does it stop, queue, or overage-bill? Silent overage is how a predictable cost becomes a surprise.
24. Is AI usage bundled or separate? Bundled looks simpler and means a cheaper model saves the vendor money, not you.

Build the comparison on your own numbers

Take your actual monthly conversation volume, your average messages per conversation, and your seat count, and ask each vendor to price that. List prices are built around different assumptions and are not comparable.

For reference on how session-based counting differs from per-message billing, the pricing page sets out one model plainly, and the free plan lets you measure your real conversation shape before committing to anyone.

Twenty-four questions produces a lot of text and no decision. Weight them, because they are not equally consequential.

Treat the first category as a gate

If inference happens somewhere your policy forbids, or training rights are ambiguous, stop. Continuing to evaluate a vendor that fails a hard constraint wastes weeks and creates internal momentum toward a choice legal will later block.

Watch how they answer, not only what they answer

A vendor who responds in writing within two days with specifics is showing you what support will feel like. One who takes two weeks and answers a different question is also showing you. The evaluation process is a sample of the relationship, and it is the only sample you get before signing.

Descriptions converge; demonstrations diverge. Ask for these five specifically, and watch rather than listen.

1. Build a branch live

Not a pre-built flow - ask them to add a question with three options and a condition while you watch. You are measuring how long an ordinary change takes, because that is what your team will do weekly forever.

2. Call an external API and use the response

The single most revealing request. Ask them to call any public endpoint, store the response, and reference it two nodes later. Vendors whose integration story is a third-party automation tool cannot do this cleanly.

3. Trigger a handover with nobody available

Out of hours is most hours. Watch what the customer sees.

4. Operate the widget with the keyboard only

Open, type, send, close. Then ask them to turn on a screen reader and send a message. This takes ninety seconds and tells you more than a VPAT.

5. Export everything

Ask to see the export file, not a description of it. A CSV of message text without timestamps, speaker or conversation IDs is not a usable export, and you will only discover that on the way out.

Some responses are more informative than any feature comparison.

"We're fully compliant" with nothing attached

Applies to accessibility, security and data protection equally. Compliance produces documents. A claim without one means nobody has been through the criteria.

Vagueness about where inference happens

Every vendor knows which model provider they call. Not answering promptly means either they have not thought about it or the answer is inconvenient. See where chatbot conversations actually go for what to ask instead.

"That's on the roadmap"

Fine for a nice-to-have, disqualifying for anything you need in the first quarter. Ask for a date in writing, and assume anything without one will not arrive.

Pricing that requires a call

Sometimes legitimate for enterprise. Often it means pricing is set by what they think you will pay. Either way, get the billable unit defined in writing before any call.

Unlimited anything

Unlimited conversations, unlimited API calls, unlimited seats. There is always a fair-use clause. Ask what the actual number is and what happens when you reach it.

Evaluation does not end at signature. Three things are worth doing early while you still have the vendor's attention.

Verify the claims that mattered

Check that inference is where they said, that the export works, and that the widget behaves as demonstrated. Discrepancies are far easier to raise in month one than month ten.

Establish the ownership

Name who reads transcripts, who owns the content, who fixes an integration, and who picks up a handover. A platform decision does not survive an unowned deployment - who should own your chatbot covers the roles and the realistic weekly hours.

Set the model decision separately

If the platform supports several providers, the model is a decision you can revisit independently of the vendor. Run the twenty-question test rather than accepting the default - choosing an LLM covers it, and the provider list sets out the options by jurisdiction.

Start narrow

One high-volume question, answered well, measured for a month. Expanding from evidence beats launching broad and discovering the gaps in public. The template library shortens the build, the visual builder makes the gaps visible, and the free plan is enough to run the pilot before you commit spend. If the evaluation concludes you should not build at all, that case is here.

Worth settling before the shortlist, because the answer changes which questions matter.

What teams underestimate about building

The model call is a weekend. The widget, conversation state, retrieval, the agent inbox, handover routing, analytics, rate limiting and accessibility are the year. Teams who have built one describe the same experience: the chat loop worked immediately and everything around it consumed the schedule.

Building makes sense where the chatbot is the product, or where a hard constraint rules out every vendor. It rarely makes sense because a platform looked expensive, since the comparison being made is usually licence cost against engineering time that was never costed.

The middle option

A platform that lets you bring your own model key and call your own APIs gives most of the control people want to build for, without the year of surrounding work. That is the combination worth checking for during evaluation, and it is why the integration question above is weighted so heavily.

Four clauses decide more than the feature list, and all four are easier to negotiate before signature.

Price protection

Ask what happens at renewal. Uncapped increases are common and are the mechanism by which a cheap first year becomes an expensive third. A cap of a stated percentage is a reasonable ask and often granted.

Data handling in the agreement, not the marketing

"We do not train on customer data" belongs in the DPA. So does the sub-processor list, notice of changes to it, and breach notification timelines. A statement on a website can change without telling you.

Exit assistance

A clause covering export format and a defined period of assistance after termination. Without it, leaving depends on goodwill at precisely the moment there is none.

Service levels that mean something

Uptime percentages are easy to publish. Ask what the remedy is, because an SLA with no credit attached is a statement of intent. Ask also whether it covers the AI features or only the platform - these are sometimes scoped differently, and the model provider's availability is not usually covered at all.

None of this requires a legal department to raise. Asking a vendor to confirm these four in writing is normal, and how they respond is itself part of the evaluation. The data questions sit alongside the wider trail covered in where chatbot conversations actually go.

Every question above is answered better by two weeks of real use than by a sales call. Most platforms have a free tier adequate for this, and a pilot changes what you are negotiating about.

What a two-week pilot should establish

The number worth getting

Average messages per conversation, from your own traffic. It is the input that makes vendor pricing comparable, and nobody can tell you what it is in advance - it depends on your customers and your flow. A platform billing per message and one billing per session look similar on a rate card and diverge by a factor of several once you have this number.

Pilot with one question, not the whole plan

Pick the highest-volume question you get and answer it properly. It exercises everything - content, retrieval, handover, analytics - without the build cost of a full deployment, and it produces a real measurement rather than a demo impression. The template library shortens the build, and the free plan is adequate for the whole exercise.

What a pilot changes about the negotiation

You stop asking whether a feature exists and start asking why something took twenty minutes that should have taken two. Vendors respond differently to specific operational feedback than to a feature checklist, and you will have found the two or three things that genuinely differ between your shortlist.