Chatbot vs Conversational AI: You Own Every Sentence

A chatbot picks its reply from a set of sentences a human already wrote. Conversational AI writes the sentence at the moment it answers. That is the whole diffe

A chatbot picks its reply from a set of sentences a human already wrote. Conversational AI writes the sentence at the moment it answers. That is the whole difference in chatbot vs conversational AI, and every cost, risk and compliance consequence falls out of it.

Our position is that the usual framing — rigid versus adaptive, keywords versus understanding — describes the machinery and hides the decision. The thing you are actually choosing is the size of your answer set: whether the complete list of things your company might say in that channel can be printed and reviewed, or whether it gets composed live and you find out afterwards. A Hacker News commenter posting as DragonStrength described the failure mode exactly in July 2025: an Amazon support flow routed them to a bot that "hallucinated a 1-800 number to call which was most definitely not Amazon." Nobody at Amazon wrote that number. It was still Amazon's answer.

The short answer: If you cannot enumerate every acceptable reply before launch, you are not buying a chatbot, you are publishing an open-ended statement channel, so choose conversational AI only where you can afford to be bound by a sentence nobody reviewed, and keep the script wherever a wrong sentence costs money, rights, or a regulator's attention.

Last updated: July 30, 2026.

Scripted flow, intent-based conversational AI and generative agent compared by answer set, authorship and provability

The three answer sets, and the test that separates them.

Chatbot vs conversational AI: the answer set is the difference

A rule-based chatbot has a closed answer set: a finite list of predefined replies, each written and approved by a person, selected by if-then logic, a decision tree or a keyword match. Conversational AI has an open input space and, when it is generative, an open answer set too. The reply is composed at request time and did not exist before the customer asked. Everything else people list as a difference in customer support content is downstream of that.

This is not a semantic quibble. The United States Consumer Financial Protection Bureau drew the same line in its June 2023 issue spotlight, defining the simpler category as chatbots that "use either decision tree logic or a database of keywords to trigger preset, limited responses," and treating machine-learning and large-language-model systems as a separate class with separate failure modes (CFPB, Chatbots in consumer finance). The same report found that in 2022 over 98 million users, roughly 37% of the US population, engaged with a bank's chatbot, and that all of the ten largest US commercial banks were running one.

Hold those two facts side by side and the decision changes shape. A third of a country has already talked to one of these things, mostly in a regulated context, and the regulator's concern was never that the bots were insufficiently conversational. It was that a scripted system might fail to notice a customer asserting a legal right, and that a less constrained one might say something untrue. Those are two different defects with two different fixes, and choosing between chatbot vs conversational AI is choosing which defect you would rather manage.

We have not run a controlled deployment of both categories on the same support queue, so nothing here is a first-hand test result. Every number below comes from a named source we fetched, and where the sources disagree or run out we say so.

Three technologies, two names

The market sells two labels over three distinct technologies, and the middle one is where most real deployments actually sit. Naming all three is the fastest way to stop arguing about definitions and start pricing the decision.

Scripted flows. A directed graph of states. The customer's input selects an edge — by button, by keyword, by regular expression — and each state emits a reply from a file. Authorship is total: someone wrote every sentence, and you can diff the file. It fails by refusing, which is annoying and visible.

Intent-based conversational AI. Natural language processing turns free text into structure, a classifier maps that structure onto a fixed list of intents, entities get extracted, slots get filled, and the system emits a reply from a template a human wrote. This is what Amazon Lex, older Dialogflow projects and Rasa deployments do, and intent recognition is the whole trick. The input space is open; the answer set is still closed. It fails by misrouting: it confidently picks the wrong intent and gives you a correct answer to a question you did not ask.

Generative conversational AI. A language model produces the reply text, usually grounded in retrieved documents, sometimes with tools attached. Nobody wrote the sentence. The failure mode is invention: the United States National Institute of Standards and Technology calls it confabulation, "the production of confidently stated but erroneous or false content," and lists it as one of twelve risks specific to generative systems in its Generative AI Profile (NIST AI 600-1, July 2024).

There is a fourth configuration that resists the taxonomy and deserves naming, because it is quietly the best answer for a lot of help centres: semantic retrieval with an authored reply. Vector search finds the right article, and the system returns that article's text verbatim. The matching is modern; the answer set is still a closed list of things a human wrote. If your problem is that customers cannot find the right page, this is the cheapest fix available and it is not generation.

Vendors blur tiers two and three under one banner, which is why buyers end up comparing a decision tree with a language model and concluding, reasonably, that the language model is better at conversation. It is. The relevant question is whether you wanted better conversation or a bounded set of company statements.

For a plain walk through the vocabulary before you take that apart, this explainer from IBM is six minutes and stays honest about where each tier stops:

Play video

What neither category is

Neither a rule-based chatbot nor conversational AI is, by itself, an agent. Both categories in their standard form read and speak; they do not change the state of your business systems. The moment a conversational system is given permission to issue a refund, reset a password or cancel a subscription, it has crossed into different territory with a different control problem, which we worked through separately in our earlier analysis of the write-access line between AI agents and chatbots.

That distinction matters here because it isolates what this article is about. This is the read-only case: the system's only output is language. And the point people miss is that a read-only system is not a low-risk system. It can still commit you to a price, misstate a policy, tell a customer their legal rights do not exist, or repeat something confidential it was shown earlier in the session. It causes all of that damage without touching a single record.

Nor is conversational AI a synonym for a large language model. Natural language processing, speech recognition, dialogue management and text-to-speech were conversational AI for a decade before anyone shipped a transformer to production, and plenty of the systems marketed as conversational AI or as a virtual assistant in 2026 are still an intent classifier with a template library behind it. Ask which tier you are buying and ask to see the reply text.

The Answer-Set Test: three questions, one sitting

The Answer-Set Test is our name for a three-question diagnostic you can finish in an afternoon with a support lead and a compliance contact in the room. It replaces the usual "is your use case complex?" conversation, which nobody can answer honestly, with three questions that have checkable answers.

Question one: can you enumerate every acceptable reply before launch? Take the last 200 real conversations in the channel. Ask whether the correct outcome for each one is a sentence you could write today and put in a file. If most of them collapse onto a short list of replies, the answer set is closed and a script is not a compromise, it is the correct engineering choice. Our working threshold is about 80%, which is a rule of thumb from reading support distributions rather than a figure from any study. The point is the shape of the curve, not the exact cut-off. If the tail is genuinely long and each conversation needs a different sentence, no amount of flow-building will cover it.

Question two: if it says something nobody wrote, are you bound by it? This is a question for legal, not for engineering, and the honest answer in most consumer-facing channels is yes. Work through the three worst sentences a plausible model could emit in that channel: a discount you do not offer, a policy that does not exist, a deadline that has passed — and price each one. If the price is a apology email, proceed. If the price is a refund obligation or a regulatory finding, you need a closed answer set at least for those topics.

Question three: six months from now, can you show exactly what it said? Not "do we have logs". Can you produce, for a specific customer on a specific date, the exact text the system emitted, what it was grounded in, and which version of the prompt or flow was live? If the answer is no, you cannot defend the channel, and the difference between the two categories becomes moot because you will lose the argument either way.

Score it plainly. Three yeses means conversational AI is a reasonable risk. Two noes and one unknown means pick the script and revisit in two quarters. The mixed cases are where hybrids belong, and we cover the assembly further down.

You own the sentence: what Moffatt v. Air Canada settled

If you take one legal fact into the chatbot vs conversational AI decision, take this one: a tribunal has already held a company liable for what its chatbot said, and rejected the argument that the bot was somehow a separate actor. In Moffatt v. Air Canada, 2024 BCCRT 149, issued 14 February 2024, British Columbia's Civil Resolution Tribunal ordered the airline to pay $812.02, made up of $650.88 in damages, $36.14 in pre-judgment interest and $125 in tribunal fees (full decision).

The facts are ordinary, which is what makes them useful. Jake Moffatt, booking travel after a grandparent's death in November 2022, asked Air Canada's chatbot about bereavement fares. The bot said a ticket could be submitted for a reduced bereavement rate within 90 days of issuance. A separate page on the same website said the policy did not apply after travel was completed. Moffatt relied on the bot, flew, applied afterwards and was refused.

Air Canada's defence is the part every deployment team should read. It argued it could not be liable for information given by its own chatbot. Tribunal Member Christopher C. Rivers called that "a remarkable submission" and held that while a chatbot has an interactive component, it "is still just a part of Air Canada's website." The tribunal added that the airline never explained why the static page should be treated as more trustworthy than the bot, or why a customer should have to cross-check one part of a website against another.

Three consequences follow, and they apply to the scripted tier as much as the generative one.

ConsequenceScripted flowIntent-basedGenerative
A wrong sentence binds youYes, but you wrote itYes, and you wrote itYes, and nobody wrote it
Root cause is findableDiff the flow fileInspect intent + confidenceOnly if you retained prompt, context and output
Fix is deployable same dayYesUsuallySometimes; the model is not a config file

Note what the Air Canada bot was. Based on the decision, this was not a frontier language model improvising; the tribunal recorded that the response even carried a hyperlink to the correct policy page. The liability came from an inconsistency between two authored surfaces. A scripted answer set does not protect you from being wrong. It protects you from being wrong in a way you cannot locate.

The generative version of the same problem arrived in April 2025 at Cursor, the AI coding tool, whose support bot told users that logging in from multiple machines was disallowed. It was not a policy. Co-founder Michael Truell replied publicly that "We have no such policy" and attributed the message to "an incorrect response from a front-line AI support bot," as reported by The Register on 18 April 2025. The user who raised it was refunded, and the company's remedy is the one worth noting: Truell said that AI responses used for email support "are now clearly labeled as such." Nobody wrote the invented rule; the company still owned it, corrected it in public, and changed its disclosure practice as a result.

What each option costs per conversation, at July 2026 list prices

Most comparisons of chatbot vs conversational AI are qualitative, adaptive versus rigid, learns versus does not, and they skip the number a buyer needs. So here are real list prices, fetched from vendor pricing pages on 30 July 2026, expressed in the unit each vendor bills in.

OptionVendor unitList priceSource
Intent-based, textPer text request$0.00075Amazon Lex pricing
Intent-based, speechPer speech request$0.004Amazon Lex pricing
Designed flows, chatPer request$0.007Google Conversational Agents pricing
Generative playbooks, chatPer request$0.012Google Conversational Agents pricing
Designed flows, voicePer second of audio$0.001Google Conversational Agents pricing
Generative playbooks, voicePer second of audio$0.002Google Conversational Agents pricing
Outcome-priced AI agentPer resolved outcome$0.99Fin pricing, Intercom
Outcome-priced qualificationPer qualification$9.99Fin pricing, Intercom

Two things in that table are worth more than the rest of this section.

The first is Google's own price gap. On the same product, the designed-flow agent type costs $0.007 per chat request and the generative playbook type costs $0.012 — a 1.7× premium for an open answer set, priced by the vendor itself. Be precise about what that compares: these are billed agent types, and a designed flow can itself invoke generative steps, so the gap is not a clean deterministic-versus-generative line. It is still the cleanest available market signal that these are different goods rather than different quality levels of the same good.

The second is the unit change. Amazon and Google bill per request; Intercom bills per outcome. At $0.99 per resolution, a conversation that takes 12 turns to resolve costs the same as one that takes three, while the same conversation on Google's generative playbooks costs about $0.14 in request charges. Outcome pricing is not a markup, it is a different risk allocation: the vendor absorbs turn-count variance and you pay for results. Which is cheaper depends entirely on your turns-per-conversation, and almost nobody knows that number before they measure it.

A worked derivation, using only the fetched prices, for a queue of 50,000 chat conversations per month at an assumed 8 turns per conversation. The turn count is our assumption, clearly marked, not a measurement:

ConfigurationPer-conversation platform cost50,000/month
Intent-based text, 8 requests at $0.00075$0.006$300
Designed flows, 8 requests at $0.007$0.056$2,800
Generative playbooks, 8 requests at $0.012$0.096$4,800
Outcome-priced, 60% of conversations reach a billable outcome$0.594$29,700

The spread from the cheapest to the most expensive row is roughly 99×, on identical volume. Two of those numbers are ours, not the vendors': the eight turns and the 60% outcome rate. Change the turn assumption to four and the request-priced rows halve while the outcome row does not move at all; drop the outcome rate to 30% and the last row falls to $14,850. Treat the table as a sensitivity model you re-run with your own two numbers, not as a quote. That sensitivity is the reason the platform bill is the wrong thing to argue about first: the variance inside your own traffic pattern is larger than the gap between two vendors.

The other cost this table does not carry is the cost of being wrong. The Air Canada award was $812.02, which is nothing to an airline; the real expenditure was the defence, the process change and the fact that the case is now the standard citation in every article on this subject, including this one. Cursor's invented policy cost a refund, a public correction by a co-founder and a permanent change to how the company labels its support replies. Neither figure is knowable in advance, which is exactly why the Answer-Set Test asks you to price the three worst sentences before you launch rather than after.

And the platform bill is the small number anyway. Conversation design, intent curation, retrieval content maintenance, review of escalated transcripts and the human queue behind the bot all cost more than the requests. We priced the wider version of this problem in our analysis of what enterprise AI implementation actually costs beyond the licence. Every figure in this section is a platform list price and none of them is a total cost of ownership.

Conversational AI reliability: accuracy is the wrong number

Ask a vendor how accurate their conversational AI is and you get a single percentage. That number is close to meaningless for a support channel, because a support channel replays the same situation thousands of times and what you need to know is whether the system handles it the same way every time. The academic benchmark that measures this directly is τ-bench, from Sierra, which evaluates agents against simulated users and a documented policy in retail and airline domains.

Its headline result should be read slowly. In the original paper, state-of-the-art function-calling agents "succeed on <50% of the tasks" and are "quite inconsistent (pass^8 <25% in retail)" (Yao et al., τ-bench, arXiv, June 2024). The pass^8 metric asks whether the agent succeeds on all eight independent attempts at the same task. Fewer than a quarter of tasks survive that. An agent that is right 60% of the time on a single try can still be unreliable on the second identical request from the second identical customer, and support volume is nothing but identical requests.

The currency check matters here, because that paper is from 2024 and the field moved. The same lineage published τ²-bench in June 2025, extending the setup to a dual-control telecom domain where the user must also act — toggle a setting, read a code back — and reported "significant performance drops when agents shift from no-user to dual-control, highlighting the challenges of guiding users" (Barres et al., arXiv, June 2025). In other words, the harder and more realistic the collaboration, the worse the numbers get. Anyone quoting only the earlier figures is quoting the easier benchmark.

What you are toldWhat to ask instead
Accuracy on a test setConsistency across repeated identical requests
Containment rateContainment rate excluding conversations the customer abandoned
Customer satisfaction on resolved chatsSatisfaction on escalated chats, where the damage is
Intent coverageBehaviour on the 20% the classifier is least confident about

One caveat that cuts against our own use of this evidence, and it is a real one. τ-bench scores agents that call tools and change database state, which is harder than answering a question read-only. A generative support agent that only speaks is doing a narrower job, so the benchmark almost certainly overstates the difficulty of the case this article is about. What transfers is not the absolute score but the shape of the finding: single-attempt accuracy and repeated-attempt reliability are different quantities, and the second one is the one a support queue exercises. If you know of a public repeated-trial benchmark for read-only grounded answering at support scale, we have not found one, and its absence is a genuine gap in the evidence available to buyers.

None of this argues for the script. It argues for a specific discipline: measure repeatability, not brilliance, and treat any vendor number that is not repeated-trial as marketing.

What an open answer set leaks that a script cannot

A scripted chatbot cannot disclose something it was never given. That is not a security control anybody designed; it is a property of a closed answer set, and it disappears the moment the reply text is generated from a context window. It is also not absolute: a flow that injects account fields into a template will happily leak them if the session is bound to the wrong customer. The difference is that a scripted leak is a bug with a location, while a generated one is a probability. Two risks arrive together at tier three.

The first is prompt injection. The Open Worldwide Application Security Project defines it as occurring "when user prompts alter the LLM's behavior or output in unintended ways," notes that the inputs need not be human-visible, and states plainly that retrieval-augmented generation and fine-tuning "do not fully mitigate prompt injection vulnerabilities" (OWASP LLM01:2025). For a support channel, the practical version is a customer pasting text that makes the bot ignore its instructions, quote its system prompt, or repeat content retrieved for a different tenant.

The second is what the same project lists next: sensitive information disclosure. A generative support agent typically has retrieval access to a knowledge base, sometimes to account data, and always to the conversation so far. Anything in that context can appear in an answer, and the answer goes to whoever is in the chat window. This is why masking personal data before the prompt leaves your network is a different control from redacting it in the log afterwards, a distinction we set out in our approach to inline data-loss prevention at the gateway.

RiskScripted flowIntent-basedGenerative
Says something untrueOnly if authored wrongOnly if authored wrongStructural risk (confabulation)
Reveals retrieved content to the wrong userOnly via a flow or session bugRare, template-boundedLive risk
Manipulable by customer inputNot meaningfullyClassifier can be gamedPrompt injection applies
Leaks its own instructionsNo instructions to leakNoYes, unless controlled
Transcript contains customer personal dataYesYesYes, plus model-provider exposure

Read the last row carefully, because it is the one people forget. Every tier collects personal data in transcripts. What changes at tier three is that the transcript has usually been sent to a third-party model provider, which turns a retention question into a data-transfer question.

Chatbot disclosure and evidence: what changes on 2 August 2026

From 2 August 2026, if you serve people in the European Union, a conversational system generally has to tell them it is a machine. Article 50 of the EU AI Act places transparency duties on both providers and deployers: providers must design systems that interact directly with natural persons so that those persons are informed they are interacting with an AI system, and the European Commission's guidance states that people "must be notified when they are interacting with an AI system from the start of the first interaction in a clear and distinguishable manner" (European Commission FAQ on Article 50). If you bought the bot rather than built it, you are the deployer, and the duty does not stop at your vendor.

There is an exemption where the AI nature is obvious, assessed against an average person who is "reasonably well-informed, circumspect, and observant," and the Commission says it reads that exemption narrowly. Do not build a compliance position on it. The Commission published 40 pages of draft guidelines on 8 May 2026 and ran a stakeholder consultation to 3 June 2026 (consultation page), which tells you the detail is still being negotiated while the date is not. Breaches outside the prohibited-practices list sit in the AI Act's penalty tier of up to EUR 15 million or 3% of worldwide annual turnover (Article 99). Our guide to what deployers face under the AI Act works through the wider obligation set.

The disclosure duty is cheap to satisfy and it lands on both categories: a scripted bot is an AI system for these purposes too. The expensive obligation is the one nobody sends an invoice for, which is evidence. If a customer disputes what your system told them, the artefact that settles it is the transcript plus the grounding plus the version of the configuration that was live. For a scripted flow that is a file in version control. For a generative agent it is a retention decision somebody has to make deliberately, before the incident, which is the argument we made in our work on audit trails that actually prove what an agent did.

One honest gap: we could not independently verify the widely quoted contact-centre savings figures attributed to industry analysts, because those pages block automated retrieval, so they are excluded rather than repeated here.

Verdicts: which one your use case actually needs

Uniform verdicts on the chatbot vs conversational AI question, so you can find yours and stop reading. Each block assumes you have run the Answer-Set Test above.

Scripted flow — verdict: choose it for bounded transactions with a legal edge. Best for: order status, store hours, appointment slots, password-reset instructions, delivery windows, anything where the right answer is one of a dozen sentences. Strengths: the answer set is printable and reviewable; the cheapest tier to run; failure is visible and gets escalated; a wrong reply is a one-line fix. Weaknesses: brittle to phrasing; the CFPB's specific criticism applies, that inflexible scripts may only recognise a dispute when the customer uses the right words; customers resent dead ends. Choose it if: your top 20 intents cover more than three quarters of volume, or a wrong sentence has a price tag.

Intent-based conversational AI — verdict: the default for high-volume support, and the most under-sold tier. Best for: channels with wide phrasing variance but a stable set of outcomes: billing questions, tier-one IT, logistics enquiries. Strengths: open input, closed answer set, which is the combination most buyers actually want; per-request costs at the bottom of the market, $0.00075 per text request on Amazon Lex; confidence scores give you a natural escalation threshold; CRM integration is a solved pattern here because the reply templates take structured fields. Weaknesses: intent curation and training data upkeep are ongoing labour; misrouting is confident and quiet; multilingual coverage multiplies the maintenance; and this is the tier with the worst public reputation, because most of the bots people hate are badly-built versions of it. Choose it if: you need to understand anything a customer types but still want to approve everything the company says. Do not choose it if you are not also fixing escalation. An intent-based system with a slow route to a human reproduces exactly the experience that made customers distrust bots in the first place, and no amount of classifier accuracy compensates.

Generative conversational AI — verdict: right where the answer genuinely cannot be pre-written, and only with retention and grounding decided first. Best for: long-tail policy questions over a large document corpus, technical troubleshooting, summarising an account history for a human agent. Strengths: covers the tail no flow can enumerate; handles multi-part questions in one turn; drafts for human review rather than replacing the human. Weaknesses: confabulation is a structural property, not a bug to be patched; prompt injection is not fully mitigable per OWASP; consistency is weak under repeated trials per τ-bench; highest per-request price of the tiers we priced. Choose it if: your tail is real, your grounding corpus is maintained, and you can answer the six-months-later question today.

Hybrid — verdict: what most mature deployments converge on, and the only configuration that answers "both". Best for: any channel with a fat head and a long tail. Shape: deterministic flows in front for the enumerable head; a generative tier behind it for the tail; a hard escalation path to a human; and, critically, the generative tier restricted to informational answers while anything transactional stays in the flow. Weaknesses: two systems to maintain, two sets of analytics, and a routing layer that becomes its own source of bugs. Choose it if: the Answer-Set Test came back mixed, which for most support queues it does.

When the scripted chatbot still wins

The market's assumption in every chatbot vs conversational AI comparison is that scripted bots are legacy and conversational AI is the upgrade. There are situations where that is straightforwardly wrong, and they are more common than vendor content admits.

The strongest case is the regulated disclosure. When the correct reply is dictated by a regulator or by counsel, such as a rights notice, a mandated wording, a complaints route — an open answer set is a liability with no compensating benefit. You do not want fluency there. You want the exact sentence legal signed off, delivered identically every time, which is precisely what a closed answer set guarantees and what pass^8 numbers suggest generation does not.

The second case is thin evidence. If you cannot yet answer the third Answer-Set Test question, a script keeps you defensible while you build the retention path. Deploying the more capable system into an environment where you cannot reconstruct what it said is choosing the harder incident.

The third is cost at genuinely high volume with a narrow intent set. At 50,000 conversations a month the gap between an intent-based tier and an outcome-priced agent, on the fetched list prices above, was two orders of magnitude. If your traffic really is order-status lookups, paying resolution prices for them is a transfer of margin with no reader-visible benefit.

The fourth is the one customers voice loudest, and it cuts both ways. On Hacker News in May 2026 a commenter wrote that as a customer their "willingness to entertain AI chatbot decision trees is rock bottom" (comment). That is an argument against bad scripts, not for unbounded generation. The same frustration attaches to a fluent bot that cannot escalate. The variable customers actually respond to is time-to-human, and either tier can get that right or wrong.

Upgrading a chatbot flow without betting the channel

If the Answer-Set Test says you have a real tail, the migration question is not which platform. It is how to expose the new tier to production traffic without making the whole channel depend on it. A sequence that follows the same logic as the gateway rollout model we use elsewhere — observe first, enforce second, optimize third — is set out in our AI Gateway approach.

  1. Pick the single worst-performing intent, not the whole channel. Choose the one bucket where the flow's fallback rate is highest. That is where the tail lives and where an improvement is measurable against a baseline you already have.
  2. Run the new tier in shadow for two weeks. It answers nothing. It generates a candidate reply alongside the live flow, and the two get compared offline. This is the cheapest step and the one teams skip. If your platform has no shadow mode, and many do not, replay a week of exported transcripts against the new tier in a batch job instead. It is the same evidence, one week later.
  3. Have a human read 200 shadow replies against the live ones. Not a sample of ten. Count three things: how many candidate replies are better, how many are wrong, and how many contain a sentence legal would not have approved. The third count is the one that decides the rollout.
  4. Go live read-only, on the tail only. The generative tier answers informational questions. Anything that touches money, entitlement or personal data routes back to the flow or to a person. This keeps the open answer set out of the topics where a wrong sentence has a price.
  5. Set the escalation threshold before launch, in writing. Below a confidence or grounding threshold, hand off. Design the handoff so the customer does not repeat themselves; the transcript goes with them. Approval and escalation gates are controls only when they can actually say no, which we examined in our piece on when a human in the loop is real control.
  6. Retain the evidence from day one. Transcript, retrieved sources, prompt version, model and version, timestamp. Retro-fitting this after an incident is not possible, which is the whole reason it belongs in step six and not step twelve.
  7. Review the wrong-reply count weekly for the first quarter, then monthly. If the count is not falling, the problem is your grounding corpus, not your model choice.

On the elapsed time: steps one to three are about three weeks of calendar with a few days of actual work, step four is a deployment, and step five should already be written before step four ships. Teams that report six-month timelines for this are usually rebuilding the whole channel at the same time, which is a different project with a different risk profile.

Notice what the sequence does not include: replacing the flow. The flow keeps serving the head of the distribution, and it stays as the fallback when the new tier is unsure. The migration is additive.

Seven numbers to instrument before you switch

If you take nothing else operational from this, take the instrumentation list. None of these needs a new platform; all of them need somebody to define them before the vendor does.

NumberWhy it decides the chatbot vs conversational AI question
Turns per conversation, median and 90th percentileConverts per-request pricing into per-conversation cost; decides whether outcome pricing is cheaper
Fallback rate per intentLocates the tail. A flat fallback rate across intents means the problem is design, not coverage
Escalation rate, and time-to-humanThe variable customers actually judge you on
Repeat-contact rate within 48 hoursCatches the containment metric lying to you. A contained conversation the customer re-opens was not resolved
Wrong-reply count from human reviewThe only direct measure of confabulation you will get without a benchmark
Share of conversations whose full evidence you can reproduceAnswers Answer-Set Test question three with a number
Cost per resolved conversation, all-inThe only cost figure that compares tiers fairly

Two of these deserve a warning. Containment rate, sometimes sold as a deflection rate, is the metric most often quoted and most often gamed: a conversation the customer abandoned in frustration counts as contained, and so does one where the customer gave up on self-service and phoned instead. And the wrong-reply count only exists if somebody reads transcripts, which means it has a headcount attached, and that headcount is part of the cost of the generative tier and belongs in the business case.

One measurement that is worth more than any of the seven: read the escalated transcripts yourself for a week. Customer service teams learn more about which tier they need from twenty transcripts than from a quarter of dashboards, because the thing you are looking for is not a rate. It is whether the sentences the system produced were sentences your company would have chosen to say.

Where this analysis is uncertain

Several things in this piece are weaker than they look, and it is worth being explicit about which.

We have not run a controlled A/B of the three tiers on one support queue, so the comparative reliability claims rest on published benchmarks rather than on our own measurement. τ-bench and τ²-bench are simulated environments with simulated users; they are the best public evidence available and they are not your traffic. A vendor-tuned production system with a well-maintained retrieval corpus may do considerably better than a general model scored on a benchmark, and there is no public dataset that would let us quantify how much better.

The pricing table is list price on a single day, in the vendor's own unit, and vendors change these. Enterprise agreements routinely land far below list. The 50,000-conversation model uses our assumed 8 turns per conversation, which is an assumption and not a measurement; your own median will move every row in it.

The legal picture is jurisdictional and unsettled. Moffatt is a small-claims decision of one Canadian provincial tribunal, not a binding appellate precedent, and its persuasive weight outside British Columbia is a question for your counsel rather than for us. Article 50's obviousness exemption has not been tested, and the Commission's guidelines were still in draft when we wrote this. Nothing here is legal advice.

We could not verify the analyst savings estimates that circulate in this topic, because the publishers block automated retrieval, and Reddit was unreachable for problem discovery, so the practitioner voices here come from Hacker News and skew technical for a topic whose main audience is support and CX operations. Finally, we have deliberately excluded voice-specific concerns such as latency budgets, barge-in and call recording law, which change the calculation enough to deserve their own treatment.

The layer this decision implies

Once you have made the choice, a second problem appears that neither vendor sells you a fix for: the conversational system is now one of many AI surfaces in the company, and the questions that decide whether it stays safe are not conversational-design questions. They are who owns this system, which data it may reach, what it said, and what it cost. That layer is what Leapforce builds: one governed endpoint in front of every model, identity and policy applied per request, and an audit record of what ran and what was refused. Our gateway rollout follows the sequence used above: observe first, enforce second, optimize third. Per our published build status, gateway endpoints, tracing and single sign-on are live, while inline data-loss prevention and dollar budgets are in development, and we would rather say that than imply otherwise. Leapforce does not sell a support chatbot or a conversation-design tool, and if a conversational platform is what you need, buy one of those.

 FAQ

Frequently asked questions

Usually the reverse of how it is asked, and getting it backwards is why the chatbot vs conversational AI comparison confuses people. Conversational AI is the capability stack: language understanding, dialogue management, sometimes generation — and a chatbot is a product surface that may or may not use it. A button-driven decision tree is a chatbot with no conversational AI in it at all. A voice assistant is conversational AI that is not a chatbot. The useful question is not which category a product belongs to but whether its answer set is closed or open.

No, and conflating them is the most common error in the chatbot vs conversational AI literature. Conversational AI predates large language models by more than a decade: natural language processing, intent recognition, slot filling and template replies are all conversational AI with a closed answer set. Generative AI composes the reply text at request time. Many products sold as conversational AI in 2026 are still intent-based, which is often the better engineering choice. Ask the vendor to show you where the reply text comes from.

Run the Answer-Set Test on real traffic. Pull the last 200 conversations from the channel, and for each one write down the reply that would have been correct. Then count how many distinct replies you wrote. If 200 conversations collapse onto 25 replies, your answer set is closed and a script or an intent-based system will serve it. If you wrote 150 different replies, the tail is real. This takes an afternoon and costs nothing, and it produces the fallback-rate baseline you will need later.

On list prices fetched 30 July 2026, the same vendor charges $0.007 per chat request for designed flows and $0.012 for generative playbooks on Google Conversational Agents, a 1.7× premium. Amazon Lex charges $0.00075 per text request for intent-based handling. Intercom's Fin charges $0.99 per resolved outcome. At an assumed eight turns per conversation, that is roughly $0.006 to $0.10 per conversation on request-priced platforms against $0.99 per resolution on outcome pricing. Platform cost is the small line; conversation design, retrieval-content maintenance and transcript review cost more.

A tribunal has already said yes. In Moffatt v. Air Canada, 2024 BCCRT 149, British Columbia's Civil Resolution Tribunal rejected the argument that a chatbot is a separate legal actor, holding it "is still just a part of Air Canada's website," and ordered the airline to pay $812.02. The tribunal also refused to accept that a customer should have to cross-check the bot against another page of the same site. This is a provincial small-claims decision rather than binding appellate authority, so ask your counsel how it reads in your jurisdiction. Design as though every sentence is a company statement.

In the European Union, from 2 August 2026, generally yes. Article 50 of the AI Act requires that people interacting directly with an AI system be informed of it, and the European Commission says notification must come at the start of the first interaction in a clear and distinguishable manner. There is a narrow exemption where it is obvious to a reasonably well-informed person, which the Commission interprets tightly. The obligation applies to scripted bots as well as generative ones, and the Commission's implementing guidelines were still in draft consultation as of June 2026.

Yes, and it is what most mature deployments end up doing. The pattern that works is deterministic flows in front for the enumerable head of your traffic, a generative or intent-based tier behind for the tail, and a hard escalation path to a human. The design decision that matters is not the routing, it is the restriction: keep the open answer set on informational topics and keep anything touching money, entitlement or personal data inside the authored flow. That way the tail gets covered without putting an unreviewed sentence in front of a transaction.

A scripted flow needs a conversation designer and someone who can maintain content; no machine learning skills at all. An intent-based system adds intent curation and annotation, which is ongoing linguistic work rather than engineering, plus threshold tuning. A generative tier adds retrieval-corpus maintenance, prompt and grounding evaluation, an incident process for wrong replies, and a data-protection review of what leaves your network. The last of those is the one most teams under-resource, because it looks like a one-off approval and behaves like a standing obligation.

Not on containment rate alone, because an abandoned conversation counts as contained. Pair it with repeat-contact rate inside 48 hours, escalation rate, time-to-human, and a human-reviewed wrong-reply count. Add cost per resolved conversation all-in, including the reviewer time. If containment rises while repeat contacts also rise, the system is deflecting rather than resolving and you have made the channel worse while the dashboard says otherwise.

No. Per-request pricing starts below a tenth of a cent and outcome pricing has no floor commitment beyond a plan seat, so the platform cost is accessible at almost any size. What does scale with size is the governance work: disclosure, retention, data-protection review and transcript audit are broadly the same effort for 5,000 conversations a month as for 500,000. Small teams should therefore be more selective about which topics get an open answer set, not less.

Shadow mode on one intent. Pick the single flow with the highest fallback rate, run the new tier alongside it generating candidate replies that nobody sees, and have a human compare 200 pairs. You get a wrong-reply count and a better-reply count against a live baseline for the cost of two weeks of requests and a few hours of review. It is a smaller commitment than a pilot and it produces the number that actually decides the rollout, which a pilot usually does not.

There is no single answer, and that is the point: retention is set by your sector's recordkeeping rules, your privacy notice and your dispute window, not by the chat vendor's default. Decide it before launch and write it down, because the evidence you need in a dispute is the transcript plus what the reply was grounded in plus which configuration version was live. A scripted flow gives you most of that from version control. A generative tier gives you only what you deliberately retained.

Ready to Govern Your AI?

Talk to LeapForce — one controlled layer for every AI tool, connector, model, and agent.

Thirty minutes · No pitch deck

Ready to turn AI experiments into measurable ROI?

Bring one outcome you'd like AI to move. We'll help you scope a pilot you can actually measure — and tell you honestly if it's not worth doing yet.

Comments