Development

AI in customer support: where it helps and where it hurts

A language model answers confidently even when it's wrong. Where AI is useful in support, what to prepare in advance and what it mustn't be trusted with.

The idea of replacing support with a language model seems obvious: questions repeat, the answers are in documents, and the model can talk coherently. In practice there really are benefits, but they don't lie quite where people look for them, and the main risk is built in such a way that it's invisible in a demo. Below is what it looks like up close.

Two ways to use it, and they're not equal

AI as the first line. The model answers the client directly. Fast, round the clock, it takes on the flow of identical questions. And that's where all the risk is: every answer goes out, and nobody has checked it.

AI as an operator's assistant. The model reads the request, finds relevant pieces in the knowledge base and suggests a draft reply. The operator edits and sends it. The time savings are smaller than in the first option, but a mistake doesn't leave the company.

The second option is almost always worth launching first. It gives you the same foundation — a knowledge base, search over it, integration with the ticketing system — while collecting statistics: how often a draft works without edits, where the model is wrong, what data it lacks. After a couple of months of this, you can see which topics can go autonomous and which can't. Launching straight to clients deprives you of this information exactly when you need it most.

The main risk: confident and wrong

A language model produces plausible text. Plausibility and truth coincide often, but not always, and you can't tell them apart by the form of the answer: a wrong answer looks exactly like a right one — just as calm, in the same tone.

For support, this means specific things:

  • The model may state a condition you don't have: a return period, a delivery option, a product capability.
  • It may assemble an answer from two correct fragments and reach a wrong conclusion — the hardest error to catch.
  • It will readily agree with a client's incorrect statement if the client asserts it confidently.
  • It doesn't know what changed on your side yesterday unless it has been told.

A client takes a promise made by a bot as a promise from the company. Arguing with that is pointless, and in human terms they're right.

The practical conclusion: reduce the risk not by persuading the model to be more careful but by how the system is built. An answer is built only from fragments found in your knowledge base; if nothing relevant is found, there's no answer — there's a hand-off to a person. It's noticeably duller than free dialogue, and noticeably safer.

What to prepare before launch

Answer quality is determined not by the choice of model but by what it answers from. Preparing the knowledge base is the main work of the project, and clients usually underestimate it.

  1. Gather the sources. Policies, past support replies, product descriptions, terms, instructions. Everything actually in use, not everything that was once written.
  2. Clean out contradictions. If two documents have different return terms, the model will pick one, and not necessarily the right one. Contradictions must be removed before, not after.
  3. Split into self-contained pieces. A fragment should be understandable without the rest of the document. References like "see the section above" don't work when searching by pieces.
  4. Assign an owner. Each section of the knowledge base has a person who updates it when things change. A knowledge base without an owner goes stale within a few months and starts doing harm.
  5. Separate public from internal. Internal comments, purchase prices, instructions for VIP exceptions mustn't end up in the set a client's answer is built from.
  6. Connect live data separately. Order status, account balance, appointment date aren't a knowledge base but a query to a system. The model mustn't guess such things; it must call a function and relay the result.

The sixth point usually determines usefulness. AI that only knows texts answers general questions. AI that can look up a specific order resolves real requests.

Hand-off to a human is a mandatory part

It's not an emergency exit but a standard element of the scenario. It's designed in advance, with rules.

The conversation should be handed to a person:

  • when there's no relevant fragment in the knowledge base;
  • when the client asks — the first time, with no attempts to hold on to them;
  • when it's about money: a refund, recalculation, compensation, a payment dispute;
  • when the message shows noticeable irritation or mentions a complaint;
  • when the question concerns health, law or safety;
  • when the client asks for the third time — which means the answers didn't help.

On hand-off, the operator should receive the entire conversation, not a "client is waiting" notification. Having to repeat the story is more irritating than the bot itself.

A note on out-of-hours. If no operator is available right now, it's more honest to say so and log the request with a clear response time than to send the person round in circles of automated phrases.

What a model mustn't be trusted with

Some decisions aren't handed to automation, even if it's technically possible.

  • Financial operations. Refunds, discounts, charges, plan changes. The model can prepare a request; a person makes the decision.
  • Legally significant wording. Answers about the contract, warranty obligations, personal data. Here you need approved text, not a paraphrase.
  • Medical and legal advice. The answer should be a route to a specialist, not advice.
  • Individual exceptions. "Fine, we'll make an exception for you" is a management decision.
  • Handling personal data without verifying identity. Before saying anything about an order, the system must know it's talking to its owner. That's a separate mechanism, not a guess based on the name in the chat.

This also covers where request data goes. If an external service is used, you need to understand exactly what's sent out, and clean unnecessary details out of conversations in advance. A related topic is covered in the security section.

How to measure whether it works

A feeling that "things got better" is unreliable. Look at clear signs, each of which can be collected from day one:

  • the share of requests closed without an operator — but only together with the next point;
  • the share of people who contacted you again about the same issue after an automated answer: if it's growing, the bot isn't resolving, it's postponing;
  • how many drafts the operator sends without edits and how many they rewrite completely;
  • the topics where hand-off to a person happens most often — a hint about which knowledge base section to expand;
  • as a separate list — cases where the bot answered wrongly. These need to be reviewed one by one, not counted as a percentage.

The last point matters more than the rest. One wrong answer about return terms costs more than a hundred saved operator minutes.

A practical sequence

  1. Start with operator assistance, not autonomous answers to clients.
  2. Gather and clean the knowledge base. This is the main part of the work; budget time from people inside the company, not just developers.
  3. Configure answers strictly from found fragments, refusing to answer when there's no source.
  4. Connect live data through calls to your systems, not through paraphrase.
  5. Describe the rules for hand-off to an operator and check that the whole conversation is passed on.
  6. Draw up a list of topics banned from automated answers and test it against real requests.
  7. After a few weeks of operation, switch to autonomous mode only the topics where drafts went out without edits.

This sequence is slower than "hooking a model up to the chat in a week", but it doesn't create commitments you'll later have to honour. If you'd like to go through your situation in detail — write to us.