All the news that's fit to malfunction
This article contains affiliate links. If you buy something through them, we may earn a small commission, at no extra cost to you. It does not influence what we write, which is more than can be said for most chatbots.
Customer Service2 min read
SAMPLE: Why chatbots invent things, whether a chatbot can make promises for a company, and what to do about it. A placeholder post for testing the blog page.
By Richard U. RossumPublished Updated
This is a sample post. Chatbots make things up because they are built to produce likely-sounding text, not to check facts. The sensible response is to treat their answers as a first draft and keep a copy of anything they promise.
This is a sample blog post. It exists to demonstrate the layout. Its examples are fictional, though the general explanation is accurate.
Chatbots make things up. They do so politely, fluently and at all hours. This post explains why, and what a sensible person can do about it.
Because they are built to produce likely-sounding text, not to check facts. A language model learns patterns from a very large amount of writing, then predicts which words usually come next. When it has no reliable information, it still predicts, and the prediction can be entirely invented.1
In practice, often yes. When a chatbot appears on a company's own website, what it says has tended to be treated as the company speaking. The chatbot is not a separate person with opinions of its own, however much it may sound like one.
Enjoying this? The Rossum Report delivers it weekly.
Usually you cannot, which is the difficulty. A guess and a fact arrive in the same confident tone. Some warning signs:
A chatbot that is wrong and a chatbot that is right sound exactly alike. That is the whole problem, in two sentences.
Treat a chatbot's answer as a first draft, not a final word. Check anything important against the company's written policy, and keep a copy of the conversation. For further reading, there is a sample book on the subject, which is where an affiliate link would go.
| Situation | Sensible response |
|---|---|
| It promises a refund | Save the conversation |
| It quotes a law | Look up the law |
| It seems very sure | Be slightly less sure |
Researchers call this "hallucination." The machines have not been asked for their view. ↩
Often, yes. When the chatbot is on the company's own website, what it says is generally treated as the company speaking.
Not on confidence alone. A chatbot sounds equally sure when it is right and when it is wrong.
The Footage
Once a week, a calm and thoroughly sourced account of the machines' latest lapses in judgment. Written by a correspondent who has waded through all the tedious details so that you, quite sensibly, don't have to.
Free. No spam. You may unsubscribe at any time, and we shall take it with quiet dignity. Mostly.
Thoughts? Tips? A bot of your own that has misbehaved? Reply to any issue of The Rossum Report, or file a report with the appropriate department.
Customer Service
Regrettability Index 3 · Rather Unfortunate