Test a website chatbot with questions it should answer, questions it should refuse, and questions your site answers badly. A safe launch means correct published information is used, missing information is not invented, and the visitor has a clear route to a person.
Test a website chatbot with questions it should answer, questions it should refuse, and questions your site answers badly. A safe launch means correct published information is used, missing information is not invented, and the visitor has a clear route to a person.
Do not begin by asking only the easy questions from your homepage. Those prove very little. Build a short test sheet from real customer situations, then record what the chatbot says and what your website actually says beside it.
Which questions should the chatbot answer before launch?
Start with ten questions whose answers are clear and current on your site. Use the wording a visitor would use, not the wording in your navigation.
Include a price, opening hours, service area, what a service includes, a cancellation rule, and the next available step. For each question, note the page, FAQ, or document that contains the answer. Chatterbox builds its knowledge by crawling a site's pages, FAQs, and documents, so the published material is the standard you test against.
Ask each question in two or three ordinary ways. “Do you cover Baner?” and “Can you come to my address in Baner?” should be judged against the same published service-area statement. You are checking whether the answer stays faithful to the material, not whether every reply uses identical words.
What should the chatbot refuse to answer?
Now write five questions your website does not answer. Try an unpublished discount, an exception to your cancellation policy, a location outside the listed area, a custom bundle, and a promise about timing that no page makes.
The right result is not a creative answer. Chatterbox says it cannot find the answer rather than inventing one. It then asks the visitor for contact details so a person can handle the question.
Treat an unsupported number, policy, or promise as a launch blocker. A polished guess is still a guess, and the customer may reasonably hear it as your business speaking.
Where can your own website create a wrong answer?
Test the weak spots in the source material. Ask about a price that appears on several location pages. Check an old FAQ against a newer service page. Look for two descriptions of the same package that include different things.
This matters because one real crawl read more than 1,000 pages and produced over 5,400 question-and-answer pairs. Most were near-duplicates caused by the same pricing language appearing across many location pages. Repetition can hide the one sentence that is no longer true.
When a test exposes a contradiction, fix the website first. Asking the chatbot again without correcting the source only repeats the test against the same problem.
Does the fallback leave a useful next step?
Run the refusal tests when nobody is available to reply immediately. Chatterbox can answer outside business hours unattended, but an unanswered question still needs a human next step.
Check that the fallback admits the gap plainly and asks for contact details. Do not judge it by how apologetic or conversational it sounds. Judge it by whether the visitor knows that no answer was found and what to do next.
Use a phone number you control while testing. Confirm only what appears in the conversation: the chatbot requested the details instead of extending a published answer into a situation it did not cover. Do not promise a callback time in your test copy unless your team can keep it.
When is the website chatbot ready to go live?
Launch when the clear questions match the published material, the missing questions produce a refusal, and every contradiction you found has been corrected at its source. Keep the test sheet. Run it again whenever prices, hours, coverage, or policies change.
If an agency is testing several client sites, repeat the sheet for each business. Chatterbox keeps each client's knowledge base isolated from every other client's, but every site still needs its own current questions and answers.
A good pre-launch test is deliberately uncomfortable. It looks for the wrong price, the absent policy, and the edge case that should reach a person. Passing the easy questions is useful. Refusing the dangerous ones is what makes the launch safe.