← All writing

Template

When to re-crawl website content for your chatbot

Re-crawl website content after you change a fact the chatbot may repeat, such as a price, opening time, service area, or policy. Update the primary source, remove stale copies, run the crawl, then test the changed answer and one edge case. Chatterbox allows each source to be re-crawled once a day.

A keyboard, mouse, cable and closed laptop on a small-business back-office desk.

A website chatbot re-crawl is due when a fact visitors rely on has changed. A new homepage photograph can wait. A new price, opening time, service area, cancellation condition, or preparation instruction cannot.

Chatterbox answers from material your business has supplied. A website source can be re-crawled once a day. Use that refresh deliberately: correct the source first, remove the old version, then test what the widget now says.

Which website changes need a chatbot re-crawl?

Re-crawl after changing information that could alter a visitor's next decision. That includes what something costs, whether you cover an address, when you open, what a service includes, what a customer must bring, and which policy applies.

You do not need to refresh the knowledge base because you changed a color, replaced a photograph, or fixed punctuation that does not change meaning. The useful test is simple: could the old page make the chatbot give an answer your team would now have to correct? If yes, refresh it.

Do not wait for a monthly maintenance day when the change concerns a price or promise. The widget can answer outside business hours without anyone watching. A stale answer can therefore keep appearing after the website team believes the correction is finished.

What should happen before you run the crawl?

Update the primary source. Change the page that your team treats as authoritative. If delivery terms belong on the delivery page, fix that page rather than adding a temporary sentence to the homepage.

Remove the stale copy. Search the public site for the old figure, time, place, or condition. An old FAQ and a new service page can both be available to the crawler. A fresh crawl does not decide which statement the business intended to keep.

Check robots.txt too if the page moved to a new path or the earlier crawl failed. Chatterbox's crawler is called ChatterboxBot and obeys that file. A source cannot refresh material the crawler is not allowed to read.

How do you refresh and test the changed answer?

A five-step path from updating the primary source to testing an edge-case question.

Re-crawl the source. Chatterbox allows a website source to be re-crawled once a day. If several related pages need editing, finish the connected changes before using that day's refresh. This avoids testing one corrected page beside another page you already know is stale.

Test the changed answer. Ask the question in the words a visitor would use. Chatterbox shows the source passage behind each answer, so confirm that the reply points to the page and sentence you intended to update. Do not stop at seeing the new number somewhere in the response; check the condition attached to it as well.

Ask one edge-case question. Change one detail so the new source no longer settles the answer. If the page lists five covered areas, ask about a sixth. If it gives a standard price, ask about a custom job. Chatterbox should say it cannot find the answer and ask for contact details rather than stretching the revised rule.

What if the answer is still old after the re-crawl?

First, open the public page in a private browser window. A content management system, page cache, or content delivery layer may still be serving the old version even though the editor shows the new one. The crawler can only read what the public URL returns.

Next, follow the source passage shown with the answer. It may reveal a second page you missed, a repeated FAQ, or an uploaded document carrying the old statement. Chatterbox can also use file uploads, pasted text, and hand-typed question-and-answer pairs. Re-crawling a website source does not update those separate sources.

Correct or remove the stale material where it lives. Then use the next available re-crawl and repeat the changed-answer and edge-case tests. The goal is not merely a successful crawl. It is one current source for the known answer, and an honest refusal where that source stops.