A crawl audit reads every public page, extracts the questions each one answers, then clusters them. What surfaces is rarely a knowledge base. It is duplication, prices that disagree with each other, policies nobody has updated, and the customer questions the site never answered at all.
Before a chatbot answers anything, something has to read the website and decide what it knows. That reading is the most honest audit a business website ever gets, because it has no interest in the site's design, its traffic or the story anyone tells about it. It only asks one question of every page: what could a customer learn here?
The answer is usually less than the page count suggests.
The audit, step by step
You can run this without a chatbot. It takes a crawler, a spreadsheet and an afternoon.
Crawl every public URL and record the ones that return content. Note what is excluded: pages behind a login, anything blocked in robots.txt, and anything only reachable after a form post. That exclusion list is a finding in itself, because it is the material your customers cannot search either.
For each page, write down the questions it answers. Not the topic. The question, phrased as a customer would ask it. A services page might answer three; a hero image and a testimonial answer none.
Cluster the questions. Group everything that is the same question in different words, and count how many pages landed in each group.
Diff the answers inside each cluster. Where two pages answer the same question with different facts, you have found a contradiction, and it is the highest-value output of the whole exercise.
List the questions with no cluster at all. Compare that against the last fifty enquiries the business actually received, which is where the gaps become undeniable.
What one crawl looked like
One authorized crawl of a customer site read more than 1,000 pages and produced over 5,400 question-and-answer pairs.
That sounds like an unusually well-documented business. It was not. Most of those pairs were near-duplicates: the same pricing language repeated across dozens of location pages, one town swapped for another, a sentence reordered here and there. Reduced to distinct answers, the real knowledge base was a small fraction of the pile.
This is a single site rather than a study, and the specific ratios will differ for yours. The shape is the part worth carrying over: page count and knowledge are barely related, and volume actively hurts when a retrieval system has to pick one answer out of forty slightly different versions of it. That is one of the ways a grounded chatbot still gets things wrong.
The five things a crawl reliably surfaces
Duplication. The same answer copied across service pages, location pages and an old landing page. Harmless to a reader who sees one of them. A problem for anything that has to choose.
Contradiction. Two live prices for the same service. A response time promised on the homepage and a different one in the FAQ. Delivery included on one page and charged on another. Nobody notices because no human reads both pages in the same minute.
Staleness. Last year's rates on a page that never got taken down. A COVID-era notice. A team page listing someone who left in 2024. Old content does not announce itself, and a crawler treats a 2022 price with exactly the same seriousness as this morning's.
Gaps. Questions customers ask every week with no published answer anywhere. Delivery area, whether GST is included, what happens if the size is wrong, whether you take UPI, weekend timings. These are the ones the owner answers on the phone so often they have stopped noticing they are unwritten.
Hedges. "Pricing varies as per requirement." "Timelines depend on the project." These pages look like answers and contain none. They are worse than gaps, because a system will retrieve them, and the customer gets a paragraph that says nothing at eleven at night.
Pages written for search rather than for customers
Every audit finds a set of pages written for a search engine and never re-read by a person. Twenty near-identical location pages. A blog post about "the benefits of modular kitchens" with no price, no process and no contact detail. A services page tuned for a keyword phrase nobody says out loud.
They are not useless: they may still bring the traffic they were built for. They are close to useless as knowledge, and they distort a knowledge base by weight of repetition, making a marginal fact look well-established because it appears forty times.
A crawl audit is the first time most businesses see that split laid out: which pages exist to be found, and which exist to answer someone. A healthy site needs both. Most sites discover they have a lot of the first and very little of the second.
What to do with the findings
Fix contradictions first. They cause the worst outcomes, and each one is a fifteen-minute edit.
Delete or date the stale pages. A page that is genuinely historical can say so in its first line.
Write the gaps. Two or three plain sentences per question, published somewhere findable. This is the bulk of the work and the whole point of the audit. The pre-launch checklist is the same job organized as a to-do list.
Rewrite the hedges into a real range with the conditions attached, or accept that those questions route to a person and design for that.
Then run the audit again in six months, because the interesting number is not what the crawl found today. It is whether the gap list got shorter. If you maintain client sites, this is also the most convincing thing you can put in front of a client, since it is their own website telling them what it fails to say. Agencies can run it as a paid first step before any chatbot conversation starts.