Why 'no website found' is usually wrong

Checking whether a business has a website sounds trivial: search for the name, see if a site comes up. Building a tool that does it automatically, at scale and accurately, taught me more about data quality than any other part of this project.

#Version one: confident and wrong

The first version searched for the business name and area. If no plausible site appeared in the results, it marked the business as having no website. It produced a long list of businesses apparently without sites.

When I checked a sample by hand, most of them had perfectly good websites.

The search provider had returned weak results, or the business name was generic, or the site simply was not well indexed. The tool had turned I didn't find it into it doesn't exist — the most common mistake in automated research.

#Design the data model to say "unknown"

The fix started in the data model, not the code. A business's web presence now has four states:

• Has a site — found and reachable.
• Weak site — reachable but missing basics (mobile layout, contact details, HTTPS).
• Broken — definitely failing.
• Unknown — not enough evidence either way.

"No website" is only assigned when a trusted, authoritative source confirms it. A missing search result can never produce it.

#Multiple independent signals

Instead of one search, the pipeline now combines:

1. A server-side search feed for the name and area.
2. Domain candidates generated from the name (fadedistrict.co.uk, fade-district.com…) and probed directly.
3. A site scan of any candidate, trying several URL variants.

Domain guessing turned out to be surprisingly strong. One search provider blocked server traffic entirely and another returned irrelevant results; simply trying the obvious domains found sites the searches missed.

#"Broken" needs a high bar too

The mirror-image problem appeared next: sites marked broken that opened fine in a browser. Causes included:

• Bot protection pages returning 403 or a challenge.
• www working while the bare domain did not, or vice versa.
• Slow servers exceeding a short timeout.

Now a site is only broken after DNS failure or a hard 404/410 on every URL variant. Protection pages count as online. Timeouts count as unknown.

#Keep the evidence with the verdict

Every verdict stores what was found: the matched domain, status codes per variant, the page title, and any photos discovered. When someone questions a result, the answer is on screen in seconds. That turned the tool from something to second-guess into something to trust.

#Respect the sites you check

An automated checker is still a visitor. It identifies itself, uses short timeouts, fetches only what it needs and never hammers a host. Images found during a scan are proxied through an endpoint that accepts only image types under a size cap and refuses private network addresses — basic protection against the endpoint being abused.

#What generalises

• Absence of evidence is not evidence of absence. Build "unknown" in from day one.
• Combine independent signals rather than trusting one.
• Set asymmetric thresholds: the costlier the wrong label, the more evidence it needs.
• Store evidence beside every automated decision.
• Spend on one trusted signal before adding many weak ones.

The architecture is described in more detail in the web-presence verification case study.