Web-presence verification pipeline
An internal research tool that checks whether a business already has a working website, using several independent signals and refusing to guess when the evidence is weak.
Technical objective
Answer one question reliably — does this business already have a website that works? — because a wrong answer in either direction wastes time and looks careless.
Constraints
- Some search engines block automated server requests or return low-quality results.
- Websites fail temporarily: timeouts, bot protection and redirects are not the same as 'broken'.
- No personal data should leave the internal tool; the public site shows architecture only.
Architecture
- 01
Business record
Name, area and any known URL from the internal list.
- 02
Search signal
Server-side search feed queried for the business name and area.
- 03
Domain candidates
Likely domains generated from the name and probed directly.
- 04
Site scan
Several URL variants (www, non-www, http, https) fetched with timeouts.
- 05
Classifier
Rules map evidence to: has site, weak site, broken, or unknown.
- 06
Human review
Unknowns stay unknown until confirmed by a trusted source.
Flow runs left to right, top to bottom.
Key implementation decisions
'Unknown' is a valid answer
The first version labelled anything it could not find as 'no website'. Most of those businesses had sites. Now a missing search result never proves absence — only a trusted maps listing or a confirmed new registration can.
Only definite failures count as broken
A site is marked broken only for DNS failure or a hard 404/410 across every URL variant. Timeouts and bot-protection pages are treated as online.
Evidence is stored with the verdict
Each check keeps what was found — matched domain, status codes, photos discovered — so a person can audit any decision in seconds.
Challenges
Blocked and noisy search
One search provider blocked server traffic entirely, another returned irrelevant results. Domain guessing plus direct probing ended up being the most reliable free signal.
False 'broken' labels
Sites that blocked automated requests looked broken. Trying multiple URL variants and treating protection pages as 'online' removed almost all false positives.
Lessons learned
- Absence of evidence is not evidence of absence — design the data model to say 'unknown'.
- Keep the evidence next to the verdict; it makes the system debuggable by non-engineers.
- Pay for one trusted signal before stacking many weak ones.