The numbers behind our two studies, as CSV and JSON
Both studies are our own measurements: one on how many licensed service businesses have a working website, one on who is really behind the AI crawler traffic in our server logs. The files hold the aggregates each study prints, with the denominator next to every percentage. Reuse them with credit.
US service-license website-presence dataset (July 2026)
Of 8,297,381 state and territory service-license rows in our database, 4,542,400 had a completed website check. After keeping genuine state and territory codes and separating business entities from individually licensed people, 1,630,689 of 1,705,674 business entities had no working website, or 95.60%. Treat that as an upper bound, not an exact rate (see the limits below).
Preview: table entity_buckets (31 rows in the full file across 3 tables)
| Bucket | Rows | Share | No working site found |
|---|---|---|---|
| Business entities | 1,705,674 | 41.90% | 95.60% |
| Individuals | 1,262,510 | 31.01% | 99.29% |
| Undetermined | 1,102,709 | 27.09% | 96.23% |
| All 56-state rows | 4,070,893 | 100% | 96.91% |
Method in three filters
- Keep only rows where our pipeline completed a website check (4,542,400 rows). A blank website field on an unchecked row is not a measured absence.
- Keep only the 56 genuine US state and territory codes (4,070,893 rows). The field held 199 distinct values; federal contractor registrations from SAM.gov and scraper source tags are not state licenses.
- Classify each row by entity and trade tokens in the business name (
LLC,INC,PLUMBING) against names that simply equal the holder’s own name. Rows that match neither stay undetermined.
Limits. The 95.60% is an upper bound. “Checked” means we looked for a website and did not find one, not that we confirmed there is none, and we have not measured how often our search misses a site that exists. Among plumbing, HVAC, roofing and electrical business records, rows from state license files (268,510) are 97.87% no working site, against 35.26% for rows from Google Maps listings (27,207). This is not a random sample. Verification coverage runs from 20.1% in New York to 99.8% in Alaska, and the two largest states are the least checked. The individual share (31.01%) is a floor, because blank owner-name fields cannot be classified. Michigan, at 85.97%, is an unexplained outlier.
Full write-up with the reasoning behind each filter: Up to 95.6% of licensed service businesses have no working website.
AI crawler request log summary (19–26 July 2026)
Between 19 and 26 July 2026, 3,727 requests to vantaweb.io carried an AI-crawler or search-engine user agent. 3,109 of them, 83.4%, asked for a path we have never published. Counting user agents alone overstated real crawler activity by about six times (3,727 divided by 618 requests for real paths is 6.03).
Preview: table legitimate_fetches_per_day (13 rows in the full file across 3 tables)
| Crawler | Operator | Legitimate fetches per day |
|---|---|---|
| ClaudeBot | Anthropic | 43 |
| OAI-SearchBot | OpenAI | 13 |
| ChatGPT-User | OpenAI | 8 |
| Googlebot | 13 |
Method
- Every HTTP request to vantaweb.io for the window, pulled per day from the Cloudflare GraphQL Analytics API (
httpRequestsAdaptiveGroups, dimensionsuserAgentandclientRequestPath). - Each requested path compared with the 105 URLs in our published sitemap plus robots.txt, sitemap.xml, llms.txt, llms-full.txt, rss.xml and our .well-known files. A request for any other path counts as spoofed, whatever its user agent claimed.
Limits. One site, eight days, modest traffic: a single scanning campaign moves the percentages more than it would on a large site. We make no claim about who sent the spoofed requests. The 83.4% is specific to us.
Full write-up: 83% of our AI crawler traffic was fake.
How the files are laid out
Each CSV is in long format with six columns: table (which table of the study the row belongs to), item (the row label), metric (what was counted), value, unit and note. The JSON file carries the same rows plus the dataset name, period, source and licence. Values are printed as in the study, without thousands separators.
Only the aggregates the studies print are published. The underlying license records and request logs are not in these files.
Licence, citation and corrections
Both datasets are licensed under Creative Commons Attribution 4.0. Use them, change them, publish with them, and credit VantaWeb with a link to this page.
VantaWeb. “Open data from VantaWeb: two first-party studies.” https://vantaweb.io/data/. Hunter Spence, founder. CC BY 4.0.
If a figure looks wrong, tell us. When we correct a number we correct it on the study page and in these files together, and the study page carries a dated note. Related reading: our AI receptionist statistics, each figure with a named source and a date.