You Get What You Sample: Evaluating Sampling Strategies for Web Security Measurements
Evaluation of 500k Tranco and 24.8M Common Crawl hosts shows Top-N domain sampling biases web security measurements; probability sampling yields unbiased estimates.
The study is the first comprehensive investigation of how sampling strategies affect web security measurement conclusions, comparing datasets and strategies across 500k Tranco domains and 24.8M Common Crawl hosts. It shows Top-N selection does not reflect the overall web distribution and may bias observed vulnerability rates, while probability-based strategies yield stable, unbiased prevalence and impact estimates. Hybrid sampling offers no advantage because its deterministic prefix consistently hurts accuracy, and the authors propose an adaptive probability-based strategy effective even when target prevalence is unknown.