You Get What You Sample: Evaluating Sampling Strategies for Web Security Measurements
Evaluation of 500k Tranco and 24.8M Common Crawl hosts shows Top-N domain sampling biases web security measurements; probability sampling yields unbiased estimates.
The study is the first comprehensive investigation of how sampling strategies affect web security measurement conclusions, comparing datasets and strategies across 500k Tranco domains and 24.8M Common Crawl hosts. It shows Top-N selection does not reflect the overall web distribution and may bias observed vulnerability rates, while probability-based strategies yield stable, unbiased prevalence and impact estimates. Hybrid sampling offers no advantage because its deterministic prefix consistently hurts accuracy, and the authors propose an adaptive probability-based strategy effective even when target prevalence is unknown.
- Top-N sampling distorts prevalence estimates and limits cross-study comparability
- Probability-based sampling yields stable, unbiased estimates
- Hybrid sampling's deterministic prefix consistently reduces accuracy
- Evaluated on 500k Tranco and 24.8M Common Crawl hosts
- Proposes adaptive probability-based strategy for unknown target prevalence
- Products
- TrancoCommon Crawl
From the original article
1 min read at arxiv.orgWeb measurement studies rely on domain datasets such as Tranco to quantify the prevalence and impact of security issues at scale, but exhaustively analyzing these datasets is often infeasible because of the cost of advanced analysis techniques, requiring the use of sampling.
By Xuenan Zhang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.