Search: “Common Crawl”

You Get What You Sample: Evaluating Sampling Strategies for Web Security Measurements

Evaluation of 500k Tranco and 24.8M Common Crawl hosts shows Top-N domain sampling biases web security measurements; probability sampling yields unbiased estimates.

The study is the first comprehensive investigation of how sampling strategies affect web security measurement conclusions, comparing datasets and strategies across 500k Tranco domains and 24.8M Common Crawl hosts. It shows Top-N selection does not reflect the overall web distribution and may bias observed vulnerability rates, while probability-based strategies yield stable, unbiased prevalence and impact estimates. Hybrid sampling offers no advantage because its deterministic prefix consistently hurts accuracy, and the authors propose an adaptive probability-based strategy effective even when target prevalence is unknown.

arXiv cs.CR·Sep 10, 08:21 UTC · 6d agoResearch1

Why No HTTPS? The 2021 Version

Troy Hunt·Aug 12, 07:55 UTC · Aug 12, 2021AI tools & infra

terms.txt: A Consent and Compensation Protocol for Agentic Web Access

terms.txt specifies a robots.txt-style protocol for per-path, per-purpose AI crawler consent and compensation, with enforcement adding 0.20-0.65 ms per request.

The paper documents that automated clients now make up most web requests, that training dominates Cloudflare-classified crawling, and that the largest AI platforms fetch thousands of pages per returned visitor while robots.txt cannot express identity, purpose, terms, or price. It specifies terms.txt plus an origin-enforced exchange using Web Bot Auth signatures, signed intent, delegation tokens, HTTP 402 negotiation, and signed receipts. A dependency-free implementation adds 0.20 to 0.65 ms per request on one vCPU.

arXiv cs.CR·Sep 10, 07:01 UTC · 6d agoResearch

WhiteHat Security introduces new feature for dynamic single-page application scanning

Help Net Security·Oct 10, 00:00 UTC · Oct 10, 2018AI tools & infra

TLD Tracker: Exploring Newly Released Top

Unit 42 tracked 19 newly released top-level domains and found large-scale phishing, unwanted program distribution, and cybersquatting tied to TLD launch dates.

Researchers analyzed 19 new generic TLDs, including .zip, .bot, .ing, and .meme, released or approaching general availability over roughly 18 months. Data from passive DNS, registry zone files, newly registered domain feeds, and the Tranco top-1M list showed phishing campaigns, potentially unwanted program distribution, and domain squatting on these TLDs. Abuse correlated with each TLD's rollout phases, indicating attackers monitor general availability dates to register and weaponize domains. The IANA root database now lists over 1,000 generic TLDs.

Palo Alto Unit 42·Aug 17, 12:47 UTC · Aug 17, 2026Research

WordPress Security Plugins: How to Choose the Right One

Sucuri's guide breaks WordPress security plugins into hardening, malware scanning, integrity monitoring, and filtering types, and explains how to evaluate and layer them.

The Sucuri guide explains that WordPress security plugins bundle five capabilities - hardening, malware detection, integrity monitoring, activity logging, and application-level filtering - and that plugins run only after WordPress loads, unlike server-level firewalls. It lists leading causes of compromise: outdated plugins and themes, weak or reused credentials, nulled premium software, insecure configuration, and shared-hosting cross-contamination. It concludes with evaluation criteria and a post-installation security checklist for owners without dedicated security teams.

Sucuri Blog·Sep 5, 03:01 UTC · 11d agoIndustry

Tactical Threat Intelligence Explained: Benefits & Use Cases

Recorded Future explains tactical threat intelligence, covering attacker TTPs, IOC collection, and use in SOC detection, response, and control hardening.

Recorded Future published an explainer on tactical cyber threat intelligence, describing how it differs from strategic and operational intelligence by focusing on attacker tactics, techniques, and procedures. The piece outlines sources such as OSINT, dark web monitoring, malware analysis, and internal telemetry, and describes the intelligence lifecycle from collection through dissemination. It argues tactical CTI improves detection tuning, incident response, preventive controls, and resource allocation for SOC teams.

Recorded Future·Aug 17, 00:00 UTC · Aug 17, 2026Industry