ZeroHour
DashboardNewsCVEsExploitsIncidentsAIBriefAnalyticsEntitiesWatchlistIOCsAsk the feedSourcesFeeds & APIKiosk mode

Search: “web crawling”

16 stories

Searchkeywordsmart Sortlatesthotimportancerelevance
DomainAllCyberAIWindow24h3d7d30dAllIn the wildImportant onlyShow duplicates
CategoryAllVulnerability 31Data breach 28Ransomware 25Malware 23Exploit / PoC 14Industry 13Phishing & fraud 13Research 12Policy & legal 12Threat actor 10AI safety & security 5AI tools & infra 4Tools 4AI industry 1
SeverityAllcritical 41high 52medium 52low 35info 17

TLD Tracker: Exploring Newly Released Top

Unit 42 tracked 19 newly released top-level domains and found large-scale phishing, unwanted program distribution, and cybersquatting tied to TLD launch dates.

Researchers analyzed 19 new generic TLDs, including .zip, .bot, .ing, and .meme, released or approaching general availability over roughly 18 months. Data from passive DNS, registry zone files, newly registered domain feeds, and the Tranco top-1M list showed phishing campaigns, potentially unwanted program distribution, and domain squatting on these TLDs. Abuse correlated with each TLD's rollout phases, indicating attackers monitor general availability dates to register and weaponize domains. The IANA root database now lists over 1,000 generic TLDs.

Palo Alto Unit 42 · Aug 17, 2026Research

You Get What You Sample: Evaluating Sampling Strategies for Web Security Measurements

Evaluation of 500k Tranco and 24.8M Common Crawl hosts shows Top-N domain sampling biases web security measurements; probability sampling yields unbiased estimates.

The study is the first comprehensive investigation of how sampling strategies affect web security measurement conclusions, comparing datasets and strategies across 500k Tranco domains and 24.8M Common Crawl hosts. It shows Top-N selection does not reflect the overall web distribution and may bias observed vulnerability rates, while probability-based strategies yield stable, unbiased prevalence and impact estimates. Hybrid sampling offers no advantage because its deterministic prefix consistently hurts accuracy, and the authors propose an adaptive probability-based strategy effective even when target prevalence is unknown.

arXiv cs.CR · 6d agoResearch1

Creepy crawlies

git.kernel.org spends more CPU rendering commit pages for abusive AI-era scrapers than on all legitimate access including git clones.

Konstantin Ryabitsev reports that abusive crawler 'background radiation' at git.kernel.org consumes more CPU cycles rendering commit HTML than all legitimate access, with 14 cores across five geo-distributed nodes dedicated to scraper traffic. Simon Willison highlights the implications for crawlable web services like Datasette.

Simon Willison · 9d agoAI industry1

terms.txt: A Consent and Compensation Protocol for Agentic Web Access

terms.txt specifies a robots.txt-style protocol for per-path, per-purpose AI crawler consent and compensation, with enforcement adding 0.20-0.65 ms per request.

The paper documents that automated clients now make up most web requests, that training dominates Cloudflare-classified crawling, and that the largest AI platforms fetch thousands of pages per returned visitor while robots.txt cannot express identity, purpose, terms, or price. It specifies terms.txt plus an origin-enforced exchange using Web Bot Auth signatures, signed intent, delegation tokens, HTTP 402 negotiation, and signed receipts. A dependency-free implementation adds 0.20 to 0.65 ms per request on one vCPU.

arXiv cs.CR · 6d agoResearch

Reflectiz Launches Agentic Pentesting for Websites: Up to 10x Coverage vs Conventional Pentests

Reflectiz launched a multi-agent AI penetration testing platform for websites, claiming up to 10x more coverage than conventional pentests.

Reflectiz, a continuous web exposure management vendor, launched a multi-agent penetration testing platform for websites under a new Offensive Hub. Four specialized agents crawl the site like a real user, fingerprint the stack, run attacks, and independently validate findings to remove false positives. The company claims up to 10x more coverage than conventional pentesting by starting from its pre-existing live model of each site. Findings integrate via REST API, CI/CD triggers and Slack, and the Atlas remediation agent guides fixes.

CSO Online · 8d agoTools

Reflectiz Launches Agentic Pentesting for Websites: Up to 10x Coverage vs Conventional Pentests

Reflectiz launched a multi-agent AI penetration testing platform for websites, claiming up to 10x more coverage than conventional pentests by leveraging existing site context.

Reflectiz, a web exposure management vendor, announced an agentic penetration testing product within a new Offensive Hub. The platform uses specialized AI agents for crawling, stack fingerprinting, attack execution, and independent validation, claiming to cover the OWASP Top 10 and reduce false positives. Findings integrate with existing workflows via REST API, CI/CD triggers, and Slack, and an AI remediation agent named Atlas guides fixes. The announcement is a vendor press release with a planned September 15 webinar demo.

GBHackers · 8d agoTools 2 sources

WordPress Security Plugins: How to Choose the Right One

Sucuri's guide breaks WordPress security plugins into hardening, malware scanning, integrity monitoring, and filtering types, and explains how to evaluate and layer them.

The Sucuri guide explains that WordPress security plugins bundle five capabilities - hardening, malware detection, integrity monitoring, activity logging, and application-level filtering - and that plugins run only after WordPress loads, unlike server-level firewalls. It lists leading causes of compromise: outdated plugins and themes, weak or reused credentials, nulled premium software, insecure configuration, and shared-hosting cross-contamination. It concludes with evaluation criteria and a post-installation security checklist for owners without dedicated security teams.

Sucuri Blog · 11d agoIndustry

Tactical Threat Intelligence Explained: Benefits & Use Cases

Recorded Future explains tactical threat intelligence, covering attacker TTPs, IOC collection, and use in SOC detection, response, and control hardening.

Recorded Future published an explainer on tactical cyber threat intelligence, describing how it differs from strategic and operational intelligence by focusing on attacker tactics, techniques, and procedures. The piece outlines sources such as OSINT, dark web monitoring, malware analysis, and internal telemetry, and describes the intelligence lifecycle from collection through dissemination. It argues tactical CTI improves detection tuning, incident response, preventive controls, and resource allocation for SOC teams.

Recorded Future · Aug 17, 2026Industry

Have it both ways: stay discoverable in search while disallowing AI training

Cloudflare launches Disallow AI Training setting letting sites block AI training via robots.txt while staying indexed in search; Apple, Google, and Microsoft honor it.

Cloudflare announced a 'Disallow AI Training' setting that publishes a no-training preference in robots.txt so sites can block AI training (including by mixed-use crawlers) without losing search indexing. Apple, Google, and Microsoft meet Cloudflare's new 'Accountable' designation, which requires training/summary opt-out mechanisms, URL-level training visibility, and assurance that opt-outs don't affect search rankings. Cloudflare cites that under 1% of its sites block search bots while 17% block AI training, and its Block settings now apply to mixed-use crawlers as well. Granular controls over how much content appears in AI summaries are planned for early next year.

Cloudflare Blog · 1d agoTools

Beneath the Surface: Detecting and Blocking Hidden Malicious Traffic Distribution Systems

Unit 42 built an ML-based detector for malicious traffic distribution systems, finding malicious TDS chains average longer redirections and more URLs than legitimate ones.

Traffic distribution systems redirect victims through chains of intermediate domains to hide final destinations, serving phishing, malvertising, and online gambling operations. Unit 42's topological analysis of redirection graphs found malicious TDS traffic uses longer chains (about 25% exceed four hops vs 10% benign), more URLs (median 126 vs 80), and fewer isolated subgraphs with higher connectivity. These features power an ML detector integrated into Advanced DNS Security and Advanced URL Filtering to identify and block malicious TDS infrastructure in customer traffic.

Palo Alto Unit 42 · Aug 17, 2026Research