Agentic Crawlers Record More Trackers Than Standard Methods
Original title: Beyond the Landing Page: Website Blocking, Tracker Visibility, and Human Alignment in Automated Web Crawls
LLM-driven agents record 1.5-1.7 times more third-party trackers than non-agentic crawlers across 2,000 websites.
Compares agentic and non-agentic web crawlers on 2,000 popular websites. Agentic crawlers detect more trackers, especially on consent pages. They recover 82-87% of human-observed trackers but half their observations not in human visits. Websites deny more with standard automation.
- Website denials increase beyond landing pages for standard browser automation
- Detection-resistant framework prevents escalation without increasing tracking
- Agentic crawlers record tracking from 1.5-1.7 times more third-party organizations
- Agentic recover 82-87% of human trackers
From the original article
1 min read at arxiv.orgWeb tracking studies largely rely on automated crawlers, yet their measurements can differ from what people encounter while browsing. Conventional crawlers follow predefined navigation rules, whereas LLM-driven agents decide how to navigate and interact with websites, potentially changing the tracking they encounter.
By A H M Nazmus Sakib · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.