GUIAuditor: Enabling Post-hoc Child Safety Forensics via Action-Guided GUI Provenance on Mobile Devices
GUIAuditor uses an on-device model to record mobile GUI activity for later guardian safety review.
Researchers present GUIAuditor, a mobile system that records GUI provenance so a guardian can later review a child's interactions. A multimodal large language model turns GUI events into a narrative, while evidence distillation cuts data requiring analysis by more than 89.2 percent versus periodic sampling. On 295 clips it reached 95.23 percent macro-F1 for significant events and returned the correct evidence first for over 90.20 percent of questions. Tests on three smartphones added about 2.1 watts, 7.4 seconds of per-event latency, and roughly 3.1 GB peak memory.
- GUIAuditor creates queryable GUI provenance from a child's mobile interactions.
- Distillation reduces analyzed data by more than 89.2 percent versus periodic sampling.
- It reached 95.23 percent macro-F1 and over 90.20 percent top-result retrieval.
- On-device inference added about 2.1W, 7.4 seconds, and 3.1GB peak memory.
Full article244 words · extracted from arxiv.org · click to collapse
The proliferation of smart devices exposes children to online risks like grooming and financial scams that are deeply embedded within legitimate applications. Current approaches rely on automated prevention and detection, a paradigm that is fundamentally limited by its inherent fallibility. Whether rule-based or AI-driven, they inevitably produce false positives and negatives, failing to provide reliable protection. In this paper, we argue for a complementary, human-in-the-loop, post-hoc forensic paradigm. We present GUIAuditor, the first system designed to realize this vision by creating GUI Provenance: a queryable, semantic record of a child's interaction sequence. To generate this, GUIAuditor leverages a Multimodal Large Language Model (MLLM) to translate the temporal sequence of GUI events into a human-understandable narrative. To make this practical on mobile devices, a novel evidence distillation pipeline reduces the data requiring analysis by over 89.2% compared to periodic sampling approaches adopted by industry standards, with negligible impact on accuracy. On a new dataset of 295 interaction clips, GUIAuditor achieves a 95.23% Macro-F1 Score in logging significant events and, crucially, its two-stage forensic query engine successfully retrieves the correct evidence as the top result for over 90.20% of natural language questions. An end-to-end evaluation on three modern smartphones shows that the full pipeline, including on-device MLLM inference, adds 2.1W of power draw and 7.4s of per-event latency, with a peak memory footprint of ${\sim}$3.1GB. These results show that post-hoc GUI forensics can run on modern mobile devices and provide useful context for guardian-led safety review.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.28205