When Agents See Differently: Exposing UI Desynchronization Threats in Mobile Agents
Researchers expose 'human-agent UI desynchronization' attacks where repackaged APKs invisibly mislead mobile AI agents into attacker-chosen actions.
The paper introduces human-agent UI desynchronization: agents ingest digital screenshots and accessibility metadata that reveal content human users cannot perceive due to occlusion and luminance-contrast limits. An automated framework embeds perturbations into repackaged APK clones that steer mobile agents toward attacker-designated actions without access to runtime user instructions or online adaptation. Evaluations across five mobile-agent frameworks and three backbone models on 546 tasks achieved average misleading rates of 77.9% and 66.9%. A questionnaire study with 186 participants found the visual perturbations difficult for humans to notice.