Relevance Does Not Imply Applicability: Experience Activation for Personal GUI Agents
ExpActivator activates only situation-applicable GUI history, lifting step success 28% while using far fewer tokens.
The paper argues that task-relevant GUI history is not automatically applicable at each decision: retrieved context mainly helps the first step and little of the remaining 90%. ExpActivator is a training-free method that matches the current screen to historical states in a frozen GUI backbone's latent space and supplies only the matching action. It also proposes a recurring intent only when time and scenario support it, otherwise abstaining. Across four backbones it raises within-trajectory step success by 28% on average, uses about one-fifth as many history tokens, and reaches about 2.3 times the Matthews correlation of the strongest proactive baseline.
- Retrieved task history mostly helps the first step, not the remaining 90%.
- Training-free matcher uses the frozen GUI backbone latent space to pick applicable actions.
- Proactive intents fire only with enough time and scenario support; otherwise it abstains.
- Step success rises 28% on average across four backbones with about one-fifth the history tokens.
- Matthews correlation is about 2.3 times the strongest proactive baseline.
- AI models
- ExpActivator
From the original article
1 min read at huggingface.co
Personal Graphical User Interface (GUI) agents rely on interaction history to infer what a user wants from ambiguous instructions and to anticipate recurring routines. Existing approaches retrieve task-relevant history and append it to the policy's context, implicitly assuming that experience relevant to a task remains useful for each decision within it.
By Fuyao Zhang, Xuan Wang, Zherui Li · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.