RTAformer targets relation changes in video scene graph generation
Original title: RTAformer: Relation-Transition-Aware Transformer for Video Scene Graph Generation
On Action Genome, gains are larger on relation-transition frames than on non-transition frames with matched backbones.
The paper describes relational semantic inertia, where video scene graph models keep an old relation prediction after the relation has changed. RTAformer is a one-stage transformer that models entity persistence separately from relation changes, using differences between adjacent-frame relation queries and cross-frame instance alignment in training. Experiments on Action Genome report higher recall and mean recall than existing methods, with ablations supporting the proposed components.
- The failure mode is called relational semantic inertia.
- Calibrated adjacent-frame differences guide predicate updates.
- Cross-frame instance alignment is used during training.
- Matched-backbone comparisons show larger gains on transition frames.
From the original article
1 min read at huggingface.co
Video scene graph generation (VidSGG) aims to capture objects and their evolving relationships in videos. This requires accounting for both the persistence of object identities and abrupt changes in relationships. Existing methods aggregate information across frames to exploit temporal context, but an emphasis on consistency can obscure these changes.
By Jaeyeon Heo, Geuntaek Lim, Yukyung Choi · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.