Origin Is All You Need: Provenance-Aware Transformers for Structural Trust-Boundary Separation
Provenance-aware transformers use origin labels to structurally block indirect prompt injection.
The paper argues ordinary transformers cannot separate system instructions from retrieved or user content because one attention mechanism processes every token, enabling indirect prompt injection. Provenance-Aware Transformers assign each token a ring ID and add origin embeddings, a learnable origin attention bias, and an origin scale that survives normalization. A two-stage fine-tune teaches released pretrained models those origin semantics under ring constraints. The authors report robust in-distribution and out-of-distribution resistance to indirect prompt injection while utility stays comparable to the base model.
- Indirect prompt injection exploits attention that treats every source alike.
- Ring IDs, origin embeddings, attention bias, and scale encode provenance.
- Two-stage fine-tuning adapts released pretrained models to ring constraints.
- Reported injection resistance holds in and out of distribution with similar utility.
Full article200 words · extracted from arxiv.org · click to collapse
Indirect prompt injection (IPI) remains a central safety and security challenge for large language model (LLM) systems because standard transformers lack architectural notion of source authority. Retrieved documents, user inputs, and system instructions are all processed through the same undifferentiated attention mechanism, forcing the model to infer from wording alone what should be obeyed and what should be treated as data. We propose Provenance-Aware Transformers, a provenance-aware defense that makes application-supplied source labels actionable inside the model. Each input token is assigned a ring ID encoding its origin, and the model is augmented with origin embeddings, a learnable origin attention bias, and a learnable origin scale that preserves provenance under normalization. The resulting architecture enforces a structural boundary between authoritative and non-authoritative sources during generation. To instantiate this architecture on released pretrained models, we propose a two-stage fine-tuning pipeline to teach the model origin semantics and task behavior under ring constraints. Evaluation shows that Provenance-Aware Transformers maintain robust resistance to IPI both in-distribution and out-of-distribution while preserving utility comparable to the base pretrained model. More broadly, our work shows that exposing provenance as a first-class architectural signal can shift LLM safety alignment from brittle pattern matching toward explicit trust separation.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.21088