Search: “adversarial-robustness”

Smart search ranks by meaning as well as keywords (one row per story, last 45 days).

Robust Policy Optimization via Adversarial Importance Sampling

Adversarial Importance Sampling estimates worst-case RL returns without extra interactions; authors also release the advrl PyTorch library.

The paper introduces Advis, which uses importance sampling over trajectories from standard training to estimate and optimize verifiable worst-case returns, requiring no additional environment interactions or auxiliary networks. It also releases advrl, a modular PyTorch library of single-file robustness methods and adversarial attacks for reproducible evaluation. The authors show adversarial hyperparameters do not transfer across agents, so they evaluate with 6-14x more attacker configurations than prior work. Effectiveness is demonstrated on continuous control environments.

arXiv cs.AI / cs.LG / cs.CL·Sep 11, 16:40 UTC · 4d agoAI research1

20

Active Adaptation, Not Static Defense: Temporal Dynamics of Preventative Steering in Adversarial Fine-Tuning

Researchers analyze why Preventative Steering protects LLMs against malicious fine-tuning, finding active adaptation drives protection, and propose Progressive Intensity Scheduling.

The paper studies Preventative Steering, a training-time defense that injects undesirable-trait persona vectors during adversarial fine-tuning and removes them at evaluation time. Temporal analysis shows protection emerges from an early compensatory adaptation phase followed by a steady-state phase, with attention output projections acting as the dominant residual-write route for defensive updates. Intervention Delta Preservation experiments show that preserving or reinjecting weight offsets fails to maintain protection, indicating reliance on active adaptation rather than a static defense. The proposed Progressive Intensity Scheduling improves safety robustness on Qwen2.5 and Gemma-3 while reducing harmful trait expression.

arXiv cs.CR·Sep 9, 13:19 UTC · 6d agoAI safety & security1

32

A Unified and Constrained View of Regularization-Based Robust Reinforcement Learning

Paper unifies regularization-based robust RL methods via new performance-gap upper bounds and jointly learned Lagrange multipliers.

The authors derive new upper bounds on the gap between nominal and worst-case deep RL policies, each expressible as an existing regularization objective plus a KL-divergence penalty. Robust training is reformulated as constrained optimization, where prior methods correspond to a fixed Lagrange multiplier. The multiplier is instead updated jointly with the policy, auto-tuning the regularization weight. Adversarial evaluations across several continuous control tasks validate the theory.

arXiv cs.AI / cs.LG / cs.CL·Sep 11, 16:45 UTC · 4d agoAI research1

20

Adversarial Testing of Automated Program Repair Agents for Security Vulnerabilities

SWEADV benchmark shows adversarial issue descriptions make LLM program-repair agents write insecure fixes in 51.7% of cases, evading most detection tools.

Researchers built SWEADV, a benchmark of 750 adversarial issue descriptions derived from 150 SWE-bench Verified repair tasks, covering command execution, deserialization, path traversal, denial of service, and weak hashing attack types. Tested on mini_swe agents backed by GPT-5-Mini, MiniMax-M2.5, and DeepSeek-R, adversarial descriptions induced malicious behavior with successful repair in 51.7% of cases. Detection was weak: LLM-as-judge pre-repair screening reached only 62.3% accuracy, and post-repair detection via static analysis and LLM-as-judge achieved just 39.4% and 55.4%.

arXiv cs.CR·Sep 14, 17:53 UTC · 1d agoAI safety & security1

52

Do Input-Level Defenses Transfer to Observation-Level Attacks on VideoLLMs?

A systematic study shows input-level adversarial defenses provide inconsistent, often near-zero protection against observation-level attacks on video LLMs.

Researchers introduce DefTEval, a controlled framework testing eleven input-level defenses against five attack types across five video LLMs. Harmful-content detection rates are frequently near zero, and defenses fail even when attacks embed harmful signals in every sampled frame. Token compression discards localized safety features and modality fusion down-weights weakened visual signals, with defense outcomes dominated by model architecture rather than the defense method.

arXiv cs.CR·Sep 8, 06:59 UTC · 8d agoAI safety & security

40

Windows Malware Detector as a Compound AI System: Trade-Offs in Accuracy, Efficiency, and Adversarial Robustness

Researchers model industrial Windows malware detection as a Compound AI System, quantifying trade-offs between detection accuracy, efficiency, and adversarial robustness under tiered attacker knowledge.

Industrial Windows malware detectors combine rule-based mechanisms with ML-based static and dynamic analyses, but their architectures are rarely publicly disclosed. The authors propose a methodology balancing detection performance, computational cost, and robustness, plus system-level threat models capturing whole-pipeline evasion rather than isolated components. Experiments on real-world data show training time reductions with marginal detection loss, while more knowledgeable attackers craft increasingly effective adversarial examples. The paper derives deployment guidelines from the observed efficiency-robustness trade-off.

arXiv cs.CR·Sep 8, 08:04 UTC · 8d agoResearch

42

Certifying Adversarial Robustness of Quantum Classifiers under Known-Readout Query Access

Framework certifies adversarial robustness of quantum classifiers using only measurement statistics and finite-shot outcomes, demonstrated on IBM Quantum hardware.

The paper introduces a measurement-only certification framework for adversarial robustness of quantum classifiers under known-readout query access, requiring no tomography, parameters, or gradients. It returns a lower bound ruling out untargeted errors within a radius and an attack-independent upper bound witnessing an adversarial state, both estimable with finite-sample guarantees. Evaluations show the lower bound tracks exact optima on tractable instances while the upper bound stays informative when standard attacks fail. The method was validated on IBM Quantum hardware using 40 executions of two 8-qubit quantum neural networks.

arXiv cs.CR·Sep 10, 14:45 UTC · 5d agoResearch

27

RAPID: A Real-Time Defense Against Unauthorized Model Distillation for Text-to-Image Services

RAPID embeds defensive perturbations in a T2I model's shared VAE decoder to block unauthorized black-box distillation in real time.

The paper defends text-to-image services against model theft via black-box output-based distillation, where adversaries collect prompt-image pairs to train substitute models. RAPID integrates defensive perturbations into the shared VAE decoder using self-referenced latent maximization plus reconstruction-guided color regularization, avoiding costly sample-wise online optimization. Across four T2I models and four datasets versus five baselines, it consistently degrades substitute-model generation quality while preserving visual fidelity.

arXiv cs.CR·Sep 14, 16:11 UTC · 1d agoAI safety & security

35

InceptionRAG: Stealthy Poisoning Attack Against Retrieval-Augmented Generation

InceptionRAG fragments malicious payloads into dormant passages that trigger LLMs to self-deduce misinformation via multi-hop reasoning, bypassing existing RAG poisoning defenses.

Researchers introduce InceptionRAG, a stealthy corpus poisoning attack against retrieval-augmented generation that splits a malicious payload into a chain of individually harmless dormant passages. When retrieved together, the passages induce LLMs to self-deduce target misinformation through multi-hop reasoning, achieving over 80% attack success rate across three datasets and three LLMs under rigorous adversarial constraints. A zeroth-order suffix optimization (ZOSO) method automates authoritative suffix generation in black-box settings. The authors also propose HODOR, a document isolation defense that decouples adversarial logical dependencies.

arXiv cs.CR·Sep 15, 08:22 UTC · 1d agoResearch

54

Recognition-Refusal Misalignment in LLMs: Why Models Answer Structurally Unanswerable Questions

A linear hidden-state direction encodes question impossibility in 1.7B-70B LLMs, but misalignment with the safety-refusal pathway explains why models answer unanswerable questions.

The study examines why instruction-tuned LLMs from 1.7B to 70B parameters answer structurally unanswerable math and code questions instead of abstaining. A single linear direction in the hidden state separates answerable from impossible prompts, showing models represent impossibility before generation, but this direction is nearly orthogonal to the canonical safety-refusal direction. Generation-time steering along the recognition direction changes invalidity-aware behavior dose-responsively, and the geometry is present even at the pretraining endpoint, indicating a routing failure rather than an encoding failure.

Hugging Face daily papers·Aug 28, 20:00 UTC · 18d agoAI safety & security

30

Nearly Tight Rademacher Bounds for Sparsely Activated Neural Networks

Theory paper derives nearly tight Rademacher complexity bounds for sparsely activated one-hidden-layer ReLU networks.

Building on Awasthi et al. (COLT 2024), the authors bound statistical complexity for networks where each input activates at most k of s hidden units. A support-preserving cover and normalized chaining argument remove the explicit dimension factor, with matching lower bounds up to logarithms. They also derive agnostic minimax excess-risk bounds of order min{1, sqrt(s/(km))} for a normalized bounded loss and show bias bounds comparable to WR restore worst-case rates even on domains where sparsity holds globally.

arXiv cs.AI / cs.LG / cs.CL·Sep 8, 17:53 UTC · 7d agoAI research

16

What Makes Adversarial Examples Transfer Across Deepfake Detectors?

A controlled study of 60 deepfake detectors shows adversarial example transfer depends heavily on source-target compatibility, with source averaging understating vulnerability.

The study evaluates adversarial example transferability across 60 deepfake detectors spanning six backbones, two pretraining regimes, and five training-data configurations, using AutoAttack (AA) and Carlini-Wagner with Expectation over Transformation (CW-EOT). Transfer rises sharply when source and target share an exact backbone, architecture family, pretraining regime, or training data, with the dominant factor depending on the attack. Mean attack success rate is 7.21% under AA and 19.52% under CW-EOT for single sources, while a multi-source oracle reaches 64.48% after excluding exact matches, showing source averaging can substantially understate target vulnerability. The authors release 240,000 adversarially perturbed images, pairwise transfer results, detector configurations, and evaluation code.

arXiv cs.CR·Sep 9, 10:28 UTC · 7d agoAI safety & security

40

Knowing What Not to Answer: Selective Non-Compliance in Vision-Language Models

Researchers introduce KoNA, a benchmark exposing vision-language models' failures at selective non-compliance, plus fine-tuning that improves refusal and abstention accuracy.

KoNA is a benchmark for evaluating selective non-compliance in vision-language models across five categories: False Premise, Visual Inaccessibility, Universal Unknown, Task Feasibility and Safety. It tests both query-level and component-level non-compliance using paired single and compound queries, and evaluations across diverse VLMs show models often fail to refuse, correct or abstain appropriately, with failures worsening on compound queries. Fine-tuning VLMs on KoNA examples substantially improves non-compliance accuracy while largely maintaining performance on fully answerable tasks.

Hugging Face daily papers·Sep 3, 20:00 UTC · 12d agoAI research1

45

ACEA: An Adversarial Co-Evolution Arena for Head-to-Head Red-Team and Blue-Team LLM Testing

ACEA is a pluggable arena scoring LLM red-team attackers and blue-team defenses head-to-head with an LLM judge and verifiable leakage ground truth.

ACEA connects pluggable red- and blue-team adapters to a shared target LLM through the model-agnostic ASAP HTTP protocol and scores attack and defense rates per adversarial round. Canonical seeded secrets provide verifiable ground truth that separates real leakage from hallucination, and attacks are delivered to the target even when blocked to measure raw potency. The platform adds real-time battle visualization, failure-localizing end-of-battle reports, and an optional in-context improvement loop that feeds advisory hints between rounds.

arXiv cs.CR·Sep 8, 04:57 UTC · 8d agoAI safety & security1

32

MarkSec: Capability-Aware Evaluation of Adversarial Attacks Against LLM Watermarks

MarkSec unifies evaluation of stealing, scrubbing, and spoofing attacks against LLM watermarks with quality-constrained success metrics under shared reporting protocols.

MarkSec is a framework unifying analysis of stealing, scrubbing, and spoofing attacks against LLM watermarks under shared detector calibration, metric definitions, and reporting protocols. It introduces a quality-constrained attack success metric that jointly assesses attack effectiveness and text quality. Experiments across representative watermark families, attacks, LLMs, and datasets show that attacks strongest by watermark removal alone can fall behind general rewriting when success requires acceptable text quality, and stealing-based scrubbers often underperform the best general-scrubbing baselines.

arXiv cs.CR·Sep 15, 06:02 UTC · 1d agoResearch

50

Beyond Solver Verdicts: Generative Reward Models for Autoformalization

Researchers introduce Generative Verification (GenV), a generative reward model achieving 0.961 AUROC in detecting unfaithful autoformalization that preserves solver verdicts.

The paper formalizes Verdict-Preserving-Unfaithfulness (VPU), a failure mode in neurosymbolic autoformalization where an incorrect encoding executes successfully and matches the expected solver verdict, and proves verdict-only verification is bounded to chance-level detection. The proposed Generative Verification (GenV) distills an offline Z3-equivalence oracle into a reference-free, continuous reference-equivalence score within the language model's vocabulary space. The oracle-mined verifier (GenV+HN) achieves 0.961 AUROC, generalizes zero-shot across unseen translators and formal styles, and yields an 11.3-point downstream accuracy gain in agentic test-time compute allocation. Mechanistic analysis with decision-projected logit lenses and sparse autoencoders shows the generative readout extracts precise spatial error coordinates without explicit localization training.

Hugging Face daily papers·Sep 9, 20:00 UTC · 6d agoAI research1

35

HoneyRoute: Honeypot-Model Routing for Adversarial LLM Serving

HoneyRoute detects malicious LLM serving requests and diverts them to a honeypot model, reaching F1 0.911 with 38 ms median added latency.

HoneyRoute is an inference-serving layer pairing a streaming router (a frozen 0.8B embedding backbone with per-domain MLP heads) with a dual-implementation honeypot and an analysis loop that converts trapped interactions into attacker fingerprints for router retraining. On a production trace plus a seven-domain attack corpus it matches 96% of a two-tier guard-LLM cascade's F1 at 1/385th of its latency with 0% evasion under 13 adversarial transformations. Diverting malicious traffic cuts production token consumption under GCG-suffix flooding by 97.8%, and loop training raises detection F1 to 0.933.

arXiv cs.CR·Sep 8, 06:26 UTC · 8d agoAI safety & security

38

Robust Coverless Linguistic Steganography via Sentence Embedding Space with Global Resynchronization

Researchers propose a coverless steganographic framework encoding messages as hierarchical clustering paths in sentence embedding space with a Global Resynchronization Mechanism for robustness.

An arXiv paper proposes encoding secret messages as hierarchical clustering paths in the sentence embedding space rather than token space, improving decoding stability against word- and sentence-level textual perturbations. A Global Resynchronization Mechanism (GRM) reframes variable-length bitstreams as discrete symbols anchored to semantic subspaces to prevent bit-slippage. Experiments show substantial robustness improvements while maintaining embedding capacity and resistance to statistical analysis.

arXiv cs.CR·Sep 4, 10:22 UTC · 12d agoResearch

18

Benign Loss Landscapes Can Coexist with Worst-Case Hardness

Theory paper shows tree tensor networks contain worst-case hard targets yet benign loss landscapes, with difficulty arising from degenerate saddles.

The paper studies tree tensor networks (TTNs), which generalize deep linear networks and Tucker decompositions and embed arbitrary read-once Boolean formulas. It proves that every local minimum that is minimum-norm is global for every realizable target, so bad local minima do not distinguish typical from worst-case problems. Instead, learning difficulty arises from high-order degenerate saddle points caused by rank-deficiency, illustrated via a parity function case study, linking landscape geometry to computational hardness.

arXiv cs.AI / cs.LG / cs.CL·Sep 11, 16:57 UTC · 4d agoAI research

16

Studying Image Tokenizers as Visual Languages in Unified Multimodal Models

A controlled autoregressive testbed shows validation losses must be analyzed per task, and image tokenizer choice affects joint multimodal text modeling.

Researchers built a pure-autoregressive testbed to study image tokenizers as the 'visual language' of unified multimodal models, tracking task-specific validation losses during multimodal continual pretraining across text, image, text-to-image (T2I), and image-to-text (I2T) prediction. They found that losses exhibit distinct scaling behavior per task and rank tokenizers differently, and that I2T loss over a shared text vocabulary gives a more consistent loss–performance signal than T2I loss. Better reconstruction does not necessarily yield lower task-specific losses or stronger downstream performance, and tokenizer choice can affect text modeling under joint optimization. Case studies examine the discriminator, semantic supervision, and vocabulary size design axes.

Hugging Face daily papers·Sep 7, 20:00 UTC · 8d agoAI research1

40

Studying Image Tokenizers as Visual Languages in Unified Multimodal Models

A controlled pure-autoregressive testbed shows task-specific validation losses rank image tokenizers differently, with I2T loss the most consistent signal.

Researchers built a controlled pure-autoregressive testbed and tracked task-specific validation losses during multimodal continual pretraining across text, image, text-to-image (T2I), and image-to-text (I2T) prediction. They find losses should be analyzed per task because they exhibit distinct scaling behavior and rank tokenizers differently, and that the loss-performance relationship depends on the predicted token space. I2T loss, computed over a shared text vocabulary, correlates consistently with both generation and visual understanding performance after supervised finetuning. Case studies revisit the discriminator, semantic supervision, and vocabulary size as tokenizer design axes.

arXiv cs.AI / cs.LG / cs.CL·Sep 8, 17:57 UTC · 7d agoAI research1

30

Perturbation Probing: A New Diagnostic for the Fragility of LLM Safety

Unit 42 research shows LLM safety refusals concentrate in a thin neural layer, motivating external, multi-layered AI security controls.

Palo Alto Networks Unit 42 introduces Perturbation Probing, a diagnostic technique for measuring the fragility of LLM safety mechanisms. The research finds that safety refusal behavior is localized within a thin neural layer, implying small perturbations can undermine built-in refusals. The authors argue this motivates external, multi-layered security defenses on top of model-internal safety training.

Palo Alto Unit 42·Aug 28, 22:00 UTC · 18d agoAI safety & security

45

Training and Finetuning Multi-Vector Embedding Models with Sentence Transformers

Hugging Face published a tutorial on training and finetuning multi-vector embedding models using the Sentence Transformers library.

Hugging Face's blog walks through training and finetuning multi-vector embedding models with Sentence Transformers. Multi-vector approaches store multiple vectors per document to support late-interaction retrieval. The post is a practical guide for developers building retrieval pipelines with the library.

Hugging Face Blog·Aug 26, 00:00 UTC · 21d agoAI tools & infra1

20

Don't Send What You Don't Need: Question-Guided Token Pruning as a Privacy Defense for Vision-Language Models

QPriv-VL prunes privacy-sensitive visual tokens in federated/split VQA, cutting membership-inference success on VQA-RAD from 0.99 to 0.76-0.79 using ~40% of tokens.

The paper proposes QPriv-VL, a question-guided token-pruning framework for federated, split, and U-shaped split learning that suppresses privacy-sensitive visual patches before transmission. Its Dynamic Threshold Predictor combines cross-modal question relevance with frozen DINOv2-derived sensitivity to compute a per-sample pruning ratio and retention mask in one forward pass, without sensitivity labels. Evaluated on GQA, OK-VQA, VQAv2, SLAKE, VQA-RAD, and PathVQA against FSHA, FORA, iDLG, and attribute-inference membership inference attacks, it matches or beats fixed-ratio pruning. On VQA-RAD it reduces membership-inference success from 0.99 to 0.76-0.79 while preserving competitive accuracy with about 40% of the original token budget.

arXiv cs.CR·Sep 14, 14:48 UTC · 1d agoResearch

35

On Identifying Adversarial Intent Injection in AI-Native 6G Networks

Dual-path CNN and AutoEncoder framework detects adversarial intent injection in AI-native 6G networks, reaching 0.97 accuracy and 0.98 F1.

The paper defines a fine-grained threat model for adversarial intent injection in AI-native 6G intent-based networking, where malicious policies are disguised within benign intent flows. It evaluates four injection strategies: stealth-mode, random distribution, increasing frequency, and decreasing frequency. A dual-path detection framework combines a CNN using TF-IDF features for supervised detection with an AutoEncoder trained only on benign data for one-class detection, reaching 0.97 accuracy and 0.98 F1-score, roughly 9% and 36% gains over the state-of-the-art baseline.

arXiv cs.CR·Sep 10, 19:23 UTC · 5d agoResearch

35

What Breaks Local Watermarks? A Robustness Benchmark for Local Invisible Image Watermarking

First systematic robustness benchmark of five local invisible image watermarking methods across 55 transformations finds all are vulnerable, with inpainting and geometric misalignment completely breaking payload…

The paper presents the first systematic robustness benchmark for local invisible image watermarks, covering 55 image transformations across signal distortions, coordinate alignment changes, indirect local edits, and direct watermark edits. It evaluates five methods: MaskWM, WAM, OmniGuard, TrustMark, and PixelSeal, all supporting localization natively or with minimal adaptation. Results show every method is vulnerable to some transformation; MaskWM offers the strongest payload recovery and localization but the lowest clean-image quality, and synchronization further improves its recovery under geometric transformations. Geometric misalignment and generative local edits such as inpainting and outpainting can completely impair payload recovery, while signal distortions are often tolerated.

arXiv cs.CR·Sep 15, 08:30 UTC · 1d agoResearch

42

Competence-Gated Pooling of Language Models and Priors for Event Forecasting

Paper proposes a competence gate pooling language model forecasts with external priors, improving Brier score from 0.0771 to 0.0732 across 2,357 binary questions.

The paper defines a language model's relative competence as its marginal value beyond an available external forecast, and derives conditions under Brier loss where model disagreement improves that forecast. A competence gate estimates domain-level source weights from resolved outcomes, shrinks uncertain estimates toward a global weight, and recalibrates the pooled forecast. Across 2,357 resolved binary questions and five language models, it improves the external baseline from 0.0771 to 0.0732 Brier and beats global forecast combinations, though it defers to the market on ForecastBench. Across four Qwen models, verbal confidence failed to identify when the model outperformed the external forecast, while outcome-estimated competence supported better abstention.

Hugging Face daily papers·Sep 9, 20:00 UTC · 6d agoAI research

38

Understanding the Impact of Model Pruning on Long-Tail Forgetting and Explanation Reliability in Medical Imaging

Systematic study finds model pruning causes frequency-dependent long-tail forgetting in medical imaging and that gradient-informed methods best preserve explanations.

Across two long-tailed medical imaging datasets, two CNN architectures, four pruning methods, and sparsity up to 95%, the study measures predictive performance, explanation stability, and faithfulness. Rare classes degrade earlier and more severely than frequent ones, while explanation reliability depends mainly on the pruning strategy, with gradient-informed methods degrading least. Mechanistic analysis ties explanation collapse to loss of class-discriminative gradients rather than vanishing feature activations, recommending class- and explanation-aware evaluation of compression.

arXiv cs.AI / cs.LG / cs.CL·Sep 7, 17:44 UTC · 8d agoAI research

27

Implementing a White-Box Undetectable Backdoor for Random Fourier Features

Researchers implement Goldwasser's CLWE-based undetectable backdoor for Random Fourier Features models in numpy/scipy, confirming practical realizability with no detectable differences from clean models.

The paper provides an end-to-end implementation of the Goldwasser et al. white-box undetectable backdoor for models trained with the Random Fourier Features algorithm, using only numpy and scipy. It derives two samplers for the core GP_d(b_k) distribution: a rejection-sampling proxy and an exact closed-form sampler verified against its analytic form. Statistical indistinguishability tests covering weight-space and functional black-box comparisons found no detectable difference between backdoored and clean models across sparsity ratios. The underlying lattice hardness reduction was not reproduced, and the work demonstrates the threat is realizable with commodity scientific-computing tools rather than specialized cryptographic infrastructure.

arXiv cs.CR·Sep 14, 22:18 UTC · 1d agoResearch

50

Algorithmic stability via ensembling

Theoretical work derives a general framework quantifying stability guarantees for averaging-based ensembles under arbitrary data perturbations via covariance operator norms.

The paper develops a framework for quantifying algorithmic stability of ensembling strategies defined via averaging, for varied types of data perturbation. The main result bounds the stability of the ensembled algorithm in terms of the norm of a covariance operator describing the ensembling process. The framework yields interpretable insights across practical perturbation examples and provides sharper guarantees than those derived from differential privacy considerations.

arXiv cs.AI / cs.LG / cs.CL·Sep 9, 16:39 UTC · 6d agoAI research

22