Search: “toxicity-detection”

Pick Your Poison: Learning to Select Poison Sets for Stronger LLM Backdoor Attacks

SAILS learns to select poison sets for LLM backdoor attacks, showing attack success ranges 3% to 80% at fixed poison counts across LLaMA-3-8B settings.

The paper shows existing backdoor evaluations that randomly sample a fixed number of poisoned examples severely underestimate worst-case vulnerability: across three LLaMA-3-8B settings, attack success ranges from 3% to 80% depending only on which poison set is chosen. SAILS formalizes poison selection as oracle-budgeted set optimization, learning a set scorer from a few hundred finetune-and-evaluate runs to rank millions of candidate sets and audit a small shortlist. It improves held-out attack success by 30 percentage points over the strongest influence baselines and transfers from small-scale to full-scale finetuning, extending to code-generation, agentic, and API-only backdoors.

arXiv cs.CR·Sep 14, 04:44 UTC · 2d agoAI safety & security 2 sources

"Shut Up and Let Me Enjoy My Otome": Understanding and Measuring the Toxicity in Otome Game Communities

First large-scale study finds 22.20% of Weibo otome game posts toxic versus 3.71% on Reddit, with LLM detectors reaching 0.82 F1.

Researchers present the first large-scale measurement of toxicity in otome game communities, introducing OtomeSCAN, which collected and analyzed 620,045 posts from Weibo and Reddit over 18 months. They manually annotated 4,308 posts, identified eight target groups, and evaluated seven toxicity detectors, with their best LLM-based model reaching F1-scores of 0.82 on Weibo and 0.78 on Reddit. The study found 22.20% of Weibo posts were toxic versus 3.71% on Reddit, and toxicity rose to 37.09% within 72 hours during an external attack on Weibo. The authors also flagged 191 potential-coordination clusters, 64.40% of which targeted game developers.

arXiv cs.CR·Sep 7, 21:44 UTC · 8d agoResearch

Bag of Tricks or Bag of Myths? Reducing Modeling Complexity with Task Knowledge in Explainable Suicide Risk Assessment

Researchers audit 31 NLP techniques for clinician-annotated suicide risk prediction, finding only 5 of 31 comparisons yield reliable gains.

A study of 1,635 clinician-annotated social media posts ran roughly 300 controlled experiments across 7 methodological families, auditing techniques such as model scaling, synthetic data, ensembling, and threshold tuning under severe class imbalance. The proposed system reformulates risk factor prediction as entailment between posts and codebook definitions, using architecturally diverse ensembles with class-balanced training and deployment-consistent calibration. It scores 0.8203 for risk, 0.7953 for evidence, and 0.7045 macro-F1 for factors, ranking third among 53 teams.

arXiv cs.AI / cs.LG / cs.CL·Sep 7, 17:14 UTC · 8d agoAI research

The History Is the Detector: Executing CVE Patch History, End-to-End

BUGSTONE-E2E converts CVE patch history into executable LLM-guided detection rules, yielding 1,033 rules and 644 runtime-verified findings across 14 programs.

The BUGSTONE-E2E framework mines reusable detection rules from verified fixing commits, organized by CWE and language, and applies them through a funnel pipeline that escalates from Tree-sitter anchors and lightweight heuristics to LLM-based agent inspection, runtime verification, and scope-checked patch generation. Built from 19,325 high-severity CVEs published 2022-2026, it produced 1,033 detection rules spanning 56 CWE families, packaged into 172 skills. Applied across 14 programs, it generated runtime evidence for 644 findings, demonstrating that vulnerability history can drive reproducible detection and repair.

arXiv cs.CR·Sep 4, 16:40 UTC · 11d agoResearch

An Empirical Analysis of ReDoS Vulnerabilities and ReDoS Detection Tools

Study of NVD data finds ReDoS vulnerabilities growing more prevalent and more likely to be exploited, while five detection tools disagree substantially.

The study compares five publicly available ReDoS detection tools and one regex correction tool across three datasets. An empirical analysis of all ReDoS vulnerabilities reported to the NVD finds they are becoming more prevalent and are much more likely to be exploited than non-ReDoS weaknesses. The detection tools exhibited substantial disagreement on whether a given regex is vulnerable.

arXiv cs.CR·Sep 9, 15:09 UTC · 7d agoResearch

BdThemes Supply Chain Attack Poisons JSON to Create Rogue WordPress Admins

The Hacker News·Aug 11, 05:48 UTC · Aug 11, 2026VulnerabilityCVE-2026-18072CVE-2026-64638

When the prompt becomes the payload: A practical pen-testing guide for GenAI, LLM and RAG applications

CSO Online publishes a practical penetration-testing guide for GenAI, LLM, and RAG applications, covering prompt injection, retrieval poisoning, and tenant isolation testing.

The guide frames LLM applications as attack graphs spanning prompts, retrieval layers, vector stores, tools, identities, and downstream APIs, arguing that conventional web testing misses instruction-vs-data channel risks. It builds on OWASP prompt injection guidance (direct vs. indirect injection) and NIST's 2025 adversarial machine-learning taxonomy, noting that RAG and fine-tuning do not remove injection risk. Recommended practices include documenting trust transitions across components, using canaries and synthetic records to avoid test side effects, running multi-turn and obfuscated injection campaigns, and verifying chains from poisoned documents to observable state changes. It also details testing RAG pipelines via controlled document poisoning across metadata, OCR layers, and code comments, plus cross-tenant isolation checks on retrieved document IDs.

CSO Online·Sep 9, 10:00 UTC · 7d agoAI safety & security1

Biology-in-the-loop: Amortized Adaptive Hit Discovery in CRISPR Screens

Researchers release AssayBench-Loop, a 1,389-screen CRISPR benchmark, and AssayLoop, a framework that learns adaptive hit discovery policies.

The paper introduces AssayBench-Loop, a large-scale benchmark of 1,389 CRISPR screens across five phenotype categories for adaptive hit discovery under budget constraints. It also introduces AssayLoop, which combines AssayFormer, a transformer-based amortized acquisition policy trained across historical screens, with LLM-derived biological priors via an adaptive handoff. On temporally held-out screens, AssayLoop achieves 5.67-fold enrichment over random selection and recovers 27.7% of hits after assaying roughly 5% of the candidate library, outperforming existing adaptive-design methods and standalone LLMs.

arXiv cs.AI / cs.LG / cs.CL·Sep 10, 17:45 UTC · 5d agoAI research1

CTEM Is Not About the Stages. It’s About the Outcome.

Horizon3 argues CTEM programs should measure continuously reduced exposure rather than mapping technologies to Gartner's five stages.

Horizon3 contends that Continuous Threat Exposure Management should be judged by one outcome: continuously reducing attacker-reachable exposure, not by mapping a technology to each of Gartner's five stages. The post argues validation and verification, not visibility or closed tickets, provide evidence that attack paths are actually broken. It describes a Discover, Validate, Prioritize, Remediate, Verify, Repeat motion as its operationalization of CTEM.

Horizon3.ai·Sep 2, 13:11 UTC · 14d agoIndustry

Perturbation Probing: A New Diagnostic for the Fragility of LLM Safety

Unit 42 research shows LLM safety refusals concentrate in a thin neural layer, motivating external, multi-layered AI security controls.

Palo Alto Networks Unit 42 introduces Perturbation Probing, a diagnostic technique for measuring the fragility of LLM safety mechanisms. The research finds that safety refusal behavior is localized within a thin neural layer, implying small perturbations can undermine built-in refusals. The authors argue this motivates external, multi-layered security defenses on top of model-internal safety training.

Palo Alto Unit 42·Aug 28, 22:00 UTC · 18d agoAI safety & security

TIER: Threat Implicitness Benchmark for Evaluating LLM Safety Behaviors

TIER benchmark shows LLM safety behaviors shift gradually across threat implicitness levels, with jailbreaks exposing the largest robustness gaps.

The TIER benchmark evaluates LLM safety behaviors across four risk domains and four threat levels, from explicit harmful requests to sophisticated jailbreaks, using a six-label behavior scale and two independent LLM judges. Experiments on six open-weight LLMs show safety behaviors evolve gradually across threat levels rather than flipping from refusal to compliance. Models with similar Attack Success Rates can exhibit distinct response distributions, arguing for behavior-aware safety evaluation.

arXiv cs.CR·Sep 4, 13:13 UTC · 12d agoAI safety & security

Do Input-Level Defenses Transfer to Observation-Level Attacks on VideoLLMs?

A systematic study shows input-level adversarial defenses provide inconsistent, often near-zero protection against observation-level attacks on video LLMs.

Researchers introduce DefTEval, a controlled framework testing eleven input-level defenses against five attack types across five video LLMs. Harmful-content detection rates are frequently near zero, and defenses fail even when attacks embed harmful signals in every sampled frame. Token compression discards localized safety features and modality fusion down-weights weakened visual signals, with defense outcomes dominated by model architecture rather than the defense method.

arXiv cs.CR·Sep 8, 06:59 UTC · 8d agoAI safety & security

PIDS-Bench: Evaluating Prompt-Injection Detectors Under Over-Defense, Obfuscation, and Distribution Shift

PIDS-Bench shows prompt-injection detectors scoring F1 above 0.98 still misclassify about one-third of external benign security-adjacent prompts, revealing provenance-sensitive over-defense.

PIDS-Bench is a frozen multi-axis benchmark that jointly evaluates prompt-injection detectors on attack detection and benign false-positive behavior at fixed thresholds, spanning in-distribution inputs, hard-benign prompts, obfuscated attacks, and domain/structural distribution shifts. It evaluates seven detectors plus a rule-based lower-bound reference. A detector exceeding F1 = 0.98 on held-out data still misclassifies roughly one-third of an externally-sourced benign security-adjacent subset, and no internal detector reaches F1 >= 0.95 with hard-benign FPR <= 0.10 on the stress distribution. Hard-negative augmentation nearly eliminates over-defense on curated stress inputs but leaves it intact on externally-sourced prompts, a pattern termed provenance-sensitive over-defense.

arXiv cs.CR·Sep 14, 04:28 UTC · 2d agoAI safety & security

SAFIRE: Safety-Critical Benchmark for Fine-grained Fire and Smoke Understanding in Multimodal LLMs

SAFIRE, an 83K-image fire and smoke benchmark, shows open-source multimodal LLMs average only 61.9% accuracy on safety-critical fire reasoning.

SAFIRE is a large-scale benchmark for fire-smoke understanding in multimodal LLMs with 83K captioned images across 20 scenarios and 193K multiple-choice VQA questions spanning 10 evaluation dimensions from perception to higher-order reasoning. Annotations were built via a GPT-5.4-assisted multi-stage pipeline with MLLM majority voting. Ten open-source MLLMs (8B-38B) average 61.9% accuracy, exposing major gaps in safety-critical reasoning. Adapting vision encoders on 7% of the domain data raises fire-scene classification from 20.1% to 64.5%.

arXiv cs.AI / cs.LG / cs.CL·Sep 7, 17:59 UTC · 8d agoAI research

UltraViolet Cyber Equinox measures detection coverage against MITRE frameworks

UltraViolet Cyber launches Equinox, an AI-automated platform mapping SIEM and EDR detection coverage against MITRE ATT&CK and ATLAS frameworks.

UltraViolet Cyber announced Equinox, a detection engineering platform from its TIDE team that maps existing detections and log sources against MITRE ATT&CK and MITRE ATLAS. It performs coverage analysis in under 30 minutes and recommends vendor or custom detections to fill gaps, with engineers reviewing and backtesting each recommendation. In one customer trial, mapped technique coverage rose from 59 of 222 techniques (26.6%) to 136 of 222 (61.3%) without increasing alert volume.

Help Net Security·Sep 15, 13:33 UTC · 1d agoTools2

MemSentry: A Framework for Detecting Persistent Memory Poisoning in Agentic AI

MemSentry intercepts persistent-memory writes in agentic AI to catch memory poisoning, reaching 91.7% accuracy with SBERT+LR classification.

Memory poisoning lets adversaries plant crafted content in an agent's long-term memory to suppress security alerts, enable privilege escalation, or override policies without modifying model weights or system prompts. The paper presents MemSentry, a configuration-driven framework that evaluates proposed persistent-memory writes on source trust, semantic risk, attack radius over a dependency DAG, access risk, and a signed security-state delta to issue deterministic Accept, Review, or Quarantine decisions. Across 1,000 GPT-4-generated scenarios on a 20-asset dependency DAG, SBERT+LR achieved 91.7% accuracy and 0.908 macro-F1, all four classifiers detected 100% of external quarantine-class threats, and verified-insider writes are escalated for human review rather than auto-quarantined.

arXiv cs.CR·Sep 8, 13:40 UTC · 8d agoAI safety & security

Why AI raises the stakes for exposure validation

Fal.Con 2026 commentary argues AI accelerates vulnerability discovery and exploitation, making evidence-based exposure validation essential for defender prioritization.

CSO Online reports on the exposure-validation theme at CrowdStrike's Fal.Con 2026 conference, where CEO George Kurtz described AI as the new cyber battlefield and emphasized AI red teaming and continuous security. The piece argues that as AI speeds up vulnerability discovery and exploitability analysis on both sides, teams must determine which exposures are actually exploitable in their environments—chained weaknesses, credential abuse, lateral movement, privilege escalation—rather than chasing theoretical risk. It points readers to Horizon3's conference perspective.

CSO Online·Sep 11, 21:49 UTC · 4d agoIndustry

Reference-Based Bias Detection in LLMs via Relative Representations of Hidden States

Researchers propose auditing LLM bias via relative hidden-state representations, detecting bias increases with 3-50x less compute than output-level benchmarks.

The paper introduces a reference-based bias auditing method that compares hidden-state representations across model variants, such as before and after fine-tuning, by encoding sentences relative to a fixed anchor set. The resulting Representational Bias Shift (Delta-B) correlates with output-level bias change in 15 of 18 tested settings, reaching |r| = 0.84 under full fine-tuning across WildGuardMix, DecodingTrust, and ToxiGen benchmarks. Thresholding Delta-B detects checkpoints whose bias increased with ROC AUC between 0.65 and 0.99 and beats a SEAT-based baseline, while auditing a model in about three minutes with 3-50x less compute.

Hugging Face daily papers·Sep 8, 20:00 UTC · 7d agoAI research

ToxicRAG: Compromising Retrieval-Augmented Generation Systems via Single-Shot Knowledge Poisoning Attacks

ToxicRAG shows a single narrative-form poisoned document can steer RAG answers, achieving 0.61-0.91 attack success rates across four LLMs.

The attack injects one document per target question written as a coherent knowledge-update narrative that acknowledges the previously accepted answer, introduces fabricated events that appear to invalidate it, and attributes the attacker-chosen answer to purported authorities. An optional answer-focused self-validation loop revises candidates when a surrogate LLM fails to reproduce the target answer. Across 100 target questions each from Natural Questions, HotpotQA, and MS-MARCO, with four victim LLMs and four dense retrievers, ToxicRAG achieves attack success rates of 0.61-0.91 and matches or exceeds the strongest baseline by 0 to 11 percentage points.

arXiv cs.CR·Sep 10, 04:42 UTC · 6d agoAI safety & security1

Why Is SHAP Not a Reliable Standalone Explanation Framework for Malware Detection?

arXiv paper shows SHAP gives unreliable standalone explanations for malware detection, with attribution dilution and sign reversal in dependent PE feature spaces.

The paper argues SHAP's formal guarantees are insufficient for reliable malware interpretation because the explained feature-coalition game is fixed only by analyst choices, not by malware behavior in the data. In static Portable Executable feature spaces, dependent feature groups cause conditional SHAP to dilute credit by a factor of 1/m across redundant features, attribute importance to features the model never uses, and even reverse attribution signs; interventional SHAP queries off-manifold coalitions no real executable exhibits. Experiments on EMBER-2018, EMBER-2024, and BODMAS with fixed LightGBM and XGBoost detectors confirm these effects. The authors position SHAP as a limited diagnostic requiring explicit data-distribution statements and domain validation, not a standalone explanation framework.

arXiv cs.CR·Sep 4, 02:00 UTC · 12d agoResearch1

Target leakage, not model class, explains reported accuracy in survey-based cardiovascular screening: a leakage-tiered audit of glass-box and tabular foundation models

Audits of 10 classifiers on BRFSS show target leakage, not model class, drives the reported 0.89 AUROC in survey-based cardiovascular screening.

The study benchmarks ten model classes, including glass-box and tabular foundation models, for prevalent myocardial infarction on 442,067 respondents of the 2022 BRFSS across five feature tiers of decreasing leakage risk. Removing two post-diagnostic features costs every model 0.049-0.051 AUROC and collapses performance into a 0.0045-wide band, and the explainable boosting machine matches all alternatives within 0.005 while scoring roughly 104x faster than the strongest foundation model. Frozen models transport within 0.002 AUROC to 2023 data; the authors conclude evaluation practice and feature sets, not model capacity, are the binding constraint.

arXiv cs.AI / cs.LG / cs.CL·Sep 10, 17:29 UTC · 5d agoAI research1

IntentFuzz: A Protocol-Aware Fuzzer for Automated Invariant Violation Detection in Intent-Based Cross-Chain Bridges

IntentFuzz protocol-aware fuzzer recovers bridge structure from unannotated Solidity and confirmed 22 invariant violations across 24 real-world deployments.

IntentFuzz formalizes a taxonomy separating invariant violations from settlement exposures in intent-based cross-chain bridges, then recovers a bridge's intent structure and deposit/fill function roles from unannotated Solidity source. It classified deposit and fill functions with 100% recall and 82% combined precision, and achieved 100% recall and precision on 23 planted-bug mutants. Across 24 real-world deployments it confirmed 17 genuine invariant violations with heuristic-only input generation, rising to 22 with its LLM-assisted tier, spanning eight vulnerable GitHub repositories with findings reproducible against public deployed bytecode.

arXiv cs.CR·Sep 11, 16:03 UTC · 4d agoResearch1

A First-Principles Evaluation of Graph-Based Network Intrusion Detection Systems

GIDS-Eval framework reveals evaluation gaps in graph-based network intrusion detection; two crafted edges fully evade three detector-dataset pairs.

Researchers introduce GIDS-Eval, a framework decomposing graph-based network intrusion detection systems into six interchangeable stages to enable controlled comparisons. Surveying nine GIDS and reimplementing five, they find two crafted edges achieve full evasion against three of eight detector-dataset pairs, snapshot windows alone cause a mean 38.3% relative swing in average precision, and none of 18 replayed detector-dataset pairs can alert as events arrive. Their encoder-free GIDS-Lite control ranks first by AP on two of four datasets at up to 575x lower runtime.

arXiv cs.CR·Sep 10, 22:38 UTC · 5d agoResearch1

Cross-Model Agreement as a Deployment-Time Reliability Signal for Automatic Polyp Segmentation

Referee-Based Quality Estimation flags unreliable polyp segmentations at inference without ground truth, reaching ROC-AUC 0.960 with SegFormer-B0 referees.

RBQE measures agreement between a primary segmentation model and an independently trained referee on a 1,223-image external benchmark drawn from four public datasets. A cross-architecture SegFormer-B0 referee achieves the strongest signal (ROC-AUC 0.960), beating a Test-Time Augmentation baseline by 0.055 ROC-AUC under an identical protocol. Excluding trivially separable empty-mask cases, ROC-AUC falls to 0.876 (SegFormer-B0) and 0.783 (same-architecture control), but RBQE's margin over baselines widens. Progressive rejection of low-agreement predictions increases mean Dice of retained outputs, supporting selective prediction at the cost of one extra forward pass.

arXiv cs.AI / cs.LG / cs.CL·Sep 9, 17:32 UTC · 6d agoAI research

How to build an exposure management program the business trusts: Lessons from Tenable’s CSO

Tenable's CSO describes an AI-driven exposure management program that consolidates tool sprawl and translates cyber risk for boards.

A Tenable blog post shares lessons from CSO Robert Huber on moving to an AI-driven exposure management program. It argues tool sprawl and data silos hinder holistic risk assessment and that exposure management unifies attack-surface data into business-level metrics for the C-suite.

Tenable Blog·Aug 27, 14:30 UTC · 20d agoIndustry

BenchShield: Formal Model-Backed Instrumentation for Reward Integrity in LLM-Agent Evaluation Infrastructure

BenchShield uses lifecycle-model-backed instrumentation to detect reward hacking in LLM-agent benchmarks, lifting full-chain recall to 77-100% at up to 65% lower cost.

The framework grounds reward-hacking detection in a finite lifecycle model of an evaluation's reward-relevant events, combining a static phase-aware taint analysis with runtime infrastructure-side evidence attribution. Evaluation used a human-labeled corpus of 456 adjudicated trajectories drawn from more than 31,000 public agent runs across three benchmarks. BenchShield improves full-chain recall from 23-94% to 77-100% and same-vector coverage from 16-56% to 43-78%, cuts per-task cost by up to 65%, and achieves 96% accuracy detecting reward hacking at runtime.

arXiv cs.CR·Sep 10, 03:10 UTC · 6d agoAI safety & security1

Not All Relations Are Equal: Relation-Balanced and Calibrated Graph Learning for Provenance-Based Intrusion Detection

RECAL improves provenance-based APT detection with relation-balanced masked graph learning and calibrated errors, reaching 99.99% F1 on DARPA E3 datasets.

RECAL is an unsupervised framework for provenance-based intrusion detection that uses relation-balanced masked graph learning to capture rare interaction patterns, addressing statistical heterogeneity where relation frequencies differ by roughly 140,000X in CADETS. It calibrates reconstruction errors against each relation's benign error distribution to produce comparable anomaly evidence and reduce false alarms. On three DARPA E3 datasets, RECAL achieves F1 scores of 99.99%, 99.93%, and 99.99%, outperforming the best baseline on each dataset, and reduces mean false positive rate by approximately 105X, 4X, and 41X versus the lowest-FPR baseline.

arXiv cs.CR·Sep 15, 00:31 UTC · 1d agoResearch

A Malicious Webpage Could Poison Your Local AI Model Behind NVIDIA NemoClaw

Oasis Security found NVIDIA NemoClaw's Ollama binding to 0.0.0.0 enables DNS rebinding attacks that let attacker pages poison model chat templates with persistent hidden instructions.

Oasis Security disclosed that NVIDIA NemoClaw on Windows/WSL paths binds Ollama to 0.0.0.0:11434 without authentication, exposing the API to browser-based DNS rebinding attacks from malicious webpages. An attacker can then modify the model's chat template via /api/create, planting hidden instructions that run on every subsequent inference and persist across conversations, invisible to API consumers. NemoClaw v0.0.35 fixed the issue on macOS and Linux; no fix exists for Windows and WSL paths beyond a warning in v0.0.34. Ollama's own 2024 fix (CVE-2024-28224) added Host header validation, but it is skipped when bound to non-loopback addresses. No exploitation has been reported as of August 25, 2026.

The Hacker News·Aug 25, 15:58 UTC · 21d agoAI safety & securityCVE-2024-282241

Your Critical Vulnerabilities Might Not Be Your Biggest Risk

Autonomous penetration testing advocates prioritize exploitable attack paths over raw vulnerability severity for continuous security validation.

The article argues that scanner severity scores lack context: a critical flaw behind strong segmentation may be low priority, while a medium flaw on internet-facing systems can provide a foothold chained toward sensitive data. It positions autonomous penetration testing and attack path validation as the execution layer for continuous security validation, replacing point-in-time assessments. The piece is vendor-authored thought leadership rather than incident or vulnerability news.

The Hacker News·Sep 11, 11:30 UTC · 5d agoIndustry1