Smart search ranks by meaning as well as keywords (one row per story, last 45 days).
An online exam tool reportedly suffered a data breach, potentially exposing student or user information, according to Infosecurity Magazine.
Infosecurity Magazine published a report indicating that an online examination platform suffered a data breach. The available metadata does not include the vendor name, number of affected users, or the types of data exposed. Scope and impact cannot be fully assessed without the full article text.
Datamimic is an open-source test data generation tool aimed at keeping coding agents from inventing their own test fixtures.
A Hacker News discussion (40 points) highlights Datamimic, an open-source rapiddweller GitHub project for generating realistic synthetic test data. The tool targets AI coding agents, aiming to prevent them from fabricating their own inconsistent test worlds. Only the repository link was shared, so details are limited.
ETSI published TR 104 180 defining 18 data quality metrics, including fairness and privacy, to assess whether datasets are fit for AI.
ETSI's technical report TR 104 180 defines 18 metrics across four groups - intrinsic quality, usability and lineage, fairness, and privacy - each with calculation formulas, plus an open-source tool that scores datasets. Testing on an aircraft engine sensor dataset and a US census dataset revealed a roughly threefold gender gap in high earners (about 31% of men versus 11% of women) and two privacy failures: re-identification via age, race, sex, and country, and sensitive fields stored in plaintext. The working group included Sejong University, EGM, TTA, Daejeon University, and CNIT.
CSO Online publishes a practical penetration-testing guide for GenAI, LLM, and RAG applications, covering prompt injection, retrieval poisoning, and tenant isolation testing.
The guide frames LLM applications as attack graphs spanning prompts, retrieval layers, vector stores, tools, identities, and downstream APIs, arguing that conventional web testing misses instruction-vs-data channel risks. It builds on OWASP prompt injection guidance (direct vs. indirect injection) and NIST's 2025 adversarial machine-learning taxonomy, noting that RAG and fine-tuning do not remove injection risk. Recommended practices include documenting trust transitions across components, using canaries and synthetic records to avoid test side effects, running multi-turn and obfuscated injection campaigns, and verifying chains from poisoned documents to observable state changes. It also details testing RAG pipelines via controlled document poisoning across metadata, OCR layers, and code comments, plus cross-tenant isolation checks on retrieved document IDs.
ExecCritic separates test generation from patching for coding agents, lifting SWE-bench Verified resolution to 72.6%.
ExecCritic pairs a test-verify-revise scaffold with role-specific reinforcement learning: a Test agent writes repository-native tests and a Repair agent fixes code from execution feedback, both using Qwen-3.5-35B-A3B backbones. Post-trained Qwen agents compose to 72.6% on SWE-bench Verified, an 11.4-point gain over the 61.2% no-test baseline, without stronger-model or oracle feedback at evaluation time. The work shows test quality is the key variable: base-agent tests lowered resolution to 57.3% while GPT-5.6-sol tests raised it to 65.3%.
Tricentis CISO Erika Dean urges keeping production data out of test environments and describes red-teaming that caught a prompt injection gap.
In a Help Net Security interview, Tricentis CISO Erika Dean recommends keeping production data out of QA and testing environments, even in finance and healthcare, where weaker QA controls create risk. Her team red-teamed an agentic chatbot before launch, found a prompt injection gap, and held the release for a week until it was fixed. She says vendors who cannot explain data residency, retention, and model training use are disqualified regardless of product quality.
Synthesized announced a Test Data Agent that provisions production-like data and system states to validate enterprise AI agents before deployment.
Synthesized unveiled its Test Data Agent, an agentic infrastructure capability that generates, masks, and subsets production-representative data for testing AI agents under realistic enterprise conditions. It integrates with agent development, evaluation, testing, and orchestration frameworks, with purpose-built support for complex SAP estates including finance, procurement, and supply-chain workflows and ECC-to-S/4HANA transformation programs. The product runs in on-premises, private-cloud, and hybrid environments and exposes REST APIs and CI/CD triggers for repeatable validation scenarios.
Audit of 22 frontier models finds widespread verbatim retrieval of published molecular property values, with higher reasoning increasing recall of memorized numbers.
An arXiv audit tests 22 frontier LLMs across 12 molecular regression benchmarks for verbatim retrieval of published values. More than 50% of the LLMs show verbatim retrieval on five datasets, and identical experiments are flagged 89% more often at a high reasoning level than at the lowest one. Suppressing retrieval moves model prediction errors closer together in relative terms, suggesting predictive capability is not determined solely by memorized values.
WearableQA benchmark tests LLM health reasoning over longitudinal wearable data; the best of 14 evaluated LLMs reaches 72.9% accuracy.
WearableQA comprises 4,084 ten-option multiple-choice questions built from wearable time series, blood biomarkers, and demographics of 200 real users with up to 500 days of daily measurements. It defines 16 question types along two axes: data versus health reasoning, and single- versus cross-signal reasoning. Evaluation of 14 proprietary and open-source LLMs shows performance from 19.6% to 72.9% against a 10% chance baseline, with most models below 60%.
ToolLoop introduces a closed-loop synthetic data framework whose 11K examples lift a 4B model to 86.40% on BFCL tool-use evaluation.
ToolLoop decomposes tool-use data synthesis into function-name sampling, backward derivation of user queries, and forward derivation of tool calls, with dynamic self-feedback at each stage. This shifts the paradigm from generate-then-filter to generate-verify-refine, reducing inefficient and imbalanced synthetic data. A 4B model trained on 11K synthetic examples reaches 86.40% accuracy on BFCL non-reasoning mode (86.07% in an Isolate variant excluding BFCL-overlapping functions) and 72.1% on ACEBench using only 18.3% of baseline training data.
DataFlex-RL benchmark of 13 RLVR data policies on Qwen2.5-7B finds none reproducibly beats uniform sampling under matched GRPO training.
DataFlex-RL is an evaluation platform comparing rollout-selection, reweighting, and mixture data policies for RLVR under a common GRPO recipe. Across 13 configurations and 12 matched seeds with Qwen2.5-7B-Base on 12 math, logic, and science benchmarks, uniform GRPO improved domain-balanced accuracy by 7.76 points, but no alternative policy achieved a statistically significant improvement. A corrected 12-seed Llama-3.1-8B-Base extension found no consistent winner, and math-heavy evaluation summaries were negatively correlated (-0.33) with domain-balanced summaries.
Dan Luu's eval finds coding-agent testing instructions (TDD, formal methods, PBT, skills) mostly fail to beat defaults on Zstd implementation correctness.
The author ran 26 prompt conditions plus 4 skills on a Zstd-in-Rust implementation eval using codex with GPT-5.6, testing TDD, fuzzing, property-based testing, formal methods (Lean 4, TLA+, Verus, Kani, SMT solvers) and community skills. Nothing dramatically outperformed the default no-instruction condition, which did above average; at xhigh effort, fuzzing and PBT conditions did slightly better than formal methods. Pre-registered predictions included TDD underperforming and popular test skills (ECC, Hegel, Trail of Bits) not outperforming. Results are averages of 80 runs per condition plotted against cost.
Horizon3 blog argues security context, not data volume, is the SOC bottleneck, promoting its NodeZero integration with CrowdStrike Falcon Next-Gen SIEM.
Horizon3.ai published a vendor blog explaining how its NodeZero Proactive Security Platform integration with CrowdStrike Falcon Next-Gen SIEM brings validated exposure findings into existing security operations workflows. The post argues SOCs are now limited by confidence rather than visibility, needing context to decide which issues matter. It cites a global chemical manufacturer that validated exploitable exposures with NodeZero before completing a $2 billion merger.
Evaluation of twelve LLMs on 222 clinical questions shows verbatim quotes rarely substantiate claims; claude-opus-5 fully substantiates only 37.1%.
The authors build a standardized harness over four clinical practice guidelines and evaluate twelve LLMs on 222 synthetic clinical questions, measuring citation attachment, verbatim quote production, and claim substantiation. Most models attach verbatim quotes to over 90% of claims from prompting alone, though lightweight models like claude-haiku-4.5 struggle. Quotes frequently fail to substantiate claims: claude-opus-5 quotes 98.0% of claims but fully substantiates only 37.1%, exposing a capability gap for verifiable clinical QA.
A retrospective study found GPT-4 over-flagged emergency department revisit cases while an LLM knowledge-graph screener achieved 83-100% positive predictive value.
In an exploratory retrospective study of 99 emergency department diagnosis pairs from a multihospital health system, clinicians and GPT-4 independently judged whether revisit pairs warranted further assessment. GPT-4 responses correlated poorly with clinicians, flagging 94% of pairs for follow-up, 4.4-13.3 times more than clinicians, though prompt engineering was minimal. An algorithm leveraging an LLM-populated knowledge graph (KGA) achieved 83-100% positive predictive value against at least one clinician rater, suggesting LLM-based screening could broaden revisit quality review without substantially increasing reviewer workload.
A study finds LLM-synthesized CodeQL queries improve average F1-score by 82% over baseline queries, offering scalable vulnerability detection versus direct LLM scanning.
Researchers conducted an empirical study evaluating whether LLMs can synthesize executable CodeQL queries from National Vulnerability Database vulnerability data. LLM-generated queries significantly enhanced baseline CodeQL suites, yielding an 82% improvement in average F1-score across a diverse set of real-world vulnerabilities. A cost-benefit analysis shows direct LLM-based scanning of entire repositories is often computationally and financially prohibitive, while LLM query synthesis offers a scalable and cost-effective alternative for large-scale vulnerability detection.
WearableQA benchmark introduces 4,084 questions over real longitudinal wearable data, showing 14 LLMs score 19.6-72.9% on health reasoning, far from solved.
WearableQA is a benchmark of 4,084 10-option multiple-choice questions built from wearable time series, blood biomarkers, and demographics of 200 real users with up to 500 days of daily measurements. It defines 16 question types along two axes: data versus health reasoning, and single- versus cross-signal reasoning, using a dual-grounding framework combining literature and population-validated patterns. Evaluations of 14 proprietary and open-source LLMs show accuracy ranging from 19.6% to 72.9% against a 10% chance baseline, with most models below 60%.
91% of 115 surveyed TEE deployments across Intel SGX, TDX, and AMD SEV fail to provide reproducible builds needed for verifiable remote attestation.
A systematic study of 115 TEE deployments found 91% were not reproducible and 80% lacked both source code and a reference build, undermining remote attestation guarantees. Interviews with 12 developers of 50 Intel SGX projects confirmed that only one participant treats reproducibility as a development priority. The authors identify technical barriers such as embedded timestamps plus ecosystem-level issues like lack of build-environment control in multi-stakeholder projects, and call for holistic, committed reproducibility practices.
Controlled mid-training experiments on Qwen3-8B-Base find each domain has a 10-40% coverage optimum and domain gaps survive alignment SFT.
Using Qwen3-8B-Base (with a 4B replication) across five semantically rule-disjoint KOR-Bench domains, the authors train 30 data allocations spanning the five-domain simplex at five seeds each. All five domains show interior optima in the moderate 10-40% coverage band, and domain gaps persist after a fixed-budget compensatory SFT pass, which raises 116/120 cells yet bridges 0/240 pairs at a 5% threshold. Zero coverage collapses mid-training-only accuracy, though a FineWeb-Edu-only control shows the collapse is partly generic drift. The results argue mid-training data composition requires principled design rather than reliance on later alignment.
Researchers introduce StudyBench, a physics benchmark showing self-evolution gains on textbook problems rarely transfer to olympiad-level questions.
StudyBench is a controlled physics benchmark splitting test data into an Application Set of difficult textbook problems and a Transfer Set of olympiad-level problems. Across three base models, representative self-evolution methods improved on the Application Set but rarely transferred to the harder Transfer Set. A guidance ablation reveals a Guidance Gap, and every method hits a Compute Plateau, indicating the remaining limits are method problems rather than data or compute problems.
Latvia's road traffic agency CSDD confirmed a breach exposing data on 1.2 million people and 200,000 businesses, prompting leadership resignations.
Latvia's Road Traffic Safety Directorate (CSDD) said hackers accessed payment receipt data dating back to 2008, covering over 1.2 million people and 200,000 legal entities, about two-thirds of Latvia's population. Stolen data includes personal ID numbers, license plates, payment amounts and addresses; phone numbers, emails and passwords were not affected. CERT.LV said attackers exploited a vulnerability in an internet-exposed CSDD system, and President Edgars Rinkevics called the breach a significant national security threat. The supervisory board resigned and chief Aivars Aksenoks said he will leave; state police opened criminal proceedings while responsibility with IT contractor Tet is disputed.
Study measures approval integrity in Lightcap LLM answer publication, finding the 14B response-act checker accepts 291 of 302 unsupported answers.
The study evaluates exact-content binding, authorization freshness, and checkpoint recovery in Lightcap's publication enforcement using 3,600 assessments over 900 human-annotated RAGTruth responses from three Ministral models. The production 14B response-act checker accepts 291 of 302 unsupported answers versus 41 for a direct-grounding baseline, with supported-answer retention of 95.2% versus 66.9%. A stateful recheck-recovery policy increases exact-match error by 9.23 percentage points relative to initial checkpoints, and controlled evidence-fingerprint changes expose asymmetric freshness enforcement between publication and recovery. A separate BIPIA prompt-injection experiment records zero target insertions among 266 valid editor outputs.
Open-source PentestGPT runs autonomous LLM-driven penetration tests via Claude Code and Codex, with legacy human-in-the-loop mode supporting many providers.
PentestGPT, originally published at USENIX Security 2024 by Gelei Deng and colleagues, is an open-source framework that lets a large language model autonomously run penetration testing stages (recon, exploit, walkthrough) with no human in the loop, driving Claude Code or Codex CLIs. A legacy interactive mode uses three cooperating LLM sessions maintaining a Pentesting Task Tree and supports OpenAI, Anthropic, Google Gemini, DeepSeek, xAI, Qwen, Moonshot, and local models via Ollama. The tool sends anonymous telemetry to Langfuse by default, excluding command outputs, credentials, and flags, and is available free on GitHub.
Prophet Security's 2026 survey of 250+ security pros: 40% use AI daily; AI users report 25%+ faster investigations and rising AI-driven attacks.
Prophet Security's State of AI in Security Operations 2026 report, based on a survey of 250+ cybersecurity professionals, found 40% of security teams use AI daily and only 4% have no adoption plans. Teams average 100 daily alerts (up to 1,000 at large firms), leave 28% of alerts uninvestigated, and 60% of respondents said missed alerts led to breaches or downtime. Among AI adopters, 72% report at least 25% faster investigations, 56% observed increased AI-driven attacks, and no respondents grant AI full unsupervised autonomy. Data privacy (44%) and explainability (41%) top the adoption hurdles.
Continual Search framework iteratively prompts LLM judges to keep searching agent execution logs, boosting long-horizon failure root-cause attribution accuracy.
The paper frames automated root-cause attribution (RCA) for long-horizon AI agent failures as a search problem, since relevant evidence is sparse and distributed across massive execution traces. The authors propose Continual Search, an iterative framework that nudges an LLM judge across successive turns to keep hunting unresolved diagnostic evidence instead of settling on an early plausible diagnosis. They introduce MegaRCA-Mix, a benchmark of 50 human-annotated failure trials on long-horizon, execution-heavy tasks. On MegaRCA-Mix, Continual Search improves GPT-5.5's F1 from 0.349 to 0.498 (over 40% gain), and lower-tier models can surpass higher-tier counterparts when search is effective.
Researchers propose auditing LLM bias via relative hidden-state representations, detecting bias increases with 3-50x less compute than output-level benchmarks.
The paper introduces a reference-based bias auditing method that compares hidden-state representations across model variants, such as before and after fine-tuning, by encoding sentences relative to a fixed anchor set. The resulting Representational Bias Shift (Delta-B) correlates with output-level bias change in 15 of 18 tested settings, reaching |r| = 0.84 under full fine-tuning across WildGuardMix, DecodingTrust, and ToxiGen benchmarks. Thresholding Delta-B detects checkpoints whose bias increased with ROC AUC between 0.65 and 0.99 and beats a SEAT-based baseline, while auditing a model in about three minutes with 3-50x less compute.
Benchmark Radar provides a living searchable database of 1,283 AI benchmark records and 12,916 score observations drawn from 37 daily discovery sources.
Benchmark Radar combines daily discovery of benchmark papers, repositories, datasets, and releases from 13 direct connectors and 24 first-party feeds into a searchable catalog with model card mentions and score histories. The catalog contains 1,283 source records drawn from 4 benchmark catalogs plus 12,916 numeric observations on 790 records. The release includes a web dashboard with leaderboard, Pareto frontier of score versus usage, saturation and trend views, daily feeds, a CLI, and reproducible analysis. The paper audits the full catalog and examines benchmark saturation and limits of score comparisons.
Horizon3.ai publishes a five-question CISO checklist demanding proof of exploitability from CTEM vendors.
Horizon3.ai released a CISO's CTEM Evaluation Checklist offering five questions for evaluating Continuous Threat Exposure Management technologies, covering proof of exploitability, demonstrated impact, remediation verification, and long-term reduction of exploitable exposure. The checklist warns against relying on scanner findings, risk scores, closed tickets, and isolated test results, urging evidence from the buyer's own environment. It is downloadable vendor marketing material.
Datadog Security Labs tested Sonnet 5, Composer 2.5, and GPT 5.5 to see if plan mode yields more secure code than default mode.
Datadog Security Labs evaluated whether plan mode produces measurably more secure code than default mode. The test covered three frontier coding models: Sonnet 5, Composer 2.5, and GPT 5.5. The results inform how engineering teams should configure AI coding assistants to reduce insecure code. This is an AI security evaluation, not an incident report.
Horizon3.ai customer story details Virginia Tech automating external pentesting via NodeZero's GraphQL API with GitLab and ServiceNow integration for remediation tracking.
Horizon3.ai published a customer story describing how Virginia Tech, whose environment serves more than 38,000 students across hundreds of independent departments and multiple cloud providers, used NodeZero's GraphQL API to automate external pentesting through GitLab. Findings are routed directly into ServiceNow for subnet-owner assignment and remediation tracking, creating a repeatable attack-validation-to-remediation workflow.