Search: “OracleZoom”

OracleZoom: On-Policy Self-Distillation Inspired Reference-Constrained Recursive Image Super Resolution

OracleZoom enables recursive extreme-scale image super-resolution via reference-constrained on-policy distillation, reducing hallucinations at deep zoom scales.

OracleZoom tackles recursive super-resolution, where repeated feeding of predictions back into the same model leaves deeper-scale outputs unsupervised as required source resolution grows geometrically. The framework trains on its own trajectory while carrying the last ground-truth evidence beyond the supervision boundary, combining direct and cross-scale supervision, a no-reference quality objective, a KL-constrained pretrained latent prior, and EMA consistency. Across seven datasets it achieves state-of-the-art SR quality across zoom scales, averaging 0.713 CLIPIQA with larger gains at deeper scales and significantly reduced hallucinations. Code, data, and models are publicly released.

Hugging Face daily papers·Sep 5, 20:00 UTC · 11d agoAI research

Toolkit Hidden Inside Oracle Database Evades Endpoint Tools

Infosecurity Magazine·Aug 6, 15:30 UTC · Aug 6, 2026Vulnerability

Benchmark Radar: A Living Database and Search Engine for AI Benchmarks and Evaluation

Benchmark Radar provides a living searchable database of 1,283 AI benchmark records and 12,916 score observations drawn from 37 daily discovery sources.

Benchmark Radar combines daily discovery of benchmark papers, repositories, datasets, and releases from 13 direct connectors and 24 first-party feeds into a searchable catalog with model card mentions and score histories. The catalog contains 1,283 source records drawn from 4 benchmark catalogs plus 12,916 numeric observations on 790 records. The release includes a web dashboard with leaderboard, Pareto frontier of score versus usage, saturation and trend views, daily feeds, a CLI, and reproducible analysis. The paper audits the full catalog and examines benchmark saturation and limits of score comparisons.

Hugging Face daily papers·Sep 9, 20:00 UTC · 7d agoAI research

The Frontier AEO Tracker: What Astra Chooses (and every other frontier model, and what you can do about it)

Latent Space launches an AEO tracker scoring 7 frontier models' product recommendations across 161 categories, revealing generational bias flips.

Latent Space built a tracker measuring Answer Engine Optimization by running 6 prompt variations across 7 frontier models with search enabled over 161 product categories, scoring first choices, alternatives, mentions, and anti-recommendations. It found 28 categories with a universally dominant primary choice and observed soft biases, such as models favoring their own lab's coding agents. Analysis of Anthropic's Sol→Astra and Opus→Fable generations showed newer models consulting fewer sources and being less likely to change answers when questions are paraphrased.

Latent Space·Sep 7, 21:32 UTC · 9d agoAI research

Autonomous Research for Open-Ended Problems: A Case Study on Telecom Ticket Retrieval

Case study shows autonomous LLM research reaches 90% of SOTA on telecom ticket retrieval in 10 weeks versus 10 months human work.

The paper explores adapting autonomous research to open-ended, industry-grade ML problems through a telecom ticket retrieval case study with commercial and open-source agents. Autonomous research reached 90% of state-of-the-art performance (0.34 vs. 0.38 Recall@1) in 10 weeks versus 10 months of human work, at up to $200 per Cursor campaign. The authors find agents excel at narrow hyperparameter optimization but lack human-like intuition, recommending human-agent collaboration.

arXiv cs.AI / cs.LG / cs.CL·Sep 11, 17:09 UTC · 5d agoAI research

A Three-Layer Caching Architecture for Low-Latency LLM Web Search on Commodity CPU Hardware

OreoLook's three-layer Redis caching architecture cuts redundant LLM calls and embedding work for CPU-hosted web-search answer generation.

The paper describes a three-layer caching architecture for OreoLook (formerly lixSearch), an open-source LLM answer engine: a Redis session context window with Huffman-compressed disk overflow, a semantic query cache matching rephrasings via embedding cosine similarity, and a URL embedding cache deduplicating embedding computations. Deployed on a single 8-vCPU Intel Cascade Lake server with 30 Hypercorn workers across three containerized replicas, it achieved an 89.3% aggregate Redis keyspace hit rate, 0.1 ms read latency, and 1.38 MB memory overhead. An LRU eviction daemon migrates idle sessions to disk and rehydrates them for resumption hours or days later.

Hugging Face daily papers·Aug 11, 20:00 UTC · Aug 11, 2026AI tools & infra1

Atlas: Efficient Verifiable Semantic Search

Atlas delivers zero-knowledge proofs for HNSW semantic search, verifying RAG retrieval in under a second on SIFT1M and 2.0 seconds at 100M vectors.

Atlas lets a search provider prove that a query was answered correctly against a committed HNSW index without revealing the index, addressing provider deviations like truncation or bias. It combines offline preprocessing, a fixed-size-state restructuring of HNSW with a correctness proof, and timestep-tagged batching of per-step arguments. The system proves queries in under a second on SIFT1M and 2.0 seconds at 100 million vectors while preserving plaintext HNSW recall, and proven retrieval maintains end-to-end RAG answer quality at lower cost than prior verifiable retrieval systems.

arXiv cs.CR·Sep 10, 17:30 UTC · 6d agoResearch1

MetroLLM-Bench: Evaluating Language Models as Transit Kiosk Runtimes

MetroLLM-Bench is a 955-case benchmark testing language models as transit kiosk tool-calling runtimes across six real metro systems.

The benchmark covers 37-414-station metro systems and eleven task categories including routing, fare calculation, disruptions, accessibility, and adversarial input, with 14 deterministic and 8 semantic scoring components. Of 26 models from six vendors, a PEFT-tuned 4B Qwen 3.5 student scored 91.3 on Tier 1, exceeding GPT-5.6 (90.6/90.0), while Muse Glimmer 30B led the composite ranking. A deterministic rule-based baseline reached 84.6, and PEFT gains over base models shrank from +7.03 points at 2B to -0.91 at 27B.

Hugging Face daily papers·Sep 8, 20:00 UTC · 8d agoAI research

What Does an LLM-Agent Leaderboard Rank Actually Compare?

A methodological study shows close LLM-agent leaderboard rank gaps on SWE-bench and similar benchmarks often do not support superiority claims.

The paper defines an estimand-aware pairwise procedure for comparing agents, checking common support and applying explicit uncertainty rules and practical margins. Across SWE-bench, AgentRewardBench, and tau2-bench, close rank differences are frequently unresolved, and proxy labels or utility rules can change which system is selected. The authors argue a leaderboard score summarizes a released evaluation but does not by itself justify pairwise superiority conclusions.

arXiv cs.AI / cs.LG / cs.CL·Sep 7, 17:31 UTC · 9d agoAI research

IDORacle: Template-Guided SQL-Sink Mediation for Object-Level Authorization in Java Applications

IDORacle intercepts MyBatis/JDBC SQL sinks to block horizontal privilege escalation in legacy Java apps with sub-millisecond guard latency.

IDORacle is a template-guided SQL-sink interception and rewriting framework that prevents IDOR/BOLA horizontal privilege escalation at runtime in Java database applications. It propagates authenticated identity context across HTTP requests, asynchronous tasks, and data-access boundaries via a server-side trace identifier, and computes dual SQL fingerprints at the MyBatis/JDBC boundary to generate reusable mediation plans. On a Java-SQL benchmark grounded in real-world CVE reports, it blocks tested violations with worst-case guard latency of 0.17 ms, reduced to 0.017 ms average for hot templates via redundancy-aware optimization.

arXiv cs.CR·Sep 11, 04:20 UTC · 5d agoResearch1

Context Engineering Inside the Harness: 4 Mechanisms That Beat Context Overflow and Goal Loss on Long-Horizon Tasks

Survey of four harness mechanisms—context budgeting, compaction, todo-state, and memory—that keep long-horizon LLM agents on task across 200+ tool calls.

The article details how agent harnesses, not larger context windows, solve context overflow and goal loss on long-horizon tasks, citing Chroma's Context Rot report showing 18 LLMs (GPT-4.1, Claude 4, Gemini 2.5, Qwen3) degrade on long inputs. Concrete implementations include LangChain Deep Agents offloading tool responses over 20,000 tokens to the filesystem and truncating old tool calls at 85% window usage, and Claude Code capping auto memory at 25KB while re-reading the 5 most recently modified files after compaction. OpenAI's Responses API now offers server-side compaction via context_management with a standalone /responses/compact endpoint, which Codex uses for long-running coding tasks. Manus reports a roughly 100:1 input-to-output token ratio per ~50-tool-call task, motivating todo.md state recitation to prevent goal drift.

MarkTechPost·Sep 13, 05:56 UTC · 3d agoAI research1

Fortunate Recall: Ontology-Driven Memory Lifecycle Management for Persistent Coherence in LLMs

Fortunate Recall introduces ontology-based lifecycle policies for LLM memory, cutting confabulation roughly in half (e.g., 45.1% to 22.4%) versus Mem0.

Fortunate Recall (FR) is a composable policy layer that classifies personal facts into a 10+1 behavioral ontology and applies category-specific lifecycle rules including differential temporal decay, slot-key supersession, event-time validity, and retrieval routing. FR-Bank scores 76.9% on the new 516-question LifecycleBench, ahead of Mem0, A-MEM, Memory-R1, and MemoryOS (61%-70.5%), and 75.2% on LongMemEval-S. End-to-end, confabulation drops from Mem0's 45.1% to 22.4% over answered queries, with the ranking replicating on open-weight Kimi K2.5 and transferring to the independent BEAM benchmark (46.8% vs 32.9%).

arXiv cs.AI / cs.LG / cs.CL·Sep 9, 16:31 UTC · 7d agoAI research

Towards Scalable and Cost-Efficient Vulnerability Detection: A Study on Automatic Query Generation

A study finds LLM-synthesized CodeQL queries improve average F1-score by 82% over baseline queries, offering scalable vulnerability detection versus direct LLM scanning.

Researchers conducted an empirical study evaluating whether LLMs can synthesize executable CodeQL queries from National Vulnerability Database vulnerability data. LLM-generated queries significantly enhanced baseline CodeQL suites, yielding an 82% improvement in average F1-score across a diverse set of real-world vulnerabilities. A cost-benefit analysis shows direct LLM-based scanning of entire repositories is often computationally and financially prohibitive, while LLM query synthesis offers a scalable and cost-effective alternative for large-scale vulnerability detection.

arXiv cs.CR·Sep 9, 16:31 UTC · 7d agoResearch1

ConvMem: Convolutional Memory for Long-Context Reasoning

Researchers propose ConvMem, a training-free framework treating LLMs as convolutional kernels for parallelizable long-context reasoning beyond fixed context windows.

ConvMem reformulates long-context reasoning as a hierarchical convolution in which the LLM summarizes text segments hierarchically, shortening the reasoning path from a linear chain to a logarithmic tree. It uses configurable strides, skip connections, and multi-kernel convolution to capture evidence, decompose queries, and enable massive parallelization across segments and reasoning threads. On RULER-HotpotQA and RULER-2WikiMultiHopQA it outperforms training-free baselines and avoids the out-of-distribution overfitting seen in RL-trained approaches like MemAgent.

arXiv cs.AI / cs.LG / cs.CL·Sep 9, 16:53 UTC · 7d agoAI research

Meta FAIR Introduces AI Research Preference Models (RPMs): Ranking ML Experiments Before Spending GPU Hours

Meta FAIR, Oxford and UCL introduce Research Preference Models that rank unexecuted ML experiments, lifting AIRS-Bench scores from 0.684 to 0.729 and cutting compute ~1.6×.

Researchers from Meta FAIR, Oxford, and UCL introduce Research Preference Models (RPMs), which use frozen pretrained LLMs (Qwen3.6-27B backbone, no fine-tuning) to rank unexecuted experiment candidates and execute only the winner of a pairwise knockout tournament. Two variants shipped: an inference-only LLM-as-a-judge and an agentic variant that runs small pilot experiments in an H200 sandbox. On AIRS-Bench (20 tasks, 24 hours on one H200, 10 seeds), scores rise from 0.684 (random) to 0.711 and 0.729 versus a 0.748 validation oracle, and both variants reach the baseline's 24-hour score in roughly 15 hours. The team reports new SOTA on WinoGrande (94.1% with Agentic RPM) and SVAMP (95.7% with inference-only).

MarkTechPost·Sep 6, 20:25 UTC · 10d agoAI research

TRACE: Trajectory-robust Admission with Evidence Ordering for Efficient GUI Agents

TRACE, a training-free visual token pruning framework, cuts GUI agent inference latency and memory while keeping trajectory-wide visual evidence reusable.

TRACE is a training-free framework for trajectory-robust admission and coverage-aware evidence ordering that prunes high-resolution screenshot tokens accumulated in GUI agent trajectories. It ranks visual evidence using a query-independent layout-derived interaction prior combined with instruction relevance and feature novelty, and reserves part of the budget for native tokens distributed across the screen to repair spatial coverage. A monotone KV contraction incrementally compresses retired frames into compact session state, avoiding repeated visual encoding or pruning. Experiments across six GUI benchmarks and diverse models verify effectiveness under tight budgets, with source code to be released.

Hugging Face daily papers·Sep 8, 20:00 UTC · 8d agoAI research

When the prompt becomes the payload: A practical pen-testing guide for GenAI, LLM and RAG applications

CSO Online publishes a practical penetration-testing guide for GenAI, LLM, and RAG applications, covering prompt injection, retrieval poisoning, and tenant isolation testing.

The guide frames LLM applications as attack graphs spanning prompts, retrieval layers, vector stores, tools, identities, and downstream APIs, arguing that conventional web testing misses instruction-vs-data channel risks. It builds on OWASP prompt injection guidance (direct vs. indirect injection) and NIST's 2025 adversarial machine-learning taxonomy, noting that RAG and fine-tuning do not remove injection risk. Recommended practices include documenting trust transitions across components, using canaries and synthetic records to avoid test side effects, running multi-turn and obfuscated injection campaigns, and verifying chains from poisoned documents to observable state changes. It also details testing RAG pipelines via controlled document poisoning across metadata, OCR layers, and code comments, plus cross-tenant isolation checks on retrieved document IDs.

CSO Online·Sep 9, 10:00 UTC · 7d agoAI safety & security1

PuzzleMask: Abusing Plain Prose as a Covert AI Attack Vector

Check Point details PuzzleMask, a plain-prose technique that bypasses LLM gatekeeper policy checks, letting hidden payloads reach target models unreviewed.

Check Point Research describes PuzzleMask, a prompt-crafting technique that hides policy-violating payloads inside plain-English prose wrappers, bypassing quick LLM-based policy checks without emojis, Base64, or invisible formatting. The researchers tested 23 automated prompts against gatekeepers including GPT-4o-mini, GPT-OSS-Safeguard 20b, Claude 3 Haiku, and Llama Guard 3, and all were classified as safe despite policies that flagged the plain versions. When submitted to GPT-5 in thinking-high mode with a Python interpreter, the target model extracted and acted on the payload in over 90% of trials. The technique is not itself a jailbreak but can carry a jailbreak prompt as payload; mitigations include input paraphrasing, hardened gatekeeper policies, and output monitoring.

Check Point Research·updated Sep 11, 05:53 UTC · 5d ago·first Sep 10, 14:32 UTC · 6d agoAI safety & security 2 sources

The AI Malware Maturity Gap

Recorded Future introduces AIM3, a five-level maturity model for AI malware, showing current attacker AI use is mostly AI-assisted rather than autonomous.

Recorded Future proposes AIM3, a five-level model defining AI malware from LLM-translated to LLM-embedded, spanning experimentation to fully autonomous agentic campaigns. Public examples remain early-stage: PROMPTFLUX uses Google Gemini to rewrite its VBScript dropper (Level 1), while Lamehug/PROMPTSTEAL, attributed to APT28, invokes the HuggingFace API to generate reconnaissance commands (Level 3). The authors argue most current AI malware augments existing tradecraft rather than enabling one-click autonomous attacks.

Recorded Future·Aug 26, 00:00 UTC · 21d agoResearch

LLMs and Contextual Integrity

Bruce Schneier highlights two papers: the CIMemories benchmark shows frontier LLMs leak memory attributes up to 69%, and an RL method reduces inappropriate disclosures.

Bruce Schneier discusses contextual integrity in LLMs, referencing the CIMemories benchmark, which uses synthetic profiles with 100+ attributes per user to test whether models with persistent memory disclose sensitive information appropriately. Evaluation showed frontier models exhibit up to 69% attribute-level violations, with GPT-5's violation rate rising from 0.1% to 9.6% across 40 tasks and reaching 25.1% with repeated prompting, showing unstable leakage behavior. A second paper introduces a reinforcement learning framework trained on a synthetic 700-example dataset that substantially reduces inappropriate disclosure while maintaining task performance, with improvements transferring to the human-annotated PrivacyLens benchmark.

Schneier on Security·Aug 18, 10:40 UTC · 29d agoAI safety & security1

Studying Without a Syllabus: Task-Agnostic Environment Preprocessing

Paper formalizes task-agnostic environment preprocessing, where agents study unfamiliar environments under a budget to build reusable artifacts for a frozen solver.

The paper formalizes task-agnostic environment preprocessing, where a studying system explores an environment under a budget and produces artifacts like indices, scripts, or procedural guidance for a frozen solver, without task examples or evaluation feedback. The authors compare unaided and archive-equipped meta-agents against fixed synthetic-practice and corpus-processing methods across six heterogeneous benchmarks. A meta-agent variant achieves the highest Avg@3 reward on five benchmarks, while fixed corpus processing remains best on the largest corpus benchmark. Studied artifacts reduce the test-time sampling needed to reach a given score, shifting computation from repeated test-time attempts to a pre-task study phase.

Hugging Face daily papers·Sep 8, 20:00 UTC · 8d agoAI research

What researchers learned about building an LLM security workflow

Oslo and FFI researchers show structured agentic workflows lift LLM alert-triage accuracy from 0% to about 93% on malicious cases.

Researchers at the University of Oslo and the Norwegian Defence Research Establishment tested GPT-5-mini, Claude 3 Haiku, Qwen3:30B, and Gemma 3:27B on alerts from the AIT Log Data Set V1.1; given only alert descriptions and log summaries, all four models correctly flagged zero percent of true-positive cases involving reconnaissance, brute-force logins, and initial access. Wrapping the same models in a workflow with constrained SQL queries over Suricata logs, an evidence summarizer, and a verdict stage with revision loops raised malicious-case accuracy to an average of 93 percent, with GPT-5-mini identifying every malicious case across 100 runs. The authors flag it as a proof-of-concept on one synthetic scenario and note models skewed conservative on benign alerts, with GPT-5-mini marking every benign case uncertain.

Help Net Security·Aug 24, 04:21 UTC · 23d agoAI research1

AI SIEM Search

Huntress launched AI Search for Managed SIEM, translating plain-English questions into ESQL log queries with schedulable custom alerts.

Huntress introduced AI Search in its Managed SIEM, letting users query logs in natural language instead of writing ESQL, KQL, or SPL. The feature generates a real ESQL query behind the scenes, exposes it for users to inspect and learn from, and allows saved searches to be scheduled as recurring alerts at hourly, daily, or weekly cadence. A planned next version will return summarized plain-language answers instead of raw logs.

Huntress·Sep 7, 14:00 UTC · 9d agoTools

What Else Needs Fixing? Exploring Cost-Effective Test-Time Compute for Revision Propagation in Artifacts Generated Through Conversation

A new benchmark shows LLMs reach 68.3-93% accuracy propagating local revisions across conversationally generated artifacts, with parallel-sample selection most cost-effective.

The paper introduces a benchmark for revision propagation: when users request a local change, LLMs must identify dependencies and update all affected parts of an artifact generated through conversation, where context lives in the chat history. Nine revision methods, including sequential reflection and parallel sampling variants, were evaluated on gpt-oss-20b/120b, gpt-5.4-mini, and qwen3.5-9b/27b/122b. Baselines scored 68.3-93% accuracy, and selecting among three parallel samples via LLM-based or medoid selection improved accuracy by 2.2-9.7% as the most cost-effective test-time compute strategy. Code and dataset are released.

Hugging Face daily papers·Sep 2, 20:00 UTC · 14d agoAI research

Feyospace-v1: How the Cyber Mercury Seven Trained Frontier Cyber Models

A seven-person independent team trained open-weight agentic cyber models via a data-centric post-training framework, ranking 10th on CyberGym and first at comparable scale.

The paper presents Feyospace-v1, a data-centric post-training framework combining five systems: Choulea (hidden reasoning analysis), SkyReal (teacher-sampling cost reduction), Hongzwang (bypassing teacher API restrictions), PSBreakup (restoring capabilities weakened by model merging), and Kreator (converting expert interventions into trainable reasoning). The data engine builds resettable coding, vulnerability, CTF, kernel-history, full-exploit, firmware, and device-backed environments, retaining only execution-verified and evidence-audited trajectories, yielding 164,269 trajectories for long-context supervised fine-tuning. Three checkpoints improve over their starting models by an average of 23.76% on the full CyberGym suite and 10.49% across pooled CTF suites. As of September 1, 2026, Feyospace-s1 achieves a 63.24% verified success rate, ranks 10th on the official CyberGym leaderboard, and all three checkpoints rank 1st among models at comparable parameter scales.

Hugging Face daily papers·Sep 7, 20:00 UTC · 9d agoAI research

Zoom Patches “Zoomsday” Zero-Click Flaw Enabling Remote Code Execution

Zoom patched CVE-2026-53413, a zero-click annotation flaw dubbed "Zoomsday" allowing remote code execution on meeting participants' devices across all platforms.

Zoom patched four vulnerabilities, including CVE-2026-53413, a stack buffer overflow in CAnnoFormatBlock::Deserialize in the annotation protocol that allows zero-click remote code execution on another participant's device. A Security also found CVE-2026-53414, a buffer overread enabling denial-of-service crashes, and CVE-2026-53415, a use-after-free Zoom had already discovered internally. Updates shipped for Workplace 7.1.5 and 7.0.6, Rooms 7.1.5, and Meeting SDK 7.1.5 across all supported platforms.

Security Affairs·Aug 11, 17:48 UTC · Aug 11, 2026VulnerabilityCVE-2026-53413CVE-2026-53414CVE-2026-53415

Zoom Annotation Flaws Could Let a Meeting Participant Hijack Another Attendee's Client

A Security researchers disclosed three Zoom annotation flaws enabling zero-click client hijacking; Zoom shipped fixes in June and July with no exploitation reported.

Researchers at A Security found three flaws in Zoom's annotation feature: CVE-2026-53413 (CVSS 8.3, buffer over-write), CVE-2026-53414 (CVSS 6.5, buffer over-read), and CVE-2026-53415 (CVSS 8.3, use-after-free). A crafted drawing object sent over the wrong message channel can overwrite adjacent memory and hijack another attendee's client with no user interaction. Fixes shipped in Zoom Workplace 7.1.5/7.0.6, VDI Client 7.0.11/6.6.16, and Zoom Rooms/Meeting SDK 7.1.0+ during June and July. No exploitation has been reported and the flaws are absent from CISA's Known Exploited Vulnerabilities catalog.

The Hacker News·Aug 11, 19:08 UTC · Aug 11, 2026VulnerabilityCVE-2026-53413CVE-2026-53414CVE-2026-53415

MeClear: Cooperative Game-Theoretic Attribution and Risk-Aware Memory Clearance for Long-Horizon LLM Agents

MeClear uses cooperative Shapley attribution to clear harmful memories from long-horizon LLM agents, boosting task recovery by 25.5 points over baselines.

The paper introduces MeClear, a task-conditioned memory clearance framework for long-horizon LLM agents that identifies and selectively suppresses memories with negative downstream utility without permanently altering the persistent memory bank. It combines Leave-One-Out screening with sampled cooperative Shapley attribution to distribute utility across interacting evidence, resolving redundant conflict masking that single-removal evaluations miss. Across ten long dialogue memory pools it achieves 85.9% target recall and 82.3% overall task recovery, a 25.5 percentage-point improvement over LOO baselines.

arXiv cs.AI / cs.LG / cs.CL·Sep 8, 17:46 UTC · 8d agoAI research

Iris-mini and Iris-pro are the strongest open-weight search agents in their class

Chinese lab AllSpark releases Iris-mini (35B) and Iris-pro (397B) open-weight search agents claiming best-in-class results on BrowseComp and other research benchmarks.

AllSpark's paper introduces Iris-mini (35B parameters, built on Qwen3.6-35B-A3B) and Iris-pro (397B parameters, built on Qwen3.5-397B-A17B), both with 256,000-token context windows. Iris-pro scores 88.6 on BrowseComp, 85.1 on BrowseComp-ZH, 92.9 on DeepSearchQA, and 56.4 on Humanity's Last Exam; Iris-mini reaches 82.2, 84.8, 86.9, and 52.3 respectively. Training tasks are reverse-engineered from web link structure, filtered by a judge model, and refined via alternating SFT and reinforcement learning ('SFT-RL climbing') against live web search. Weights are available on Hugging Face, and the Iris Harness with agent loop, tools, and all four benchmarks is on GitHub.

The Decoder·Sep 13, 12:58 UTC · 3d agoModel release1

PIA-Bench: Towards Automated Privacy Impact Assessment with Large Language Models

Researchers release PIA-Bench, the first open benchmark evaluating how accurately LLMs can automate privacy impact assessments using 73 curated federal PIAs.

PIA-Bench is the first open benchmark for evaluating large language models on real-world privacy impact assessments (PIAs). The authors audited 499 expert-authored PIAs published by US federal agencies and curated 73 structured PIAs comprising 451 privacy risk items and 831 mitigation items. Off-the-shelf LLMs were found to produce meaningful assessments while identifying clear avenues for improvement. The paper calls for domain-specific LLM agent workflows, accountable LLM infrastructure, and new quality standards for PIAs.

arXiv cs.CR·Sep 11, 08:16 UTC · 5d agoResearch1