Search: “bcjr”

30 stories

DomainAll Cyber AIWindow24h 3d 7d 30d AllIn the wild Important only Show duplicates

SeverityAll critical 0 high 0 medium 0 low 0 info 1

Smart search ranks by meaning as well as keywords (one row per story, last 45 days).

Pre-Whitening and BCJR Posterior Distillation for Bi-LSTM Detection in Faster-than-Nyquist Signaling

Study shows nested-window Bi-LSTM architectures do not improve faster-than-Nyquist detection; pre-whitening plus BCJR distillation cuts bit error rates.

Across roughly 260 controlled trainings, processing nested intersymbol-interference windows in separate recurrent branches never significantly beat a plain Bi-LSTM at matched parameter budgets. The authors attribute the limitation to the observation model rather than architecture, and instead pre-whiten inputs and distill BCJR soft posteriors into the network. With 3.4% more parameters, the method reaches 1.05x the BCJR bit error rate at compression factor 0.8 and 1.89x at 0.7, improving to 1.47x with a wider whitened window.

arXiv cs.AI / cs.LG / cs.CLSep 7, 17:07 UTC · 9d agoAI research

K-Bench: a clinically calibrated benchmark for evaluating large language models in high-risk mental health conversations

Clinician-calibrated K-Bench evaluates 125 LLM configurations on 200 high-risk mental health vignettes, exposing wide variation in suicide and violence risk handling.

K-Bench is a clinician-calibrated, protected benchmark evaluating 125 model configurations from 33 base models across 14 providers on 200 multi-turn vignettes covering suicide, self-harm, domestic violence, substance misuse and no-risk presentations. A frozen GPT-4o judge achieved 94.2% exact agreement with clinician consensus across 6,751 eligible comparisons from 151 clinician-rated transcripts. Leading models combined supportive conversation with combined-risk scores above 95, while risk exploration varied substantially among weaker configurations; therapeutic prompting helped weaker models and elevated reasoning produced no average improvement. A continuously updated public leaderboard is hosted at k-bench.ai with protected test materials.

arXiv cs.AI / cs.LG / cs.CLSep 14, 16:49 UTC · 2d agoAI safety & security

JSCeal Hides Crypto Malware in V8 Bytecode

Check Point details JSCeal, a crypto-stealing malware delivered as compiled V8 bytecode, and built a decompiler pipeline to expose its capabilities.

JSCeal, tracked by Check Point Research since early 2025, is a cryptocurrency stealer delivered as heavily obfuscated JavaScript compiled to V8 bytecode (.jsc) and run by a bundled Node.js runtime. Check Point, presenting at Black Hat USA 2026, extended the open-source View8 decompiler into a pipeline that recovered readable code from 23 samples, revealing theft of browser credentials and cookies, Telegram sessions, keystrokes, screenshots, HTTPS traffic interception with attacker-controlled certificates, and automated Google account takeover. Targeted overrides rewrite content for Binance, Bybit, and Ledger users. Later samples added AES-256-CBC encryption with externally supplied keys and expanded targeting to macOS.

Security AffairsSep 7, 10:19 UTC · 9d agoMalware1

Retrofitting Code Using LLMs to Support Exceptional Behavior

EXCODER combines static/dynamic analysis with LLMs to retrofit exception-handling code, achieving 85.92% pass@1 with Qwen 2.5 Coder 32B on Java benchmarks.

The paper introduces the task of retrofitting existing code with Exception Related Code (throw statements, guarding conditions, try/catch blocks) so that given Exceptional Behavior Tests pass. EXCODER performs context engineering by integrating static and dynamic program analysis output with LLMs; it was evaluated on a benchmark built from 304 methods across 75 GitHub Java projects. Combined with Qwen 2.5 Coder 32B, EXCODER achieves pass@1, 5, and 10 rates of 85.92%, 86.18%, and 86.51%, roughly 13 percentage points over baseline, and manual inspection reveals remaining limitations.

arXiv cs.AI / cs.LG / cs.CLSep 9, 16:18 UTC · 7d agoAI research1

ActReview: Rebuttal-Guided Training Data and Rubric Rewards for Actionable Peer Review Generation

ActReview post-trains Qwen3-8B-Base on OpenReview rebuttals to generate actionable peer-review feedback with grounded revision suggestions, benchmarked on 1,000 curated instances.

The paper defines Actionable Peer-review Generation as diagnostic claim generation plus revision suggestion generation and introduces ActReview, a rebuttal-guided post-training framework. From OpenReview review-rebuttal threads the authors build ActReview-40K, aligning reviewer weaknesses with author responses grounded in localized paper evidence, and post-train Qwen3-8B-Base with multi-task SFT followed by GRPO using weakness-specific rubric rewards. They also release ActReview-Bench, a human-curated 1,000-instance benchmark, on which ActReview outperforms prior specialized review-generation models on actionability and grounding while remaining competitive with strong prompt-based LLMs. Human evaluation confirms improved revision usefulness but identifies a remaining gap in technical accuracy.

Hugging Face daily papersSep 7, 20:00 UTC · 9d agoAI research

Payara 7.2026.1.RC1 Arbitrary EJB Method Invocation via Insecure Reflection in Payara Server

Payara Server 7.2026.1.RC1 HTTP EJB endpoints rely on attacker-controlled reflection and JNDI lookups without authorization, enabling arbitrary EJB method invocation.

Payara Server exposes multiple HTTP-accessible EJB invocation mechanisms that depend on attacker-controlled reflection, dynamic class loading, and unsafe deserialization. Remote clients can perform arbitrary JNDI lookups, resolve attacker-supplied class names, and invoke EJB business methods without sufficient authorization enforcement or input restriction. Both the deprecated InvokeEJBServlet and other endpoints are affected in version 7.2026.1.RC1.

Full DisclosureSep 4, 00:13 UTC · 12d agoVulnerability 2 sources

[AINews] Jev: a “System One Model” that only decides/classifies/routes/scores — >100x faster, >200x cheaper than small frontier LLMs

TypeSafe launches Jev, an RLCD-trained decision model claiming 20-200x faster, 40-400x cheaper classification than frontier LLMs, alongside Gemini 3.8 Live and Neon.

TypeSafe's Jev is a 'System One' decision model trained with RLCD, claiming 20-200x faster and 40-400x cheaper classification and routing than frontier LLMs with free output tokens and no hallucinated text. Google launched Gemini 3.8 Live and 3.8 Live Extended Thinking, supporting 97 languages and async tool calls, debuting #1 on Artificial Analysis' speech-to-speech index at 82.6. Periodic Labs' Neon is a ~1T-parameter XRD analysis model trained with RL on proprietary lab data using 1,300 H200s, lifting FrontierXRD success from 2.7% to 55.3% and beating GPT-6 Astra at lower inference cost.

Latent Space11:09 UTC · 9h agoModel release1

Towards a Deterministic Math Solver for Clinical Language Models

Paper shows handing arithmetic to a deterministic Python solver beats direct model calculation at 32B but not reliably at 7B on MedCalc-Bench.

Researchers test a Program-Solve interface where clinical LLMs write case-specific Python executed by a restricted local solver instead of doing arithmetic directly. On MedCalc-Bench Verified (1,100 cases, 55 calculators), Qwen2.5-32B-AWQ scored 90.53% with solver handoff versus 83.47% with direct arithmetic (+7.05 points), while Qwen2.5-7B gained an unreliable +3.29 points with a confidence interval spanning zero. The authors audited the benchmark against clinical guidelines and flagged 16 of 55 calculators for version, use, or coefficient concerns.

Hugging Face daily papersSep 8, 20:00 UTC · 8d agoAI research

The 2026 PNPL Competition: Word Classification and Efficient Cross-Subject Generalisation in LibriBrain100

The 2026 PNPL competition releases LibriBrain100, a MEG speech dataset with 32 extra subjects, targeting word classification and cross-subject BCI generalization.

The 2025 PNPL competition on non-invasive speech decoding from MEG achieved F1-macro scores of 95.6% for speech detection and 73.6% for phoneme classification, built on LibriBrain's ~50 hours of single-subject data. The 2026 edition extends this with LibriBrain100, adding 32 subjects (~40 minutes each) plus ~80 hours of within-subject data. Two tracks target within-subject word classification at scale and cross-subject generalization with subject-specific fine-tuning shrinking from ~40 to ~20 to ~10 minutes, aiming at clinically feasible non-invasive BCIs for people with profound paralysis.

Hugging Face daily papersSep 2, 20:00 UTC · 14d agoAI research

ModaLens: Measuring Image Sensitivity in Report-Conditioned Medical VLMs

ModaLens image-swap audit shows report availability cuts MedGemma-27B image sensitivity on MIMIC-CXR from 20.94% to 4.26% answer changes.

ModaLens is a paired image-swap audit measuring how report availability affects image sensitivity in report-conditioned medical VLMs. On MedGemma-27B across 3,199 paired MIMIC-CXR cases from 293 patients (14 questions per case), generated answers changed on 4.26% of image-swap trials with the report versus 20.94% without it, a 16.7-point paired difference (95% CI 15.6-17.7). The original prompt with a lowercase first-token readout gave 4.70% versus 17.07%, and the direction replicated in two further model lineages. Labels derived from reports limit conclusions about visual correctness; code, prompts, and run records are publicly released.

Hugging Face daily papersSep 13, 20:00 UTC · 3d agoAI research

[0day-rubbish] QuantaStor 6.8.3.018 Command injection in the alert-mail command via the smtpPassword field (8.8)

QuantaStor 6.8.3.018 has a CVSS 8.8 command injection in its alert-mail command, exploitable via the smtpPassword field.

0day Rubbish Research Team disclosed a command injection (CWE-78) in QuantaStor 6.8.3.018's alert-mail command, reachable through the smtpPassword field. The flaw scores CVSS 8.8 (AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H). No CVE identifier or evidence of active exploitation is mentioned in the disclosure.

Full DisclosureSep 8, 17:31 UTC · 8d agoVulnerability 2 sources

IndicTriMix: Developing Language Identification Datasets and Models for Tri-Language Code-Mixing

Researchers release IndicTriMix benchmarks and fine-tuned MuRIL and XLM-RoBERTa models for token-level language identification in tri-language code-mixed text.

The paper formulates token-level language identification in code-mixed text as a sequence labeling task and fine-tunes MuRIL and XLM-RoBERTa transformer models for Indian languages. It evaluates on Hindi, Gujarati, and Bengali configurations with manually annotated test sets and proposes two code-mixed generation approaches using parallel trilingual sentences. A public benchmark, annotated test sets, and fine-tuned models are released for reproducibility.

arXiv cs.AI / cs.LG / cs.CLSep 10, 17:36 UTC · 6d agoAI research1

Reference-Based Bias Detection in LLMs via Relative Representations of Hidden States

Researchers propose auditing LLM bias via relative hidden-state representations, detecting bias increases with 3-50x less compute than output-level benchmarks.

The paper introduces a reference-based bias auditing method that compares hidden-state representations across model variants, such as before and after fine-tuning, by encoding sentences relative to a fixed anchor set. The resulting Representational Bias Shift (Delta-B) correlates with output-level bias change in 15 of 18 tested settings, reaching |r| = 0.84 under full fine-tuning across WildGuardMix, DecodingTrust, and ToxiGen benchmarks. Thresholding Delta-B detects checkpoints whose bias increased with ROC AUC between 0.65 and 0.99 and beats a SEAT-based baseline, while auditing a model in about three minutes with 3-50x less compute.

Hugging Face daily papersSep 8, 20:00 UTC · 8d agoAI research1

Smishing Triad Hackers Use JWR Phishing Kit to Steal Cards, OTPs and Bank Credentials

Group-IB attributes large-scale smishing using the JWR real-time phishing kit to the Smishing Triad's Outsider cluster, harvesting card data, OTPs, and bank credentials.

Group-IB attributes a large-scale SMS phishing campaign to Outsider, an operator sub-cluster within the Smishing Triad phishing-as-a-service ecosystem, using a kit dubbed JWR. The Vue 2-based platform maintains real-time WebSocket communication with operators, enabling them to adapt pages live and harvest roughly 70 PII fields, card data, PINs, OTPs, identity document images, and digital wallet credentials via a dedicated PayPal sub-funnel. Unit 42 previously tied 194,345 malicious domains across 136,933 root domains to the broader operation since January 2024. Defenders can hunt for /api/open/ endpoints, /webSocket/QT/ paths, JWR-prefixed storage artifacts, and a hard-coded WebSocket token.

GBHackersupdated 13:39 UTC · 7h agofirst 11:57 UTC · 9h agoPhishing & fraud in the wild 2 sources

Breaking the Seal: Static Deobfuscation of JSCeal’s Compiled V8 Bytecode

Check Point's hasherezade details static deobfuscation of JSCeal, a V8-bytecode stealer targeting cryptocurrency applications since March 2024.

JSCeal is an infostealer distributed as compiled V8 bytecode (.jsc) executed by a bundled Node.js runtime, aimed at cryptocurrency applications. Other vendors track the same family under the names WEEVILPROXY or MeadowLocust. Check Point Research has tracked the campaign since early 2024, with activity dating back to March 2024. The write-up presents a static approach to unpacking the bytecode without executing it.

Check Point ResearchAug 31, 13:38 UTC · 16d agoMalware in the wild1

Lightweight Vision Transformer Compression for On-Device Plant Disease Detection in Resource-Constrained Agricultural Field Conditions

A unified ViT compression pipeline (H-BAC pruning, quantization, distillation) cuts plant-disease models 54.5x to 6.01 MB while keeping 95.13% accuracy.

Researchers combined Hessian-Balanced Adaptive Block Pruning (H-BAC), guided by second-order sensitivity estimation, with quantization and attention-based knowledge distillation to compress Vision Transformers for on-device chilli plant disease detection in India. On a 3-class cross-village, cross-device out-of-distribution dataset, the integrated pipeline reduced model size from 327.42 MB to 6.01 MB (54.5x) at 95.13 +/- 2.32% accuracy, matching the 95.13% FP32 baseline. Ablations also show a directly trained 6.01 MB INT8 student reaches 94.87% accuracy, indicating where pruning and distillation add limited value.

arXiv cs.AI / cs.LG / cs.CLSep 4, 16:40 UTC · 12d agoAI research

Emergency Department Revisit Quality Review Screening: Exploring Human Decision-Making and Artificial Intelligence Support

A retrospective study found GPT-4 over-flagged emergency department revisit cases while an LLM knowledge-graph screener achieved 83-100% positive predictive value.

In an exploratory retrospective study of 99 emergency department diagnosis pairs from a multihospital health system, clinicians and GPT-4 independently judged whether revisit pairs warranted further assessment. GPT-4 responses correlated poorly with clinicians, flagging 94% of pairs for follow-up, 4.4-13.3 times more than clinicians, though prompt engineering was minimal. An algorithm leveraging an LLM-populated knowledge graph (KGA) achieved 83-100% positive predictive value against at least one clinician rater, suggesting LLM-based screening could broaden revisit quality review without substantially increasing reviewer workload.

arXiv cs.AI / cs.LG / cs.CLSep 9, 16:37 UTC · 7d agoAI research1

Version Control DFIR: a Cheatsheet to GitHub, GitLab, Bitbucket, and Azure DevOps

Wiz published a DFIR cheatsheet covering log visibility, incident readiness, and threat hunting across GitHub, GitLab, Bitbucket, and Azure DevOps.

Wiz researchers released a practitioner's guide to version control system forensics, incident response, and threat hunting. The cheatsheet maps log sources, audit capabilities, and hunting workflows across GitHub, GitLab, Bitbucket, and Azure DevOps. It aims to improve incident readiness for source code and CI/CD compromise scenarios.

Wiz BlogAug 27, 12:00 UTC · 20d agoResearch1

Towards Scalable and Cost-Efficient Vulnerability Detection: A Study on Automatic Query Generation

A study finds LLM-synthesized CodeQL queries improve average F1-score by 82% over baseline queries, offering scalable vulnerability detection versus direct LLM scanning.

Researchers conducted an empirical study evaluating whether LLMs can synthesize executable CodeQL queries from National Vulnerability Database vulnerability data. LLM-generated queries significantly enhanced baseline CodeQL suites, yielding an 82% improvement in average F1-score across a diverse set of real-world vulnerabilities. A cost-benefit analysis shows direct LLM-based scanning of entire repositories is often computationally and financially prohibitive, while LLM query synthesis offers a scalable and cost-effective alternative for large-scale vulnerability detection.

arXiv cs.CRSep 9, 16:31 UTC · 7d agoResearch1

7th September – Threat Intelligence Report

Check Point weekly bulletin: exploited SonicWall SMA 1000 zero-days (CVE-2026-83548/49), JFrog Artifactory auth bypass, and a 2.8M-patient Baylor Genetics breach.

Check Point's weekly bulletin covers breaches including Thomson Reuters' C-Track platform (court records across 11 US states and Canada), Baylor Genetics (2.8M patients and employees, SSNs included), Hit casinos in Slovenia, and Dropbox (~5,000 accounts via abused Lenovo email verification). SonicWall patched two exploited SMA 1000 zero-days, including CVE-2026-83548, a CVSS 10.0 pre-authentication SSRF, while JFrog fixed exploited authentication bypass CVE-2026-82329 (CVSS 9.8). Malware and actor coverage includes Gambling Goblin, JSCeal, Mirage Kitten's NodeRabbit/PollCat campaign, and DPRK Contagious Interview macOS activity, plus a FalconFlank privilege-escalation PoC against CrowdStrike Falcon.

Check Point ResearchSep 7, 14:54 UTC · 9d agoAdvisory in the wildCVE-2026-83548 CVE-2026-83549 CVE-2026-823291

An Empirical Analysis of CodeQL False Positives and Query Refinements for Java Vulnerabilities

Study of 167 Java CVE instances finds CodeQL false positives follow recurring patterns; query refinements remove 81.8% of reviewed ones.

Researchers ran CodeQL's Java security query suite on 167 CVE instances from 110 projects, manually reviewing 500 sampled false-positive paths and building a five-category taxonomy led by Missed Path Constraint or Sanitization (36.6%), Benign Execution Context (29.4%), and Missing Trust Boundary Modeling (27.6%). Guided by the taxonomy, query-level refinements removed 81.8% of reviewed false positives and 15.8% of reported paths across the selected queries while retaining 7 of 8 true positives. To address generalization, agentic coding tools given the refinement patterns as templates adapted them to new projects successfully in 56% and 62% of tasks, versus 28% without guidance.

arXiv cs.CRSep 3, 22:48 UTC · 12d agoResearch1

Approval Integrity and Recovery in LLM Answer Publication

Study measures approval integrity in Lightcap LLM answer publication, finding the 14B response-act checker accepts 291 of 302 unsupported answers.

The study evaluates exact-content binding, authorization freshness, and checkpoint recovery in Lightcap's publication enforcement using 3,600 assessments over 900 human-annotated RAGTruth responses from three Ministral models. The production 14B response-act checker accepts 291 of 302 unsupported answers versus 41 for a direct-grounding baseline, with supported-answer retention of 95.2% versus 66.9%. A stateful recheck-recovery policy increases exact-match error by 9.23 percentage points relative to initial checkpoints, and controlled evidence-fingerprint changes expose asymmetric freshness enforcement between publication and recovery. A separate BIPIA prompt-injection experiment records zero target insertions among 266 valid editor outputs.

arXiv cs.CRSep 14, 13:48 UTC · 2d agoAI safety & security

CVE-2026-80351: Apache Camel K: Camel K Tenant repositories reach Maven execution inside operator pod

Apache Camel K CVE-2026-80351 lets tenant-controlled Maven repository content execute code inside the operator pod; fixed in 2.9.3/2.10.2.

Apache disclosed a critical eval injection vulnerability (CVE-2026-80351) in Apache Camel K, where improperly neutralized directives in dynamically evaluated Maven configuration let tenant-controlled repository content reach Maven execution inside the operator pod. This can allow tenant-controlled content to influence code execution in the operator context. Affected versions are 2.0.0 before 2.9.3 and 2.10.1 before 2.10.2; fixes are available in 2.9.3 and 2.10.2.

oss-securitySep 10, 14:06 UTC · 6d agoVulnerability 3 sources CVE-2026-803512

Evaluating Time-Series Foundation Models and Multimodal Dietary Context for CGM Forecasting

Study finds zero-shot time-series foundation models underperform on CGM forecasting; fine-tuned Chronos-Bolt cuts RMSE up to 18.4% and dietary context adds signal.

The paper evaluates time-series foundation models for continuous glucose monitoring forecasting across eight public datasets covering Type 1 diabetes, Type 2 diabetes, and non-diabetes populations. Under a unified protocol, zero-shot foundation models did not consistently outperform baselines like Elastic Net and PatchTST, but lightweight fine-tuning did, with fine-tuned Chronos-Bolt reducing RMSE by 6.5%-18.4% in the T1D cohort and 8.6%-18.2% in the non-diabetes/T2D cohort. A residual-based fusion framework adding dietary context from CGMacros reduced overall RMSE by about 3% and postprandial RMSE by about 15% versus CGM-only baselines.

arXiv cs.AI / cs.LG / cs.CLSep 10, 17:43 UTC · 6d agoAI research

Molecular Déjà Vu: Digit-Level Retrieval of Published Values in Frontier Language Models

Audit of 22 frontier models finds widespread verbatim retrieval of published molecular property values, with higher reasoning increasing recall of memorized numbers.

An arXiv audit tests 22 frontier LLMs across 12 molecular regression benchmarks for verbatim retrieval of published values. More than 50% of the LLMs show verbatim retrieval on five datasets, and identical experiments are flagged 89% more often at a high reasoning level than at the lowest one. Suppressing retrieval moves model prediction errors closer together in relative terms, suggesting predictive capability is not determined solely by memorized values.

arXiv cs.AI / cs.LG / cs.CLSep 4, 17:32 UTC · 12d agoAI research1

[20260801] - Core - Response header injection in download views

Joomla fixes CVE-2026-71572, response header injection in download views enabling reflected file download attacks, in 5.4.8/6.1.3.

Joomla disclosed a response header injection flaw (CVE-2026-71572) in multiple download views, caused by lack of output processing, enabling reflected file download and content-type confusion. It is rated low impact, severity, and probability and affects Joomla CMS 3.0.0-5.4.7 and 6.0.0-6.1.2. The fix ships in Joomla 5.4.8 and 6.1.3 on 2026-08-18.

Joomla Security CentreAug 17, 14:00 UTC · Aug 17, 2026Advisory CVE-2026-71572

Cross-Model Agreement as a Deployment-Time Reliability Signal for Automatic Polyp Segmentation

Referee-Based Quality Estimation flags unreliable polyp segmentations at inference without ground truth, reaching ROC-AUC 0.960 with SegFormer-B0 referees.

RBQE measures agreement between a primary segmentation model and an independently trained referee on a 1,223-image external benchmark drawn from four public datasets. A cross-architecture SegFormer-B0 referee achieves the strongest signal (ROC-AUC 0.960), beating a Test-Time Augmentation baseline by 0.055 ROC-AUC under an identical protocol. Excluding trivially separable empty-mask cases, ROC-AUC falls to 0.876 (SegFormer-B0) and 0.783 (same-architecture control), but RBQE's margin over baselines widens. Progressive rejection of low-agreement predictions increases mean Dice of retained outputs, supporting selective prediction at the cost of one extra forward pass.

arXiv cs.AI / cs.LG / cs.CLSep 9, 17:32 UTC · 7d agoAI research

[0day-rubbish] Accurate Online Private Cloud on-prem (current) Unauthenticated Hessian deserialization leading to JNDI remote class loading (9.8)

0day Rubbish disclosed an unauthenticated Hessian deserialization flaw in Accurate Online Private Cloud on-prem allowing JNDI remote class loading, rated 9.8.

The 0day Rubbish Research Team publicly disclosed an unauthenticated Hessian deserialization vulnerability in the current on-premises release of Accurate Online Private Cloud. The flaw lets unauthenticated attackers trigger JNDI remote class loading, a path that typically yields remote code execution. The issue carries a CVSS 9.8 rating. No CVE identifier or evidence of in-the-wild exploitation was included in the disclosure.

Full DisclosureSep 8, 17:31 UTC · 8d agoVulnerability

Target leakage, not model class, explains reported accuracy in survey-based cardiovascular screening: a leakage-tiered audit of glass-box and tabular foundation models

Audits of 10 classifiers on BRFSS show target leakage, not model class, drives the reported 0.89 AUROC in survey-based cardiovascular screening.

The study benchmarks ten model classes, including glass-box and tabular foundation models, for prevalent myocardial infarction on 442,067 respondents of the 2022 BRFSS across five feature tiers of decreasing leakage risk. Removing two post-diagnostic features costs every model 0.049-0.051 AUROC and collapses performance into a 0.0045-wide band, and the explainable boosting machine matches all alternatives within 0.005 while scoring roughly 104x faster than the strongest foundation model. Frozen models transport within 0.002 AUROC to 2023 data; the authors conclude evaluation practice and feature sets, not model capacity, are the binding constraint.

arXiv cs.AI / cs.LG / cs.CLSep 10, 17:29 UTC · 6d agoAI research1

IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications

IdeaAMBIG benchmark of 660 specification-gap instances shows LLMs localize implementation-critical research gaps poorly, with best model at 9.6% defect recovery.

IdeaAMBIG is a benchmark of 660 evidence-grounded instances evaluating whether research-method specifications provide enough information for faithful implementation: 163 real-world gaps from reproducibility reports and GitHub issues plus 497 controlled synthetic gaps. It tests codification-readiness assessment, defect localization, and clarification action generation across 13 LLMs. The best model achieves only a 9.6% Macro Defect Recovery Rate on real-world instances, though 80.6% clarification success when given the annotated defect, and an oracle study shows gold resolutions raise codification-ready rates from 14% to 98%. Defect localization emerges as the main bottleneck across all evaluated models.

Hugging Face daily papersSep 8, 20:00 UTC · 8d agoAI research