Search: “BigQuery”

Data access: the hidden cost of security vendor lock-in

Elastic compares SIEM data egress cost, latency, and fidelity across CrowdStrike, Microsoft, Google, and Splunk, arguing vendors engineer lock-in.

Elastic Security Labs published an opinion piece comparing how major SIEM and security vendors handle data egress, based on each vendor's public documentation as of September 2026. It rates CrowdStrike Falcon Data Replicator and Palo Alto Networks XSIAM Event Forwarding as restricted (paid add-ons with batch delays), Microsoft as partially open, Splunk as open, and Elastic as open with no export license. The piece argues frictionless ingestion paired with licensed or delayed egress is an intentional lock-in business model, and cites CrowdStrike's 2026 Global Threat Report eCrime breakout time of 29 minutes to argue real-time telemetry access is now essential.

Elastic Security Labs·Sep 4, 00:00 UTC · 12d agoIndustry

Towards Scalable and Cost-Efficient Vulnerability Detection: A Study on Automatic Query Generation

A study finds LLM-synthesized CodeQL queries improve average F1-score by 82% over baseline queries, offering scalable vulnerability detection versus direct LLM scanning.

Researchers conducted an empirical study evaluating whether LLMs can synthesize executable CodeQL queries from National Vulnerability Database vulnerability data. LLM-generated queries significantly enhanced baseline CodeQL suites, yielding an 82% improvement in average F1-score across a diverse set of real-world vulnerabilities. A cost-benefit analysis shows direct LLM-based scanning of entire repositories is often computationally and financially prohibitive, while LLM query synthesis offers a scalable and cost-effective alternative for large-scale vulnerability detection.

arXiv cs.CR·Sep 9, 16:31 UTC · 7d agoResearch1

Now everyone can put data to work

OpenAI launched a Data agent in ChatGPT Work that connects to enterprise warehouses and builds shareable analysis dashboards without SQL.

OpenAI introduced a Data agent in ChatGPT Work that connects to approved sources including Snowflake, BigQuery, Databricks, Redshift, ClickHouse, MongoDB, and Datadog, plus files from Google Drive and SharePoint. It investigates metric changes, builds interactive dashboards, and integrates with BI tools such as Power BI, Tableau, Omni, Sigma, and ThoughtSpot using semantic layers from dbt, Databricks Genie Ontology, and Snowflake Horizon. Queries enforce the connected account's existing table, row, and column permissions, with administrators controlling access via Workspace settings. OpenAI says nearly all of its product team and over two-thirds of its GTM organization use it internally, and NTT Data, Thermo Fisher, and ServicePiston are Alpha customers.

OpenAI News·Sep 10, 15:00 UTC · 6d agoAI industry

Bulk AbuseIPDB reporting using command-line tools

A blogger shows how to bulk-report distributed botnet IPs scraping a self-hosted Gitea instance to AbuseIPDB using bash, awk, and jq.

The author's self-hosted Gitea instance was scraped by distributed bots using random IPs worldwide and randomized, often nonexistent User-Agent strings, generating heavy request load for days. The write-up demonstrates filtering Traefik access logs (CLF and JSON) with grep, awk, and jq to extract offending IPs, tagging them with AbuseIPDB category 19 (Bad Web Bot), deduplicating them, and building a CSV for the Bulk Reporter tool. Reports can then be submitted through the AbuseIPDB bulk-report API, respecting the 5,000-report daily limit.

Lobsters · security·Sep 9, 20:26 UTC · 6d agoTools1

Google's new AI model predicts the future from sales data, weather, and discount schedules

Google Research released TimesFM-3, a 330M-parameter multivariate time series forecasting model that tops Gift-Eval, FEV-Bench, and Time benchmarks and is on Hugging Face.

Google Research released TimesFM-3, a 330-million-parameter Transformer-based time series forecasting model trained on more than one trillion real and synthetic data points. It works zero-shot and adds multivariate support, ingesting related series, historical-only covariates, and known future events such as discount schedules and weather forecasts, while filling all future time steps in a single one-shot pass. Google reports first place among pretrained forecasting models on Gift-Eval, FEV-Bench, and Time, ahead of Amazon's Chronos-2, the Toto-2.0 family, and its own TimesFM-2.5. Weights are available on GitHub and Hugging Face, with BigQuery integration planned in the coming weeks.

The Decoder·Sep 12, 09:26 UTC · 4d agoModel release2

Toolkit Hidden Inside Oracle Database Evades Endpoint Tools

Infosecurity Magazine·Aug 6, 15:30 UTC · Aug 6, 2026Vulnerability

Introducing WeatherNext 3, our most advanced and accurate global weather AI model

Google DeepMind launches WeatherNext 3, an AI weather model delivering hourly 5-km forecasts from live satellite data, now integrated across Google products.

WeatherNext 3 ingests live geostationary satellite mosaics and station observations through a Functional Generative Network (FGN) mesh transformer, producing hourly forecasts at 5-km surface resolution versus WeatherNext 2's 25-km, 6-hour grid. Independent live evaluations by Brightband rate it the most accurate global weather model to date. It adds renewable-energy variables such as 100-meter turbine-height wind speeds and solar radiation, and is integrated across Search, Gemini, Maps, Google Maps Platform, and Cloud.

Google DeepMind·Sep 3, 15:02 UTC · 13d agoModel release

LongAgent: History-Guided Agentic Search for Longitudinal Outcome Prediction

LongAgent autonomously searches variable sets and temporal windows to predict longitudinal medical outcomes, beating the strongest non-agent baseline on synthetic data.

The paper proposes LongAgent, an agent-based method that searches over combinations of variable sets, temporal windows and aggregation functions for outcome prediction on heterogeneous medical longitudinal data. It uses a history memory of previous searches and numerical evidence to guide exploration. On synthetic data it achieves mean RMSE 1.7376, improving over the best non-agent baseline by 0.0151 (95% CI [0.0045, 0.0260]; p=0.0273), and performs comparably to the best baseline on a real clinical dataset.

arXiv cs.AI / cs.LG / cs.CL·Sep 14, 16:51 UTC · 2d agoAI research

Atlas: Efficient Verifiable Semantic Search

Atlas delivers zero-knowledge proofs for HNSW semantic search, verifying RAG retrieval in under a second on SIFT1M and 2.0 seconds at 100M vectors.

Atlas lets a search provider prove that a query was answered correctly against a committed HNSW index without revealing the index, addressing provider deviations like truncation or bias. It combines offline preprocessing, a fixed-size-state restructuring of HNSW with a correctness proof, and timestep-tagged batching of per-step arguments. The system proves queries in under a second on SIFT1M and 2.0 seconds at 100 million vectors while preserving plaintext HNSW recall, and proven retrieval maintains end-to-end RAG answer quality at lower cost than prior verifiable retrieval systems.

arXiv cs.CR·Sep 10, 17:30 UTC · 6d agoResearch1

Grouped Value Attention: Efficient KV Caching via On-Demand Key Reconstruction

Grouped Value Attention stores grouped values and reconstructs content keys via a learned linear map, cutting KV-cache size about 45-47% versus GQA.

GVA stores only grouped values and reconstructs content keys with a learned linear map absorbed into the query at decode time, while a small shared decoupled RoPE channel preserves positional information via a separately cached positional key. At 350M parameters trained on 30B FineWeb-Edu tokens, the 16-dimensional positional variant scores 44.18 average accuracy across five tasks versus 44.36 for GQA and 43.88 for MLA. Custom decoding kernels are in development with an open-source release planned.

Hugging Face daily papers·Sep 7, 20:00 UTC · 8d agoAI research

From Parameters to Answers: How LLMs Retrieve and Use Their Internal Knowledge

Interpretability study traces how Qwen, Llama, and Gemma route query information and internal knowledge across layers when answering questions.

Researchers used layerwise interventions on hidden states to separate query-routing signals from target knowledge in language models, testing Qwen, Llama, and Gemma on country-continent questions with varied answer types. A pair-conditioned request direction strengthens before interventions alter downstream knowledge, opening a causal window while answer-supporting content is still forming. Trajectories differ by model: Gemma shows a partially overlapping mid-layer routing profile, while Llama has no sustained routing-effect window under the same gates.

arXiv cs.AI / cs.LG / cs.CL·Sep 10, 17:39 UTC · 6d agoAI research1

Bad Likert Judge: A Novel Multi-Turn Technique to Jailbreak LLMs by Misusing Their Evaluation Capability

Unit 42 details the Bad Likert Judge multi-turn jailbreak that abuses LLMs' evaluation capability, raising attack success rates over 60% across six frontier models.

Palo Alto Networks Unit 42 describes the Bad Likert Judge technique, a multi-turn jailbreak that asks a target LLM to act as a Likert-scale judge scoring the harmfulness of example responses. The highest-rated example in each scale can carry harmful content, bypassing the model's internal guardrails. Testing across six state-of-the-art text-generation LLMs showed an average attack success rate increase of more than 60% versus plain attack prompts, with tested models anonymized. The technique targets edge cases rather than typical use, and the article positions the work as guidance for defenders on potential jailbreak risks.

Palo Alto Unit 42·Aug 17, 12:34 UTC · Aug 17, 2026AI safety & security

An Empirical Analysis of CodeQL False Positives and Query Refinements for Java Vulnerabilities

Study of 167 Java CVE instances finds CodeQL false positives follow recurring patterns; query refinements remove 81.8% of reviewed ones.

Researchers ran CodeQL's Java security query suite on 167 CVE instances from 110 projects, manually reviewing 500 sampled false-positive paths and building a five-category taxonomy led by Missed Path Constraint or Sanitization (36.6%), Benign Execution Context (29.4%), and Missing Trust Boundary Modeling (27.6%). Guided by the taxonomy, query-level refinements removed 81.8% of reviewed false positives and 15.8% of reported paths across the selected queries while retaining 7 of 8 true positives. To address generalization, agentic coding tools given the refinement patterns as templates adapted them to new projects successfully in 56% and 62% of tasks, versus 28% without guidance.

arXiv cs.CR·Sep 3, 22:48 UTC · 12d agoResearch1

When Should LLMs Abstain? Chain-of-Self-Questioning for Selective Risk Control

Chain-of-Self-Questioning prompting cuts LLM wrong-answer commitments 32% relative while raising answered accuracy, holding across eleven model families.

The paper introduces Chain-of-Self-Questioning (CoSQ), a prompt-only framework that makes LLM answer commitment conditional on an explicit assessment of the information required to answer. On an 817-item TruthfulQA multiple-choice set, Grounded-CoSQ at τ=0.90 reduced mean unconditional wrong-commitment rate from 13.1% under chain-of-thought to 8.9% (a 32.1% relative reduction), while raising answered accuracy from 86.9% to 89.7% at 87.6% coverage. Improvements held across eleven open-weight and hosted model families and at every evaluated threshold, with convergent evidence from a Natural Questions short-answer evaluation.

arXiv cs.AI / cs.LG / cs.CL·Sep 15, 17:52 UTC · 1d agoAI research

PARSER: Read in Parallel, Reason in Depth for Long-Context LLM Agents

PARSER uses parallel reader subagents and an RL-trained lead agent for long-context QA, beating baselines and cutting latency up to 11x.

The PARSER paper decouples reading from reasoning: frozen subagents each read one document chunk in parallel while an RL-optimized lead agent iteratively broadcasts queries and aggregates evidence in scatter-gather rounds. On multi-hop QA with 7K to 896K token contexts, a 4B-backbone PARSER beats the strongest sequential memory baseline by 5.7 points on average and 12.0 points at 896K tokens, and a 9B version surpasses DeepSeek-V4-Pro by 6.3 points. Controlled experiments show robustness to evidence position, order, and distance perturbations, with inference latency reduced by up to 11x.

Hugging Face daily papers·Sep 5, 20:00 UTC · 10d agoAI research1

PostgreSQL Fixes 12-Year-Old Logical Decoding Flaw Enabling Replication-Role Code Execution

PostgreSQL fixed CVE-2026-6471 (CVSS 7.2), a 12-year-old logical decoding flaw letting replication-role accounts execute arbitrary code as the database OS user.

PostgreSQL patched CVE-2026-6471 (CVSS 7.2), a flaw present since logical decoding shipped in 9.4 (2014) that lets an account with the REPLICATION attribute run arbitrary code as the database server's OS user when wal_level is logical. Versions before 18.6, 17.11, 16.15, 15.19, and 14.24 are affected; exploitation requires such an account, which backup, CDC, and monitoring tooling routinely holds. The fix adds an output_plugin_libraries whitelist, so administrators using non-default plugins such as wal2json and decoderbufs must add them and reload configuration after updating. Cyera Research, whose write-up names the flaw PostGREShell, demonstrated escalating a replication user to superuser with persistence; no PoC or KEV entry existed as of September 4.

The Hacker News·Sep 4, 15:20 UTC · 12d agoVulnerabilityCVE-2026-64711

IDORacle: Template-Guided SQL-Sink Mediation for Object-Level Authorization in Java Applications

IDORacle intercepts MyBatis/JDBC SQL sinks to block horizontal privilege escalation in legacy Java apps with sub-millisecond guard latency.

IDORacle is a template-guided SQL-sink interception and rewriting framework that prevents IDOR/BOLA horizontal privilege escalation at runtime in Java database applications. It propagates authenticated identity context across HTTP requests, asynchronous tasks, and data-access boundaries via a server-side trace identifier, and computes dual SQL fingerprints at the MyBatis/JDBC boundary to generate reusable mediation plans. On a Java-SQL benchmark grounded in real-world CVE reports, it blocks tested violations with worst-case guard latency of 0.17 ms, reduced to 0.017 ms average for hot templates via redundancy-aware optimization.

arXiv cs.CR·Sep 11, 04:20 UTC · 5d agoResearch1

Molecular Déjà Vu: Digit-Level Retrieval of Published Values in Frontier Language Models

Audit of 22 frontier models finds widespread verbatim retrieval of published molecular property values, with higher reasoning increasing recall of memorized numbers.

An arXiv audit tests 22 frontier LLMs across 12 molecular regression benchmarks for verbatim retrieval of published values. More than 50% of the LLMs show verbatim retrieval on five datasets, and identical experiments are flagged 89% more often at a high reasoning level than at the lowest one. Suppressing retrieval moves model prediction errors closer together in relative terms, suggesting predictive capability is not determined solely by memorized values.

arXiv cs.AI / cs.LG / cs.CL·Sep 4, 17:32 UTC · 12d agoAI research1

PLEASE_READ_ME: The Opportunistic Ransomware Devastating MySQL Servers

Guardicore Labs uncovers the PLEASE_READ_ME ransomware campaign targeting MySQL servers, using double extortion and publishing stolen victim data.

Guardicore Labs at Akamai uncovered an opportunistic ransomware campaign dubbed PLEASE_READ_ME that targets MySQL servers. The attackers employ double extortion, publishing stolen data to pressure victims into paying. The campaign is devastating internet-facing MySQL deployments, making exposed database servers the primary at-risk population.

Akamai Blog·Sep 8, 15:13 UTC · 8d agoRansomware in the wild

Future-Proofing the Internet: Akamai Achieves End-to-End PQC

Akamai says it achieved end-to-end post-quantum cryptography across its network to future-proof internet traffic.

Akamai announced it has achieved end-to-end post-quantum cryptography (PQC) across its services. The company frames the move as future-proofing the internet against quantum decryption threats. No article text was available; details are limited to the announcement title.

Akamai Blog·Aug 19, 08:00 UTC · 28d agoIndustry

A Three-Layer Caching Architecture for Low-Latency LLM Web Search on Commodity CPU Hardware

OreoLook's three-layer Redis caching architecture cuts redundant LLM calls and embedding work for CPU-hosted web-search answer generation.

The paper describes a three-layer caching architecture for OreoLook (formerly lixSearch), an open-source LLM answer engine: a Redis session context window with Huffman-compressed disk overflow, a semantic query cache matching rephrasings via embedding cosine similarity, and a URL embedding cache deduplicating embedding computations. Deployed on a single 8-vCPU Intel Cascade Lake server with 30 Hypercorn workers across three containerized replicas, it achieved an 89.3% aggregate Redis keyspace hit rate, 0.1 ms read latency, and 1.38 MB memory overhead. An LRU eviction daemon migrates idle sessions to disk and rehydrates them for resumption hours or days later.

Hugging Face daily papers·Aug 11, 20:00 UTC · Aug 11, 2026AI tools & infra1

CVE-2026-73334: Apache Parquet Hadoop: File-controlled KMS URL is forwarded to pluggable KmsClient that skips host validation

Apache Parquet Hadoop CVE-2026-73334: a file-controlled KMS URL reaches pluggable KmsClients without host validation in parquet-java 1.12 through 1.18.0.

Apache disclosed CVE-2026-73334, a moderate issue in the org.apache.parquet.crypto.keytools package of parquet-java, versions 1.12 through 1.18.0. The package implements envelope encryption that wraps data keys via a Key Management Service. A KMS URL controlled by the Parquet file is forwarded to a pluggable KmsClient that skips host validation, which could allow crafted files to redirect KMS requests.

oss-security·Sep 8, 15:46 UTC · 8d agoVulnerabilityCVE-2026-733341

PostgreSQL Hit by 12-Year-Old Vulnerability Allowing Server Takeover

PostgreSQL flaw CVE-2026-6471, present since 2014, lets low-privilege replication accounts execute code and take over database servers; patches released.

Cyera researchers disclosed PostGREShell (CVE-2026-6471, CVSS 7.2), a missing-authorization bug in PostgreSQL logical decoding that lets users with REPLICATION privilege dlopen arbitrary files and execute code as the server account. The flaw affects PostgreSQL 9.4 through 18 and can yield privilege escalation, permanent superuser access and persistent backdoors on Windows, Linux and macOS. Fixes are available in versions 18.6, 17.11, 16.15, 15.19 and 14.24. Defenders should update and remove unnecessary Replication privileges.

Security Affairs·Sep 4, 13:41 UTC · 12d agoVulnerabilityCVE-2026-64711

Characterizing Language Generation in the Limit: Finite Witnesses and a Separation-Width Hierarch

New work characterizes language generation in the limit via finite witnesses, proves a full separation-width hierarchy, and formalizes all results in Lean.

The paper fully characterizes when language generation in the limit is possible for arbitrary families over a countable universe: each target must admit a finite positive witness such that targets activated by any finite sample share an infinite common intersection. It defines positive separation width and proves every level of the resulting hierarchy occurs, with countable families admitting singleton witnesses and unions of families with infinite common cores requiring unbounded finite witnesses. The characterization, a universal normalization, and a diagonal capture lemma are machine-checked in the Lean proof assistant, with the development maintained on GitHub.

arXiv cs.AI / cs.LG / cs.CL·Sep 9, 17:54 UTC · 7d agoAI research1

ChainDrop Worm Hits 400 npm Packages with Two Billion Monthly Installs

Infosecurity Magazine·Aug 5, 10:00 UTC · Aug 5, 2026Malware

AI SIEM Search

Huntress launched AI Search for Managed SIEM, translating plain-English questions into ESQL log queries with schedulable custom alerts.

Huntress introduced AI Search in its Managed SIEM, letting users query logs in natural language instead of writing ESQL, KQL, or SPL. The feature generates a real ESQL query behind the scenes, exposes it for users to inspect and learn from, and allows saved searches to be scheduled as recurring alerts at hourly, daily, or weekly cadence. A planned next version will return summarized plain-language answers instead of raw logs.

Huntress·Sep 7, 14:00 UTC · 9d agoTools

The Fraud Ecosystem: A Transition From Known Marketplaces to a Fragmented Environment

Rapid7 analyzes how fraud marketplaces are fragmenting into specialized shops after larger marketplaces were dismantled, aided by new MITRE F3 framework

Rapid7 reports a shift from large known fraud marketplaces to a fragmented environment of smaller specialized storefronts such as Xleet, Blackpass, Infodig, and Styx, operating across dark web channels, Telegram, and P2P options. These Fraud-as-a-Service shops sell stolen accounts, PII, synthetic identity generation, infrastructure, and money laundering support, supporting schemes like business email compromise. MITRE's Fraud Fighting Framework (F3), introduced in early 2026, aims to help security teams prioritize monitoring of fraud TTPs, particularly account takeover techniques. Fraud damages are anticipated to approach hundreds of billions of USD.

Rapid7 Blog·Sep 11, 13:33 UTC · 5d agoPhishing & fraud

LLM Agents as Computational Typologists

AUTOTYPOLOGIST is an LLM agent that performs evidence-grounded linguistic typology analysis over 25 open-source reference grammars.

The agent retrieves relevant grammar sections, analyzes interlinear glossed text (IGT), and iteratively reasons over typological hypotheses in a ReAct-style workflow. It was evaluated on typological feature coding against expert annotations and hypothesis testing against universals using 25 open-source reference grammars. Results suggest LLM agents can support scalable, inspectable crosslinguistic analysis but still require expert validation.

arXiv cs.AI / cs.LG / cs.CL·Sep 7, 17:35 UTC · 9d agoAI research1

Beyond Top-k Skill Retrieval: Diversity-Aware Skill Routing for LLM Agents

DSR reranks LLM agent skills with Determinantal Point Processes to favor complementary, non-redundant sets, improving multi-skill query coverage.

The paper proposes Diverse Skill Routing (DSR), a diversity-aware reranking framework for LLM agent skill routing that uses a Determinantal Point Process to balance query relevance and non-redundancy across large skill registries. DSR introduces a query-residual diversity kernel that penalizes redundant skill overlap while avoiding penalties arising only from shared query relevance. On the SkillRouter benchmark, DSR improves recall and full coverage over a strong pointwise reranking baseline, with the largest gains on multi-skill queries. The authors argue skill routing should be treated as complementary set selection, not just relevance ranking.

Hugging Face daily papers·Sep 4, 20:00 UTC · 11d agoAI research

Benchmark Radar: A Living Database and Search Engine for AI Benchmarks and Evaluation

Benchmark Radar provides a living searchable database of 1,283 AI benchmark records and 12,916 score observations drawn from 37 daily discovery sources.

Benchmark Radar combines daily discovery of benchmark papers, repositories, datasets, and releases from 13 direct connectors and 24 first-party feeds into a searchable catalog with model card mentions and score histories. The catalog contains 1,283 source records drawn from 4 benchmark catalogs plus 12,916 numeric observations on 790 records. The release includes a web dashboard with leaderboard, Pareto frontier of score versus usage, saturation and trend views, daily feeds, a CLI, and reproducible analysis. The paper audits the full catalog and examines benchmark saturation and limits of score comparisons.

Hugging Face daily papers·Sep 9, 20:00 UTC · 6d agoAI research

Meme Coin Factories: Uncovering Large-Scale Manipulations on pump.fun

Large-scale pump.fun study of 15 million meme coins identifies five manipulation classes including wash trading and a Market-Manipulation-as-a-Service ecosystem.

Researchers analyzed all 15 million coins launched on pump.fun over the last two years plus large random samples of transaction data, identifying five manipulation classes: wash trading, creator address obfuscation, coordinated sells, copycat coins, and social media manipulation. Strategic actors bypass the platform interface and implement strategies in a highly automated, low-latency way by interacting directly with the blockchain. The study also uncovers Market-Manipulation-as-a-Service (MMaaS) third-party tools that let non-technical users run these manipulations, and proposes mitigations for traders, pump.fun, and regulators.

arXiv cs.CR·Sep 9, 14:35 UTC · 7d agoResearch

Decade-old PostgreSQL flaw turns backup account into a backdoor

Cyera disclosed CVE-2026-6471, a decade-old PostgreSQL flaw letting REPLICATION-attribute accounts achieve RCE and superuser compromise, patched August 13.

Cyera Research's 'PostGREShell' flaw exists in PostgreSQL's logical replication output plugin loading, where the replication code path never called the library-path security check, allowing path traversal or UNC paths to load arbitrary code. A low-privilege backup account can achieve code execution on Windows, Linux, and macOS, then escalate to superuser with persistent backdoor access. CVE-2026-6471 affects versions back to PostgreSQL 9.4 (2014) and was patched in releases 18.6, 17.11, 16.15, 15.19, and 14.24 on August 13. CVSS is 7.2, and no exploitation of this specific CVE has been confirmed, though VirusTotal hunting found 114 malicious PostgreSQL plugins.

CSO Online·Sep 3, 11:38 UTC · 13d agoVulnerabilityCVE-2026-6471

Panic builds over bankrupt Spirit’s looming data sale to Google

Startups object to Google's bankruptcy-auction purchase of Spirit Airlines operational data, claiming proprietary IP is being sold without consent.

Google won an auction to acquire a large enterprise dataset from bankrupt Spirit Airlines, which it says will help improve its products and AI models, with no personal information included. Springshot, whose airline logistics platform powered Spirit's stack, filed a limited objection arguing the vaguely defined data categories could transfer third-party IP and trade secrets it owns; International Aero Engines filed a similar objection. The EFF called it the first public bankruptcy proceeding over selling company and employee data as an asset, and objectors warn of a precedent letting large companies acquire startup IP through bankruptcy courts.

Ars Technica · AI·Sep 10, 18:14 UTC · 6d agoAI industry

K-Bench: a clinically calibrated benchmark for evaluating large language models in high-risk mental health conversations

Clinician-calibrated K-Bench evaluates 125 LLM configurations on 200 high-risk mental health vignettes, exposing wide variation in suicide and violence risk handling.

K-Bench is a clinician-calibrated, protected benchmark evaluating 125 model configurations from 33 base models across 14 providers on 200 multi-turn vignettes covering suicide, self-harm, domestic violence, substance misuse and no-risk presentations. A frozen GPT-4o judge achieved 94.2% exact agreement with clinician consensus across 6,751 eligible comparisons from 151 clinician-rated transcripts. Leading models combined supportive conversation with combined-risk scores above 95, while risk exploration varied substantially among weaker configurations; therapeutic prompting helped weaker models and elevated reasoning produced no average improvement. A continuously updated public leaderboard is hosted at k-bench.ai with protected test materials.

arXiv cs.AI / cs.LG / cs.CL·Sep 14, 16:49 UTC · 2d agoAI safety & security