Search: “llm arithmetic”

Smart search ranks by meaning as well as keywords (one row per story, last 45 days).

Rethinking Heterogeneous System Disaggregation for Subquadratic Attention

SQD disaggregates LLM inference by quadratic versus subquadratic attention layers, improving energy efficiency up to 56% on heterogeneous systems versus GPU-only baselines.

SQD (SubQuadratic Disaggregation) splits decode not by operator type but by quadratic versus subquadratic attention, matching their distinct arithmetic intensity and memory footprints. For sparse attention LLMs it separates top-k selection (requiring full KV indexing) from top-k attention plus FFN; for linear and sliding-window models it separates dense attention layers from subquadratic layers plus FFN. On an adjusted 8xB200 heterogeneous proxy, tokens-per-joule improves 53% on GLM 5.2, 31% on Nemotron 3 Ultra, and 56% on Gemma 4 31B. A Rubin plus LPX analytical model shows 1.2x-1.5x tighter achievable latencies and up to 3.6x higher throughput versus attention-FFN disaggregation.

arXiv cs.AI / cs.LG / cs.CL·Sep 11, 17:55 UTC · 4d agoAI research

28

llm 0.34

Version 0.34 of Simon Willison's llm CLI adds response-duration metrics to log output, plus bug fixes and faster log querying.

The open-source llm command-line tool for interacting with large language models released version 0.34. The headline change adds response duration in milliseconds and human-readable form to llm logs --usage Markdown output, plus a new duration_ms field in llm logs --short. The release includes several contributed bug fixes and a significant performance improvement to llm logs, alongside the related llm-openrouter 0.7.1 release.

Simon Willison·Sep 2, 19:23 UTC · 13d agoAI tools & infra

28

How well do agents use test/verification techniques?

Dan Luu's eval finds coding-agent testing instructions (TDD, formal methods, PBT, skills) mostly fail to beat defaults on Zstd implementation correctness.

The author ran 26 prompt conditions plus 4 skills on a Zstd-in-Rust implementation eval using codex with GPT-5.6, testing TDD, fuzzing, property-based testing, formal methods (Lean 4, TLA+, Verus, Kani, SMT solvers) and community skills. Nothing dramatically outperformed the default no-instruction condition, which did above average; at xhigh effort, fuzzing and PBT conditions did slightly better than formal methods. Pre-registered predictions included TDD underperforming and popular test skills (ECC, Hegel, Trail of Bits) not outperforming. Results are averages of 80 runs per condition plotted against cost.

Hacker News · security·Sep 8, 02:58 UTC · 8d agoAI researchHN 131↑ · 50 comments

38

Training a 3.8B LLM to 0.384 CORE for $998 – Hugo Vergnes

Independent developer Hugo Vergnes trained a 3.8B-parameter Llama-style model to 0.384 CORE on 65B tokens for $998 in 43 hours on rented B200s.

Hugo Vergnes trained little-lm, a 3.848B-parameter decoder-only LLM, on 65.3B tokens in 43 hours for $998 using rented NVIDIA B200s, scoring 0.384 on the CORE benchmark and beating nanochat d32 (0.310) at similar cost. The Llama-style architecture uses RMSNorm, RoPE, GQA with 24 query and 8 KV heads, relu-squared MLPs, QK-norm, and ResFormer-style value embeddings that account for 19% of parameters. Key wins included the Muon optimizer for matrix parameters, a trapezoidal learning-rate schedule with linear cooldown, FP8 training plus vocabulary padding for roughly 33% throughput gains, and the ClimMix dataset over FineWeb-Edu. The project, inspired by Karpathy's nanochat, was built as a config-driven YAML framework for small LLM training.

Hacker News · security·Sep 10, 02:04 UTC · 6d agoAI researchHN 28↑ · 2 comments1

35

Towards a Deterministic Math Solver for Clinical Language Models

Paper shows handing arithmetic to a deterministic Python solver beats direct model calculation at 32B but not reliably at 7B on MedCalc-Bench.

Researchers test a Program-Solve interface where clinical LLMs write case-specific Python executed by a restricted local solver instead of doing arithmetic directly. On MedCalc-Bench Verified (1,100 cases, 55 calculators), Qwen2.5-32B-AWQ scored 90.53% with solver handoff versus 83.47% with direct arithmetic (+7.05 points), while Qwen2.5-7B gained an unreliable +3.29 points with a confidence interval spanning zero. The authors audited the benchmark against clinical guidelines and flagged 16 of 55 calculators for version, use, or coefficient concerns.

Hugging Face daily papers·Sep 8, 20:00 UTC · 7d agoAI research

25

llm 0.35

llm CLI tool version 0.35 adds support for OpenAI's new GPT-6 Astra model exposed as gpt-6-astra.

Simon Willison released llm 0.35, which adds an OpenAI model definition for GPT-6 Astra under the model ID gpt-6-astra. No other release details were provided in the post.

Simon Willison·Sep 7, 23:54 UTC · 8d agoAI tools & infra

25

Towards Scalable and Cost-Efficient Vulnerability Detection: A Study on Automatic Query Generation

A study finds LLM-synthesized CodeQL queries improve average F1-score by 82% over baseline queries, offering scalable vulnerability detection versus direct LLM scanning.

Researchers conducted an empirical study evaluating whether LLMs can synthesize executable CodeQL queries from National Vulnerability Database vulnerability data. LLM-generated queries significantly enhanced baseline CodeQL suites, yielding an 82% improvement in average F1-score across a diverse set of real-world vulnerabilities. A cost-benefit analysis shows direct LLM-based scanning of entire repositories is often computationally and financially prohibitive, while LLM query synthesis offers a scalable and cost-effective alternative for large-scale vulnerability detection.

arXiv cs.CR·Sep 9, 16:31 UTC · 6d agoResearch1

32

Towards Tackling Application Logic Flaws through Autonomous Formal-Logic Modeling and Automated Reasoning

LL-Verifier combines LLMs with logic model checking to automatically discover logic flaws, uncovering vulnerabilities in 27 IoT access-control protocols.

Researchers present LL-Verifier, a framework that uses LLMs to autonomously convert natural-language protocol descriptions and security goals into formal logic models in a new logic language built on Maude, then applies logic model checking for exhaustive verification. The framework targets application-logic flaws that are tied to business semantics and hard to scale with manual analysis. Evaluation on 27 access-control protocols of widely used IoT devices uncovered a range of sophisticated logic vulnerabilities with security and privacy implications.

arXiv cs.CR·Sep 9, 17:58 UTC · 6d agoResearch1

42

Recognition-Refusal Misalignment in LLMs: Why Models Answer Structurally Unanswerable Questions

A linear hidden-state direction encodes question impossibility in 1.7B-70B LLMs, but misalignment with the safety-refusal pathway explains why models answer unanswerable questions.

The study examines why instruction-tuned LLMs from 1.7B to 70B parameters answer structurally unanswerable math and code questions instead of abstaining. A single linear direction in the hidden state separates answerable from impossible prompts, showing models represent impossibility before generation, but this direction is nearly orthogonal to the canonical safety-refusal direction. Generation-time steering along the recognition direction changes invalidity-aware behavior dose-responsively, and the geometry is present even at the pretraining endpoint, indicating a routing failure rather than an encoding failure.

Hugging Face daily papers·Aug 28, 20:00 UTC · 18d agoAI safety & security

30

Beneath the Surface of Chains-of-Thought: A Mechanistic Interpretation of Reasoning Operations in LLMs

Study shows LLM reasoning operations like planning and deduction are geometrically separable in hidden states, with separability peaking in middle layers.

Researchers investigate whether functional reasoning operations — problem formulation, goal decomposition, deduction — have corresponding geometric structure in LLM hidden representations. They find operations are separable in held-out representations with separability peaking in middle layers, ruling out lexical and positional confounds; token-wise operation alignment becomes more distributed across layers, and identical surface tokens are represented differently depending on their surrounding chunk. Attention-masking interventions show chunk-onset operation-aligned representations depend on preceding reasoning context; code is released on GitHub (naver-ai/beneath-cot).

Hugging Face daily papers·Sep 3, 20:00 UTC · 12d agoAI research1

40

I accidentally turned LLM memory into program analysis

A pwning.systems write-up describes how LLM memory functionality was unexpectedly repurposed into a program analysis technique.

A security research post on pwning.systems describes the author's discovery that LLM memory behavior effectively functioned as program analysis. The write-up is hosted on a security-focused blog and surfaced via a security-tagged link aggregator. Detailed technical content is not included in this feed, limiting verifiable specifics.

Lobsters · security·Aug 29, 03:16 UTC · 18d agoResearch1

30

HyQuant: Hybrid-Precision Quantization for LLM Attention

HyQuant keeps most LLM attention states low-bit while preserving vertical-line tokens and local windows in high precision, maintaining near-lossless accuracy.

HyQuant is a hybrid-precision quantization framework for LLM attention that quantizes most attention states to low bits while keeping accuracy-critical vertical-line tokens and local-window states in full precision, selected via lightweight attention-pattern signals. In the prefill stage it uses a hybrid-precision attention operator, and in the decode stage it applies the same principle to KV-cache compression with fused dequantization and attention computation. Across diverse tasks, models, and datasets it maintains nearly lossless accuracy; code is available on GitHub.

Hugging Face daily papers·Aug 27, 20:00 UTC · 19d agoAI tools & infra1

33

Bad Likert Judge: A Novel Multi-Turn Technique to Jailbreak LLMs by Misusing Their Evaluation Capability

Unit 42 details the Bad Likert Judge multi-turn jailbreak that abuses LLMs' evaluation capability, raising attack success rates over 60% across six frontier models.

Palo Alto Networks Unit 42 describes the Bad Likert Judge technique, a multi-turn jailbreak that asks a target LLM to act as a Likert-scale judge scoring the harmfulness of example responses. The highest-rated example in each scale can carry harmful content, bypassing the model's internal guardrails. Testing across six state-of-the-art text-generation LLMs showed an average attack success rate increase of more than 60% versus plain attack prompts, with tested models anonymized. The technique targets edge cases rather than typical use, and the article positions the work as guidance for defenders on potential jailbreak risks.

Palo Alto Unit 42·Aug 17, 12:34 UTC · 29d agoAI safety & security

60

When Should LLMs Abstain? Chain-of-Self-Questioning for Selective Risk Control

Chain-of-Self-Questioning prompting cuts LLM wrong-answer commitments 32% relative while raising answered accuracy, holding across eleven model families.

The paper introduces Chain-of-Self-Questioning (CoSQ), a prompt-only framework that makes LLM answer commitment conditional on an explicit assessment of the information required to answer. On an 817-item TruthfulQA multiple-choice set, Grounded-CoSQ at τ=0.90 reduced mean unconditional wrong-commitment rate from 13.1% under chain-of-thought to 8.9% (a 32.1% relative reduction), while raising answered accuracy from 86.9% to 89.7% at 87.6% coverage. Improvements held across eleven open-weight and hosted model families and at every evaluated threshold, with convergent evidence from a Natural Questions short-answer evaluation.

arXiv cs.AI / cs.LG / cs.CL·Sep 15, 17:52 UTC · 15h agoAI research

30

Unlocking Lossless Speedups in LLMs via Discrete Diffusion

Uno pairs autoregressive LLMs with lightweight diffusion weights to draw multiple tokens in parallel, delivering up to 3x lossless speedup without a draft model.

The paper introduces diffusion-augmented LLMs: autoregressive weights trained with the standard next-token objective plus lightweight diffusion weights trained via a Diffusion Distillation phase to emit multiple tokens in parallel. Psi-Spec samplers enable lossless acceleration without the separate draft model required by speculative decoding. The 8B Uno model outperforms the 26B open DiffusionGemma and proprietary Mercury 2 on agentic tool use, coding, and long-context reasoning benchmarks, with up to 3x throughput gains over the base model at all evaluated batch sizes. Code and checkpoints are released publicly.

Hugging Face daily papers·Sep 2, 20:00 UTC · 13d agoAI research

55

Litelm: LiteLLM Without the Bloat

New open-source project Litelm launched on GitHub as a leaner alternative to the LiteLLM multi-provider LLM gateway.

Litelm is a community-published GitHub project positioned as a stripped-down alternative to LiteLLM, a widely used gateway/SDK for routing requests to multiple LLM providers. The post drew modest attention on Hacker News with 45 points and 17 comments. The listing provides no further technical details, benchmarks, or feature specifics.

Hacker News · AI·Sep 11, 18:10 UTC · 4d agoAI tools & infraHN 45↑ · 17 comments2

14

Ask HN: Anyone still coding like 2021? Where do you work?

Hacker News users debate coding without LLMs, with one developer fired for refusing AI tools and others describing daily hand-coding practice to counter skill atrophy.

An Ask HN thread collects experiences of developers who still write code without LLM assistance. One contributor says he was fired for political reasons after refusing to use LLMs despite adequate stated performance, and observes fewer job ads now require LLM use. Others describe starting each day with a LeetCode problem or 30-60 minutes of hand-coding to stay sharp, contractual bans on AI-generated code for a government-adjacent embedded product over unresolved copyright issues, and inconsistent corporate policies where ChatGPT or Codex use flip-flops between allowed and blocked while a CIO mandates 70-80% AI-generated code next year.

Hacker News · AI·Sep 9, 13:41 UTC · 6d agoAI industryHN 47↑ · 54 comments2

12

Large Language Models for HVAC Operations in Building Energy Systems: A Critical Review of Methods, Applications, and Deployment Readiness

Systematic review of 66 studies finds LLMs for HVAC operations are mostly research-stage, with no ready-now deployment and only four pilot-level studies.

A critical review of 66 peer-reviewed studies from 2023 to March 2026 examines LLMs for HVAC operations in building energy systems. Only four studies reach pilot-level evidence, none reports sustained operational deployment, and 63 of 66 are research-only. Conventional ML, MPC, and RL remain dominant for high-frequency control and short-horizon forecasting, and the evidence supports LLMs primarily as semantic and workflow layers rather than autonomous controllers.

arXiv cs.AI / cs.LG / cs.CL·Sep 4, 16:08 UTC · 11d agoAI research

45

Retrofitting Code Using LLMs to Support Exceptional Behavior

EXCODER combines static/dynamic analysis with LLMs to retrofit exception-handling code, achieving 85.92% pass@1 with Qwen 2.5 Coder 32B on Java benchmarks.

The paper introduces the task of retrofitting existing code with Exception Related Code (throw statements, guarding conditions, try/catch blocks) so that given Exceptional Behavior Tests pass. EXCODER performs context engineering by integrating static and dynamic program analysis output with LLMs; it was evaluated on a benchmark built from 304 methods across 75 GitHub Java projects. Combined with Qwen 2.5 Coder 32B, EXCODER achieves pass@1, 5, and 10 rates of 85.92%, 86.18%, and 86.51%, roughly 13 percentage points over baseline, and manual inspection reveals remaining limitations.

arXiv cs.AI / cs.LG / cs.CL·Sep 9, 16:18 UTC · 6d agoAI research1

25

What Else Needs Fixing? Exploring Cost-Effective Test-Time Compute for Revision Propagation in Artifacts Generated Through Conversation

A new benchmark shows LLMs reach 68.3-93% accuracy propagating local revisions across conversationally generated artifacts, with parallel-sample selection most cost-effective.

The paper introduces a benchmark for revision propagation: when users request a local change, LLMs must identify dependencies and update all affected parts of an artifact generated through conversation, where context lives in the chat history. Nine revision methods, including sequential reflection and parallel sampling variants, were evaluated on gpt-oss-20b/120b, gpt-5.4-mini, and qwen3.5-9b/27b/122b. Baselines scored 68.3-93% accuracy, and selecting among three parallel samples via LLM-based or medoid selection improved accuracy by 2.2-9.7% as the most cost-effective test-time compute strategy. Code and dataset are released.

Hugging Face daily papers·Sep 2, 20:00 UTC · 13d agoAI research

30

LLMs and Contextual Integrity

Bruce Schneier highlights two papers: the CIMemories benchmark shows frontier LLMs leak memory attributes up to 69%, and an RL method reduces inappropriate disclosures.

Bruce Schneier discusses contextual integrity in LLMs, referencing the CIMemories benchmark, which uses synthetic profiles with 100+ attributes per user to test whether models with persistent memory disclose sensitive information appropriately. Evaluation showed frontier models exhibit up to 69% attribute-level violations, with GPT-5's violation rate rising from 0.1% to 9.6% across 40 tasks and reaching 25.1% with repeated prompting, showing unstable leakage behavior. A second paper introduces a reinforcement learning framework trained on a synthetic 700-example dataset that substantially reduces inappropriate disclosure while maintaining task performance, with improvements transferring to the human-annotated PrivacyLens benchmark.

Schneier on Security·Aug 18, 10:40 UTC · 28d agoAI safety & security

48

Φ-Bench: Can Large Language Models Engineer the Infrastructure That Powers Them?

Researchers release Phi-Bench, a benchmark evaluating frontier LLMs on open-ended, long-horizon engineering and optimization of the LLM infrastructure stack.

Phi-Bench evaluates LLMs on open-ended engineering of the LLM infrastructure stack, derived from optimization problems studied in frontier research and grounded in real-world code repositories. Tasks range from localized kernel-level function completion to long-horizon implementation and end-to-end system optimization. Experiments on frontier LLMs reveal current capabilities and limitations on the path toward autonomous optimization of future AI infrastructure.

Hugging Face daily papers·Sep 8, 20:00 UTC · 7d agoAI research1

28

Procedural Graphs: Self-Evolving Execution Structures for LLM Agents

Researchers introduce Procedural Graphs, self-evolving (procedure, relation, procedure) structures guiding LLM agent tool use and planning.

Procedural Graphs organize procedural knowledge into (procedure, relation, procedure) triplets to guide LLM agent actions, addressing drift such as lost objectives, out-of-order tool calls, and repeated unproductive steps. At each decision step the framework localizes the active node and a guidance model translates the surrounding subgraph into step-level situational guidance. An LLM refiner edits graph topology by contrasting failed with successful trajectories, and across datasets, task types and LLMs the approach outperforms memory-based baselines and matches or surpasses hand-designed graphs.

arXiv cs.AI / cs.LG / cs.CL·Sep 8, 17:59 UTC · 7d agoAI research1

25

Claude Fable Solves a Historical Cipher

Bruce Schneier's blog highlights that the Claude Fable AI model solved a historical cipher, demonstrating LLM capabilities in cryptanalysis.

Bruce Schneier's blog post discusses the Claude Fable AI model successfully deciphering a historical cipher. The post frames the result as a notable example of LLMs applied to classical cryptanalysis. The published text provides limited technical detail beyond the headline.

Schneier on Security·updated Sep 13, 21:06 UTC · 2d ago·first Sep 9, 11:08 UTC · 6d agoAI research 2 sources

20

llm-openrouter 0.7.1

Simon Willison released llm-openrouter 0.7.1, a performance fix for loading OpenRouter models in his LLM CLI plugin.

Version 0.7.1 of the llm-openrouter plugin addresses a performance problem when loading OpenRouter models in the LLM command-line tool. The fix was contributed by GitHub user waveplate. It is a minor maintenance release with no security implications noted.

Simon Willison·Sep 2, 20:23 UTC · 13d agoAI tools & infra

14

The Router Within: Eliciting Native Skill Routing from a Frozen LLM

Gavel reads native skill-routing signals from a frozen LLM's forward passes with two linear maps, beating retrieve-and-rerank pipelines by up to 21.9 points on Qwen3-32B.

Gavel (Glance And Verdict from a frozen LLM) elicits skill routing from a frozen agent LLM using two trained linear maps that read mid-layer states, keeping all skill text out of context. A glance step scores the full library against compact per-skill banks built in one forward pass at installation; a verdict step resumes shortlisted skills' forward passes and fuses likelihood and yes/no judgments as a product of experts. It transfers zero-shot to three public benchmarks plus SkillTraj, a new benchmark of 372 simulated agent trajectories. On Qwen3-32B it beats progressive disclosure and retrieve-and-rerank pipelines adding 1.2B–16B external parameters by up to 13.4 points on written tasks and 21.9 when skills are needed mid-rollout.

Hugging Face daily papers·updated Sep 14, 17:58 UTC · 1d ago·first Sep 13, 20:00 UTC · 2d agoAI research 2 sources

40

Learning to Solve Hard Problems in RL for LLMs by Never Giving Up

Paper introduces Never Give Up adaptive sampling, fixing RL's 'Matthew Effect' where compute is wasted on easy problems and hard problems see little improvement.

Researchers identify a 'Matthew Effect' in reinforcement learning for LLMs, where RL yields large gains on easy problems but minimal improvement on hard ones because compute is misallocated. They propose Never Give Up (NGU), an adaptive sampling method that keeps generating samples for a problem until one is correct, using asynchronous RL to filter easy problems cheaply and concentrate compute on hard ones. NGU improves performance per compute on the Deepscaler math benchmark and iteratively solves the Manufactoria coding task where standard GRPO with per-test reward fails.

Hugging Face daily papers·Sep 10, 20:00 UTC · 5d agoAI research

45

Necessary or Sufficient? Evaluating LLM Explanations With Behavioural Evidence

New framework tests whether LLM-cited explanation factors are necessary or sufficient, finding weak correlation across Claude, GPT, and Gemini models.

An arXiv paper introduces black-box intervention tests measuring whether factors LLMs cite in their explanations are necessary or sufficient for their outputs in agent oversight workflows. Across eight models from the Claude, GPT, and Gemini families, Spearman correlations between cited rankings and measured influence ranged from 0.349-0.354 (advisor recommendation) to 0.431-0.580 (prompt monitoring). Uncited factors scored above the lowest cited factor in up to 57.6% of advisor responses, showing cited top-three factors do not reliably identify the most influential inputs.

arXiv cs.AI / cs.LG / cs.CL·Sep 4, 17:37 UTC · 11d agoAI safety & security1

44

OpenArch – PyTorch implementations of modern LLM architectures

OpenArch provides PyTorch reference implementations of modern LLM architectures for developers and researchers.

OpenArch is a GitHub project offering PyTorch implementations of modern large language model architectures. The repository attracted 43 points and 7 comments on Hacker News. It targets developers and researchers who want readable, runnable versions of current LLM architectures.

Hacker News · AI·Sep 14, 07:55 UTC · 2d agoAI tools & infraHN 43↑ · 7 comments

15

llm-gemini 0.34

llm-gemini 0.34 adds support for Google's new Gemini 3.8 Flash model with configurable low, medium and high thinking levels.

Simon Willison released llm-gemini 0.34, a plugin for the LLM CLI that adds the gemini-3.8-flash model, including low, medium and high thinking levels, and fixes an issue where async responses failed to record the resolved model version. The release coincided with Google's launch of Gemini 3.8 Flash, plus a Gemini 3.8 Flash Cyber variant restricted to trusted defenders. Willison noted the Flash tier's speed, low cost and competence at HTML and JavaScript generation tasks.

Simon Willison·Sep 2, 16:39 UTC · 13d agoAI tools & infra

30

When LLM Decompilers Recompile More and Preserve Less

Researchers show LLM decompiler outputs can recompile yet diverge behaviorally, proposing the Decompile-Diverge fuzzing oracle to catch hidden changes.

The paper demonstrates that LLM-based decompilers can produce code that recompiles and passes all shipped tests yet diverges on other legitimate inputs—4.9% overall and up to 13% for one system—and can make disclosed vulnerabilities vanish without a visible crash. Across 300 real GitHub functions and 287 CVE-grounded functions, a refinement LLM lifted Ghidra's build rate from 75% to 90% while Matched rate fell from 74% to 62%, with up to one tenth of vulnerabilities showing Crash Absence. Decompile-Diverge detects these gaps by synthesizing drivers, growing fuzzing corpora from the reference, and rerunning decompiled code on identical inputs.

arXiv cs.CR·Sep 4, 17:16 UTC · 11d agoResearch

42

Verifiable by Construction: Claim-Level Evaluation of Verbatim Citation in Clinical Question Answering

Evaluation of twelve LLMs on 222 clinical questions shows verbatim quotes rarely substantiate claims; claude-opus-5 fully substantiates only 37.1%.

The authors build a standardized harness over four clinical practice guidelines and evaluate twelve LLMs on 222 synthetic clinical questions, measuring citation attachment, verbatim quote production, and claim substantiation. Most models attach verbatim quotes to over 90% of claims from prompting alone, though lightweight models like claude-haiku-4.5 struggle. Quotes frequently fail to substantiate claims: claude-opus-5 quotes 98.0% of claims but fully substantiates only 37.1%, exposing a capability gap for verifiable clinical QA.

arXiv cs.AI / cs.LG / cs.CL·Sep 14, 17:53 UTC · 1d agoAI research

38