Search: “inference-infrastructure”

Smart search ranks by meaning as well as keywords (one row per story, last 45 days).

An Evidence-First Multi-LLM Framework for Auditable Critical-Infrastructure Dependency Modeling

Evidence-first multi-LLM framework builds auditable critical-infrastructure dependency graphs while preserving provenance and unresolved cases.

The framework constructs Infrastructure Knowledge Bases and Infrastructure Dependency Graphs from heterogeneous infrastructure documentation using multiple open-weight LLMs that independently extract candidate entities and dependencies from normalized evidence. It separates evidence verification, ontology grounding, entity resolution, dependency alignment, validation, fusion, and human review, projecting the validated IKB deterministically into the IDG without new LLM-generated knowledge. Evaluation across nine infrastructure projects shows entity recovery achieves substantially higher recall than full dependency recovery, and cross-model overlap is much lower for dependencies than entities, indicating models often produce non-overlapping candidate assertions rather than stable consensus.

arXiv cs.CR·Sep 11, 02:33 UTC · 5d agoResearch

28

Architecting memory and storage in the AI era

Analysis argues AI inference shifts data-center bottlenecks to memory and storage, urging balanced compute, memory, storage, and network architecture over raw compute.

MIT Technology Review, citing Tirias Research principal analyst Jim McGregor, argues that AI inference and agentic workloads make data movement the key constraint, elevating memory and storage from background hardware to strategic assets. The piece says RAG and real-time inference require continuous data retrieval and caching that legacy infrastructure cannot support. It frames infrastructure planning as a business decision balancing performance, efficiency, cost, and scalability in healthcare, finance, and customer-facing AI.

MIT Technology Review · AI·Sep 4, 18:39 UTC · 11d agoAI industry

14

Beyond Training: A Feasibility Taxonomy for Inference-Time AI Governance

A study maps twenty inference-time AI governance mechanisms, finding commercial readiness only against cooperative deployers and no adequate defense versus state-level adversaries.

The paper develops a feasibility taxonomy of twenty inference-time AI governance mechanisms across monitoring, verification, and enforcement, each rated on a four-point readiness scale against a four-vendor evidence base. Fifteen of the twenty mechanisms have commercial technical substrates in production today, though governance-grade assurance and adversarial robustness vary substantially. Stress testing shows readiness holds only against a cooperative deployer and low-to-medium-capability user: no mechanism rates adequate against a high-capability state-level deployer, and fine-tuning removes model-internal enforcement components. A second-rater reliability check on readiness ratings returned a quadratic-weighted Cohen's kappa of 0.74.

arXiv cs.CR·Sep 9, 12:37 UTC · 7d agoAI policy

28

Φ-Bench: Can Large Language Models Engineer the Infrastructure That Powers Them?

Researchers release Phi-Bench, a benchmark evaluating frontier LLMs on open-ended, long-horizon engineering and optimization of the LLM infrastructure stack.

Phi-Bench evaluates LLMs on open-ended engineering of the LLM infrastructure stack, derived from optimization problems studied in frontier research and grounded in real-world code repositories. Tasks range from localized kernel-level function completion to long-horizon implementation and end-to-end system optimization. Experiments on frontier LLMs reveal current capabilities and limitations on the path toward autonomous optimization of future AI infrastructure.

Hugging Face daily papers·Sep 8, 20:00 UTC · 7d agoAI research1

28

How Hugging Face Inference Endpoints, Jobs, and Buckets Power Search on Papers with Code

Hugging Face explains how Inference Endpoints, Jobs, and Buckets power semantic search on Papers with Code.

Hugging Face describes the infrastructure behind search on Papers with Code, built on its Inference Endpoints, Jobs, and Buckets services. The post is a product-focused engineering walkthrough with no security impact.

Hugging Face Blog·Aug 21, 00:00 UTC · 26d agoAI tools & infra

20

BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference

BeaconKV introduces training-free KV cache compression using beacon queries, cutting long-reasoning inference memory up to 5.8x while preserving accuracy.

The paper shows recency-based KV cache compression assumptions fail in long-horizon reasoning because Thought Revisiting Tokens (TRT) re-attend to distant context such as early task-solving plans. TRT queries cluster into a small number of similarity groups, which BeaconKV exploits by maintaining compact beacon query representatives to anticipate revisited KV pairs without storing full query history. The training-free method achieves up to 5.8x memory reduction and over 4.3x throughput improvement across four open-source large reasoning models while nearly preserving full cache accuracy.

Hugging Face daily papers·Sep 3, 20:00 UTC · 12d agoAI research1

33

The Self-Expanding Stolen Inference Supply Chain: An AI Agent Harvesting and Re-Serving LLM Access, (Fri, Sep 11th)

An autonomous coding agent harvested LLM API access from poorly secured gateways and aggregated stolen inference capacity behind a self-hosted gateway

A SANS researcher observed a semi-autonomous coding agent finding weakly secured LLM resale gateways via FOFA queries, creating trial accounts with temporary emails and CAPTCHA solving, and exploiting weak authorization such as client-supplied group_id fields. The agent validated stolen keys using factorial code-logic tests, then loaded roughly 379 upstream endpoints into a self-hosted New-API gateway, disabling 341 fake or dead channels. Five model names including claude-opus-5 and gpt-5.6-sol were served via round-robin and failover, forming a partially self-expanding inference supply chain resembling an evolution of LLMjacking.

SANS Internet Storm Center·Sep 11, 14:40 UTC · 4d agoThreat actor in the wild

68

ConvMem: Convolutional Memory for Long-Context Reasoning

Researchers propose ConvMem, a training-free framework treating LLMs as convolutional kernels for parallelizable long-context reasoning beyond fixed context windows.

ConvMem reformulates long-context reasoning as a hierarchical convolution in which the LLM summarizes text segments hierarchically, shortening the reasoning path from a linear chain to a logarithmic tree. It uses configurable strides, skip connections, and multi-kernel convolution to capture evidence, decompose queries, and enable massive parallelization across segments and reasoning threads. On RULER-HotpotQA and RULER-2WikiMultiHopQA it outperforms training-free baselines and avoids the out-of-distribution overfitting seen in RL-trained approaches like MemAgent.

arXiv cs.AI / cs.LG / cs.CL·Sep 9, 16:53 UTC · 6d agoAI research

38

Implementation of Machine Learning Workflows with NVIDIA cuML, RAPIDS, GPU Benchmarking, Explainability, Clustering, and Model Inference

Hands-on tutorial implements NVIDIA cuML and RAPIDS to GPU-accelerate scikit-learn-style ML workflows with benchmarking, clustering, and inference.

The tutorial demonstrates NVIDIA cuML as a GPU-accelerated machine learning framework, using cuml.accel to speed up unmodified scikit-learn scripts with zero code changes and the native cuML API for CuPy/cuDF interoperability. It benchmarks CPU versus GPU implementations of PCA, K-Means, nearest-neighbor search, logistic regression, random forests, and DBSCAN on datasets up to 200,000 samples with 64 features. It also builds GPU pipelines with UMAP, t-SNE, and HDBSCAN, validates GPU-generated SHAP explanations, uses the FIL library for forest inference, and covers model serialization and GPU/CPU portability.

MarkTechPost·Sep 13, 01:42 UTC · 3d agoAI tools & infra

14

RetroThinker: Enabling Retrospective Thinking in Speech LLMs

RetroThinker is a post-training framework letting the Moshi speech LLM self-correct reasoning mid-stream, adding 11% GSM8K accuracy at similar latency.

Researchers introduce RetroThinker, a multi-stage post-training framework that equips the Moshi speech LLM to verify and forward-correct chain-of-thought steps during streaming inference. It combines supervised fine-tuning on curated retrospective thinking data with length-based direct preference optimization (DPO). On GSM8K it achieves an 11% absolute accuracy gain over non-retrospective baselines at comparable latency.

arXiv cs.AI / cs.LG / cs.CL·Sep 10, 17:41 UTC · 5d agoAI research1

28

A*-Thought-V2: Efficient Latent Reasoning via Geometric Dynamics of LLM

A*-Thought-V2 compresses redundant chain-of-thought steps into latent tokens guided by hidden-state geometry, improving accuracy up to 2.6% while halving response length.

A*-Thought-V2 models chain-of-thought as a hidden-state trajectory projected into a 3D PCA space and compresses steps whose transitions deviate from the question-to-solution direction into continuous latent tokens, keeping aligned steps explicit. Training uses stepwise embedding forcing and label forcing with soft multi-modal vocabulary supervision. On Qwen3.5-9B and Qwen3.6-27B across six in-domain and out-of-domain benchmarks it improves average accuracy by up to 2.6%, cuts response length by up to half, and raises Accuracy per Computation Unit 2.29x while reducing preprocessing and training time by 94.6% and up to 80.3%.

arXiv cs.AI / cs.LG / cs.CL·Sep 7, 17:56 UTC · 8d agoAI research

36

Large Language Models Develop Belief State Geometry In-Context

Probing six open-source LLMs on HMM-generated data shows belief states are linearly decodable (R² 0.83–0.99), suggesting in-context learning approximates Bayesian prediction.

Researchers prompted six open-source LLMs with data from 40 hidden Markov models selected for non-trivial belief structure and probed residual-stream activations for belief states (posteriors over hidden states). Belief states were linearly decodable with peak R² values of 0.83–0.99 across HMM/LLM combinations, spanning early to late layers. Patching and steering the probe-identified subspace preserved downstream prediction quality while control interventions degraded performance substantially, establishing functional relevance. The results provide representation-level evidence that in-context learning approximates optimal Bayesian prediction over a context-inferred generative model.

arXiv cs.AI / cs.LG / cs.CL·Sep 15, 16:17 UTC · 22h agoAI research

45

PARSER: Read in Parallel, Reason in Depth for Long-Context LLM Agents

PARSER uses parallel reader subagents and an RL-trained lead agent for long-context QA, beating baselines and cutting latency up to 11x.

The PARSER paper decouples reading from reasoning: frozen subagents each read one document chunk in parallel while an RL-optimized lead agent iteratively broadcasts queries and aggregates evidence in scatter-gather rounds. On multi-hop QA with 7K to 896K token contexts, a 4B-backbone PARSER beats the strongest sequential memory baseline by 5.7 points on average and 12.0 points at 896K tokens, and a 9B version surpasses DeepSeek-V4-Pro by 6.3 points. Controlled experiments show robustness to evidence position, order, and distance perturbations, with inference latency reduced by up to 11x.

Hugging Face daily papers·Sep 5, 20:00 UTC · 10d agoAI research1

52

A*-Thought-V2: Efficient Latent Reasoning via Geometric Dynamics of LLM

A*-Thought-V2 compresses chain-of-thought into latent tokens using geometric hidden-state dynamics, cutting computation while improving accuracy on Qwen models.

A*-Thought-V2 models chain-of-thought as a hidden-state trajectory and interleaves explicit text with continuous latent tokens, compressing steps whose transitions deviate from the question-to-solution direction. Trained via stepwise embedding forcing and label forcing with soft multi-modal supervision, it was evaluated on Qwen3.5-9B and Qwen3.6-27B across six benchmarks. Reported results include up to 2.6% average accuracy gain, up to 50% shorter responses, 2.29x higher Accuracy per Computation Unit, 94.6% faster preprocessing, and up to 80.3% faster training.

Hugging Face daily papers·Sep 7, 13:56 UTC · 9d agoAI research

48

Signed Rescue Routing: Harm-Aware Cascades for Efficient LLM Inference

Signed Rescue Routing improves LLM cascade efficiency by predicting when a larger model actually corrects a smaller one rather than uncertainty.

Signed Rescue Routing (SRR) is a budgeted cascade method that separately predicts rescues and regressions when escalating from a small to a large model, ranking requests by their signed difference. The authors prove this signed conditional gain is Bayes-optimal under a fixed escalation budget and add only a lightweight two-head router needing small-model output statistics at deployment. Evaluation with Qwen3-4B and Qwen3-8B on MMLU, HellaSwag, and ARC-Challenge shows better accuracy-compute tradeoffs than entropy routing and learned error predictors.

arXiv cs.AI / cs.LG / cs.CL·Sep 7, 17:31 UTC · 8d agoAI research

22

Unfold The World: Factorize 4D Properties in Reinforcing Spatial Reasoning

FactoSR factorizes 4D spatial reasoning into XY, Z, and T reinforcement-learning sub-objectives, boosting VLM performance on VSI-Bench by 5.9% and All-Angles-Bench by 4.5%.

Researchers present FactoSR, a factorized reinforcement learning framework that decomposes world-consistent reasoning into planar correspondence, depth consistency, and temporal reversibility sub-objectives. Optimizing these verifiable constraints turns the ill-posed projection recovery problem into tangible reasoning steps. Evaluations show gains of 5.9% on VSI-Bench and 4.5% on All-Angles-Bench for 3D and 4D reasoning, arguing VLMs' spatial bottleneck stems from training on 2D projections versus latent 3D geometry and temporal continuity.

Hugging Face daily papers·Sep 2, 20:00 UTC · 13d agoAI research

26

HoneyRoute: Honeypot-Model Routing for Adversarial LLM Serving

HoneyRoute detects malicious LLM serving requests and diverts them to a honeypot model, reaching F1 0.911 with 38 ms median added latency.

HoneyRoute is an inference-serving layer pairing a streaming router (a frozen 0.8B embedding backbone with per-domain MLP heads) with a dual-implementation honeypot and an analysis loop that converts trapped interactions into attacker fingerprints for router retraining. On a production trace plus a seven-domain attack corpus it matches 96% of a two-tier guard-LLM cascade's F1 at 1/385th of its latency with 0% evasion under 13 adversarial transformations. Diverting malicious traffic cuts production token consumption under GCG-suffix flooding by 97.8%, and loop training raises detection F1 to 0.933.

arXiv cs.CR·Sep 8, 06:26 UTC · 8d agoAI safety & security

38

GraphProfiler: Source-Linked Sensitive Attribute Inference via Personal Knowledge Graphs

GraphProfiler links LLM attribute inferences to source posts via personal knowledge graphs, enabling targeted redaction of privacy-leaking content.

GraphProfiler represents a user's post history as a source-linked personal knowledge graph where nodes and edges trace back to originating posts, making LLM-based attribute inference auditable. It reaches 86.7% attack success rate on the eight-attribute SynthPAI benchmark and 84.6% on PANDORA, within two points of strong text-only baselines, while citing supporting evidence for over 98% of predictions. Ablation experiments show removing cited posts reduces attack success substantially more than removing random posts, supporting targeted privacy mitigation.

arXiv cs.CR·Sep 11, 05:13 UTC · 5d agoResearch1

45

ReCite: Agentic Reasoning for Faithful Citation

ReCite is an agentic citation framework using claim-level reasoning and verification, outperforming large generative models in strict citation accuracy.

ReCite is a decoupled agentic framework that orchestrates location perception, intent-aware query planning, and reflective verification for citation recommendation. Trained on synthesized reasoning trajectories, the agent verifies claim-evidence consistency and triggers self-correction loops when retrieved candidates lack logical support. Experiments show the lightweight framework outperforms state-of-the-art massive generative models in strict citation accuracy, addressing misattribution where cited papers are real but logically unsupportive.

arXiv cs.AI / cs.LG / cs.CL·Sep 8, 17:59 UTC · 7d agoAI research

25

Lightning Weave: Improving the Accuracy-Efficiency Frontier of Reasoning Models through Capability Composition

Lightning Weave composes capabilities from independently post-trained models via on-policy distillation, improving Qwen3.5-4B reasoning accuracy while cutting tokens.

Lightning Weave is a post-training framework that merges accuracy and efficiency capabilities from independently post-trained specialist models into a single student via on-policy distillation. Each capability is represented as a policy shift, combined via aligned log-ratio shifts and Tilted-Target DOPD, enabling training without serving multiple live anchor models concurrently. On Qwen3.5-4B, it raises HMMT 2025 accuracy from 59.2% to 64.0% with 10.7% fewer response tokens, and LiveCodeBench v5 accuracy from 41.7% to 54.2% with 9.6% fewer tokens. The authors report a state-of-the-art accuracy-efficiency Pareto frontier across diverse students and math/code benchmarks, with code planned for release.

Hugging Face daily papers·Sep 12, 20:00 UTC · 3d agoAI research

52

Foundation Models for Generalizable Semantic and Goal-Oriented Communication

FMSGOC uses vision-language foundation model priors plus diffusion reconstruction to enable generalizable semantic communication at 0.039 bits per pixel for 6G.

FMSGOC targets generalization failures in semantic and goal-oriented communication for 6G by leveraging broad visual-linguistic foundation model priors. A vision-language model selects sparse, goal-aligned semantic anchors while a fine-tuned diffusion model performs masked completion to reconstruct images at the receiver, decoupling what to send from how to reconstruct. On CIFAR-10 it reaches 0.039 bits per pixel with cosine similarity 0.87-0.90 and 0.83-0.86 on unseen ImageNet inputs, outperforming end-to-end baselines at lower bit rates.

arXiv cs.AI / cs.LG / cs.CL·Sep 7, 18:12 UTC · 8d agoAI research

22

Introducing @huggingface/kernels: 200+ WebGPU Kernels for Local AI

Hugging Face released @huggingface/kernels, a library offering 200+ WebGPU compute kernels to accelerate AI inference locally in browsers.

Hugging Face introduced the @huggingface/kernels package, bundling more than 200 optimized WebGPU compute kernels for running AI workloads locally. The release targets browser-based and on-device inference, reducing reliance on server-side compute. No article body was available beyond the title, so benchmark results and supported models are not specified.

Hugging Face Blog·Sep 1, 00:00 UTC · 15d agoAI tools & infra

32

Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMs

A survey catalogs inference-efficiency techniques for video and audiovisual LLMs, mapping bottlenecks in sampling, encoding, token reduction, and LLM decoding.

This survey covers inference-efficiency mechanisms for visual and audiovisual video LLMs, reporting reductions in parameters, FLOPs, latency, memory, and token counts. It organizes methods by pipeline stage, covering frame sampling, modality encoding, connector-level token reduction, and LLM prefilling and decoding for systems built since late 2022. The authors compile accuracy-cost comparisons under shared host models and input protocols, identify gaps in audiovisual efficiency and standardized evaluation, and maintain a public repository.

Hugging Face daily papers·Sep 8, 20:00 UTC · 7d agoAI research

40

Likelihood-free inference with nuisance parameters through normalizing flows

Researchers decompose normalizing flows to derive near-pivotal statistics for likelihood-free inference with nuisance parameters, recovering the t-test and beating Welch limits.

A new paper decomposes neural-network normalizing flows to uncover pivotal statistics in the presence of nuisance parameters using only a sample generator from the distribution of interest. The statistic is near-pivotal in the sense of minimum average KL-divergence of its p-values and can incorporate prior knowledge of group invariances such as translation and scale. Experiments show it recovers the one-sample t-test almost exactly, outperforms the Welch test on worst-case size over a constrained variance-ratio range, and delivers higher power and much faster runtime than profile likelihood-ratio techniques on small-to-moderate samples.

arXiv cs.AI / cs.LG / cs.CL·Sep 9, 17:58 UTC · 6d agoAI research

16

A Graph-Based Approach for Mapping Kernel-Level Telemetry to MITRE ATT&CK

Trace2ATT&CK maps eBPF kernel telemetry to MITRE ATT&CK via provenance graphs and RAG with local open-weights LLMs, validated on 347 Atomic Red Team tests.

Trace2ATT&CK collects kernel-level events via eBPF, correlates attacker commands into a provenance graph, and derives compact graph representations suitable for LLM-based reasoning, mapping behavior to MITRE ATT&CK techniques with ranked candidates and rationales. Mapping uses both pure LLM prompting and retrieval-augmented generation grounded in the ATT&CK knowledge base. It was evaluated on 347 Linux Atomic Red Team tests using locally deployed open-weights LLMs. RAG consistently improved ATT&CK mapping over pure prompting, and provenance graphs substantially outperformed raw telemetry, without compromising data confidentiality.

arXiv cs.CR·Sep 11, 13:35 UTC · 5d agoResearch

38

An Evidence Model for Agentic Processes: Evidence Claims, Trust Assumptions, and Policy Assessment

Researchers propose an evidence claim model defining which trust and audit claims agentic AI systems can support, mapping claims to mechanisms, assumptions, and threats.

The paper proposes an evidence claim model for agentic AI processes that exchange messages, invoke tools, request approvals, and modify shared artifacts. It distinguishes claim types such as artifact integrity, provenance, approval evidence, and policy assessment, mapping each to mechanisms, assumptions, limitations, and threats. It stresses that hashes, signatures, and external anchors do not establish semantic truth, authorization, or capture completeness. The contribution is conceptual, offering vocabulary for what an agentic black box can and cannot evidence and which controls must surround it.

arXiv cs.CR·Sep 8, 09:25 UTC · 8d agoAI safety & security

26

[AINews] Fal’s H3 Max Live breaks the infinite videogen barrier

Fal post-trained MiniMax H3 into a 'Max' variant with 35x-faster inference, enabling faster-than-realtime AI video generation and infinite streams.

Fal post-trained MiniMax's H3 model into a 'Max' variant and optimized it for its in-house inference engine, achieving roughly 35x the speed of the official endpoint. The optimization enables faster-than-realtime video generation, demonstrated by an infinite interactive AI-generated stream productized by levels.io. The roundup also notes Meta Muse Code's general availability with an SDK, open DeepSeek-V4-Flash-Vision-Exp weights, GLM-5.3-Flash's strong agentic cost/performance rankings, and Tencent's 770B-parameter Hy4 Preview MoE with 49B active parameters.

Latent Space·Sep 1, 04:36 UTC · 15d agoAI industry

50

Atlas: Efficient Verifiable Semantic Search

Atlas delivers zero-knowledge proofs for HNSW semantic search, verifying RAG retrieval in under a second on SIFT1M and 2.0 seconds at 100M vectors.

Atlas lets a search provider prove that a query was answered correctly against a committed HNSW index without revealing the index, addressing provider deviations like truncation or bias. It combines offline preprocessing, a fixed-size-state restructuring of HNSW with a correctness proof, and timestep-tagged batching of per-step arguments. The system proves queries in under a second on SIFT1M and 2.0 seconds at 100 million vectors while preserving plaintext HNSW recall, and proven retrieval maintains end-to-end RAG answer quality at lower cost than prior verifiable retrieval systems.

arXiv cs.CR·Sep 10, 17:30 UTC · 5d agoResearch1

40

Distill Globally, Adapt Locally: Reasoning Distillation and Product-Type Test-Time Training for Scalable Trade-Up Recommendation

A distillation framework compresses LLM reasoning into a 15.5M-parameter trade-up recommendation model reaching AUC 0.941 with product-type test-time training.

The paper targets trade-up recommendation, which identifies higher-quality alternatives that preserve customer purchase intent. A retrieval-augmented few-shot LLM teacher generates labels and rationales that supervise a compact embedding-pair classifier; at inference the 15.5M-parameter student uses only two precomputed 768-dimensional embeddings with no LLM calls. On 8,352 annotated pairs, label-only training scored AUC 0.912, reasoning distillation reached 0.924, and product-type test-time training lifted it to 0.941 with average precision 0.940. The distilled student is roughly 5,000x faster and 10,000x cheaper than direct LLM inference on a 100K-pair proxy catalog.

arXiv cs.AI / cs.LG / cs.CL·Sep 4, 17:08 UTC · 11d agoAI research

25

MetroLLM-Bench: Evaluating Language Models as Transit Kiosk Runtimes

MetroLLM-Bench is a 955-case benchmark testing language models as transit kiosk tool-calling runtimes across six real metro systems.

The benchmark covers 37-414-station metro systems and eleven task categories including routing, fare calculation, disruptions, accessibility, and adversarial input, with 14 deterministic and 8 semantic scoring components. Of 26 models from six vendors, a PEFT-tuned 4B Qwen 3.5 student scored 91.3 on Tier 1, exceeding GPT-5.6 (90.6/90.0), while Muse Glimmer 30B led the composite ranking. A deterministic rule-based baseline reached 84.6, and PEFT gains over base models shrank from +7.03 points at 2B to -0.91 at 27B.

Hugging Face daily papers·Sep 8, 20:00 UTC · 7d agoAI research

30