MOAI speeds non-interactive homomorphic transformer inference
It cuts BERT-base evaluation time 52.8% versus THOR and reports 2.36 minutes amortized per input on one GPU.
Stories per day · 30 days
3 RSS
Relatedattention3language-models3privacy3sparsity3dinov32elastic-inference2generative-models2huggingface2llm2pretraining2pruning2scaling-laws23d-gaussian-splatting1arxiv1autonomous-driving1bert1bgl1bptt1
74 stories, newest first · one card per story
Stories tagged #transformers
LogCDM detects log anomalies on CPU with an 8M-parameter model
It reports F1 of 0.984 on HDFS and scores a session in 5.5-10.5 ms without a GPU.
Spectral unlearning applies matrix-structured updates to transformers
Applied atop existing objectives, it narrows most of the gap to a retain-trained reference on memorization, privacy, and utility.
FWL-ToPM Enables Real-Time Full-Waveform LiDAR Transformers
FWL-ToPM provides over 8x speedup and nearly 9-point F1 gain on Ghost-FWL benchmark.
GaussianRenderFormer renders 3D Gaussian splats without a rasterizer
On 300 unseen objects it scores 38.08 dB PSNR, versus 44.02 dB when rasterizing the same pruned splats.
LoopVQ sub-1-bit KV cache quantization for looped transformers
LoopVQ jointly quantizes channels across multiple loop steps in looped transformers to reduce memory while preserving accuracy.
Zyphra introduces Hierarchical Attention for subquadratic attention
The available text names ZHA but gives no architecture, benchmarks, or release details.
Transformers Model Integration Contracts
Hugging Face Blog announces Transformers model integration contracts for AI tooling.
Fast Relative-Position Attention Algorithm Speeds Up Transformers
Exact tiled computation incorporates additive position terms into FlashAttention without redundant indexing or memory overhead.
Hierarchical Residuals Decouple Residual and Compute Width
Block-wise grouping with wider residuals and learned projections improves FLOPs efficiency while preserving standard layer structure.
TACFormer generates full PROTACs conditioned on target protein sequences
A GPT-style decoder uses ESM-2 embeddings of the protein of interest instead of designing only the linker.
SpecSlot cuts global attention cost in pixel-space transformers
Only the lowest-frequency slot per spatial anchor attends globally; higher-frequency detail stays local.
RAG-CSI retrieves channel history to improve CSI prediction
On a multi-scene dataset, the method cut normalized mean squared error 15.42% and improved all five metrics by more than 10%.
Pairwise graph gains track length and local signs, not topology
After controlling for sequence length, the reported edge over model confidence disappears; topology shifts ROC-AUC by at most 0.004.
Ariadne transfers fine-tuning effects across differently shaped Transformers
The training-free method synthesizes a native target update from activation changes instead of moving parameters.
Nestor nested sparsity beats MoE transformers at equal compute
Attention-side sparsity nests with mixture-of-experts sparsity and adapts to new domains with less forgetting.
Paper derives attention architectures from geometric knowledge spaces
The framework recovers standard, causal, linear, and tensor attention plus recurrent networks, then proposes new geometric spaces.
Scaling law favors full BPTT over truncated looped transformers
Two fully trained loops beat the compute-optimal non-looped model with about 27% fewer non-embedding parameters.
HRoPE encodes patch surfaces as slanted planes for multi-view attention
Plane parameters need no ground-truth geometry and can be learned from view synthesis alone or taken from foundation models.
Transformers forget prompts as generated context grows
Hidden-state distance between two prompts decays as a power law; trained models forget faster than untrained ones.
Designing Transformers for Continuous Diffusion Language Models
LangDiT redesigns Transformers for continuous diffusion LMs, cutting OpenWebText generation perplexity to 11.33–21.72.
Your Transformer Is Secretly Elastic
A retraining-free method turns transformers into elastic models that prune smoothly up to 60% sparsity.
Continuum Attention
Continuum Attention beats softmax and focal attention by integrating a continuous memory curve.
Show HN: Building an LLM from Scratch in TypeScript
A TypeScript guide shows how to build a GPT-style language model from scratch in Node.js.