Search: “speech-language-modeling”

Smart search ranks by meaning as well as keywords (one row per story, last 45 days).

Mind2Dialogue: Training Human-Aware Language Models by Simulating User Mental States

Mind2Dialogue simulates users' mental states to generate privileged supervision, boosting personalization and preference-following in Qwen, Llama, and OLMo assistants.

The Mind2Dialogue framework uses a psychology-guided simulator that preserves personal characteristics while updating user mental states through interaction, driving coherent conversations and an Oracle assistant's responses. Privileged distillation trains models on the Oracle's well-informed responses so they can assist users without direct access to mental states at deployment. Training on the full corpus improves every reported personalization metric over Qwen, Llama, and OLMo instruction-tuned baselines, including 26.6 to 40.9 percentage point gains in preference-following generation.

Hugging Face daily papers·updated Sep 14, 17:55 UTC · 2d ago·first Sep 13, 20:00 UTC · 3d agoAI research 2 sources

45

Building and Evaluating Fixed-Voice Thai TTS from Synthetic Speech

Researchers distill a compact 82M-parameter Thai TTS from synthetic OmniVoice data, enabling on-device fixed-voice synthesis without reference audio.

The paper uses a large voice-cloning model (OmniVoice) as a synthetic data source to train Wayu-Paxa-TTS-Edge, an 82M-parameter fixed-voice Thai TTS student. The model achieves 68.2% Challenge-Set Keyword Accuracy (85.5% of Gemini 3.1), 91.4% pause precision, and CERs of 3.7% on Thai and 1.1% on English. It outperforms its teacher on pause placement and is open-sourced with its evaluation framework.

Hugging Face daily papers·Sep 2, 20:00 UTC · 14d agoAI research

25

Do speech foundation models really learn words?

Researchers show via residualization that later layers of HuBERT and wav2vec 2.0 encode word identity and semantics independently of phonetic content.

The study argues that discriminative ability on words does not imply specialized word representations, since good word discrimination can be explained by phoneme encoding alone. By partialling out phoneme information using residualization, the authors show that later layers of HuBERT and wav2vec 2.0 encode words with reasonable fidelity independently of local phonetic content. Applying this disentanglement approach enhances higher-order linguistic information in word discovery tasks, informing analysis of speech foundation models used for recognition and speech tokens.

arXiv cs.AI / cs.LG / cs.CL·Sep 9, 16:47 UTC · 7d agoAI research

28

Discrete Beckmann Transport Models for One-Step Language Modeling and Reasoning

DBTM achieves one-step text generation via a time-independent transport map trained directly from data, removing pretrained teacher distillation.

Discrete Beckmann Transport Models (DBTM) build a time-independent flow whose autonomous transport map provably carries any point in ambient space to a fixed point on simplex vertices in a single step. The fixed-point property is characterized by a conservation equation whose residual can be minimized directly from data, eliminating the need for a teacher flow, distillation, and time conditioning. A partial-context interpolant extension turns additional function evaluations into refinement steps rather than ODE integration steps. On language modeling and reasoning tasks, DBTM's one- and few-step generation improves quality and accuracy over discrete diffusion and continuous flow baselines.

arXiv cs.AI / cs.LG / cs.CL·Sep 14, 17:19 UTC · 2d agoAI research1

35

Training and Finetuning Multi-Vector Embedding Models with Sentence Transformers

Hugging Face published a tutorial on training and finetuning multi-vector embedding models using the Sentence Transformers library.

Hugging Face's blog walks through training and finetuning multi-vector embedding models with Sentence Transformers. Multi-vector approaches store multiple vectors per document to support late-interaction retrieval. The post is a practical guide for developers building retrieval pipelines with the library.

Hugging Face Blog·Aug 26, 00:00 UTC · 21d agoAI tools & infra1

20

Studying Image Tokenizers as Visual Languages in Unified Multimodal Models

A controlled autoregressive testbed shows validation losses must be analyzed per task, and image tokenizer choice affects joint multimodal text modeling.

Researchers built a pure-autoregressive testbed to study image tokenizers as the 'visual language' of unified multimodal models, tracking task-specific validation losses during multimodal continual pretraining across text, image, text-to-image (T2I), and image-to-text (I2T) prediction. They found that losses exhibit distinct scaling behavior per task and rank tokenizers differently, and that I2T loss over a shared text vocabulary gives a more consistent loss–performance signal than T2I loss. Better reconstruction does not necessarily yield lower task-specific losses or stronger downstream performance, and tokenizer choice can affect text modeling under joint optimization. Case studies examine the discriminator, semantic supervision, and vocabulary size design axes.

Hugging Face daily papers·Sep 7, 20:00 UTC · 9d agoAI research1

40

Nuha-Speech: Building General-Purpose Arabic Speech-LLMs

Nuha-Speech initiative builds general-purpose Arabic speech-LLMs using a 1.5M-sample speech QA corpus and fine-tuned Qwen-Omni variants.

The paper introduces Nuha-Speech, an initiative covering dataset construction, model training, and evaluation for Arabic speech large language models. The authors built an Arabic Speech Question-Answering corpus of over 1.5 million training samples and used it for supervised fine-tuning of Qwen-Omni model variants at multiple scales. A tailored evaluation framework with diverse tasks and metrics is designed to assess Arabic speech capabilities under limited resource constraints.

arXiv cs.AI / cs.LG / cs.CL·Sep 10, 17:50 UTC · 6d agoAI research1

30

Multi-Vector (Late Interaction) Embedding Models with Sentence Transformers

Hugging Face details building and using multi-vector late-interaction embedding models with Sentence Transformers for retrieval workloads.

Hugging Face published a guide on multi-vector, late-interaction embedding models (ColBERT-style) supported through Sentence Transformers. The post covers how practitioners can build and use these models for retrieval and RAG pipelines. It is a developer tooling and technique write-up, not a security advisory.

Hugging Face Blog·Aug 18, 00:00 UTC · 29d agoAI tools & infra1

42

Measuring benchmark optimization in speech recognition

Hugging Face examines how much speech recognition systems overfit benchmarks and how to measure benchmark optimization in ASR.

A Hugging Face post on measuring benchmark optimization in automatic speech recognition, analyzing how model improvements on benchmarks reflect genuine capability gains versus overfitting. It is evaluation methodology research with no direct security impact.

Hugging Face Blog·Aug 21, 00:00 UTC · 26d agoAI research

22

AuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editing

Open-source speech foundation model AuK unifies generation and editing, trained on 1.95 million hours, with distilled AuK-Flash achieving 4.5x speedup.

AuK is an open-source foundational model that unifies speech generation and editing through natural-language instructions and audio context, trained on approximately 3.03 billion instruction-audio instances and 1.95 million hours of supervision across five task families including generation, content editing, and acoustic editing. It combines a multimodal LLM for semantic conditioning, a VAE jointly trained on speech, general audio, and music, and a hybrid rectified-flow Transformer using dual-stream MMDiT blocks followed by unified single-stream DiT blocks. Post-training applies human-feedback preference optimization for editing and reward-based reinforcement learning for generation, and the distilled AuK-Flash performs 4-step inference without classifier-free guidance at a 4.5x wall-clock speedup. Source code and model weights are released.

Hugging Face daily papers·Sep 7, 20:00 UTC · 9d agoModel release2

42

What Did I Just Say? Self-Listening for Full-Duplex Speech Models

Researchers propose Self-Listening, a full-duplex speech approach feeding realized model speech back as input to improve interruption recovery.

Full-duplex spoken language models can listen and speak simultaneously, but asynchronous text generation, speech synthesis, and playback cause mismatches between what a model believes it said and what the user heard. The paper defines the resulting recovery problem as anchor interruption and proposes Self-Listening, which interleaves user speech, model text, and played speech as input streams. The authors also release AnchorSpeech, a benchmark with homogeneous training and test splits tracking which ordered response items were actually spoken. Experiments show self-listening models achieve better anchoring performance than full-duplex baselines.

Hugging Face daily papers·Sep 3, 20:00 UTC · 13d agoAI research

22

Motion-Omni: End-to-End Joint Speech and Full-Body Motion for Spoken Dialogue

Researchers introduce Motion-Omni, an end-to-end model generating speech with synchronized full-body motion, responding 5.4x faster than cascade pipelines.

Motion-Omni is an end-to-end framework in which a spoken dialogue model outputs facial expressions and hand, upper-body, and lower-body motion directly from the hidden states that produce speech, replacing two-stage cascade pipelines. Trained on 422,856 quality-ranked pseudo-labeled pairs (1,402 hours) with a Qwen2.5-7B-Instruct backbone, Motion-Omni-Q7 matches its teacher cascade within 2% on reference-free motion metrics, achieves a 2.62% word error rate, and runs faster than real time (RTF=0.78). The authors also release the SwDA-500 dataset and the first public evaluation protocol for stochastic open-ended full-body spoken dialogue.

Hugging Face daily papers·Aug 27, 20:00 UTC · 20d agoAI research1

45

Kraken: LLM-based Speech-to-Speech Translation via Low-bitrate VQ and Dual-path Source Conditioning

Kraken speech-to-speech translation model builds on Qwen3-8B with low-bitrate vector quantization and source-conditioned vocoding.

Kraken augments a pre-trained Qwen3-8B LLM with speech feature inputs and low-bitrate single-layer vector-quantized tokens trained to reconstruct SSL features. A separate token-to-waveform decoder, Autowave-X, is conditioned on source speech to improve non-linguistic transfer. Training used 150k hours of multilingual and multitask speech data. The model reportedly beats SeamlessM4T-Large v2 and Qwen2.5-Omni in translation quality and speaker/prosody transfer.

arXiv cs.AI / cs.LG / cs.CL·Sep 11, 16:41 UTC · 5d agoAI research1

25

Large Language Models Develop Belief State Geometry In-Context

Probing six open-source LLMs on HMM-generated data shows belief states are linearly decodable (R² 0.83–0.99), suggesting in-context learning approximates Bayesian prediction.

Researchers prompted six open-source LLMs with data from 40 hidden Markov models selected for non-trivial belief structure and probed residual-stream activations for belief states (posteriors over hidden states). Belief states were linearly decodable with peak R² values of 0.83–0.99 across HMM/LLM combinations, spanning early to late layers. Patching and steering the probe-identified subspace preserved downstream prediction quality while control interventions degraded performance substantially, establishing functional relevance. The results provide representation-level evidence that in-context learning approximates optimal Bayesian prediction over a context-inferred generative model.

arXiv cs.AI / cs.LG / cs.CL·Sep 15, 16:17 UTC · 1d agoAI research1

45

StepAudio 3 Realtime Technical Report

StepAudio 3 Realtime debuts an audio-language model with Think-While-Speaking reasoning, delivering full-duplex voice dialogue with top benchmark results.

StepAudio 3 Realtime is an audio-language foundation model built around a continuous listen-converse-think-act loop for real-time spoken interaction. Think-While-Speaking runs private reasoning in parallel with speech, reaching a 73.0 macro average on StepAudioChat in reasoning mode. The model reports 90.6 on MMSU, 98.9 overall on the Artificial Analysis Full-Duplex Bench, and 56.0% macro task success on tau-Voice. An integrated Voice Agent handles asynchronous tool execution without disrupting dialogue flow.

Hugging Face daily papers·Sep 11, 20:00 UTC · 5d agoAI research

45

NCP-ArchPreview Technical Report: Moving towards Latent Space Language Models through Next Concept Prediction

An 8.9B-parameter latent-space language model using next-concept prediction matches OLMo-3-7B pretraining loss with only 51.3% of the training tokens.

NCP-ArchPreview augments next-token prediction with Next Concept Prediction over a product-quantized concept vocabulary built from hidden states, trained jointly end-to-end. The 8.9B model was trained on 5.73T tokens from the Dolma-3 dataset, the largest latent-space language model demonstration to date. It consumes 51.3% of the tokens to reach OLMo-3-7B's final pretraining loss and outperforms it by 2.45 points on the downstream macro-average, including a 5.99-point GSM8K gain. The learned latent space also enables lightweight domain adaptation via a 17M-parameter VQ module and improves speculative drafting accepted length by 4.17%.

Hugging Face daily papers·Sep 8, 20:00 UTC · 8d agoAI research1

52

Augustinian BabyLM: What Ostensive Definition Can and Cannot Teach a Small Language Model

Study shows visually grounded token embeddings in a small masked LM persist through training and improve object-property knowledge, but escape standard BabyLM benchmarks.

The paper implements ostensive definition for a small DeBERTa masked language model trained on 10M words, seeding visually grounded tokens with embeddings derived from labeled image regions before training. Visual initialization leaves a persistent, seed-replicated advantage on object-property knowledge (COMPS) and a corpus-tailored Visual-Property Swap benchmark covering color, material, size, and shape, but has no effect on most BabyLM grammar benchmarks. Synthetic grounding of previously unseeded words causally transfers the advantage to exactly those words.

arXiv cs.AI / cs.LG / cs.CL·Sep 10, 17:43 UTC · 6d agoAI research

20

Studying Image Tokenizers as Visual Languages in Unified Multimodal Models

A controlled pure-autoregressive testbed shows task-specific validation losses rank image tokenizers differently, with I2T loss the most consistent signal.

Researchers built a controlled pure-autoregressive testbed and tracked task-specific validation losses during multimodal continual pretraining across text, image, text-to-image (T2I), and image-to-text (I2T) prediction. They find losses should be analyzed per task because they exhibit distinct scaling behavior and rank tokenizers differently, and that the loss-performance relationship depends on the predicted token space. I2T loss, computed over a shared text vocabulary, correlates consistently with both generation and visual understanding performance after supervised finetuning. Case studies revisit the discriminator, semantic supervision, and vocabulary size as tokenizer design axes.

arXiv cs.AI / cs.LG / cs.CL·Sep 8, 17:57 UTC · 8d agoAI research1

30

IndicTriMix: Developing Language Identification Datasets and Models for Tri-Language Code-Mixing

Researchers release IndicTriMix benchmarks and fine-tuned MuRIL and XLM-RoBERTa models for token-level language identification in tri-language code-mixed text.

The paper formulates token-level language identification in code-mixed text as a sequence labeling task and fine-tunes MuRIL and XLM-RoBERTa transformer models for Indian languages. It evaluates on Hindi, Gujarati, and Bengali configurations with manually annotated test sets and proposes two code-mixed generation approaches using parallel trilingual sentences. A public benchmark, annotated test sets, and fine-tuned models are released for reproducibility.

arXiv cs.AI / cs.LG / cs.CL·Sep 10, 17:36 UTC · 6d agoAI research1

15

Build Low-Latency Multilingual Voice Agents: Open Weights & Full Deployment Control with NVIDIA Magpie TTS

NVIDIA's Magpie TTS open-weight multilingual speech model enables low-latency voice agents with full deployment control.

Hugging Face's blog highlights NVIDIA Magpie TTS, an open-weights multilingual text-to-speech model designed for building low-latency voice agents. The open licensing gives developers full deployment control, allowing self-hosted multilingual speech for agentic applications. The post walks through building voice agents with the model.

Hugging Face Blog·Aug 10, 16:25 UTC · Aug 10, 2026Model release

45

Competence-Gated Pooling of Language Models and Priors for Event Forecasting

Paper proposes a competence gate pooling language model forecasts with external priors, improving Brier score from 0.0771 to 0.0732 across 2,357 binary questions.

The paper defines a language model's relative competence as its marginal value beyond an available external forecast, and derives conditions under Brier loss where model disagreement improves that forecast. A competence gate estimates domain-level source weights from resolved outcomes, shrinks uncertain estimates toward a global weight, and recalibrates the pooled forecast. Across 2,357 resolved binary questions and five language models, it improves the external baseline from 0.0771 to 0.0732 Brier and beats global forecast combinations, though it defers to the market on ForecastBench. Across four Qwen models, verbal confidence failed to identify when the model outperformed the external forecast, while outcome-estimated competence supported better abstention.

Hugging Face daily papers·Sep 9, 20:00 UTC · 7d agoAI research

38

Puppeteer: Object-Grounded Posture-Aware Co-Speech Gesture Generation

Researchers present Puppeteer, a posture-aware, object-grounded diffusion model generating physically consistent co-speech gestures with temporal control.

Puppeteer decomposes long gestures into structured primitives encoded by a causal variational autoencoder into temporally ordered latent tokens. Conditional diffusion in the causal latent space conditions on speech signals, motion history, an initial posture reference, and object geometry to synthesize physically consistent gestures. The authors also introduce new evaluation metrics and release SceneGes, the first curated synthetic 3D dataset of embodied co-speech gestures with corresponding 3D objects.

Hugging Face daily papers·Aug 30, 20:00 UTC · 17d agoAI research1

18

ECHO: A Matched-Contrast Benchmark for Context-Sensitive Turn-Taking in Full-Duplex Dialogue

ECHO benchmark pairs identical-overlap Chinese dialogue examples with contrasting contexts, revealing most full-duplex systems bias toward yielding the floor.

ECHO is a paired diagnostic benchmark for Chinese full-duplex turn-taking that matches examples with identical overlap transcripts but contrasting preceding multi-turn contexts, one requiring Yield and the other Keep. It also includes off-talk examples for diagnosing unnecessary yielding and introduces pair accuracy, which grants no credit to constant-action policies. Experiments across multiple full-duplex systems show most exhibit a pronounced Yield bias, performing substantially better on interruptions than backchannels. The benchmark and metadata will be publicly released.

arXiv cs.AI / cs.LG / cs.CL·Sep 15, 15:58 UTC · 1d agoAI research

40

Technical Manual for a Toolkit for Measuring Contextual Individuation in Transformer Language Models

An open methodology toolkit measures whether transformer language models contextualize fixed word forms across domains using bridge forms and layer-wise silhouette analysis.

The manual documents an open toolkit built around 'bridge forms' - identical written words recurring across two or more subject domains with a different sense in each - to test whether transformer language models individuate word occurrences by context beyond the embedding layer. It covers declarative specification of bridge forms, Wikipedia corpus acquisition, occurrence localization, layer-wise representation extraction, domain-pairwise silhouette measurement, and visualization, justifying each choice against failure modes such as sense contamination and subword-tokenization misalignment. It is a methodological and implementation reference and reports no empirical results.

arXiv cs.AI / cs.LG / cs.CL·Sep 4, 16:38 UTC · 12d agoAI research

14

RetroThinker: Enabling Retrospective Thinking in Speech LLMs

RetroThinker is a post-training framework letting the Moshi speech LLM self-correct reasoning mid-stream, adding 11% GSM8K accuracy at similar latency.

Researchers introduce RetroThinker, a multi-stage post-training framework that equips the Moshi speech LLM to verify and forward-correct chain-of-thought steps during streaming inference. It combines supervised fine-tuning on curated retrospective thinking data with length-based direct preference optimization (DPO). On GSM8K it achieves an 11% absolute accuracy gain over non-retrospective baselines at comparable latency.

arXiv cs.AI / cs.LG / cs.CL·Sep 10, 17:41 UTC · 6d agoAI research1

28

The Semantic Bottleneck: Leveraging Semantic Representations for Non-Invasive Speech Decoding

Researchers introduced Brain2Semantics2Text, decoding sentence meaning from non-invasive MEG brain recordings via a semantic bottleneck, improving on prior Brain2Text methods.

The paper proposes Brain2Semantics2Text, a non-invasive speech decoding method that maps sentence-level magnetoencephalography (MEG) responses into a semantic embedding space and inverts those embeddings into natural language. Motivated by evidence that high-level semantic representations are distributed across cortex and evolve on slower timescales, the approach targets meaning rather than phonemes or words, avoiding the need for word-level alignment. The authors report improved sentence-level results compared to prior non-invasive Brain2Text methods despite the low signal-to-noise ratio of neural recordings.

Hugging Face daily papers·Sep 8, 20:00 UTC · 8d agoAI research2

25

Knowing What Not to Answer: Selective Non-Compliance in Vision-Language Models

Researchers introduce KoNA, a benchmark exposing vision-language models' failures at selective non-compliance, plus fine-tuning that improves refusal and abstention accuracy.

KoNA is a benchmark for evaluating selective non-compliance in vision-language models across five categories: False Premise, Visual Inaccessibility, Universal Unknown, Task Feasibility and Safety. It tests both query-level and component-level non-compliance using paired single and compound queries, and evaluations across diverse VLMs show models often fail to refuse, correct or abstain appropriately, with failures worsening on compound queries. Fine-tuning VLMs on KoNA examples substantially improves non-compliance accuracy while largely maintaining performance on fully answerable tasks.

Hugging Face daily papers·Sep 3, 20:00 UTC · 13d agoAI research1

45

RelateAnything: Real-Time Open-Vocabulary Relation Prediction From Any Inputs

RelateAnything is a 53M-parameter open-vocabulary relation prediction model running at 20 ms/frame, with 2.3-3.5x higher mean recall than comparable open-vocabulary methods.

RelateAnything predicts scored relations between image regions using any predicate vocabulary supplied at inference as text embeddings, with object labels never required as input, so region sources can change without retraining. Training covers 19,103 predicates using positive-unlabeled supervision; the authors release RA-4M (474k images, 4.3M geometrically verified relations over 10,102 free-text predicates) and the OV-SGG-Bench evaluation suite. The 53M-parameter model runs at 20 ms/frame and achieves 2.3-3.5x the mean recall of the strongest comparable open-vocabulary method across cross-dataset and zero-shot benchmarks. Model, corpus, and benchmark are public.

Hugging Face daily papers·Sep 10, 20:00 UTC · 6d agoAI research1

40

PhysBrain 1.5: From Vision-Language Models to Physical Foundation Models

PhysBrain 1.5, an 8B physical foundation model, sets open-source state of the art across 28 embodied understanding benchmarks.

The paper presents PhysBrain 1.5, a unified 8B model for understanding physical environments, generating actions, and predicting future states, built from a vision-language model with joint autoregressive next-token prediction over language, end-effector motion, and dense visual targets. Pre-training uses embodied supervision from human interaction videos, followed by supervised fine-tuning on human demonstrations, robot trajectories, and simulated experience. The model averages 72.5 across 28 embodied benchmarks, setting a new open-source state of the art and performing on par with proprietary GPT-6-Astra and Gemini 3.6 Flash, with best open-source results on 14 benchmarks.

Hugging Face daily papers·Sep 13, 20:00 UTC · 3d agoAI research1

45