Distillation aligns item and code spaces for generative recommenders
Item Distribution Matching lifts average Recall@10 by 15.39%, and Code Marginal Matching lifts NDCG@10 by 8.28%.
Stories per day · 30 days
3 RSS
Relatedai-security1backdoor1candidate-scoring1class-activation-maps1clip1computer-vision1contrastive-learning1credit-complaints1cross-architecture1dense-retrieval1embeddings1encoders1famteb1generative-recommendation1goibert1imagenet1llm1long-tailed-recognition1
29 stories, newest first · one card per story
Stories tagged #knowledge-distillation
Yalda Embedding released as a 595.8M Persian retrieval model
Adapted from Qwen3-Embedding-0.6B on more than one million pairs, it scores 0.5488 macro-average on eight FaMTEB tasks.
GoiBERT packs Vietnamese encoders into 26M and 49M parameters
GoiBERT-large leads non-teacher averages with under half the parameters of the strongest compact baseline and twice its throughput.
SRCD distills head-class CAMs to resist shortcuts on tail classes
The method aggregates a few head-class activation maps into a teacher and reports consistent gains on long-tailed benchmarks.
Exact risk analysis finds students inherit teacher bias
On 231,521 CFPB complaints, outcome-trained students beat the teacher's decision cost by 8%.
ReDiST builds student-conditioned traces for reasoning distillation
Offline selection ranks traces by answer recovery and verified-answer probability before distilling only the student model.
SimOPD: Relational Knowledge Transfer for On-Policy Distillation
SimOPD distills teacher knowledge through response similarities, gaining up to 3.1 points cross-architecture.
Clean Data Can Still Carry Backdoors: Support-Persistent Backdoors for Model Reuse
DAR plants support-persistent backdoors that stay above 96 percent success after clean-data distillation.
DisaggKD splits teacher and student across nodes for distillation
On the Qwen3.5-35B-A3B to 9B pair, the split matches colocated LM loss and is faster.
Rethinking Relative Policy Differences Transfer of On-Policy Distillation
Relative policy differences, not expert strength alone, determine useful on-policy distillation transfer.
When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better
Semi-OPD, distilling on offline student rollouts, beats on-policy distillation in 14 of 17 teacher-student pairs.
Universal Textual Teaching for LLMs
Universal Textual Teaching distills teacher-student gaps into a reusable natural-language Primer without updating student weights.
Distilling Graph Geometry: Knowledge Gap from GNNs to MLPs
Researchers propose G2MLP, a curvature-guided distillation method that transfers GNN geometry into graph-free MLPs.
Smaller Models, Better Rejects: Preference Distillation Scaling
Smaller frozen models produce more useful preference-distillation rejects than a student's own failures on code and math.
Scaling Properties of Same-Family On-Policy Distillation
On-policy distillation lets larger students surpass compact RL teachers in an early useful-transfer regime.
GTR: Gated Token Recurrence for Efficient Dense Prediction
GTR introduces a softmax-free recurrent vision backbone reaching 58.9 box AP on COCO with sub-2ms latency, targeting high-resolution dense prediction and edge deployment.
LastOPD: Taming Collapse in Latent On-Policy Distillation
LastOPD briefly aligns only the last-layer latent state when distilling Qwen3 into Qwen3-1.7B, beating token-only OPD on MATH-500.
Calibrating Teacher--Student Discrepancy for On-Policy Distillation
Researchers propose Cal-OPD, a calibrated on-policy distillation method that filters teacher self-deviation and improves reasoning model training.
Training Specialist Models without Reasoning Trajectories for Domain Expert Distillation
Study shows specialists trained on question-answer pairs implicitly select latent reasoning trajectories, and tuning choices control the precision-generalization trade-off in distillation.
Label-Guided Knowledge Distillation for 3D-CNNs in Action Recognition
LGKD uses ground-truth labels to guide feature distillation for 3D-CNNs, combining sample-wise and class-wise distillation for action recognition.
US Agencies Warn Chinese AI Firms Are Extracting Advanced AI Models
NSA, CISA, and FBI accuse six Chinese AI firms including DeepSeek and Alibaba of industrial-scale distillation of US frontier models.
Chinese AI firms are siphoning capabilities from American models, CISA warns
CISA, NSA and FBI warn Chinese AI firms including DeepSeek and Moonshot AI extracted billions of tokens from US frontier models via distillation campaigns.
CISA Warns Chinese AI Firms Extract Billions of Tokens From Claude, GPT, Gemini and Grok
CISA, NSA and FBI advisory says six Chinese AI firms extracted billions of tokens from Claude, GPT, Gemini and Grok via API proxies since late 2024.
DeepSeek, Alibaba and Chinese AI Firms Extract Billions of Tokens From U.S. AI Models
NSA, CISA and FBI advisory AA26-251A accuses DeepSeek, Alibaba and four other Chinese AI firms of industrial-scale distillation of US frontier models.