GoiBERT packs Vietnamese encoders into 26M and 49M parameters
Original title: GoiBERT: Compact Vietnamese Language Models via Two-Stage Knowledge Distillation
GoiBERT-large leads non-teacher averages with under half the parameters of the strongest compact baseline and twice its throughput.
The paper presents GoiBERT, two compact Vietnamese encoders of 26 million and 49 million parameters that run on raw text without external word segmentation. A same-tokenizer Vietnamese encoder teaches general representations, then a larger teacher supplies task soft labels without labeling its own training data. Across eight benchmarks, GoiBERT-large has the highest average among non-teacher models. GoiBERT-base keeps 99% of that baseline's performance at a quarter of its size, while compression remains costly for inference and entity recognition.
- The family has models of 26 million and 49 million parameters.
- Benchmarks cover classification, aspect sentiment, inference, and named entity recognition.
- GoiBERT-base retains 99% of the compact baseline at a quarter of its size.
- Code and models are at github.com/Nhantran65/GoiBERT.git.
From the original article
1 min read at arxiv.orgVietnamese pre-trained encoders have improved rapidly, but the strongest ones are large, multilingual-vocabulary models or depend on external word segmentation, which makes them costly to deploy. We present GoiBERT, a family of two compact Vietnamese encoders with 26M and 49M parameters that operate directly on raw text. GoiBERT learns from two complementary teachers.
By Quang Anh Kiet Tran · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.