IndicTriMix: Developing Language Identification Datasets and Models for Tri-Language Code-Mixing
Researchers release IndicTriMix benchmarks and fine-tuned MuRIL and XLM-RoBERTa models for token-level language identification in tri-language code-mixed text.
The paper formulates token-level language identification in code-mixed text as a sequence labeling task and fine-tunes MuRIL and XLM-RoBERTa transformer models for Indian languages. It evaluates on Hindi, Gujarati, and Bengali configurations with manually annotated test sets and proposes two code-mixed generation approaches using parallel trilingual sentences. A public benchmark, annotated test sets, and fine-tuned models are released for reproducibility.
- Treats code-mixed language identification as sequence labeling.
- Covers Hindi, Gujarati, and Bengali tri-language mixing.
- Publicly releases benchmark, annotated test sets, and models.
- Products
- IndicTriMix
- Countries
- India
- AI models
- MuRILXLM-RoBERTa
From the original article
1 min read at arxiv.orgLanguage identification in code-mixed text, largely observed in social media, is highly essential when users frequently switch between multiple languages within a single utterance. Accurately identifying the languages of code-mixed tokens becomes an urgent necessity.
By Pruthwik Mishra · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.