IndicTriMix: Developing Language Identification Datasets and Models for Tri-Language Code-Mixing
Researchers release IndicTriMix benchmarks and fine-tuned MuRIL and XLM-RoBERTa models for token-level language identification in tri-language code-mixed text.
The paper formulates token-level language identification in code-mixed text as a sequence labeling task and fine-tunes MuRIL and XLM-RoBERTa transformer models for Indian languages. It evaluates on Hindi, Gujarati, and Bengali configurations with manually annotated test sets and proposes two code-mixed generation approaches using parallel trilingual sentences. A public benchmark, annotated test sets, and fine-tuned models are released for reproducibility.
15