Climate-ModernBERT: Revisiting Corpus Composition for Domain-Adaptive Continued Pretraining
Climate-ModernBERT domain-adapted encoders reach 76.3 average F1 across nine climate benchmarks, 2.8 points above vanilla ModernBERT-Base.
The authors continue pretraining ModernBERT-Base on three climate corpora - academic text, climate-filtered web data, and synthetic documents - and compare joint mixtures against parameter-space merging of specialized checkpoints. The best model achieves 76.3 average F1 across nine climate NLP benchmarks, a 2.8-point improvement over the vanilla baseline. Academic climate corpora provide the strongest adaptation signal, and parameter-space merging outperforms joint multi-source training while preserving complementary corpus information; all variants are released.
- Continued pretraining of ModernBERT-Base on three climate corpora
- 76.3 average F1 across nine climate NLP benchmarks, +2.8 over baseline
- Parameter merging beats joint multi-source training for heterogeneous corpora
- Academic climate text is the strongest adaptation signal among sources
- Products
- Climate-ModernBERT
- AI models
- ModernBERT-Base
From the original article
1 min read at arxiv.orgNatural Language Processing (NLP) in the climate domain requires models to process heterogeneous text sources, including scientific literature, policy disclosures, and synthetic reports. However, how to effectively combine diverse domain corpora during continued pretraining (CPT) remains underexplored.
By Yongan Yu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.