JEPA pretraining underperforms masked models on MIMIC-IV EHRs
Original title: Are Joint-Embedding Predictive Objectives Effective for Longitudinal EHRs?
Three joint-embedding formulations trained without collapse but still lost to standard masked language modelling downstream.
The paper tests whether joint-embedding predictive architectures can replace masked language modelling for longitudinal electronic health records. On MIMIC-IV, all three JEPA variants trained without representational collapse yet consistently underperformed MLM. Larger latent targets raised prediction error, and shrinking them did not restore downstream performance.
- MLM remains the dominant pretraining objective for EHR transformer encoders.
- Target size, not target heterogeneity, increased prediction error.
- Stable latent prediction alone did not yield competitive EHR representations.
From the original article
1 min read at arxiv.orgMasked language modelling (MLM) is the dominant pretraining objective for transformer encoders applied to longitudinal electronic health records (EHRs). However, reconstructing individual events may overemphasise noisy or weakly informative observations and does not explicitly encourage learning higher-level patient states.
By Ashvin Gupta · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.