From Retrieval to Typed Decisions: Calibrated System One Models from Biomedical Sentence Encoders
Researchers turn biomedical sentence encoders into calibrated typed-decision models and release MEDLINE-S1.
The paper tests whether biomedical sentence encoders trained for retrieval are good bases for typed decision models that answer schema-constrained questions in one forward pass. SBERT2S1 converts Sentence-Transformers into bi-encoder, cross-head, and prior-fused residual models, evaluated on BIODECIDE and MEDLINE-S1, 243,000 decisions from NLM indexing. After fine-tuning, retrieval pretraining helps the prior-fused head in 10 of 15 comparisons but often hurts the cross-head, which beats it under every objective. The released RLCD recipe trails cross-entropy by 2.5-3.0 points, largely from reward normalisation; code, labels, and a model are released.
- SBERT2S1 converts Sentence-Transformers into cross-head and prior-fused residual decision models.
- MEDLINE-S1 supplies 243,000 training decisions derived from NLM indexing.
- Cross-head outperforms prior-fused residual under every training objective.
- The released RLCD recipe trails cross-entropy by 2.5 to 3.0 points.
Full article200 words · extracted from huggingface.co · click to collapse
Typed decision models answer schema-constrained questions about a text in one forward pass and return probabilities meant to be thresholded. We ask whether biomedical sentence encoders trained for retrieval are good starting points for such models. We present SBERT2S1, which converts Sentence-Transformers encoders into bi-encoder, cross-head (C) and prior-fused residual (PFR) decision models, together with BIODECIDE, a biomedical typed-decision suite, and MEDLINE-S1, 243k training decisions derived from NLM indexing. Across six parent-retriever pairs, retrieval training improves zero-shot matching of content-bearing options. After fine-tuning, its effect depends on the head: across five pairs and three training-set sizes, retrieval training significantly helps PFR, which keeps the retrieval prior, in 10 of 15 comparisons, but helps C in one and hurts it in five. A matched grid of two heads and five training objectives shows that C outperforms PFR under every objective, and that the released RLCD recipe of open System One models trails cross-entropy by 2.5-3.0 points. The deficit stems mainly from its reward normalisation, which inflates the noisy score-function term 3.6-15-fold; an unbiased leave-one-out estimator recovers most of the gap. After temperature scaling, no objective is clearly better calibrated than cross-entropy. We release the code, the MEDLINE-S1 labels and a model.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2610.02486