IntReal dataset and suite benchmark interleaved text-image generation
Original title: IntReal: A Real-World Dataset and Benchmark for Multimodal Interleaved Generation
IntReal-100K and a seven-dimension suite, with 83.3% human agreement, show unimodal training can also help interleaved generation.
Researchers introduce IntReal to address missing tasks, training data, and fine-grained evaluation for multimodal interleaved generation. IntReal-100K contains real-world interleaved samples in four categories, and the evaluation suite scores models on seven dimensions with expert models. The authors report low metric redundancy, 83.3% agreement with human judgments, and Spearman correlation above 0.9 across evaluators. Benchmarks of four unified multimodal model families, plus post-training, find distinct strengths and gains from modality-specific optimization.
Unconfirmed: Benchmark scores and human-agreement figures are reported by the paper's authors.
- Maximum pairwise absolute Spearman correlation between metrics is at most 0.42.
- Human agreement is reported at 83.3% accuracy.
- Evaluator consistency exceeds a Spearman correlation of 0.9.
- Post-training shows unimodal generation training can improve interleaved output.
From the original article
1 min read at huggingface.co
Humans routinely consume multimodal content, where textual and visual information is naturally interleaved. Although Unified Multimodal Models (UMMs) can jointly generate text and images, multimodal interleaved generation remains largely underexplored.
By Shaohan Yu, Jie Qin, Wende Tan · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.