BackTrend: Evaluating Scientific Weak-Signal Prediction via Backward Reconstruction
BackTrend finds frontier systems recover few validated early scientific weak signals.
BackTrend is a retrospective benchmark for recovering early scientific weak signals that later become central to mature topics. It includes 25 artificial intelligence and machine learning topics and 66 human-validated signals grounded in 2019-2024 publication-frequency trajectories, covering underrecognized problems and emerging methods. Frontier LLMs, RAG systems, and agentic research systems often drift or miss the reference signals; the strongest F1 is 10.1%, and Coverage10 reaches at most 18.5%. Added retrieval or web-search evidence helps only up to a moderate budget.
- Benchmark covers 25 mature AI and ML topics.
- It includes 66 human-validated weak signals from 2019-2024.
- Best system reaches only 10.1% F1.
- Coverage10 is at most 18.5% of reference signals.
Full article188 words · extracted from arxiv.org · click to collapse
Scientific weak signals are early, low-visibility research directions that later become central to mature scientific topics, yet existing resources such as trend tracking, citation forecasting, and foresight reports rarely provide validated reference sets that link concrete early precursors to later paradigms. We introduce BackTrend, a retrospective benchmark in which, given a mature target topic and a temporal evidence constraint, systems must recover two types of precursors: problem-space signals, underrecognized research problems, and solution-space signals, emerging methods for known problems. BackTrend contains 25 mature target topics in artificial intelligence and machine learning and 66 human-validated weak signals, reconstructed from large-scale literature by grounding each candidate in its 2019-2024 publication-frequency trajectory. We evaluate frontier LLMs, RAG systems, and agentic research systems using semantic matching and coverage-based metrics. Current systems often generate plausible but misaligned precursors, exhibiting topic drift, granularity mismatch, near-miss matching, and incomplete coverage; the strongest system achieves only 10.1% F1, while Coverage10 reaches at most 18.5% of the reference signals. Our budget analyses show that additional retrieval and web-search evidence can improve performance up to a moderate budget, but does not by itself close the substantial performance gap.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.24921