Frontier AI agents propose experiments with limited ranking gains
Original title: rTaste: Do Research Agents Have Taste? Evaluating Early Stage Research Judgment
Six frontier models propose MVEs from NLP papers but show minimal improvement in experiment selection over random picks.
Researchers evaluated six frontier models as research agents that propose minimum viable experiments, forecast outcomes and rank them for execution. Of 415 proposals, 194 were judged sound and valuable while only 73 were novel and beyond the seed paper. Agents matched auditors' preferences more often than random but did not outperform the original first choice. The evaluation harness and dataset of all MVEs are released.
- 194 sound and valuable experiments judged
- 73 novel and non-obvious experiments
- forecasts beat base rate but not more accurate than fresh model instances
- teams increase cost without clear quality gains
From the original article
1 min read at huggingface.co
Do research agents have taste, the judgment of which experiments are worth running before their outcomes are known? We give six frontier models seven recent NLP papers. Each model, working as a research agent alone or as a team, proposes minimum viable experiments (MVEs), forecasts whether each will pass a threshold it sets and ranks them.
By Art Kanke · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.