arxiv:2608.13417

Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development

Published on Aug 13

· Submitted by

Wanli Yang on Aug 17

#3 Paper of the day

LongCat

Upvote

Authors:

Abstract

Frontier autonomous agents excel at engineering optimization but show unstable performance, limited novelty, and variable experience reuse across long-horizon tasks.

Generated by thinkingmachines/Inkling-Small

Autonomous agents are increasingly capable of improving models, systems, and other technical artifacts through long-horizon experimentation. To understand the current state of this capability, however, evaluation must go beyond final scores, which neither reveal where progress is gained or lost nor indicate whether accumulated experience improves later decisions. We therefore present a systematic evaluation of seven frontier models on 36 long-horizon tasks based on a new framework that uses rule-based metrics to characterize within-run behavior through Solution Framing, Execution, and Feedback Control and controlled comparisons to assess experience reuse within and across tasks. The results show that current agents operate more like engineering optimizers than fully autonomous researchers: they can formulate and implement practical solutions, but their performance varies substantially across runs, their strongest solutions mainly adapt or combine established techniques, and genuine methodological novelty remains rare. Detailed analysis reveals that observed performance is shaped by multiple factors, including distinct process bottlenecks behind similar final outcomes, experience reuse that can help or mislead subsequent decisions, and harness designs that affect performance stability. These findings suggest concrete directions for improving model training, inference-time strategies, experience management, and harness design.

View arXiv page View PDF Project page Add to collection

Community

WenDingY

Paper submitter 1 day ago

🚀 We are excited to share Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development.

🔍 As AI agents increasingly tackle long-horizon research and engineering tasks, evaluating them by final scores alone is no longer enough. How do they actually approach, execute, and improve throughout the research process?

We systematically evaluate 7 frontier models across 36 long-horizon tasks, going beyond final performance to examine how agents frame solutions, execute experiments, respond to feedback, and reuse experience.

🧑‍🔬 Our results suggest that today's agents are better characterized as engineering optimizers than fully autonomous researchers: they can formulate and implement practical solutions, but performance remains highly variable across runs, strong solutions largely adapt or combine established techniques, and genuine methodological novelty is still rare.

📊 We also find that: