Could you clarify whether "retry@5" is equivalent to pass@5?
If it is, we are concerned that this metric may not accurately reflect context management ability.
Since success is determined by an external evaluator over five attempts, rather than by the model selecting, integrating, or improving its own previous trajectories.
It is not context management
Thanks for your interest in our work.
pass@5 is a sampling-based evaluation metric. The model generates 5 independent attempts for a given task, and the task is considered solved if at least one attempt produces a correct answer. It measures the probability that the model can solve a problem given multiple chances.
retry@5 is a context management strategy not yet included in our code repo. It refers to the discard-all strategy described in the DeepSeek V3.2 Tech Report. You can also find an implementation in the MiroThinker codebase, where it corresponds to the context compressing limit. It works as follows:
- The model is given a maximum context window of 300 tool calls, but during inference, only the 5 most recent tool calls are retained (a sliding window).
- If the model exhausts the maximum context length without producing an answer, all prior context is cleared and a fresh run begins from scratch.
- This retry process repeats until the model either produces an answer or reaches the retry limit of 5 (i.e., the maximum number of times the context can be discarded).
- Crucially, for a single task, the model produces at most one final answer — or no answer at all if all 5 retries are exhausted without success.
Key distinction: pass@5 generates 5 independent candidate answers and succeeds if any one is correct — it is a multi-sample evaluation metric. retry@5, by contrast, is a context recovery mechanism that restarts execution when the model gets stuck, ultimately yielding a single answer. The retries are not independent parallel samples but sequential recovery attempts aimed at producing one definitive result.
In our experiments, the average number of tool calls under retry@5 is 400.9, whereas under pass@5, each independent attempt averages 119.7 tool calls.
You can reproduce the BrowseComp result using the MiroThinker code by replacing its tools with ours.
Maybe I misunderstood something, but I couldn't find the implementation details of "discard all" in this repository: https://github.com/InternScience/Agents-A1/tree/main/evaluation/Search. I'm not sure whether I was looking in the wrong place.
Maybe I misunderstood something, but I couldn't find the implementation details of "discard all" in this repository: https://github.com/InternScience/Agents-A1/tree/main/evaluation/Search. I'm not sure whether I was looking in the wrong place.
Hi flust,
Currently, the codebase doesn't include discard-all since it's mainly for evaluating HLE and other search benchmarks. BrowseComp requires additional context management, which we haven't open‑sourced yet. We plan to add it later. For now, you can refer to MiroThinker's implementation or use their evaluation framework for reproduction.