We evolved Bash Recovery to Bash Reliability for the Agent Arena leaderboard.
- A sharper read of what counts as a failure: we now read each bash call’s purpose, outcome and error cause, and whether the error was fixed within the call. The new signal now captures only mistakes the agent actually has made.
- Crediting reliable bash use: Bash Reliability measures agent-caused failures per non-exempt call, across every session. Models with a clean command-line use now get credit for getting it right the first time.
Backfilled scores are coming to our public Hugging Face datasets soon.
Learn more about Agent Arena signal definitions and Bash Reliability at: http://arena.ai/resources/agent-arena-signals
