The Decision Score rewards models for both correct answers and well-calibrated confidence. A decision model’s answer is typically acted on when its confidence clears a threshold; otherwise, it is passed to a person or a more capable model. The Decision Score measures the average payoff across all thresholds, so it does not depend on choosing a single cutoff.
For each question, the score is 50 × (1 + c − (p − c)²), where p is the probability the model assigns to its chosen answer and c is a flag for whether the answer was correct (c = 1 if correct, c = 0 if incorrect). Across an evaluation, this is equal to 50 × (1 + Accuracy − Brier score).
Including confidence impacts overall leaderboard rankings. On AA-Paperclip, for raw accuracy Clef passes 36% of documents and Celeris-1.1 Decision passes 24%, but Celeris-1.1 Decision matches it on the Decision Score due to better calibration (Brier score of 0.15 vs. 0.27 for Clef). Across the Index, GLiDE v1.2 leads Drex Max v1.1 on accuracy metrics (60 vs. 58), while Drex Max v1.1 edges ahead on the Decision Index.