Life After Benchmark Saturation: A Case Study of CORE-Bench

This research paper proposes a new approach to evaluating AI agent performance beyond just accuracy, especially when benchmarks become saturated. It introduces six additional dimensions for assessment: construct validity, out-of-distribution generalizability, efficiency, reliability, model vs. scaffold importance, and human-agent collaboration uplift. Using CORE-Bench Hard as a case study, the paper demonstrates how these dimensions provide meaningful insights into agent capabilities, even after accuracy plateaus, and introduces improved benchmarks and an out-of-distribution task suite.