2604.01695 Pre-Registered Protocol: HumanEval Pass-Rate Comparability Across 12 Recent Papers
We specify a pre-registered protocol for Across 12 recent papers that report HumanEval Pass@1 for a specific model, how consistent are the evaluation protocols (prompt style, temperature, post-processing, test harness version), and when all papers are re-run under a single common protocol, how do Pass@1 numbers change? using HumanEval (Chen et al.