Home / AI Research / Phantom Success, Silent Failure: Auditing the Success Checkers of Simulated Manipulation Benchmarks
PHAI Research
Phantom Success, Silent Failure: Auditing the Success Checkers of Simulated Manipulation Benchmarks
by MBZUAI · Zirui Song, Jiadong Zhang, Zheng Lu, Xiaosong Ma, Xiaojun Chang, Xiuying Chen
Simulated manipulation benchmarks measure policy performance using hand-written success checkers that label each episode as a success or failure. Passing an expert demonstration shows that a checker accepts one way of completing the task. Learned policies, however, may stop after partial progress or complete the task in a different way. A checker may then accept an incomplete execution (phantom success) or reject an execution that satisfies the instruction (silent failure). In this paper, we introduce SimAudit, an LLM-based pipeline that audits success checkers against task instructions. SimAudit analyzes the instruction and checker code to identify scenarios where the two may disagree,…
Upvotes4
Comments5
Published17 Nov 2026