AI-SOVEREIGN
Home / AI Research / Phantom Success, Silent Failure: Auditing the Success Checkers of Simulated Manipulation Benchmarks
PHAI Research

Phantom Success, Silent Failure: Auditing the Success Checkers of Simulated Manipulation Benchmarks

by MBZUAI · Zirui Song, Jiadong Zhang, Zheng Lu, Xiaosong Ma, Xiaojun Chang, Xiuying Chen

Simulated manipulation benchmarks measure policy performance using hand-written success checkers that label each episode as a success or failure. Passing an expert demonstration shows that a checker accepts one way of completing the task. Learned policies, however, may stop after partial progress or complete the task in a different way. A checker may then accept an incomplete execution (phantom success) or reject an execution that satisfies the instruction (silent failure). In this paper, we introduce SimAudit, an LLM-based pipeline that audits success checkers against task instructions. SimAudit analyzes the instruction and checker code to identify scenarios where the two may disagree,…

Upvotes4
Comments5
Published17 Nov 2026

More from MBZUAI