AI-SOVEREIGN
Home / AI Research / Online Mistake Detection in Instruction-Conditioned Procedural Video
ONAI Research

Online Mistake Detection in Instruction-Conditioned Procedural Video

by UoHaifa · Amit Feldman, Ilan Shimshoni, Rotem Dror

An assistant monitoring a person performing a procedure is most useful when it can detect mistakes while there is still time to recover. This requires judging actions against task instructions as the video unfolds, identifying the attempted step, locating it in time, and assessing its execution. Existing online detectors do not use the instruction text, while instruction-conditioned methods operate offline. In this paper, we study online, instruction-conditioned mistake detection in egocentric videos. Using only observed video, task instructions, and prior predictions, our model identifies and localizes completed action attempts and classifies them as correct, modified, slip, corrections,…

Upvotes11
Comments2
Published18 Nov 2026