Home / AI Research / Suppressing the Shortcut Hidden Behind the Gains of VideoLLM Fine-Tuning
SUAI Research
Suppressing the Shortcut Hidden Behind the Gains of VideoLLM Fine-Tuning
· Youngwoo Shin, Jongsuk Kim, Junmo Kim
Supervised fine-tuning (SFT) is the stage in which Video Large Language Models (VideoLLMs) advance their capability to reason over video. However, the VideoLLM also learns to answer from textual regularities of its training data, a shortcut that bypasses the video. The regularities behind the shortcut are hard to define at the data level, and the shortcut forms regardless of training epoch, data size, or backbone. The shortcut is hidden behind a rise in accuracy within the training distribution, yet it is not required for the rise and even costs SFT the gain the video offers under distribution shift. We therefore propose Video-Evidence Training via Orthogonal projection (VETO), which…
Upvotes4
Comments2
Published18 Nov 2026