Authors: Shijie Wang, Samaneh Azadi, Rohit Girdhar, Saketh Rambhatla, Chen Sun, Xi Yin
Abstract: Text-Image-to-Video (TI2V) generation aims to generate a video from an image
following a text description, which is also referred to as text-guided image
animation. Most existing methods struggle to generate videos that align well
with the text prompts, particularly when motion is specified. To overcome this
limitation, we introduce MotiF, a simple yet effective approach that directs
the model’s learning to the regions with more motion, thereby improving the
text alignment and motion generation. We use optical flow to generate a motion
heatmap and weight the loss according to the intensity of the motion. This
modified objective leads to noticeable improvements and complements existing
methods that utilize motion priors as model inputs. Additionally, due to the
lack of a diverse benchmark for evaluating TI2V generation, we propose TI2V
Bench, a dataset consists of 320 image-text pairs for robust evaluation. We
present a human evaluation protocol that asks the annotators to select an
overall preference between two videos followed by their justifications. Through
a comprehensive evaluation on TI2V Bench, MotiF outperforms nine open-sourced
models, achieving an average preference of 72%. The TI2V Bench is released in
https://wang-sj16.github.io/motif/.
Source: http://arxiv.org/abs/2412.16153v1
