Authors: Dantong Niu, Yuvan Sharma, Haoru Xue, Giscard Biamby, Junyi Zhang, Ziteng Ji, Trevor Darrell, Roei Herzig
Abstract: Foundation models pre-trained on massive unlabeled datasets have
revolutionized natural language and computer vision, exhibiting remarkable
generalization capabilities, thus highlighting the importance of pre-training.
Yet, efforts in robotics have struggled to achieve similar success, limited by
either the need for costly robotic annotations or the lack of representations
that effectively model the physical world. In this paper, we introduce ARM4R,
an Auto-regressive Robotic Model that leverages low-level 4D Representations
learned from human video data to yield a better pre-trained robotic model.
Specifically, we focus on utilizing 3D point tracking representations from
videos derived by lifting 2D representations into 3D space via monocular depth
estimation across time. These 4D representations maintain a shared geometric
structure between the points and robot state representations up to a linear
transformation, enabling efficient transfer learning from human video data to
low-level robotic control. Our experiments show that ARM4R can transfer
efficiently from human video data to robotics and consistently improves
performance on tasks across various robot environments and configurations.
Source: http://arxiv.org/abs/2502.13142v1
