Authors: Pu Hua, Minghuan Liu, Annabella Macaluso, Yunfeng Lin, Weinan Zhang, Huazhe Xu, Lirui Wang
Abstract: Robotic simulation today remains challenging to scale up due to the human
efforts required to create diverse simulation tasks and scenes.
Simulation-trained policies also face scalability issues as many sim-to-real
methods focus on a single task. To address these challenges, this work proposes
GenSim2, a scalable framework that leverages coding LLMs with multi-modal and
reasoning capabilities for complex and realistic simulation task creation,
including long-horizon tasks with articulated objects. To automatically
generate demonstration data for these tasks at scale, we propose planning and
RL solvers that generalize within object categories. The pipeline can generate
data for up to 100 articulated tasks with 200 objects and reduce the required
human efforts. To utilize such data, we propose an effective multi-task
language-conditioned policy architecture, dubbed proprioceptive point-cloud
transformer (PPT), that learns from the generated demonstrations and exhibits
strong sim-to-real zero-shot transfer. Combining the proposed pipeline and the
policy architecture, we show a promising usage of GenSim2 that the generated
data can be used for zero-shot transfer or co-train with real-world collected
data, which enhances the policy performance by 20% compared with training
exclusively on limited real data.
Source: http://arxiv.org/abs/2410.03645v1
