Authors: Yifei Zhou, Qianlan Yang, Kaixiang Lin, Min Bai, Xiong Zhou, Yu-Xiong Wang, Sergey Levine, Erran Li
Abstract: The vision of a broadly capable and goal-directed agent, such as an
Internet-browsing agent in the digital world and a household humanoid in the
physical world, has rapidly advanced, thanks to the generalization capability
of foundation models. Such a generalist agent needs to have a large and diverse
skill repertoire, such as finding directions between two travel locations and
buying specific items from the Internet. If each skill needs to be specified
manually through a fixed set of human-annotated instructions, the agent’s skill
repertoire will necessarily be limited due to the quantity and diversity of
human-annotated instructions. In this work, we address this challenge by
proposing Proposer-Agent-Evaluator, an effective learning system that enables
foundation model agents to autonomously discover and practice skills in the
wild. At the heart of PAE is a context-aware task proposer that autonomously
proposes tasks for the agent to practice with context information of the
environment such as user demos or even just the name of the website itself for
Internet-browsing agents. Then, the agent policy attempts those tasks with
thoughts and actual grounded operations in the real world with resulting
trajectories evaluated by an autonomous VLM-based success evaluator. The
success evaluation serves as the reward signal for the agent to refine its
policies through RL. We validate PAE on challenging vision-based web
navigation, using both real-world and self-hosted websites from WebVoyager and
WebArena.To the best of our knowledge, this work represents the first effective
learning system to apply autonomous task proposal with RL for agents that
generalizes real-world human-annotated benchmarks with SOTA performances. Our
open-source checkpoints and code can be found in https://yanqval.github.io/PAE/
Source: http://arxiv.org/abs/2412.13194v1
