Authors: Brandon Trabucco, Gunnar Sigurdsson, Robinson Piramuthu, Ruslan Salakhutdinov
Abstract: The predominant approach for training web navigation agents gathers human
demonstrations for a set of popular websites and hand-written tasks, but it is
becoming clear that human data are an inefficient resource. We develop a
pipeline to facilitate Internet-scale training for agents without laborious
human annotations. In the first stage, an LLM generates tasks for 150k diverse
websites. In the next stage, LLM agents complete tasks and produce
trajectories. In the final stage, an LLM reviews the trajectories and judges
their success. Language models are competitive with human annotators, detecting
and filtering out harmful content with an accuracy of 97%, generating feasible
tasks with an 89% rate, and judging successful trajectories with an 82.6%
accuracy. Scaling the pipeline, agents based on Llama 3.1 70B solve 16.7% of
tasks for 150k sites. Training on the data generated by our pipeline is
competitive with training on human demonstrations. In data-limited settings
derived from Mind2Web and WebLINX, we improve Step Accuracy by up to +89.5% and
+122.1% respectively for agents trained on mixtures of data from our pipeline,
and human data. When training agents with all available human data from these
benchmarks, agents fail to generalize to diverse real sites, and adding our
data improves their generalization by +149.0% for WebLINX and +156.3% for
Mind2Web. Code will be available at: data-for-agents.github.io.
Source: http://arxiv.org/abs/2502.06776v1
