Authors: Guangzhi Xiong, Qiao Jin, Xiao Wang, Yin Fang, Haolin Liu, Yifan Yang, Fangyuan Chen, Zhixing Song, Dengyu Wang, Minjia Zhang, Zhiyong Lu, Aidong Zhang
Abstract: Retrieval-augmented generation (RAG) has shown great potential for
knowledge-intensive tasks, but its traditional architectures rely on static
retrieval, limiting their effectiveness for complex questions that require
sequential information-seeking. While agentic reasoning and search offer a more
adaptive approach, most existing methods depend heavily on prompt engineering.
In this work, we introduce RAG-Gym, a unified optimization framework that
enhances information-seeking agents through fine-grained process supervision at
each search step. We also propose ReSearch, a novel agent architecture that
synergizes answer reasoning and search query generation within the RAG-Gym
framework. Experiments on four challenging datasets show that RAG-Gym improves
performance by up to 25.6\% across various agent architectures, with ReSearch
consistently outperforming existing baselines. Further analysis highlights the
effectiveness of advanced LLMs as process reward judges and the transferability
of trained reward models as verifiers for different LLMs. Additionally, we
examine the scaling properties of training and inference in agentic RAG. The
project homepage is available at https://rag-gym.github.io/.
Source: http://arxiv.org/abs/2502.13957v1
