Authors: Xiangyu Zhao, Chengqian Ma
Abstract: Large Language Models (LLMs) exhibit remarkable proficiency in addressing a
diverse array of tasks within the Natural Language Processing (NLP) domain,
with various prompt design strategies significantly augmenting their
capabilities. However, these prompts, while beneficial, each possess inherent
limitations. The primary prompt design methodologies are twofold: The first,
exemplified by the Chain of Thought (CoT), involves manually crafting prompts
specific to individual datasets, hence termed Expert-Designed Prompts (EDPs).
Once these prompts are established, they are unalterable, and their
effectiveness is capped by the expertise of the human designers. When applied
to LLMs, the static nature of EDPs results in a uniform approach to both simple
and complex problems within the same dataset, leading to the inefficient use of
tokens for straightforward issues. The second method involves prompts
autonomously generated by the LLM, known as LLM-Derived Prompts (LDPs), which
provide tailored solutions to specific problems, mitigating the limitations of
EDPs. However, LDPs may encounter a decline in performance when tackling
complex problems due to the potential for error accumulation during the
solution planning process. To address these challenges, we have conceived a
novel Prompt Recursive Search (PRS) framework that leverages the LLM to
generate solutions specific to the problem, thereby conserving tokens. The
framework incorporates an assessment of problem complexity and an adjustable
structure, ensuring a reduction in the likelihood of errors. We have
substantiated the efficacy of PRS framework through extensive experiments using
LLMs with different numbers of parameters across a spectrum of datasets in
various domains. Compared to the CoT method, the PRS method has increased the
accuracy on the BBH dataset by 8% using Llama3-7B model, achieving a 22%
improvement.
Source: http://arxiv.org/abs/2408.01423v1
