Authors: Yaoxiang Wang, Haoling Li, Xin Zhang, Jie Wu, Xiao Liu, Wenxiang Hu, Zhongxin Guo, Yangyu Huang, Ying Xin, Yujiu Yang, Jinsong Su, Qi Chen, Scarlett Li
Abstract: Effective instruction tuning is indispensable for optimizing code LLMs,
aligning model behavior with user expectations and enhancing model performance
in real-world applications. However, most existing methods focus on code
snippets, which are limited to specific functionalities and rigid structures,
restricting the complexity and diversity of the synthesized data. To address
these limitations, we introduce a novel feature tree-based synthesis framework
inspired by Abstract Syntax Trees (AST). Unlike AST, which captures syntactic
structure of code, our framework models semantic relationships between code
elements, enabling the generation of more nuanced and diverse data. The feature
tree is constructed from raw data and refined iteratively to increase the
quantity and diversity of the extracted features. This process enables the
identification of more complex patterns and relationships within the code. By
sampling subtrees with controlled depth and breadth, our framework allows
precise adjustments to the complexity of the generated code, supporting a wide
range of tasks from simple function-level operations to intricate multi-file
scenarios. We fine-tuned widely-used base models to create the EpiCoder series,
achieving state-of-the-art performance at both the function and file levels
across multiple benchmarks. Notably, empirical evidence indicates that our
approach shows significant potential in synthesizing highly complex
repository-level code data. Further analysis elucidates the merits of this
approach by rigorously assessing data complexity and diversity through software
engineering principles and LLM-as-a-judge method.
Source: http://arxiv.org/abs/2501.04694v1
