Authors: Xiaolong Jin, Kai Wang, Dongwen Tang, Wangbo Zhao, Yukun Zhou, Junshu Tang, Yang You
Abstract: Generative models have achieved remarkable success in image, video, and text
domains. Inspired by this, researchers have explored utilizing generative
models to generate neural network parameters. However, these efforts have been
limited by the parameter size and the practicality of generating
high-performance parameters. In this paper, we propose COND P-DIFF, a novel
approach that demonstrates the feasibility of controllable high-performance
parameter generation, particularly for LoRA (Low-Rank Adaptation) weights,
during the fine-tuning process. Specifically, we employ an autoencoder to
extract efficient latent representations for parameters. We then train a
conditional latent diffusion model to synthesize high-performing model
parameters from random noise based on specific task conditions. Experimental
results in both computer vision and natural language processing domains
consistently demonstrate that COND P-DIFF can generate high-performance
parameters conditioned on the given task. Moreover, we observe that the
parameter distribution generated by COND P-DIFF exhibits differences compared
to the distribution obtained through normal optimization methods, indicating a
certain level of generalization capability. Our work paves the way for further
exploration of condition-driven parameter generation, offering a promising
direction for task-specific adaptation of neural networks.
Source: http://arxiv.org/abs/2408.01415v1
