Authors: Haotian Tang, Yecheng Wu, Shang Yang, Enze Xie, Junsong Chen, Junyu Chen, Zhuoyang Zhang, Han Cai, Yao Lu, Song Han
Abstract: We introduce Hybrid Autoregressive Transformer (HART), an autoregressive (AR)
visual generation model capable of directly generating 1024×1024 images,
rivaling diffusion models in image generation quality. Existing AR models face
limitations due to the poor image reconstruction quality of their discrete
tokenizers and the prohibitive training costs associated with generating 1024px
images. To address these challenges, we present the hybrid tokenizer, which
decomposes the continuous latents from the autoencoder into two components:
discrete tokens representing the big picture and continuous tokens representing
the residual components that cannot be represented by the discrete tokens. The
discrete component is modeled by a scalable-resolution discrete AR model, while
the continuous component is learned with a lightweight residual diffusion
module with only 37M parameters. Compared with the discrete-only VAR tokenizer,
our hybrid approach improves reconstruction FID from 2.11 to 0.30 on MJHQ-30K,
leading to a 31% generation FID improvement from 7.85 to 5.38. HART also
outperforms state-of-the-art diffusion models in both FID and CLIP score, with
4.5-7.7x higher throughput and 6.9-13.4x lower MACs. Our code is open sourced
at https://github.com/mit-han-lab/hart.
Source: http://arxiv.org/abs/2410.10812v1
