Authors: Y. Isabel Liu, Windsor Nguyen, Yagiz Devre, Evan Dogariu, Anirudha Majumdar, Elad Hazan
Abstract: This paper describes an efficient, open source PyTorch implementation of the
Spectral Transform Unit. We investigate sequence prediction tasks over several
modalities including language, robotics, and simulated dynamical systems. We
find that for the same parameter count, the STU and its variants outperform the
Transformer as well as other leading state space models across various
modalities.
Source: http://arxiv.org/abs/2409.10489v1
