Authors: Jathushan Rajasegaran, Xinlei Chen, Rulilong Li, Christoph Feichtenhofer, Jitendra Malik, Shiry Ginosar
Abstract: This paper explores Masked Autoencoders (MAE) with Gaussian Splatting. While
reconstructive self-supervised learning frameworks such as MAE learns good
semantic abstractions, it is not trained for explicit spatial awareness. Our
approach, named Gaussian Masked Autoencoder, or GMAE, aims to learn semantic
abstractions and spatial understanding jointly. Like MAE, it reconstructs the
image end-to-end in the pixel space, but beyond MAE, it also introduces an
intermediate, 3D Gaussian-based representation and renders images via
splatting. We show that GMAE can enable various zero-shot learning capabilities
of spatial understanding (e.g., figure-ground segmentation, image layering,
edge detection, etc.) while preserving the high-level semantics of
self-supervised representation quality from MAE. To our knowledge, we are the
first to employ Gaussian primitives in an image representation learning
framework beyond optimization-based single-scene reconstructions. We believe
GMAE will inspire further research in this direction and contribute to
developing next-generation techniques for modeling high-fidelity visual data.
More details at https://brjathu.github.io/gmae
Source: http://arxiv.org/abs/2501.03229v1
