Generative Adversarial Networks (GANs) hold the state-of-the-art when it comes to image generation.
However, while the rest of computer vision is slowly taken over by transformers or other attention-based architectures, all working GANs to date contain some form of convolutional layers. This paper changes that and builds TransGAN, the first GAN where both the generator and the discriminator are transformers. The discriminator is taken over from ViT (an image is worth 16×16 words), and the generator uses pixelshuffle to successfully up-sample the generated resolution. Three tricks make training work: Data augmentations using DiffAug, an auxiliary superresolution task, and a localized initialization of self-attention.
Their largest model reaches competitive performance with the best convolutional GANs on CIFAR10, STL-10, and CelebA.
Yannic Kilcher explains