Perhaps the most mysterious aspect of modern deep generative models of images is that they work even when the number of training examples is much smaller than the dimensionality of the input. Often this is attributed to the “manifold hypothesis” which argues that the models estimate a low dimensional manifold that best fits the training distribution, but I will show that this explanation is flawed. Rather I will present theoretical and empirical results which demonstrate that architectural choices made in successful GANs and diffusion models make them learn the distribution of patches rather than the distribution of images. Finally, I will show work in progress where we apply this insight (“patches are all you need”) to classical methods for image generation.
Joint work with: Ariel Elnekave, Roy Friedman, Itamar Harel and Antonio Torralba.
Yair Weiss is the Dieter Schwarz Professor of Artificial Intelligence at the Hebrew University and the former Dean of the School of Computer Science and Engineering. His research interests include Human and Machine Vision, Machine Learning and Neural Computation.