Latent Space

A latent space is the learned internal representation where a neural network encodes information about its inputs. 'Latent' means hidden — these representations are not directly observable in the raw data but are learned by the network during training. For example, an autoencoder compresses images into a low-dimensional latent vector and then reconstructs them. The latent space that emerges organizes images by high-level semantic features — faces are near other faces, landscapes near landscapes — even though the network was never told to do this.

Latent spaces are where the magic of generative AI happens. In a Variational Autoencoder (VAE), the latent space is designed to be smooth and continuous — meaning you can sample random points and decode them into plausible new images. In GANs, the generator maps random latent vectors to realistic images. In Stable Diffusion, the diffusion process operates in a compressed latent space (hence 'latent diffusion') rather than pixel space, making generation much faster. Moving smoothly through latent space produces smooth transitions between concepts — for example, morphing between a cat and a dog by interpolating between their latent representations.

The structure of a latent space reveals what a model has learned. If a model's latent space clusters similar concepts together and separates different ones, it has learned meaningful representations. Techniques like t-SNE and UMAP can visualize high-dimensional latent spaces in 2D, revealing clusters and relationships. In language models, the latent representations (hidden states) capture grammatical structure, semantic meaning, and world knowledge. Understanding latent spaces is key to understanding how neural networks represent and reason about information.