Encoder-Decoder Architecture

The Encoder-Decoder architecture is a fundamental design pattern in deep learning for tasks where the input and output have different structures, lengths, or modalities. The encoder processes the entire input (a sentence, an image, a document) and compresses it into a fixed or variable-length internal representation — often called a latent vector, context vector, or hidden state. The decoder then takes this representation and generates the output step by step or all at once, depending on the architecture.

This pattern was originally popularized for machine translation using RNNs: the encoder reads a source sentence word by word and produces a context vector, then the decoder generates the target translation word by word from that vector. The addition of the Attention Mechanism was a breakthrough that allowed the decoder to selectively focus on relevant parts of the encoder's output at each generation step, dramatically improving translation quality. The Transformer architecture generalized this with self-attention in both encoder and decoder, leading to models like T5, BART, and the original GPT approach.

The encoder-decoder pattern extends far beyond text. In computer vision, U-Net uses an encoder-decoder structure for image segmentation — the encoder extracts features at decreasing resolutions and the decoder reconstructs a pixel-level segmentation map. Autoencoders use this pattern for unsupervised learning, compressing data to a low-dimensional bottleneck and reconstructing it. Variational Autoencoders (VAEs) and Diffusion Models also follow encoder-decoder principles. Understanding this architecture provides a unified lens for many seemingly different AI systems.