Autoencoder from scratch in 60 minutes | Intuition + coding

If you are interested in image generation today, whether it is Variational Autoencoders or diffusion models or even large multimodal models that generate images from text, then spending serious time understanding a plain autoencoder is not optional, it is foundational, because almost every modern generative model is built by either extending, correcting, or compensating for what a simple autoencoder can and cannot do. At its core, an autoencoder is a very honest model. You give it an image, it compresses that image into a smaller representation called the latent space, and then it tries to reconstruct the original image from that compressed representation. There is no magic here. The encoder is learning what information it can afford to keep, and the decoder is learning how to rebuild the image using only that limited information. When this works well, it tells us something very deep: the model has discovered a compact structure underlying the data. When it fails, it also tells us something equally important: either the latent space is too small, the architecture is not expressive enough, or the task itself demands uncertainty that a deterministic mapping cannot capture. This is exactly where the real learning starts. A standard autoencoder learns a deterministic mapping from image to latent vector and back to image. For a given input image, you always get the same latent code. This makes autoencoders excellent for representation learning, dimensionality reduction, denoising, compression, and anomaly detection. If you train an autoencoder on normal data, say healthy medical scans or defect free manufactured parts, it becomes very good at reconstructing those patterns, and very bad at reconstructing anything abnormal. That reconstruction error becomes a signal. This is not a limitation, this is a strength, and many industrial systems rely on exactly this property. However, the same property becomes a limitation the moment you ask a different question. Can I randomly sample a latent vector and generate a new realistic image? In a vanilla autoencoder, the answer is usually no, or at least not reliably. The latent space learned by an autoencoder is not guaranteed to be continuous, smooth, or well behaved. Points between two valid latent codes may decode into garbage. Sampling from it blindly is like picking random coordinates in a city without knowing where the roads are. This is not a bug. The model was never trained to support generation from random latent samples. Once you understand this limitation clearly, the motivation behind Variational Autoencoders becomes obvious and almost inevitable. VAEs do not discard autoencoders. They take the same encoder decoder idea and add a probabilistic structure to the latent space. Instead of mapping an image to a single point, the encoder maps it to a distribution. Instead of hoping the latent space behaves nicely, the training objective forces it to align with a known prior distribution. In simple words, VAEs are trying to fix the one thing autoencoders are not designed to do, which is structured and reliable generation. Diffusion models take this line of thinking even further. They often operate in a learned latent space rather than raw pixel space because autoencoder style compression makes the problem computationally tractable. Many modern diffusion pipelines first learn a powerful autoencoder to map images into a lower dimensional latent representation, and then learn a diffusion process in that latent space. If you do not understand what information is preserved, lost, or distorted by the autoencoder, then the diffusion model becomes a black box on top of another black box. If you do understand it, suddenly the entire system feels logical. This is why autoencoders are such an important conceptual checkpoint. They teach us what it means to learn representations, what compression really costs us, why reconstruction loss shapes behavior so strongly, and why generation is not just about decoding but about structuring the latent space itself. They also teach humility. An autoencoder will happily memorize if you let it. It will overfit if the bottleneck is too wide. It will fail silently if the latent dimension is too small. All of these behaviors show up again in more sophisticated generative models, just in more subtle forms. What autoencoders can do very well is learn compact, task useful representations of images, remove noise, detect anomalies, and act as a backbone for larger systems. What they cannot do by design is model uncertainty or provide a principled way to sample new data. Understanding this boundary is what prepares you to truly appreciate why VAEs introduce randomness, why diffusion models rely on noise schedules, and why generative modeling is as much about probability as it is about neural networks.