How Gaussian Processes Work | Bayesian Regression Explained

How can we model an unknown function without specifying a fixed functional form in advance? In this video, we explore Gaussian Processes (GPs), a flexible Bayesian approach for modelling functions and making predictions with uncertainty. Instead of defining a regression function through a fixed set of coefficients, a Gaussian Process places a probability distribution over possible functions. Observed data then update this prior distribution to produce a posterior distribution over functions. Topics include: • What a Gaussian Process is • Gaussian Processes as distributions over functions • Bayesian nonparametric regression • Mean functions • Covariance functions and kernels • How kernels encode assumptions about functions • Similarity between observations • Prior functions • Conditioning on observed data • Posterior Gaussian Processes • Gaussian Process regression • Predictive mean and predictive uncertainty • Interpolation and extrapolation • The effect of kernel hyperparameters • Length scale and smoothness • Uncertainty away from observed data • Advantages and limitations of Gaussian Processes The central idea is: Prior over functions + Observed data → Posterior over functions A Gaussian Process is characterized by a mean function and a covariance function, or kernel. The kernel determines how strongly function values at different input locations are related. Points that are considered similar by the kernel tend to have similar function values. Before observing data, the Gaussian Process represents a range of plausible functions. After conditioning on observations, functions that are compatible with the data receive more probability, resulting in a posterior distribution that can be used for prediction. One of the major strengths of Gaussian Processes is that they provide both: Prediction + Uncertainty Rather than producing only a fitted curve, a GP naturally quantifies uncertainty about the unknown function — including how that uncertainty changes across the input space. Gaussian Processes are an important example of Bayesian nonparametric modelling and provide a powerful way to model nonlinear relationships without choosing a rigid functional form beforehand. This video is part of my Bayesian Statistics Explained series, where we build Bayesian concepts and modelling methods step by step. Subscribe for clear and intuitive explanations of statistics, probability, Bayesian modelling, and data science.