Eigenvalues & Eigenvectors Explained: Spectral Theorem & PCA | Ink into Stars: Lecture 12

Welcome to Lecture 12 of "Ink into Stars: 108 Lectures on Machine Learning"! Under the night sky, every star traces a circular arc — except Polaris, which sits directly on Earth's rotational axis. On a potter's wheel, the clay spins — yet the central axis never changes direction. When a matrix warps space, most arrows both stretch and rotate off their lines, but a few special directions refuse to turn: they only stretch or shrink along their own line. Those invariant directions are Eigenvectors, and their scaling factors are Eigenvalues (`A v = lambda v`)! In this lecture, we master the geometry and algebra of eigenvalues and eigenvectors: • The Eigenvector Equation (`A v = lambda v`): Why eigenvectors stay on their span, why `v != 0`, and why any nonzero multiple of an eigenvector is still an eigenvector along the same line. • Characteristic Equation (`det(A - lambda I) = 0`): Rewriting `A v = lambda v` as `(A - lambda I) v = 0`, why a nonzero solution requires the matrix `A - lambda I` to collapse space (`det = 0`), and solving a 2×2 example by hand. • Trace & Determinant Identities: Why `lambda_1 + lambda_2 = tr(A)` and `lambda_1 * lambda_2 = det(A)`, plus the four eigenvalue regimes (expanding, shrinking, flipping orientation, collapsing space, and pure rotation). • Eigendecomposition (`A = V Lambda V^-1`), Power Iteration & PageRank: How diagonalization turns `A^k` into rotate-scale-rotate, why repeated multiplication `A^k x` converges to the principal eigenvector, and how Google's PageRank finds the stationary distribution `M pi = 1 * pi`. • The Spectral Theorem (`A = Q Lambda Q^T`) & Quadratic Forms (`x^T A x`): Why real symmetric matrices (like covariance matrices and Hessians!) always have real eigenvalues and mutually orthogonal eigenvectors, eliminating cross terms and classifying bowls vs. saddles. • Preview of PCA on the Wine Dataset: Using `np.linalg.eigh` on the 13×13 Wine covariance matrix to capture 55.4% of total 13D variance in just two orthogonal principal directions (achieving 94.8% k-NN accuracy in 2D!). ⏱️ Chapters: 00:00 Cold Open: Polaris Star Trails, The Potter's Wheel & Invariant Axes 00:46 Lecture 12 Title: Invariant Directions — Eigenvalues & Eigenvectors 00:58 Most Directions Rotate; Two Special Directions Only Scale (Av = lambda v) 02:56 Characteristic Equation: Why det(A - lambda I) = 0 Finds Eigenvalues 03:35 Solving Eigenvalues & Eigenvectors by Hand on A = [[2, 1], [1, 2]] 04:38 Four Eigenvalue Regimes & Pure Rotation Matrices 05:32 Trace = Sum of Eigenvalues, Determinant = Product of Eigenvalues 06:35 Eigendecomposition A = V Lambda V^-1 & Pause & Think #2 07:50 Power Iteration (A^k x) & Google PageRank Stationary Distribution 09:38 Symmetric Matrices & The Spectral Theorem (A = Q Lambda Q^T) 10:42 Quadratic Forms (x^T A x): Positive Definite Bowl vs. Indefinite Saddle 12:15 Maximizing Projected Variance (u^T Sigma u) & Pause & Think #3 13:25 Python: np.linalg.eigh vs. eig & Wine 13x13 Covariance Matrix 15:00 Top 2 Eigenvectors (94.8% Accuracy), Three Takeaways & Preview of SVD 📚 Course Playlist (Ink into Stars: 108 Lectures on Machine Learning): • Full Playlist:    • Ink into Stars: 108 Lectures on Machine Le...   • Lecture 01: Ink into Stars — How Machines Learn from Experience:    • What Is Machine Learning? From Stargazers ...   • Lecture 02: Data, Features & Labels — Reading the Raw Stone:    • Raw Materials: Data, Features & Labels | I...   • Lecture 03: Three Waves — A Brief History of Machine Learning:    • Three Waves of Machine Learning: Perceptro...   • Lecture 04: Translating Real Problems into Models:    • Translating Business Problems into ML Mode...   • Lecture 05: Taking Up the Brush — Python from Zero:    • Python for Machine Learning from Scratch: ...   • Lecture 06: The Power of Arrays — NumPy & Vectorization:    • NumPy & Vectorization: Why Arrays Run 126x...   • Lecture 07: The Language of Tables — Pandas Data Processing:    • Pandas for Machine Learning: DataFrames, M...   • Lecture 08: Developing the Image — Matplotlib & Seaborn:    • Data Visualization with Matplotlib & Seabo...   • Lecture 09: One Shared Grammar — Scikit-Learn & Pipelines:    • Scikit-Learn Complete Guide: Pipelines, Cr...   • Lecture 10: Arrows in Space — Vectors, Norms & Dot Products:    • Vectors, Norms & Dot Products: The Geometr...   • Lecture 11: Warping Space — Matrices & Linear Transformations:    • Matrices as Linear Transformations: Column...   • Lecture 12: Invariant Directions — Eigenvalues & Eigenvectors (This Video) 📖 Free Companion Book & Code Repository: • GitHub: https://github.com/augustfu-droid/ml108 #Eigenvalues #Eigenvectors #LinearAlgebra #SpectralTheorem #PCA #PageRank #MachineLearning #MathematicsForML #NumPy #InkIntoStars