Unsupervised Learning: k-Means, PCA & Nearest Neighbors

Every model so far learned from labels. What if nobody gave it the answers? We hide the labels on 1,797 real handwritten digits and let the machine find structure: k-means step by step, choosing k, PCA squeezing 64 numbers into 2, k-nearest neighbors, scaling, the curse of dimensionality, and anomaly detection. Plus the trap: k-means finds "groups" even in pure noise. Every number on screen is real output from code you can run. Episode 8 of "From a Straight Line to ChatGPT", a full series on AI, machine learning and deep learning for developers and interview prep. ⏱ Chapters 0:00 Intro 0:50 Data without answers 1:28 k-means, step by step 2:11 Ten groups, ten digits? 2:54 Choosing k 3:33 What can go wrong: clusters in pure noise 4:20 PCA: fewer numbers 4:59 How many components? 5:42 k-nearest neighbors 6:16 Neighbors need scale 6:56 The curse of dimensionality 7:37 Anomaly detection 8:16 What can go wrong: the wrong shape 8:52 Try it yourself 9:29 Interview corner 10:39 Recap and next stop 🧪 Try it yourself (pip install numpy scikit-learn) import numpy as np from sklearn.cluster import KMeans from sklearn.datasets import load_digits from sklearn.decomposition import PCA from sklearn.metrics import adjusted_rand_score, silhouette_score X, y = load_digits(return_X_y=True) # y = the true digits, never shown to k-means noise = np.random.default_rng(0).uniform(0, 16, size=X.shape) # same shape, pure static print(f"images: {len(X)}, numbers per image: {X.shape[1]}") for name, data in {"digits": X, "noise": noise}.items(): kmeans = KMeans(n_clusters=10, random_state=0).fit(data) rerun = KMeans(n_clusters=10, random_state=1).fit(data) # same data, a different start print(f"{name}: 10 clusters, sizes {sorted(np.bincount(kmeans.labels_).tolist())}") print(f" silhouette {silhouette_score(data, kmeans.labels_):.3f}, " f"agreement with a rerun {adjusted_rand_score(kmeans.labels_, rerun.labels_):.2f}") if name == "digits": print(f" match with the hidden labels: {adjusted_rand_score(y, kmeans.labels_):.2f}") pca = PCA().fit(X) # no labels here either print("variance kept by each component:", np.round(pca.explained_variance_ratio_[:5], 3).tolist(), "...") print(f"the first 2 components keep {pca.explained_variance_ratio_[:2].sum():.1%}") print("components needed to keep 90%:", PCA(n_components=0.9).fit(X).n_components_) What we got: on the digits, silhouette 0.186, a rerun agrees 0.72, and the clusters match the hidden labels at 0.62. On static of the same shape, k-means still returns 10 tidy groups, but silhouette 0.012 and rerun agreement 0.03. PCA: the first 2 components keep 28.5%; 21 keep 90%. 🧠 Cheat sheet k-means: assign each point to its nearest center, move each center to its points' mean, repeat k-means always returns k groups, even in noise: compare against a same-shape baseline Choosing k: elbow (inertia), silhouette, rerun stability; a score needs a baseline k-means assumes round, similar-size clusters and needs scaled features; DBSCAN finds other shapes PCA: directions of most variance; variance is not the same as importance for prediction k-NN: no training step, label by the k nearest examples' vote; scale your features Curse of dimensionality: in high dimensions, nearest and farthest get almost equally far Anomaly detection: learn what normal looks like, flag what's far from it 📚 Sources Optical Recognition of Handwritten Digits, UCI (CC BY 4.0): https://archive.ics.uci.edu/dataset/8... Wine, UCI (CC BY 4.0): https://archive.ics.uci.edu/dataset/1... MacQueen, k-means (1967): https://projecteuclid.org/euclid.bsms... Arthur & Vassilvitskii, k-means++ (2007): https://doi.org/10.1145/1283383.1283494 Pearson, lines and planes of closest fit (1901): https://doi.org/10.1080/1478644010946... Cover & Hart, nearest neighbor (1967): https://doi.org/10.1109/TIT.1967.1053964 Ester et al., DBSCAN (1996): https://www.osti.gov/biblio/421283 Wattenberg et al., How to use t-SNE effectively (2016): https://distill.pub/2016/misread-tsne/ 📚 Series: From a Straight Line to ChatGPT AI06 Is Your Model Any Good? Metrics That Matter AI07 Decision Trees, Random Forests & Boosting AI08 Learning Without Labels: Clustering, PCA & Nearest Neighbors ← you are here AI09 Neurons & Neural Networks … full playlist:    • Artificial Intelligence (AI)   Just Known: from unknown to known. A channel by Just Unknown. Note: the handwritten digits and wines are real data; the static, shuffled pixels, random points and half-moons are made-up data. Pictures marked "illustration" are not data. #ai #machinelearning #unsupervisedlearning #python #computerscience #codinginterview