AI Is Getting Too Good at the Wrong Thing

Why do AI systems cheat? Because we train them on scores, and a score is never quite the thing we want. This is the story of Goodhart's law, from the cobra bounty to reward hacking in today's AI labs, and of a race to build systems nobody can fully test. CHAPTERS 0:00 Intro: The Evolved Circuit 4:12 I. The Cobra Effect 7:31 II. Goodhart's Law 12:07 III. Regressional Goodhart 19:07 IV. Extremal Goodhart 22:17 V. Causal Goodhart 25:42 VI. Reward Hacking 32:36 VII. Adversarial Goodhart 40:05 VIII. The Measure Becomes the Business 45:37 IX. A Race No One Can Win 49:32 X. No Second Try PAPERS AND BOOKS Bird & Layzell, “The Evolved Radio” (2002) Siebert, Der Kobra-Effekt (2001) Vann, “Of Rats, Rice, and Race: The Great Hanoi Rat Massacre” (2003) Goodhart, “Problems of Monetary Management: the U.K. Experience” (1975) Strathern, “‘Improving ratings’: audit in the British University system” (1997) Locker & Mason, “Analysis of the distribution of time that patients spend in emergency departments” (2005) Kinnard, The War Managers (1977) Manheim & Garrabrant, “Categorizing Variants of Goodhart's Law” (2018) Galton, “Typical Laws of Heredity” (1877); “Regression Towards Mediocrity in Hereditary Stature” (1886) Kahneman, Thinking, Fast and Slow (2011) Kane & Staiger, “Volatility in School Test Scores” (2002) Tinbergen, The Study of Instinct (1951) Gwynne & Rentz, “Beetles on the Bottle: Male Buprestids Mistake Stubbies for Females” (1983) Stiennon et al., “Learning to summarize from human feedback” (2020) Gao et al., “Scaling Laws for Reward Model Overoptimization” (2022) Singhal et al., “A Long Way to Go: Investigating Length Correlations in RLHF” (2023) Sharma et al., “Towards Understanding Sycophancy in Language Models” (2023) Wei et al., “Jailbroken: How Does LLM Safety Training Fail?” (2023) Krakovna et al., “Specification gaming examples in AI” (2018-) Lehman et al., “The Surprising Creativity of Digital Evolution” (2020) Christiano et al., “Deep reinforcement learning from human preferences” (2017) Bondarenko et al., “Demonstrating specification gaming in reasoning models” (2025) Baker et al., “Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation” (2025) Korbak et al., “Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety” (2025) Zhong et al., “ImpossibleBench” (2025) MacDiarmid et al., “Natural emergent misalignment from reward hacking in production RL” (2025) Schiestl et al., “Orchid pollination by sexual swindle” (1999) Searle, “Minds, Brains, and Programs” (1980) Dennett, The Intentional Stance (1987) Goldstein & Stanovsky, “Do Zombies Understand?” (2024) Greenblatt et al., “Alignment faking in large language models” (2024) Potter et al., “Peer-Preservation in Frontier Models” (2026) Singh et al., “The Leaderboard Illusion” (2025) Bastounis et al., “Navier-Stokes lost in translation” (2026) Paglieri et al., “A Case Study on Emergent Cheating and Whistleblowing in Autonomous Research Swarms” (2026) Omohundro, “The Basic AI Drives” (2008) Bostrom, “The Superintelligent Will” (2012) Turner et al., “Optimal Policies Tend to Seek Power” (2021) Russell, Human Compatible (2019) MUSIC J.S. Bach: The Well-Tempered Clavier I: Fugue in G minor, BWV 861; Prelude in G minor, BWV 861; Prelude in C major, BWV 846; Prelude in C minor, BWV 847; Prelude in E minor, BWV 855 J.S. Bach: Fugue in G minor, BWV 578 (“Little”), arr. for saxophone quartet Saint-Saëns: Danse macabre, Op. 40 Satie: Gnossiennes No. 1 and No. 2 Javanese gamelan, court musicians of the Kraton Ngayogyakarta (Yogyakarta), excerpt 1 Schubert: Piano Trio No. 2 in E-flat major, D. 929 (Op. 100), II. Andante con moto J.S. Bach: Goldberg Variations: Aria; Variations 1, 4, 7, 9, 13, 15, 24, 25 Beethoven: Symphony No. 3 “Eroica”, II. Marcia funebre Brahms: Symphony No. 4 in E minor, Op. 98, IV. Allegro energico e passionato J.S. Bach: Harpsichord Concerto No. 1 in D minor, BWV 1052, I. Allegro and II. Adagio Brahms: Tragic Overture, Op. 81 Borodin: In the Steppes of Central Asia Grieg: Piano Concerto in A minor, Op. 16, I. Allegro molto moderato Used under Creative Commons Attribution licenses: Gamelan (Kraton Ngayogyakarta), recording by RTB45, CC BY 4.0 (creativecommons.org/licenses/by/4.0), commons.wikimedia.org/wiki/File:Gamelan_angka_1_Kraton_Ngayogyakarta.ogg Schubert, Piano Trio No. 2, II: P. Thiemann, S. Ostrovsky, S. Myong (Gran Teatre del Liceu), CC BY 3.0 (creativecommons.org/licenses/by/3.0),    • Segon moviment del Trio n. 2 per a piano, ...  , excerpt Bach, Harpsichord Concerto No. 1, BWV 1052: L. Mravunac Fabijanić, Komorni orkestar Jastrebarsko, CC BY 3.0 (creativecommons.org/licenses/by/3.0),    • Komorni Orkestar Jastrebarsko - J. S. Bach...  , excerpts