The Fix That Caused the 2012 Leap Second Bug

In July 2007 a Linux commit removed a call to clock_was_set() from the leap second code, because it could deadlock the machine. The commit message expected one rare side effect, a timer firing one second late on a leap second deletion, and added: "It will never fire too early." On the leap second at the end of 30 June 2012, absolute CLOCK_REALTIME timers on affected Linux kernels fired one second early, and programs waiting on futexes with short deadlines spun at full CPU. The fix in July 2012 put the same call back on the leap second path, as clock_was_set_delayed(), so the update runs after the timer interrupt, where it is safe to ask the other CPUs to do work. Chapters 0:00 A sentence from 2007 0:41 A minute with sixty-one seconds 1:24 The thirtieth of June 2:19 Two clocks, one second apart 4:08 Removed on purpose 5:38 Same call, one step later 6:28 Time going backwards in your own code 7:15 Smear it, then stop Details the video keeps short The early reports blamed Java and Cassandra, then a Linux deadlock with a patch from March. The March 2012 commit (6b43ae8a619d) fixed a different livelock in the leap second code and was already in 3.4.4, the kernel in the first report. The fix for this bug landed in v3.5-rc7 and was backported to 3.4-stable. John Stultz wrote that the bug had likely been there since the 2007 commit (v2.6.22). The record doesn't say whether the leap second at the end of 2008 caused the same spike. What fired early, per John Stultz: absolute timers on the realtime clock (TIMER_ABSTIME CLOCK_REALTIME), and the sub-second ones returned immediately. Cloudflare, 1 January 2017: a smoothed round trip time went negative across the leap second and reached rand.Int63n, which panics on negative input. Since Go 1.9, time values carry a monotonic reading. BIPM Resolution 5 (2026): continuous UTC from 20 May 2027, with the maximum UT1-UTC difference raised to 3,600 seconds. Sources (search by title; links will be added once the channel can post them) Main sources: Linux commit 746976a301ac, NTP, remove clock_was_set() call to prevent deadlock (Thomas Gleixner, 3 July 2007) LKML, Leap second insertion causes futex to repeatedly timeout (1 July 2012) LKML, John Stultz, Potential fix for leapsecond caused futex issue (v2), 1 July 2012 Linux commit 4873fa070ae8, timekeeping, Fix leapsecond triggered load spike issue (John Stultz, 10 July 2012) LKML, John Stultz, Fix for leapsecond caused hrtimer/futex issue (updated), 10 July 2012 Linux commit 6b43ae8a619d, ntp, Fix leap-second hrtimer livelock (March 2012, a different bug) Cloudflare, How and why the leap second affected Cloudflare DNS (January 2017) Go 1.9 release notes (monotonic time) BIPM, Resolution 5 of the 28th CGPM (2026), continuity of UTC Background: LWN, Leaping seconds and looping servers (Jonathan Corbet, 2 July 2012) POSIX.1-2024, Seconds Since the Epoch Google Public NTP, Leap Smear Engineering at Meta, It's time to leave the leap second in the past (2022) AWS EC2 User Guide, leap seconds The Register, Leap second bug cripples Linux servers at airlines, Reddit, LinkedIn (2 July 2012) Music and sound effects: Freesound, Damiano Caudullo, YouTube Audio Library. Tracks: 2018-04-18 2.wav (Doctor_Dreamchip), Decision (The Tower of Light), Red Shift, Rapid Unscheduled Disassembly and Event Horizon (The Grey Room / Density & Time), Under The Rug (Density & Time), When the Ash Settles (The Westerlies), Minor Arp Simple (mariannadeoliveira). Narrated with an AI clone of my own voice. Spotted a mistake? Tell me in the comments, with a source if you have one. Corrections get pinned. #linux #softwareengineering #programming