It Looked Right — The Black Box Precedent, Track 12

Fluent and correct look identical, right up until they don't — usually in the novel situation nobody trained for. The cheap, fast test everyone runs before deploying has no mechanism for catching that difference in advance. Track 12 of "The Black Box Precedent." Lyrics: [Verse 1] There's a test we run on everything before we let it in the door Does it look like something trustworthy, does it look like something sure Does the answer sound like wisdom, does the letter read as right Does the diagnosis land the way a steady hand would write That test is fast, that test is cheap, and everybody runs it first Before the deeper question ever gets a chance to be rehearsed And it works just fine on Tuesday, works just fine a thousand times Right up until the moment when the fluent thing and the true thing don't align [Chorus] It looked right It looked right, right up until it wasn't true It looked right That's the test that's fooling me and you Fluent isn't the same as sound Plausible isn't the same as real It looked right That's the danger underneath the wheel [Verse 2] The system never lies to you the way a liar might It doesn't know the difference between the wrong thing and the right It only knows the shape of what an answer's meant to be And it builds you one that's fluent as the tide comes off the sea So the ones who check the output, checking fast and checking light Can't see the seam between the reasoned and the merely built to sound polite Two things that could not be more different, standing in the same disguise Fluency wearing wisdom's clothing, walking past us, telling lies [Chorus] It looked right It looked right, right up until it wasn't true It looked right That's the test that's fooling me and you Fluent isn't the same as sound Plausible isn't the same as real It looked right That's the danger underneath the wheel [Verse 3] And here's the cruelest part of it, the part that keeps me up at night It isn't in the easy places where the difference comes to light It's in the case nobody trained for, the edge outside the learned terrain The novel situation where the pattern starts to strain That's exactly where the fluency and truth begin to pull apart That's exactly where the stakes run highest, closest to the heart And that's exactly where the fast cheap test was never built to see The moment when the answer's confident but wrong as wrong can be [Verse 4] A new employee's failure shows up bounded, shows up slow Probation, narrow scope, a supervisor watching close, you know The system's failure hides inside the fluency that made it seem Trustworthy enough to hand the keys before we'd built the scheme Of checking, narrowing, and earning what we should have made it earn Deployed at speed no onboarding process ever built to learn The competence and the incompetence look exactly the same Right up until the fluent voice is speaking into the flame [Bridge] Fluent and correct Look the same from where we stand Until the edge, until the case No one accounted for in the plan [Verse 5] So here's the warning wrapped inside the wonder I began with here It isn't that the guessing's evil, isn't cause for constant fear It's that the only test we're running is the cheapest one we've got And cheap tests catch the easy failures, miss the costly ones a lot Ask it for the reasoning before, not the reasoning as decor Ask it what the narrow scope is, ask who's checking, ask what's for The consequence, ask what would show if something started drifting wrong That's the only way that fluent ever gets to holding true for long [Final Chorus] It looked right It looked right, right up until it wasn't true It looked right That's the test that's fooling me and you Fluent isn't the same as sound Plausible isn't the same as real It looked right That's the danger underneath the wheel Underneath the wheel [Outro] It looked right Until the moment that it broke It looked right That's the danger we still hold