You have more data than you think, and need less than you realize | Douglas Hubbard | ep 7

This is an audio interview that predates ChatGPT, copilots, and any AI budget anyone had to defend to a board. That is the reason to play it now rather than a caveat about it: nothing in it needed updating. A decade ago in 2016, I interviewed Douglas Hubbard for a podcast, now recut with a new introduction and close. He took my answer to measuring collaboration apart in about ninety seconds, and ten years later that is still the first answer people give. It is now on dashboards with the word AI at the top of them. Douglas Hubbard invented Applied Information Economics and wrote "How to Measure Anything: Finding the Value of Intangibles in Business" (the book people hand you when you say something cannot be measured). He has measured drought resistance in the Horn of Africa, the effect of pesticide regulation on endangered species, and the economic impact of restoring a desert in Inner Mongolia. Signals: Doug: there is no such thing as a statistically significant sample size. Measurement is uncertainty reduction, not an exact number. Doug: you have more data than you think, and you need less than you think. The more uncertain you are, the more the first few observations buy you. Doug: define the thing by the decision it changes. Why do you care, what would you do differently, and how far off would the answer have to be before you acted. Doug: you have already priced the thing you refuse to price. Your approvals and rejections imply the number, and anyone with a little algebra can recover it. Sam: concept, object and method are the three ways an AI value case falls over. Demanding one exact ROI figure, measuring "productivity" without saying what you would see more of, and assuming you need the whole population before you can start. Subtractions: Doug: stop treating measurement as a query against data you already hold. The scientific revolution did not wait for a populated database. Doug: stop latching onto the first observable thing. Message volume is not collaboration; it is what is easy to count. Doug: stop trusting your own ninety percent. People who say 90% are right about 60 to 65% of the time, and calibration is trainable in half a day. Doug: stop treating the refusal to quantify as the ethical position. All it buys is a number that comes out different every time. Sam: stop running pilots whose result will not change a decision either way. If no outcome flips the call, it is not a measurement. The tape is from Doable Change, and the archive is public at https://snapsynapse.com/DoableChange/ Doug is not predicting anything here, and this is not a "look how prescient" post. He is describing how measurement has always worked. The striking part is that it needed no updating. Links from this episode: Douglas Hubbard: https://howtomeasureanything.com Episode page and full transcript: https://sigsub.show/episodes/ep-007/ Issue 64 of the newsletter, the written companion to this episode: https://sigsub.show/newsletter/measur... New here? Follow the show on Apple Podcasts or Spotify and future episodes arrive without you doing anything else. Livestream Wednesdays. Episode Fridays. Newsletter Sundays at https://sigsub.substack.com SigSub.show Chapters: 00:00 There is no statistically significant sample size 00:28 Why a 2016 tape, and why now 03:10 How would you measure collaboration? 04:01 Count the messages: the answer everybody gives 05:22 How Doug started measuring intangibles 07:31 What "statistically significant" actually means 08:38 Define it by the decision it changes 10:06 More data than you think, less than you need 13:35 The more uncertain you are, the more the first look buys 14:34 The speed of light, and the spot sample 17:59 Control groups, placebos, and "is this team just better?" 20:08 The value of a human life 22:08 Recall bias, and why you feel more risk-averse 23:09 Your ninety percent is really sixty-five 24:49 Three illusions: concept, object, method 26:34 How those three break an AI business case 27:48 Stop running pilots that change no decision 29:00 You have already priced a human life