The FACTS Benchmark Explained: How Google DeepMind Tests Whether LLMs Tell the Truth

About This Episode Most AI benchmarks ask whether a model is smart. FACTS, from Google DeepMind and Kaggle, asks whether it tells the truth. In this educational episode, Sunil, Sajjad and Sam walk through the four FACTS benchmarks (parametric, search, multimodal and grounding), explain how FACTS Grounding uses long expert-written passages and two LLM judges that must agree, and show why even state-of-the-art models score in the low-to-mid 80s. They also cover where the benchmark falls short, from judge bias to rewarding refusals, why smaller models can be more truthful than large reasoning models, and where domain-specific benchmarks like legal are heading. Episode Summary Responding to listener feedback asking for broader, more educational episodes, Sunil opens a discussion of AI benchmarks, starting with FACTS, the benchmark DaaX has been competing on. Sajjad places it in a long line of language benchmarks, from Stanford's SQuAD through GLUE, MNLI and SNLI to Stanford's HELM, before describing the FACTS suite from Google DeepMind and Kaggle. It has four parts: FACTS Multimodal, for questions about images; FACTS Parametric, a closed-book quiz of what a model already knows; FACTS Search, which tests whether a model can find and report the right answer using web search; and FACTS Grounding, which asks a model to answer questions using only a supplied document. Sajjad explains why FACTS Grounding is so demanding. Expert-written passages, much like SAT reading passages and up to about 32,000 tokens long, span domains such as legal, financial, technical and medical. Of roughly 920 questions, 860 are public. Most require long-form answers that combine several facts from across the passage, follow per-question instructions on format and length, and preserve the source's tone and intent; calling a knife "the best" when the source says "one of the better knives" is penalized. In version 2, two mandated judges, Gemini Flash and GPT-5, must both agree that an answer is eligible and factually accurate. Even the leading model scores around 85%. Sam gives a non-technical framing: a brilliant lawyer who makes up one fact in roughly one brief in three, and four exams told as an analyst's working day. Examples show why partial credit doesn't exist, from a locomotive dated 1939 instead of 1949 to a passage implying nine items without ever stating the number. Sajjad explains that DaaX separates language skills from numerical reasoning and knowledge retrieval, and verifies every claim, regenerating claims or whole answers when needed. The hosts also look at the benchmark's limits. Sam notes that LLM judges agree with human raters only about three-quarters of the time, that self-preference bias is reduced but not eliminated, and that grounding rewards a correct refusal rather than knowing more. Sajjad confirms that the FACTS Grounding leaderboard is effectively a RAG test of each model. The conversation turns to domain-specific benchmarks, especially legal, where DaaX could use legal vocabularies and ontologies to guide answers. Sam closes with a surprising finding: smaller, cheaper models have at times topped FACTS rankings, because being smart and being truthful are different skills, and large reasoning models trained to be confident and complete can be too good for their own good. Full transcript at https://daax.ai/podcast/episode-30-fa... Chapters ● 0:00 — Episode 30 and a recommendation: Signalful ● 0:48 — A broader, educational format: AI benchmarks ● 1:51 — What is FACTS? ● 2:39 — A short history of language benchmarks: SQuAD, GLUE, HELM ● 3:29 — Google DeepMind and Kaggle's FACTS suite ● 3:58 — The four FACTS benchmarks ● 5:15 — How FACTS Grounding works ● 6:51 — Two judges that must agree ● 7:55 — What makes it grueling ● 9:50 — How the judges decide ● 10:20 — Mandated judges: Gemini Flash and GPT-5 ● 11:06 — Public vs. private datasets ● 11:49 — Domain knowledge, or reading comprehension? ● 13:01 — Long-form answers and instruction following ● 14:08 — Matching the source's tone and intent ● 15:22 — FACTS Grounding v1 vs. v2 ● 15:55 — The leaderboard: no model near 90% ● 16:38 — The brilliant lawyer who makes things up ● 17:41 — Does it tell the truth? Four exams as a day in the life ● 19:07 — Numerical and logical reasoning traps ● 20:11 — Non-determinism, and why DaaX verifies every claim ● 21:08 — One wrong date, zero credit ● 22:06 — Eight items on the table, one in the pocket ● 23:10 — Sorting kitchen knives and the cognitive surface ● 24:26 — Parametric scoring: accuracy, hedging and F1 ● 25:10 — Where the benchmark falls short: LLM judges and bias ● 26:00 — Grounding rewards refusals ● 26:34 — FACTS Grounding is effectively a RAG test ● 27:05 — Domain-specific benchmarks: legal ● 28:44 — Legal vocabularies, ontologies and Harvey ● 29:39 — Small models winning on truthfulness ● 31:35 — Wrap-up