Can Doctors Trust Black-Box Healthcare AI?

Serelora co-founders Spencer Wozniak and Luis Cisneros discuss what trustworthy clinical AI should look like, prompted by a dispute involving OpenEvidence and Doximity. Drawing on clinical experience and product-building lessons, they argue for transparent reasoning, traceable sources, physician oversight, and AI that augments rather than replaces medical judgment. Reach out to us at info@serelora.com! Keywords clinical AI, healthcare AI, medical AI transparency, explainable AI, clinical decision support, AI scribes, physician liability, evidence-based medicine Key Topics Spencer Wozniak and Luis Cisneros introduce Serelora and explain their clinical backgrounds and motivation for building healthcare AI. The OpenEvidence-Doximity dispute raises questions about data privacy, competitive intelligence, and whether clinical AI systems should protect their reasoning. Luis argues that clinical AI should expose its reasoning and the evidence behind recommendations so physicians can assess risk drivers and trust outputs. They discuss how clinical differences across specialties, institutions, and populations make a single centralized reasoning approach inadequate. Source links, patient documentation, benchmarks, and physician feedback can help reveal hallucinations and improve clinical systems. They distinguish qualitative clinical reasoning from quantitative analysis, arguing that calculations should use validated tools rather than rely on language models over unstructured records. AI scribes should preserve physician judgment: reviewable notes, addenda, edits, transcripts, and audit trails can support learning and legal traceability. A physician's experience, intuition, and ability to notice what a patient does not say remain essential; AI is best treated as an augmenting tool. A failure during an 80-minute scribe session exposed a server memory limit; processing transcripts in chunks fixed the issue. Patient skepticism about outreach messages prompted the founders to consider testing clearer communications and alternative channels. Takeaways “We need to glass box it.” “The AI is an extension of you.” “The doctor is gonna be able to capture what the patient didn't say.” “They are tools to augment our capacity.” “AI is nothing more than a machine to translate your ideas.” “You still need to review it.” “The only way to get better is to be able to have a system that provides you that.” “There’s not one school of thought that exists in medicine.” “The real question is, how do you make it useful” “We want to build and learn.” "There's an art to it too." Chapters 00:00 Why two healthcare builders started talking clinical AI 01:22 The OpenEvidence-Doximity dispute and data privacy concerns 04:12 What counts as a moat in clinical AI? 07:43 Why black-box reasoning undermines physician trust 12:01 The case for decentralized, transparent clinical reasoning 15:21 What doctors should look for in trustworthy AI 19:02 Trace recommendations to patient records and evidence 22:26 Benchmarks, bias, and hallucinated clinical data 24:20 Why unstructured records cause drift 26:24 Clinical intuition and the limits of AI replacement 32:16 Scribes, cognitive surrender, and physician review 34:08 Addenda and audit trails for AI-generated notes 39:09 AI as a translator of clinician thinking 44:11 Who is responsible for reviewing AI output? 45:22 Qualitative judgment versus quantitative risk analysis 49:12 Fixing the 80-minute scribe session failure 52:19 Patient trust, outreach messages, and adoption 54:20 Why the founders want to share the building process 57:53 What’s next for the clinical AI conversation