15-Year Data Scientist Interview: What Would You Build vs Buy?

What does 15 years of Data Science experience actually look like when the job is no longer just building models? In this 15-Year Data Scientist Engineering Interview, we go into Principal-level territory: experimentation, causal inference, build-vs-buy decisions, metrics governance, production incidents, training-serving skew, platform ownership, LLM-assisted analytics, security, scaling, and the judgment required to make technical decisions that still work years later. This is not a generic list of interview questions. The discussion is built around realistic situations where the hardest part is understanding the trade-off, the failure mode, and the organizational consequences. 🔥 IN THIS EPISODE • A/B testing with heavy-tailed revenue metrics • Heterogeneous treatment effects and targeted rollouts • Causal inference when the target population changes • External validity and population drift • Build vs buy for an experimentation platform • Hybrid architecture and reversibility as a design constraint • Experiment guardrails and sequential testing • Bonferroni vs CUSUM vs always-valid inference • Centralized metrics layer design and organizational buy-in • dbt semantic layer vs purpose-built metrics platforms • Silent data-quality failures in production feature pipelines • Production incident caused by a pandas dependency update • Experimentation platform architecture at scale • Assignment, Kafka ingestion and computation layers • Production Python code review • SQL injection and why parameterized queries matter • Composite cache keys and silent correctness bugs • Organizational ownership transfer from Data Science to ML Engineering • Evaluating LLM-assisted analytics and natural-language-to-SQL tools • SQL correctness vs result correctness • Why vendor benchmark accuracy is not enough • Training-serving skew after a data-source migration • Feature registry design and preventing pipeline drift • Pandas-to-Spark migration and null/forward-fill differences • Scaling a churn model from 1 million to 30 million users • Distributed feature computation and serving architecture 🚨 THE PRODUCTION INCIDENT A recommendation system looks healthy. The model retrained successfully. Training metrics passed. The serving layer shows no obvious problem. Then customers start seeing items they have already purchased. The investigation reveals a classic training-serving skew problem: the training pipeline moved to a new purchase-history table, while serving continued reading from a legacy table that had stopped updating. The interview walks through the recovery, temporary model rollback, validation, and the longer-term architectural fix: a feature registry that keeps training and serving definitions aligned. 📈 THE 30X SCALING CHALLENGE A churn model needs to grow from 1 million monthly predictions to 30 million. What breaks first? The discussion covers batch computation, pandas limitations, Postgres write pressure, distributed processing, PySpark/pandas-on-Spark, Polars, BigQuery/Snowflake, Redis/DynamoDB, and the importance of validating numerical equivalence before migrating production feature pipelines. 🤖 EVALUATING LLM-ASSISTED ANALYTICS A vendor claims a 40% reduction in analyst time. But what happens when the tool generates plausible SQL that is subtly wrong? The interview proposes evaluating: • SQL correctness • Result correctness • Whether analysts can detect silent errors • Performance on real internal queries The core question is not simply “Can the LLM write SQL?” It is: “Can we trust the answer enough to make a business decision?” 🎯 WHO THIS IS FOR • Data Scientists • Senior Data Scientists • Principal Data Scientists • ML Engineers • Data Science Engineers • MLOps Engineers • AI Engineers • Analytics Engineers • Technical Leads • Anyone preparing for senior/principal-level Data Science interviews At 15 years, the interview moves beyond individual model development. The focus becomes judgment: choosing what to build, what to buy, what to keep boring, how to manage reversibility, how to design organizational interfaces, and how to make production systems resilient. 📌 SERIES Kritya AI Technical Interview Podcast Data Scientist Engineering Interview Series — 1 to 20 Years Experience This is the 15-year / Principal-level episode, continuing the progression from technical implementation toward architecture, production ownership, organizational design, and long-term technical bets. 💬 INTERVIEW CHALLENGE Your model's training metrics are healthy. Your serving infrastructure is healthy. But users suddenly receive obviously wrong recommendations. Where do you investigate FIRST? Comment with your investigation sequence — not just the final answer. Subscribe to Kritya AI for realistic Data Science, Machine Learning, AI Engineering, MLOps and system-design interview discussions. #DataScience #DataScientist #MachineLearning #MLOps #DataEngineering