8-Year Data Scientist Interview: The Migration Mistakes That Break Production

What happens when your new ML system is better — but replacing the old system could break the business? Welcome to Episode 8 of the Kritya AI Technical Interview Podcast — a Senior Data Scientist interview calibrated to 8 years of real-world experience. At this level, the interview isn't just about building models — it's about understanding what happens when models live inside real production systems: legacy pipelines, downstream consumers, data contracts, feature stores, and teams that quietly built assumptions around systems nobody documented. This episode focuses on one of the hardest parts of senior Data Science Engineering: MIGRATION. How do you build the new system while the old one still serves real traffic, prove the new pipeline is equivalent, discover undocumented dependencies, and know when you finally have enough evidence to shut the legacy system down? 🔥 WHAT WE COVER • A/B testing: p = 0.03 — stopping rules vs practical significance • Python migration-validation interface: score differences, NULL bands, floating-point edge cases • Job completion vs output quality — observability for a "green but wrong" pipeline • Versioned metrics-layer design across four years of undocumented SQL views • Lineage as a practical tool for blast-radius estimation • Causal inference: did the discount actually cause retention, or is it selection bias? • Snowflake vs a dedicated feature store at 60,000 accounts • LLM security: what data is safe to send into a prompt 🔥 MODEL MIGRATION: THE MODEL ISN'T THE FIRST THING YOU CHECK A three-year-old churn model is being replaced by a gradient-boosted model with better offline AUC. The obvious question is "is the new model better?" — but the senior-level questions are different. Before switching traffic: feature parity, score distribution alignment, shadow-mode validation, downstream contracts, and existing intervention thresholds. A model can have better offline metrics and still create a worse business outcome — if a CRM threshold stays at 0.6 while the new model triples positive predictions, the downstream intervention queue can explode overnight. That's the difference between model performance and real-world impact. 🏗️ THE LEGACY ML MIGRATION PROBLEM A customer health scoring system has run for five years: notebook → cron → EC2 → Postgres → 12 downstream consumers. The company wants Airflow, versioned models, and monitoring — but the old system can't simply be switched off. The old table may hide undocumented assumptions about column names, NULL handling, and downstream thresholds, so the new system can technically reproduce the same logic while still breaking consumers. The fix: run both systems in parallel, writing the new pipeline to a shadow table while an automated job compares outputs every run — separating "reproduce the pipeline" from "change the model" into two distinct steps. 🚨 REAL MIGRATION INCIDENT: 40 ACCOUNTS MISCLASSIFIED Both systems are running. Customer Success reports that around 40 accounts marked "at risk" had already been successfully recovered. The first hypothesis is data freshness, but timestamps look correct. The real cause: a feature called "days since last success" existed in the legacy notebook but was missing from the new feature-store specification — the new pipeline was built from documentation, and the documentation was incomplete. The old source code was the real ground truth. 💀 WHEN DO YOU FINALLY KILL THE LEGACY SYSTEM? Maybe the most important question of the episode. Do you decommission after six months, after a year, after 30 clean days? The candidate argues for criteria over an arbitrary timeline: multiple clean migration runs, every consumer accounted for, a documented owner, the new system surviving a real incident, and a tested rollback path. Shutting down the old system is itself a production change. 🧠 THE 8-YEAR DATA SCIENTIST MINDSET A 4-year Data Scientist focuses on how to build the new system. An 8-year Data Scientist starts asking: who depends on the old system, what breaks if the systems disagree, how do we roll back, and when can we safely turn it off? The hard part isn't always building the new architecture — sometimes it's keeping the lights on while you build it. 🎙️ THE 1–20 YEARS SERIES 1 Year → Fundamentals 2 Years → Production Reasoning 3 Years → Security & Production 4 Years → Scaling & Performance 5 Years → Data Quality & Evaluation 6 Years → Reliability & Incident Response 7 Years → Architecture & Decision Making 8 Years → Migration, Legacy Integration & Data Quality If you're preparing for a Senior Data Scientist, Data Scientist Engineer, Machine Learning Engineer, MLOps, or ML System Design interview, don't just listen for the answers — listen for the reasoning. Comment below: would you shut down the legacy system, or keep both running? What evidence would you require first? Subscribe to Kritya AI Labs and follow the complete 1–20 Years journey. #DataScience #DataScientist #MLOps