Orchestrating a Machine Learning Pipeline with Microsoft Fabric | Abby Kaplan (SLCC)

Abby Kaplan, Data Scientist at Salt Lake Community College, shows how her team took predictive ML models that had no reliable way to run on a schedule and built a Microsoft Fabric process around them. The pipeline pulls data from an on-prem warehouse, shapes it, refits the model only when new data arrives, generates batch predictions, and puts the predictions and model metrics in front of end users. She builds it live with a toy version of SLCC's "course fill" model, which predicts which class sections will fill up and which will be underenrolled. What's covered: • Data pipelines that copy on-prem warehouse data into a Lakehouse • Notebooks and custom environments for data prep • Fitting a scikit-learn model, tracking it in MLflow, and saving artifacts to the Lakehouse • An If Condition that refits the model only when new labeled data exists • Passing model IDs between activities with parameter cells and pipeline variables • Refreshing the SQL analytics endpoint before querying it • Surfacing predictions in a dashboard • Q&A: batch scoring vs. endpoints, AutoML vs. transparency, monitoring for feature drift, and where Fabric ML could improve Recorded October 15, 2026 at the Microsoft Fabric & Data Engineering Group, Salt Lake City (online). Chapters 0:00 Intro 1:15 The problem: ML models with no way to run them 4:10 Requirements checklist 5:14 Use case: predicting which classes fill up 6:44 Pipeline: copy on-prem warehouse data to a Lakehouse 11:31 Notebooks to prep the data 15:16 Fit the model with scikit-learn and MLflow 21:12 Only refit when there is new data 25:08 If Condition on notebook and script outputs 27:34 Step forward a year and fit model version 2 29:19 Parameter cells to pass model IDs 32:22 Refresh the SQL analytics endpoint first 33:50 Batch predictions with a pipeline variable 39:04 Design choices: why keep it modular 40:52 Predictions in a dashboard 41:49 Q&A 52:08 Wrap-up and what's next Join the group (free) for upcoming events: Meetup: https://www.meetup.com/microsoft-fabr... LinkedIn group:   / 17113088   Fabric Community: https://community.fabric.microsoft.co... Hosted by Iurii Iurchenko and Nathan Giullian. #MicrosoftFabric #MachineLearning #DataScience