Apache Airflow for ML | DeployBytes 11 - Line by Line

Real MLOps, step by step. Part 11 schedules retraining with Apache Airflow. The rakuten_retrain DAG runs @daily: make_baseline, check_features and run_decay_check feed a branch that either skips the retrain or retrains the head and runs promote_if_better. A second DAG, rakuten_serve_monitor, fires traffic at the load balancer and reads decay from what was actually served. At this commit promote_if_better only prints PROMOTE or HOLD on macro-F1 and moves nothing. The gated promotion that really moves the @champion alias in the MLflow registry lands in the rebuilt repo (commit 21a865e), the fix the Big Picture song for this part is about. Why this series exists: This video is part of a series created to make central concepts easier to understand and hold onto. It was built with accessibility in mind, particularly for learners with learning differences, so the pace and style are kept simple and supportive. There may be a few mispronunciations in the spoken narration, and occasionally a small mistake in the content. These do not take away from the purpose of the series, which is simply to help the learning stick. Work with me:   / sandeep-grover-b3192a16   Productionizing a real 7-encoder multimodal Rakuten classifier (84,916 products, weighted-F1 0.9147), from git to Kubernetes. Chapters: 0:00 Why this series exists 0:29 Step 11 - Airflow and monitoring 1:37 The shape of this step 3:06 Why Airflow, and why it comes last 3:53 What Airflow runs: three services and one database 5:20 The two DAGs: one offline, one on served traffic 6:52 The decay rule that gates the retrain 9:04 Live run 10:15 11a - the Airflow image (docker/airflow/Dockerfile) 11:06 11b - the compose anchor and the three Airflow services 12:55 11c - the decay check (src/monitoring/decay_check.py) 14:11 11d - bring Airflow up, unpause and trigger the retrain DAG 18:14 11e - the served-traffic path: fire at the load balancer, read decay 20:38 11f - commit 22:28 Walkthrough: docker/airflow/Dockerfile 25:13 Walkthrough: docker-compose.yml 39:43 Walkthrough: src/config.py 43:53 Walkthrough: src/monitoring/decay_check.py 1:02:55 Walkthrough: src/monitoring/fire_and_log.py 1:11:33 Walkthrough: src/monitoring/monitor_served.py 1:18:16 Walkthrough: airflow/dags/rakuten_retrain.py 1:34:31 Walkthrough: airflow/dags/rakuten_serve_monitor.py 1:45:19 Reality check: the DAGs start paused, and three errors at this step 1:47:40 Lessons worth keeping 1:49:24 Takeaway: the pipeline is closed Series stack (all 12 episodes): Python, cookiecutter-data-science, Git/GitHub, Docker, Docker Compose, PostgreSQL, MinIO (S3), MLflow, DVC, FastAPI, Prometheus, pytest, Nginx, Apache Airflow, Kubernetes, Argo Rollouts, PyTorch, Hugging Face Transformers. #MLOps #Airflow #ApacheAirflow #Orchestration #Retraining #MLflow #MachineLearning #Python #DevOps #MLEngineering