Перейти к содержимому

MinIO S3 Buckets for ML | DeployBytes 5 - Line by Line

Dr. Sandeep Grover

0:00 / 0:00

MinIO S3 Buckets for ML | DeployBytes 5 - Line by Line

33 просмотра · 1 месяц назад
Dr. Sandeep Grover
81 подписчик
33 просмотра · 1 месяц назад
Why this series exists: This video is part of a series created to make central concepts easier to understand and hold onto. It was built with accessibility in mind, particularly for learners with learning differences, so the pace and style are kept simple and supportive. There may be a few mispronunciations in the spoken narration, the occasional formatting error, and sometimes a small mistake in the content. None of these take away from the purpose of the series, which is simply to help the learning stick. Real MLOps, step by step. Part 5 adds MinIO, a self-hosted S3-compatible object store, and creates the two buckets the rest of the pipeline depends on - one for MLflow artifacts, one for DVC data. A single init container makes both, so the stack comes up correct from a cold start every time. Recommended Learning Order: The Big Picture (visual overview, 6 min):    • MinIO S3 Buckets for ML | DeployBytes 5 - ...   Line by Line (full code walkthrough, 27 min) - you are here Why this series exists: This video is part of a series created to make central concepts easier to understand and hold onto. It was built with accessibility in mind, particularly for learners with learning differences, so the pace and style are kept simple and supportive. There may be a few mispronunciations in the spoken narration, and occasionally a small mistake in the content. These do not take away from the purpose of the series, which is simply to help the learning stick. Work with me:   / sandeep-grover-b3192a16   Productionizing a real 7-encoder multimodal Rakuten classifier (84,916 products, weighted-F1 0.9147), from git to Kubernetes. Chapters: 0:00 Step 5 - MinIO buckets and DVC_BUCKET 1:41 The shape of this step 3:31 Why MinIO comes before MLflow 4:27 What is MinIO, and what is a bucket? 5:46 Why exactly two buckets, and the plan 6:57 The config that does it: minio and minio-init 9:19 The minio service the init container waits on 11:24 The fix: minio-init makes both buckets in one line 15:06 Bring the stack up again: both buckets are simply there 17:34 Look inside: the console, and a throwaway mc listing 20:37 The one code change, and the commit 24:09 Lessons worth keeping 25:42 Takeaway, and on to Step 6 Series stack (all 12 episodes): Python, cookiecutter-data-science, Git/GitHub, Docker, Docker Compose, PostgreSQL, MinIO (S3), MLflow, DVC, FastAPI, Prometheus, pytest, Nginx, Apache Airflow, Kubernetes, Argo Rollouts, PyTorch, Hugging Face Transformers. #MLOps #MachineLearning #MinIO #S3 #ObjectStorage #Docker #DVC #MLflow #Python #DataEngineering #ModelDeployment #DataScience #DevOps