WALMART Data Engineering End-To-End Project [Airflow + DBT + Databricks]

Thanks to GHOST for sponsoring this video. Try GHOST now - https://b.link/ansh-gost Data Engineering End-To-End Project | Databricks | Airflow | DBT | AWS | End-To-End Data Engineer Project What you will learn? This is a complete End-To-End Data Engineering project which included all the modern tech stack such as Apache Airflow, DBT, Databricks, AWS and covers scenarios like Incremental Ingestion, Metadata Driven Pipelines, Slowly Changing Dimensions, STAR Schema and much more. This is a beginner friendly project which will help you learn all the tech stack by doing it. This is perfect learning resource for anyone wants to grow in the Data Engineering space and wants to learn the modern tech stack. Timestamps: 0:00 Introduction 05:07 Data Architecture 15:08 Environment Setup 24:13 Source Database OLTP 32:10 Adding Data Using Agent 52:35 Lakehouse Architecture 1:00:57 Databricks Overview 1:07:50 Incremental Data Ingestion into Databricks 1:18:34 DBT Overview 1:31:37 DBT Setup 1:48:15 Query Databricks Using DBT 2:03:05 Silver Technical Layer 2:11:44 Incremental Models in DBT 2:45:20 Silver Business Layer 2:53:46 Metadata Driven Ingestion Pipeline 3:11:56 Test Cases With DBT 3:27:32 Ephemeral Models in DBT 3:50:56 DBT Snapshots 4:00:00 Slowly Changing Dimensions 4:24:37 Apache Airflow Overview 4:34:25 Airflow With Docker 4:45:21 Orchestration With Airflow DAGs 5:11:59 Trigger DBT Models With DAG 5:29:37 Trigger Databricks Pipelines With Airflow 5:54:32 AWS Overview 5:58:47 AWS S3 6:01:23 Data Ingestion From S3 Data Lake 6:12:59 Deploy To GitHub UDEMY Courses: Apache Airflow Bootcamp https://www.udemy.com/course/apache-a... Apache Spark 4.0 Bootcamp https://www.udemy.com/course/databric... Databricks Data Engineer https://www.udemy.com/course/databric... Fabric Data Engineer https://www.udemy.com/course/databric... Watch My Previous Videos Spark Pipelines Full Course    • Spark Declarative Pipelines Full Course (N...   Claude Code Full Course    • CLAUDE CODE Full Course For Beginners (DAT...   Databricks Full Course    • Databricks Tutorial (From Zero to Hero) | ...   DBT Full Course    • [Data Build Tool] DBT - The Ultimate Guide...   ADF Full Course    • Azure Data Factory Full Course (From Begin...   GitHub Repo https://github.com/anshlambagit/Walma... ======================================= Join this channel to SUPPORT MY HARDWORK:    / @anshlambajsr   ======================================= Connect with ME LinkedIn -   / ansh-lamba-793681184   Telegram - https://t.me/anshlambadatafam ========================= For COLLABORATION 👇 anshlambacollab@gmail.com ========================= ⭐Hashtags⭐