Перейти к содержимому

How to Design a Multi-Hop Medallion Architecture on Azure | ADF + ADLS Gen2 | Part 1

Cloud BI Academy

0:00 / 0:00

How to Design a Multi-Hop Medallion Architecture on Azure | ADF + ADLS Gen2 | Part 1

223 просмотра · 3 недели назад
Cloud BI Academy
272 подписчика
223 просмотра · 3 недели назад
How would you design a multi-hop Medallion Architecture for a large-scale data platform? In this Azure Data Engineering series, we build a practical end-to-end data platform using Azure Data Factory, ADLS Gen2, Azure Databricks, Apache Spark, Delta Lake and Power BI. In Part 1, we focus on designing the foundation of the architecture using Azure Data Factory and Azure Data Lake Storage Gen2. In this video, you will learn how to design the architecture, configure ADLS Gen2, create the Landing/Raw layer, configure Azure Data Factory, create Linked Services and datasets, connect ADF with ADLS Gen2, and build data ingestion pipelines. The architecture we are building is: Source Systems → Azure Data Factory → ADLS Gen2 → Bronze → Silver → Gold → Power BI This series focuses not only on individual Azure services, but on how these components work together to build a scalable and production-oriented data engineering platform. 🚀 COMING NEXT — PART 2 In Part 2, we move into Azure Databricks and Apache Spark to implement the processing layer and build the Bronze → Silver → Gold transformation flow. Topics will include: • Azure Databricks & Apache Spark • PySpark transformations • Bronze, Silver & Gold layers • Data cleansing and validation • Deduplication • Business transformations • Delta Lake • Data quality and optimization 🔜 PART 3 We will take the solution further with Delta Lake, optimization, production architecture and Power BI analytics. If you are interested in Azure Data Engineering, ADF, ADLS Gen2, Databricks, PySpark, Delta Lake, SQL, SSIS and modern data platforms, subscribe to Cloud BI Academy and follow the complete series. ‪@CloudBIAcademy‬ 💬 Want Part 2? Comment "PART 2" below. #Azure #AzureDataEngineering #MedallionArchitecture #AzureDataFactory #ADLSGen2 #AzureDatabricks #PySpark #DeltaLake #DataEngineering #MicrosoftAzure #DataArchitecture #ApacheSpark #PowerBI #Lakehouse