Metadata Driven processing with Notebooks in Microsoft Fabric

Metadata-driven processing with Spark notebooks in Microsoft Fabric. I’ve covered metadata-driven patterns before with RunMultiple and with Fabric pipelines, this time it’s notebooks. I set up a simple demo: three source tables (customers, products, orders) and a metadata table describing what to load and whether each table is active. Then I build an orchestrator notebook that reads the metadata, constructs a DAG of load activities, and fans them out to a generic child notebook with concurrency control, so adding or changing what gets loaded is a metadata edit, not a code change. I also show what happens when you flip a table to inactive and drop concurrency to 1: the orchestrator picks it up automatically without touching any notebook code. This is the small-scale version of a pattern that matters a lot more at 300+ tables, where duplicating notebook logic per table just isn’t viable. A follow-up video will cover load groups and dependency-based concurrency (some notebooks running in parallel, others waiting on a batch to finish first). Chapters: 00:00 — Intro: metadata-driven processing with notebooks 00:27 — Setup notebook: creating the source tables 01:22 — Creating the metadata table 03:17 — The orchestrator notebook explained 04:25 — Building the DAG from metadata 05:36 — Visualizing the DAG structure 05:54 — The child notebook and its parameters 07:50 — Running the DAG (and fixing a couple of typos) 09:11 — Checking per-activity results 10:03 — The real use case: changing behavior via metadata, not code 10:39 — Deactivating a table and lowering concurrency 11:56 — Confirming the updated run (two tables, sequential) 12:45 — Recap and teaser: load groups and dependency-based concurrency