Databricks Job Taking Too Long? Fix It in Query Profile (No Spark UI Needed)

our Databricks job on serverless compute is running slow, and you can't find the Spark UI to figure out why — because it doesn't exist on serverless. In this video, I diagnose a genuinely slow job (20 million rows, 4+ minutes) using only Query History and Query Profile, then fix it live. You'll see: ✅ Why the Spark UI isn't available on serverless compute (and what replaces it) ✅ How to read the Scan node metrics panel — files read, files pruned, cloud storage request duration ✅ How to spot the "small file problem" from real numbers (400 files, 0 pruned, 39s of pure I/O wait) ✅ The exact fix: OPTIMIZE + Z-ORDER, re-run, and compare ✅ Real before/after numbers: 4:01 down to 0:58 — same query, same logic, zero code changes If you're running Databricks on serverless compute and have never opened Query Profile, this is the first place you should look before touching cluster size or code. If this helped, like and subscribe — next up, we dig into the Aggregate and Sort nodes in Query Profile. #Databricks #DataEngineering #ApacheSpark