Deletion Vectors in Delta Lake: How Soft Deletes Really Work

In a file-based lakehouse, deleting a single row could mean rewriting an entire Parquet file full of untouched records. Deletion Vectors in Delta Lake change that — they mark a row as logically removed without immediately rewriting the file. In this BricksNotes explainer, Krithi walks through how Deletion Vectors work behind the scenes, why they matter for data engineering, and — just as importantly — when and how you actually clean up the deleted data. We cover the key distinction most people miss: a row can be logically deleted but still physically stored, and that difference drives everything about table maintenance. You’ll see where Deletion Vectors shine (frequent small deletes, updates, merges, and CDC pipelines), where they help less (append-only workloads), and the maintenance operations you need to keep straight — DELETE (changes what the table shows), OPTIMIZE (compacts files), REORG TABLE … APPLY PURGE (rewrites files to physically remove soft-deleted rows), and VACUUM (removes obsolete files after the retention period) — plus why shortening retention can quietly break time travel. Chapters 0:00 — One DELETE, a whole file rewritten 0:31 — What are Deletion Vectors? 0:49 — Why Parquet makes row edits expensive 1:49 — How Deletion Vectors change it 1:59 — The book-page analogy 2:25 — What it is: the vector and the transaction log 3:02 — Logically deleted vs. physically erased 4:28 — Table maintenance: DELETE, OPTIMIZE, PURGE 5:26 — VACUUM and the delete sequence 5:45 — Don’t confuse DELETE / PURGE / VACUUM 6:03 — When to use them (and when not) 7:08 — The one idea to remember Follow BricksNotes for practical Data Engineering, Databricks, and Delta Lake concepts — keep learning, keep building, keep growing. 🌐 bricksnotes.com Educational content from BricksNotes. “Databricks” and “Delta Lake” are trademarks of Databricks, Inc.; this video is independent commentary and is not affiliated with or endorsed by Databricks. Feature behavior and syntax change — always check the current Delta Lake / Databricks documentation before applying this in production. #DeltaLake #DataEngineering #Databricks #vectorsindeltalake #datascience #data #dataanalytics #dataanalysis