Postgres CDC to Parquet Tables on S3, with PII Masking | Onehouse OneFlow

Stream a Postgres table into Parquet tables on S3 that stay in sync with every insert, update and delete, with customer emails hashed on the way in. No pipeline code. An online retailer's analysts need revenue by order status from tables that stay current, without ever seeing a customer's email. Normally that means a change-capture tool, a message queue, a streaming job and masking code. Here it's one source and one flow in Onehouse OneFlow. IN THIS VIDEO • Create a Postgres CDC source: the database connection, a replication user, change logs through Amazon MSK (managed Kafka, low latency), and an AWS Glue schema registry. • Create a flow in OneFlow: Mutable write mode so updates and deletes are applied in place, Pipeline Quarantine for bad records, and Data Masking to hash customer_email. • Verify with Apache Spark: after the first snapshot, and after an INSERT, UPDATE and DELETE, revenue by status in the Parquet table matches Postgres to the cent. • Why hash instead of drop: the same email always gives the same hash, so you can still count unique customers and join tables. CHAPTERS (0:00) The problem (0:51) The orders in Postgres (1:11) Creating a Postgres CDC source (2:09) The retailer's source (2:24) Creating a flow in OneFlow (2:49) Mutable vs append-only (3:12) Table config, quarantine and masking (4:15) First snapshot (4:31) Querying the Parquet table with Spark (4:54) INSERT: a new order (5:18) UPDATE: the order ships (5:37) DELETE: the order is cancelled (6:00) Why hash instead of drop (6:25) Recap LEARN MORE ► Onehouse: https://www.onehouse.ai/ CONNECT Subscribe:    / @onehousehq   X:   / onehousehq   Linkedin:   / onehousehq   #Postgres #CDC #DataEngineering