Real-Time Data Ingestion with Debezium, Kafka & HDFS | Final Data Engineering Project Demo

🚀 Data Engineering (CDC Pipeline Demo) This demo showcases a complete Change Data Capture (CDC) pipeline using: Debezium PostgreSQL Source Connector *Apache Kafka* (Zookeeper, Kafka Broker) *Kafka Connect* *HDFS Sink Connector* *Hadoop HDFS (NameNode + DataNode)* 💡 *Goal:* Capture real-time changes in a PostgreSQL table (`crypto_prices`) and stream them through Kafka into an HDFS data lake. 🎥 *Demo Steps Covered:* 1. Docker Compose setup of Kafka, Zookeeper, PostgreSQL, Connect, and Hadoop 2. Debezium CDC configuration 3. Kafka topic stream verification 4. HDFS Sink Connector setup 5. Data written into HDFS (`/data/lake/`) 6. Troubleshooting & validation via Kafka CLI and HDFS shell 📦 *Technologies Used:* Docker & Docker Compose Apache Kafka 7.0.1 Kafka Connect Debezium 2.1.1.Final Confluent HDFS Sink Connector Hadoop 3.3.1 PostgreSQL 14 🧠 *Keywords:* Kafka Connect, Debezium, HDFS Sink, Data Engineering Project, Real-Time Ingestion, PostgreSQL to HDFS, Dockerized Kafka Pipeline, CDC Architecture 📁 Project files: https://github.com/yassineMtg/enterprise-d... 👍 Like • 💬 Comment • 🔔 Subscribe