Kafka Connect - 4.2. Offset Storage (File vs Internal Topics)

https://centralmesh.io Chapter 4: Deep Dive into Connect Internals In this section, we focus on how Kafka Connect stores offsets so connectors can resume reliably after restarts or failures. Standalone workers use a local file, while distributed workers use a shared, replicated Kafka topic that supports failover and horizontal scaling. You’ll learn: How source and sink offsets record connector processing progress. How standalone mode persists offsets in a local file. How distributed mode stores offsets in the connect-offsets topic. Why replication makes offset state durable across worker failures. How log compaction retains the latest offset for each key. By the end of this section, you’ll be able to: Explain how offsets enable resumable data transfer. Compare file-based and Kafka-backed offset storage. Choose an offset storage model for development or production. Describe how replication and compaction protect distributed offset state.