🎄Twelve Days of SMT 🎄 - Day 8: TimestampConverter

The TimestampConverter Single Message Transform lets you work with timestamp fields in Kafka messages. You can convert a string into a native Timestamp type (or Date or Time), as well as Unix epoch - and the same in reverse too. This is really useful to make sure that data ingested into Kafka is correctly stored as a Timestamp (if it is one), and also enables you to write a Timestamp out to a sink connector in a string format that you choose. The TimestampConverter takes three arguments; the name of the field holding the timestamp, the data type to which you want to convert it, and the format of the timestamp to parse (if reading a string and target.type is unix/timestamp/date/time) or write (if you’re writing a string and target.type is string). The timestamp format is based on SimpleDateFormat. "transforms" : "convertTS", "transforms.convertTS.type" : "org.apache.kafka.connect.transforms.TimestampConverter$Value", "transforms.convertTS.field" : "txn_date", "transforms.convertTS.format" : "EEE MMM dd HH:mm:ss zzz yyyy", "transforms.convertTS.target.type": "Timestamp" --- ⏱️Time codes: 00:00:00 Brief introduction 00:01:16 Kafkacat - the swiss-army knife of Kafka commandline tools 00:02:45 Inspecting the schema for a topic in the Schema Registry 00:04:56 Setting up the basic sink connector (no transform) 00:05:38 Using the TimestampConverter to write a string to a timestamp in a database table 00:08:56 Write the date portion of a string timestamp to a Date type in target sink connector 00:10:54 Write the date portion of a string timestamp to a Date type in target sink connector 00:11:43 Converting a string timestamp into unix timestamp (epoch) 00:14:10 Using the TimestampConverter with a Source connector 00:17:00 A brief tangent about useful bash keyboard shortcuts 00:18:19 Converting at ingest vs egress 00:18:50 Recap --- 👾 Demo code and details: https://github.com/confluentinc/demo-... 🗒️ SMT reference: https://docs.confluent.io/platform/cu... Learn more about Kafka Connect here: 🏃‍♂️ Quick:    • Kafka Connect in 60 seconds   🚶 More detail: https://rmoff.dev/kafka-connect-zero-... --- ☁️ Confluent Cloud: https://confluent.cloud/signup?utm_so... 💾Download Confluent Platform: https://www.confluent.io/download/?ut... 📺 Kafka Connect connector deep-dives:    • Kafka Connect   ✍️Kafka Connect documentation: https://docs.confluent.io/current/con... 🧩Confluent Hub: https://www.confluent.io/hub/?utm_sou...