Friday Night Database Recovery Crash Course (A True Story)

A story about database backups and restore drills, Point-In-Time Recovery with the Write-Ahead Log, RPO and RTO, alert fatigue, and incident response. This video is built for learning! To keep things clear, some details have been oversimplified. Chapters 0:00 A Friday night incident 0:44 Friday afternoon 1:28 The CEO's request and log_statement 3:08 The disk space alert 3:38 Self-hosted Postgres on Kubernetes 4:35 Deleting the log files 5:54 rm -rf on an empty variable 7:04 Incident response and communication 8:29 Daily backups and the lost day 8:53 Write-Ahead Log and Point-In-Time Recovery 9:37 No restore procedure 10:13 Restore drills, RPO and RTO 11:16 No backup for 12 days 12:16 Alert fatigue 12:39 The CEO joins the call 13:41 Restoring from a 12 day old backup 15:07 Rotating shifts 16:04 Monday morning hi@tajpouria.com https://github.com/tajpouria #postgresql #devops #sre #database #incidentresponse