Перейти к содержимому

Query Parquet on S3 with Plain SQL: No Spark, No Warehouse

Synapcores

0:00 / 0:00

Query Parquet on S3 with Plain SQL: No Spark, No Warehouse

119 просмотров · 7 дней назад
Synapcores
4 подписчика
119 просмотров · 7 дней назад
Build a data lake on AWS S3 in about 20 minutes: export MySQL tables to Parquet in your own bucket on a schedule, then query the files in place with plain SQL. No Spark, no ETL tool, no warehouse. ▶ Free download: https://synapcores.com/download?utm_s... ▶ Step-by-step recipe (MySQL → S3 Parquet → SQL, ~20 min): https://synapcores.com/recipes/data-l... Why it matters: most analytics bills aren't storage bills, they're compute bills. A typical marketing agency pays ~$4,800/month to analyze its own data, and only $23 of that is storage. Here's how to run the same reports from Parquet in your own S3 bucket for ~$505/month. In this data engineering breakdown we follow a performance-marketing agency: 3 million ad events a day, ~90 million new rows a month, 40 client dashboards refreshing every hour. We split the bill line by line: 💸 Storage: ~$23/month — the same on Snowflake or S3 💸 Moving the data: ~$1,800/month in per-row ETL fees 💸 Running reports: ~$3,000/month in data warehouse compute Then we rebuild it: export straight from the app database to Parquet in your S3 data lake on an hourly schedule, query the files in place with SQL, and run it all on one fixed-price server. Result: ~85–89% lower cost — up to ~$80,000 a year — with a fixed monthly bill instead of a variable one. What you'll learn: ✅ How to build a data lake on S3 with Parquet files ✅ How to query Parquet in S3 directly with SQL and partition filtering ✅ Why Snowflake costs grow with dashboards and refreshes, not with data size ✅ How per-row (monthly active rows) ETL pricing scales with your growth ✅ The real trade-offs: hourly batch vs streaming, fixed capacity vs auto-scaling ✅ When you should KEEP Snowflake (and when you're overpaying) ✅ How to test it risk-free: one table, one dashboard, one month ⏱ Chapters 00:00 The $4,800 analytics bill 00:39 Where the money actually goes 01:20 It's an architecture problem 01:33 Trap #1: per-row ETL pipelines 02:22 Hourly batch vs streaming 02:48 Trap #2: data warehouse compute 03:10 Query Parquet directly in S3 03:34 Fixed server vs auto-scaling 04:10 Setup in 3 steps (with self-verification) 04:39 $4,800 vs $505: the savings 04:57 The trade-off: you run the server 05:15 When to keep Snowflake 05:57 Try it: one dashboard, one month Numbers are an illustrative scenario using public list prices (checked October 2026): Snowflake $2–$4 per credit and $23/TB/month storage, Amazon S3 $0.023/GB-month, an AWS m7g.4xlarge server (~$477/month on-demand), and an assumed ETL rate of $0.01–$0.03 per 1,000 monthly active rows. Your costs depend on your workload — run the free recipe and compare against your own bill. 🔗 More • SynapCores Community Edition (free — macOS, Linux, Docker): https://synapcores.com/download?utm_s... • Recipe: https://synapcores.com/recipes/data-l... • Replace Fivetran + Snowflake (blog): https://synapcores.com/blog/replace-f... This video includes AI-generated visuals and narration. #DataLake #DataEngineering #AWSS3 #Parquet #Snowflake