Diving into Apache Spark Internals (built with Scala) - Dean Chen

Presented by Dean Chen. For more tech talks and to network with other engineers, check out our site https://www.hakkalabs.co/logs Full post here: https://www.hakkalabs.co/articles/div... Apache Spark is a next generation engine for large scale data processing built with Scala. Dean Chen, software engineer at ebay, discusses how Spark takes advantage of Scala's function idioms to produce an expressive and intuitive API for big data analysis. Dean covers the design of Spark RDDs and the abstraction enables the Spark execution engine to be extended to support a wide variety of use cases: Spark SQL, Spark Streaming, MLib and GraphX. ABOUT DATA COUNCIL: Data Council (https://www.datacouncil.ai/) is a community and conference series that provides data professionals with the learning and networking opportunities they need to grow their careers. Make sure to subscribe to our channel for more videos, including DC_THURS, our series of live online interviews with leading data professionals from top open source projects and startups. FOLLOW DATA COUNCIL: Twitter:   / datacouncilai   LinkedIn:   / datacouncil-ai   Facebook:   / datacouncilai   Eventbrite: https://www.eventbrite.com/o/data-cou... - 🎟️ GET YOUR TICKET TO AI COUNCIL 2026 🎟️ Meet the world's top AI infrastructure minds where architects of AI share what works. Three days of high-quality technical talks and meaningful interactions. → https://aicouncil.com/sf-2026 ⚡ FIND US: X: https://x.com/AICouncilConf LinkedIn:   / aicouncilconf   Website: https://aicouncil.com/