The Spice Cluster-Sidecar Architecture with Apache DataFusion: Seattle DataFusion Meetup, April 2026

Spice AI Founder & CEO Luke Kim presents at the April 2026 Apache DataFusion Meetup in Seattle, covering why the shift to agentic AI demands a new approach to data infrastructure, and how Spice addresses it with the cluster-sidecar architecture. Luke walks through why centralized data platforms weren't built for the query volume, latency requirements, and security constraints of AI agents, and demonstrates Spice's approach: a lightweight sidecar giving agents fast local data access, backed by a distributed cluster for large-scale queries. The demo covers: Single-node vs. distributed query execution on 100M+ Amazon review rows using Apache Ballista Acceleration with Spice Cayenne, taking a 22-second query to under 3 seconds Distributed vector search and embedding generation across cluster nodes - Inline AI classification using LLM UDFs, pushed down to executor nodes Also covered: multi-active HA scheduler architecture using object store coordination, Vortex columnar format integration, and SpiceBench - an open source benchmark for continuous ingestion and mutation workloads. Spice AI is open source and runs anywhere from a Raspberry Pi to a multi-node cluster. References: The Spice Cluster-Sidecar Architecture: https://spice.ai/blog/cluster-sidecar... Apache DataFusion Ballista at Spice AI: https://spice.ai/blog/apache-ballista... Spice GitHub: github.com/spiceai/spiceai