HDF5 at the speed of Zarr

Pangeo Showcase talk on Wed Mar 13, 2024 by Luis Lopez. As flexible and powerful as HDF5 can be, it traditionally comes with big tradeoffs when it’s accessed from remote storage systems, mainly because the file format and the client I/O libraries were designed for local and supercomputing workflows. As scientific data and workflows migrate to the cloud, efficient access to data stored in HDF5 format is a key factor that will accelerate or slow down “science in the cloud” across all disciplines. We have been working on testing implementation of recently available features in the HDF5 stack that results in performant access to HDF5 from remote cloud storage. This performance seems on par with modern cloud-native formats like Zarr but with the advantage of not having to reformat the data or generate metadata sidecar files (DMR++, Kerchunk).