kCephFS: RDMA-based CephFS I/O Acceleration for AI Workloads

Speaker: Daegon Yang (Storage Team Lead @ Dnotitia) AI training workloads require storage to handle both massive sequential reads during the training phase and massive sequential writes during the checkpoint phase. This session covers the work of extending the Linux kernel CephFS client (ceph.ko) based on RDMA and optimizing each of these loads. Since the mainline kernel CephFS client does not support RDMA (only ceph-fuse supports it), we first implemented RDMA transport paths in the kernel client and Ceph Messenger. With the application of RDMA, read bandwidth improved by approximately 1.6 times, and write bandwidth and P99 latency were also improved. In addition to this, for the read path, the structure where the OSD reassembled and copied data read from the disk into its own buffer before transmission was reconfigured into a zero-copy path based on a buffer pool pre-registered with the NIC (O_DIRECT performs direct DMA to the application buffer, while buffered performs zero-copy to the page cache). This further improved read performance. In the write path, multiple 4KB pages were grouped into a single large unit using large folio to reduce the cost of page-unit iteration, and fsync drains were distributed to multiple workers using parallel range-split drains. As a result, writeback drain bandwidth was improved by approximately 11 times. In addition, the rationale for choosing a kernel client instead of a user-space daemon, OSD crashes caused by the lack of application-level backpressure in the RDMA path, and the necessity of server-side flow control are explained, and various benchmark results are covered.