Presentation Details |
|||||
Name: | (RP13) Enhanced Adaptive Compression in Lustre | ||||
Time: | Tuesday, June 20, 2017 08:35 am - 09:45 am |
||||
Room: | Substanz 1+2 | ||||
Breaks: | 07:30 am - 10:00 am Welcome Coffee | ||||
Presenter: | Anna Fuchs, University of Hamburg | ||||
Abstract: | The poster presents the design of transparent compression in the Lustre filesystem. Compression can increase the throughput and save costs for storage systems. In Lustre it will be based on stripes which are divided into chunks corresponding to the ZFS records. The most important challenges are potential performance losses related to bad read-ahead performance and read-modify-write behavior. ZFS offers solutions for the read-ahead problem when using the tree structure and saving the compressed data without gaps. RMW requires complex strategies, however, its impact can be reduced significantly by using chunks instead of complete stripes. First benchmarks showed that lz4 and zstd are well suited for the approach due to their very high compression speed, although it requires parallel compression to satisfy fast interconnects. Authors: Anna Fuchs, Universität Hamburg Michael Kuhn, Universität Hamburg Julian Kunkel, DKRZ Thomas Ludwig, Universität Hamburg / DKRZ |
||||
Download | RP13_Fuchs.pdf (425 KB) |
||||