Towards End-to-End Compression in Lustre

Link:
Autor/in:
Erscheinungsjahr:
2024
Medientyp:
Text
Beschreibung:
  • Scientific applications generate massive amounts of data, posing storage limitations and network traffic challenges. While scientists struggle with the usage of application-side compression and parallel I/O, we design a transparent feature. Our Lustre-based prototype automatically applies lossless compression, offering flexibility in compression-related decisions to minimize computational costs and optimize application performance. We outline the challenges posed by our prototype and illustrate through a comprehensive assessment how integrating Lustre and ZFS as backend solutions provides the essential elements for performance and scalability: specifically, asynchronous operations and parallel processing of compression. In our evaluation, we illustrate the interaction of different buffer levels within a distributed system. Additionally, we showcase how the I/O pattern, hardware setup, and various system software optimizations can impact overall performance and influence the choice of compression strategy.
Lizenz:
  • info:eu-repo/semantics/closedAccess
Quellsystem:
Forschungsinformationssystem der UHH

Interne Metadaten
Quelldatensatz
oai:www.edit.fis.uni-hamburg.de:publications/807f9a53-cf49-4ded-b636-ebfabdf294e5