Please use this identifier to cite or link to this item:
Title: OctopuSFS: A distributed file system with tiered storage management
Authors: Kakoulli, Elena 
Herodotou, Herodotos 
Keywords: Cluster computing;Data handling;Digital storage;Distributed computer systems;Distributed database systems;Fault tolerance;File organization;Molluscs;Multiobjective optimization;Multiprocessing systems;Storage management
Category: Electrical Engineering - Electronic Engineering - Information Engineering
Field: Engineering and Technology
Issue Date: May-2017
Publisher: Association for Computing Machinery
Source: Proceedings of the 2017 ACM International Conference on Management of Data, pp. 65-78
Journal: Proceedings of the ACM SIGMOD International Conference on Management of Data 
Conference: International Conference on Management of Data 
Abstract: The ever-growing data storage and I/O demands of modern large-scale data analytics are challenging the current distributed storage systems. A promising trend is to exploit the recent improvements in memory, storage media, and networks for sustaining high performance and low cost. While past work explores using memory or SSDs as local storage or combine local with network-attached storage in cluster computing, this work focuses on managing multiple storage tiers in a distributed setting. We present OctopusFS, a novel distributed file system that is aware of heterogeneous storage media (e.g., memory, SSDs, HDDs, NAS) with different capacities and performance characteristics. The system offers a variety of pluggable policies for automating data management across the storage tiers and cluster nodes. The policies employ multi-objective optimization techniques for making intelligent data management decisions based on the requirements of fault tolerance, data and load balancing, and throughput maximization. At the same time, the storage media are explicitly exposed to users and applications, allowing them to choose the distribution and placement of replicas in the cluster based on their own performance and fault tolerance requirements. Our extensive evaluation shows the immediate benefits of using OctopusFS with data-intensive processing systems, such as Hadoop and Spark, in terms of both increased performance and better cluster utilization.
ISBN: 9781450341974
ISSN: 0730-8078
DOI: 10.1145/3035918.3064023
Rights: © ACM
Type: Book Chapter
Appears in Collections:Κεφάλαια βιβλίων/Book chapters

Show full item record


checked on Oct 13, 2019

Page view(s) 5

Last Week
Last month
checked on Oct 16, 2019

Google ScholarTM



Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.