Please use this identifier to cite or link to this item:
Title: Automating distributed tiered storage management in cluster computing
Authors: Herodotou, Herodotos 
Kakoulli, Elena 
Major Field of Science: Natural Sciences
Field Category: Computer and Information Sciences
Keywords: Hadoop;Distributed File System;Mapreduce
Issue Date: 2020
Source: Proceedings of the VLDB Endowment, 2020, vol. 13, no. 1, pp. 43-56
Volume: 13
Issue: 1
Start page: 43
End page: 56
Journal: Proceedings of the VLDB Endowment 
Abstract: Data-intensive platforms such as Hadoop and Spark are routinely used to process massive amounts of data residing on distributed le systems like HDFS. Increasing memory sizes and new hardware technologies (e.g., NVRAM, SSDs) have recently led to the introduction of storage tiering in such settings. However, users are now burdened with the additional complexity of managing the multiple storage tiers and the data residing on them while trying to optimize their workloads. In this paper, we develop a general framework for automatically moving data across the available storage tiers in distributed le systems. Moreover, we employ machine learning for tracking and predicting le access patterns, which we use to decide when and which data to move up or down the storage tiers for increasing system performance. Our approach uses incremental learning to dynamically rene the models with new le accesses, allowing them to naturally adjust and adapt to workload changes over time. Our extensive evaluation using realistic workloads derived from Facebook and CMU traces compares our approach with several other policies and showcases signicant bene ts in terms of both workload performance and cluster effciency.
Description: Presented at 46th International Conference on Very Large Data Bases, 31 August - 4 September 2020, Japan
ISSN: 2150-8097
DOI: 10.14778/3357377.3357381
Rights: This work is licensed under the Creative Commons AttributionNonCommercial-NoDerivatives 4.0 International License.
Attribution-NonCommercial-NoDerivatives 4.0 International
Type: Article
Affiliation : Cyprus University of Technology 
Appears in Collections:Άρθρα/Articles

Files in This Item:
File Description SizeFormat
3357377.3357381.pdfFulltext974.4 kBAdobe PDFView/Open
CORE Recommender
Show full item record


checked on Aug 2, 2021


Last Week
Last month
checked on Apr 22, 2021

Page view(s)

Last Week
Last month
checked on Aug 3, 2021


checked on Aug 3, 2021

Google ScholarTM



This item is licensed under a Creative Commons License Creative Commons