Please use this identifier to cite or link to this item: https://hdl.handle.net/20.500.14279/19364
Title: Automating distributed tiered storage management in cluster computing
Authors: Herodotou, Herodotos 
Kakoulli, Elena 
Major Field of Science: Natural Sciences
Field Category: Computer and Information Sciences
Keywords: Hadoop;Distributed File System;Mapreduce
Issue Date: 2020
Source: Proceedings of the VLDB Endowment, 2020, vol. 13, no. 1, pp. 43-56
Volume: 13
Issue: 1
Start page: 43
End page: 56
Journal: Proceedings of the VLDB Endowment 
Abstract: Data-intensive platforms such as Hadoop and Spark are routinely used to process massive amounts of data residing on distributed le systems like HDFS. Increasing memory sizes and new hardware technologies (e.g., NVRAM, SSDs) have recently led to the introduction of storage tiering in such settings. However, users are now burdened with the additional complexity of managing the multiple storage tiers and the data residing on them while trying to optimize their workloads. In this paper, we develop a general framework for automatically moving data across the available storage tiers in distributed le systems. Moreover, we employ machine learning for tracking and predicting le access patterns, which we use to decide when and which data to move up or down the storage tiers for increasing system performance. Our approach uses incremental learning to dynamically rene the models with new le accesses, allowing them to naturally adjust and adapt to workload changes over time. Our extensive evaluation using realistic workloads derived from Facebook and CMU traces compares our approach with several other policies and showcases signicant bene ts in terms of both workload performance and cluster effciency.
Description: Presented at 46th International Conference on Very Large Data Bases, 31 August - 4 September 2020, Japan
URI: https://hdl.handle.net/20.500.14279/19364
ISSN: 21508097
DOI: 10.14778/3357377.3357381
Rights: This work is licensed under the Creative Commons AttributionNonCommercial-NoDerivatives 4.0 International License.
Attribution-NonCommercial-NoDerivatives 4.0 International
Type: Article
Affiliation : Cyprus University of Technology 
Publication Type: Peer Reviewed
Appears in Collections:Άρθρα/Articles

Files in This Item:
File Description SizeFormat
3357377.3357381.pdfFulltext974.4 kBAdobe PDFView/Open
CORE Recommender
Show full item record

SCOPUSTM   
Citations

14
checked on Nov 6, 2023

WEB OF SCIENCETM
Citations

10
Last Week
1
Last month
1
checked on Oct 29, 2023

Page view(s)

318
Last Week
0
Last month
2
checked on Dec 23, 2024

Download(s) 50

85
checked on Dec 23, 2024

Google ScholarTM

Check

Altmetric


This item is licensed under a Creative Commons License Creative Commons