Please use this identifier to cite or link to this item:
Title: Cost-based Data Prefetching and Scheduling in Big Data Platforms over Tiered Storage Systems
Authors: Herodotou, Herodotos 
Kakoulli, Elena 
Major Field of Science: Engineering and Technology
Field Category: Electrical Engineering - Electronic Engineering - Information Engineering
Issue Date: 13-Nov-2023
Source: ACM Transactions on Database Systems, 2023, vol. 48, iss. 4, pp. 1-40
Volume: 48
Issue: 4
Start page: 1
End page: 40
Journal: ACM Transactions on Database Systems 
Abstract: The use of storage tiering is becoming popular in data-intensive compute clusters due to the recent advancements in storage technologies. The Hadoop Distributed File System, for example, now supports storing data in memory, SSDs, and HDDs, while OctopusFS and hatS offer fine-grained storage tiering solutions. However, current big data platforms (such as Hadoop and Spark) are not exploiting the presence of storage tiers and the opportunities they present for performance optimizations. Specifically, schedulers and prefetchers will make decisions only based on data locality information and completely ignore the fact that local data are now stored on a variety of storage media with different performance characteristics. This article presents Trident, a scheduling and prefetching framework that is designed to make task assignment, resource scheduling, and prefetching decisions based on both locality and storage tier information. Trident formulates task scheduling as a minimum cost maximum matching problem in a bipartite graph and utilizes two novel pruning algorithms for bounding the size of the graph, while still guaranteeing optimality. In addition, Trident extends YARN's resource request model and proposes a new storage-tier-aware resource scheduling algorithm. Finally, Trident includes a cost-based data prefetching approach that coordinates with the schedulers for optimizing prefetching operations. Trident is implemented in both Spark and Hadoop and evaluated extensively using a realistic workload derived from Facebook traces as well as an industry-validated benchmark, demonstrating significant benefits in terms of application performance and cluster efficiency.
ISSN: 03625915
DOI: 10.1145/3625389
Rights: Attribution-NonCommercial-NoDerivatives 4.0 International
Type: Article
Affiliation : Cyprus University of Technology 
Appears in Collections:Άρθρα/Articles

Files in This Item:
File Description SizeFormat
Herodotou et al.-2023.pdfopen access4.02 MBAdobe PDFView/Open
CORE Recommender
Show full item record

Page view(s)

Last Week
Last month
checked on Jul 23, 2024


checked on Jul 23, 2024

Google ScholarTM



This item is licensed under a Creative Commons License Creative Commons