Please use this identifier to cite or link to this item:
https://hdl.handle.net/20.500.14279/32659
Title: | Cost-based Data Prefetching and Scheduling in Big Data Platforms over Tiered Storage Systems | Authors: | Herodotou, Herodotos Kakoulli, Elena |
Major Field of Science: | Engineering and Technology | Field Category: | Electrical Engineering - Electronic Engineering - Information Engineering | Issue Date: | 13-Nov-2023 | Source: | ACM Transactions on Database Systems, 2023, vol. 48, iss. 4, pp. 1-40 | Volume: | 48 | Issue: | 4 | Start page: | 1 | End page: | 40 | Journal: | ACM Transactions on Database Systems | Abstract: | The use of storage tiering is becoming popular in data-intensive compute clusters due to the recent advancements in storage technologies. The Hadoop Distributed File System, for example, now supports storing data in memory, SSDs, and HDDs, while OctopusFS and hatS offer fine-grained storage tiering solutions. However, current big data platforms (such as Hadoop and Spark) are not exploiting the presence of storage tiers and the opportunities they present for performance optimizations. Specifically, schedulers and prefetchers will make decisions only based on data locality information and completely ignore the fact that local data are now stored on a variety of storage media with different performance characteristics. This article presents Trident, a scheduling and prefetching framework that is designed to make task assignment, resource scheduling, and prefetching decisions based on both locality and storage tier information. Trident formulates task scheduling as a minimum cost maximum matching problem in a bipartite graph and utilizes two novel pruning algorithms for bounding the size of the graph, while still guaranteeing optimality. In addition, Trident extends YARN's resource request model and proposes a new storage-tier-aware resource scheduling algorithm. Finally, Trident includes a cost-based data prefetching approach that coordinates with the schedulers for optimizing prefetching operations. Trident is implemented in both Spark and Hadoop and evaluated extensively using a realistic workload derived from Facebook traces as well as an industry-validated benchmark, demonstrating significant benefits in terms of application performance and cluster efficiency. | URI: | https://hdl.handle.net/20.500.14279/32659 | ISSN: | 03625915 | DOI: | 10.1145/3625389 | Rights: | Attribution-NonCommercial-NoDerivatives 4.0 International | Type: | Article | Affiliation : | Cyprus University of Technology | Publication Type: | Peer Reviewed |
Appears in Collections: | Άρθρα/Articles |
Files in This Item:
File | Description | Size | Format | |
---|---|---|---|---|
Herodotou et al.-2023.pdf | open access | 4.02 MB | Adobe PDF | View/Open |
CORE Recommender
Page view(s)
111
Last Week
1
1
Last month
6
6
checked on Dec 23, 2024
Download(s) 50
104
checked on Dec 23, 2024
Google ScholarTM
Check
Altmetric
This item is licensed under a Creative Commons License