Παρακαλώ χρησιμοποιήστε αυτό το αναγνωριστικό για να παραπέμψετε ή να δημιουργήσετε σύνδεσμο προς αυτό το τεκμήριο:
https://hdl.handle.net/20.500.14279/32659
Τίτλος: | Cost-based Data Prefetching and Scheduling in Big Data Platforms over Tiered Storage Systems | Συγγραφείς: | Herodotou, Herodotos Kakoulli, Elena |
Major Field of Science: | Engineering and Technology | Field Category: | Electrical Engineering - Electronic Engineering - Information Engineering | Ημερομηνία Έκδοσης: | 13-Νοε-2023 | Πηγή: | ACM Transactions on Database Systems, 2023, vol. 48, iss. 4, pp. 1-40 | Volume: | 48 | Issue: | 4 | Start page: | 1 | End page: | 40 | Περιοδικό: | ACM Transactions on Database Systems | Περίληψη: | The use of storage tiering is becoming popular in data-intensive compute clusters due to the recent advancements in storage technologies. The Hadoop Distributed File System, for example, now supports storing data in memory, SSDs, and HDDs, while OctopusFS and hatS offer fine-grained storage tiering solutions. However, current big data platforms (such as Hadoop and Spark) are not exploiting the presence of storage tiers and the opportunities they present for performance optimizations. Specifically, schedulers and prefetchers will make decisions only based on data locality information and completely ignore the fact that local data are now stored on a variety of storage media with different performance characteristics. This article presents Trident, a scheduling and prefetching framework that is designed to make task assignment, resource scheduling, and prefetching decisions based on both locality and storage tier information. Trident formulates task scheduling as a minimum cost maximum matching problem in a bipartite graph and utilizes two novel pruning algorithms for bounding the size of the graph, while still guaranteeing optimality. In addition, Trident extends YARN's resource request model and proposes a new storage-tier-aware resource scheduling algorithm. Finally, Trident includes a cost-based data prefetching approach that coordinates with the schedulers for optimizing prefetching operations. Trident is implemented in both Spark and Hadoop and evaluated extensively using a realistic workload derived from Facebook traces as well as an industry-validated benchmark, demonstrating significant benefits in terms of application performance and cluster efficiency. | URI: | https://hdl.handle.net/20.500.14279/32659 | ISSN: | 03625915 | DOI: | 10.1145/3625389 | Rights: | Attribution-NonCommercial-NoDerivatives 4.0 International | Type: | Article | Affiliation: | Cyprus University of Technology | Publication Type: | Peer Reviewed |
Εμφανίζεται στις συλλογές: | Άρθρα/Articles |
Αρχεία σε αυτό το τεκμήριο:
Αρχείο | Περιγραφή | Μέγεθος | Μορφότυπος | |
---|---|---|---|---|
Herodotou et al.-2023.pdf | open access | 4.02 MB | Adobe PDF | Δείτε/ Ανοίξτε |
CORE Recommender
Page view(s)
111
Last Week
1
1
Last month
6
6
checked on 22 Δεκ 2024
Download(s)
104
checked on 22 Δεκ 2024
Google ScholarTM
Check
Altmetric
Αυτό το τεκμήριο προστατεύεται από άδεια Άδεια Creative Commons