Please use this identifier to cite or link to this item: https://ktisis.cut.ac.cy/handle/10488/13849
Title: Profiling, what-if analysis, and costbased optimization of mapreduce programs
Authors: Babu, Shivnath 
Herodotou, Herodotos 
Major Field of Science: Engineering and Technology
Field Category: Electrical Engineering - Electronic Engineering - Information Engineering
Keywords: Hadoop;Distributed File System;Mapreduce
Issue Date: Aug-2011
Source: Proceedings of the VLDB Endowment, 2011,vol. 4, no 11, pp. 1111-1122
Volume: 4
Issue: 11
Start page: 1111
End page: 1122
Journal: Proceedings of the VLDB Endowment
Abstract: MapReduce has emerged as a viable competitor to database systems in big data analytics. MapReduce programs are being written for a wide variety of application domains including business data processing, text analysis, natural language processing, Web graph and social network analysis, and computational science. However, MapReduce systems lack a feature that has been key to the historical success of database systems, namely, cost-based optimization. A major challenge here is that, to the MapReduce system, a program consists of black-box map and reduce functions written in some programming language like C++, Java, Python, or Ruby. We introduce, to our knowledge, the first Cost-based Optimizer for simple to arbitrarily complex MapReduce programs. We focus on the optimization opportunities presented by the large space of configuration parameters for these programs. We also introduce a Profiler to collect detailed statistical information from unmodified MapReduce programs, and a What-if Engine for fine-grained cost estimation. All components have been prototyped for the popular Hadoop MapReduce system. The effectiveness of each component is demonstrated through a comprehensive evaluation using representative MapReduce programs from various application domains. © 2011 VLDB Endowment.
ISSN: 2150-8097
Rights: © ACM
Type: Article
Affiliation : Duke University 
Appears in Collections:Άρθρα/Articles

CORE Recommender
Show full item record

SCOPUSTM   
Citations

246
checked on Jun 15, 2021

Page view(s)

46
Last Week
1
Last month
2
checked on Jun 16, 2021

Google ScholarTM

Check


Items in KTISIS are protected by copyright, with all rights reserved, unless otherwise indicated.