Please use this identifier to cite or link to this item:
Title: Profiling, what-if analysis, and costbased optimization of mapreduce programs
Authors: Babu, Shivnath 
Herodotou, Herodotos 
Major Field of Science: Engineering and Technology
Field Category: Electrical Engineering - Electronic Engineering - Information Engineering
Keywords: Hadoop;Distributed File System;Mapreduce
Issue Date: Aug-2011
Source: Proceedings of the VLDB Endowment, 2011,vol. 4, no 11, pp. 1111-1122
Volume: 4
Issue: 11
Start page: 1111
End page: 1122
Journal: Proceedings of the VLDB Endowment
Abstract: MapReduce has emerged as a viable competitor to database systems in big data analytics. MapReduce programs are being written for a wide variety of application domains including business data processing, text analysis, natural language processing, Web graph and social network analysis, and computational science. However, MapReduce systems lack a feature that has been key to the historical success of database systems, namely, cost-based optimization. A major challenge here is that, to the MapReduce system, a program consists of black-box map and reduce functions written in some programming language like C++, Java, Python, or Ruby. We introduce, to our knowledge, the first Cost-based Optimizer for simple to arbitrarily complex MapReduce programs. We focus on the optimization opportunities presented by the large space of configuration parameters for these programs. We also introduce a Profiler to collect detailed statistical information from unmodified MapReduce programs, and a What-if Engine for fine-grained cost estimation. All components have been prototyped for the popular Hadoop MapReduce system. The effectiveness of each component is demonstrated through a comprehensive evaluation using representative MapReduce programs from various application domains. © 2011 VLDB Endowment.
ISSN: 2150-8097
Rights: © ACM
Type: Article
Affiliation : Duke University 
Appears in Collections:Άρθρα/Articles

CORE Recommender
Show full item record


checked on Jun 15, 2021

Page view(s)

Last Week
Last month
checked on Jun 16, 2021

Google ScholarTM


Items in KTISIS are protected by copyright, with all rights reserved, unless otherwise indicated.