Please use this identifier to cite or link to this item:
|Title:||Profiling, what-if analysis, and costbased optimization of mapreduce programs||Authors:||Babu, Shivnath
|Major Field of Science:||Engineering and Technology||Field Category:||Electrical Engineering - Electronic Engineering - Information Engineering||Keywords:||Hadoop;Distributed File System;Mapreduce||Issue Date:||Aug-2011||Source:||Proceedings of the VLDB Endowment, 2011,vol. 4, no 11, pp. 1111-1122||Volume:||4||Issue:||11||Start page:||1111||End page:||1122||Journal:||Proceedings of the VLDB Endowment||Abstract:||MapReduce has emerged as a viable competitor to database systems in big data analytics. MapReduce programs are being written for a wide variety of application domains including business data processing, text analysis, natural language processing, Web graph and social network analysis, and computational science. However, MapReduce systems lack a feature that has been key to the historical success of database systems, namely, cost-based optimization. A major challenge here is that, to the MapReduce system, a program consists of black-box map and reduce functions written in some programming language like C++, Java, Python, or Ruby. We introduce, to our knowledge, the first Cost-based Optimizer for simple to arbitrarily complex MapReduce programs. We focus on the optimization opportunities presented by the large space of configuration parameters for these programs. We also introduce a Profiler to collect detailed statistical information from unmodified MapReduce programs, and a What-if Engine for fine-grained cost estimation. All components have been prototyped for the popular Hadoop MapReduce system. The effectiveness of each component is demonstrated through a comprehensive evaluation using representative MapReduce programs from various application domains. © 2011 VLDB Endowment.||ISSN:||2150-8097||Rights:||© ACM||Type:||Article||Affiliation :||Duke University|
|Appears in Collections:||Άρθρα/Articles|
checked on Jun 15, 2021
checked on Jun 16, 2021
Items in KTISIS are protected by copyright, with all rights reserved, unless otherwise indicated.