Optimizing Hadoop for MapReduce : learn how to configure your Hadoop cluster to run optimal MapReduce jobs

120 Pages·2014·2.15 MB·English

by  Tannir

Optimizing Hadoop for
MapReduce
Learn how to configure your Hadoop cluster to run
optimal MapReduce jobs
Khaled Tannir
BIRMINGHAM - MUMBAI
Optimizing Hadoop for MapReduce
Copyright © 2014 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval
system, or transmitted in any form or by any means, without the prior written
permission of the publisher, except in the case of brief quotations embedded in
critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy
of the information presented. However, the information contained in this book is
sold without warranty, either expressed or implied. Neither the author, nor Packt
Publishing, and its dealers and distributors will be held liable for any damages
caused or alleged to be caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the
companies and products mentioned in this book by the appropriate use of capitals.
However, Packt Publishing cannot guarantee the accuracy of this information.
First published: February 2014
Production Reference: 1140214
Published by Packt Publishing Ltd.
Livery Place
35 Livery Street
Birmingham B3 2PB, UK.
ISBN 978-1-78328-565-5
www.packtpub.com
Cover Image by Khaled Tannir ([email protected])
Credits
Author Project Coordinator
Khaled Tannir Aboli Ambardekar
Reviewers Proofreaders
Włodzimierz Bzyl Simran Bhogal
Craig Henderson Ameesha Green
Mark Kerzner
Indexer
Rekha Nair
Acquisition Editor
Joanne Fitzpatrick
Graphics
Yuvraj Mannari
Commissioning Editor
Manasi Pandire
Production Coordinators
Manu Joseph
Technical Editors
Mario D'Souza Alwin Roy
Rosmy George
Pramod Kumavat Cover Work
Alwin Roy
Arwa Manasawala
Adrian Raposo
Copy Editors
Kirti Pai
Laxmi Subramanian
About the Author
Khaled Tannir has been working with computers since 1980. He began
programming with the legendary Sinclair Zx81 and later with Commodore home
computer products (Vic 20, Commodore 64, Commodore 128D, and Amiga 500).
He has a Bachelor's degree in Electronics, a Master's degree in System Information
Architectures, in which he graduated with a professional thesis, and completed his
education with a Master of Research degree.
He is a Microsoft Certified Solution Developer (MCSD) and has more than 20 years
of technical experience leading the development and implementation of software
solutions and giving technical presentations. He now works as an independent IT
consultant and has worked as an infrastructure engineer, senior developer, and
enterprise/solution architect for many companies in France and Canada.
With significant experience in Microsoft .Net, Microsoft Server Systems,
and Oracle Java technologies, he has extensive skills in online/offline applications
design, system conversions, and multilingual applications in both domains: Internet
and Desktops.
He is always researching new technologies, learning about them, and looking for
new adventures in France, North America, and the Middle-east. He owns an IT and
electronics laboratory with many servers, monitors, open electronic boards such
as Arduino, Netduino, RaspBerry Pi, and .Net Gadgeteer, and some smartphone
devices based on Windows Phone, Android, and iOS operating systems.
In 2012, he contributed to the EGC 2012 (International Complex Data Mining forum at
Bordeaux University, France) and presented, in a workshop session, his work on "how
to optimize data distribution in a cloud computing environment". This work aims to
define an approach to optimize the use of data mining algorithms such as k-means and
Apriori in a cloud computing environment.
He is the author of RavenDB 2.x Beginner's Guide, Packt Publishing.
He aims to get a PhD in Cloud Computing and Big Data and wants to learn more
and more about these technologies.
He enjoys taking landscape and night time photos, travelling, playing video games,
creating funny electronic gadgets with Arduino/.Net Gadgeteer, and of course,
spending time with his wife and family.
You can reach him at [email protected].
Acknowledgments
All praise is due to Allah, the Lord of the Worlds. First, I must thank Allah for giving
me the ability to think and write.
Next, I would like to thank my wife, Laila, for her big support, encouragement, and
patience throughout this project. Also, I would like to thank my family in Canada
and Lebanon for their support during the writing of this book.
I would like to thank everyone at Packt Publishing for their help and guidance, and
for giving me the opportunity to share my experience and knowledge in technology
with others in the Hadoop and MapReduce community.
Thank you as well to the technical reviewers, who provided great feedback to ensure
that every tiny technical detail was accurate and rich in content.
About the Reviewers
Włodzimierz Bzyl works at the University of Gdańsk, Poland. His current
interests include web-related technologies and NoSQL databases. He has a passion
for new technologies and introduces his students to them. He enjoys contributing
to open source software and spending time trekking in the Tatra mountains.
Craig Henderson graduated in 1995 with a degree in Computing for Real-time
Systems and has spent his career working on large-scale data processing and
distributed systems. He is the author of an open source C++ MapReduce library
for single server application scalability, which is available at https://github.
com/cdmh/mapreduce, and he currently researches image and video processing
techniques for person identification.
Mark Kerzner holds degrees in Law, Mathematics, and Computer Science. He has
been designing software for many years and Hadoop-based systems since 2008. He is
the President of SHMsoft, a provider of Hadoop applications for various verticals, a
co-founder of the Hadoop Illuminated training and consulting, and also the
co-author of the open source book, Hadoop Illuminated. He has also authored and
co-authored other books and patents.
I would like to acknowledge the help of my colleagues, in particular
Sujee Maniyam, and last but not least, my multitalented family.
www.PacktPub.com
Support files, eBooks, discount offers and more
You might want to visit www.PacktPub.com for support files and downloads related
to your book.
Did you know that Packt offers eBook versions of every book published, with PDF
and ePub files available? You can upgrade to the eBook version at www.PacktPub.
com and as a print book customer, you are entitled to a discount on the eBook copy.
Get in touch with us at [email protected] for more details.
At www.PacktPub.com, you can also read a collection of free technical articles, sign
up for a range of free newsletters and receive exclusive discounts and offers on Packt
books and eBooks.
TM
http://PacktLib.PacktPub.com
Do you need instant solutions to your IT questions? PacktLib is Packt's online
digital book library. Here, you can access, read and search across Packt's entire
library of books.
Why Subscribe?
• Fully searchable across every book published by Packt
• Copy and paste, print and bookmark content
• On demand and accessible via web browser
Free Access for Packt account holders
If you have an account with Packt at www.PacktPub.com, you can use this to access
PacktLib today and view nine entirely free books. Simply use your login credentials
for immediate access.
Table of Contents
Preface 1
Chapter 1: Understanding Hadoop MapReduce 7
The MapReduce model 7
An overview of Hadoop MapReduce 10
Hadoop MapReduce internals 10
Factors affecting the performance of MapReduce 12
Summary 14
Chapter 2: An Overview of the Hadoop Parameters 15
Investigating the Hadoop parameters 16
The mapred-site.xml configuration file 17
The CPU-related parameters 17
The disk I/O related parameters 18
The memory-related parameters 20
The network-related parameters 21
The hdfs-site.xml configuration file 22
The core-site.xml configuration file 25
Hadoop MapReduce metrics 26
Performance monitoring tools 26
Using Chukwa to monitor Hadoop 27
Using Ganglia to monitor Hadoop 27
Using Nagios to monitor Hadoop 28
Using Apache Ambari to monitor Hadoop 29
Summary 29
Chapter 3: Detecting System Bottlenecks 31
Performance tuning 32
Creating a performance baseline 33
Identifying resource bottlenecks 36
Identifying RAM bottlenecks 37