Big Data Analytics with Microsoft HDInsight in 24 Hours, Sams Teach Yourself

1049 Pages·2015·42.06 MB·English

About This E-Book
EPUB is an open, industry-standard format for e-books. However, support for EPUB
and its many features varies across reading devices and applications. Use your device or
app settings to customize the presentation to your liking. Settings that you can customize
often include font, font size, single or double column, landscape or portrait mode, and
figures that you can click or tap to enlarge. For additional information about the settings
and features on your reading device or app, visit the device manufacturer’s Web site.
Many titles include programming code or configuration examples. To optimize the
presentation of these elements, view the e-book in single-column, landscape mode and
adjust the font size to the smallest setting. In addition to presenting code and
configurations in the reflowable text format, we have included images of the code that
mimic the presentation found in the print book; therefore, where the reflowable format
may compromise the presentation of the code listing, you will see a “Click here to view
code image” link. Click the link to view the print-fidelity code image. To return to the
previous page viewed, click the Back button on your device or app.
Sams Teach Yourself: Big Data Analytics
with Microsoft HDInsight® in 24 Hours
Arshad Ali
Manpreet Singh
800 East 96th Street, Indianapolis, Indiana, 46240 USA
Sams Teach Yourself Big Data Analytics with Microsoft HDInsight® in 24
Hours
Copyright © 2016 by Pearson Education, Inc.
All rights reserved. No part of this book shall be reproduced, stored in a retrieval system,
or transmitted by any means, electronic, mechanical, photocopying, recording, or
otherwise, without written permission from the publisher. No patent liability is assumed
with respect to the use of the information contained herein. Although every precaution has
been taken in the preparation of this book, the publisher and author assume no
responsibility for errors or omissions. Nor is any liability assumed for damages resulting
from the use of the information contained herein.
ISBN-13: 978-0-672-33727-7
ISBN-10: 0-672-33727-4
Library of Congress Control Number: 2015914167
Printed in the United States of America
First Printing November 2015
Editor-in-Chief
Greg Wiegand
Acquisitions Editor
Joan Murray
Development Editor
Sondra Scott
Managing Editor
Sandra Schroeder
Senior Project Editor
Tonya Simpson
Copy Editor
Krista Hansing
Editorial Services, Inc
Senior Indexer
Cheryl Lenser
Proofreader
Anne Goebel
Technical Editors
Shayne Burgess
Ron Abellera
Publishing Coordinator
Cindy Teeter
Cover Designer
Mark Shirar
Compositor
codeMantra
Trademarks
All terms mentioned in this book that are known to be trademarks or service marks have
been appropriately capitalized. Sams Publishing cannot attest to the accuracy of this
information. Use of a term in this book should not be regarded as affecting the validity of
any trademark or service mark.
HDInsight is a registered trademark of Microsoft Corporation.
Warning and Disclaimer
Every effort has been made to make this book as complete and as accurate as possible, but
no warranty or fitness is implied. The information provided is on an “as is” basis. The
authors and the publisher shall have neither liability nor responsibility to any person or
entity with respect to any loss or damages arising from the information contained in this
book.
Special Sales
For information about buying this title in bulk quantities, or for special sales opportunities
(which may include electronic versions; custom cover designs; and content particular to
your business, training goals, marketing focus, or branding interests), please contact our
corporate sales department at [e-mail] or [phone].
For government sales inquiries, please contact
[e-mail].
For questions about sales outside the U.S., please contact
[e-mail].
Contents at a Glance
Introduction
Part I: Understanding Big Data, Hadoop 1.0, and 2.0
HOUR 1 Introduction of Big Data, NoSQL, and Business Value Proposition
2 Introduction to Hadoop, Its Architecture, Ecosystem, and Microsoft Offerings
3 Hadoop Distributed File System Versions 1.0 and 2.0
4 The MapReduce Job Framework and Job Execution Pipeline
5 MapReduce—Advanced Concepts and YARN
Part II: Getting Started with HDInsight and Understanding Its Different
Components
HOUR 6 Getting Started with HDInsight, Provisioning Your HDInsight Service Cluster,
and Automating HDInsight Cluster Provisioning
7 Exploring Typical Components of HDFS Cluster
8 Storing Data in Microsoft Azure Storage Blob
9 Working with Microsoft Azure HDInsight Emulator
Part III: Programming MapReduce and HDInsight Script Action
HOUR 10 Programming MapReduce Jobs
11 Customizing the HDInsight Cluster with Script Action
Part IV: Querying and Processing Big Data in HDInsight
HOUR 12 Getting Started with Apache Hive and Apache Tez in HDInsight
13 Programming with Apache Hive, Apache Tez in HDInsight, and Apache
HCatalog
14 Consuming HDInsight Data from Microsoft BI Tools over Hive ODBC Driver:
Part 1
15 Consuming HDInsight Data from Microsoft BI Tools over Hive ODBC Driver:
Part 2
16 Integrating HDInsight with SQL Server Integration Services
17 Using Pig for Data Processing
18 Using Sqoop for Data Movement Between RDBMS and HDInsight
Part V: Managing Workflow and Performing Statistical Computing
HOUR 19 Using Oozie Workflows and Job Orchestration with HDInsight
20 Performing Statistical Computing with R
Part VI: Performing Interactive Analytics and Machine Learning
HOUR 21 Performing Big Data Analytics with Spark
22 Microsoft Azure Machine Learning
Part VII: Performing Real-time Analytics
HOUR 23 Performing Stream Analytics with Storm
24 Introduction to Apache HBase on HDInsight
Index
Table of Contents
Introduction
Part I: Understanding Big Data, Hadoop 1.0, and 2.0
HOUR 1: Introduction of Big Data, NoSQL, and Business Value Proposition
Types of Analysis
Types of Data
Big Data
Managing Big Data
NoSQL Systems
Big Data, NoSQL Systems, and the Business Value Proposition
Application of Big Data and Big Data Solutions
Summary
Q&A
HOUR 2: Introduction to Hadoop, Its Architecture, Ecosystem, and Microsoft
Offerings
What Is Apache Hadoop?
Architecture of Hadoop and Hadoop Ecosystems
What’s New in Hadoop 2.0
Architecture of Hadoop 2.0
Tools and Technologies Needed with Big Data Analytics
Major Players and Vendors for Hadoop
Deployment Options for Microsoft Big Data Solutions
Summary
Q&A
HOUR 3: Hadoop Distributed File System Versions 1.0 and 2.0
Introduction to HDFS
HDFS Architecture
Rack Awareness
WebHDFS
Accessing and Managing HDFS Data
What’s New in HDFS 2.0
Summary
Q&A
HOUR 4: The MapReduce Job Framework and Job Execution Pipeline
Introduction to MapReduce
MapReduce Architecture
MapReduce Job Execution Flow
Summary
Q&A
HOUR 5: MapReduce—Advanced Concepts and YARN
DistributedCache
Hadoop Streaming
MapReduce Joins
Bloom Filter
Performance Improvement
Handling Failures
Counter
YARN
Uber-Tasking Optimization
Failures in YARN
Resource Manager High Availability and Automatic Failover in YARN
Summary
Q&A
Part II: Getting Started with HDInsight and Understanding Its Different
Components
HOUR 6: Getting Started with HDInsight, Provisioning Your HDInsight Service
Cluster, and Automating HDInsight Cluster Provisioning
Introduction to Microsoft Azure
Understanding HDInsight Service
Provisioning HDInsight on the Azure Management Portal
Automating HDInsight Provisioning with PowerShell
Managing and Monitoring HDInsight Cluster and Job Execution
Summary
Q&A
Exercise
HOUR 7: Exploring Typical Components of HDFS Cluster
HDFS Cluster Components
HDInsight Cluster Architecture
High Availability in HDInsight
Summary
Q&A
HOUR 8: Storing Data in Microsoft Azure Storage Blob
Understanding Storage in Microsoft Azure
Benefits of Azure Storage Blob over HDFS
Azure Storage Explorer Tools
Summary
Q&A
HOUR 9: Working with Microsoft Azure HDInsight Emulator
Getting Started with HDInsight Emulator
Setting Up Microsoft Azure Emulator for Storage
Summary
Q&A
Part III: Programming MapReduce and HDInsight Script Action
HOUR 10: Programming MapReduce Jobs
MapReduce Hello World!
Analyzing Flight Delays with MapReduce
Serialization Frameworks for Hadoop
Hadoop Streaming
Summary
Q&A
HOUR 11: Customizing the HDInsight Cluster with Script Action
Identifying the Need for Cluster Customization
Developing Script Action
Consuming Script Action
Running a Giraph job on a Customized HDInsight Cluster
Testing Script Action with HDInsight Emulator
Summary