Binary Analysis Cookbook: Actionable recipes for disassembling and analyzing binaries for security risks

388 Pages·2019·8.194 MB·English

by  Michael Born

Binary Analysis Cookbook
Actionable recipes for disassembling and analyzing binaries
for security risks
Michael Born
BIRMINGHAM - MUMBAI
Binary Analysis Cookbook
Copyright © 2019 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form
or by any means, without the prior written permission of the publisher, except in the case of brief quotations
embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the information presented.
However, the information contained in this book is sold without warranty, either express or implied. Neither the
author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to
have been caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the companies and products
mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy
of this information.
Commissioning Editor: Pavan Ramchandani
Acquisition Editor: Prachi Bisht
Content Development Editor: Ronn Kurien
Senior Editor: Rahul Dsouza
Technical Editor: Komal Karne
Copy Editor: Safis Editing
Project Coordinator: Vaidehi Sawant
Proofreader: Safis Editing
Indexer: Rekha Nair
Production Designer: Nilesh Mohite
First published: September 2019
Production reference: 1190919
Published by Packt Publishing Ltd.
Livery Place
35 Livery Street
Birmingham
B3 2PB, UK.
ISBN 978-1-78980-760-8
www.packt.com
I dedicate this book to my friend, Joe Blackshaw, for his encouragement and positive
reinforcement up until he breathed his last breath. I miss you my friend and look forward to
seeing you in Heaven some day. This world is not the same without you, and your friendship
meant so much to me over the years, and especially while working on this project. May you
truly rest in peace my friend.
Packt.com
Subscribe to our online digital library for full access to over 7,000 books and videos, as well
as industry leading tools to help you plan your personal development and advance your
career. For more information, please visit our website.
Why subscribe?
Spend less time learning and more time coding with practical eBooks and Videos
from over 4,000 industry professionals
Improve your learning with Skill Plans built especially for you
Get a free eBook or video every month
Fully searchable for easy access to vital information
Copy and paste, print, and bookmark content
Did you know that Packt offers eBook versions of every book published, with PDF and
ePub files available? You can upgrade to the eBook version at www.packt.com and as a print
book customer, you are entitled to a discount on the eBook copy. Get in touch with us at
[email protected] for more details.
At www.packt.com, you can also read a collection of free technical articles, sign up for a
range of free newsletters, and receive exclusive discounts and offers on Packt books and
eBooks.
Contributors
About the author
Michael Born is a senior security consultant for SecureSky, Inc. Michael has earned several
industry certifications and has co-taught offensive-focused Python programming classes at
OWASP AppSec USA, and AppSec Europe. He enjoys coding in Python, IA32, IA64,
PowerShell, participating in, and designing, capture the flag (CTF) challenges, teaching
and mentoring others looking to embark on a career in information security, and presenting
on various information security topics at local chapters of well-known information security
groups. Michael has served on the chapter board for his local OWASP chapter, is a lifetime
OWASP member, and participates in the local DC402 group.
I would like to thank my wife and children for their patience, support, love, and
encouragement throughout this process, during what was already a challenging year for
us. Thank you also to my friends in DC402, OWASP, and my colleagues across the U.S.
for their continued encouragement and support. Finally, thank you to my Heavenly
Father, for your love, grace, and the talents you bless each of us with. To you be the glory
forever and ever. Amen.
About the reviewer
Andrew Freeborn has been involved in security and IT for over 20 years across multiple
industries and countries. By anticipating the latest threats with the help of research, he
specializes in looking at things from the perspective of an attacker in order to identify
specific threats in each organization. Andrew enjoys speaking at conferences, learning, and
baking.
I would like to thank my family for their love and support, and Michael for providing me
with this amazing opportunity.
Packt is searching for authors like you
If you're interested in becoming an author for Packt, please visit authors.packtpub.com
and apply today. We have worked with thousands of developers and tech professionals,
just like you, to help them share their insight with the global tech community. You can
make a general application, apply for a specific hot topic that we are recruiting an author
for, or submit your own idea.
Table of Contents
Preface 1
Chapter 1: Setting Up the Lab 7
Installing VirtualBox on Windows 8
Getting ready 9
How to do it... 9
How it works... 10
There's more... 10
See also 11
Installing VirtualBox on Mac 11
Getting ready 11
How to do it... 11
How it works... 12
There's more... 13
See also 13
Installing VirtualBox on Ubuntu 13
Getting ready 13
How to do it... 15
How it works... 16
There's more... 16
See also 16
Installing a 32-bit Ubuntu 16.04 LTS Desktop virtual machine 16
Getting ready 17
How to do it... 17
How it works... 19
There's more... 19
See also 19
Installing a 64-bit Ubuntu 16.04 LTS Desktop virtual machine 20
Getting ready 20
How to do it... 20
How it works... 23
There's more... 23
See also 24
Installing the dependencies and the tools 24
Getting ready 24
How to do it... 25
How it works... 25
There's more... 26
See also 26
Installing the code examples 26
Table of Contents
Getting ready 27
How to do it... 27
How it works... 27
There's more... 28
See also 28
Installing the EDB Debugger 29
Getting ready 29
How to do it... 29
How it works... 31
There's more... 31
See also 31
Taking a snapshot of the virtual machines 32
Getting ready 32
How to do it... 32
How it works... 33
There's more... 33
See also 34
Chapter 2: 32-bit Assembly on Linux and the ELF Specification 35
Technical requirements 36
Differences between Intel and AT&T syntax 36
Getting ready 36
How to do it... 36
How it works... 37
There's more... 38
See also 39
Introduction to the IA-32 registers 39
Getting ready 40
How to do it... 40
How it works... 42
There's more... 45
See also 46
Introducing common IA-32 instructions 46
Getting ready 46
How to do it... 47
How it works... 49
There's more... 51
See also 51
Making IA-32 system calls on Linux 52
Getting ready 52
How to do it... 52
How it works... 56
There's more... 60
See also 60
Introducing the ELF 32-bit specification 60
[ ii ]
Table of Contents
Getting ready 61
How to do it... 61
How it works... 68
There's more... 74
See also 75
Chapter 3: 64-bit Assembly on Linux and the ELF Specification 76
Technical requirements 76
Introducing the IA64 registers 77
Getting ready 77
How to do it... 77
How it works... 78
There's more... 78
See also 80
Introducing common IA64 instructions 80
Getting ready 81
How to do it... 81
How it works... 83
There's more... 85
See also 85
Making IA64 system calls on Linux 86
Getting ready 86
How to do it... 86
How it works... 88
There's more... 92
See also 92
Introducing the ELF 64-bit specification 93
Getting ready 93
How to do it... 94
How it works... 96
There's more... 99
See also 100
Chapter 4: Creating a Binary Analysis Methodology 101
Technical requirements 102
Performing binary discovery 102
Getting ready 103
How to do it... 103
How it works... 104
There's more... 105
See also 105
Information gathering 106
Getting ready 107
How to do it... 107
How it works... 109
There's more... 111
[ iii ]