Mastering Malware Analysis: A malware analyst's practical guide to combating malicious software, APT, cybercrime, and IoT attacks

573 Pages·2022·39.182 MB·English

by  Alexey Kleymenov, Amr Thabet

Mastering Malware Analysis
Second Edition
A malware analyst's practical guide to combating malicious
software, APT, cybercrime, and IoT attacks
Alexey Kleymenov
Amr Thabet
BIRMINGHAM—MUMBAI
Mastering Malware Analysis
Second Edition
Copyright © 2022 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted
in any form or by any means, without the prior written permission of the publisher, except in the case
of brief quotations embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the information
presented. However, the information contained in this book is sold without warranty, either express
or implied. Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable
for any damages caused or alleged to have been caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the companies and
products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot
guarantee the accuracy of this information.
Group Product Manager: Mohd Riyan Khan
Publishing Product Manager: Mohd Riyan Khan
Content Development Editor: Adrija Mitra
Technical Editor: Nithik Cheruvakodan
Copy Editor: Safis Editing
Project Coordinator: Ashwin Kharwa
Proofreader: Safis Editing
Indexer: Manju Arasan
Production Designer: Ponraj Dhandapani
Marketing Coordinator: Ankita Bhonsle
First published: June 2019
Second edition: September 2022
Production reference: 1010922
Published by Packt Publishing Ltd.
Livery Place
35 Livery Street
Birmingham
B3 2PB, UK.
978-1-80324-024-4
www.packt.com
I dedicate this book to my family and friends – your continuous support means
so much to me.
– Alexey Kleymenov
To my family.
– Amr Thabet
Co n t r i b u t o r s
About the authors
Alexey Kleymenov started working in the information security industry in his second year at university
and now has more than 14 years of practical experience at several international cybersecurity companies.
He is a malware analyst and software developer who is passionate about reverse engineering, automation,
and research. Alexey has taken part in numerous investigations analyzing all types of malicious samples,
has developed various systems to perform threat intelligence activities in the IT, OT, and IoT sectors,
and has authored several patents. Alexey is a member of the (ISC)² organization and holds the CISSP
certification. Finally, he is a founder of the RE and More project, teaching people all over the world
how to perform malware analysis in the most efficient way.
I would like to deeply thank all my family, especially my beloved mom, Olga,
and wife, Anastasia, for all your love and support. Big thanks to Amr, who
turned this project into enjoyable cooperative work. I’m much obliged to the
Packt team for addressing all of our inquiries, and to the readers and reviewers
for their invaluable feedback. Finally, thanks to everyone who contributed to
my personal development, served as an inspiration, or was next to me when I
needed them.
Amr Thabet is a malware researcher and an incident handler with over 10 years of experience. He
has worked in several Fortune 500 companies, including Symantec and Tenable. Currently, he is the
founder of MalTrak, providing real-world in-depth training in malware analysis, incident response,
threat hunting, and red teaming to help the next generation of cybersecurity enthusiasts to build their
careers in cybersecurity.
Amr is also a speaker and trainer at some of the top security conferences all around the world, including
Blackhat, DEFCON, Hack In Paris, and VB Conference. He was also featured in Christian Science
Monitor for his work on Stuxnet.
I’d like to thank my parents for helping me and believing in me throughout this
journey. And a big thanks to my book partner, friend, and former colleague,
Alexey. Without his expertise, hard work, and dedication, this book wouldn’t
have come to light. We put our experience, expertise, and hearts into this work
and we really hope it changes your life and your career as this knowledge once
changed ours.
About the reviewer
Ahmed Neil is a well-known thought leader in the cybersecurity domain whose work focuses on
information security, threat hunting, threat intelligence, malware analysis, and digital forensics. He
also has a passion for academic research in the field of cybersecurity. He holds an MSc in computer
forensics and is currently working at IBM as a cybersecurity engineer (operations).

Table of Contents
Preface xvii
Part 1: Fundamental Theory
1
Cybercrime, APT Attacks, and Research Strategies 3
Why malware analysis? 4 APT attack 13
Malware analysis in collecting threat intelligence 4 Zero-day attack 13
Malware analysis in incident response 5 Fileless malware 14
Malware analysis in threat hunting 5 Choosing your analysis strategy 14
Malware analysis in creating detections 6
Understand your audience 14
Exploring types of malware 6 Answer your audience’s questions 15
A short history of malware development 6 Define your goals 16
Malware categories 7 Avoid unnecessary technical details 16
Naming conventions 10 Example structures 16
Typical analysis workflow 18
The MITRE ATT&CK framework
explained 10 Setting up the environment 18
Basic terminology 10 Choosing the virtualization software 19
Enterprise Matrix 11 Safety features 19
APT and zero-day attacks and fileless Summary 21
malware 13
viii Table of Contents
2
A Crash Course in Assembly and Programming Basics 23
Basics of informatics 24 Basics 53
Numeral systems 24 The instruction set 54
Basic data units and data types 25 Diving deep into PowerPC 56
Bitwise operations 26
Basics 56
Architectures and their assembly 30 The instruction set 58
Registers 30 Covering the SuperH assembly 59
Memory 31
Basics 60
Instructions (CISC and RISC) 32
The instruction set 60
Becoming familiar with x86 (IA-32
Working with SPARC 62
and x64) 34
Basics 62
Registers 34
The instruction set 63
The instruction structure 36
Moving from assembly to high-level
The instruction set 38
programming languages 64
Arguments, local variables, and calling
conventions (in x86 and x64) 42 Arithmetic statements 64
If conditions 66
Exploring ARM assembly 45
While loop conditions 68
Basics 47
Instruction sets 49 Summary 68
Basics of MIPS 52
Part 2: Diving Deep into Windows Malware
3
Basic Static and Dynamic Analysis for x86/x64 71
Working with the PE header structure 72 Static linking 81
Why PE? 72 Dynamic linking 82
Exploring PE’s structure 73 Dynamic link libraries 82
PE+ (x64 PE) 78 Application programming interface (API) 83
PE header analysis tools 79 Using PE header information for
Static and dynamic linking 80 static analysis 84
Table of Contents ix
How to use the PE header for incident handling 84 Setting labels and comments 104
How to use a PE header for threat hunting 85 Differences between OllyDbg and x64dbg 104
PE loading and process creation 87 Debugging malicious services 105
Basic terminology 87 What is a service? 105
Process creation step by step 91 Attaching to services 107
PE file loading step by step 92
Essentials of behavioral analysis 109
WOW64 processes 93
File operations 109
Basics of dynamic analysis using Registry operations 110
OllyDbg and x64dbg 94
Process operations 111
Debugging tools 95 WinAPIs 111
How to analyze a sample with OllyDbg 97 Network activity 112
Types of breakpoints 100 Sandboxes 112
Modifying the program’s execution 102
Summary 114
List strings, APIs, and cross-references 104
4
Unpacking, Decryption, and Deobfuscation 115
Exploring packers 116 Technique 3 – monitoring memory allocated
spaces for unpacked code 130
Exploring packing and encrypting tools 116
Technique 4 – in-place unpacking 132
Identifying a packed sample 117
Technique 5 – searching for and transferring
Technique 1 – using static signatures 118 control to OEP 133
Technique 2 – evaluating PE section names 118 Technique 6 – stack restoration-based 133
Technique 3 – using stub execution signs 119
Dumping the unpacked sample and
Technique 4 – detecting a small import table 119
fixing the import table 134
Automatically unpacking packed
Dumping the process 134
samples 120
Fixing the import table 135
Technique 1 – the official unpacking process 120
Identifying simple encryption
Technique 2 – using OllyScript with OllyDbg 121
algorithms and functions 137
Technique 3 – using generic unpackers 121
Types of encryption algorithms 137
Technique 4 – emulation 121
Basic encryption algorithms 139
Technique 5 – memory dumps 122
Identifying encryption functions in
Manual unpacking techniques 123 disassembly 140
Technique 1 – memory breakpoint on String search detection techniques for simple
execution 123 algorithms 141
Technique 2 – call stack backtracing 126 Identifying the RC4 encryption algorithm 143