Using Probabilistic Techniques To Aid In Password Cracking Attacks

140 Pages·2015·2.71 MB·English

by  Charles Matthew Weir

Florida State University Libraries
Electronic Theses, Treatises and Dissertations The Graduate School
2010
Using Probabilistic Techniques to Aid in
Password Cracking Attacks
Charles Matthew Weir
Follow this and additional works at the FSU Digital Library. For more information, please contact [email protected]
THE FLORIDA STATE UNIVERSITY

COLLEGE OF ARTS AND SCIENCES

USING PROBABILISTIC TECHNIQUES TO AID IN PASSWORD CRACKING
ATTACKS

By

CHARLES MATTHEW WEIR

A Dissertation submitted to the
Department of Computer Science
in partial fulfillment of the
requirements for the degree of
Doctor of Philosophy

Degree Awarded:
Spring Semester, 2010

Copyright © 2010
Charles Matthew Weir
All Rights Reserve
The members of the committee approve the dissertation of Charles Matthew Weir defended on
March 31, 2010.

__________________________________
Sudhir Aggarwal
Professor Directing Dissertation

__________________________________
Washington Mio
University Representative

__________________________________
Breno De Medeiros
Committee Member

__________________________________
Mike Burmester
Committee Member

__________________________________
Xiuwen Liu
Committee Member

The Graduate School has verified and approved the above-named committee members.
ii
To my parents:
I’d have a longer dedication but they told me
to focus on the rest of this paper instead…
Thanks for all the good advice.

iii
ACKNOWLEDGEMENTS

This research would not have been possible without the help, support and advice provided to me
by Professor Aggarwal, and Professor De Medeiros. They are the reason why I use the word “we” more
than “I”. The original idea to use probabilistic context free grammars to represent password creation
was theirs, and I can‟t count the number of hours they spent with me as we developed new techniques
to move it from theory to reality. They challenged me to be my best and then provided the freedom for
me to explore new avenues of research. In short, I was extremely fortunate to work with them, and I
plan on continuing our collaborations in the future.
I‟d also like to thank Bill Glodek, another graduate student, for his assistance. I had the pleasure
of working with him on password cracking for a year before he graduated, and his research showed the
validity of using probabilistic grammars to attack password creation strategies.
Finally, I would like to thank the National Institute of Justice for funding this research, along with
the Florida Department of Law Enforcement and National White Collar Crime Center with providing
support and direction. When explaining my research to other people, the most frequent question I‟m
asked is, “Why are you trying to improve password cracking techniques?” It gives me great pride to be
able to respond that my work is helping to catch and prosecute criminals whose crimes range from child
exploitation to insider trading. This focus on helping to solve real world crimes by assisting law
enforcement bypass strong encryption has been one of the driving forces behind my research.

iv
TABLE OF CONTENTS

List of Tables ................................................................................................................ viii
List of Figures ................................................................................................................ x
Abstract ................................................................................................................ xiii

1. Introduction ................................................................................................................ 1
1.1 Overview of Password Cracking Research Detailed in this Paper ....................... 1
1.2 Overview of a Password Cracking Session ......................................................... 3
1.3 Password Hashes and Salts ................................................................................ 4
1.3.1 The UNIX Crypt Family of Hashes .............................................................. 7
1.3.2 The LANMAN Hash ..................................................................................... 7
1.3.3 The NTLM Hash .......................................................................................... 8
1.3.4 The WPA and WPA2 Hashes ...................................................................... 8
1.3.5 Website Password Hashes ......................................................................... 9
1.4 Existing Password Cracking Tools ...................................................................... 9
1.4.1 John the Ripper ........................................................................................... 10
1.4.2 Cain & Able ................................................................................................. 11
1.4.3 L0phtCrack .................................................................................................. 12
1.4.4 Elcomsoft Distributed Password Recovery .................................................. 13
1.4.5 AccessData Password Recovery Toolkit ..................................................... 13
1.5 Obtaining the Datasets ........................................................................................ 14
1.6 Description of Datasets ........................................................................................ 19
1.6.1 The MySpace List........................................................................................ 20
1.6.2 The PhpBB List ........................................................................................... 21
1.6.3 The Hotmail List .......................................................................................... 22
1.6.4 The 78k Finnish List .................................................................................... 23
1.6.1 The RockYou List ........................................................................................ 25

2. Applying Probability to Brute Force Attacks .................................................................. 26
2.1 Description of Brute Force Techniques................................................................ 26
2.1.1 Pure Brute Force ......................................................................................... 26
2.1.2 Letter Frequency Analysis ........................................................................... 27
2.1.3 Markov Models ............................................................................................ 29
2.1.4 Targeted Brute Force .................................................................................. 34
2.2 Comparative Results ........................................................................................... 37

3. Evaluating Dictionary Based Attacks ............................................................................ 42
3.1 Overview .............................................................................................................. 42
3.2 Creating Input Dictionaries .................................................................................. 43
3.3 Using Edit Distance to Estimate Coverage .......................................................... 44
3.4 Identifying Word Mangling Rules ......................................................................... 47
v
4. Dictionary Based Rainbow Tables ................................................................................ 52
4.1 Overview of Rainbow Tables ............................................................................... 52
4.2 Rainbow Table Construction ................................................................................ 53
4.3 Rainbow Table Collisions .................................................................................... 55
4.4 Previous Work ..................................................................................................... 56
4.5 Indexing Dictionary Based Rules ......................................................................... 59
4.5.1 Overview of Drcrack .................................................................................... 59
4.5.2 Performing a Plain Dictionary Based Guess ............................................... 62
4.5.3 Appending/Pre-pending a Value ................................................................. 62
4.5.4 Case Mangling Rules .................................................................................. 63
4.5.5 Letter Replacement Rules ........................................................................... 64
4.6 Results ................................................................................................................ 65

5. Using Probabilistic Grammas for Password Cracking................................................... 71
5.1 Overview of Probabilistic Password Cracking ...................................................... 71
5.2 The Mechanics of Probabilistic Password Cracking ............................................ 73
5.2.1 Using Probabilistic Grammars ..................................................................... 73
5.2.2 Basic Preprocessing .................................................................................... 77
5.2.3 Case and Dictionary Preprocessing ............................................................ 81
5.2.4 Probability Smoothing ................................................................................. 83
5.2.5 Effectively Generating a Next Function ....................................................... 84
5.2.6 Proof of Correctness ................................................................................... 87
5.2.7 The Deadbeat Dad Algorithm ...................................................................... 89
5.3 Experiments and Results ..................................................................................... 92
5.3.1 Description of Input Dictionaries .................................................................. 93
5.3.2 Password Cracking Results ........................................................................ 94
5.3.3 Time and Memory Requirements ................................................................ 98
5.4 Future Work ......................................................................................................... 103

6. Evaluating Entropy as a Metric for the Strength of Passwords ..................................... 106
6.1 The NIST Model of Password Entropy................................................................. 106
6.2 Evaluating the NIST Model of Entropy against Password
Cracking Attacks ........................................................................................................ 109

7. Conclusion ................................................................................................................ 118

vi
APPENDICES ................................................................................................................ 119

A Pseudo-Code for the Probabilistic Password Cracker
Next Algorithm ....................................................................................... ...... 119
B Pseudo-Code for the Probabilistic Password Cracker
Deadbeat Dad Algorithm ....................................................................... ...... 120

REFERENCES ......................................................................................................... ...... 121

BIOGRAPHICAL SKETCH ......................................................................................... ...... 126

vii
LIST OF TABLES

Table 1.3.1: Demonstration of Password Salts ................................................................... 6

Table 1.6.1.1: Password Characteristics of the MySpace Training List .............................. 20

Table 1.6.2.1: Password Characteristics of the Cracked PhpBB List ................................. 22

Table 1.6.3.1: Password Characteristics of the Hotmail Training List ................................. 23

Table 1.6.4.1: Password Characteristics of the Finnish78k Training List ............................ 24

Table 1.6.4.2: Password Characteristics of the Finnish78k MD5 Cracked Passwords ....... 24

Table 1.6.5.1: Password Characteristics of the RockYou32 training list ............................. 25

Table 2.1.1.1: Example Character Sets for Pure Brute Force Attacks ................................ 27

Table 2.1.2.1: Letter Frequency Analysis of Various Password Lists ................................. 29

Table 3.3.1: Example Modified Edit Distance Rules ........................................................... 46

Table 3.4.1: Matches between Common Input Dictionaries and the MySpace
Training Set ........................................................................................................................ 47

Table 3.4.2: Top 10 Rules Found in the MySpace Training Set Using Edit Distance ......... 49

Table 3.4.3: Interesting Rules Found in the MySpace Training Set Using Edit Distance .... 50

Table 4.1.1: Example Hash Lookup Table .......................................................................... 52

Table 4.2.1 Cracking a Hash Using Rainbow Tables with a Chain Length of Four ............. 55

Table 4.5.3.1: Index Size Created by Appending Keyboard Combinations to Each
Other ............................................................................................................................... 63

Table 4.6.1: Publicly Available Rainbow Tables for the NTLM Password Hash ................. 65

Table 4.6.2: Results of Rainbow Table Attacks against 100 PhpBB MD5 Password
Hashes ............................................................................................................................... 68

Table 4.6.3: Reasons Why the Dictionary Based Table did not Crack Passwords
from the PhpBB List ............................................................................................................ 69

Table 5.1.1: First Ten Word Mangling Rules in John the Ripper ........................................ 71
viii
Table 5.2.1.1: Example probabilistic context-free grammar ................................................ 74

Table 5.2.1.2: Listing of different grammar structures ........................................................ 75

Table 5.2.2.1: Listing of Different String Types ................................................................... 78

Table 5.2.2.2: Probabilities of One-Digit Numbers .............................................................. 79

Table 5.2.2.3: Probabilities of Top 10 Two-Digit Numbers.................................................. 79

Table 5.2.2.4: Probabilities of Top 10 Base Structures ....................................................... 80

Table 5.2.3.1: Probabilities of Top 5 Case Mangling Masks for Six Character Alpha
Strings ............................................................................................................................... 81

Table 5.2.5.1: Initial Priority Queue for the Grammar in Table 5.2.1.1 ................................ 86

Table 5.2.5.2: Priority Queue after the First Entry is Popped .............................................. 86

Table 5.3.2.1: Size of Input Dictionaries ............................................................................. 94

Table 5.3.3.1: Size of the Stored Grammar Training Set .................................................... 99

Table5.3.3.2: Total Search Space ...................................................................................... 100

Table 6.2.1: Composition of the Different RockYou1 Sub Lists .......................................... 111

ix