Open Source Intelligence Tools and Resources Handbook

510 Pages·2020·4.228 MB·English

by  i-intelligence

OPEN SOURCE INTELLIGENCE
TOOLS AND RESOURCES HANDBOOK
2020
OPEN SOURCE INTELLIGENCE

TOOLS AND RESOURCES HANDBOOK

2020

Aleksandra Bielska

Noa Rebecca Kurz, Yves Baumgartner, Vytenis Benetis

2
Foreword

I am delighted to share with you the 2020 edition of the OSINT Tools and Resources
Handbook. Once again, the Handbook has been revised and updated to reflect the evolution
of this discipline, and the many strategic, operational and technical challenges OSINT
practitioners have to grapple with.

Given the speed of change on the web, some might question the wisdom of pulling together
such a resource. What’s wrong with the Top 10 tools, or the Top 100? There are only so many
resources one can bookmark after all. Such arguments are not without merit. My fear,
however, is that they are also shortsighted. I offer four reasons why.

To begin, a shortlist betrays the widening spectrum of OSINT practice. Whereas OSINT was
once the preserve of analysts working in national security, it now embraces a growing class
of professionals in fields as diverse as journalism, cybersecurity, investment research, crisis
management and human rights. A limited toolkit can never satisfy all of these constituencies.

Second, a good OSINT practitioner is someone who is comfortable working with different
tools, sources and collection strategies. The temptation toward narrow specialisation in
OSINT is one that has to be resisted. Why? Because no research task is ever as tidy as the
customer’s requirements are likely to suggest.

Third, is the inevitable realisation that good tool awareness is equivalent to good source
awareness. Indeed, the right tool can determine whether you harvest the right information. It
follows that the more tools you have in your portfolio, the more flexible your OSINT
capabilities are likely to be.

Finally, the process of compiling this handbook is an intelligence exercise in its own right,
alerting us to where we are at as a community and the challenges we are likely to face going
forward – not least of which are disinformation, the fracturing of the internet, the proliferation
of niche social media platforms, and the urgent need for better tools to monitor and analyse
the content they provide.

Our hope is that this Handbook helps you plug any gaps in your collection efforts and alerts
you to the many tools and resources you can consult for better intelligence, including on the
discipline of OSINT itself. Of course, the Handbook makes no claim to being complete.
Something new will doubtless pop-up tomorrow, and I suspect there are hundreds of tools
we haven’t even come across at all. With your help, we can address any omissions.

In any event, the usual caveats apply. While every effort has been made to test and validate
these resources, we caution that today’s wonder tool may be tomorrow’s security liability.
Thus, whenever testing a new script, application or browser extension, please do so in a
secure environment to minimise any risk to your assets or operations.

3
I would like to thank my colleagues at i-intelligence for their efforts in compiling this
Handbook. Much of the credit belongs to Aleksandra Bielska who, as our Head of Training,
evaluates hundreds of tools every year to determine which ones make it into our courses. I
am also indebted to Yves Baumgartner and Vytenis Benetis for routinely updating us on new
solutions. Thanks too are also due to Noa Kurz who worked tirelessly over the summer to
help us get this document ready.

I close by acknowledging the contributions of the many students, practitioners and Twitter
followers we’ve interacted with over the past two years. Your generosity, insights, suggestions
and support are much appreciated.

Chris Pallaris
Director, i-intelligence GmbH

4
Table of Contents

Foreword ............................................................................................................................. 3
Search Tools ..................................................................................................................... 17
General Search .......................................................................................................................... 17
Meta Search ............................................................................................................................... 18
Visual Search and Clustering Search Engines ......................................................................... 20
Decentralised Search Engines .................................................................................................. 21
Charitable Search Engines ........................................................................................................ 22
Code Search .............................................................................................................................. 23
File and FTP Search ................................................................................................................... 25
Open Database, Directories and Services Search .................................................................. 26
Similar Sites Search ................................................................................................................... 27
Children-Friendly Search Engines ............................................................................................ 28
National Search Engine Directories ......................................................................................... 29
Custom Search ........................................................................................................................... 30
Specialty Search Engines .......................................................................................................... 31
Search Tools ............................................................................................................................... 32
Social Media ..................................................................................................................... 34
Major Social Networks .............................................................................................................. 34
Social Media Search and Monitoring ...................................................................................... 37
Working with Hashtags ............................................................................................................. 39
Finding Influencers and Other Social Media Users ................................................................ 41
Social Media Management and Content Discovery ............................................................... 42
Social Media Analytics .............................................................................................................. 43
Social Media Tools: Twitter ...................................................................................................... 44
Social Media Tools: Facebook ................................................................................................. 49
Social Media Tools: Instagram ................................................................................................. 51
5
Social Media Tools: Pinterest ................................................................................................... 55
Social Media Tools: Reddit ....................................................................................................... 56
Social Media Tools: Voat .......................................................................................................... 59
Social Media Tools: VKontakte ................................................................................................. 60
Social Media Tools: LinkedIn .................................................................................................... 61
Social Media Tools: Github ...................................................................................................... 63
Social Media Tools: TikTok ....................................................................................................... 65
Messaging and Chat Tools ....................................................................................................... 66
Social Media Tools: Snapchat .................................................................................................. 68
Social Media Tools: WhatsApp ................................................................................................ 69
Social Media Tools: Skype ........................................................................................................ 70
Social Media Tools: Kik ............................................................................................................. 71
Social Media Tools: Telegram .................................................................................................. 72
Social Media Tools: IRC ............................................................................................................ 74
Dating Apps and Sites .............................................................................................................. 75
Fitness Apps and Communities ............................................................................................... 76
Working with Blogs .......................................................................................................... 77
Blog Search ................................................................................................................................ 77
Working with Forums ....................................................................................................... 78
Forum, Newsgroup and Discussion Board Search Tools ....................................................... 78
Working with Chan Boards .............................................................................................. 79
Chan Search Tools ..................................................................................................................... 79
Working with Q&A Sites .................................................................................................. 80
Q&A Sites ................................................................................................................................... 80
People Investigations ....................................................................................................... 81
People Search Tools ................................................................................................................. 81
Name Research .......................................................................................................................... 84
Public Records ........................................................................................................................... 85
Document Checking ................................................................................................................. 87
Ancestry Research ..................................................................................................................... 88
Missing People .......................................................................................................................... 89
Address and Contact Information Search ............................................................................... 91
Contact Detail Recovery Apps ................................................................................................. 93
6
Investigating Phone Numbers .................................................................................................. 95
Cell Tower Search ...................................................................................................................... 99
E-mail Search ........................................................................................................................... 100
Validating E-mails .................................................................................................................... 102
Analysing E-mails ..................................................................................................................... 104
Analysing E-mail Headers ....................................................................................................... 105
Matching E-mails to Social Media Profiles ............................................................................ 106
IP Grabbers and E-mail Trackers ............................................................................................ 107
Username Check ..................................................................................................................... 108
CV and Resume Search ........................................................................................................... 110
Expert Search ........................................................................................................................... 111
Business Research .......................................................................................................... 113
Company Search Tools ........................................................................................................... 113
Business Registers Lists and Directories ................................................................................ 117
Job Search Resources ............................................................................................................. 118
Company Reviews ................................................................................................................... 120
Salary Search ............................................................................................................................ 121
Patent Research ....................................................................................................................... 123
Trademark Search .................................................................................................................... 125
Tenders ..................................................................................................................................... 127
Charity, NGO, and Nonprofit Research ................................................................................. 129
Researching Leaks ................................................................................................................... 131
Real Estate Research ...................................................................................................... 132
Real Estate Research ............................................................................................................... 132
Online Marketplaces ...................................................................................................... 135
Classifieds and Online Marketplaces ..................................................................................... 135
Cryptocurrency and Banking Research ........................................................................ 138
Cryptocurrency Research ........................................................................................................ 138
Banking and Credit Card Information ................................................................................... 141
Currency Conversion ............................................................................................................... 143
Legal Research ............................................................................................................... 144
Legal Research Tools and Resources .................................................................................... 144

7
Researching Countries ................................................................................................... 147
Country Profiles ....................................................................................................................... 147
Researching Terrorism, Crime, and Cybersecurity ...................................................... 148
Researching Terrorism ............................................................................................................ 148
Researching Crime .................................................................................................................. 151
Researching Drug Trade ......................................................................................................... 153
Researching Money Laundering and Corruption ................................................................. 154
Researching Human Trafficking ............................................................................................. 155
Requests for User Information: Information for Law Enforcement ...................................... 158
Researching Cybersecurity ..................................................................................................... 159
Military Research ..................................................................................................................... 161
Crisis Alerts and Travel Advisory ............................................................................................ 163
Researching Stolen Items .............................................................................................. 164
Researching Stolen Items ....................................................................................................... 164
Transportation ................................................................................................................ 166
Airplanes, Airports and Flight Trackers ................................................................................. 166
Drones ...................................................................................................................................... 169
Cars ........................................................................................................................................... 171
Railways .................................................................................................................................... 173
Public Transport ....................................................................................................................... 176
Ships ......................................................................................................................................... 178
Container and Cargo Tracking ............................................................................................... 181
Web Intelligence ............................................................................................................ 182
Researching Domains ............................................................................................................. 182
IP / IPv6 ..................................................................................................................................... 187
SEO Tools ................................................................................................................................ 191
Analysing Unique IDs .............................................................................................................. 193
Subdomain Enumeration ........................................................................................................ 194
Identifying Backlinks ................................................................................................................ 195
Site Mapping Tools ................................................................................................................. 196
Identifying Publish Date .......................................................................................................... 197
Extracting Files ........................................................................................................................ 198
Analysing URLs ........................................................................................................................ 199
8
Identifying Technologies Behind Websites .......................................................................... 200
Finding SSL Certificates .......................................................................................................... 201
Checking Website Status ........................................................................................................ 202
HTTPS Status ........................................................................................................................... 203
Researching Cyber Threats ..................................................................................................... 204
IoT Search Engines .................................................................................................................. 208
Exploit Search Engines ........................................................................................................... 209
General Webint Tools and Toolkits ....................................................................................... 210
Multipurpose OSINT Frameworks ......................................................................................... 212
Web History and Site Capture ............................................................................................... 214
Wifi Search ............................................................................................................................... 217
Dark Web ........................................................................................................................ 218
Researching the Dark Web ..................................................................................................... 218
Alternatives to Tor ................................................................................................................... 220
Event Search ................................................................................................................... 221
Event Search ............................................................................................................................ 221
Working with Images ..................................................................................................... 222
Image Search and Reverse Image Search ............................................................................. 222
Searching within Screenshots ................................................................................................. 225
Facial Recognition ................................................................................................................... 226
Logo Search and Identification Tools .................................................................................... 227
Investigating Objects and Symbols ....................................................................................... 228
Image Analysis ......................................................................................................................... 229
Stock Images ............................................................................................................................ 232
Image and Photo Editing ........................................................................................................ 234
Creating World Clouds ........................................................................................................... 237
OCR and Image Translation Tools ......................................................................................... 238
Other Image Processing Tools ............................................................................................... 239
Working with Videos ...................................................................................................... 240
Video Search and Reverse Video Search ............................................................................... 240
Live Streaming and Webcams ................................................................................................ 242
YouTube Tools ......................................................................................................................... 245
General Video Downloaders and Archivers .......................................................................... 247
9
Downloading Videos from YouTube ..................................................................................... 249
Downloading Videos from Twitter ......................................................................................... 250
Downloading Videos from Facebook .................................................................................... 251
Downloading Videos from Instagram .................................................................................... 252
Downloading Videos from TikTok .......................................................................................... 253
Downloading Videos from Pinterest ...................................................................................... 254
Downloading Videos from Reddit .......................................................................................... 255
Video Players ........................................................................................................................... 256
Video Editors and Converters ................................................................................................ 257
Working with Video Metadata ............................................................................................... 259
Working with GIFs and Memes ..................................................................................... 260
Finding GIFs and Memes ........................................................................................................ 260
Working with Audio ....................................................................................................... 261
Working with Radio ................................................................................................................. 261
Podcasts ................................................................................................................................... 263
Speech to Text Tools .............................................................................................................. 265
Text to Speech Tools .............................................................................................................. 266
Audio Editors ........................................................................................................................... 267
Working with Audio Metadata ............................................................................................... 268
3D Models and Designs ................................................................................................ 269
3D Models and Designs .......................................................................................................... 269
Working with Documents .............................................................................................. 270
Searching Web-based Documents ........................................................................................ 270
Searching Personal Document Collections ........................................................................... 271
Searching for Slides and Presentations ................................................................................. 272
Searching Pastebins ................................................................................................................ 273
Document and Reference Management ............................................................................... 274
PDF Management ................................................................................................................... 275
Working with Document Metadata ....................................................................................... 277
Academic Resources and Grey Literature .................................................................... 278
Academic and Grey Literature Research Tools ..................................................................... 278
Medical Search Engines .......................................................................................................... 284

10