Python for Secret Agents - Volume II

180 Pages·2015·1.599 MB·English

by  Lott, Steven

Python for Secret Agents
Volume II
Gather, analyze, and decode data to reveal hidden
facts using Python, the perfect tool for all aspiring
secret agents
Steven F. Lott
BIRMINGHAM - MUMBAI
Python for Secret Agents
Volume II
Copyright © 2015 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval
system, or transmitted in any form or by any means, without the prior written
permission of the publisher, except in the case of brief quotations embedded in
critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy
of the information presented. However, the information contained in this book
is sold without warranty, either express or implied. Neither the author,nor Packt
Publishing, and its dealers and distributors will be held liable for any damages
caused or alleged to be caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the
companies and products mentioned in this book by the appropriate use of capitals.
However, Packt Publishing cannot guarantee the accuracy of this information.
First published: August 2014
Second edition: December 2015
Production reference: 1011215
Published by Packt Publishing Ltd.
Livery Place
35 Livery Street
Birmingham B3 2PB, UK.
ISBN 978-1-78528-340-6
www.packtpub.com
Credits
Author Project Coordinator
Steven F. Lott Sanchita Mandal
Reviewer Proofreader
Shubham Sharma Safis Editing
Commissioning Editor Indexer
Julian Ursell Priya Sane
Acquisition Editors Graphics
Subho Gupta Kirk D'Penha
Sam Wood
Content Development Editor Production Coordinator
Riddhi Tuljapurkar Komal Ramchandani
Technical Editor Cover Work
Danish Shaikh Komal Ramchandani
Copy Editor
Vibha Shukla
About the Author
Steven F. Lott has been programming since the 70s, when computers were large,
expensive, and rare. As a contract software developer and architect, he has worked
on hundreds of projects from very small to very large. He's been using Python to
solve business problems for over 10 years.
He's currently leveraging Python to implement microservices and ETL pipelines.
His other titles with Packt Publishing include Python Essentials, Mastering
Object-Oriented Python, Functional Python Programming, and Python for Secret Agents.
Steven is currently a technomad who lives in various places on the East Coast of the
U.S. His technology blog is http://slott-softwarearchitect.blogspot.com.
About the Reviewer
Shubham Sharma holds a bachelor's degree in computer science engineering
with specialization in business analytics and optimization from UPES, Dehradun.
He has a good skill set of programming languages. He also has an experience in
web development ,Android, and ERP development and works as a freelancer.
Shubham also loves writing and blogs at www.cyberzonec.in/blog. He is currently
working on Python for the optimal specifications and identifications of mobile
phones from customer reviews.
www.PacktPub.com
Support files, eBooks, discount offers, and more
For support files and downloads related to your book, please visit www.PacktPub.com.
Did you know that Packt offers eBook versions of every book published, with PDF
and ePub files available? You can upgrade to the eBook version at www.PacktPub.com
and as a print book customer, you are entitled to a discount on the eBook copy. Get in
touch with us at [email protected] for more details.
At www.PacktPub.com, you can also read a collection of free technical articles, sign
up for a range of free newsletters and receive exclusive discounts and offers on Packt
books and eBooks.
TM
https://www2.packtpub.com/books/subscription/packtlib
Do you need instant solutions to your IT questions? PacktLib is Packt's online digital
book library. Here, you can search, access, and readPackt's entire library of books.
Why subscribe?
• Fully searchable across every book published by Packt
• Copy and paste, print, and bookmark content
• On demand and accessible via a web browser
Free access for Packt account holders
If you have an account with Packt at www.PacktPub.com, you can use this to access
PacktLib today and view 9 entirely free books. Simply use your login credentials for
immediate access.
Table of Contents
Table of Contents
Preface v
Chapter 1: New Missions – New Tools 1
Background briefing on tools 2
Doing a Python upgrade 3
Preliminary mission to upgrade pip 5
Background briefing: review of the Python language 6
Using variables to save results 7
Using the sequence collections: strings 8
Using other common sequences: tuples and lists 11
Using the dictionary mapping 12
Comparing data and using the logic operators 13
Using some simple statements 14
Using compound statements for conditions: if 15
Using compound statements for repetition: for and while 16
Defining functions 17
Creating script files 18
Mission One – upgrade Beautiful Soup 20
Getting an HTML page 21
Navigating the HTML structure 22
Doing other upgrades 24
Mission to expand our toolkit 25
Scraping data from PDF files 26
Sidebar on the ply package 28
Building our own gadgets 28
Getting the Arduino IDE 29
Getting a Python serial interface 31
Summary 32
[ i ]
Table of Contents
Chapter 2: Tracks, Trails, and Logs 33
Background briefing – web servers and logs 34
Understanding the variety of formats 34
Getting a web server log 35
Writing a regular expression for parsing 35
Introducing some regular expression rules and patterns 37
Finding a pattern in a file 38
Using regular expression suffix operators 41
Capturing characters by name 43
Looking at the CLF 44
Reading and understanding the raw data 47
Reading a gzip compressed file 48
Reading remote files 50
Studying a log in more detail 50
What are they downloading? 52
Trails of activity 53
Who is this person? 55
Using Python to run other programs 56
Processing whois queries 57
Breaking a request into stanzas and lines 58
Alternate stanza-finding algorithm 60
Making bulk requests 60
Getting logs from a server with ftplib 61
Building a more complete solution 62
Summary 63
Chapter 3: Following the Social Network 65
Background briefing – images and social media 66
Accessing web services with urllib or http.client 68
Who's doing the talking? 71
Starting with someone we know 74
Finding our followers 76
What do they seem to be talking about? 79
What are they posting? 81
Deep Under Cover – NLTK and language analysis 83
Summary 85
[ ii ]
Table of Contents
Chapter 4: Dredging up History 87
Background briefing–Portable Document Format 88
Extracting PDF content 90
Using generator expressions 90
Writing generator functions 92
Filtering bad data 93
Writing a context manager 94
Writing a PDF parser resource manager 96
Extending the resource manager 97
Getting text data from a document 100
Displaying blocks of text 101
Understanding tables and complex layouts 103
Writing a content filter 105
Filtering the page iterator 107
Exposing the grid 108
Making some text block recognition tweaks 110
Emitting CSV output 111
Summary 112
Chapter 5: Data Collection Gadgets 115
Background briefing: Arduino basics 116
Organizing a shopping list 118
Getting it right the first time 118
Starting with the digital output pins 119
Designing an external LED 121
Assembling a working prototype 124
Mastering the Arduino programming language 126
Using the arithmetic and comparison operators 127
Using common processing statements 128
Hacking and the edit, download, test and break cycle 130
Seeing a better blinking light 130
Simple Arduino sensor data feed 132
Collecting analog data 135
Collecting bulk data with the Arduino 138
Controlling data collection 140
[ iii ]