Python High Performance

264 Pages·2017·4.47 MB·English

by  Gabriele Lanaro

Python High Performance
Second Edition

Build robust applications by implementing concurrent and
distributed processing techniques

Gabriele Lanaro

BIRMINGHAM - MUMBAI
Python High Performance
Second Edition
Copyright © 2017 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or
transmitted in any form or by any means, without the prior written permission of the
publisher, except in the case of brief quotations embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the
information presented. However, the information contained in this book is sold without
warranty, either express or implied. Neither the author, nor Packt Publishing, and its
dealers and distributors will be held liable for any damages caused or alleged to be caused
directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the
companies and products mentioned in this book by the appropriate use of capitals.
However, Packt Publishing cannot guarantee the accuracy of this information.
First published: December 2013
Second edition: May 2017
Production reference: 2250517
Published by Packt Publishing Ltd.
Livery Place
35 Livery Street
Birmingham
B3 2PB, UK.
ISBN 978-1-78728-289-6
www.packtpub.com
Credits
Author Copy Editor
Gabriele Lanaro Shaila Kusanale
Reviewer Project Coordinator
Will Brennan Ulhas Kambali
Commissioning Editor Proofreader
Kunal Parikh Safis Editing
Acquisition Editor Indexer
Chaitanya Nair Tejal Daruwale Soni
Content Development Editor Graphics
Vikas Tiwari Abhinash Sahu
Technical Editor Production Coordinator
Jijo Maliyekal Shantanu Zagade
About the Author
Dr. Gabriele Lanaro has been conducting research to study the formation and growth of
crystals using medium and large-scale computer simulations. In 2017, he obtained his PhD
in theoretical chemistry. His interests span machine learning, numerical computing
visualization, and web technologies. He has a sheer passion for good software and is the
author of the chemlab and chemview open source packages. In 2013, he authored the first
edition of the book “High Performance Python Programming”.
I'd like to acknowledge the support from Packt editors, including Vikas Tiwari. I would also
like to thank my girlfriend, Harani, who had to tolerate the way-too-long writing nights,
and friends who provided company and support throughout. Also, as always, I’d love to
thank my parents for giving me the opportunity to pursue my ambitions.
Lastly, I would like to thank Blenz coffee for powering the execution engine of this book
through electricity and caffeine.
About the Reviewer
Will Brennan is a C++/Python developer based in London with previous experience in
writing molecular dynamics simulations. He is currently working on high-performance
image processing and machine learning applications. You can refer to his repositories at htt

ps://github.com/WillBrennan.
www.PacktPub.com
For support files and downloads related to your book, please visit www.PacktPub.com.
Did you know that Packt offers eBook versions of every book published, with PDF and
ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a
print book customer, you are entitled to a discount on the eBook copy. Get in touch with us
at [email protected] for more details.
At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a
range of free newsletters and receive exclusive discounts and offers on Packt books and
eBooks.
https://www.packtpub.com/mapt

Get the most in-demand software skills with Mapt. Mapt gives you full access to all Packt
books and video courses, as well as industry-leading tools to help you plan your personal
development and advance your career.
Why subscribe?
Fully searchable across every book published by Packt
Copy and paste, print, and bookmark content
On demand and accessible via a web browser
Customer Feedback
Thanks for purchasing this Packt book. At Packt, quality is at the heart of our editorial
process. To help us improve, please leave us an honest review on this book's Amazon page
at https://www.amazon.com/dp/1787282899.

If you'd like to join our team of regular reviewers, you can e-mail us at
[email protected]. We award our regular reviewers with free eBooks and
videos in exchange for their valuable feedback. Help us be relentless in improving our
products!
Table of Contents
Preface
1
Chapter 1: Benchmarking and Profiling
7
Designing your application 8
Writing tests and benchmarks 14
Timing your benchmark 16
Better tests and benchmarks with pytest-benchmark 19
Finding bottlenecks with cProfile 22
Profile line by line with line_profiler 27
Optimizing our code 29
The dis module 31
Profiling memory usage with memory_profiler 32
Summary 35
Chapter 2: Pure Python Optimizations
36
Useful algorithms and data structures 37
Lists and deques 38
Dictionaries 40
Building an in-memory search index using a hash map 42
Sets 44
Heaps 45
Tries 47
Caching and memoization 49
Joblib 52
Comprehensions and generators 53
Summary 55
Chapter 3: Fast Array Operations with NumPy and Pandas
56
Getting started with NumPy 57
Creating arrays 57
Accessing arrays 59
Broadcasting 64
Mathematical operations 67
Calculating the norm 68
Rewriting the particle simulator in NumPy 68
Reaching optimal performance with numexpr 72
Pandas 74
Pandas fundamentals 74
Indexing Series and DataFrame objects 76
Database-style operations with Pandas 78
Mapping 79
Grouping, aggregations, and transforms 81
Joining 83
Summary 84
Chapter 4: C Performance with Cython
86
Compiling Cython extensions 86
Adding static types 89
Variables 89
Functions 91
Classes 92
Sharing declarations 94
Working with arrays 95
C arrays and pointers 95
NumPy arrays 98
Typed memoryviews 99
Particle simulator in Cython 101
Profiling Cython 105
Using Cython with Jupyter 109
Summary 112
Chapter 5: Exploring Compilers
114
Numba 115
First steps with Numba 115
Type specializations 117
Object mode versus native mode 118
Numba and NumPy 121
Universal functions with Numba 121
Generalized universal functions 123
JIT classes 126
Limitations in Numba 129
The PyPy project 130
Setting up PyPy 131
Running a particle simulator in PyPy 132
Other interesting projects 133
Summary 134
Chapter 6: Implementing Concurrency
135
[ ii ]