Cython: a guide for Python programmers

253 Pages·2015·3.724 MB·English

by  Smith, Kurt W

Cython
Build software that combines Python’s expressivity with the performance “ Cython has proven itself C
and control of C (and C++). It’s possible with Cython, the compiler and in many foundational y
hybrid programming language used by foundational packages such as t
Python projects where h
NumPy, and prominent in projects including Pandas, h5py, and scikits- Cython
performance is critical. o
learn. In this practical guide, you’ll learn how to use Cython to improve
n
Python’s performance—up to 3000x—and to wrap C and C++ libraries in In this book, Kurt Smith
Python with ease.
walks you through
Author Kurt Smith takes you through Cython’s capabilities, with sample everything you need to
code and in-depth practice exercises. If you’re just starting with Cython, or
know to tap into Cython's
want to go deeper, you’ll learn how this language is an essential part of any
power for your own
performance-oriented Python programmer’s arsenal.
projects.”
■ Use Cython’s static typing to speed up Python code
—Robert Bradshaw
■ Gain hands-on experience using Cython features to boost your Lead Cython Developer
numeric-heavy Python A GUIDE FOR PYTHON PROGRAMMERS
■ Create new types with Cython—and see how fast object-
oriented programming in Python can be
■ Effectively organize Cython code into separate modules and
packages without sacrificing performance
■ Use Cython to give Pythonic interfaces to C and C++ libraries
■ Optimize code with Cython’s runtime and compile-time
profiling tools
■ Use Cython’s prange function to parallelize loops transparently
with OpenMP
Kurt W. Smith discovered Cython’s power during his graduate research in com-
putational plasma physics. Today, he uses Cython extensively in his consulting
work at Enthought, training scientists, engineers, and researchers in Python,
NumPy, Cython, and parallel and high-performance computing.
S
m
it
h
PROGRAMMING/PYTHON Twitter: @oreillymedia
facebook.com/oreilly
US $29.99 CAN $34.99
ISBN: 978-1-491-90155-7
Kurt W. Smith
Cython
Build software that combines Python’s expressivity with the performance “ Cython has proven itself C
and control of C (and C++). It’s possible with Cython, the compiler and in many foundational y
hybrid programming language used by foundational packages such as t
Python projects where h
NumPy, and prominent in projects including Pandas, h5py, and scikits- Cython
performance is critical. o
learn. In this practical guide, you’ll learn how to use Cython to improve
n
Python’s performance—up to 3000x—and to wrap C and C++ libraries in In this book, Kurt Smith
Python with ease.
walks you through
Author Kurt Smith takes you through Cython’s capabilities, with sample everything you need to
code and in-depth practice exercises. If you’re just starting with Cython, or
know to tap into Cython's
want to go deeper, you’ll learn how this language is an essential part of any
power for your own
performance-oriented Python programmer’s arsenal.
projects.”
■ Use Cython’s static typing to speed up Python code
—Robert Bradshaw
■ Gain hands-on experience using Cython features to boost your Lead Cython Developer
numeric-heavy Python A GUIDE FOR PYTHON PROGRAMMERS
■ Create new types with Cython—and see how fast object-
oriented programming in Python can be
■ Effectively organize Cython code into separate modules and
packages without sacrificing performance
■ Use Cython to give Pythonic interfaces to C and C++ libraries
■ Optimize code with Cython’s runtime and compile-time
profiling tools
■ Use Cython’s prange function to parallelize loops transparently
with OpenMP
Kurt W. Smith discovered Cython’s power during his graduate research in com-
putational plasma physics. Today, he uses Cython extensively in his consulting
work at Enthought, training scientists, engineers, and researchers in Python,
NumPy, Cython, and parallel and high-performance computing.
S
m
it
h
PROGRAMMING/PYTHON Twitter: @oreillymedia
facebook.com/oreilly
US $29.99 CAN $34.99
ISBN: 978-1-491-90155-7
Kurt W. Smith
Cython
Kurt W. Smith
Cython
by Kurt W. Smith
Copyright © 2015 Kurt W. Smith, PhD. All rights reserved.
Printed in the United States of America.
Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.
O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are
also available for most titles (http://safaribooksonline.com). For more information, contact our corporate/
institutional sales department: 800-998-9938 or [email protected].
Editors: Mike Loukides and Meghan Blanchette Indexer: Ellen Troutman Zaig
Production Editor: Matthew Hacker Cover Designer: Ellie Volckhausen
Copyeditor: Rachel Monaghan Interior Designer: David Futato
Proofreader: Rachel Head Illustrator: Rebecca Demarest
January 2015: First Edition
Revision History for the First Edition:
2015-01-09: First release
See http://oreilly.com/catalog/errata.csp?isbn=9781491901557 for release details.
The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Cython, the cover image of a South African
python, and related trade dress are trademarks of O’Reilly Media, Inc.
While the publisher and the author have used good faith efforts to ensure that the information and instruc‐
tions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors
or omissions, including without limitation responsibility for damages resulting from the use of or reliance
on this work. Use of the information and instructions contained in this work is at your own risk. If any code
samples or other technology this work contains or describes is subject to open source licenses or the intel‐
lectual property rights of others, it is your responsibility to ensure that your use thereof complies with such
licenses and/or rights.
ISBN: 978-1-491-90155-7
[LSI]
For Ellen, Zélie, Leo, and Hugh—my muses.
An inconvenience is only an adventure wrongly considered; an adventure
is an inconvenience rightly considered.
— GKC
A dead thing can go with the stream, but only a living thing can go against it.
— GKC

Table of Contents
Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi
1. Cython Essentials. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Comparing Python, C, and Cython 2
Function Call Overhead 5
Looping 6
Math Operations 6
Stack Versus Heap Allocation 6
Tempering Our Enthusiasm 7
Wrapping C Code with Cython 8
Summary 10
2. Compiling and Running Cython Code. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
The Cython Compilation Pipeline 12
Installing and Testing Our Setup 13
The Standard Way: Using distutils with cythonize 14
Our distutils Script 15
Compiling with distutils on Mac OS X and Linux 15
Compiling with distutils on Windows 16
Using Our Extension Module 17
Interactive Cython with IPython’s %%cython Magic 19
Compiling On-the-Fly with pyximport 21
Controlling pyximport and Managing Dependencies 22
pyximport Example with External Dependencies 23
Rolling Our Own and Compiling by Hand 24
Using Cython with Other Build Systems 26
CMake and Cython 26
SCons and Cython 26
Make and Cython 26
v
Compiler Directives 28
Summary 29
3. Cython in Depth. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
Interpreted Versus Compiled Execution 31
Dynamic Versus Static Typing 32
Static Type Declaration with cdef 34
Automatic Type Inference in Cython 36
C Pointers in Cython 37
Mixing Statically and Dynamically Typed Variables 39
Statically Declaring Variables with a Python Type 41
Static Typing for Speed 43
Reference Counting and Static String Types 45
Cython’s Three Kinds of Functions 46
Python Functions in Cython with the def Keyword 46
C Functions in Cython with the cdef Keyword 49
Combining def and cdef Functions with cpdef 50
Functions and Exception Handling 51
Functions and the embedsignature Compiler Directive 53
Type Coercion and Casting 55
Declaring and Using structs, unions, and enums 56
Type Aliasing with ctypedef 59
Cython for Loops and while Loops 61
Guidelines for Efficient Loops 61
Loop Example 62
The Cython Preprocessor 63
Bridging the Python 2 and Python 3 Divide 64
str, unicode, bytes, and All That 66
Summary 67
4. Cython in Practice: N-Body Simulation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
Overview of the N-Body Python Code 69
Converting to Cython 71
Python Data Structures and Organization 72
Converting Data Structures to structs 73
Running the Cythonized Version 75
Summary 76
5. Cython and Extension Types. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
Comparing Python Classes and Extension Types 79
Extension Types in Cython 80
Type Attributes and Access Control 83
vi | Table of Contents
C-Level Initialization and Finalization 85
cdef and cpdef Methods 86
Inheritance and Subclassing 89
Casting and Subclasses 90
Extension Type Objects and None 91
Extension Type Properties in Cython 92
Special Methods Are Even More Special 94
Arithmetic Methods 94
Rich Comparisons 96
Iterator Support 98
Summary 99
6. Organizing Cython Code. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
Cython Implementation (.pyx) and Declaration (.pxd) Files 102
The cimport Statement 105
Predefined Definition Files 107
Include Files and the include Statement 109
Organizing and Compiling Cython Modules Inside Python Packages 110
Summary 113
7. Wrapping C Libraries with Cython. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
Declaring External C Code in Cython 115
Cython Does Not Automate Wrapping 117
Declaring External C Functions and typedefs 118
Declaring and Wrapping C structs, unions, and enums 119
Wrapping C Functions 121
Wrapping C structs with Extension Types 122
Constants, Other Modifiers, and Controlling What Cython Generates 125
Error Checking and Raising Exceptions 128
Callbacks 128
Callbacks and Exception Propagation 133
Summary 134
8. Wrapping C++ Libraries with Cython. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
Simple Example: MT_RNG Class 135
The Wrapper Extension Type 137
Compiling with C++ 138
Using Our Wrapper from Python 139
Overloaded Methods and Functions 140
Operator Overloading 142
C++ Exceptions 144
Stack and Heap Allocation of C++ Instances 145
Table of Contents | vii
Working with C++ Class Hierarchies 146
C++ Templates 147
Templated Functions and Cython’s Fused Types 148
Templated Classes 149
Iterators and Nested Classes 150
Included STL Container Class Declarations 151
Memory Management and Smart Pointers 154
Summary 157
9. Cython Profiling Tools. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
Cython Runtime Profiling 159
Performance Profiling and Annotations 164
Summary 170
10. Cython, NumPy, and Typed Memoryviews. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
The Power of the New Buffer Protocol 172
The memoryview Type 173
Typed Memoryviews 176
Typed Memoryview Example 176
C-Level Access to Typed Memoryview Data 177
Trading Safety for Performance 178
Declaring Typed Memoryviews 179
Using Typed Memoryviews 183
Beyond Buffers 187
Wrapping C and C++ Arrays 189
Correct (and Automatic) Memory Management with Cython and C Arrays 189
Summary 192
11. Cython in Practice: Spectral Norm. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
Overview of the Spectral Norm Python Code 193
Performance Profiling 196
Cythonizing Our Code 197
Adding Static Type Information 198
Using Typed Memoryviews 198
Comparing to the C Implementation 200
Summary 200
12. Parallel Programming with Cython. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
Thread-Based Parallelism and the Global Interpreter Lock 201
The nogil Function Attribute 202
The with nogil Context Manager 203
Using prange to Parallelize Loops 204
viii | Table of Contents