Fluent Python

1.93 MB·English

by  Luciano Ramalho

Preface Page: 5
Who This Book Is For Page: 5
Who This Book Is Not For Page: 5
Five Books in One Page: 5
How the Book Is Organized Page: 5
Hands-On Approach Page: 6
Soapbox: My Personal Perspective Page: 6
Companion Website: fluentpython.com Page: 6
Conventions Used in This Book Page: 6
Using Code Examples Page: 7
O’Reilly Online Learning Page: 7
How to Contact Us Page: 7
Acknowledgments Page: 7
Acknowledgments for the First Edition Page: 8
I. Data Structures Page: 9
1. The Python Data Model Page: 10
What’s New in This Chapter Page: 10
A Pythonic Card Deck Page: 10
How Special Methods Are Used Page: 11
Emulating Numeric Types Page: 12
String Representation Page: 12
Boolean Value of a Custom Type Page: 13
Collection API Page: 13
Overview of Special Methods Page: 13
Why len Is Not a Method Page: 14
Chapter Summary Page: 14
Further Reading Page: 14
2. An Array of Sequences Page: 16
What’s New in This Chapter Page: 16
Overview of Built-In Sequences Page: 16
List Comprehensions and Generator Expressions Page: 17
List Comprehensions and Readability Page: 17
Listcomps Versus map and filter Page: 17
Cartesian Products Page: 17
Generator Expressions Page: 18
Tuples Are Not Just Immutable Lists Page: 18
Tuples as Records Page: 18
Tuples as Immutable Lists Page: 19
Comparing Tuple and List Methods Page: 19
Unpacking Sequences and Iterables Page: 20
Using * to Grab Excess Items Page: 20
Unpacking with * in Function Calls and Sequence Literals Page: 20
Nested Unpacking Page: 20
Pattern Matching with Sequences Page: 21
Pattern Matching Sequences in an Interpreter Page: 22
Slicing Page: 24
Why Slices and Ranges Exclude the Last Item Page: 24
Slice Objects Page: 24
Multidimensional Slicing and Ellipsis Page: 24
Assigning to Slices Page: 25
Using + and * with Sequences Page: 25
Building Lists of Lists Page: 25
Augmented Assignment with Sequences Page: 25
A += Assignment Puzzler Page: 26
list.sort Versus the sorted Built-In Page: 26
When a List Is Not the Answer Page: 27
Arrays Page: 27
Memory Views Page: 28
NumPy Page: 29
Deques and Other Queues Page: 29
Chapter Summary Page: 30
Further Reading Page: 31
3. Dictionaries and Sets Page: 34
What’s New in This Chapter Page: 34
Modern dict Syntax Page: 34
dict Comprehensions Page: 34
Unpacking Mappings Page: 34
Merging Mappings with | Page: 35
Pattern Matching with Mappings Page: 35
Standard API of Mapping Types Page: 36
What Is Hashable Page: 36
Overview of Common Mapping Methods Page: 36
Inserting or Updating Mutable Values Page: 37
Automatic Handling of Missing Keys Page: 37
defaultdict: Another Take on Missing Keys Page: 37
The __missing__ Method Page: 38
Inconsistent Usage of __missing__ in the Standard Library Page: 39
Variations of dict Page: 39
collections.OrderedDict Page: 39
collections.ChainMap Page: 39
collections.Counter Page: 40
shelve.Shelf Page: 40
Subclassing UserDict Instead of dict Page: 40
Immutable Mappings Page: 41
Dictionary Views Page: 41
Practical Consequences of How dict Works Page: 41
Set Theory Page: 42
Set Literals Page: 42
Set Comprehensions Page: 43
Practical Consequences of How Sets Work Page: 43
Set Operations Page: 43
Set Operations on dict Views Page: 44
Chapter Summary Page: 44
Further Reading Page: 45
4. Unicode Text Versus Bytes Page: 47
What’s New in This Chapter Page: 47
Character Issues Page: 47
Byte Essentials Page: 47
Basic Encoders/Decoders Page: 48
Understanding Encode/Decode Problems Page: 49
Coping with UnicodeEncodeError Page: 49
Coping with UnicodeDecodeError Page: 49
SyntaxError When Loading Modules with Unexpected Encoding Page: 50
How to Discover the Encoding of a Byte Sequence Page: 50
BOM: A Useful Gremlin Page: 50
Handling Text Files Page: 51
Beware of Encoding Defaults Page: 52
Normalizing Unicode for Reliable Comparisons Page: 54
Case Folding Page: 55
Utility Functions for Normalized Text Matching Page: 55
Extreme “Normalization”: Taking Out Diacritics Page: 55
Sorting Unicode Text Page: 56
Sorting with the Unicode Collation Algorithm Page: 57
The Unicode Database Page: 57
Finding Characters by Name Page: 57
Numeric Meaning of Characters Page: 58
Dual-Mode str and bytes APIs Page: 58
str Versus bytes in Regular Expressions Page: 58
str Versus bytes in os Functions Page: 59
Chapter Summary Page: 59
Further Reading Page: 59
5. Data Class Builders Page: 62
What’s New in This Chapter Page: 62
Overview of Data Class Builders Page: 62
Main Features Page: 63
Classic Named Tuples Page: 63
Typed Named Tuples Page: 64
Type Hints 101 Page: 65
No Runtime Effect Page: 65
Variable Annotation Syntax Page: 65
The Meaning of Variable Annotations Page: 65
More About @dataclass Page: 66
Field Options Page: 67
Post-init Processing Page: 68
Typed Class Attributes Page: 69
Initialization Variables That Are Not Fields Page: 69
@dataclass Example: Dublin Core Resource Record Page: 69
Data Class as a Code Smell Page: 70
Data Class as Scaffolding Page: 70
Data Class as Intermediate Representation Page: 71
Pattern Matching Class Instances Page: 71
Simple Class Patterns Page: 71
Keyword Class Patterns Page: 71
Positional Class Patterns Page: 71
Chapter Summary Page: 72
Further Reading Page: 72
6. Object References, Mutability, and Recycling Page: 75
What’s New in This Chapter Page: 75
Variables Are Not Boxes Page: 75
Identity, Equality, and Aliases Page: 76
Choosing Between == and is Page: 76
The Relative Immutability of Tuples Page: 76
Copies Are Shallow by Default Page: 77
Deep and Shallow Copies of Arbitrary Objects Page: 77
Function Parameters as References Page: 78
Mutable Types as Parameter Defaults: Bad Idea Page: 78
Defensive Programming with Mutable Parameters Page: 79
del and Garbage Collection Page: 80
Tricks Python Plays with Immutables Page: 80
Chapter Summary Page: 81
Further Reading Page: 81
II. Functions as Objects Page: 84
7. Functions as First-Class Objects Page: 85
What’s New in This Chapter Page: 85
Treating a Function Like an Object Page: 85
Higher-Order Functions Page: 85
Modern Replacements for map, filter, and reduce Page: 86
Anonymous Functions Page: 86
The Nine Flavors of Callable Objects Page: 87
User-Defined Callable Types Page: 87
From Positional to Keyword-Only Parameters Page: 87
Positional-Only Parameters Page: 87
Packages for Functional Programming Page: 88
The operator Module Page: 88
Freezing Arguments with functools.partial Page: 89
Chapter Summary Page: 90
Further Reading Page: 90
8. Type Hints in Functions Page: 92
What’s New in This Chapter Page: 92
About Gradual Typing Page: 92
Gradual Typing in Practice Page: 92
Starting with Mypy Page: 93
Making Mypy More Strict Page: 93
A Default Parameter Value Page: 93
Using None as a Default Page: 94
Types Are Defined by Supported Operations Page: 94
Types Usable in Annotations Page: 96
The Any Type Page: 96
Simple Types and Classes Page: 96
Optional and Union Types Page: 97
Generic Collections Page: 97
Tuple Types Page: 98
Generic Mappings Page: 99
Abstract Base Classes Page: 99
Iterable Page: 100
Parameterized Generics and TypeVar Page: 100
Static Protocols Page: 102
Callable Page: 104
NoReturn Page: 104
Annotating Positional Only and Variadic Parameters Page: 105
Imperfect Typing and Strong Testing Page: 105
Chapter Summary Page: 105
Further Reading Page: 106
9. Decorators and Closures Page: 109
What’s New in This Chapter Page: 109
Decorators 101 Page: 109
When Python Executes Decorators Page: 109
Registration Decorators Page: 110
Variable Scope Rules Page: 110
Closures Page: 111
The nonlocal Declaration Page: 112
Variable Lookup Logic Page: 112
Implementing a Simple Decorator Page: 113
How It Works Page: 113
Decorators in the Standard Library Page: 113
Memoization with functools.cache Page: 113
Using lru_cache Page: 114
Single Dispatch Generic Functions Page: 115
Parameterized Decorators Page: 116
A Parameterized Registration Decorator Page: 116
The Parameterized Clock Decorator Page: 117
A Class-Based Clock Decorator Page: 117
Chapter Summary Page: 118
Further Reading Page: 118
10. Design Patterns with First-Class Functions Page: 121
What’s New in This Chapter Page: 121
Case Study: Refactoring Strategy Page: 121
Classic Strategy Page: 121
Function-Oriented Strategy Page: 122
Choosing the Best Strategy: Simple Approach Page: 123
Finding Strategies in a Module Page: 124
Decorator-Enhanced Strategy Pattern Page: 124
The Command Pattern Page: 125
Chapter Summary Page: 125
Further Reading Page: 126
III. Classes and Protocols Page: 127
11. A Pythonic Object Page: 128
What’s New in This Chapter Page: 128
Object Representations Page: 128
Vector Class Redux Page: 128
An Alternative Constructor Page: 129
classmethod Versus staticmethod Page: 129
Formatted Displays Page: 130
A Hashable Vector2d Page: 131
Supporting Positional Pattern Matching Page: 132
Complete Listing of Vector2d, Version 3 Page: 132
Private and “Protected” Attributes in Python Page: 133
Saving Memory with __slots__ Page: 133
Simple Measure of __slot__ Savings Page: 134
Summarizing the Issues with __slots__ Page: 135
Overriding Class Attributes Page: 135
Chapter Summary Page: 136
Further Reading Page: 136
12. Special Methods for Sequences Page: 139
What’s New in This Chapter Page: 139
Vector: A User-Defined Sequence Type Page: 139
Vector Take #1: Vector2d Compatible Page: 139
Protocols and Duck Typing Page: 140
Vector Take #2: A Sliceable Sequence Page: 141
How Slicing Works Page: 141
A Slice-Aware __getitem__ Page: 141
Vector Take #3: Dynamic Attribute Access Page: 142
Vector Take #4: Hashing and a Faster == Page: 143
Vector Take #5: Formatting Page: 145
Chapter Summary Page: 146
Further Reading Page: 147
13. Interfaces, Protocols, and ABCs Page: 150
The Typing Map Page: 150
What’s New in This Chapter Page: 150
Two Kinds of Protocols Page: 150
Programming Ducks Page: 151
Python Digs Sequences Page: 151
Monkey Patching: Implementing a Protocol at Runtime Page: 152
Defensive Programming and “Fail Fast” Page: 152
Goose Typing Page: 153
Subclassing an ABC Page: 155
ABCs in the Standard Library Page: 156
Defining and Using an ABC Page: 157
ABC Syntax Details Page: 158
Subclassing an ABC Page: 158
A Virtual Subclass of an ABC Page: 159
Usage of register in Practice Page: 159
Structural Typing with ABCs Page: 160
Static Protocols Page: 160
The Typed double Function Page: 161
Runtime Checkable Static Protocols Page: 161
Limitations of Runtime Protocol Checks Page: 161
Supporting a Static Protocol Page: 162
Designing a Static Protocol Page: 162
Best Practices for Protocol Design Page: 163
Extending a Protocol Page: 163
The numbers ABCs and Numeric Protocols Page: 164
Chapter Summary Page: 165
Further Reading Page: 165
14. Inheritance: For Better or for Worse Page: 169
What’s New in This Chapter Page: 169
The super() Function Page: 169
Subclassing Built-In Types Is Tricky Page: 170
Multiple Inheritance and Method Resolution Order Page: 171
Mixin Classes Page: 172
Case-Insensitive Mappings Page: 172
Multiple Inheritance in the Real World Page: 173
ABCs Are Mixins Too Page: 173
ThreadingMixIn and ForkingMixIn Page: 173
Django Generic Views Mixins Page: 174
Multiple Inheritance in Tkinter Page: 175
Coping with Inheritance Page: 175
Favor Object Composition over Class Inheritance Page: 175
Understand Why Inheritance Is Used in Each Case Page: 175
Make Interfaces Explicit with ABCs Page: 176
Use Explicit Mixins for Code Reuse Page: 176
Provide Aggregate Classes to Users Page: 176
Subclass Only Classes Designed for Subclassing Page: 176
Avoid Subclassing from Concrete Classes Page: 176
Tkinter: The Good, the Bad, and the Ugly Page: 176
Chapter Summary Page: 177
Further Reading Page: 177
15. More About Type Hints Page: 180
What’s New in This Chapter Page: 180
Overloaded Signatures Page: 180
Max Overload Page: 180
Takeaways from Overloading max Page: 181
TypedDict Page: 182
Type Casting Page: 184
Reading Type Hints at Runtime Page: 185
Problems with Annotations at Runtime Page: 185
Dealing with the Problem Page: 186
Implementing a Generic Class Page: 186
Basic Jargon for Generic Types Page: 187
Variance Page: 187
An Invariant Dispenser Page: 187
A Covariant Dispenser Page: 188
A Contravariant Trash Can Page: 188
Variance Review Page: 188
Implementing a Generic Static Protocol Page: 189
Chapter Summary Page: 190
Further Reading Page: 190
16. Operator Overloading Page: 194
What’s New in This Chapter Page: 194
Operator Overloading 101 Page: 194
Unary Operators Page: 194
Overloading + for Vector Addition Page: 195
Overloading * for Scalar Multiplication Page: 197
Using @ as an Infix Operator Page: 198
Wrapping-Up Arithmetic Operators Page: 198
Rich Comparison Operators Page: 198
Augmented Assignment Operators Page: 199
Chapter Summary Page: 201
Further Reading Page: 201
IV. Control Flow Page: 204
17. Iterators, Generators, and Classic Coroutines Page: 205
What’s New in This Chapter Page: 205
A Sequence of Words Page: 205
Why Sequences Are Iterable: The iter Function Page: 205
Using iter with a Callable Page: 206
Iterables Versus Iterators Page: 206
Sentence Classes with __iter__ Page: 207
Sentence Take #2: A Classic Iterator Page: 207
Don’t Make the Iterable an Iterator for Itself Page: 208
Sentence Take #3: A Generator Function Page: 208
How a Generator Works Page: 209
Lazy Sentences Page: 210
Sentence Take #4: Lazy Generator Page: 210
Sentence Take #5: Lazy Generator Expression Page: 210
When to Use Generator Expressions Page: 210
An Arithmetic Progression Generator Page: 211
Arithmetic Progression with itertools Page: 212
Generator Functions in the Standard Library Page: 212
Iterable Reducing Functions Page: 216
Subgenerators with yield from Page: 216
Reinventing chain Page: 217
Traversing a Tree Page: 217
Generic Iterable Types Page: 218
Classic Coroutines Page: 219
Example: Coroutine to Compute a Running Average Page: 219
Returning a Value from a Coroutine Page: 220
Generic Type Hints for Classic Coroutines Page: 222
Chapter Summary Page: 222
Further Reading Page: 222
18. with, match, and else Blocks Page: 226
What’s New in This Chapter Page: 226
Context Managers and with Blocks Page: 226
The contextlib Utilities Page: 227
Using @contextmanager Page: 228
Pattern Matching in lis.py: A Case Study Page: 229
Scheme Syntax Page: 229
Imports and Types Page: 230
The Parser Page: 230
The Environment Page: 230
The REPL Page: 231
The Evaluator Page: 231
Procedure: A Class Implementing a Closure Page: 233
Using OR-patterns Page: 234
Do This, Then That: else Blocks Beyond if Page: 234
Chapter Summary Page: 235
Further Reading Page: 235
19. Concurrency Models in Python Page: 239
What’s New in This Chapter Page: 239
The Big Picture Page: 239
A Bit of Jargon Page: 239
Processes, Threads, and Python’s Infamous GIL Page: 240
A Concurrent Hello World Page: 241
Spinner with Threads Page: 241
Spinner with Processes Page: 242
Spinner with Coroutines Page: 243
Supervisors Side-by-Side Page: 244
The Real Impact of the GIL Page: 245
Quick Quiz Page: 245
A Homegrown Process Pool Page: 246
Process-Based Solution Page: 246
Understanding the Elapsed Times Page: 246
Code for the Multicore Prime Checker Page: 247
Experimenting with More or Fewer Processes Page: 248
Thread-Based Nonsolution Page: 248
Python in the Multicore World Page: 248
System Administration Page: 249
Data Science Page: 249
Server-Side Web/Mobile Development Page: 249
WSGI Application Servers Page: 250
Distributed Task Queues Page: 250
Chapter Summary Page: 251
Further Reading Page: 251
Concurrency with Threads and Processes Page: 251
The GIL Page: 252
Concurrency Beyond the Standard Library Page: 252
Concurrency and Scalability Beyond Python Page: 253
20. Concurrent Executors Page: 256
What’s New in This Chapter Page: 256
Concurrent Web Downloads Page: 256
A Sequential Download Script Page: 257
Downloading with concurrent.futures Page: 257
Where Are the Futures? Page: 258
Launching Processes with concurrent.futures Page: 259
Multicore Prime Checker Redux Page: 260
Experimenting with Executor.map Page: 261
Downloads with Progress Display and Error Handling Page: 262
Error Handling in the flags2 Examples Page: 263
Using futures.as_completed Page: 264
Chapter Summary Page: 265
Further Reading Page: 265
21. Asynchronous Programming Page: 267
What’s New in This Chapter Page: 267
A Few Definitions Page: 267
An asyncio Example: Probing Domains Page: 267
Guido’s Trick to Read Asynchronous Code Page: 268
New Concept: Awaitable Page: 268
Downloading with asyncio and HTTPX Page: 269
The Secret of Native Coroutines: Humble Generators Page: 270
The All-or-Nothing Problem Page: 270
Asynchronous Context Managers Page: 270
Enhancing the asyncio Downloader Page: 271
Using asyncio.as_completed and a Thread Page: 271
Throttling Requests with a Semaphore Page: 272
Making Multiple Requests for Each Download Page: 273
Delegating Tasks to Executors Page: 274
Writing asyncio Servers Page: 275
A FastAPI Web Service Page: 275
An asyncio TCP Server Page: 276
Asynchronous Iteration and Asynchronous Iterables Page: 278
Asynchronous Generator Functions Page: 278
Async Comprehensions and Async Generator Expressions Page: 280
async Beyond asyncio: Curio Page: 281
Type Hinting Asynchronous Objects Page: 282
How Async Works and How It Doesn’t Page: 283
Running Circles Around Blocking Calls Page: 283
The Myth of I/O-Bound Systems Page: 283
Avoiding CPU-Bound Traps Page: 283
Chapter Summary Page: 283
Further Reading Page: 284
V. Metaprogramming Page: 286
22. Dynamic Attributes and Properties Page: 287
What’s New in This Chapter Page: 287
Data Wrangling with Dynamic Attributes Page: 287
Exploring JSON-Like Data with Dynamic Attributes Page: 287
The Invalid Attribute Name Problem Page: 289
Flexible Object Creation with __new__ Page: 289
Computed Properties Page: 290
Step 1: Data-Driven Attribute Creation Page: 290
Step 2: Property to Retrieve a Linked Record Page: 291
Step 3: Property Overriding an Existing Attribute Page: 292
Step 4: Bespoke Property Cache Page: 292
Step 5: Caching Properties with functools Page: 293
Using a Property for Attribute Validation Page: 294
LineItem Take #1: Class for an Item in an Order Page: 294
LineItem Take #2: A Validating Property Page: 294
A Proper Look at Properties Page: 295
Properties Override Instance Attributes Page: 295
Property Documentation Page: 296
Coding a Property Factory Page: 296
Handling Attribute Deletion Page: 297
Essential Attributes and Functions for Attribute Handling Page: 297
Special Attributes that Affect Attribute Handling Page: 297
Built-In Functions for Attribute Handling Page: 298
Special Methods for Attribute Handling Page: 298
Chapter Summary Page: 299
Further Reading Page: 299
23. Attribute Descriptors Page: 301
What’s New in This Chapter Page: 301
Descriptor Example: Attribute Validation Page: 301
LineItem Take #3: A Simple Descriptor Page: 301
LineItem Take #4: Automatic Naming of Storage Attributes Page: 303
LineItem Take #5: A New Descriptor Type Page: 304
Overriding Versus Nonoverriding Descriptors Page: 304
Overriding Descriptors Page: 305
Overriding Descriptor Without __get__ Page: 305
Nonoverriding Descriptor Page: 306
Overwriting a Descriptor in the Class Page: 306
Methods Are Descriptors Page: 306
Descriptor Usage Tips Page: 307
Descriptor Docstring and Overriding Deletion Page: 308
Chapter Summary Page: 308
Further Reading Page: 308
24. Class Metaprogramming Page: 310
What’s New in This Chapter Page: 310
Classes as Objects Page: 310
type: The Built-In Class Factory Page: 310
A Class Factory Function Page: 311
Introducing __init_subclass__ Page: 312
Why __init_subclass__ Cannot Configure __slots__ Page: 314
Enhancing Classes with a Class Decorator Page: 314
What Happens When: Import Time Versus Runtime Page: 315
Evaluation Time Experiments Page: 316
Metaclasses 101 Page: 317
How a Metaclass Customizes a Class Page: 317
A Nice Metaclass Example Page: 318
Metaclass Evaluation Time Experiment Page: 319
A Metaclass Solution for Checked Page: 320
Metaclasses in the Real World Page: 322
Modern Features Simplify or Replace Metaclasses Page: 322
Metaclasses Are Stable Language Features Page: 322
A Class Can Only Have One Metaclass Page: 322
Metaclasses Should Be Implementation Details Page: 322
A Metaclass Hack with __prepare__ Page: 322
Wrapping Up Page: 323
Chapter Summary Page: 323
Further Reading Page: 324
Afterword Page: 327
Further Reading Page: 327
Index Page: 328
About the Author Page: 344