Foundations of Python Network Programming, Second Edition

370 Pages·2010·2.19 MB·English

by  John Goerzen

BOOKS FOR PROFESSIONALS BY PROFESSIONALS® THE EXPERT’S VOICE® IN OPEN SOURCE
Companion
eBook
Available
Foundations of Python Network
Foundations
Programming of
P
This second edition of Foundations of Python Network Programming targets y
Python 2.5 through Python 2.7, the most popular production versions of the lan-
t
guage. Python has made great strides since Apress released the first edition of
h
this book back in the days of Python 2.3. The advances required new chapters to Foundations of
be written from the ground up, and others to be extensively revised. o
Brandon Rhodes
You will learn fundamentals like IP, TCP, DNS and SSL by using working
Python programs; you will also be able to familiarize yourself with infrastruc- n
ture components like memcached and message queues. You can also delve into

Network server designs, and compare threaded approaches with asynchronous N Python Network
event-based solutions. But the biggest change is this edition's expanded treat-
ment of the web. e
The HTTP protocol is covered in extensive detail, with each feature accom-
panied by sample Python code. You can use your HTTP protocol expertise by t
studying an entire chapter on screen scraping and you can then test lxml and w Programming
BeautifulSoup against a real-world web site. The chapter on web application pro-
gramming now covers both the WSGI standard for component interoperability, o
as well as modern web frameworks like Django.
Finally, all of the old favorites from the first edition are back: E-mail proto- r
John Goerzen cols like SMTP, POP, and IMAP get full treatment, as does XML-RPC. You can still k

learn how to code Python network programs using the Telnet and FTP protocols,
but you are likely to appreciate the power of more modern alternatives like the
P
paramiko SSH2 library. If you are a Python programmer who needs to learn the
r
network, this is the book that you want by your side. o
g
r
a The comprehensive guide to building network
THE APRESS ROADMAP m applications with Python
Foundations of m
Beginning Pro
Python Network
Companion eBook THE APRPEySthSo nROADMAP Programming Python i
n
Foundations of g
Dive BinePtgyoit nhPnoyintnhgon 3 PFoyPPtuyhrntoohdngoa rnDtai oemNnvesmet lwooinfpo gmAr kgeinlet PythoPny PAthrlgooonr ithms SECOND SECOND EDITION
EDITION
Foundations of Agile
Dive into Python 3 Python Development Python Algorithms
G
R Brandon Rhodes John Goerzen
SOURCE CODE ONLINE oeh and
www.apress.com rzod
ee
ns
Shelve in:
Python
User level:
Intermediate–Advanced
>
m
o
ok.c
o
b
e
w
o
w
w.
w
w
<
ok
o
B
e
w!
o
W
m
o
d fr
a
o
nl
w
o
D
Foundations of Python
Network Programming
The comprehensive guide to building
network applications with Python
Second Edition

■ ■ ■
Brandon Rhodes
John Goerzen

i
Foundations of Python Network Programming: The comprehensive guide to building network
applications with Python
Copyright © 2010 by Brandon Rhodes and John Goerzen
All rights reserved. No part of this work may be reproduced or transmitted in any form or by any
means, electronic or mechanical, including photocopying, recording, or by any information
storage or retrieval system, without the prior written permission of the copyright owner and the
publisher.
ISBN-13 (pbk): 978-1-4302-3003-8

Printed and bound in the United States of America (POD)
Trademarked names, logos, and images may appear in this book. Rather than use a trademark
symbol with every occurrence of a trademarked name, logo, or image we use the names, logos, and
images only in an editorial fashion and to the benefit of the trademark owner, with no intention of
infringement of the trademark.
The use in this publication of trade names, trademarks, service marks, and similar terms, even if
they are not identified as such, is not to be taken as an expression of opinion as to whether or not
they are subject to proprietary rights.
President and Publisher: Paul Manning
Lead Editor: Frank Pohlmann
Development Editor: Matt Wade
Technical Reviewer: Michael Bernstein
Editorial Board: Steve Anglin, Mark Beckner, Ewan Buckingham, Tony Campbell, Gary Cornell,
Jonathan Gennick, Michelle Lowman, Matthew Moodie, Jeffrey Pepper, Frank Pohlmann,
Ben Renow-Clarke, Dominic Shakeshaft, Matt Wade, Tom Welsh
Coordinating Editor: Laurin Becker
Copy Editors: Mary Ann Fugate and Patrick Meador
Compositor: MacPS, LLC
Indexer: Potomac Indexing, LLC
Cover Designer: Anna Ishchenko
Distributed to the book trade worldwide by Springer Science+Business Media, LLC., 233 Spring
Street, 6th Floor, New York, NY 10013. Phone 1-800-SPRINGER, fax [phone], e-mail
[email protected], or visit www.springeronline.com.
For information on translations, please e-mail [email protected], or visit www.apress.com.
Apress and friends of ED books may be purchased in bulk for academic, corporate, or promotional
use. eBook versions and licenses are also available for most titles. For more information, reference
our Special Bulk Sales–eBook Licensing web page at www.apress.com/info/bulksales.
The information in this book is distributed on an “as is” basis, without warranty. Although every
precaution has been taken in the preparation of this work, neither the author(s) nor Apress shall
have any liability to any person or entity with respect to any loss or damage caused or alleged to be
caused directly or indirectly by the information contained in this work.
The source code for this book is available to readers at www.apress.com.
ii
To the Python community
for creating a programming language, libraries, and packages
that are freely written and freely gifted
from one programmer to another

To small Persephone-cat for keeping me warm
while revising chapters late at night

And, most of all, to my Jackie

iii
Contents at a Glance

■Contents ................................................................................................................ v(cid:1)
■About the Authors ............................................................................................... xv(cid:1)
■About the Technical Reviewer ............................................................................ xvi(cid:1)
■Acknowledgments ............................................................................................. xvii(cid:1)
■Introduction ..................................................................................................... xviii
■Chapter 1: Introduction to Client/Server Networking ............................................ 1(cid:1)
■Chapter 2: UDP .................................................................................................... 15(cid:1)
■Chapter 3: TCP ..................................................................................................... 35(cid:1)
■Chapter 4: Socket Names and DNS ..................................................................... 51(cid:1)
■Chapter 5: Network Data and Network Errors ..................................................... 71(cid:1)
■Chapter 6: TLS and SSL ....................................................................................... 87(cid:1)
■Chapter 7: Server Architecture ........................................................................... 99(cid:1)
■Chapter 8: Caches, Message Queues, and Map-Reduce ................................... 125(cid:1)
■Chapter 9: HTTP ................................................................................................. 137(cid:1)
■Chapter 10: Screen Scraping ............................................................................ 163(cid:1)
■Chapter 11: Web Applications ........................................................................... 179(cid:1)
■Chapter 12: E-mail Composition and Decoding ................................................. 197(cid:1)
■Chapter 13: SMTP .............................................................................................. 217(cid:1)
■Chapter 14: POP ................................................................................................ 235(cid:1)
■Chapter 15: IMAP .............................................................................................. 243(cid:1)
■Chapter 16: Telnet and SSH ............................................................................... 263(cid:1)
■Chapter 17: FTP ................................................................................................. 291(cid:1)
■Chapter 18: RPC ................................................................................................ 305(cid:1)
■Index ................................................................................................................. 323(cid:1)
iv
Contents

■Contents at a Glance ............................................................................................ iv(cid:1)
■About the Authors ............................................................................................... xv(cid:1)
■About the Technical Reviewer ............................................................................. xv(cid:1)
■Acknowledgments .............................................................................................. xvi(cid:1)
■Introduction ....................................................................................................... xvii

■Chapter 1: Introduction to Client/Server Networking ............................................ 1(cid:1)
The Building Blocks: Stacks and Libraries ...................................................................... 1(cid:1)
Application Layers ........................................................................................................... 4(cid:1)
Speaking a Protocol ........................................................................................................ 5(cid:1)
A Raw Network Conversation ......................................................................................... 6(cid:1)
Turtles All the Way Down ................................................................................................ 8(cid:1)
The Internet Protocol ....................................................................................................... 9(cid:1)
IP Addresses ................................................................................................................. 10(cid:1)
Routing .......................................................................................................................... 11(cid:1)
Packet Fragmentation ................................................................................................... 13(cid:1)
Learning More About IP ................................................................................................. 14(cid:1)
■Chapter 2: UDP .................................................................................................... 15(cid:1)
Should You Read This Chapter? .................................................................................... 16(cid:1)
Addresses and Port Numbers ....................................................................................... 16(cid:1)
Port Number Ranges ..................................................................................................... 17(cid:1)
Sockets ......................................................................................................................... 19(cid:1)
v
■ CONTENTS
Unreliability, Backoff, Blocking, Timeouts .................................................................... 22(cid:1)
Connecting UDP Sockets ............................................................................................... 25(cid:1)
Request IDs: A Good Idea .............................................................................................. 27(cid:1)
Binding to Interfaces ..................................................................................................... 28(cid:1)
UDP Fragmentation ....................................................................................................... 30(cid:1)
Socket Options .............................................................................................................. 31(cid:1)
Broadcast ...................................................................................................................... 32(cid:1)
When to Use UDP .......................................................................................................... 33(cid:1)
Summary ....................................................................................................................... 34(cid:1)
■Chapter 3: TCP ..................................................................................................... 35(cid:1)
How TCP Works ............................................................................................................. 35(cid:1)
When to Use TCP ........................................................................................................... 36(cid:1)
What TCP Sockets Mean ............................................................................................... 37(cid:1)
A Simple TCP Client and Server .................................................................................... 38(cid:1)
One Socket per Conversation ........................................................................................ 41(cid:1)
Address Already in Use ................................................................................................. 42(cid:1)
Binding to Interfaces ..................................................................................................... 43(cid:1)
Deadlock ....................................................................................................................... 44(cid:1)
Closed Connections, Half-Open Connections ................................................................ 48(cid:1)
Using TCP Streams like Files ........................................................................................ 49(cid:1)
Summary ....................................................................................................................... 49(cid:1)
■Chapter 4: Socket Names and DNS ..................................................................... 51(cid:1)
Hostnames and Domain Names .................................................................................... 51(cid:1)
Socket Names ............................................................................................................... 52(cid:1)
Five Socket Coordinates ............................................................................................... 53(cid:1)
IPv6 ............................................................................................................................... 54(cid:1)
Modern Address Resolution .......................................................................................... 55(cid:1)
vi
■ CONTENTS
Asking getaddrinfo() Where to Bind .............................................................................. 56(cid:1)
Asking getaddrinfo() About Services ............................................................................. 56(cid:1)
Asking getaddrinfo() for Pretty Hostnames ................................................................... 57(cid:1)
Other getaddrinfo() Flags .............................................................................................. 58(cid:1)
Primitive Name Service Routines .................................................................................. 59(cid:1)
Using getsockaddr() in Your Own Code ......................................................................... 60(cid:1)
Better Living Through Paranoia .................................................................................... 61(cid:1)
A Sketch of How DNS Works ......................................................................................... 63(cid:1)
Why Not to Use DNS ...................................................................................................... 65(cid:1)
Why to Use DNS ............................................................................................................ 66(cid:1)
Resolving Mail Domains ................................................................................................ 68(cid:1)
Zeroconf and Dynamic DNS .......................................................................................... 70(cid:1)
Summary ....................................................................................................................... 70(cid:1)
■Chapter 5: etwork Data and Network Errors ....................................................... 71(cid:1)
Text and Encodings ....................................................................................................... 71(cid:1)
Network Byte Order ...................................................................................................... 73(cid:1)
Framing and Quoting .................................................................................................... 75(cid:1)
Pickles and Self-Delimiting Formats ............................................................................. 79(cid:1)
XML, JSON, Etc. ............................................................................................................ 80(cid:1)
Compression ................................................................................................................. 81(cid:1)
Network Exceptions ...................................................................................................... 82(cid:1)
Handling Exceptions ..................................................................................................... 83(cid:1)
Summary ....................................................................................................................... 85(cid:1)
■Chapter 6: TLS and SSL ....................................................................................... 87(cid:1)
Computer Security ........................................................................................................ 87(cid:1)
IP Access Rules ............................................................................................................. 88(cid:1)
Cleartext on the Network .............................................................................................. 90(cid:1)
vii
■ CONTENTS
TLS Encrypts Your Conversations ................................................................................. 92(cid:1)
TLS Verifies Identities ................................................................................................... 93(cid:1)
Supporting TLS in Python .............................................................................................. 94(cid:1)
The Standard SSL Module ............................................................................................. 95(cid:1)
Loose Ends .................................................................................................................... 98(cid:1)
Summary ....................................................................................................................... 98(cid:1)
■Chapter 7: Server Architecture ........................................................................... 99(cid:1)
Daemons and Logging .................................................................................................. 99(cid:1)
Our Example: Sir Launcelot ......................................................................................... 100(cid:1)
An Elementary Client ................................................................................................... 102(cid:1)
The Waiting Game ....................................................................................................... 103(cid:1)
Running a Benchmark ................................................................................................. 106(cid:1)
Event-Driven Servers .................................................................................................. 109(cid:1)
Poll vs. Select .............................................................................................................. 112(cid:1)
The Semantics of Non-blocking .................................................................................. 113(cid:1)
Event-Driven Servers Are Blocking and Synchronous ................................................ 114(cid:1)
Twisted Python ........................................................................................................... 114(cid:1)
Load Balancing and Proxies ........................................................................................ 117(cid:1)
Threading and Multi-processing ................................................................................. 117(cid:1)
Threading and Multi-processing Frameworks ............................................................ 120(cid:1)
Process and Thread Coordination ............................................................................... 122(cid:1)
Running Inside inetd ................................................................................................... 123(cid:1)
Summary ..................................................................................................................... 124(cid:1)
■Chapter 8: Caches, Message Queues, and Map-Reduce ................................... 125(cid:1)
Using Memcached ...................................................................................................... 126(cid:1)
Memcached and Sharding .......................................................................................... 128(cid:1)
Message Queues ......................................................................................................... 130(cid:1)
viii