Table Of Content

Code Connected Volume 1 Pieter Hintjens Code Connected Volume 1 Learning ZeroMQ Professional Edition for C/C++ To my children Noémie, Freeman, and Gregor Code Connected Volume 1 Copyright © 2010-2013 Pieter Hintjens First Printing: January 6th, 2013 Licensed under CC-BY-SA-3.0 Published by iMatix Corporation “ Code Connected” is a trademark of Pieter Hintjens. “ ØMQ” , “ ØMQ” , “ ZeroMQ” , and the ØMQ logo are trademarks of iMatix Corporation. Edited by Andy Oram at O'Reilly This text originated as “ ZeroMQ - The Guide” at http://zguide.zeromq.org, with examples from the ØMQ community in C, C++, C#, Clojure, Delphi, Erlang, Go, Haskell, Haxe, Java, Lisp, Lua, Node.js, Objective-C, Perl, PHP, Python, Ruby, Scala, Tcl, and other languages. ISBN 978-1481262651 Contents 7 Preface 7 ØMQ in a Hundred Words 7 How It Began 7 The Zen of Zero 8 How This Book Came To Be 9 Audience 9 Code Connected 11 Chapter 1. Basics 11 Fixing the World 12 Starting Assumptions 13 Getting the Examples 13 Ask and Ye Shall Receive 16 A Minor Note on Strings 17 Version Reporting 18 Getting the Message Out 22 Divide and Conquer 26 Programming with ØMQ 28 Why We Needed ØMQ 32 Socket Scalability 32 Upgrading from ØMQ v2.2 to ØMQ v3.2 34 Warning: Unstable Paradigms! 35 Chapter 2. Sockets and Patterns 35 The Socket API 40 Messaging Patterns 56 Handling Errors and ETERM 60 Handling Interrupt Signals 62 Detecting Memory Leaks 63 Multithreading with ØMQ 67 Signaling Between Threads (PAIR Sockets) 69 Node Coordination 72 Zero-Copy 73 Pub-Sub Message Envelopes 75 High-Water Marks 76 Missing Message Problem Solver v vi Code Connected Volume 1 79 Chapter 3. Advanced Request-Reply Patterns 79 The Request-Reply Mechanisms 83 Request-Reply Combinations 86 Exploring ROUTER Sockets 88 The Load Balancing Pattern 98 A High-Level API for ØMQ 107 The Asynchronous Client/Server Pattern 111 Worked Example: Inter-Broker Routing 135 Chapter 4. Reliable Request-Reply Patterns 135 What is “ Reliability” ? 136 Designing Reliability 137 Client-Side Reliability (Lazy Pirate Pattern) 141 Basic Reliable Queuing (Simple Pirate Pattern) 144 Robust Reliable Queuing (Paranoid Pirate Pattern) 152 Heartbeating 155 Contracts and Protocols 156 Service-Oriented Reliable Queuing (Majordomo Pattern) 178 Asynchronous Majordomo Pattern 184 Service Discovery 185 Idempotent Services 186 Disconnected Reliability (Titanic Pattern) 198 High-Availability Pair (Binary Star Pattern) 214 Brokerless Reliability (Freelance Pattern) 234 Conclusion 235 Chapter 5. Advanced Pub-Sub Patterns 235 Pros and Cons of Pub-Sub 237 Pub-Sub Tracing (Espresso Pattern) 239 Last Value Caching 244 Slow Subscriber Detection (Suicidal Snail Pattern) 247 High-Speed Subscribers (Black Box Pattern) 249 Reliable Pub-Sub (Clone Pattern) 311 Postface 311 Tales from Out There 312 How This Book Happened 314 Removing Friction 315 Licensing 316 Code Connected Volume 2 Preface ØMQ in a Hundred Words ØMQ (also known as ZeroMQ, ØMQ, or zmq) looks like an embeddable networking library but acts like a concurrency framework. It gives you sockets that carry atomic messages across various transports like in-process, inter-process, TCP, and multicast. You can connect sockets N-to-N with patterns like fan-out, pub-sub, task distribution, and request-reply. It’ s fast enough to be the fabric for clustered products. Its asynchronous I/O model gives you scalable multicore applications, built as asynchronous message-processing tasks. It has a score of language APIs and runs on most operating systems. ØMQ is from iMatix1 and is LGPLv3 open source. How It Began We took a normal TCP socket, injected it with a mix of radioactive isotopes stolen from a secret Soviet atomic research project, bombarded it with 1950-era cosmic rays, and put it into the hands of a drug-addled comic book author with a badly-disguised fetish for bulging muscles clad in spandex (see figure 1). Yes, ØMQ sockets are the world-saving superheroes of the networking world. The Zen of Zero The Ø in ØMQ is all about tradeoffs. On the one hand this strange name lowers ØMQ’ s visibility on Google and Twitter. On the other hand it annoys the heck out of some Danish folk who write us things like “ ØMG røtfl” , and “ Ø is not a funny looking zero!” and “ Rødgrød med Fløde!” , which is apparently an insult that means “ may your neighbours be the direct descendants of Grendel!” Seems like a fair trade. 1 http://www.imatix.com 7 8 Code Connected Volume 1 - Preface Originally the zero in ØMQ was meant as “ zero broker” and (as close to) “ zero latency” (as possible). Since then, it has come to encompass different goals: zero administration, zero cost, zero waste. More generally, “ zero” refers to the culture of minimalism that permeates the project. We add power by removing complexity rather than by exposing new functionality. How This Book Came To Be In the summer of 2010, ØMQ was still a little-known niche library described by its rather terse reference manual and a living but sparse wiki. Martin Sustrik and myself were sitting in the bar of the Hotel Kyjev in Bratislava plotting how to make ØMQ more widely popular. Martin had written most of the ØMQ code, and I’ d put up the funding and organized the community. Over some Zlaty Bazants, we agreed that ØMQ needed a new, simpler web site and a basic guide for new users. Martin collected some ideas for topics to explain. I’ d never written a line of ØMQ code before this, so it became a live learning documentary. As I worked through simple examples to more complex ones, I tried to answer many of the questions I’ d seen on the mailing list. Because I’ d been building large-scale architectures for 30 years, there were a lot of problems at which I was keen to throw ØMQ. Amazingly the results were mostly simple and elegant, even when working in C. I felt a pure joy learning ØMQ and using it to solve real problems, which brought me back to programming after a few years' pause. And often, not knowing how it was “ supposed” to be done, we improved ØMQ as we went along. From the start, I wanted the ØMQ guide to be a community project, so I put it onto GitHub and let others contribute with pull requests. This was considered a radical, even vulgar approach by some. We came to a division of labor: I’ d do the writing and make the original C examples, and others would help fix the text and translate the examples into other languages. This worked better than I dared hope. You can now find all the examples in several languages and many in a dozen languages. It’ s a kind of programming language Rosetta stone and a valuable outcome in itself. We set up a high score: reach 80% translation and your language got its own Guide. PHP, Python, Lua, and Haxe reached this goal. People asked for PDFs, and we created those. People asked for ebooks, and got those. About a hundred people contributed to the examples to date. The book, in its on-line version “ The Guide” , achieved its goal of popularizing ØMQ. The style pleases most and annoys some, which is how it should be. In December 2010, my work on ØMQ and this guide stopped, as I found myself going through late-stage cancer, heavy surgery, and six months of chemotherapy. When I picked up work again in mid-2011, it was to start using ØMQ in anger for one of the largest use cases imaginable: on the mobile phones and tablets of the world’ s biggest electronics company. But the goal of the ØMQ book was, from the start, a printed work. So it was exciting to get an email from Bill Lubanovic in January 2012 introducing me to his editor, Andy Code Connected Volume 1 - Preface 9 Oram, at O’ Reilly, suggesting a ØMQ book. Of course! Where do I sign? How much do I have to pay? Oh, I get money for this? All I have to do is finish it? Of course as soon as O’ Reilly announced a ØMQ book, other publishers started sending out emails to potential authors. You’ ll probably see a rash of ØMQ books coming out next year. That’ s good. Our niche library has hit the mainstream and deserves its six inches of shelf space. My apologies to the other ØMQ authors. We’ ve set the bar horribly high, and my advice is to make your books complementary. Perhaps focus on a specific language, platform, or pattern. This is the magic and power of communities: be the first community in a space, stay healthy, and you own that space for ever. Audience This book is written for professional programmers who want to learn how to make the massively distributed software that will dominate the future of computing. We assume you can read C code, because most of the examples here are in C even though ØMQ is used in many languages. We assume you care about scale, because ØMQ solves that problem above all others. We assume you need the best possible results with the least possible cost, because otherwise you won’ t appreciate the trade-offs that ØMQ makes. Other than that basic background, we try to present all the concepts in networking and distributed computing you will need to use ØMQ. Code Connected This book, “ Code Connected Volume 1” , is the first volume covering the theme of how to build distributed software over ØMQ. In this book, you’ ll learn how to use the ØMQ library. We’ ll cover the basics, the API, the different socket types and how they work, reliability, and a host of patterns you can use in your applications. You’ ll get the best results by working through the examples and text from start to end. This is the Professional Edition for C/C++ developers. But whatever your programming language, the patterns and techniques that Code Connected explains are relevant to you. Get the examples online in C, C++, C#, Clojure, Delphi, Erlang, Go, Haskell, Haxe, Java, Lisp, Lua, Node.js, Objective-C, Perl, PHP, Python, Ruby, Scala, and Tcl.