Musings about random stuff. No attempt at scientific rigor. Take with a grain of salt.
All the opinions stated here are my own. Any resemblance to opinions of other people, living or dead, is purely coincidental.
Heartbeating is a common technique to check whether network connection is alive. The idea is that each end of the connection sends small packet od data called hearetbeat once in a while. If the peer doesn't receive a heartbeat for some time (typically a multiply of the interval between the heartbeats), the connection is considered broken. Interestingly, TCP protocol doesn't provide heartbeats (there are optional keep-alives that are operating on scale of hours, but these are not really useful...
Comments: 27
How to Write a Language Binding for nanomsg
FoldUnfold Table of Contents Reusing ZeroMQ bindings Forking ZeroMQ bindings Writing a new binding Plug-ins vs. native code Plug-ins Native code Run-time retrieval of constants Managing socket lifetime Sending and receiving messages Zero-copy Error codes Handling EINTR Polling Modularity Language bindings are pieces of infrastructure that allow nanomsg to be used from different programming languages. This article is meant to provide some hints that may be helpful for binding developers....
Comments: 0
Using Survey Protocol for High Availability
I've already wrote about survey scalability protocol in this blog. The article explained how the survey protocol can be used to collect information from a set of computers. This article, on the other hand, shows how to use survey protocol to combine high reliability with low latency. First, let's have a look how REQ/REP protocol handles reliability. The basic idea is that there are multiple instances of the service, so if one of them fails, others are still available for processing requests....
Comments: 0
Optimising Subscriptions in nanomsg
When I was writing ZeroMQ's subscription subsystem, my assumption was that there will be thousands or in the worst case tens of thousands subscriptions at any single time. The assumption reflected my background in financial services, where subscriptions are mostly used for subscribing for stock quotes. The topic you subscribe to is the name of the stock and there are, typically, some tens of thousands of those, even if you take into account derivatives like futures and options. However, it...
Comments: 8
I've originally written this article in September 2011. Given that I've had several discussions about multiplexing inside of messaging solutions in past few days, I've decided to re-publish a slightly improved version. Introduction Some messaging technologies (e.g. AMQP) allow for multiplexing several data streams on top of a single TCP connection. Distributed messaging systems such as ZeroMQ which assume lot of small independent services running on a single box or even inside a single process...
Comments: 0
The idea underlying bus messaging pattern is to provide the semantics similar to those of the hardware bus everyone connected to the bus gets any data sent to the bus just to do so on the higher layer. So, to use messaging terminology, everyone connected to the message bus gets any message sent to the bus. This pattern doesn't scale well at some point the number of applications connected to the bus will grow to the point where they will be overloaded by sheer amount of messages they...
Comments: 7
Implementing Network Protocols in User Space
Implementing network protocols in user space is not as uncommon as it may seem. The advantages when compared to implementing them in the kernel space range from easier development, more freedom to experiment and greater portability to less hassle with the distribution (the pain of getting the patch into the mainline kernel vs. simply uploading a user space library to GitHub), improved performance thanks to kernel by-pass and no way to provide a kernel space implementation for proprietary...
Comments: 20
Why Communication Infrastructure Should Use Permissive Licenses
In past days I was challenged couple of times about my decision to publish nanomsg under MIT/X11 license. Other folks, on the other hand, were extremely happy about the license. So, I written this article to give a short explanation of the licensing choice. First, there are two different meanings of freedom in software world. No, I don't mean free as in speech vs. free as in beer . What I mean is freedom as in freedom to run, inspect the code, redistribute and modify vs. freedom as in ...
Comments: 9
Priotitised Load Balancing with nanomsg
Load balancing is one of the typical features of messaging systems. Although some don't do it (MQTT), most of them provide some way to spread a workload among cluster of boxes. In ZeroMQ load balancing is done via REQ (requests that require replies) or, alternatively, by PUSH socket (requests that don't require replies). When designing it, I've opted for completely fair load-balancer. What it means is that if there are two peers able to process requests, first request goes to the first one,...
Comments: 0
Location-independent Addressing with Messaging Middleware
When I'm thinking about messaging, usually I assume is that its goal is to deliver nicely packaged tools for implementing different clever scaling algorithms. An example may be pub-sub pattern (or JMS topic) to do one-to-many data distribution, or req-rep pattern (or JMS queue) to implement a service provided by a cluster of stateless workers. However, messaging is quite often used in a very different manner. Instead of providing a distributed algorithm among N processes, it is used for a simple...
Comments: 5
EINTR is one of the POSIX errors that you can get from different blocking functions (send, recv, poll, sem_wait etc.) The POSIX standard unhelpfully describes it as Interrupted function. Googling for EINTR returns mainly random questions like I am getting teh EINTR error. What now? answered mostly by Just restart the interrupted function. None of this helps much when you want to correctly handle EINTR, actually understand what you are doing and why. In this blog post I'll try to explain...
Comments: 30
Couple of AMQP-related blog posts have arrived lately. Reading it, I've realised that although I've been involved with AMQP from the very beginning I've never explicitly expressed my thoughts on it, on the split between AMQP/0-9-1 and AMQP/1-0 and related technical issues. Also, I believe that designing ZeroMQ gave me a pretty unique point of view on AMQP, that some of the readers may find interesting to explore. 8 Years in Making To start with, the goal of AMQP was breaking the IBM and TIBCO...
Comments: 13
Getting Rid of Domain Registrars Using Bitcoin Algorithm
Recently I've stumbled upon an though-provoking article here. What it says is that whatever the benefits of Bitcoin as a distributed currency, the biggest deal is that it provided us with a viable algorithm for reaching consensus in highly distributed global-scale systems. I would also say that actually proving that the algorithm works on global scale in real world is a great and almost unprecedented feat. The article goes on to hint that the Bitcoin algorithm may be used to fully decentralise...
Comments: 6
In my last blog post I've ranted about problems with implementing intrusive containers in C++. The main problem is that to allow an object to be included into an intrusive container you have to modify the object itself, for example, add 'prev' and 'next' member variables to it. That in turn breaks the encapsulation principle. In rigorous object-oriented design the item is owned by the container, and thus the container should be aware of the item but not vice versa. There have been a lot of...
Comments: 40
Why should I have written ZeroMQ in C, not C++ (part II)
In my previous blog post I've discussed how the need for rigorous error handling in low-level infrastructure software prevents the usage of many fundamental C++ features (exceptions, constructors, destructors). The conclusion was that with such a severely limited feature set, writing the program in C yields shorter and more readable codebase. A side effect is eliminating the dependency on C++ runtime library, which shouldn't be easily dismissed, especially in embedded environments. In this blog...
Comments: 184
I've run into a problem recently. As explained elsewhere, my long-term goal was to use Crossroads/ZeroMQ as a first step on the road to messaging as native layer of the Internet stack. The idea is that in addition to routing layer like IP and reliability layer like TCP there should be a scalability layer, similar to Crossroads/ZeroMQ available out of the box, as part of operating system, everywhere and without a need to care about it. That, obviously, means that the scalability layer should be...
Comments: 28
If you ever skimmed linux kernel code may have noticed that condition expressions are often marked as either 'likely' or 'unlikely': if (likely (i == 0)) { ... } What is it good for? These directives are translated into the following GCC directives: #define likely(x) __builtin_expect ((x), 1) #define unlikely(x) __builtin_expect ((x), 0) These in turn instruct the compiler which branch of the program should be faster and which may execute slower. To understand how it works we need to...
Comments: 14
Distributed Computing: The Survey Pattern
Survey is a well established pattern both in real world and in distributed computing. Think of census. You send census forms to everyone, collect the results and move on with computing statistics or whatever. In distributed computing, the common usage of survey pattern is service discovery. You ask a generic question not aimed at a particular peer: Are there any services out there? Then you collect the responses ( I can process payments! , I can render images! , I can add two integers! ...
Comments: 5
Why should I have written ZeroMQ in C, not C++ (part I)
Just to be clear from the very beginning: This is not going to be a Torvalds-ish rant against C++ from the point of view of die-hard C programmer. I've been using C++ whole my professional career and it's still my language of choice when doing most projects. Naturally, when I started ZeroMQ project back in 2007, I've opted for C++. The main reasons were: Library of data structures and algorithms (STL) is part of the language. With C I would have to either depend on a 3rd party library or had to...
Comments: 265
Silicon Valley, Hollywood and Iceland as the New Superpower
When following the news about restrictive legislation, the likes of SOPA, PIPA and ACTA, when watching the strife between Hollywood and Silicon Valley , when thinking about the struggle between service providers and content owners, between the new and old economy, one wonders whether a country that would firmly stand behind the Silicon Valley model and lovingly embrace the emerging post-scarcity economy wouldn't reap a great benefit from doing so, the same way as England did during the...
Comments: 0
Website powered by Wikidot.