Couple of AMQP-related blog posts have arrived lately. Reading it, I've realised that although I've been involved with AMQP from the very beginning I've never explicitly expressed my thoughts on it, on the split between AMQP/0-9-1 and AMQP/1-0 and related technical issues. Also, I believe that designing ZeroMQ gave me a pretty unique point of view on AMQP, that some of the readers may find interesting to explore.
8 Years in Making
To start with, the goal of AMQP was breaking the IBM and TIBCO duopoly in enterprise messaging market and commoditisation of messaging middleware products. Both companies have charged exorbitant prices for the software and even wannabe competitor products were extremely costly. The idea was to define an interoperable messaging protocol and thus allow enterprises to switch messaging implementations without breaking the existing applications. Therefore, if incumbent vendor would go rogue with the licensing fees, the enterprise could smoothly migrate to a different vendor with more favourable pricing scheme.
By the time the work on AMQP have begun (2004) there was already an established API standard for enterprise messaging called Java Message Service, or JMS for short. Of course, JMS works only with Java, but at least it defines a reasonable set of functionality to be supported by enterprise messaging product. Thus, the real challenge of AMQP was to define a wire protocol that could work as a communication medium between JMS clients and servers. And early versions of AMQP did just that. The protocol was basically reverse-engineered from JMS. You could browse through the specification and map AMQP concepts to JMS concepts in straightforward 1:1 manner. AMQP/0-9-1 standard (as used, for example, by RabbitMQ) is still very much JMS gone binary.
There's one digression to make here: The 1:1 mapping between JMS and AMQP/0-9-1 is somewhat obscured by replacement of JMS' Queue and Topic model with AMQP's Exchange-Binging-Queue model. The refactoring seemed to make sense at the time as it clearly separated the routing functionality (exchanges) from message queueing functionality (queues). From today's perspective though, I would say that Exchange-Binding-Queue model is rather a leakage of internal broker architecture (separate modules to route and queue messages) into the protocol. While the model allows for novel ways of interlinking different routing and queueing behaviours, in practice there are only two models that actually make sense: request/reply model embodied in JMS' "Queue" concept and publish/subscribe model modelled as "Topic" in JMS. All the additional flexibility provided by Exchange-Binding-Queue model is more or less good only for letting the user shoot himself in the foot.
Anyway, back to the story: By the end of 2007 I've started working on ZeroMQ. ZeroMQ had different goals from AMQP. Instead of trying to provide interoperability for traditional enterprise messaging (IBM WebSphereMQ and various copycat products) it rather tried to address the low-latency messaging market (TIBCO RendezVous, 29 West's LBM and alike) as well as messaging in 'common' rather than 'enterprise' scenarios. At first I've tried to use AMQP/0-9-1 model in ZeroMQ. I've even written a whitepaper about it. However, AMQP never aligned well with the goals of ZeroMQ and I've finally dropped the AMQP support when releasing version 2.0 of ZeroMQ.
In 2008 AMQP/1-0 emerged. I was not much involved with AMQP working group by then so I have no idea why it have entirely dumped the previous versions of the protocol and sent all the work done by early adopters of the protocol down the drain.
The most puzzling aspect is that such a move seriously undermines the credibility of AMQP as a standard. After all, why should you invest in AMQP/1-0 if the working group is going to do the same thing next year, dump AMQP/1-0, publish completely unrelated AMQP/2-0 and make your investment worthless? It sounds counter-intuitive. Why would members of AMQP Working Group want to destroy the credibility of their standard?
There's one conspiration theory that comes to mind: Around the time AMQP/1-0 was first discussed, Microsoft joined the AMQP working group. By that time AMQP Working Group consisted of users of messaging technology (banks and alike) and few small messaging vendors. Red Hat was part of the group, but its Qpid product was relatively new, not established and have already strayed away from the original standard. RabbitMQ had an AMQP implementation, but back then it was not yet acquired by VMware and was thus a small and insignificant member of the group. Same applies to iMatix with its OpenAMQ implementation. Thus, the thinking of those in power in the Working Group may have been that credibility gained by getting established messaging vendor like Microsoft (DCOM, MSMQ, SOAP, WCF etc.) on board would compensate for the loss of credibility caused by ruining the investments made be early adopters.
The question, of course, is why not get Microsoft on board and preserve the standard at the same time. The answer seems quite obvious once you think of Microsoft joining the AMQP Working Group as of economic transaction. Microsoft offers credibility that goes with its membership in the standards group. The standard groups offers the "AMQP-compliant" stamp. Thus, to make the transaction acceptable by both parties, existing Microsoft products have to be stamped as "AMQP-compliant".
The facts seem to support the above explanation. Most importantly, the whole "broker model" part of the specification was dropped. AMQP doesn't define how the message broker should behave any more. It deals mostly with what happens on the wire between two TCP endpoints. As long as the bytes on wire are syntactically correct, broker is free to do anything and still be called AMQP-compliant. In practice it means the vendor can take an existing messaging product, implement a stand-alone well-encapsulated AMQP adaptor and there he goes without a need to re-write the whole product. If the broker model stayed in place, Microsoft would have to either create a new AMQP-compliant messaging technology or heavily re-factor its existing technologies. Both options are extremely expensive, not only in terms of the development cost, but primarily in the terms of credibility. New non-compatible product would dilute the credibility accumulated in the existing technologies. Refactoring existing deployed technologies means offending the customers. None of the two is acceptable, thus, AMQP standard has to be shoehorned to comply with existing Microsoft products.
The downside of dropping the broker model, of course, is that user can't rely on standardised semantics of the broker any more. Thus, if one AMQP-compliant broker implementation is replaced by another, the application will most likely break. The fact that original goals of AMQP project, interoperability and avoidance of vendor lock-in, are missing from the current official statement of goals is in line with the above explanation.
Recently, I've seen couple of comments from Working Group members asserting that broker model is to be added to the specification in the future. Although there's no much point in discussing vapourware, there are three possible outcomes: First and most probably, the broker model never materialises. Second, the new broker model could be shoehorned to fit a particular Microsoft technology, say WCF. Third and least likely, if another big vendor, such as IBM, joins the group there's a non-zero chance that a normal standardisation process driven by multiple parties will emerge.
Anatomy of Messaging: Connection vs. Broker
Having shortly reviewed the history of AMQP, let's now have a closer look at it from technical point of view.
If you look at any messaging protocol from 10,000 feet perspective, you'll almost always see two functionally distinct layers.
The lower one deals with what happens between exactly two TCP endpoints. It basically defines a "better TCP" to build the messaging solution on top of. This layer consists of features like initial handshaking, authentication, message delimitation, multiplexing, heartbeating etc.
Second layer deals with communication between more than two endpoints and is often referred to as "broker model" (although this is a misnomer: ZeroMQ implements this layer without a need for broker). Broker model defines the behaviour of queues shared between multiple connections as well as algorithms to route messages from clients to the shared queues and vice versa.
The following table may be of interest. I've compiled the list of better-known messaging standards and marked whether they deal with connection layer or the broker model layer:
| Connection Layer | Broker Model | |
|---|---|---|
| AMQP/0-9-1 | Yes | Yes |
| AMQP/1-0 | Yes | No |
| ZeroMQ | No | Yes |
| XMPP | Yes | Yes |
| MQTT | Yes | Yes |
| STOMP | Yes | No |
Remark: ZeroMQ is a product and does not define a protocol. I've included it into the list as it is the only example of "broker model" running straight on top of raw TCP, with no intermediate "connection layer". (You may consider this article, where I've defined some basic concepts of distributed messaging, to be an initial step towards a formal protocol.)
Merit of AMQP/1-0
To assess the merit of AMQP, it can't judged based on its original goals. With no broker model the goal of interoperability was not reached and we would have to write the protocol off as a failure. There would be nothing to blog about.
However, let's assume that the new goal (although not officially stated) is to standardise the connection layer ("better TCP") and that AMQP is thus a competitor to SCTP rather then to full-blown messaging protocols like XMPP or MQTT.
From this point of view there are many technical questions to consider such as: How does AMQP deal with back-pressure? Does channelling work? What added benefit it brings over raw TCP? Is heartbeating defined is a versatile way or is it just a niche solution? What about reliability guarantees?
I would like to consider these questions in the next part(s) of this article.
Martin Sústrik, November 1st, 2012
Martin, thanks for this. An interesting perspective - I never thought of Exchange-Binding-Queue as a leakage of the internal broker design. It makes sense when I remember how we built OpenAMQ.
You mention two recent articles on AMQP, I presume one of them is this one by @kellabyte, what's the other one?
There was @kellabytes' article and one by Pieter in past few days.
Nice article, Martin. Very interesting analysis of the reasons for dumbing down AMQP, and one I'd agree with. Ironic that AMQP was once meant to run on top of SCTP. I'm impatient to read your next piece…
Portfolio
One thing about the exchange-binding-queue refactor is that it let us plug in arbitrary routing logic as exchanges, which did prove useful. There have been quite a few different exchanges, and this model also made it trivial (in the sense of taking several years to find the simple answer) to do pattern-independent federation, which was impossible with the JMS semantics, and missing from the 0MQ patterns.
Portfolio
Yes. As I said, the model made sense back then. The amount of different use cases just asked for some flexible model. It's pretty easy to spot that something is not flexible enough. Spotting that it is too flexible can take years.
This is an excellent piece. My only criticism is that you speculate about motivations. This does not add to your analysis.
True. It's a conspiration theory and thus worthless. Unfortunately, that's the best I have. Loosening the standard to accommodate vendors (not necessarily Microsoft) seems to be the only plausible explanation of what happened. Also, I don't believe people actually conspired to loosen the standard. It probably just followed from social dynamics of the working group.
By the way, the described scenario is nothing unheard of in standards development. In fact, it exists, to some extent, in any standardisation effort. See, for example, here: http://hueniverse.com/2012/07/oauth-2-0-and-the-road-to-hell/ The only aspect where AMQP stands out is the scope of the phenomenon (dropping the whole semi-established standard).
I'd be curious to hear an alternative explanation though.
As a JMS vendor I can tell you that you're wrong with your statement that 0.9.1 is 1:1 JMS. 0.9.1 is quite incompatible with JMS and - beside that it is a version looking quite amateurish and lacking many features like exactly-once delivery, flow control and ack windowing - was the reason why existing messaging systems like those from Microsoft and JMS vendors did not implement it. This has changed with AMQP 1.0. We have now 2 JMS implementations (SwiftMQ and ActiveMQ) implementing it, we have a JMS-compliant client from Qpid (which passed the JMS TCK running against SwiftMQ and Qpid Java Broker over AMQP) and others will follow. I'm very confident now since the Department of Homeland Security joined the club. If you want to do business with them in future, be sure to provide AMQP 1.0, which, by the way, should not be a problem as we will have Proton which is a AMQP 1.0 toolkit which makes it a snap to add AMQP 1.0 support to anything in any popular language.
I don't know whether Microsoft was the reason to abandon the 0.9.1 model but you may be right here. Market power is what is important and if Microsoft initiated the move then I can only pass a big thanks to them. I don't think, however, that AMQP 1.0 will be changed that much in future. It is an OASIS Standard and it took about a year for the transition from the AMQP Working Group release with only editorial changes in the spec (no functional or protocol changes). Next step is the ISO Standard so every new release has to go through these stages. But even a change to make a protocol better is no problem for a versioned protocol.
Beside personal-psychological or business reasons to save an existing user base I really don't get why there is such a hate against AMQP 1.0. Don't spend your time with such a negative stuff. Instead be part of the move and just add AMQP 1.0 support to ZeroMQ and your fork. ;-)
@Andreas, I think one of Martin's points was exactly how easy it is to add AMQP/1.0 support to old products like SwiftMQ and ActiveMQ, and how little value that gives those who expected interoperability. You kind of underscored that argument.
Portfolio
I was not trying to be negative, just to summarise the history of AMQP as I remember it. Anyway, politics is boring. Hopefully, part II will be more interesting as I want to focus on technical aspects of messaging protocols in general and AMQP in particular.
(1) Your assertion that AMQP 1.0 does not address the 'broker model' layer
is false; it does.
The specification defines a 'distribution node' as 'a node that stores
messages for distribution', along with the logical message states
through which messages pass as they are distributed and two
'distribution modes' corresponding to competing and non-competing
consumer patterns.
There is therefore a standardised aproach to the basic pub-sub and
shared queue functionality ('Topics' and 'Queues' in JMS parlance).
(2) "I would say that Exchange-Binding-Queue model is rather a leakage
of internal broker architecture"
It is indeed! That is precisely the reason 1.0 moved away from that
model, to avoid tying the protocol to a particular internal
architecture.
Allowing existing messaging products to support the protocol without
having to undertake a complete rewrite seems to me perfectly
reasonable given the original goal as you describe it.
We are already seeing the fruits of that with new implementations of
AMQP offered or in the pipeline. Greater adoption provides users with
greater choice which is exactly what AMQP is supposed to achieve.
(3) "JMS […] defines a reasonable set of functionality […] the
real challenge of AMQP was to define a wire protocol that could
work as a communication medium between JMS clients and
servers. And early versions of AMQP did just that. The protocol
was basically reverse-engineered from JMS. You could browse
through the specification and map AMQP concepts to JMS concepts in
straightforward 1:1 manner."
This is a misleading picture. Early versions of AMQP were poor in
their support for JMS. The 'basic' class was a very naive translation
of the JMS API into an RPC-ish 'protocol' without sufficient
consideration of the functionality JMS offers. As more fully
functional JMS implementations were developed, the limitations became
clear.
I would argue the central 'concept' in JMS is that of the
'Destination', a symmetric redezvous point to which messages can be
sent and from which they can be received. The mapping of this to the
'distribution node' model is much more straightforward than it was to
the 'Exchange-Binding-Queue' model.
Hi Gordon!
Well, yes, there's the 'distribution mode' bit, but calling that a broker model is a bit exaggerated. Same way, ZeroMQ does a simple message delimitation on top of TCP, but I am not calling it a 'connection layer' just because of that.
And yup, early versions of AMQP had many deficiencies. However, it's not clear how AMQP/1-0, by stressing the connection layer and ignoring the broker model does a better job.
I would say that people working on messaging should focus on defining a generic, consistent and scalable broker model, but for some reason nobody is doing that. We are still stuck with JMS model. JMS is 11 years old now and the concepts it standardises are ~20 years old. No progress seems to happen. (Just recall how the Internet changed during that time!)
If you have any idea of how to move the broker model forward into XXI. century, I would love to discuss it with you. I have some ideas myself. I'll try to express them in a paper at some point.
Surely a standardised approach for queues (including browsing) and topics (in the pub-sub sense) *does* count as a 'broker model' under your definition? If not I think you need to more clearly describe the criteria by which you are excluding it.
Fundamental to any 'broker model' is a message abstraction and a standard way for producers and consumers/subscribers to interact with the broker. After all, the purpose of brokering in many cases is to decouple the producers and consumers so the producer shouldn't have to care what the broker does with the messages it sends. (It may of course want to verify that certain capabilities are supported on the node to which it is sending, and a means of doing this both for producers and consumers is important).
On top of this foundation you can then describe the behaviour of different node types (nodes are simply the sources/sinks for messages, i.e. the rendezvous points). For example you want to specify how a broker will allocate messages published to a node that has multiple subscribers - whether the subscribers compete for the message (as in typical queue behaviour) or whether they each get a copy (fanout or pub-sub behaviour). You will want to specify a set of states and transitions for messages through which storage models can be described (e.g. classic store and forward, reliable pub-sub). AMQP 1.0 does all this. In addition you would likely want selective subscriptions, where subscribers to the same node can specify interest in a subset of the messages published to that node (e.g. the wildcard matching in a pre 1.0 'topic exchange' as compared to the 'fanout exchange' or JMS style selectors). And of course there are other aspect of behaviour (nodes that retain the last value for a certain key, nodes that honour message priority when choosing which message to deliver etc).
Now, in my view, not every 'broker' is going to need/want to support all the same behaviours (e.g. with the pre 1.0 AMQP model support for transaction, durability or priorities was optional, as was support for the topic exchange type. Moreover some implementations decided not to support acknowledgements but to specialise in uses cases that didn't require them etc). I don't think the job of the standard is to force them to do so, but rather to provide standardised approaches for and expanding set of behaviours (perhaps aggregated into profiles for easing comparisons).
The 1.0 specification does provide a way to register extensions. This allows a catalogue of existing behaviours to be built up that then provides a good basis for further standardisation efforts, hopefully driven by real needs and working solutions. I suspect the first thrust here will be on the filters required for the JMS mapping. I can also see initiatives launched to standardise some of the existing extensions from ActiveMQ, SwiftMQ, Qpid etc (message groups, last value 'queues', sticky consumers, probably informally through the respective communities to begin with.
Also relevant here - a different dimension of 'broker' behaviour - is global addressing and standards for creating networks for co-operating components.
I'd certainly be very keen to hear your thoughts on what might be needed for a more 'modern' broker model!
(PS: Sorry this is rather a long post! as someone famous once said, I didn't have time to write a shorter one!).