RabbitMQ Is Boring, and I Love It
RabbitMQ is boring. Very boring. And we tend not to think about boring things. RabbitMQ, like the electrical grid, is entirely uninteresting — until it stops working. The human brain is conditioned to recognize and respond to pain and peril more than peace, so we tend only to remember the traumas in life. In this post, I want to try to change that. Let’s talk about RabbitMQ, an open source message broker I’ve been using for the better part of 15 years — happily and bored.
My background is in, among other things, messaging and integration technologies. Unfortunately, legacy systems are often hostile, mostly because those who came before us did not foresee the highly distributed nature of today’s modern, API-dominant architectures, such as cloud native computing and microservices.
There are many ways to approach integration. In their book, “Enterprise Integration Patterns,” Gregor Hohpe and Bobby Woolf talk about four approaches: shared databases, messaging, remote procedure call (RPC) and file transfer. Integration is all about optionality and coupling: How do we take services that don’t know about each other and make them work together without overly coupling them by proximity or time? Messaging, that’s how. Messaging is the integration approach that comes with batteries included. It has all the benefits and few, if any, of the drawbacks of the three other integration styles. Messaging means I can sleep at night. Messaging is boring. I love boring.
With the support of multiple open protocols, such as AMQP 0.9, 1.0, MQTT, STOMP and others, RabbitMQ gives people options and flexibility and interoperability, so a service written in Python can communicate with another in C# or Java, and both can be none the wiser.
A Beautiful Indirection
RabbitMQ has a straightforward programming model: Clients send messages to exchanges. The exchange acts as the broker’s front door, accepting incoming messages and routing them onward. An exchange looks at the incoming message and the message’s headers — and sometimes at one special header in particular, called a routing key — and decides to which queue (or queues) it should send the message. These exchanges can even send messages to other brokers. Queues are the thing consumers consume from. This beautiful indirection is why inserting an extra hop between a producer and a consumer is possible without affecting the producer or the consumer.
It all seems so straightforward — and boring! — when you think about it. But you wouldn’t believe how many people got this stuff wrong from the get-go. Let’s look at Java Message Service (JMS), the Java-standardized API for messaging. It has no concept of an exchange, so it is impossible to reroute a message (without sidestepping the JMS interfaces) once a producer and a consumer connect. Meanwhile, some JMS brokers couple the consumer and the producers by the Java driver client to talk to the broker. If the client supports version X of the broker, and someone has upgraded the broker to X+1, then the producer and the consumer may need to upgrade their Java client drivers to X+1.
Born in 2007, RabbitMQ was conceived due to the need for large banks to standardize their digital systems so they and their customers (that’s us) can transact more easily. RabbitMQ implemented the AMQP protocol from the jump; it’s still the most popular way to connect to the broker today. But it’s not the only way.
Let Me Count the Ways
As mentioned, RabbitMQ supports multiple protocols, which certainly offers choice, but there are other benefits as well. Like MQTT, popular in the Internet-of-Things space, where millions of clients — think microwaves, refrigerators and cars — might need to communicate with a single broker in a lightweight, efficient way. This work’s ongoing and keeps getting better by the day. For example, Native MQTT was recently announced, dramatically reducing memory footprint and increasing scalability.
RabbitMQ supports federation and active/passive deployments. It has various approaches to storing the messages in RAM or on disk. It supports transactions. It’s speedy, and it guarantees the consistency of your data. However, RabbitMQ has traditionally served messaging and integration use cases, not stream processing pipelines.
The community around RabbitMQ is vibrant, burgeoning and fast-moving, and the last few years have been incredibly prolific. Have you tried RabbitMQ Streams? Streams are a new persistent and replicated data structure that models an append-only log with nondestructive consumer semantics. You can use Streams from a RabbitMQ client library as a plain ol’ queue or through a dedicated binary protocol plugin and associated clients for even better throughput and performance.
To say it’s been successful would be an understatement. StackShare states that, among others, Reddit, Robinhood, Zillow, Backbase, Hello Fresh and Alibaba Travels all use RabbitMQ.
There are drivers for virtually every language, platform and paradigm. For example, I work on the Spring team, and we have several increasingly abstract ways by which you can use RabbitMQ, starting with the Spring for RabbitMQ foundational layer and going all the way up to the support in Spring Cloud Data Flow, our stream and batch processing stack.
RabbitMQ is open and extensible, supporting unique features with plugins to the server and extensions to the AMQP protocol.
The RabbitMQ site has a nonexhaustive list of some of the best ones. They include things like publisher confirms, dead letter exchanges, priority queues, per-message and per-queue TTL (time to live values tell RabbitMQ how long an unacknowledged message is allowed to remain in a queue before being deleted), exchange-to-exchange bindings and so much more. These are extensions to the protocol itself, implemented in the broker.
Plugins are slightly different. Numerous plugins extend the broker proper and introduce new management, infrastructure and engine capabilities. For example, there are plugins to support Kubernetes service discovery, OAuth 2, LDAP, WAN federation, STOMP and so much more.
What does all this mean for me? It means that, like PostgreSQL, RabbitMQ is a Swiss army knife. Does it do everything that the costly alternatives from Tibco or IBM do? No. But I’ll bet it can do 95% of whatever I’d need, and it’ll do so in a way that leaves my options open in the future. (And you can’t beat the price!)
Maybe it’s just all those years spent wearing a pager or running into data centers with winter jackets on at 3 a.m., but I actively avoid running anything I can’t charge for directly. I prefer to have someone run RabbitMQ for me. It’s cheap and easy enough to do so on several different cloud providers or the Kubernetes distribution of your choice.
As a developer, RabbitMQ couldn’t be more boring. As an operator in production, RabbitMQ couldn’t be more boring. I love boring, and I love RabbitMQ.
If boring appeals to you too, I encourage you to learn more about RabbitMQ, how it works as an event-streaming broker, how it compares to Kafka and about the beta of RabbitMQ as a service.
