Putting all the noise aside from fans of each service, who make it seem like their way is the only choice, this article will serve as an authoritative answer based on experience and knowledge of both systems. The insights shared in this article are based on in-depth discussions with experienced developers who had sound reasons for choosing one service over another.

The Terms

Both systems are message queues, though Kafka refers to their queue as a log. To simplify the information for better comparison, this article will refer to queues for both services. Messages in Kafka are often called records, but again, to simplify, this article will only refer to messages. Kafka also uses the word topic, which is essentially a categorization inside a queue. Topics are divided into partitions containing sequential records that cannot be changed.

The Similarities

Both RabbitMQ and Apache Kafka pass messages between producers and consumers through queues (topics). Messages can include any kind of information, such as a simple text message that triggers a chain of events on other applications. A messaging system is ideal for building a microservice platform, connecting different components, passing work onto remote workers, and real-time data streaming.

Both message services are considered mature, with RabbitMQ on the market since 2007 and Kafka operating since 2011. The reliability and scalability of both services are on pretty much equal footing, which is why the two are often compared. The differences between the two and your applications needs are what dictate which service to use when.

The Main Difference

Unlike most messaging systems, Kafka’s queue is persistent. This means that data sent to Kafka is stored until a specific time period has passed or a size limit has been reached. Until one of those two things happens, the message remains in the queue even after it is consumed. In Kafka, messages can be replayed or consumed multiple times; an adjustable setting useful in different scenarios.

RabbitMQ, on the other hand, stores messages until a receiving application connects to the queue and receives it. The client can acknowledge the message at that point or when the processing is complete; either way, once the message is acked, it’s gone from the queue.

To replay or not to replay

It is possible to replay messages in Kafka, meaning that your application can read a message multiple times. Taking advantage of the ability to replay a message in Kafka should be done mindfully. For example, saving a customer order multiple times is not usually a good idea. On the other hand, say your consumer has a bug and a newer version needs to be deployed. Being able to reprocess some or all of the messages at that point is a large benefit.

Messages in RabbitMQ can not be replayed, since they are removed once they are ack:ed. However, a RabbitMQ client can nack (negative acknowledgement) a message when it fails to handle the message; this is useful in case of a temporary failure on the consumer side. The message will simply be added back to the queue.

Protocol

Another difference between RabbitMQ and Apache Kafka is the protocol. While RabbitMQ supports several different protocols such as AMQP, MQTT, STOMP, etc., Kafka uses a custom protocol on top of TCP/IP to communicate between the applications and the cluster. RabbitMQ’s versatility in protocol gives it the advantage in more scenarios over Kafka.

Routing

The complexity with which messages are routed is another difference between the two message services. While RabbitMQ has a more complex routing method, Kafka is very simple. Simple is good, right? Not in this case; one of the biggest benefits of RabbitMQ is the flexibility it has to route messages.

RabbitMQ has four different options for routing messages - direct, topic, fanout, and header exchanges. Direct exchanges route the message to all queues that match its routing key exactly. Topics allow for wildcard matching and exact matching through a routing key. Fanout exchanges broadcast messages to every queue that is bound to the exchange. Header exchanges use the information found in the message headers and optional values, very similar to topic exchanges but without the routing keys. See this article for more information on different exchange types.

Kafka, on the other hand, does not support routing and instead relies upon partitions that contain messages in a sequence that is completely unchangeable. Creating your own dynamic routing with the help of Kafka streams is an option but not a default with Kafka and requires the use of consumer groups and persistent topics.

Priority

In RabbitMQ, a queue can be set up to have a range of priorities. Depending on the message’s priority, it is placed in the appropriate queue.

In Kafka, a message cannot be sent with a priority level, nor can it be delivered in priority order. All messages in Kafka are treated as equals and delivered in the order in which they are received no matter how busy the consumer is.

Acking

Acknowledgment - or acking - is the signal two processes give each other of receipt of the message sent or handled. Both RabbitMQ and Apache Kafka support producer acknowledgments to confirm messages have safely reached the broker. In RabbitMQ, a message can be considered delivered once it is sent or when it has been received by the consumer. Conversely, RabbitMQ clients can also nack (negative acknowledge) a message if it fails to be handled, returning it to the queue as if it were new.

Kafka maintains an offset for each message in a partition, with the committed position the last offset saved. If the process fails and restarts, it is this offset that it will recover to. Consumers in Kafka can commit offsets periodically or manually. How this is kept track of differs in different versions of Kafka.

Queue speed

For RabbitMQ, the speed of the queue is at its greatest when the queue is empty. Kafka is designed for holding large volumes of messages, so emptiness is not a factor in the speed. Enabling lazy queues in RabbitMQ is a great way to make a more stable cluster if you think your consumers won’t keep up with the speed of the publishers.

Scaling

Both RabbitMQ and Kafka are scalable in their own ways, giving you the ability to adjust the number of consumers, the power of the broker, or add more nodes as required.

Consumers

Publishing faster than your consumers perform in RabbitMQ may lead to the queue growing into millions of messages and eventually running out of memory. In this case, scaling the number of consumers that are handling the message is an easy way to accommodate this scenario. Each queue in RabbitMQ can have many consumers, who can all compete to consume the messages on the queue.

Kafka requires topic partitions to distribute consumers more efficiently, where each consumer in a group is dedicated to one or more partitions. Partitions can be set to send different sets of messages depending on the user id, location, or other factors.

Scaling broker

Kafka has the advantage when it comes to scaling. It's build with large scaling in mind. Adding more nodes to the cluster or adding more partitions to the topics are easy ways to scale up in Kafka. In RabbitMQ, vertical scaling - adding more power - is the easiest way to scale up. Because there will always be a limit on how massive the machines you can buy are, the horizontal scaling in Kafka is an advantage.

However, both message services can support large message volumes per second without any issues, as the scale where RabbitMQ or Kafka would run out of space is a rare scenario.

Log compaction (Kafka as a Database)

This does not exist in RabbitMQ but is a feature that makes Apache Kafka stand out. Log compaction ensures that the last known value for each message key is kept within the queue for a single topic partition. In other words, Kafka keeps the latest version of a message, deleting the older versions that have the same key.

Log compaction ensures that the latest information is available immediately, such as if we are showing the latest status of one cluster among thousands running. Instead of storing whether a cluster is responding or not each and every time, only the final status is stored.

Monitoring

RabbitMQ’s interface allows for monitoring and handling of the server from a web browser, including queues, connections, channels, exchanges, users, permissions and more can be created, deleted, and listed in the interface. While there are open-source and commercial tools available for monitoring administering Kafka, they are separate. Find more information about the tools available here.

Push/Pull

In RabbitMQ, messages are pushed to the consumer, making a prefetch limit configuration necessary to prevent the consumer from becoming overwhelmed by too many messages. Pulling messages from RabbitMQ is possible, but not recommended. Kafka uses a pull model, where consumers request messages in batches from an offset.

License

Both RabbitMQ and Kafka are free and open-source software licenses. Kafka components such as Rest Proxy, Schema Registry, and KSL are covered by another license called Confluent Community License, which still allows for free download, modification, and redistribution, but does not allow anyone to provide the software as a service (SaaS) offering. If Kafka changes the license again to something stricter, RabbitMQ will have the advantage of being easily replaceable by another AMQP broker while Kafka will not have this advantage.

Complexity

The opinion of the developers we work with is that Kafka’s architecture is more complex, as it includes more concepts such as topics, partitions, and message offsets. Being familiar with consumer groups and handling offsets is a prerequisite of working with Kafka. Failure handling, in particular, is a complication of working with Kafka, which requires more time and is more complicated than in RabbitMQ.

Ecosystem

Kafka is more than a broker - it is a streaming platform with many different tools that can be integrated outside the main platform. These tools include Kafka Core, Kafka Streams, Kafka Connect, Kafka REST Proxy, and the Schema Registry. Most of these additional tools come from Confluent, which is not a part of Apache.

The benefit of these tools is that a huge system can be configured before it becomes necessary to write any code whatsoever. In the case of Kafka Connect, integrating other data sources with Kafka expands processing and storage capabilities quickly and easily. Kafka REST proxy adds the possibility of receiving metadata from a cluster and produce or consumer messages over a simple REST API, a feature easily enabled from the cluster control panel.

Use Cases

After all the information about what each system can or can’t do, here are some use cases that have been written after real customer experiences with the RabbitMQ and Kafka, and why they decided on one over the other.

Use Cases - RabbitMQ

Two main use cases stand out to make the choice for RabbitMQ: long-running tasks and integration between microservice applications. In general, however, RabbitMQ is a simple and traditional pub/sub message broker. It will scale more than most systems will require it to and is easy to use immediately after installation. RabbitMQ is also a good choice for systems with simple requirements and those that do not need retention and streaming.

Long-running tasks

Message queues are, by nature, asynchronous processing of data - meaning that they allow messages to be placed in queues without dealing with it immediately. An example of such a scenario is a image scaling service that allows users to upload images on a website, with the intention of scaling the image and emailing it back to the user. Event-based microservice architectures handling hundreds of millions of users are perfect for RabbitMQ.

Integration between applications

Services like MapQuest, supporting 23.1 million unique mobile users on a monthly basis, rely on RabbitMQ to support topics spread over multiple queues. In acting as the middle-man for communication between applications, RabbitMQ enables systems to avoid bottlenecks when passing messages back and forth.

Use Cases - Apache Kafka

The perfect framework for storing, reading, re-reading, and analyzing streaming data is where Apache Kafka is the better choice. Systems that are audited or that need to permanently store messages are where Kafka is right at home.

Analysis, tracking, ingestion, logging, security

In all of these scenarios, large amounts of data must be collected, stored, and handled. Platforms that need to provide search features, analysis, or auditing of large amounts of data are completely satisfied with the performance of Kafka. Going back to the roots of Kafka, it was originally meant to track website activity such as page views, uploads, and other user actions.

With Kafka, producers send data to a single place, where it is then taken charge of by a host of backend services that consume the data as required. Major data analytical systems and storage/search systems integrate their functions with Kafka.

Real-Time processing

The high-throughput capacity of a Kafka system pushes streams of data into the target services, which are pulling the data through in real-time. Publishing services like Spotify and Rabobank publish information using the real-time services of Kafka. High-throughput and real-time combine to equal a powerful application for logs, metrics, and more.

RabbitMQ

Apache Kafka

What it is?

RabbitMQ is a solid, mature, general purpose message broker

Apache Kafka is a message bus optimized for high-ingress data streams and replay

Primary use

Message queue for communication and integration within, and between applications. For long-running tasks, or when you need to run reliable background jobs.

A framework for storing, reading (re-reading), and analyzing streaming data. Optimal for

License

Open Source: Mozilla Public License

Open Source: Apache License 2.0

Written in

Erlang

Scala (JVM)

First Version Released

2007

2011

Persistence

Persist messages until they are dropped on the acknowledgement of receipt

Persists messages with an option to delete after a retention period

Replay

No

Yes

Routing

Supports flexible routing which can return information to a consumer node

Does not support flexible routing, must be done through separate topics

Message Priority

Supported

Not supported

Monitoring

Available through a built-in UI

Available through third-party tools such as when deployed on CloudKarafka or through Confluent

Language Support

Most languages are supported

Most languages are supported

Secure Authentication

Supports standard authentication and OAuth2

Supports Kerberos, OAuth2, and standard authentication

Need a bit more guidance or a recommendation? Get in touch with us at [email protected].