Battle of the Brokers: Kafka vs. RabbitMQ vs. SQS in Production

The Core Mental Models#

Before comparing features, you must understand the two fundamentally different paradigms at play: Message Queuing and Event Streaming.

The Message Queue (RabbitMQ & SQS)#

Think of a message queue like a post office. The broker takes on the heavy lifting (Smart Broker). It tracks exactly who gets what message. When a consumer successfully processes a message, it acknowledges it, and the broker destroys the message.

• State: Maintained by the broker.
• Replayability: Zero. Once it is gone, it is gone.
• Concurrency: You just add more consumers to the queue, and the broker round-robins the work.

The Event Stream (Kafka)#

Think of an event stream like a corporate ledger. The broker is just a dumb, fast, append-only file system. It simply writes events to disk sequentially. It does not track message deletion. Instead, consumers are responsible for remembering their place in the log (Smart Consumer).

• State: Maintained by the consumer (via offsets).
• Replayability: High. Just rewind your offset and read the history again.
• Concurrency: Bound by the number of partitions.

Deep Dive: RabbitMQ#

RabbitMQ is the Swiss Army knife of traditional messaging. It implements the AMQP protocol and introduces a powerful concept: Exchanges.

Instead of publishing directly to a queue, producers publish to an Exchange. The Exchange uses Binding Rules to route messages to zero, one, or many queues based on routing keys, headers, or pattern matching.

# Declaring a direct exchange and binding a queue for specific routing
channel.exchange_declare(exchange='media_events', exchange_type='direct')
channel.queue_declare(queue='video_processing')
 
# Only route messages with the key 'video.uploaded' to this queue
channel.queue_bind(
    exchange='media_events', 
    queue='video_processing', 
    routing_key='video.uploaded'
)

When to use RabbitMQ:

• You need complex, granular routing logic (e.g., "route this message to the European billing queue only if the user is a premium member").
• You require strict point-to-point delivery with fast acknowledgments.
• You are not doing stream processing or massive historical replay.

Deep Dive: Amazon SQS#

Amazon Simple Queue Service (SQS) is the ultimate "boring technology." It is a fully managed, serverless queuing service. You don't provision nodes, you don't manage clusters, and you don't worry about ZooKeeper or KRaft.

SQS's superpower is its Visibility Timeout. When a worker pulls a message, SQS hides it from other workers for a set period. If the worker processes it, it deletes it. If the worker crashes, the timeout expires, and the message reappears for another worker to grab.

When to use SQS:

• You are entirely in the AWS ecosystem.
• You need infinite scaling without capacity planning.
• You are building simple task queues (e.g., asynchronous PDF generation, email sending).

Deep Dive: Apache Kafka#

Kafka (as explored deeply in our previous architecture guide) is built for massive, sequential disk I/O. It thrives when you have multiple disparate systems that all need to react to the same source of truth over time.

When to use Kafka:

• Event Sourcing: The stream is the database.
• Massive Fan-out: You have one topic (e.g., user_clicks) and 15 different microservices that need to read it independently without impacting each other.
• High Throughput: You are processing millions of messages per second.

Hybrid Architectures: Better Together#

A common mistake is assuming you must pick exactly one broker for your entire company. In mature, 10+ year systems, the most robust architectures combine the raw throughput of Kafka with the granular error handling of a queuing system like SQS or RabbitMQ.

The Problem: Kafka and Poison Pills#

Kafka requires strict ordering within a partition. If your consumer crashes on a malformed message (a "poison pill"), you cannot simply tell Kafka, "retry this one message later and move on to the next one." You either block the whole partition, or you discard the message.

The Solution: The Event Backbone + Worker Queues#

We use Kafka as the central nervous system for inter-domain events, but we use SQS as the muscle for flaky, third-party integrations (like sending emails or charging credit cards) that require granular retries and Dead Letter Queues (DLQs).

How it works:

1. The Order Service drops an event into Kafka.
2. The Email Consumer microservice reads from Kafka. Its only job is to immediately take the payload and write it into an SQS Queue, then advance its Kafka offset. It never blocks.
3. Fleet of serverless SMTP Workers poll SQS. If a worker fails to send the email due to a network glitch, SQS's visibility timeout handles the retry.
4. If it fails 5 times, SQS natively routes it to a DLQ for manual inspection.

{
  "deadLetterTargetArn": "arn:aws:sqs:us-east-1:123456789012:email-dlq",
  "maxReceiveCount": 5
}

This hybrid pattern guarantees you never block your high-throughput Kafka partitions while still getting enterprise-grade per-message retry semantics.

In Production#

Test your understanding#

Interview questions#