Message Queues Explained: RabbitMQ, SQS, Kafka and When to Use Them

When a user uploads a video, you don't want them staring at a spinner while you transcode it into five formats. When a payment succeeds, you don't want the checkout request to fail because the email service is slow. Message queues let one part of a system hand work to another asynchronously, so each can run at its own pace and fail independently.

#Key takeaways

• A message queue decouples producers from consumers in time, scale and availability.
• Use queues for background jobs, spike absorption, fan-out and integrations between services.
• Most systems deliver at least once, so consumers must be idempotent.
• Configure retries with backoff and dead-letter queues for messages that keep failing.
• Traditional queues (RabbitMQ, SQS) suit task distribution; logs (Kafka) suit high-throughput event streams with replay.

#How a queue works

textProducer ──publish──▶ [ queue: m1 m2 m3 m4 ] ──deliver──▶ Consumer(s)
                                                    │
                                              ack when done

1. A producer publishes a message (for example, {"job": "resize_image", "image_id": 42}).
2. The broker stores it durably.
3. A consumer receives it, processes it and acknowledges it.
4. If the consumer crashes before acknowledging, the message becomes visible again and is redelivered.

#Why use a queue?

1. Responsiveness. Return to the user immediately; do slow work in the background.

2. Load leveling. Traffic spikes fill the queue instead of overwhelming downstream services. Workers drain it at a sustainable rate.

3. Decoupling. Producers don't need consumers to be online. The email service can be down for ten minutes without failing checkouts.

4. Scalability. Add consumers to process faster; scale workers independently from web servers (horizontal vs vertical scaling).

5. Reliability. Durable messages survive crashes; failed jobs retry automatically.

#Messaging patterns

• Work queue (point-to-point): each message is processed by one consumer. Used for background jobs.
• Publish/subscribe (fan-out): each message goes to all subscribers. OrderPlaced → billing, email, analytics (event-driven architecture).
• Request/reply: asynchronous RPC with a reply queue. Use sparingly.
• Delayed and scheduled messages: retry later, or send reminders.
• Priority queues: urgent jobs first.

#Delivery guarantees

Because duplicates happen (a consumer processes a message but crashes before acking, a network blip causes redelivery), design consumers to be idempotent:

pythondef handle_payment_succeeded(msg):
    event_id = msg["event_id"]
    with db.transaction():
        if db.processed_events.exists(event_id):
            return                                   # duplicate: already handled
        db.orders.mark_paid(msg["order_id"])
        db.processed_events.insert(event_id)         # same transaction as the effect
    ack(msg)

Other idempotency techniques: upserts keyed by business IDs, conditional updates ("set status = paid where status = pending"), and idempotency keys when calling external APIs (API design best practices).

#Retries and dead-letter queues

Not every failure is permanent. A robust consumer:

• Retries transient failures (timeouts, 503s) with exponential backoff and jitter.
• Doesn't retry permanent failures (invalid data) forever.
• Moves messages that exceed max attempts to a dead-letter queue (DLQ) for inspection.

Monitor DLQ size, alert on it, and build tooling to inspect, fix and redrive messages. A DLQ nobody watches is just a slower way to lose data.

#Ordering

• Many queues don't guarantee strict order, especially with multiple consumers and retries.
• FIFO queues or partitioned logs (Kafka partitions, SQS FIFO message groups) provide order per key.
• Design so that order matters only per entity (per order, per customer), then partition by that key.
• Alternatively, make handlers tolerant: include version numbers or timestamps and ignore stale updates.

#Backpressure and visibility

Watch these metrics:

• Queue depth / consumer lag: is work piling up?
• Age of oldest message: are users waiting too long?
• Processing time and error rate per message type.
• DLQ size.

Autoscale workers on queue depth, and alert when the oldest message exceeds your latency SLO (observability explained).

#The dual-write problem

Writing to the database and then publishing a message can fail halfway, leaving the database updated but no message sent (or vice versa). Use the transactional outbox pattern: write the event to an outbox table in the same transaction, and publish it from there (change data capture explained).

#RabbitMQ vs SQS vs Kafka

Also consider Redis-based job queues for simple background jobs in smaller apps, and cloud pub/sub services on other providers. For Kafka specifics, see Apache Kafka explained.

#When not to use a queue

• The caller needs the result immediately to respond. Use a synchronous call, or return a job ID with 202 Accepted.
• The work is trivial and fast. A queue adds latency, infrastructure and debugging complexity.
• You'd be using a queue to paper over a slow query that should simply be fixed.

#FAQ

#What's the difference between a message queue and an event stream?

A queue typically deletes messages after they're consumed and distributes work among consumers. An event stream (log) retains events for a period, allowing multiple independent consumers and replay. Kafka is the best-known stream; RabbitMQ and SQS are classic queues.

#How do I guarantee a message is processed exactly once?

End-to-end exactly-once is hard. The practical approach is at-least-once delivery with idempotent consumers, so processing a duplicate has no additional effect.

#Should I use my database as a queue?

For low volumes, a database table with row locking (for example, SELECT ... FOR UPDATE SKIP LOCKED in PostgreSQL) can work well and avoids another system. At higher throughput or with complex routing, a dedicated queue is usually better.

#How many consumers should I run?

Enough to keep queue depth and message age within your latency targets at peak. Autoscale on queue depth, and remember that downstream systems (databases, APIs) must handle the combined load.