Message Queue Comparison
Apache Kafka
Distributed log. Ordered within partitions, replay-able, high throughput (millions/sec). Best for event streaming, log aggregation, CDC. Not great for task queues.
RabbitMQ
Traditional message broker (AMQP). Rich routing (direct, topic, fanout, headers). Best for task queues, RPC, complex routing. Lower throughput than Kafka.
Amazon SQS
Fully managed, serverless. Standard (at-least-once, best-effort order) or FIFO (exactly-once, strict order). Best for decoupling AWS services. Max 256KB message.
Redis Streams
Append-only log in Redis. Consumer groups, acknowledgment, range queries. Best for lightweight streaming when you already run Redis. Limited by memory.
Kafka vs RabbitMQ
Kafka: pull-based, persistent log, replay. RabbitMQ: push-based, message deleted after ack, no replay. Kafka for events, Rabbit for tasks.
Delivery Guarantees
At-most-once: fire and forget. At-least-once: retry until ack (may duplicate). Exactly-once: hardest — Kafka supports it via idempotent producers + transactions.
Ordering Guarantees
Kafka: ordered per partition (use same key). SQS Standard: no order. SQS FIFO: ordered per message group. RabbitMQ: ordered per queue (single consumer).
Backpressure
Kafka: consumers pull at their own pace. RabbitMQ: prefetch count limits unacked messages. SQS: visibility timeout. All prevent slow consumers from being overwhelmed.
Dead Letter Queue (DLQ)
Messages that fail processing N times are moved to a DLQ for inspection. Supported natively by SQS, RabbitMQ. Kafka needs custom implementation.
When to Use What
Event streaming/replay → Kafka. Task distribution/RPC → RabbitMQ. Serverless/AWS → SQS. Simple + already have Redis → Redis Streams. High volume logs → Kafka.