Distributed Messaging
Why Messaging?
Messaging decouples producers and consumers, enabling asynchronous communication, load leveling, and fault tolerance.
Without messaging (tight coupling):
Service A ──sync call──> Service B ──sync call──> Service C
(A blocked until C responds; C down = A down)
With messaging (loose coupling):
Service A ──> [Message Queue] ──> Service B ──> [Queue] ──> Service C
(A publishes and continues; B/C process independently)
Message Queue Fundamentals
Point-to-Point (Queue)
Each message consumed by exactly one consumer.
Producer ──> [ msg3 msg2 msg1 ] ──> Consumer A (gets msg1)
──> Consumer B (gets msg2)
──> Consumer C (gets msg3)
Load balancing: messages distributed across consumers
Publish-Subscribe (Topic)
Each message delivered to all subscribers.
Publisher ──> [ Topic: orders ]
│
┌─────┼─────┐
▼ ▼ ▼
Sub A Sub B Sub C
(each gets every message)
RabbitMQ
AMQP-based message broker. Routes messages through exchanges to queues.
Architecture:
Producer ──> Exchange ──(binding)──> Queue ──> Consumer
Exchange Types:
Direct: route by exact routing key match
Fanout: broadcast to all bound queues
Topic: route by routing key pattern (*.orders.#)
Headers: route by message header attributes
Producer Exchange Queues
│ ┌──────┐
│── rk="order.new" ─>│Direct│──> [order-processing]
│ │ │
│── rk="order.new" ─>│Topic │──> [order-processing] (order.*)
│ │ │──> [analytics] (*.*)
│ │ │──> [audit-log] (#)
│ └──────┘
Acknowledgment modes:
- Auto-ack: message removed on delivery (at-most-once)
- Manual ack: consumer explicitly acknowledges (at-least-once)
- Publisher confirms: broker confirms receipt to publisher
Apache Kafka
Distributed commit log. Designed for high-throughput, durable, ordered message streaming.
Architecture
Kafka Cluster
┌──────────────────────────────────────────────┐
│ Topic: orders (3 partitions, RF=3) │
│ │
│ Partition 0: [0][1][2][3][4][5][6]──> │
│ Leader: Broker 1 │
│ Replicas: Broker 2, Broker 3 │
│ │
│ Partition 1: [0][1][2][3][4]──> │
│ Leader: Broker 2 │
│ Replicas: Broker 1, Broker 3 │
│ │
│ Partition 2: [0][1][2][3][4][5]──> │
│ Leader: Broker 3 │
│ Replicas: Broker 1, Broker 2 │
└──────────────────────────────────────────────┘
Partitions and Ordering
Messages within a partition are strictly ordered. No ordering guarantee across partitions.
Producer sends: key="user-42" → hash(key) mod 3 = partition 1
Partition 1: [msg1] [msg2] [msg3] [msg4] →
offset 0 1 2 3
All messages for user-42 go to same partition
→ guaranteed order for this user's events
Consumer Groups
Consumer Group "order-processors":
┌─────────┐
│Consumer 1│ ← Partition 0
│Consumer 2│ ← Partition 1
│Consumer 3│ ← Partition 2
└─────────┘
Consumer Group "analytics":
┌─────────┐
│Consumer A│ ← Partition 0, 1
│Consumer B│ ← Partition 2
└─────────┘
Each partition assigned to exactly one consumer per group.
Adding consumers beyond partition count = idle consumers.
In-Sync Replicas (ISR)
ISR = set of replicas that are caught up to the leader
Partition 0:
Leader (Broker 1): [0][1][2][3][4][5]
Follower (Broker 2): [0][1][2][3][4][5] ← in ISR
Follower (Broker 3): [0][1][2][3] ← lagging, removed from ISR
acks=all: producer waits for ALL ISR members to replicate
acks=1: producer waits for leader only
acks=0: fire and forget
min.insync.replicas=2: at least 2 ISR members required
for acks=all writes to succeed
Log Compaction
Retains only the latest value for each key. Useful for changelogs and state snapshots.
Before compaction:
[k1:v1] [k2:v1] [k1:v2] [k3:v1] [k2:v2] [k1:v3]
After compaction:
[k3:v1] [k2:v2] [k1:v3]
Only the latest value per key survives.
Tombstone (key with null value) → key deleted after retention.
// Simplified Kafka consumer logic
STRUCTURE KafkaConsumer
group_id: string
assigned_partitions: list of TopicPartition
committed_offsets: map of TopicPartition → integer
STRUCTURE TopicPartition
topic: string
partition: integer
ASYNC PROCEDURE POLL(consumer, timeout) → list of Record
records ← empty list
FOR EACH tp IN consumer.assigned_partitions DO
offset ← consumer.committed_offsets.GET(tp, default ← 0)
// Fetch from broker starting at offset
fetched ← AWAIT FETCH(consumer, tp, offset, timeout)
APPEND fetched TO records
RETURN records
ASYNC PROCEDURE COMMIT(consumer) → success or error
// Commit current offsets to __consumer_offsets topic
FOR EACH (tp, offset) IN consumer.committed_offsets DO
AWAIT COMMIT_OFFSET(consumer, tp, offset)
PROCEDURE PROCESS_RECORD(consumer, record)
// Process the record...
// Update committed offset
tp ← TopicPartition(topic ← record.topic, partition ← record.partition)
consumer.committed_offsets[tp] ← record.offset + 1
Apache Pulsar
Multi-tenant, geo-replicated messaging. Separates serving (brokers) from storage (BookKeeper).
Architecture:
┌──────────────────────────────────────┐
│ Pulsar Brokers (stateless) │
│ - Handle produce/consume │
│ - Topic ownership (load balance) │
└──────────────┬───────────────────────┘
│
┌──────────────┴───────────────────────┐
│ Apache BookKeeper (storage) │
│ - Append-only log segments │
│ - Distributed, replicated │
│ - Tiered storage (offload to S3) │
└──────────────────────────────────────┘
Advantages over Kafka:
- Broker is stateless → easier scaling/rebalancing
- Millions of topics (topic = metadata pointer)
- Built-in multi-tenancy
- Tiered storage (hot/cold separation)
- Built-in schema registry
Delivery Semantics
At-Most-Once
Send and forget. Messages may be lost.
Producer ──msg──> Broker (no ACK waited)
──msg──> Broker (msg lost in transit, no retry)
At-Least-Once
Retry until acknowledged. Messages may be duplicated.
Producer ──msg──> Broker ──ACK──> Producer ✓
Producer ──msg──> Broker ──ACK lost──> Producer
Producer ──msg──> Broker (duplicate!)──ACK──> Producer
Consumer must handle duplicates (idempotent processing)
Exactly-Once
Each message processed exactly once. Hardest to achieve.
Kafka's Exactly-Once (Idempotent Producer + Transactions):
Producer assigns sequence number per partition:
[msg seq=0] [msg seq=1] [msg seq=2] [msg seq=1 retry]
↑ broker detects
duplicate, dedupes
Transactional produce + consume:
begin_transaction()
produce(output_topic, result)
commit_offsets(input_topic, offset) // atomic with produce
commit_transaction()
Either BOTH happen or NEITHER happens.
// Idempotent consumer pattern
STRUCTURE IdempotentProcessor
processed_ids: set of string
// In production: use a persistent store (DB, Redis)
PROCEDURE PROCESS(processor, msg_id, payload) → boolean
IF msg_id IN processor.processed_ids THEN
RETURN FALSE // already processed, skip
// Process the message
DO_WORK(payload)
ADD msg_id TO processor.processed_ids
RETURN TRUE
PROCEDURE DO_WORK(payload)
// actual business logic
Ordering Guarantees
Total order: All consumers see messages in exactly the same order
(single partition, or consensus-based)
Partition order: Messages within a partition are ordered
(Kafka, Pulsar per-partition)
Causal order: If msg A caused msg B, A delivered before B
(requires tracking dependencies)
No order: Messages delivered in any order
(most queue systems across multiple queues)
Backpressure
Mechanisms to handle producers outpacing consumers.
Strategies:
1. Blocking: Producer blocks when queue is full
2. Dropping: Drop oldest/newest messages
3. Buffering: Spill to disk (Kafka's strength)
4. Rate limiting: Throttle producer
5. Reactive: Consumer signals capacity upstream
Reactive Streams (pull-based):
Consumer ──request(n)──> Producer
Producer ──send(n items)──> Consumer
Consumer ──request(m)──> Producer
...
Consumer controls flow rate.
Comparison
| Feature | RabbitMQ | Kafka | Pulsar | |---|---|---|---| | Model | Queue + Pub/Sub | Commit log | Commit log | | Ordering | Per-queue | Per-partition | Per-partition | | Retention | Until consumed | Time/size-based | Tiered storage | | Throughput | ~50K msg/s | ~1M msg/s | ~1M msg/s | | Replay | No (consumed = gone) | Yes (offset seek) | Yes (cursor) | | Protocol | AMQP | Custom binary | Custom binary | | Multi-tenancy | vhosts | Limited | Native | | Use case | Task queues, RPC | Event streaming | Multi-tenant streaming |
Real-World Patterns
| Pattern | System | Use Case | |---|---|---| | Event sourcing | Kafka | Persist state as event log | | CQRS | Kafka + DB | Separate read/write models | | Change data capture | Debezium + Kafka | DB changes to stream | | Dead letter queue | RabbitMQ/Kafka | Failed message handling | | Saga orchestration | Kafka/Temporal | Distributed transactions | | Stream processing | Kafka Streams/Flink | Real-time analytics |
Key Takeaways
- Kafka's append-only log model is fundamentally different from traditional message queues: messages persist, consumers control their position, and replay is a first-class feature.
- Exactly-once semantics require cooperation between producer, broker, and consumer. In practice, idempotent consumers are the most reliable approach.
- Partition count determines maximum parallelism. Choose it carefully -- increasing partitions later requires rebalancing.
- Backpressure is essential in production. Without it, a fast producer can overwhelm consumers, broker memory, or disk.
- Pulsar's separation of compute (brokers) and storage (BookKeeper) enables independent scaling -- a pattern increasingly adopted in modern systems.