Pub-Sub Patterns

Publish-subscribe decouples message producers from consumers by introducing an intermediary that routes messages based on subscriptions. Publishers emit events without knowing who receives them, and subscribers express interest without knowing who produces them.

Fan-Out

Fan-out delivers a single message to multiple consumers simultaneously. When an event occurs, every interested service receives its own copy.

Simple Fan-Out

A publisher sends one message to an exchange or topic, and the broker copies it to every bound queue or subscription.

Order Service publishes "order.placed"
  --> Inventory Service (reserve stock)
  --> Payment Service (charge customer)
  --> Notification Service (send confirmation email)
  --> Analytics Service (track conversion)

Amazon SNS combined with SQS is the classic fan-out pattern on AWS. SNS receives the message once and delivers copies to multiple SQS queues, each owned by a different service. Uber uses this pattern extensively: a single trip completion event fans out to billing, driver payments, rider receipts, surge pricing updates, and fraud detection.

Weighted Fan-Out

Not all subscribers need to process every message. Some systems route a percentage of traffic to specific consumers for A/B testing or gradual rollouts.

Event: "page.viewed"
  --> Analytics Service (100% of messages)
  --> New Recommendation Engine (10% of messages, canary)
  --> Legacy Recommendation Engine (90% of messages)

Fan-In

Fan-in aggregates messages from multiple producers into a single consumer or processing pipeline. This is the inverse of fan-out.

Mobile App    --> [Topic: user.events] --> Event Aggregation Service
Web App       --> [Topic: user.events] --> Event Aggregation Service
IoT Devices   --> [Topic: user.events] --> Event Aggregation Service

Fan-in is common in logging and metrics collection. Datadog ingests metrics from thousands of agents across customer infrastructure, fanning them into a central pipeline that processes, aggregates, and stores the data.

Competing Consumers with Fan-In

When multiple producers generate high volume, a single consumer cannot keep up. Use a consumer group where members split the work.

Thousands of producers --> [Topic with 64 partitions]
  --> Consumer Group (16 instances, each reads 4 partitions)
  --> Aggregated output

Topic-Based Filtering

Subscribers select messages by subscribing to specific topics or topic patterns. This is the most straightforward filtering model.

Hierarchical Topics

Many brokers support topic hierarchies with wildcard subscriptions.

Topic hierarchy:
  orders.created
  orders.shipped
  orders.cancelled
  payments.completed
  payments.failed

Subscriptions:
  "orders.*"      --> gets all order events
  "orders.created" --> gets only creation events
  "*.failed"       --> gets all failure events across domains

RabbitMQ implements this with topic exchanges. MQTT, used extensively in IoT, supports hierarchical topics natively. Tesla's vehicle fleet publishes telemetry to hierarchical MQTT topics, with different backend services subscribing to the specific data streams they need.

Partitioned Topics

Kafka uses partitioned topics where a partition key determines message placement. Consumers subscribe to entire topics but are assigned specific partitions. This is not filtering per se, but it controls which consumer instance processes which subset.

Topic: "user-actions" (12 partitions)
  Partition key: userId
  Consumer Group A (3 instances): each reads 4 partitions
  Consumer Group B (6 instances): each reads 2 partitions

Content-Based Filtering

Content-based filtering routes messages by inspecting message attributes or body content, rather than relying on topic names.

Attribute Filtering

SNS supports message attribute filtering. Subscribers declare filter policies, and only matching messages are delivered.

Publisher sends:
  Topic: "orders"
  Attributes: { region: "eu-west", priority: "high", amount: 5000 }

Subscriber A filter: { region: ["us-east"] }          --> does NOT receive
Subscriber B filter: { priority: ["high"] }            --> receives
Subscriber C filter: { region: ["eu-west"], priority: ["high"] } --> receives

This reduces unnecessary message processing. Without filtering, Subscriber A would receive, deserialize, and discard messages it does not care about.

SQL-Based Filtering

Azure Service Bus and Apache ActiveMQ support SQL-like filter expressions on message properties.

Subscription filter: "region = 'eu-west' AND amount > 1000"

Google Cloud Pub/Sub supports attribute-based filtering. Shopify uses this to route webhook events: merchants subscribe to specific event types (order creation, inventory updates) rather than receiving all events and discarding irrelevant ones.

Tradeoffs of Content-Based Filtering

Content-based filtering moves routing logic to the broker, which increases broker CPU usage. Topic-based filtering is cheaper because routing is a simple lookup. Use content-based filtering when the number of distinct routing criteria would create an unmanageable number of topics.

Topic-based: 1 topic per event type (manageable at 20 types, unwieldy at 2000)
Content-based: 1 topic with filter expressions (handles high cardinality)

At-Least-Once vs Exactly-Once Delivery

Delivery semantics determine how the system handles failures during message transmission and processing.

At-Least-Once Delivery

The publisher retries until it receives acknowledgment from the broker. The broker retries until the consumer acknowledges. Messages may be delivered more than once, but never lost.

At-least-once flow:
  Publisher -> Broker: send message
  Broker -> Publisher: ack
  (if no ack, publisher retries --> possible duplicate at broker)

  Broker -> Consumer: deliver message
  Consumer -> Broker: ack
  (if no ack, broker redelivers --> possible duplicate at consumer)

This is the default for most systems: SQS, SNS, RabbitMQ, Kafka (with acks=all). It requires consumers to be idempotent.

Exactly-Once Delivery

True exactly-once delivery across distributed systems is impossible in the general case due to the two generals problem. What systems actually provide is exactly-once processing semantics through a combination of deduplication and idempotent operations.

Exactly-once processing strategies:
  1. Idempotent consumer: processing the same message twice produces same result
  2. Deduplication: broker or consumer tracks message IDs, discards duplicates
  3. Transactional outbox: database transaction + message publish are atomic

Kafka Streams provides exactly-once semantics within the Kafka ecosystem by combining idempotent producers, transactional writes, and consumer offset management in a single atomic operation. This works because both input and output are Kafka topics controlled by the same system.

Stripe's payment processing uses idempotency keys. Even if a payment creation message is delivered twice, the second attempt returns the result of the first without charging the customer again.

Choosing Delivery Semantics

| Semantic       | Guarantee                | Consumer Requirement | Use Case                |
|----------------|--------------------------|----------------------|-------------------------|
| At-most-once   | May lose messages         | None                 | Metrics, telemetry      |
| At-least-once  | No loss, possible dupes   | Idempotency          | Most business logic     |
| Exactly-once   | No loss, no dupes         | Transactional support | Financial transactions  |

Slow subscriber blocking fast subscribers. In fan-out, one slow consumer should not create backpressure on others. Use independent queues per subscriber.
Message ordering assumptions in fan-out. Different subscribers may process the same set of messages in different orders, leading to inconsistent state if not handled.
Filter explosion. Creating thousands of fine-grained topic subscriptions adds broker overhead. Consolidate with content-based filtering when cardinality is high.
Ignoring poison messages in pub-sub. A malformed message delivered to many subscribers causes failures everywhere simultaneously. Validate at the publisher.
Confusing delivery with processing. At-least-once delivery does not mean at-least-once successful processing. Consumers can receive a message and still fail to handle it correctly.
No backpressure mechanism. Unbounded fan-out to slow consumers fills queues and eventually crashes the broker or exhausts disk.

Key Takeaways

Fan-out delivers one event to many consumers; fan-in aggregates many sources into one pipeline. Both are fundamental to event-driven systems.
Topic-based filtering is efficient for coarse routing. Content-based filtering handles high-cardinality routing criteria at the cost of broker CPU.
At-least-once delivery is the practical default. Exactly-once processing is achievable through idempotency and deduplication, not through the messaging layer alone.
Independent subscriber queues prevent slow consumers from affecting others during fan-out.
Pub-sub decouples services in time and space, but introduces complexity in ordering, delivery guarantees, and failure handling that must be explicitly designed for.