Publish-Subscribe Systems
One event, many independent listeners — without the publisher needing to know who's listening or how many there are.
Intermediate
TopNotchNote
Notes that matter
one message, many independent recipients
A messaging queue connects one producer to consumers pulling from the same stream of work. A different, very common shape shows up when one event needs to reach several independent, unrelated systems at once: when a PlateRoute order is placed, Dispatch Service needs to know, Notification Service needs to know, and the Analytics pipeline needs to know — three unrelated consumers, all reacting to the same single event.
A publish-subscribe (pub-sub) system is built for exactly this: publishers write to a named topic without knowing who's listening, and any number of subscribers receive a copy of every message on that topic.
why not just call each interested service directly
Order Service could, in principle, call Dispatch, Notification, and Analytics directly. But that means Order Service needs to know about every current and future consumer, and adding a new consumer (say, a fraud-detection service six months from now) means modifying Order Service's code. Pub-sub decouples this completely: Order Service publishes "order placed" once, and new subscribers can be added later without Order Service ever changing.
a naive first design, and why it doesn't scale
The simplest implementation: give every subscriber its own dedicated queue, and have a "message director" copy each published message into every subscriber's queue. This works, but at scale it means potentially millions of individual queues (one per subscriber) and duplicating every message once per subscriber — expensive in both storage and operational complexity.
the broker-and-partition design most real pub-sub systems use
| Component | Role |
|---|---|
| Broker | Stores messages for a topic and serves both writes (from publishers) and reads (to subscribers). |
| Partition | A topic is split into multiple partitions, each an ordered, append-only log — this is what lets a topic scale write throughput horizontally, the same idea as partitioning a database or a message queue. |
| Cluster manager | Tracks which broker owns which partition, manages replication (a leader-follower model per partition), and handles failover. |
| Consumer manager | Tracks subscriber authorization and, in some designs, which offset each subscriber has consumed up to within a partition. |
A message published to a topic is routed to one of its partitions (often via a hash of a partitioning key, or round-robin). Within a partition, order is preserved by simple append — new messages always go at the end, identified by an increasing offset.
offsets: how consumers track their own progress
Unlike a queue where a consumed message is removed, pub-sub systems typically keep messages in the partition log for a configurable retention period, and each subscriber independently tracks its own offset — the position up to which it has read. This has a genuinely useful property: a new subscriber added later can, if the retention window still covers it, replay history from the beginning rather than only seeing messages published after it subscribed. It also means a slow subscriber falling behind doesn't affect other subscribers — each one's progress is independent.
pub-sub vs. a plain message queue
| Message queue | Pub-sub | |
|---|---|---|
| Consumer relationship | Competing consumers — each message is processed by exactly one consumer in the group | Fan-out — every subscriber gets its own copy of each message |
| Typical use | Distributing units of work across a worker pool | Broadcasting an event to multiple independent, unrelated systems |
| PlateRoute example | A pool of workers processing image-resize jobs for uploaded restaurant photos | "Order placed" fanning out to Dispatch, Notification, and Analytics |
The two patterns aren't mutually exclusive — a pub-sub topic can have a subscriber that's itself a worker pool doing competing consumption, combining both patterns at different layers.