Design a Push Subscription & Pub/Sub System

Question

Design a global publish/subscribe system with millions of subscribers, durable delivery, and filtering.

Accepted Answer

Designing a global publish/subscribe system for millions of subscribers with durable delivery and filtering presents a significant engineering challenge. It requires a robust architecture that balances scalability, reliability, and message guarantees. **Messaging Model & Fanout:** We'd likely employ a topic-based publish/subscribe model rather than a queue-based one. Topics allow for natural fanout, where a single message published to a topic can be delivered to multiple distinct subscriber groups. This avoids the complexity of managing individual queues for each subscriber and scales better for a large number of subscribers. A distributed log like Apache Kafka or AWS Kinesis would be a strong candidate for the underlying messaging backbone due to its inherent durability and high throughput. **Durable Storage & Delivery Guarantees:** Durable delivery is paramount. The messaging backbone must persist messages to disk (e.g., Kafka's replicated logs) until they are successfully acknowledged by subscribers. We would implement at-least-once delivery semantics. Subscribers would process messages and send explicit acknowledgments. If an acknowledgment is not received within a timeout, the message is retried. This requires careful handling on the subscriber side to ensure idempotency to avoid duplicate processing. **Filtering Mechanism:** Filtering can be implemented in a few ways. We could implement server-side filtering within the broker, where subscribers subscribe to specific filtered views of a topic. This is efficient but adds complexity to the broker. Alternatively, and often more scalable, we can implement client-side filtering where subscribers receive all messages and filter them locally. For more sophisticated filtering, a dedicated stream processing engine (e.g., Apache Flink, Spark Streaming) could be employed to pre-filter messages before they reach subscribers, publishing filtered streams to new topics. **Scalability & Distribution:** To handle millions of subscribers globally, the system must be distributed and horizontally scalable. This involves sharding topics across multiple broker nodes. Subscribers would connect to specific partitions of topics, allowing for parallel processing. Geographic distribution would be handled through multi-region deployments, potentially using cross-region replication or specialized global load balancers and DNS to route subscribers to their nearest regional endpoint. Client libraries would need to be designed to handle disconnections and reconnections seamlessly. **Acknowledgments, Retries, and Ordering:** Acknowledgments are crucial for durable delivery. Subscribers must explicitly confirm message processing. Retries for failed deliveries will be implemented with an exponential backoff strategy to avoid overwhelming subscribers or the broker. Global ordering guarantees are extremely difficult and expensive to maintain at scale. We would likely aim for per-partition ordering, which is naturally provided by systems like Kafka. If global ordering is a strict requirement, it would significantly impact performance and complexity, potentially requiring a distributed consensus mechanism or a single writer per topic, which isn't feasible for millions of subscribers. Robust monitoring is essential for detecting issues early, such as high latency, increased error rates, or unacknowledged messages. Proactive measures like automated scaling and circuit breakers can help prevent failures before they impact the system.

Design a Push Subscription & Pub/Sub System

Ready to prove your AWS SysOps knowledge?

Scaling Message Queues

Exactly-Once Effects with the Outbox Pattern

Design a Multi-Channel Notification Service

Design a Distributed Job Scheduler

K8s Readiness vs Liveness Probes

Taming Toil: Eliminating Repetitive Work to Scale SRE Teams

The Pragmatic SRE Guide to SLOs: From Business Goals to Error Budgets

Kubernetes Production Readiness Checklist