Design a Social News Feed

Question

Design a personalized news feed for a social network with billions of events per day. Cover write fan-out vs read fan-out, ranking, and cold start.

Accepted Answer

Designing a personalized news feed at the scale of billions of events per day presents significant system design challenges, particularly around data propagation, storage, and intelligent ranking. My approach would involve a hybrid fan-out model, robust storage, and an iterative ML-driven ranking system to deliver a highly relevant and fresh user experience. **1. Hybrid Fan-out Architecture (Write vs. Read Fan-out):** For a system with billions of events, a hybrid approach is optimal. We'd use **fan-out-on-write (push model)** for the majority of users (e.g., those with under 100k followers). When these users post, content is pushed to the inboxes (timeline caches) of their followers, ensuring low read latency. For **super-users** with massive follower counts (e.g., over 100k), we'd employ **fan-out-on-read (pull model)**. Their content would be stored in a dedicated 'hot content' pool, and follower feeds would pull this content during aggregation. This minimizes write amplification for popular creators while maintaining good read performance for most users. **2. Data Storage and Timeline Service:** All user activities are initially published to a distributed, immutable **activity log** (e.g., Kafka) for real-time processing and archival. User content is also stored in an **Outbox** (e.g., sharded Cassandra) for direct access. For fan-out-on-write, followers receive entries in their personalized **Inboxes** (often a low-latency key-value store like Redis). These inboxes primarily store content references (IDs) and minimal metadata. To manage data volume and ensure relevance, we'd implement **TTL (Time-to-Live)** on inbox entries (e.g., 2-7 days), archiving older content to cheaper storage. **Deduplication** logic would be applied both during content propagation and at read time to ensure unique content display. **3. ML-driven Ranking and Personalization:** A sophisticated machine learning model would power personalization. **Offline training** would leverage historical user interaction data (likes, shares, comments, dwell time) and diverse features: **content features** (topic, media type, virality), **user features** (demographics, interests, past behavior), **graph signals** (social proximity to content creator, strength of connection), and **freshness signals** (decay factor for older content). During **online scoring**, when a user requests their feed, a candidate set of posts (from their inbox, pulled super-user content, and recommendations) is gathered. The trained model then scores each candidate, ranking them by predicted engagement. **A/B testing** is crucial for continuous iteration and evaluation of new features, models, and feed layouts. **4. Cold Start Problem:** Addressing cold start is critical for new user adoption and retention. For **new users**, we'd guide them through an onboarding flow to follow interests or contacts. Their initial feed could be populated with popular, trending content or broad interest categories. **Collaborative filtering** (recommending popular items among similar users) and **content-based filtering** (matching initial interests to content) would be leveraged. For **sparse activity users**, we would balance showing their limited follows with trending content and content from categories they've shown minimal engagement in. Graph-based recommendations (from friends' activities) and periodic injection of diverse content would encourage exploration and gather more interaction signals. **5. Caching and Performance Optimization:** To handle billions of events and minimize latency, extensive caching is essential. A **CDN** would serve static assets (images, videos). A dedicated **feed cache** (e.g., Redis cluster) would store fully or partially rendered feeds for immediate retrieval, significantly reducing the load on downstream services. This cache is invalidated on new posts or significant profile changes. **Service-level caching** would store results from upstream services (e.g., user profiles, content metadata). **Client-side caching** on mobile apps and web clients would further improve perceived performance. Additionally, **asynchronous processing** via message queues (Kafka) for fan-out and other non-critical operations decouples services, enhancing system resilience and responsiveness. Ultimately, a scalable and personalized news feed relies on a resilient, fault-tolerant infrastructure, intelligent data partitioning, and continuous optimization. Robust monitoring, A/B testing, and an iterative development cycle are critical to adapt to user behavior changes and maintain high performance and relevance.

Design a Social News Feed

Ready to prove your AWS ML knowledge?

Design a URL Shortener (TinyURL)

Design an E-commerce Checkout & Cart

Handling Distributed Cache Invalidation

Caching Strategy Deep Dive

Design a Global Distributed Rate Limiter

Edge Computing Meets Edge Caching: Building Real-Time Applications at the Network Edge

Edge Compute with CDNs: Caching, Workers, and Safe Gradual Rollouts

Aurora vs RDS PostgreSQL vs EC2: Costs, Performance & Multi-Region Compared (2025)